Testooj user`s manual
Contents
1. 2 Add eal Remove With the value 0 must throw java langulllegalArgumentException M Open file Save and exit Exit no saving Number of values for random testing 0 _ Use values as described Figure 21 With the specification made in Figure 21 all test cases with a value zero will be prepared to catch the corresponding exceptions The file or files with the test cases 1s written on the results folder of the working directory in our example this one was c paper once we close the window of Figure 20 Depending on the number of test cases this task can be immediate or to take a few seconds Each time we generate test cases testooj builds a folder and an additional file under the temp folder of its working directory directory Next figure shows the two ses sions created in the two examples we have developed in these pages the first one folder paper Triangulo_20 10 2007_17 0 and file paper Triangulo_20 10 2007_17 0 session corresponds to a test case generation session of the paper Triangulo class executed on October 2007 20 at 17 hours The second one contains the same data but it was performed at 17 15 45 x Nombre paper A paper Triangulo 20 10 2007 _17 0 g oo results B paper Triangulo_20 10 2007_17 15 mn constraints E paper Triangula_20 10 2007_17 0 session fn paper Triangulo paper Trianguloa 20 10 2007 17 15 session fm creators O required 9 results g2. Additional options for test case execution Additional test files Class path Split JUnit files every aooaa bytes Split Mujava files every 200000 bytes Figure 1 Set up window During its execution both the user and testooj create files and saves results in several folders These ones will be saved in several folders under the Working path field Moreover for some functionalities testooj may require the location of the MuJava or JUnit tools whose paths can be established in the corresponding fields For the test case execution functionality testooj may require the location of more test files besides those of MuJava what can be established in the Additional test files field The class path required generating or executing test cases for classes with dependencies is estab lished in the Class path field For concluding the size of the files with test cases that testooj generates in MuJava or JUnit formats can be limited with the two remaining fields In our example we have the MuJava tool at c mujava06 and the JUnit tool at C eclipse plugins org junit4_4 3 1 Therefore we ll leave the working directory at c paper results the default location proposed by testooj and the MuJava and JUnit paths pointing to their corresponding locations Testooj3 setup Working path Cfpapernresults JUnit location CJeclipsepluginsiorg junit4_ 4 3 1 unitjar MuJava root CJmujavatg Additional options for test c
3. setltint l i EdiestTs_3 arene Build JUnit file Fy sapertianguled Build MuJava file setkint setkint setkin Build JUnit amp MuJava L getTipag S testTS_ 4 _ Build states file 7 al Ly paper Triangulo Pass selection to regular expression Build file 39 test templates generated in 0 016 seconds Figure 13 Test templates Test templates proceed from the expansion of the regular expression wirtten Before generating test cases the tool user must assign test values to the parame ters of the operations involved in the regular expression This is made selecting File gt JUnit assertions in the menu bar The following screen is shown JUnit assertions paper Triangulo E Constructors B paper Triangulog EJ Methods B getTipo CJ setlfint C fri o CJ setJi nt CJ setk nt With the value must throw Open file Number of values for random testing la _ Use values as described Figure 14 For now it is important to know that this window is used to assign values to the parameters of the operations Let us suppose we want to assign the values 0 1 2 3 and 5 to the parameter of the three set operations of Triangulo Then we use the tree on the 11 left side to locate the parameter of setl click on it write the first value 0 in the Test value text box and press the Add button We repeat this procedure with the remaining values and the remaining parameters B JUnit as
4. Testooj user s manual October 25 2007 A testing tool developed by Alarcos Research Group Department of Information Systems and Technologies University of Castilla La Mancha Project leader Macario Polo Usaola macario polo at uclm es 1 Introduction testooj is a tool developed in Java for testing Java programs It allows to main functionalities 1 Generation of test cases based on regular expressions 2 Execution of test cases to perform different types of result analysis testooj is a useful tool easy to use The test case generation functionality is ap propriated both for practitioners and for researchers In our opinion the second one is more research oriented 2 Launching testooj testooj can be launched 1 double clicking the testooj3Eclipse jar file 2 Opening a console and executing the command java jar testooj3Eclipse jar The first time is executed testooj shows the setup window shown in Figure 1 In our example the testooj3Eclipse jar is saved in the c paper folder of a Microsoft Win dows machine The user must fulfil the window with some paths required for its normal execution In the example of the figure testooj proposes a set of default values for the configuration which are calculated from its home directory this 1s from the directory where testooj3jEclipse jar is saved c paper B Testooj3 setup Working path CJpaperresults JUnit location Cpapenlib unit jar Clpapeniibimujayal
5. watching the test case called test_TS_1S_24 this is the test case number 24 of the 18 test template remind that test templates can be seen in the main screen of testooj Note 14 that the instance of the class under test in the code of the test cases 1s referred with the word obtained If you observe the code of the Triangulo class Figure 6 you ll see that all the setter operations throw and IllegalArgumentException when the value of the pa rameter is less or equal than zero Thus the user could instrument the test case shown in Figure 19 to throw that exception since its right behaviour is to throw it This can be made selecting the Must fail with check box note that testooj surrounds the test case with a try catch block and adds two JUnit s fail instructions one at the end of the try and another one in a generic catch block If the test case throws the expected exception it is catched 1n the catch java lang IllegalArgumentException ex and for this test case JUnit would show the green bar JUnit test cases testTS_18_20 a testTS_18_21 testTS_18_24 testTS_18_22 testTS_18_23 _ Remove method v Must fail with java lang lllegalArgumentException testTS_18_24 z testTS 18 25 public void testT5_168 24 throws Exception CE testTS_18_26 eA paper Triangulo obtained null testTS_ 18 27 testTS 18 28 obtained new paper Triangulo testTS 18 29 testTS_18_30 testTS_18 31 obtained setI argl testTS_18_
6. amount public void transfer double amount String targetAccount this withdraw amount Account target new Account targetAccount target deposit amount public void transfer double amount Account targetAccount this withdraw amount targetAccount deposit amount public String toString return mNumber mBalance public double getBalance return mBalance Figure 24 Source code of a banking Account Let us generate some test cases to check the right behaviour of this class Thus for example after executing deposit with a positive amount the balance of the instance must be increased in the value of the amount parameter If the parameter is zero or negative the instance should throw an exception For this we use the Precode and Post code areas of the screen shown in Figure 15 The deposit operation can be annotated as follows e Before executing deposit we ll save the account balance in a variable called for example balancePre e After executing it we ll check with a JUnit oracle that the account bal ance is the previous balance saved in balancePre plus the amount de posited Figure 25 illustrates this e Inthe Precode area we write the Java sentences we want execute before calling the operation in this case the declaration of the balancePre vari 18 able and the assignment of a value Note that we use the word obtained to reference the instance of the class under
7. j AQIS_106 fee eo Ee ico Hh its LAr aT Antic i ri exi meri ni MET its a u iuiuiuiIiiiiiIiIIIIIIIIIIIIIIIIIIIIIIIIiIiIIIIIIIIImmIImmHN il Figure 54 Now click on the original version and press the Set as original button Now press the Mix button and after a few seconds the text area in the bottom part of the window will show a message You will also note that the Mix button is released If you go now to the folder pointed by the Directory of target versions you Il see the following B miva O B 2 Trianqula java classes 3 mix paper Triangulo O 1 40R6_5 L01_53 om O 2 ROR_4 4015 126 G 3 40RB_10 L01_35 D 4 LOI_22 ROR_32 G S 40RB7 4015_69 6 4015_166 4015_1 G 7 ROR_24 ROR_40 G 3 4075_136 40RB_ D 9 4015_57 4015_99 O 10 4015_70 40RB_1 Figure 55 As you see testooj has created a folder called as the class under test and under this folder a folder containing a 2 order mutant The highlighted 2 order mutant of 35 Figure 55 corresponds to the mutant proceeding from the mixing of the 1 order mutants AORB 6 and LOL 53 Now you need to compile these mutants For this press the Compile button tes tooj compiles them one by one showing a progress bar during this process Usually it takes only a few seconds Mutant mixer Full CUT name including package paper Triangulo java Directory of source versions Cimujavadbresult Directory of target versions CUmujavaleimix CLASSP
8. option File gt Object crea tion in its menu bar which shows the following window Manual creation of objects Class path CADocuments and Settings iMacoiMis documentosicodigoExperimentos _ Consider inherited operations Select class lpaper Account Load members paper Account import java io FileOutputstream amp c Constructors import java io ObjectOutputStream CI Methods public class CreateAccount public static void create try Account o new Account i f f TODO insert here the construction code FileOutputStream f new FileOutputStream C paper results serialized0bj cts paper Account ser ObjectOutputStream os new ObjectOutputsStream f os write bject io f close catch Exception ex System err printlin ex toString public static void main String args CreateAccount create aCe Save Compile Figure 32 Deprecated window to create and serialize instances In this window the user writes the class path the name of the class under test and when the Load members button is pressed testooj fulfils the text area with a piece of code ready to create and serialize an instance In our opinion the recommended method in the previous subsection is more comfortable and easy than this one more over this deprecated method requires that of the complex parameter to be serializable Anyway if this method i
9. paper Triangulofraditional_mutants void_setK int ROR_13 A Sse C mujava06 result paper Trianguloftraditional_mutants void_setK int ROR_14 L gt C mujava06 result paper Triangulotraditional_mutants void_setK int ROR_15 hd Result analysis Increasing Decreasing Random Time comparison ltestTS_1 testTS 1 testTS_1 testTS 1 testTS_1 testTS 1 testTS_1 testTS_1 testT _1 testTS_1 testTS_1 testTS 1 testTS_1 testTS_1 testTS_1 testTS_1 testTS_1 C imuijay Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok C imuijay Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok C imujay Ok Ok Ok Ok Ok ok Ok Ok Ok Ok ok Ok Ok Ok Ok C imujay Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok C imujay Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok Cmujav Ok Ok Ok Ok Ok Ok Ok Ok Ook Ok Ok Ok Ok Ok Ok C imujay Ok ok Ok Ok Ok ok Ok Ok Ok Ok Ok Ok Ok Ok Ok C imujav Ok ok Ok ok Ok ok Ok Ok Ok Ok ok Ok Ok Ok Ok C imujav Ok Ok Ok ok Ok Ok Ok ok Ok Ok ok Ok ok Ok ok C imujav Ok Ok Ok Ok Ok Ok Ok Ok Ok Ok ok Ok Ok Ok Ok C imujav Ok Ok Ok Ok Ok Ok Ok ok Ok Ok ok Ok Ok Ok ok C imujav Ok ok Ok ok Ok Ok Ok Ok Ok Ok Ok Ok ok Ok ok C mujav Ok ok Ok ok Ok ok Ok ok Ok Ok Ok Ok ok Ok ok Ci
10. test e Inthe Postcode area we write the Java sentences that must be executed after calling the operation in this example the JUnit oracle Note we use again the obtained word To reference the 1 argument of the operation arg is used if there are more arguments we would use arg2 arg3 etc If we want to reference the result returned by the operation we could use the word result JUnit assertions for deposit double CI paper Account Precode C Constructors double balancePre obtained getBalance CE Methods gt deposit double o CJ withdraw double 4 getBalanced toString C transferfdouble paper Account C transferfdouble java lang String Postcode assertTrue obtained getBalance balancePretargl1 Test value Generate With the value must throw od Open file mee aiian Number of values for random testing C Use values as described Figure 25 Writing the oracle Moreover we can assign the values 0 and 100 to the parameter of the deposit operation Also we can specify that deposit must throw and exception when 0 is used Pod With the value 0 must throw java lang lllegalArgumentException m Number of values for random testing b Use values as described Figure 26 In the same way we can assign test values to the remaining operations For ex ample let us assign the st
11. 13 test cases of 125 testTS_1_38 testT5_1_98 testTS_1 95 testTS_1 99 testTS_1 82 testTS_1 32 testTS 1 37 Re analysis excluding alive Save times table Build stub file Figure 49 Result analysis window The table in the top side shows what mutants are killed by what test cases The table can be saved in CSV or HTML formats with the Save as button The Save time s table produces a similar table but now it also contains the time in milliseconds de voted to execute each test case In the bottom half of Figure 49 testooj shows a report with the best test cases this is a reduced suite of test cases that kill the same mutants than the original suite a brief summary about the percentage of mutants killed and the list of alive mutants Close to the Result analysis button there are three radio buttons for selecting the algorithm which will calculate the best test cases in general Decreasing produces the most reduced suite Increasing Decreasing Random Wh Plocel Goal cs flies sal ole e flene Mee ecalllee e e e local lool calle bose Figure 50 If you want to remake a result analysis you do not need to reexecute the test cases testooj uses the contents of a folder saved under its temp folder whose name coincides with the test file class 32 g paper A E C mujava0 result paper Tri gt results p E c mujavad result paper T
12. 32 testTS_18_33 testTS_18_34 obtained setd arg2 testTS_18_35 testTS_18_36 int arg3 int 1l testTS_18_37 obtained setK arg3 testTS 18 38 festTS_18_39 ii result3 obtained getTipo itestTS_18_40 i itestTS 18 41 fail testTS_168 24 java lang IllegalArgqumentException expected itestTS 18 42 testTS 18 43 testTS_18_44 catch java lang IllegalarqumentException ex testTS_18_45 testTS 18 46 catch Exception ex int argl f int 5 int arg2 int 0 testTS_18_47 Eail testTS_16 24 java lang Illegal ryumentException expected testTS_18_48 testTS_18_49 testTS_18_50 _ Compile file when leaving Figure 20 The window in Figure 15 can be also used to instrument testooj for surrounding with try catch bocks many test cases For example to say that the right behaviour of Triangulo is the throwing of an IllegalArgumentException with the value zero we can specify that with the value 0 that exception must be thrown see the bottom right side of next figure testooj lists all the exceptions declared in the signature of the method which is selected in the left side tree 15 JUnit assertions for setK int Ci paper Triangulo Precode C Constructors C Methods a getTipodg CE setlfint int CE setJdint CS int CE setkK int CS int Postcode Test value Generate J
13. ATH Source versions Set as original Ciimujaval result paper Triangulo original Ciimujaval result paper Triangulotraditional mutantsint_getTipo j AOM_ 100 Cimujavat result paper Triangulotraditional_mutants int_getTipo AQIS_101 Compile Cimujaval result paper Triangulofraditional_ mutants int_getTipo j AOIS_102 CUmujaval result paper Triangulofraditional_mutants int_getTipo AQS_103 C imujaval result paper Triangulofraditional mutantsint_getTipo j AOM_ 104 Ciimujava0 result paper TriangulojAraditional_mutants int_getTipo AOIS_105 Cimujaval Tesult paper Triangulofraditional_ mutants int_getTipo j AQIS_106 ia fee es ES fer ee a ies z iarr T irtis mu riante int annoi Tin iFa n E Compiling C muyavaliimiopaper Triangulo l 09 AQIS 148 LOL 60 13 of 156 Compare Figure 56 Note the difference with Figure 55 where the mutants have been compiled T zJ f bal 3 mujavat F Triangulo java 9 classes E Trianquio class O mix paper Triangulo E 1 40RB 5 L01_53 me paper C ROR 4 4075 126 O 3 40RB_10 L01_35 C 4 L01_2 ROR_3 CO 5 40RB 7 4015 69 O 6 4015 166 4015 1 C 7 ROR_24 R0R_40 C 6 4015 136 40RB C 9 4015 57 4015 99 O 10 4015 70 40RB_1 Figure 57 Now you are ready to execute test cases against 2 order mutants Simply in the test case executor window select the mix folder instead of the result folder 36 B Test case executor File
14. The value of this string is a call toe the toString method of the class under test In the example of the Triangulo class note that it has no toString method and thus it uses the inherited version from the Object class Therefore e If the class under test has a toString method the test cases generated will probably work correctly e If the class under test has not a toString method you should rewrite the return instruction of MuJava test cases such as in the following exam ple left side shows a test case generated by testooj in the test case of the right hand side the call to toString has been substituted by a string repre sentation of the state of the class under test 24 public String testTS_1_1Q0 public String testTS_1_10 try try paper Triangulo obtained null paper Triangulo obtained null obtained new paper Triangulo obtained new paper Triangulo int arg1 int 0 int arg1 int 0 obtained setI arg1 obtained setI arg1 int arg2 int 0 int arg2 int 0 obtained setJ arg2 obtained setJ arg2 int arg3 int 0 int arg3 int 0 obtained setK arg3 obtained setK arg3 int result3 obtained getTipo int result3 obtained getTipo return obtained toString return obtained i obtained j obtained k obtained tipo catch Exception ex return ex toString catch Exception ex return ex toString Figure 34 4 Test case execution This s
15. WN order mutation Test file MBisectOK class C imujava0 mix paper Trianguilo 1 AQRB 5 LOL 53 Cimujaval mix paper Trianguio 10 A0I O AQRB_ 1 Ciimujava06 mix paper Triangulo 100 LOL 65 4018 56 Cumujaval mix paper Trianguio 101 A018 63 4018 159 Climujaval mix paper Trianguio 1027 ROR_22 A018_ 45 Figure 58 5 2 States definition Also for research purposes we have added to testooj a new functionality that al lows to describe with state machines the desired behaviour of class instances Let us suppose you want to represent with a state machine the behaviour of accounts To be continued 37
16. a0 testset 4 Result analysis J Increasing Decreasing Y Random Time comparison a Figure 36 Main screen of the Test case executor In the test file combo box testooj shows the list of class files contained both in the testset folder of MuJava and in the Additional test files path Figure 1 For this ex ample we will select a file called MujavaTriangulo_I class which has been generated with testooj for the Triangulo class using the regular expression ACBDE and the test values 0 1 2 3 and 5 In the Full CUT name including package field we must write the complete name of the class under test in this example paper Triangulo The Versions directory field contains the path to the folder where the different mutants or versions of the class under test can be located In this example this path points to the result folder of the Mu Java root directory Now we can press the Search button to make that testooj looks for the mutants of paper Triangulo As we saw in Figure 35 MuJava correctly generated the mutants however testoo is not capable of finding them because it requires that the mutants have a folder structure similar to its package structure this is 1f the complete name of the class under test is edu uclm esi Foo the mutants must be in a directory fulfilling the structure edu uclm esi Foo testooj can create this structure using the Fil
17. aditional_mutants nt_getTipo AOIS_106 Result analysis Increasing Decreasing Random Time comparison Figure 41 When the Execute selected button is pressed the following two warnings may appear Seleccionar una opcion Warning paper Triangulo class already exists inthe classpath C Documents and Settings Maco Mis documentos codigoExperimentos This can lead to no killing mutants Continue CE ctw encor Figure 42 Seleccionar una opci n Warning MujavaTriangulo_ 1 class exists several times inthe classpath This can lead to unexpected results C Documents and Settings Maco Mis documentos codigoExperimentos Ciimujavalbtestset CE te em Figure 43 In general there is no problem with these advertences However if you see that you don t kill any mutant or that you kill all the mutants maybe you should take the advertences into account 29 During the execution of test cases against the program the window shows a pro gress bar and some messages e In the area pointed by the arrow testooj reports about the current action it is executing e In the are below testooj reports about the last error occurred during the execution of test cases In general these errors are not important since they may mean that the fault introduced in the mutant causes for exam ple an infinite loop in this case testooj reports of dead threat Th
18. ase execution Additional test files Class path Split JUnit files every 6000 bytes Split Mujava files every 00000 bytes Figure 2 If we press the buttons Working path JUnit location etc we can navigate through our system to select the required paths Once the configuration has been established we press the Save and exit button Then testooj creates the following folder structure in the location established in the Working path field ao E results constraints creators O required O results 9 serializedObjects temp 3 templates Figure 3 The goal of these folders will be explained in the rest of this manual For now note that testooj shows the following welcome screen Testooj A Java testing tool for Java JUnit MuJava test case generator Test case executor Figure 4 From Figure 4 we can go to the two main functionalities of testooj test case generation and test case execution 3 Test case generation This functionality is launched by pressing the left button of Figure 4 Then tes too shows the next window JUnit and Mujava test case generator c Alarcos Group UCLM File Externaltools Test case mutation Algorithms Compatibility States Fault pronenness Class path _ Consider inherited operations B Class not selected bode Test templates Max length Mo templates Get test templates Build JUnit file O Build MuJava file Build JUnit amp MuJ
19. ava Build states file Pass selection to regular expression Build file JUnit for mutation Figure 5 Let us see two important fields in this window e Class path testooj shows here the class path established in the setup window Figure 2 testooj uses this field for looking the class under test CUT If the CUT requires additional libraries or folders this 1s the place where these values must be established e Select class with this button we can navigate through the classes of our class path to find the class under test Alternatively we can write the full name of the class under test in the corresponding text box Let us suppose we want to generate test cases for the following class which represents the famous triangle type problem of Bertrand Myers package paper public class Triangulo public int 1 J K public int tipo public static final int ESCALENO 1 ISOSCELES 2 EQUILATERO 3 NO_TRIANGULO 0 public Triangulo public void setI int v throws legalArgumentException if v lt 0 throw new IllegalArgumentException 1 V public void setJ int v throws IllegalArgumentException if v lt 0 throw new IllegalArgumentException j v public void setK int v throws IllegalArgumentException if v lt 0 throw new IllegalArgumentException k v x return if scalene 2 if isosceles 3 if equilateral O if not a triangle public int getTipo if G
20. codigoExperimentos Figure 40 Now test cases can be executed against the original class and against the mu tants e For executing test cases against just a mutant double click on that mu tant e For executing test cases against a set of mutants make a multiple selec tion and press the Execute selected button in Figure 41 the test cases in 28 cluded in the MujavaTriangulo_l class test file will be executed against the six files highlighted in the list box Test case executor File N order mutation Test file MujavaTriangulo_1 class 000 milis O of original Full CUT name including package paper Triangulo class C Remote Versions directory Cmujava0S result i Search CLASSPATH Locations of the CUT versions Add Remove Set as original L Execute selected CADocuments and Settings MacoiMis documentosicodigoExperimentos C C mujava06 result paper Triangulo original C mujava06 result paper Triangulo traditional_mutants nt_getTipo AOIS_100 C mujava06 result paper Triangulofraditional_mutants int_getTipo AOIS_101 C mujava06 result paper Triangulofraditional_mutants int_getTipo AOIS_102 C mujava06 resutt paper Triangulofraditional_mutants nt_getTipo AOIS_103 mujava06 result paper Trianguloftraditional_mutants nt_getTipo AOIS_104 C mujava06 result paper TrianguloAraditional_mutants nt_getTipo AOIS_105 C mujava06 result paper TrianguloAr
21. e DOrganize versions folder option in the menu bar which looks as in Figure 37 26 CUT versions arrangement Versions directory ic Jmujavalblresult Organize folders Full CUT name including package lpaper Triangulo Figure 37 In the window of Figure 37 we press the Organize folders button and then tes tooj builds the adequate folder structure for each mutant The result of this organization is Shown in Figure 38 compare it to Figure 35 to note that under the folder correspond ing to each mutant testooj has created a paper folder and inside it it has put the source and the compiled code of the corresponding mutant E mujawal A E Triangulo class classes P 4 Triangulo java mi E result paper Triangulo class_mutants I original E traditional mutants E int_getTipot ADIS 25 paper E ATS 26 I paper El 0 acrs_27 I paper AOTS 26 I paper ACTS 29 Figure 38 Folder organization required by testooj Now testooj can find the mutants by pressing the Search button The result is shown in Figure 39 27 Test case executor File N order mutation Test file MujavaTriangulo_1 class v Full CUT name including package paper Triangulo class Versions directory cumujavad iresult Search CLASSPATH Locations of the CUT versions Add Remove Set as original Execute selected Culmujavalbitestset C imujava06 resu
22. ection explains how to use the Test case execution functionality included in testooj Therefore we do not explain here how to use the JUnit tool The functionality of test case execution in testooj uses test cases in MuJava for mat These test cases can be executed against a set of mutants generated with MuJava and then it is possible to do different types of analysis 4 1 Mutant preparation testooj is not a mutant generation tool For this you must use the MuJava tool To know more about MuJava please check the web site of Prof A J Offut at XXX In this example we will use testooj to do mutation testing on the Triangulo class We ll suppose we already have a set of mutants of Triangulo on the correspond ing MuJava folder in our example this location is at c mujava06 result E mujavan A E Triangulo class l fm classes Triangulo java O mix E result E paper Triangulo class mukants 9 original E 2 traditional mutants 5 int_getTipor 7 ADIS 26 9 4015_27 9 4015 25 Figure 35 Some of the mutants generated by MuJava ZS Now we press the Test case executor button of the welcome screen of testooj Figure 4 and the following screen appears Test case executor File N order mutation Test file MBisectOK class Full CUT name including package Search CLASSPATH Locations of the CUT versions Add Remove Setasoriginal Execute selected C imujav
23. ents and SettingsiMacoimMis docurmentosicadigoExperimentas _ Consider inherited operations Regular expression Constructors oT Cy paper Triangulod Max length a Mo templates EA Methods 4 qetTipot Get test templates setiint Build JUnit file setJ nt C Build MuJava file C Build JUnit amp MuJdava Build states file Pass selection to regular expression Build file Figure 10 setktint Note if the CUT has static methods that return an object of the CUT testooj considers these operations as factory methods and adds them to the Constructors node In this moment we can already write a regular expression for generating test cases for the class under test Let us suppose that we can generate test cases that create an instance of Triangulo assign values to 1 2 or the 3 sides in any order calculate the triangle type Test cases with this structure can be generated with the following regular expression Triangulo setl setJ setK getTipo Then what we must write in the Regular expression field is a expression match ing with this structure testooj considers the first operation in the tree as the operation called A the second operation is B the third one is C testooj is capable of manipulat ing classes with an arbitrary number of operations however in a test session it only can deal with a maximum of 50 operations Thus if the class under test has more than 50 operations the fol
24. expected _ Compile file when leaving Figure 30 Obviously the transfer operation can be annotated with an oracle that considers both the instance of account that sends the transference as well as the instance of the account that receives it This is shown in Figure 31 note that in the precode and post code areas we reference the target account with the arg2 word JUnit assertions for transfer double paper Account paperAccaunt Precode c Constructors double sourceBalance Pre obtained qetBalance EJ Methods amp CJ depositidouble o amp EJ withdrawidouble B qetbalancery toString s o CJ transferfdouble paper Account c transferfdouble java lang String double targetBalancePre argz qetBalancel Postcode asserctTrue obtained getBalance sourceBalancePre argl assert True larg getEalance i targetBalancePretargql With the value must throw Open file Number of values for random testing _ Use values as described Figure 31 22 3 2 2 Deprecated method In the first versions of testooj the way of dealing with non primitive data types was more complex Supposing the operation public void transfer double amount Ac count targetAccount the user required to create so many instances of the parameter as test values he she desired to use Then these instances should be serialized and saved in the serialized folder of testooj For this testooj includes the
25. ified in the configuration window Figure 1 In this exam ple since the number of test templates is 39 see the bottom side of Figure 13 we have fixed 5 values for each parameter and have selected the All combinations algorithm Figure 17 testooj generates a huge number of test cases 3615 When the test cases are generated they are shown in the next screen JUnit test cases testTS_18_20 a testTS_18_21 testTS_18_24 testTS_18_22 testTS_18_23 _ Remove method Must fail with java lang lllegalArgumentException estTS_18 24 testTS_18_25 public void testTS_18_24 throws Exception it stTS 18 26 paper Triangulo obtained null e a itestTS_18 27 obtained new paper Triangqulo itestTS_18_ 28 itestTS_18_ 29 int argl int 5 testTS_18_30 testTS 18 31 testTS_18_32 testTS_18_33 testTS_18_34 testTS_18_35 testTS_18_36 testTS_18_37 testTS_18_38 testTS_18_39 testTS_18_40 testTS 1841 testTS_18_42 testTS_18_43 testTS_18_44 testTS_18_45 testTS_18_46 testTS_18_47 testTS_18_48 testTS_18_49 testTS_18_50 testTS_18_51 testTS_18_52 an A A ES obtained setI argl int arg2 int 0 obtained setJ arg2 int arg3 int 1l obtained setK arg3 int result3 obtained getTipo C Compile file when leaving lt lt Figure 19 Test cases generated Each test case can be seen from the previous screen In this example the user is
26. imix CLASSPATH Versions Set as original Compile Compare Figure 53 Mutant mixer window Let us suppose you want to build 2 order mutants for the paper Triangulo class fill in the name of the class in the Full cut name including package field fill in the class path af the class path is not established in the Set up window of Figure 1 and if the class under test requires additional libraries or directories to be compiled and press the Search button corresponding to the Directory of source versions The Versions list wid get shows the list of 1 order mutants found in the source versions directory 34 Mutant mixer A lB Ed Full CUT name including package paper Triangulo java Directory of Source versions Coimujavadbresult Directory of target versions CUmujavaleimix CLASSPATH Source versions Set as original Ciimujaval result paper Triangulo original Ciimujaval result paper Triangulotraditional mutantsint_getTipo j AOMS_ 100 Cimujavat result paper Triangulotraditional_mutants int_getTipo AOMS_101 Compile Cimujaval result paper Triangulofraditional_ mutants int_getTipo j AOIS_102 CUmujaval result paper Triangulotraditional_mutants int_getTipo AOS_103 C imujaval result paper Triangulotraditional_ mutantsint_getTipo j AOIS_ 104 Cimujaval result paper Triangulojtraditional_mutants int_getTipo AOIS_105 Compare Ci imujaval Tesult paper Triangulojraditional_ mutants int_getTipo
27. imuiav Ok Ok Ok Ok lok Ok Ok Ok lok Ok Ok Ok Ok Ok Ok Ok Figure 44 Test case are executing While test cases are executing testooj fulfils the table in the bottom side of Figure 44 each cell specifies the result of executing the test case in the column against the mutant included in the row in Figure 45 we see that the test case testTS_ _ has been successfully executed against the AOIS_101 mutant 30 tesiTS_1 te Loimujavalbfresultpaper Trianguiaforiginall Ok Gp Cymujaval6resultipaper Triangulotraditional_mutantsint_getTipow AOIS 100 Ok JOH Cymujaval6resultipaper Triangulotraditional_mutantsiint_getTipow AOS 1017 Ok JO Cymujaval6resultipaper Triangulotraditional_mutantsint_getTipow AOIS 102 Ok JOH Cymujaval6resultipaper Triangulotraditional_mutantsint_getTipow AOIS 103 Ok JOH C amuiava lA resutnaner Triangulatrarditional mutantstint aetlinnw AgS may ok Cb Figure 45 As we have mentioned some paragraphs ago some mutants may cause infinite loops and therefore when a test case reaches it the execution of the test case would never finish To avoid this and to recognize these mutants as killed mutants testooj can be instrumented to consider a mutant is killed after a certain number of milliseconds Figure 46 although there are several ways of setting up the time out we recommend to use only the first one with no remoting The
28. inherited operations Figure 7 Now we can either to write the complete name this is including package or subpackages in the Select class field or to press the Select class button to find the compiled code of the class under test If we press the button the following window ap pears with it we can locate the paper Triangulo class then we press the Select button in Figure 8 and the name of the class fulfils the corresponding text box Figure 9 Class path explorer Class path EJ C Documents and Settings MacofMis documentos icallectionsiArrayList eclass 4 IrollectionsiWvectorm class 4 MBisectOk class MBisectOKR class 4 MBubCorectoOk class 4 MBubCorrectoOKR class MFindOk class MFindOKR class amp MFourballsOk class 4 MF ourballsOKR class Figure 8 JUnit and Mujava test case generator c Alarcos Group UCLM File Externaltools Test case mutation Algorithms Compatibility States Fault pronenness Class path C Documents and SettingsiMacoimMis dacumentosicadigaExperimentas _ Consider inherited operations Figure 9 Now we can load the members of the class under test by pressing the Load members button The list of public constructors and operators of the class under test appear on the left side tree JUnit and Mujava test case generator c Alarcos Group UCLM File Externaltools Test case mutation Algorithms Compatibility States Fault pronenness Class path C Docum
29. lowing window is shown and the user must remove those operations which will not be used in the regular expression Class with more than 50 operations paper ManyMethods n di I w K al z aa ab Figure 11 A class with more than 50 operations Continuing with our example of the Triangulo class we want now to write a tes tooj regular expression corresponding to Triangulo setl set setK getTipo We can 1 To write directly the desired regular expression which is A CIDIE B 2 To select the corresponding operations on the tree and make double click on them 3 To select the corresponding operations on the tree and press the button Pass selection to regular expression paper Triangulo Regular expression Gq Constructors A C DIE B paper Triangulod Max length EJ Methods L getTipaa Get test templates A mea Build JUnit file s ae C Build MuJava file setkiin Build JUnit amp MuJava Figure 12 10 If we press now the Get test templates button testooj shows in the right side tree a set of templates that later can be used to generate test cases Note that in the text are below testooj says the number of test templates generated Regular expression paper Triangulo na 4 Constructors AICIDIE 6 paper Triangulo Max length est terrniplates gt a Wethods o i Cc testTs_1 LS getTipog Get test templates 0 EjtestTs_2
30. lt paper Triangulo original C mujava06 result paper TrianguloAraditional_mutants nt_getTipo AOIS_100 4 C mujava06 result paper Triangulo raditional_mutants nt_getTipo AOIS_101 C mujava06 result paper TrianguloAraditional_mutants nt_getTipo AOIS_102 C mujava06 result paper Triangulo raditional_mutants nt_getTipo AOIS_103 C mujava06 result paper TrianguloAraditional_mutants nt_getTipo AOIS_104 C mujava06 result paper TrianguloAraditional_mutants nt_getTipo AOIS_105 C mujava06 result paper TrianguloAraditional_mutants nt_getTipo AOIS_106 LA aw LAMAR A Result analysis Increasing Decreasing Random Time comparison Figure 39 testooj loads the mutants from the Versions directory field 4 2 Execution of test cases Note the preparation we have made in the previous epigraph and note the differ ent values we have in Figure 39 here we can also establish the class path variable which will be required if the class under test needs additional libraries or directories In the example of Figure 39 the class path 1s pointing to the testset folder of MuJava Ad ditionally we must add the place where the compiled code of the class under test is lo cated which is C Documents and Settings Maco Mis documentos codigoExperimentos Thus we add this path to the class path field CLASSPATH CDoacuments and SettingsiMacoimis documentosi
31. mportant note that the check box labelled Use values as described has been checked JUnit assertions for transfer double paper Account paperAccaunt c Constructors EJ Methods CJ deposit double amp CJ withdrawidouble D qetbalancer toString EJ transferfdouble paper Account B double B paper Account cJ transferfdouble java lang String new Account deposit 1000 With the value must throw i Open file Number of values for random testing lo Use values as described Figure 29 Note To assign values to non primitive parameters always check the Use values as described check box For this example testooj generates two test cases One of them is shown in Figure 30 21 A JUnit test cases testTS_1_1 _ Remove method Must fail with java lang IllegalArgumentException M public void testTS_11 throws Exception try paper Account obtained null java lang String argl java lang String 12345 obtained new paper Account argl double arg2 double 0 double balancePre obtained getBalance obtained deposit arg2 assertTrue obtained getBalance balancePre arg2 double arg3 double 100 paper Account arg4 new Account deposit 1000 obtained transfer arg3 arg4 rere i java lang IllegalArqumentException expected catch java lang IllegalArgumentException ex catch Exception ex fail testTS_1_1 java lang IllegalArgqumentException
32. other options have not been tested enough Timeout ia 3000 milis OO o Of original i None Remote fr ffir a Tee Figure 46 Setting up the timeout 4 3 Result analysis During the execution of test cases testooj generates a set of files under its temp folder which can be later utilized to analyze the results For this press the Result analy sis button 1f nothing happens testooj reports us about the necessity of selecting the original version of the class under test Figure 47 E Select the original CLT Figure 47 Therefore click on the original version and press the Set as original button as shown in Figure 48 Set as original dlo original Hotraditional mutantsjint_getTipo V AOIs 100 Hlotraditional mutantsjnt_getTipo j AQMh 101 Figure 48 31 Now we can press the Result analysis button which shows the following screen FB Result analyzer a re 7h 1 4 i Faja a a a a a b bd bd bd bd dd E bd baba bd aaa Bab abd bab abd bd dD A H JI SEP IK RRL RR SESE SK F SK gt FK 28 eee E J PA K R PR AR K K aa Wir jaf Summary of results Minimal set of test cases
33. ri 3 constraints E c mujavad result paper Tri fm creators E C mujavade result paper Tri mn required E C mujaval result paper Tri fn results E C mujaval result paper Tri fm serializedObjects E C mujavade result paper Tri 3 temp E c mujavad result paper Tri MujavaTriangula_ 1 E C mujavat result paper Tri fm templates E C mujaval result paper Tri m E l EA iava A resulk naner Tri Figure 51 Thus simply select the test class in this example MujavaTriangulo_l class in the list box write the name of the class under test and press the search button Then you can make the result analysis 5 Other features 5 1 Mutant mixing Due to a research work we have started and which will be published in a few months we hope testooj is capable of mixing MuJava mutants this is it can take two mutants generated by the MuJava tool and mixing them in just a mutant with two faults These mutants are called 2 order mutants To do this we need to create a folder called mix under the MuJava directory mujavans classes mix C result C src testset Figure 52 Then from the Test executor window select N order mutation gt Open mix win dow in the menu bar The following window appears 33 Mutant mixer E 5 Ed Full CUT name including package Directory of source versions Cimujavadbresult Directory of target versions CUmujavale
34. ring 12345 to the parameter of the second constructor 19 HaperAccount Precode EJ Constructors Ei paper Accountijava lang String double i java lang String i double Cd paperAccountiava lang String A java lang String amp EJ Methods Postcode Test value 12345 1 Figure 27 Let us return to the main screen If we write the regular expression BC for gen erating test cases we get the following set of JUnit test cases FA JUnit test cases testTS_1_2 _ Remove method Must fail with java lang lllegalArgumentException M public void testTS_l 2 throws Exception paper Account obtained null java lang String argl java lang String 12345 obtained new paper Account argl double arg2 double 100 double balancePre obtained getBalance obtained deposit arg2 assertTrue obtained getBalance balancePre arg2 _ Compile file when leaving Figure 28 20 3 2 Dealing with non primitive data types 3 2 1 Recommended method The Account class shown in Figure 24 has an operation which takes a non primitive parameter public void transfer double amount Account targetAccount With testooj is possible to create test values for these parameters from the JUnit asser tions window In the next figure the user is assigning this parameter a new instance of Account on which 1000 monetary units are deposited note the test value written and very i
35. s for test case generation Select the algorithm All combinations O Each choice O Antitrandom C Pair wise AC O Pair wise AC step by step C Base choice O Anti random for pair wise _ Pair wise first combination Figure 17 Selection of the combination algorithm for test case generation Note the default algorithm is All combinations Now we can press the Build file button in the main testooj screen to generate the test cases Note that you can select the format of the test cases through the radio buttons Figure 18 e Build JUnit file generates one or more files with JUnit test cases e Build MuJava file generates one or more files with MuJava test cases These test cases are also understood by the second main functionality of testooj which is presented in Section 4 Test case execution e Build JUnit amp MuJava generates the two types of files e Build states file creates a file with a special JUnit file In this case test case contain instructions to leave a trace of the execution on a file that can be later interpreted by testooj to perform reverse engineering tasks 13 Regular expression AICIDIE B iene Max length Get test templates Build JUnn file C Build MuJava file C Build JUnit amp MuJava _ Build states file 1i C emme Build file file Figure 18 When we press the Build file testooj leaves the file or files with the test cases at the Working directory spec
36. s used the serialized instances are taken from the seri alized folder during test case generation 3 3 Comparing JUnit and Mu Java test cases JUnit and MuJava test cases are very similar but they have some important dif ferences 23 1 Whereas JUnit test cases return a void type MuJava ones return a String which corresponds to the string representation of the instance manipulated in the test method 2 The assert instructions included in JUnit test cases are transformed into a String expression which is later used by MuJava or testooj to check what mutants are killed by each test case Figure 33 shows a brief but illustrative example JUnit Mujava tesooj public void testDeposit public String testDeposit Account o new Account Account o new Account o deposit 1000 o deposit 1000 assertTrue o getBalance 1000 return o getBalance j Figure 33 A JUnit test case left and its corresponding one for MuJava testooj JUnit and MuJava test cases are mutually complementary with JUnit test cases we find faults in the class under test when no faults are found MuJava test cases pro vide us with a very good measure of the coverage reached on the class under test if we have mutants Note testooj leaves the source code of the MuJava files at the testset folder of MuJava 3 4 An important note about the MuJava oracle MuJava test cases generated by testooj return a String
37. sertions for setJ int J paper Triangulo Precode EJ Constructors paper Triangulo EJ Methods G getTipo EJ setlfint int EJ setini int CJ setkiint Ci int Postcode Test value 0 2 With the value Open file Number of values for random testing g must throw _ Use values as described Figure 15 Assigning test values This configuration of the class under test can be saved in order to recover it in future uses of testooj If we press the Save and exit button testooj creates a folder in the constraints folder of its working directory with a set of files containing this specifica tion paper A E get Tipo constraints results E Invariants constraints gt constraints iQ paper Triangulo 9 creators 9 required results gt serializedObjects 3 temp templates Figure 16 observer observer paper Triangulot constraints setIiinti constraints setJiint constraints setk int constraints states states At this moment we are ready to generate test cases There are several strategies for this which are selected through the Algorithms gt Select algorithm option in the menu bar which shows the menu shown in Figure 17 For further descriptions of these algorithms see the paper Grindal M Offutt J and Andler SF Combination testing 12 strategies a survey Software Testing Verification and Reliability 2005 15 p 167 199 Algorithm
38. t serializedObjects E temp 9 templates e 1 Figure 22 16 With these data the testooj user can reproduce a test session by the File gt Re build test file option of the menu bar paper Triangulo 20 10 2007 1 7 0 session paper Triangulo 20 10 2007 17 15 session Session id paper Triangulo 20 10 2007 17 15 Class under test paper Triangulo Session type junit Number of test cases 3615 Date 20 10 2007 17 15 Open file Figure 23 Rebuilding test sessions Note Test session files may use a huge space on disk If you won t rebuild test files you can remove the contents of the session folder 3 1 Writing the oracle testooj allows to write the test oracle in a generic way and then to use it for many test cases Let us change the example for this section Now we ll use an Account class that simulates a banking account It has the following code package paper public class Account protected String mNumber protected double mBalance public Account String number mNumber number mBalance 0 0 public Account String number double amount mNumber number mBalance amount 17 public void deposit double amount throws IllegalArgumentException if amount lt 0 throw new IllegalArgumentException mBalance amount public void withdraw double amount throws IlegalArgumentException if amount lt 0 throw new IllegalArgumentException mBalance
39. tipo tipo 1 if G k tipo tipo 2 if G k tipo tipo 3 if i lt 0 II j lt 0 II k lt 0 tipo Triangulo NO_TRIANGULO return tipo if tipo 0 if i j lt k II j k lt 1 Il i k lt j tipo Triangulo NO_TRIANGULO return tipo else tipo Triangulo ESCALENO return tipo if tipo gt 3 tipo Triangulo EQUILATERO return tipo else 1f tipo 1 amp amp 14 gt k tipo Triangulo SOSCELES return tipo else if tipo 2 amp amp 1 k gt tipo Triangulo ISOSCELES return tipo else if tipo 3 amp amp j k gt 1 tipo Triangulo ISOSCELES return tipo return Triangulo NO_TRIANGULO Figure 6 The Bertrand Myers triangle type problem In our computer this class is in the following location C Documents and Set tings Maco Mis documentos codigoExperimentos Note that the class is included in a package called paper Thus the compiled code of this class is actually at C Documents and Settings Maco Mis documentos codigoExperimentos paper However the classpath we must write does not include the name of the package or of the subpackages There fore we write the following value in the class path field JUnit and Mujava test case generator c Alarcos Group UCLM File Externaltools Test case mutation Algorithms Compatibility States Fault pronenness Class path C Documents and SettingsiMacoimis documentosicadigoExperimentos _ Consider
40. us in general these messages do not require your attention excepting when the message announces that a expected library is not found you should add it to the class path or when you select a test file prepared for a class under test different to that you have selected for example you try to execute the test cases in the MBisectOK class file shown in Figure 36 which contains test cases for a Bisect class against the Triangulo class Test case executor File N order mutation Test file MTrianguloOKR class v Full CUT name including package paper Triangulo class Versions directory Cumujavad result Search CLASSPATH Locations of the CUT versions Add Remove Set as original C Documents and SettinggMacoiMis documentosicodigoExperimentos C CALLIS baike hnde CoM papers Mirangare tt Ganronar Bora _VSouQniyTriomw_x_4T C mujava06 result paper TriangulofAraditional_mutants void_setK int AOIU_3 C mujava06 result paper Triangulo traditional_mutants void_setK int LOI7 C mujava06 result paper Triangulofraditional_mutants void_setK int LOI_9 C mujava06 result paper TrianguloAraditional_mutants void_setK int ROR_11 Exe cuting MTrianguloOKR against paper T J i angul o class in C mujava06 resultipaper Triangulotraditional_mutantsivoid_setK int ROR_12 C SMT RTANG PSU Papar TERETE VEALE LONA CaNT A ste 5 t getTipo A0IS 5l Executing testTS 1 186 C mujava06 result
Download Pdf Manuals
Related Search