Home

User's Guide SOTA 1.0 - 1 Introduction

1. insertIntoDB 0 00 0 00 0 00 0 00 deleteFromDB getColumns getLecturerld getPresentationld E De meel wese o ro 5 a SE mae TableColumnProperty int int String int int boolean getalignment EEN ma setAlignment UE amp ESSI getColumnNumber O ma setColumnNumber in o ma oetResizabe O EEN ma setResizable boolean EEN ma gewing dE ma setwidth int O Ud i getText 6 gt EEN setText string ie ma veinde 0 ES Ss eC Ta Te wee wee upa ss NN NN HR EENENNEEEEEEEEE cati TT vee foco 000 om om 00 000 000 0 00 getMinutes 0 00 0 00 000 0 00 ooo 0 00 0 00 0 00 000 TDI 0 00 0 00 0 00 0 00 0 00 0 00 0 00 000 000 E ee L at aR adeTimechangelistener TimeChangetistener Pgo eo o oo TimeChandel istener setTime Time setTime Time Time e EI NN EEE eeeteees op String Sting eed RON 00 Fsettoometen Sting Mm EM E eme o pa a wee we 65 Tom o getFormData Object int Object int Object 0 83 33 87 50 66 67 80 00 12 35 12 35 int Object int isvalidPercentage int A EE createTextTab TabFolder String Font int si EE QI se sor se alignRight Control OASI ECM 33 33 getlmage String 100 initimage String 00 00 __ df E ee a o wwoc wee meas ai Class sem
2. Instructions can be found here e Installation reference for the program for organising seminars HUSemOrg e Installation reference for ATOS e ATOS project setup for seminar organisation 6 2 2 SOTA and ATOS General Procedure he testing procedure is similar to the manual program test with the exception that the actual testing is done using the external testing system This leads to the following phases 1 Eclipse Create Program 2 SOTA Preparatory Phase 3 ATOS Testing Phase 4 SOTA Evaluation Phase As in 6 1 3 Eclipse is only used to create the program and plays no further role in the program test as an Ant buildfile is used The following data flow diagram indicates the phases transferred to the external testing system Eclipse Work spa celS erm Org SOTA SOTA AS CLogger jar create project Semorg import classes Semorg project export build xml Create Project Semorg import java and parse java SUL java Sourcecode CFG java backup static metrics veg r T T e A i Scheme level 2 GUI java Instr SourceCode CFG 45 cLogger ini E ie f compile SenOrg ae ant build ml static metrics ATOS reqression test start Gem ug execute test script sil vai GUI java Ra eh ee Sourcecode CFG restore origina static metrics SU java read test lags and with coverage evaluate coverage SourceCode CFG metrics cover
3. Pana Zi Zi sd 1 LOC ja Ja ER 2D Collapse All Sorting the table also works through clicking the column heads ModBIP ER data 19 Bran 10 BIP 43 14 10 ER 43 19 LU ER 43 1 ses 1 1 16 10 ee 42 Abb View Metrics 4 3 Preferences 4 3 1 Preferences View CFG In the first options block of the preferences page of the view CFG the user can determine which coverage metrics will be displayed as small square labels next to each node of the control flow graph The default setting shows only four labels for the condition coverage metrics C2 MMDC MCDC C3 next to all nodes containing non trivial conditions and at the function node If the display of path coverage metrics are enabled second option three more labels will appear next to ii Preferences Five Coverageview General Report Source view Fig EX CFGYiew Preferences For the CFG vigm Show labels for condition coverage nodes with conditions Show labels For path coverage ifunction and iterationnodes Show labels For basic coverage ifunctionnode visited CFG Mode Not visited CFG Node Highlight branching nodes wo all exits taken Branching CFG nodes visited CFG Edge EI Not visited CFG Edge EI Width of CFG Edges ipt 2pt value Default CFG Mode Default CFG Edge apt Current sourcode line MI Restore Defaults Apply Preferencees
4. QU Activating the menu item Show Coverage leads to a colored display of the coverage information in the source code and coverage views 4 1 3 Menu Configuration Configure Project Project Tasks RER Help GC E Configure Project Project specific settings can be made via the menu item Configure Project In detail these are the two paths of the test Preferences project relevant to SOTA namely the root directory and the Fig menu Configuration execution directory and underneath them the optional scripts imported for the purpose of compilation and start of the fi Select Sources project Confirming these values by selecting OK overwrites the Configure Project settings made when Configure Project Ziffer the project was created Select Sources Project directory O Developmentiworkspace Digit G ka Execution directory DiDevelopmentiworkspace Digit colin rog bee Do the source files included in the Run Script PO project the user has to select the menu iui Sources Subsequently a dialog identical to the second page of the wizard for creating a project appears The source files can be selected analogically After confirming the selection the current source files are replaced by the new ones Fig dialog Configure Project Preferences Under the menu item Preferences the user can make changes to the general settings of SOTA that apply to all projects and will also be saved when exiting the program A com
5. e C2 true evalutations of all atoms false evaluations of all atoms 2 atoms 7 2 5 Minimal Multiple Condition Coverage MMCC The minimal multiple condition coverage has established itself as a practible condition coverage which also includes the branch coverage Analogous to C2 all evaluation of the logical atoms are regarded here as well as all compound complex conditions These have to be evaluated as true as well as false during the tests The number of logical structures which have to be analysed corresponds to the number of logical conditions listed under 6 1 10 e MMCC true evaluations of all conditions false evaluations of all conditions 2 conditions 7 2 6 Modified Condition Decision Coverage MCDC An even more exact test criterion 19 the modified condition decision coverage test To fulfill this coverage 1s not only necessary that all logical atoms of every condition adopt the values true and false Additionally it should apply for each atom that configurations of these conditions exist which only differ in this atom and lead to an alternative evaluation of the complete condition This ensures that the test checked whether changing the logical value of each atom would have an influence on the total condition The two truth vectors of a condition fulfilling these requirements for an atom are called MCDC couple The coverage metric is then calculated using the MCDC couples as follows e MCDC Modified Condition D
6. setInputComponentsEnabled boolean createButtonListener Composite 84 62 50 00 50 00 50 00 50 00 50 00 50 00 AssociationTabControl checkInput 54 55 50 00 50 00 50 00 50 00 commitinputToDB 50 00 50 00 50 00 SE ET ewe wee OE tO Class semorg sdl tables Companyl nternalPresentation CompanylnternalPresentation int int Integer Date Date Time Time Time Time String String String String String String boolean Float Integer Timestamp Timestamp getVector ResultSet 84 85 60 00 62 50 62 50 62 50 getVectorFromDB Vector String tableOK createCompanyinternalPresentationTable Statement String CNRC aa FEE Class semorg gul list Company l nternalPresentationListWindow Sho 23 2 50 00 50 00 50 00 CompanyInternalPresentationListWindow ENE EN Fa CompanylnternalPresentationListWindow JL ForeignKeyConstraintSelector CompanylnternalPresentationListWindow AssociationTabControl createSpecializedMainTableListener 00 01 createS pecializedMainTableListener AssociationTabControl createSpecializedMainTableListener ForeignKeyConstraintSelector createSpecializedToolBarListeners 50 00 createChooseButtonListener AssociationTabControl createChooseButtonListener ForeignKeyConstraintSelector openList createWindow Shell 00 0 90 01 EEN EE EN EN 00 00 10 00 ES le Le Le Ep meoc es mes 5 O Clas
7. Ep Java ES Pr te Hi p 0 gl E O LI DigitTestjava o Finished after 0 006 seconds SA 1 package hu demo D n GE GP NE 3 import junit framework TestCase Runs 1 1 B Errors O B Failures 0 4 pe 5 public class DigitTest extends TestCase 6 RC hu demo DigitTest Runner JUnit 3 0 L public void testEvaluateDigitSequence EE testScanDigitSequence 0 000 s 3 feec test sequence to obtain full FEEC coverage 9 ce test sequence to obtain full C coverage 10 cl test sequence to obtain full C1 coverage LL c2 test sequence to obtain full C2 coverage Re 12 ie mmec test sequence to obtain full MMCC coverage 13 mcdc test sequence to obtain full MCDC coverage 14 c3 test sequence to obtain max C3 coverage 15 16 176 public void feec 18 System out println z Digit evaluateDigitSequence z 19 System out println Digit evaluateDigitSequence 9 4 In j 20 Sa 21 EE aoe JS 22 public void c 23 System out println 9 Digit evaluateDigitSequence 9 24 System out println z Digit evaluateDigitSequence z 25 26 27 public void c1 gt 4 b 7 Problems Javadoc B Declaration El Console Si Progress x Ex bE 536 pie Pie lt terminated gt DigitTest JUnit C Program Files x86 Java jre6 bin javaw exe 30 09 2010 11 03 51 iz 1 0 E o 9 0 E 4 p DS Writable Smart Insert 34 69 Fig JUnit test ca
8. The case node also iteration node contains this value as nrSelects which can differ from the number of hits The tooltip for iteration nodes lists next to the number of hits details about how often the loop body was skipped nrSkips the number of times it was executed exactly one time nrSingleLoops and two or more times nrMultipleLoops as well as the overall number of executions nrLoops For the try nodes which initiate the exception handling it is mentioned how often the try block could be completed without an exception Information on condition coverage is viewable in two ways On the one hand located on the right side of Node Information each node containing a Iteration Node 22 non trivial condition are Line 15 in Digit java Function _ evaluateDigitSeguence String four small boxes representing the different degrees of condition coverage From top to bottom these are the single minimal multiple MC DC and multiple condition coverage The color indicates the degree of coverage from green meaning 100 to dark red meaning 0 The threshold values and colors can be set in the preferences nrHits nrSkips nrSingleLoops nrMultipleLoops nrLoops 1 2 C2 100 0 4 4 100 0 6 6 50 0 112 75 0 3 4 20 4 20 where their presentation Condition position inDigitString lenath amp errorfree can also be turned off and the user can select two Truthvector evaluates to
9. They are determined while parsing the source code and it 1s not necessary to execute the program contrary to the coverage metrics The metrics provide on the one hand a means of estimating the complexity of the source code in terms of different criteria thus giving the user an indicator for enhancing the structure of the source code On the other hand they enable the user to assess the costs of testing and the number of different tests for individual criteria respectively The static metrics are visible in the viewMetrics for all structures of the projects right after loading it The values of the cyclomatic and essential complexity for classes files and the project are the maximum of the values of their subordinate functions for all other metrics these values are summed up Notes on the ModBI and BI values The entire scope of exception handling eliminate the possibility to identify paths precisely Therefore the computed value is always a lower bound 1 e the minimal number of paths and sub paths respectively that will be reached during the ModBI and BI test 7 1 1 Cyclomatic Complexity Cyclomatic complexity 1s computed using the control flow graph which represents all paths that might be traversed during program execution and their branching habits cf view CFG The cyclomatic complexity z G is defined as z G e n 2 where e is the number of edges and n is the number of nodes of the control flow G Therefore a function without branches
10. Time R lt oon getStreet EN e setStreet String BEE gt e lt ooo oo getZipCode EN RR setZipCode String 0 00 000 0 00 0 00 Class semora gui list PresentationListWindow E Presentetiontistwindow O EE EEN PresentationListWindow Associationtabcontro EEN ma PresentationListWindow createSpecializedMainTableListener createSpecializedMainTableListener AssociationTabControl createSpecializedMainTableListener ForeignKeyConstraintSelector createChooseButtonListener AssociationTabControl createChooseButtonListener ForeignKeyConstraintSelector Ton createSpecializedToolBartisteners O EE MR openust0 EES getTableinputFromDB O E ET RE had KR KS HEN C ass semorg qui provider PresentationTableProvider Patte 61 36 12 91 50 00 50 00 ER 50 00 66 67 66 67 EE EE ERNEST removeListener ILabelProviderListener O ASA oe pa EER CN CC CO OO EE iti Time Time Time Time String String String 12 90 25 00 16 67 16 67 16 67 4 55 4 55 String String String boolean String Float Timestamp Timestamp getVector ResultSet getVectorFromDB Vector String tableOK createPublicPresentationTable Statement insertIntoDB updateDB getColumns 00 01 D C 100 00 getColumnsWithSemtype getCoopPartner setCoopPartner String getCa
11. a o momo meas ai Class semorg sql tables Associate 56 25 67 71 Associate int String String String String String String String String String String String String String Date Date int String String String String Timestamp Timestamp getVector ResultSet 93 33 75 00 50 00 50 00 getVectorFromDB Vector String tableOK createAssociateTable Statement getEntitlement getOccupation getPassword setEntitlement int setOccupation String setPassword String insertIntoDB updateDB getNext int getPrevious int getColumns ES lol al mw mo upa Class semorg gui list AssociateListWindow 50 00 50 00 AssociateListWindow AssociateListWindow ForeignKeyConstraintS elector AssociateListWindow AssociationTabControl createSpecializedMainTableListener createSpecializedMainTableListener AssociationTabControl createSpecializedMainTableListener ForeignKeyConstraintSelector createSpecializedToolBarListeners 50 00 createChooseButtonListener AssociationTabControl createChooseButtonListener GE aan elector emo Fm N SEE IE of ja fi H pa pa me e we a EE EE EF my E E EE EI ET eiseres Oues seror JS een E E E E Es SS ole Pomoc coe e spa EI rT 000 oaa f oro vae peer eer 3333 ME N MEE EE I asas sas IRR Gegee oso om AE N EN EE
12. 16 67 createWindow Shell a a e wee wese es um e dee al Lt o eoo deo eo de getColumnImage Object int 00 0 D O EEN EEN EN EN DO OI 10 0 addListener ILabelProviderListener EEN OS DI 10 00 isLabelProperty Object String EEN EEN EN EN removeListener ILabelProviderListener 00 O EEN EEN EN EN EN 00 0 E pro a e we mo ws Class semorg qui CompanyWindow CompanyWindow Shell SES CompanyWindow Shell Company JU UI U ForeignKeyConstraintSelector TT set nputComponentsEnabled boolean JO LI VA EEN EEN EN EN commitinputToDB OO 00 00 0 00 0 0 0 confirmClose 7 E D a o wwoc wee e meas ai Class semorg sql tables Conduct Conduct int int Timestamp Timestamp tableOK createButtonListener ForeignKeyConstraintSelector createConductTable Statement insertIntoDB deleteFromDB getColumns getLecturerld getSeminartypeld E Ce a o wee wcoc cs mp Class semorg gui util Configuration Configuration boolean makeStdConfiguration boolean setProperty String String getProperty String BT RE a ea moe wese wear Br Class semorg gui util ConfigureListDlg EEN EN CancelPressed CreateDialogArea Composite getColumnProperties configureShell Shell CU oe CORE ConfigureListWidget Composite TableColumn int setListViewerlnputs TableColu
13. 22 86 15 04 7 69 Class semord dui list PublicPresentationListWindow 38 46 37 14 33 33 Class semorg qui provider PublicPresentationTableProvider 0 00 0 00 Class semorg gui PublicPresentationWindow Class semorg gui SelecttanguageWindow Class semorg app SemOrg 100 00 53 85 33 33 Class semorg app SemOrgStarter 50 00 Class semorg sal tables SeminarType 16 67 Class semorg gui list Seminar TypeListWindow 53 55 Class semorg gui provider SeminarTypeTableProvider 75 86 Class semorg gui SeminarTypeWindow Class semorg sql tables Simplel DKey ENS 60 00 50 00 4 76 5 11 29 41 71 43 60 00 50 00 Doll 29 41 71 43 16 67 50 00 50 00 16 67 50 00 50 00 Class semorg sql tables Supervisor Class semorg qui util TableColumnProperty Class semorg qui util TimeControl Class semorg qui util UtilityGUI Class semorg sql util UtilitySOL Detailed Coverage E Geo a je wwoc wese meas ai Class semorg sgl tables AbstractTable AbstractTable Timestamp Timestamp getCreationDate getModificationDate setModificationDate Timestamp createExtendedQueryString String Vector String getColumns String tableOK String getNext String DBColumn int getPrevious String DBColumn int fireTableChangedEvent int addDBTableChangedListener DBTableChangedListener removeDBTableChangedListener DBTableChangedListener E eel o
14. Conditions 16 Coverage MCDC Multiple Condition Coverage C3 46 43 Modified Boundary Interior Path Coverage ModBI 26 09 Boundary Interior Path Coverage BI 26 09 Tests test Level 2 instrumentation test 2 Level 2 instrumentation test 1 Level 2 instrumentation test 1 1 Level 2 instrumentation test empty Level 2 instrumentation Coverage of Classes Project Digit 10 Class Digit l U Detailed Coverage main String evaluateDigitSequence String SOTA User s Guide Version 1 0 1 Introduction 2 Overview 3 Installation and Start of Program 4 User Interface and Functionality 5 Files 6 Tutorials 6 1 Manual Program Test Program Digit 6 1 1 Program Digit 6 1 2 SOTA and Eclipse 6 1 3 SOTA with Ant Buildfile and Start Script 6 2 1 ATOS and HUSemOrg 6 2 2 SOTA and ATOS 6 3 Automatic Testing System SOTA ATM 6 3 1 SOTA ATM via command line call 6 2 2 SOTA ATM API 7 Appendix 6 Tutorials The following three tutorials explain the behaviour of the three ways to use SOTA in manual program testing program testing with an external test program and the test as a library in an automatic testing system 6 1 Manual Program Test Program Digit 6 1 1 Program Digit The basis for the tutorial of the manual program test is a simple Java program which tries to read a positive rational number from a string The program consists of the class Dig
15. DiiDevelopmentiworkspace Digit Language Fig first page of wizard 3 SOTA Preparatory Phase Instrumentation The next step is to instrument the project for the next test run i e adding instructions that effect in writing data into the log file during program execution allowing its complete reconstruction For this reason the user has to select the menu item Es Start Test A dialog opens and a test name as well as an instrumentation scheme have to be selected The test name also determines the name of the log file where all the log data is saved The instrumentation scheme specifies the manner of instrumentation for all structures of the project Usually the user should select the IScheme Level 2 instrumentation which leads to a minimal instrumentation but computes all coverage metrics for all files After confirming the dialog the file Digit java is saved as Digit java backup and then the instrumentation directions are added to the original file With this the preparation of the instrumentation in SOTA is completed and the testing phase may begin 4 Eclipse Testing Phase Compilation El New Project Source Files Please select the source files to import into this project Project directory Di DevelopmentimorkspacelDiqit 5 me pis t bin Q lib GJhel src Digit java Expand all Collapse all Fig second page of wizard Start test Press OK to augment the project Test name Test
16. E ECC eel soas gee serron srn fee eel ao Tesan feaae ooo on 00 10000 BEE 7 oso poso 10000 BEESTE VEER CE este iseg EC75 ee poso 10000 getShortName setZipCode String fi rec a o wee coc TG od al CompanyBooking int Date Date Date Date EN N EN Date int int Timestamp Timestamp CreateCompanyBookingTable Statement insertlntoDB updateDB S getCompanyBooking int Class semorg qui list CompanyBookingListWindow 50 00 50 00 50 00 9 4 CompanyBookingListWindow ForeignKeyConstraintS elector CompanyBookingListWindow AssociationTabControl createSpecializedMainTableListener createSpecializedMainTableListener AssociationTabControl createSpecializedMainTableListener ForeignKeyConstraintS elector createSpecializedToolBarListeners 50 00 createChooseButtonListener AssociationTabControl createChooseButtonListener ForeignKeyConstraintS elector openList getTablelnputFromDB createWindow Shell ES Prec co a wwe wee o upa Class semorg gui provider CompanyBookingTableProvider 52e I 68 00 20 00 20 00 30 00 60 87 60 87 E ce Cu isa re a To woe wee cs mos Class semorg gui CompanyBookingWindow CompanyBookingWindow Shell AssociationTabControl CompanyBookingWindow Shell CompanyBooking AssociationTabControl TO
17. Ed 0 00 000 000 LE getPrevious int 0 00 0 00 0 00 0 00 O EEN fe getColumns 10 0 getEmployerld setEmployerld int getTask setTask String getTurnover setTurnover Float getClient int getEmployees int rc co ci cz moc moc c mam a Class semorg sgl tables ClientBooking 17 02 18 18 6 67 4 55 4 55 0 00 4 55 10 00 10 00 ClientBooking int Date Date Date Date Rees o BIS Date int int int int Timestamp Timestamp 100 00 21 05 50 00 50 00 50 00 TOBE 50 00 25 00 2500 getVector ResultSet getVectorFromDB Vector String tableOK createClientBookingTable Statement insertIntoDB updateDB getColumns getClientBooking int getNext int getPrevious int getClientid setClientld int aa esagerati Elma ETIENNE MEM Ema oers om Fsetresentationid ini MM ome getBookingsOfClient int 0 0 ENE EN EN 0 00 0 0 getBookingsForPesentation em EN El setSubstituteld int 0 00 0 00 ee Sie 0 00 0 00 EECHER E IC mejo o we ea Class semorg gui list ClientBookingListWindow HR 50 00 50 00 E EET Ee ClientBookingListWindow ForeignKeyConstraintSelector ClientBookingListWindow AssociationTabControl createSpecializedMainTableListener createSpecializ
18. Presentation int int Integer Date Date Time Time Time Time String String String String String String boolean Timestamp Timestamp T eetSeminarTypeDeseroton 0 sooo 7000 5000 5000 EE eegene Genen EE EEE 2 cerns ooo oo ooo oes 000 ooo 000 000 0 00 Cavers eso foco oco ooo 0 00 0 00 000 000 0 00 removeFromDB Iterable getColumns getVector ResultSet getPresSupervisedBy int getPresl nstructedBy int getPresForSemType int getPresentationVectorFromDB Vector String getPresentation int getColumnsWithSemtype hashCode equals Object getShortDescription ovo 0 00 57 50 50 00 50 00 50 00 GER 50 00 25 00 25 00 Coco 00 0 00 getSeminarTypeForPresid int getAnnex setAnnex String getBeginningDay setBeginningDay Date isCancelled setCancelled boolean getCity setCity String getCountry setCountry String getDuration setDuration Integer getEndingDay setEndingDay Date getEndingTime setEndingTime Time getFirstStartingTime setFirstStartingTime Time getLastEndingTime setLastEndingTime Time getLocation ie EN N setLocation Strind I IJ Jr Wee getSeminarTypeld R e ooo oo setSeminarTypeld int R lt ooo oo getStartingTime e e E setStartingTime
19. Procedure The detailed procedure matches the procedure described in point 6 1 2 and only differing steps are ecplained 1 Create Project Additionally to creating the project as in 6 1 2 an xml build file is exported from Eclipse This can be done by selecting A ES Export in the context menu A dialog opens where the user should select General gt Ant Buildfiles In the second dialog window only the corresponding project here Digit has to be chosen and after finishing an xml file named build xml is created in the root directory of the project which enables compiling the complete project Select Generates Ant Buildfiles for Eclipse Java projects Select an export destination type filter text E L2 General Ant Buildfiles JE Archive File 13 File System E Preferences gt Cl iu co Fig Eclipse export dialog Export Generate Ant Buildfiles Generates Ant buildfiles based on the configuration of the Java projects KH q lect All Select the projects to use to generate the Ant buildfiles Deselect All Check projects for Ant compatibility Create target to compil project using Eclipse compiler Name For Ant buildfile build xml Fig Eclipse Ant build file dialog Finally a batch file Digit bat has to be created in the root direcotry of the project Digit This 19 necessary to start the program The file has to contain the following Java command including the class path java
20. Which method stubs would you like to create public static void main String args Constructors from superclass Inherited abstract methods Do you want to add comments Configure templates and default value here a Generate comments Fig Eclipse New Java Class Next it is necessary to include the logging component as a library in the project In order to do this the user has to copy the file ASCLogger jar from the SOTA directory into a folder lib in the root directory of the porject workspace Digit On refreshing the project overview in Eclipse F5 this file as well as the folder lib appear see figure Now the file has to be added to the build path of the project by right clicking ASCLogger jar in the context menu and selecting Build Path gt fora Add to Build Path The library is then included in the Referenced Libraries With this the first phase writing the program and preparing it for instrumentation by SOTA is completed HE Digit HE Digit GES src HG src E default package EF default package J Digit java J Digit java E Digit java backup B Digit java backup HEL JRE System Library jre6 H 4 JRE System Library jre6 EL lib F Referenced Libraries La asclogger jar H b ASCLogager jar Fig Proejct Digit with gt lib ASCLogger jar Fig and with ASCLogger jar added to build path 2 SOTA Preparatory Phase Create Project After starting SOTA via SOTA exe the user has to cre
21. again Recover Project GI The menu item Recover Project s task is to restore corrupted projects Analogous to the menu item Open Project a dialog for selecting a project appears As a result all source files of the chosen project are restored and then opened Exit Clicking the menu item Exit closes SOTA 4 1 2 Menu Tasks Re ra Ei Ken New InstrumentationScheme Project Configuration Help KS G New InstrumentationScheme Upon clicking the menu item New Instrumentation Scheme a Be Start Test dialog allowing the creation of a new instrumentation scheme Stop Test short IScheme for the current project appears An IScheme Create Report represents an instrumentation pattern for all source files of the project It can adopt different degrees Depending on the de Build Project test case the memory requirements of the log files can be e Run Project very high The purpose of this configuration option is to GM Restore Sources allow the user to set a reasonable limit for the size of those 3 Read Testlogs files Show Coverage For creating an Scheme a name for management of the Fig menu Tasks scheme inside the project must be specified in the dialog Entering a description is optional The field below the description field represents the hierarchical structure of the current project in form of a tree It shows all files their classes and the individual functions A level of instrumentation Create new instru
22. comparing the number of executed statements with the number of all statements 7 1 5 Number of Branches Branches The number of branches is defined functionally in SOTA as a way of computing the branch coverage While the number of branches in a function equates to the cyclomatic complexity 1 in this case the number of branches is defined as the sum of the outputs of all branching nodes So for a function without branches the number of branches is zero for each added if statement the number increases by two 7 1 6 Number of Modified Boundary Interior Paths ModBI The number of modified boundary interior paths corresponds to the number of subpaths through the control flow graph which have to be tested to fully execute the modified boundary interior paths coverage test The different kinds of subpaths are defined according to Liggesmeyer Software Qualitdt 2002 as follows e all executable paths through a function which do not enter pre test loops nor repeat post test loops e all executable subpaths of each loop which execute the body of the loop exactly once disregarding the behavior for closed loops all executable subpaths of each loop which execute the body of the loop exactly twice disregarding the behaviour for enclosed loops and possible following cycles of the loop body In the View CFG the user can find the number of subpaths which have to be tested according to the above definition for each loop in the node info dou
23. cp bin lib ASCLogger jar Ziffer 2 Preparatory Phase Create Project El New Project Project configuration Please enter properties of the project Project name Project directory Execution directory Language Ant build File Ce Developmentiworkspacel Digit build ml Run script Di DevelopmentimorkspacelDigitiDigit bat Fig create project with scripts Creating the project in SOTA works as in point 6 1 2 with the only difference that the files build xml and Digit bat created in the first step have to be imported on the first wizard page However it is also possible to add these files at a later point via the menu item dan Configure Project In order to succesfully compile the project using the Ant build files an Ant file ant bat that SOTA can execute also has to be included in the preferences under Preferences gt General gt Location of Ant Since the settings in the preferences are effective for all projects it is sufficient to make this entry once 3 Preparatory Phase Instrumentation 4 Testing Phase Compilation 5 Testing Phase Program Test By including the two files the manual program test is available in SOTA The dialog Start Test now offers the option Build Project and as soon as this has been marked also the option Run Project If the first option is active the first Ant build file will be executed thus compiling the instrumented sources The second option also runs the start script
24. down while selecting the desired test logs Selected test logs can be deleted with the A Delete TestLogs button That not only removes them from the project but from the system This option is not available for test case data 1 e individual parts of a test log On double clicking onto one of the listed test logs or test case data a dialog appears containing detailed TestLog test 1 information on the test data Listed first are the name and description of the test as well as the name of the instrumentation scheme Next the number of paths 1 e function cycles contained in total in the test case and a list of all functions the test hit are indicated For each function the number of associated paths 1S also listed By Function Paths ll Lyle Led clicking the column headers Digit main Strina 3 the presentation of the list can Digit evaluateDigitSeguence String a be sorted in ascending or descending alphabetical order by function name and number of paths Test name Test description IScheme Level 2 instrumentation fr of Paths in Test Fig dialog TestLogs 4 2 3 View I Scheme The view Schemes contains only the instrumentation schemes short ISchemes related to the project For every 29 TestLogs MA Ischeme x project three Schemes are created automatically is ii OE instrumenting the entire project into the levels 1 2 and 3 R Level 2 instrumentation respectively The user can create additi
25. it is possible to select the option Build Fig Preferences General Project in the dialog Start Test Be and the menu item Build Project as well as its equivalent in the toolbar are enabled In Eclipse an Ant buildfile for compiling the project can be generated via File gt Export gt Ant Buildfile If Create log file is activated system messages of SOTA will be saved as log files of the format sota_ lt YY MM DD gt _ lt index gt log on each program boot Also enabling the option Overwrite existing log file causes SOTA to create only one log file sota log and overwrite it every time the program is booted The last two options define more general aspects of the behavior of SOTA If Parse instrumented source code 1s activated the project will be parsed again after launching the test the view of coverage metrics will be disabled and the presentation of the project in all views will be based on the instrumented sources In this case the source code displayed in the view Source is always identical with the current sources 1 e the two status indicators on the status bar always show the same value Should the option be disabled the backup will be parsed and displayed instead of the instrumented file This allows an evaluation of the test logs independent of the status of the source files The last option determines whether the test project will be compiled automatically provided that a corresponding Ant script was included in addition t
26. iteration do while loop do iter body iteration iter end e selection control switch statement switch case default switch end e exceptions try catch block try try block catch finally try end e jump instruction break continue return throw For assigning nodes to their corresponding parts of the source code it 19 sufficient to click on a node Then the bottom of the view CFG focuses on the appropriate section of the source code and the line is highlighted in yellow If the option er Pin SourceView is selected in the toolbar of the view only on the first selection of a node the corresponding source code line will be displayed afterwards the source code remains pinned and does W gt 1 not scroll automatically anymore Selecting the option n Show Number of Paths displays the number of edge traversals during the current test on the left of every edge of the control flow graph In order to be able to also view large graphs clearly the user can scale the illustration of the control flow graph via the buttons Zoom Out and A Zoom In SOTA offers seven zoom levels The first three zoom levels still display the condition coverage and the fourth level still shows the caption of the nodes The three smallest zoom levels reduce the graph to blank squares More detailed information is available via the tooltip or the node information dialog see below E source Fri FG E Coverage a
27. necessary a lower level can be used in order to save capacity Using level 3 is suggested when programs or functions are not terminating properly and the user wishes to identify the exact point of abortion or where the exception is called After confirming the dialog the instrumentation scheme as well as all selected settings are stored in the project specific ischeme file which is from then on available for the program test Start Test L Restart Test die By clicking the menu item Start Test the program test 1s initiated in SOTA A dialog box for configuring the test appears Start test The user has to enter a name Press OK to augment the project for the test which also serves as a name for the test log SOTA creates at the end of a test 1 e Test name limitations for file names set by Test description the respective operating system must be considered Adding a description of the test is also Select IScheme main Mt _evaluateDigitSeguence Ji k possible but not mandatory KE Digit java The instrumentation of the Digit project has to be configured 2 main String next In addition to the user 2 evalusteDigitSequence String generated ISchemes the 3 basic ISchemes which instrument to whole project according to Level 1 Level 2 and Level 3 are also available The hierarchical project list shows the specific instrumentation of the individual project Structures Level 1 Instruments cfg relevant bl
28. ooo 00 o ooo 0 00 ModBl 60 00 Bhe CN CES EE Ek 60 00 top FEEC CO MMDC MCDC C3 ModBl B C1 C2 E oe wee pa SeminarType int String String String String String String Integer String String String Float Integer Integer Timestamp Timestamp tableOK createSeminarTypeTable Statement insertIntoDB updateDB removeFromDB Iterable getVector ResultSet EEN EE CI EET css cs getVectorFromDB Vector String getSeminarTypeFromDB int getColumns getAudience setAudience String getCharge setCharge Float getDocuments setDocuments String getDuration setDuration Integer getMaxEntrants setMaxEntrants Integer getMethodology setMethodology String getMinEntrants setMinEntrants Integer getObjective setObjective String getRequirements setRequirements String getRoutine setRoutine String getShortTitle setShortTitle String getTitle setTitle String getTopic setTopic String getld getNext int getPrevious int getSemTypeConductedBy int hashCode equals SeminarType equals Object top FEEE CO C Class semorqa qui list SeminarTypeListWindow SeminarTypeListWindow C2 50 00 SeminarTypeListWindow AssociationTabControl Semina
29. possible to compute all coverage metrics except for the condition coverage Level 2 Additionally to level 1 the instrumentation according to level 2 also saves the configuration for each atom provided it would also be evaluated in the program in the log file These data allow SOTA to compute the metrics as in level 2 as well as the condition coverage metrics for the program test Level 3 Finally SOTA offers a full instrumentation of the source code with the instrumentation according to level 3 Next to the evaluated atoms the log file will also include entries about the execution of all individual statements Therefore the log file 1s considerably larger compared with the other instrumentation levels This option of instrumentation 19 not only offered for the sake of completeness but also permits a detailed analysis of the control flow for programs terminating in an unusual way and exception handling 7 4 More Terms and Definitions Glossary Ant Ant Buildfile Apache Ant is a common tool comparable to make in Java development for automatically compiling source projects Destinations and commands for the compilation are stored in an XML file the Ant buildfile which Ant can read and then execute the compilation When using Eclipse it is possible to easily export an Ant buildfile via File gt Export gt Ant Buildfile zu exportieren ASC Logger ini The testing of Java programs requires a logging ASCLogger jar component named
30. program 1s its root directory where all source files and binaries possibly in subdirectories are located From here are the sources and the project imported and the coverage report is put into this directory The base directory of SOTA is SOTA Here are the executable SOTA exe and the library ASCLogger jar stored Aditionally the project file lt projectname gt project as well as the log file of SOTA with all program outputs are created in this directory Every test of a program that requires the program to be executed is a dynamic program test Amongst those are functional Black Box and structure oriented White or Glass Box Tests As a tool for structure oriented testing SOTA calculates the nine different coverage metrics for each test SOTA offers several levels of instrumentation in order to enable limiting the overhead evoked by the instrumentation An instrumentation scheme short IScheme contains information about a specific way of instrumenting the project 1 e it provides a mapping of all functions of the project to an instrumentation level SOTA always includes the three basic Schemes allowing instrumentation according to levels 1 2 and 3 If a new IScheme 1s created these data will be saved in the project file lt projectname gt project in the base directory of SOTA and will be available for usage in this project The start script a batch file under Windows is a file causing the start of the test progra
31. to be written in Eclipse and then tested using a structure oriented program test This procedure is split into 4 phases 1 Eclipse Create Program 2 SOTA Preparatory Phase 3 Eclipse Testing Phase 4 SOTA Evaluation Phase The following data flow diagram reflects all important actionas as well as inputs and outputs The four phases result from switching between Eclipse and SOTA and are depicted in differing colors The interface between Eclipse and SOTA is realised only with the specified files Eclipse weork space Vi fier SOTA SOT AL AS Logger jar Digit java create project Digit import ASCLogger jar create class Digit Digit project create project Digit Import Digit java Digit ischeme and parse GUI Digit java SourceCode CFG static metriese Digit java backup Digit java cinstrumentedi e 22 C Logger ini start program test test name ti Scheme level 2 GUI Digit java instr SourceCode CFG Static metrics update project FS recompile GUI Digit java SourceCode CFG Static metrics N Digit class start program Digit ne a finish program test H log restore original Digit java GUI Digit java with coverage SourcetCode CFG coverage metrics static metrics read test lags and evaluate coverage data report html create repart Fig DF diagram manual program test with Eclipse Detailed
32. z Digit evaluateDigitSequence z A T_FEEC log 25 E Tl log 26 B T10 log 275 public void cl E TiLlog 28 System out println 9 Digit evaluateDigitSequence 9 E T12 log 29 System out println z Digit evaluateDigitSequence z D T2 log ji 30 System out printin 9 Digit evaluateDigitSequence 9 E T3 log 31 E T4 log 32 Tp kers 336 public void c2 iis 34 System out println 9 Digit evaluateDigitSequence 9 HI 4 p n Writable Smart Insert 14 48 s Bi es Dm Fig An example for JUnit test case within Eclipse As can be noticed in this test file all of the test cases are grouped according to the cover measure they should cover which is stressed by the title of a method Additionally some formatting was added so that test cases are more readable For each test case the form of a source code line 1s System out println z Digit evaluateDigitSequence so that the resulting line in a log file would be test case test result After this user should perform the actual test This is performed by starting the test run with Run As JUnit Test The test will run as long as necessary and the results of that run can be observed in a Console window of Eclipse With the above mentioned set of test cases the results are Navigate Search Project Run Window Help 3 lt 1 c LA ESC oT Ve DH H AL 11 ES Debug
33. 0 100 00 0 00 100 00 100 0 R C NEE NE iO NGI IED aana oo 000 000 000 000 000 000 0007 S 50 00 sooo sooo E 50 00 5000 50001 HR NOI NG Ne OE O O IE ere sma ovo as ooo 000 000 000 000 000 000 eel a o wwoc wcoc meas ai foe SSCS Class ae I l 41 18 44 30 21 05 35 11 235 71 28 57 35 71 24 39 24 39 semorg gui util ForeignKeyConstraintSelector ForeignKeyConstraintSelector Composite int IA TE oo CC Na ForeignKeyConstraintSelector Composite int boolean a HA MM oo o _ 000 0 09 EE EN EA ER 0 00 000 EE o 215 35 22 fe fe EP 0 00 10 uote im Fonberse0 Jen T getermneneieteWessage 0 ION Comecemos 0 DS se Toe coe RR tableOK createlnstructorTable Statement getLecturerld getPresentationld insertintoDB 0 00 SET RT Be moe E TS mapa a EN E E E E E E e BT RE a mee wese me a Class semorg sal util KeyPair EN KeyPair int int equals KeyPair getKey1 getKey2 E Ce a o wee wee e meas ai Class semorg sql tables Lecturer Lecturer int String String String String 0 0C 0 0C 0 0C JO OC String String String String String String String String String Date Date String Float Float String String Timestamp Timestamp getVector ResultSet of 94 12 87 50 getVectorFromDB Vector String tableOK createLecturerTable Stateme
34. 0 00 sooo de N ee co ci c2 mmoc meoc cs modi Bi Class semorg gui util CalendarControt eer 50 00 155 00 ea NOOO 0 50 soo CalendarControl Composite int 00 EEN EEN EN EN 100 01 0 0 onopen o N 0 00 0 00 0 00 ooo ooo 000 ooo 000 000 F fresurreslendarPopupciosei 0 110 oes oco ooo A oes ooo oe oe Z fresWTCalendarPopupopenei 10 000000 a 000 000 0 00 0 00 ewen ener 6667 5000 5000 50 00 Ep 50 00 50 00 50 00 a 50 00 5000 REEN 5000 7500 75 00 a setDate Date addSWTCalendarPopuplistener SWTCalendarPopupListener removeSWT CalendarPopuplistener SWTCalendarPopupListener addSWTCalendarlistener SWTCalendarListener Factosenorupeutono RES Es removeSWTCalendarlistener SWTCalendarListener 0 00 0 00 isPopupOpen 0 00 0 00 j i 00 000 000 0 00 0 00 0 00 0 00 0 00 000 EE B d WE Lee ea wee coe s meas a Client int String String String String String OO D O O String String String String String String String String Date Date String Float int String String Timestamp Timestamp getEmployerpescription a000 Feel a 2500 getVector ResultSet 93 75 83 33 of 37 50 25 00 getVectorFromDB Vector String tableOK createClientTable Statement 0 00 insertIntoDB 25 00 25 00 updateDB getNext int 0 00 000 000
35. 1 1 Test description Select Scheme Level 2 instrumentation ww a SE Digit java E Didit ds main String ds evaluateDigitSeguence String Expand all Collapse all Level 0 No instrumentation leaves sourcecode as it is es Level 1 Instruments cFa relewant blocks standard coverage analysis dE Level 2 Instruments also conditions enables analysis of condition coverage 2 Level 3 Full instrumentation every single statement is instrumented Rerun configuration Don t change sources or binaries Instruments the project with the chosen IScheme Fig dialog Start Test At first the modified source code of the entire project has to be loaded in Eclipse To do this the project Digit has to be selected in the project overview and then refreshed by clicking F5 or via the context menu gt Refresh Subsequently Eclipse automatically compiles the new source files The instrumented sources require the library ASCLogger jar which was included in step 2 Without the correct integration of the library the compilation process will evoke error messages 5 Eclipse Testing Phase Program Test After succesfully compiling the program it is ready for the test In Eclipse it is started via the button Run As On the first start Eclipse prompts whether the program is supposed to be started as an application or applet Here the user should choose Application The following dialog asks for the application where the correct cho
36. ASCLogger jar which has to be included into the project then it administers saving the coverage data The inclusion in Eclipse is done via Project gt Properties gt Java Build Path gt Add JARs or Add External JARs depending on whether the user included the ASCLogger library into the project or is loading it from the SOTA directory Information about the individual test cases 1 e project name test name description and used IScheme are provided via the initialization file named Execution Directory of the Test Program Base Directory of the Test Program Base Directory of SOTA Dynamic Program Test Instrumentation Scheme IScheme Start Script Batch File Static Program Analysis ASCLogger ini which is created on starting the test written into the execution directory of the test program and then read by ASCLogger The execution directory of the test program is the directory from where the program is started 1 e the directory where java cp classname is executed or when using a start script the directory containing this batch file In RCP development with Eclipse the RCP program 1s started from the base directory of the platform 1 e Eclipse In this case the execution directory is eclipse The ASCLogger ini is put into the execution directory This file contains information about the test for the logging component The log files are also written into this directory The base directory of the test
37. Class semorg gui util ConfigureListDlg OOO ooo ooo 000 000 EE N NN Glass semora sahaccess DBAccess eer legal sooo orma sna Class semora saluti DBConstrant s000 ers0 sooo 60 00 Class semora saluti DBTableChangedustense E Cass semora gulutilDelavedPaymentitem NN Class semorg quiorovider DelavesPaymentitenTableProvder 40 00 EE ass semora guiar earomanases N SHR EDE Class semorg quiEnumerationiados E Cass semora saltablesinstoctor MM o Cisssemorgigui items E E E EEE EEE DD Class semorg gui provider LecturerTableProvider 92 19 66 67 88 57 Class semorg gui LecturerWindow 0 00 0 00 0 00 0 00 JO 66 67 Class semorg gui list ListWindow 80 00 85 83 100 00 ol1c Class semorg gui MainWindow 85 71 98 05 83 33 50 00 71 43 71 43 50 00 50 00 Class semorg gui util Messages 50 00 83 33 75 00 26 67 12 36 Class semorg sql tables Person 47 83 54 29 26 67 Class semorg sal tables Presentation 6 67 Class semorg gui list PresentationListWindow 0 00 0 00 Class semorg gui provider PresentationTableProvider 72 97 66 67 29 41 2 15 24 39 13 46 29 41 88 57 60 00 42 86 50 00 12 36 1 56 75 00 50 00 26 67 3 70 78 57 50 00 26 67 3 10 66 67 8 51 29 41 50 00 5 00 50 00 50 00 5 00 50 00 Class semorg sql tables PublicPresentation
38. Dir PCDATA gt lt ELEMENT AntLocation FPCDATA gt lt ELEMENT AntBuildFile FPCDATA gt lt ELEMENT RunScript PCDATA gt SOTA Coverage Report Project husemorg created 2009 03 23 14 14 49 Function Entry Exit Coverage FEEC 35 46 Files 17 Statement Coverage CO 37 99 Classes TopLevel inner Classes 420 77 343 Decision Coverage C1 26 29 Functions 1379 Condition Coverage C2 20 42 Lines 31915 Minimal Multiple Decision Coverage MMDC 20 12 Statements 10747 Modified Condition Decision Coverage MCDC 8 67 Conditions 1178 Multiple Condition Coverage C3 20 21 Modified Boundary Interior Path Coverage 0 00 ModBI Boundary Interior Path Coverage BI 0 00 Tests 1 A L 2 AA husemorg test 1 Level 2 instrumentation Coverage of Classes A EAS up DEE Project husemorg Class semorg sql tables AbstractTable 58 06 Class semorg sql tables Associate 56 25 67 71 Class semorg gui list AssociateListWindow 50 00 50 00 50 00 86 05 90 00 89 19 75 00 75 00 50 00 75 00 86 49 86 49 Class semorg qui provider AssociateTableProvider Class semorg gui AssociateWindow 96 00 90 84 70 24 52 38 58 95 Class semorg gui util AssociationTabControl Class semorg sal tables Booking 50 00 52 67 Class semorg qui util CalendarControl Class semorg sal tables Client Class semorg sal tables ClientBooking Class semorg gui list ClientBookingLis
39. E Source E LFL Coverage FI Metrics Mame Project Digit 2 LI Digit Java S 0 Digit main String 1 00 a evaluateDigitSegu 66 67 4 2 7 View Metrics FEEC 80 00 80 00 80 00 co 39 47 39 47 39 47 38 89 cl 30 00 80 00 80 00 op 005 C2 75 00 75 00 75 00 75 00 Fig view Coverage MME 68 75 68 75 68 75 68 75 JE E SOTA calculates a number of static metrics during the syntactic analysis of the source code while parsing These can be evaluated immediately after the project was read in the view Metrics Similar to the view Coverage this view consists of a project tree and a mapping of the following metrics to each project unit cyclomatic complexity essential complexity lines of code number of instructions number of branches number of modified boundary interior path segments number of boundary interior paths number of instructions evaluating conditions number of elements in all conditions and number of conditions Explanations as well as definitions of individual metrics are listed in the Annex As in the view Coverage the user can change the presentation of the project structure in the menu of the view and expand and collapse the presentation by clicking the buttons E LI Expand AIT and a KE Source L LFL Coverage Metrics Cyclic lt El E Digit java a SS Digit lt 7 main Strina 1 evaluateDigitSeguence Cp
40. Gaal is activated in the toolbar the values will be highlighted in favour of clarity The individual colors and thresholds can be defined in the preferences The listed coverage metrics are Function Entry Exit Coverage FEEC instruction coverage CO branch coverage C1 single condition coverage C2 minimal multiple condition coverage MMDC modified condition decision coverage MCDC multiple condition coverage C3 modified boundary interior path coverage MBI and boundary interior path coverage BI The definitions of the individual coverage metrics are listed in the Annex If a line instead of a value is displayed in the view the corresponding metric is not appicable to the structure because the class does not include e g conditions or instructions The user can change the Da Change Info from percentage to ratio representation and back again With the two buttons E Expand All and presentation of the value itself via the option Collapse All the hierarchical presentation of the project structure can be fully collapsed or expanded The user can sort the entire table by each column i e by name and coverage ratio In order to do that it is necessary to click the corresponding column head The sorting is based solely on the root elements of the tree 1 e in hierarchical representation on the values of the files and classes respectively However in flat representation the structures can be sorted by functions K
41. Metrics DEE S A evaluateDigitSeguence String DIDOU 2 O if if O 0 private static double evaluateDigtSequence String inDigitSeguente t 612168 double wert 0 0 Fig view CFG If test logs or test case data are selected in the view TestLogs the color of the nodes and edges will change according to the coverage of the selected test case data Also the coverage data is updated automatically Covered nodes and edges are colored green whereas not covered ones are colored red To detect covered nodes whith multiple outputs of which not all are covered which might be the cause for not covered sections of the source code these are colored yellow The user can configure this just like all other colors and also line style and thickness in the preferences The user can obtain additional information by navigating the cursor over a node The tooltip that opens indicates the type and number of hits nrHits by test case data of each node as well as its internal project ID and the line number where it can be found Branching nodes also contain iteration nrHits 5 nrSkips O nrSingleLoops 3 nrMultipleLoops 2 information about the number of times each branch was taken Therefore e the tooltip for if nodes lists a value for each the true and the false branch ID 22 Line 15 whereas switch nodes contain an overview of the chosen selections and a Fig tooltip list of the number of times each case was called
42. PETT EVER nooo 10000 cwo ECC ECC ESA poso 10000 Fey Stine oo eel 100 00 10000 Tsao E E 0000 10000 HETTIE NN EN ooo nooo 10000 PESTO iso osso roc HEET NN EYE EN nooo 10000 BESSE ooo poso 10000 FseGpdeparment tsin 10090 0000 Bono 10000 Fetcpemaio EE E nooo 10000 HENN EVER ooo 10000 H EN EIEN roue HE sun EN osso ooo 10000 DESEO E rono nooo 10000 FseGprirsname srn EYE EN aooo 200 00 BESCHE ooo ooo 10000 Fsetcpmobie Sini Jaonoofuoof nooo 10000 sacon E EH 0000 10000 BEE oso ooo 10000 Faetcpttone o EC ECC e soas 10000 HEET NN EYE EN ev 10000 DESEE i ooe ioooo nooo 10000 HETTIE INN EYE EN nooo 10000 T 7 EIEN 10 00 10000 H EN ooo ev 10000 Coac ECC osso nooo 10000 seme sre EC757 ooo gaas 10000 PESE feoee osso 00 00 10000 FsetGustomersinee 0a 10090 ee roo 10000 aerie fee eel poso 10000 PET EC ooo 0000 10000 BESCHE E EEN EEN EN poso 10000 BE EYE e EEN poso 10000 Terraneo EC ooo 1000 10000 sarramen 175 ooo EE EE EC ECC e E nooo 10000 PET EYE EN poso 10000 H EC ECC e EE E E E CE sename sting ooo ooo E OE Comece EN EIEN E EE PESTO E ESO EEN EE
43. SOTA User s Guide Version 1 0 1 Introduction 2 Overview 3 Installation and Start of Program 4 User Interface and Functionality 5 Files 6 Tutorials 7 Appendix 1 Introduction SOTA is a tool for static program analysis and structure oriented program testing Structure Oriented Testing and Analysis In the course of a structure oriented program test the tool determines the source code coverage during the test calculates the corresponding coverage metrics and provides a visual representation of the results This allows the user to evaluate the program test with regards to source code coverage Source code sections that have not been covered or conditions that have not been tested adequatly can be identified easily SOTA 1s not directly responsible for testing the program but serves as a tool for evaluating test cases and developing additional tests SOTA determines the coverage by source code instrumentation ie the program to be tested must be available as compilable source code SOTA 1 0 only works on Java programs but was developed to be able to support all major imperative and object oriented programming languages To use SOTA for other programming languages the user has to provide a parser as well as various classes for mapping the structure of the programming language to a more abstract structure This is specified in detail in the developer documentation The program was developed as a standalone Eclipse RCP application and run
44. View CFG the function node indicating the overall modified boundary interior path coverage the coverage of modified boundary interior paths by the function body and the boundary interior path coverage Similar to the second label each iteration node also receives a label for the coverage of modified boundary interior paths corresponding to the iteration The last option causes the remaining three coverage metrics FEEC CO C1 to appear at the function node In the second block the user can configure the presentation of the nodes of the control flow graph Colors for normal nodes without coverage information covered and not covered nodes can be selected For a more differentiated presentation branching nodes which are covered but whose outputs are not covered completely are highlighted The color of the highlights can be set under Branching CFG nodes Removing the tick on the option above switches the differentiated presentation off Additionally in the third block the color of the edges of the control flow graph for the presentation of edges without coverage as well as for covered and not covered edges can be configured The line thickness of the edges can also be set to a value from one to three Finally the user can also change the color for highlighting lines in the source code corresponding to the selected node of the control flow graph 4 3 2 Preferences View Coverage Here the colors for highlighting the percentage of the c
45. age metrics i static metrics report htm Create Report Fig DF diagramm Manual Test with Scripts Detailed Procedure The detailed procedure matches the procedure of the manual test with only a few exception which are described below 1 Create Program The program HUSemOrg should be unpacked into the Eclipse folder workspace and is already adapted to usage with SOTA As in point 6 1 2 a project husemorg has to be created in Eclipse and the library ASCLogger jar added to its build path 2 Preparatory Phase Create Project 3 Preparatory Phase Instrumentation 4 Testing Phase Compilation The next three steps only refer to the preparation in SOTA and are identical to the manual test 5 Testing Phase Program Test After the instrumented source files have been compiled ATOS is started for the program test itself The regression tests are performed on the instrumented classes SOTA may be closed for this The usage of ATOS is in no way affected by the preperation phase with SOTA there is no interaction between the two programs 6 Evaluation Phase Reconstruction 7 Evaluation Phase Evaluating the Test 8 Evaluation Phase Create Report The remaining steps match the description in 6 1 2 6 3 Automatic Testing System SOTA ATM SOTA ATM Automatic Test Module is a library containing the test functionality of SOTA and needing no GUI at all This permits controlling the instrumentation of projects as well as
46. age value is naturally limited to 1 e ModBI traversed MBI paths possible MBI paths 7 2 9 Boundary Interior Path Coverage Bl The boundary interior path coverage is computed like the modified boundary interior path coverage However the BI paths are computed only for the paths going through a funtion and then this value 1s compared to the number of possible BI paths e BI traversed BI paths possible BI paths 7 3 Instrumentation Level In order to allow the user to limit the memory requirements of the log files sensibly and variably the source code can be instrumented in different levels A configuration of the instrumentation is combined in an instrumentation scheme short IScheme and saved for the corresponding project For all projects three basic ISchemes which correspond to instrumenting the code according to the respective levels are provided by SOTA Level 0 Assigning level 0 as an instrumentation level for a structure causes this structure to be excluded from the instrumentation This 1s sensible for functions which create a lot of log information due to frequent execution or complex function flows but have been tested adequately and can be excluded from further testing Level 1 The basic instrumentation is offered via level Here all function entires exits and all branching structures are instrumented so that the control flow through the functions 1s can be reconstructed from these data With these data it is
47. and creates a report file in the root directory of the test program 6 1 3 SOTA with Ant build file and start script General Procedure SOTA offers the possibility to execute the manual program test in SOTA by using two scripts The procedure matches the procedure described in point 6 1 2 in principle but the third phase the testing phase is not done in Eclipse which is only needed for creating the project The following data flow diagram represents an overview of the steps taken in SOTA and the points at wich the scipts complement the procedure Eclipse work space ti fer SOTA ASOT A create project Digit create class Digit export build xml create project Digit se CLogger ar Digit project Digit ischeme Digit java L import Digit java and parse GUI Digit java Digit java backup sourcelode CFG i Static metrics ait Let e start program test Digit java instrumented test name 11 Scheme level 2 GUI Digit java mst sourcecode CFG Static metrics compile Digit ant build mil start program Digit Domo ie Pa finish program test Digit Java NY restore original GUI Digit java read test lags i with coverage and evaluate sourcelode CFG coverage metrics coverage metrics static metrics start hat GUI Digit java coce Code CFG static metrics create report Abb DF diagramm manual test with scripts Detailed
48. ant buildfile is provided i Levell Level2 Level3 lt ischeme name gt instrument sourcefiles according to chosen level or IScheme n lt name gt lt lang gt lt project dir gt lt exec dir gt lt src dir create new project file p lt name gt project open the project file r lt report file gt create report file if no name is provided it will be stored in report html s lt runscript gt start project only if runscript is provided t lt testname gt log lt testname gt log name of testlog to create or to import sz restore original sources Parameters detailed c lt ant buildfile gt The parameter c evokes the compilation of the project after the file operations have been completed This option requires the project file to contain a reference to Apache Ant and to an Ant build file The build file may also be passed via command line right after c i Levell Level2 Level3 lt ischeme name gt With the parameter i all sources of the project are saved as a backup and then instrumented It has to be followed by either a name of an IScheme specified in the project file or one of the values Level Level Level3 which stand for a complete instrumentation of the entire project according to the respective levels The project will only be instrumented if none of the source files is instrumented already The parameter i requires the declaration of a test name via the parameter t and
49. ate a project in SOTA for the test program Selecting the menu item ES Ci New Project opens a two page wizard guiding the creation process On the first page the name of the project Digit as well as the root directory project directory of the proejct created in point 1 have to be specified The execution directory of the project is automatically linked to the same directory by SOTA needing no user input since both directories are identical in this case The first page of the wizard is now completed and clicking the Next button calls the second page On the second page all sources of the project need to be imported The project Digit only consists of one file so marking the root directory is sufficient The Finish button closes the wizard and then SOTA reads and parses the source file Saving the project succesfully completes the creation process When SOTA is started again the project can be loaded via the menu item Open Project Immediately after creating a project or loading an exisiting project respectively the source code can be viewed in the view Source and the control flow graph of each function is visible in the view CFG after selecting one in the view Project The static metrics that were computed while parsing the sources are now listed in the view Metrics Ei New Project Project configuration Please enter properties of the project Project name Project directory Di Developmentiworkspace Digit Execution directory
50. ava Expand all Collapse all Fig second page of wizard Open Project Clicking the Open Project menu item opens a project created previously A standard dialog for loading a file from the root directory of SOTA appears where the desired project can be selected After confirming the selection by clicking OK SOTA opens the project and loads all the associated source files Open Project Suchen in KE configuration V i DI plugins Zuletzt workspace verwendete D husemoarg project MOdBIPathTesterz project E ModbiPathTester project E Digit project LI 7 is f T Eigene Dateien Arbeitsplatz E a Dateiname Digit project fnen AAA me Hetzwerkumgeb Dateityp Project files project Abbrechen Fig dialog Open Project Save Project Lal Saving the project by clicking Save Project creates if done for the first time a project file in the root directory of SOTA The name of the file is identical with the project and ends on project This file contains the project data and all instrumentation schemes If the file already exists it will be overwritten with the current data Close Project Clicking the menu item Close Project closes the current project Consequently SOTA is in its starting state
51. ble click on the appropriate node Here the value is listed under ModBl In the node info of the function node the value for the entire function is listed as well as the values for subpaths of loops and subpaths through the entire funtion 7 1 7 Number of Boundary Interior Paths BI Analogous to the metric above here the number of boundary interior paths is specified for each funtion and accordingly for classes files and the project the sum of all values contained in them The corresponding paths are defined as all executable paths through the funtion in which the limit of the number of paths applies so that on occurence of loops only those paths need to be tested where for each loop e the loop is skipped i e the loop body is not executed impossible for do while loops e the loop body is traversed exactly once e the loop body is traversed at least twice only regarding the first two iterations 7 1 8 Number of Statements with Logical Conditions ConditionStmts To compute the number of statements with logical conditions all occurences of statements with evaluable logical conditions in the source code are summed up Infinite loops while true and loops iterating over a set for Item item set are not counted explicitly 7 1 9 Number of Logical Atoms Atoms This metric corresponds to the sum of evaluable atomic conditions from all logical conditions The logical atoms true and false are not counted since they are no
52. corresponds to workspace project name in Eclipse projects This directory contains the imported source code as well as the reports created by SOTA at the end of a test In the run directory the test program is started In most cases the run directory corresponds to the home directory of the test program except for Eclipse RCP projects where it is the Eclipse base directory eclipse In the run directory the ASCLogger ini is created which is read by the ASCLogger during the test After the test the testlogs are read from the run directory as well Choosing a programming language in the wizard is necessary even though the current version of SOTA only supports JAVA Finally the user can include an Ant build file for compiling and a batch file for creating the test project On the second page the user has to include the source files to be considered for the test They can either select a directory thus including all source files in all sub directories or simply individual files After finishing the wizard SOTA creates the project and loads all the selected source files fi New Project Project configuration Please enter properties of the project Project name Project directory Execution directory Language Cancel 19 New Project Source Files Please select the source files to import into this project Project directory Di DeveloprentimorkspacelDic it 5 Oe RES TG bin CIE lib iel src Digit j
53. createListWindows centerShell createStatusBar getShell getinstance op prec co a wmoc wese o moss a Class semorg gui util Messages top FEEC CO Cl C2 MMDC MCDC C3 ModBl BI Class semorg sql tables Person Person int String String String String String String String String String String String String String Date Date String String Timestamp Timestamp getAnnex getBirthday getCity getCountry getEmail getFax getFirstname getMobile getName getNotices getld getPhone getSalutation getShortinfo getStreet getTitle getZipCode getFirstContact setAnnex String setBirthday Date setCity String setCountry String setEmail String setFax String setFirstContact Date setFirstname String setMobile String setName String setNotices String setNumber int setPhone String setSalutation String setShortinfo String setStreet String setTitle String setZipCode String tableOK createPersonTable Statement insertIntoDB updateDB EE EER removeFromDB Iterable getColumns e EF EE hashCode Pc EN AE EN E CR a o wwoc wee mw a E Class semorg sql tables Presentation 36 19 25 19 6 67 3 70 3 70 0 00 3 70 1 56 1 56
54. e project files used by SOTA are simple XML files which contain the project specific information as values of individual entities Their format is specified by scheme definition project dtd The project file defines a project which is at least defined by the following values Name The name of the project which must be identical with the filename without ending Language The language of the project which must be listed in the language specification Prefix The prefix which enables marking variables introduced by SOTA during the instrumentation process this avoids name collisions BackupExtension The ending used for backup files generated by SOTA ProjectDir The root directory of the project ExecDir The execution directory of the project SourceFiles A list of source files as SourceFile belonging to the project The following values may be used optionally to enable special features of the program AntLocation The path to the Apache Ant installation This is necessary to compile the project automatically AntBuildFile The Ant buildfile which enables the compilation of the buildfile RunScript The script used for booting the project Schemes A list of instrumentation schemes as IScheme which enable the variable instrumentation of the project An IScheme consists of a name and a mapping of each structure of the project data class function to an instrumentation level and optionally a description of the IScheme The followi
55. ecision Coverage MCDC couples atoms 7 2 7 Multiple Condition Coverage C3 The multiple condition coverage test requires the most comprehensive condition test since all truth vectors of every condition need to be tested This means the costs for the test grow exponentially with the number of conditions Additionally in most cases it is not possible to apply all combinations of truth values However these impossible combinations usually cannot be recognized easily The costs for testing 2 atoms is merely reduced by using short circuit operators which stop evaluating the condition as soon as the result of the complete condition was determined irrevocably e C3 evaluated truth vectors possible truth vectors 7 2 8 Modified Boundary Interior Path Coverage ModBI The modified boundary interior path coverage test 1s a test proposed by Liggesmeyer which reduces the test cases compared to the boundary interior path test see definition in 7 2 9 In order to compute the coverage metric it is necessary to compute the MBI paths covering the paths for every one of them using a function during program testing Then the sum of these covered subpaths is compared with the number of possible MBI paths as defined in 7 2 9 Since the number of ModBI paths is only a minimum of possible subpaths according to this criterion in practice more MBI paths may be traversed e g due to exceptions than defined by this minimal bound In this case the cover
56. edMainTableListener createSpecializedMainTableListener AssociationTabControl createSpecializedMainTableListener ForeignKeyConstraintSelector createSpecializedToolBarListeners 50 00 createChooseButtonListener AssociationTabControl createChooseButtonListener TT elector SES gt os aa sooo saen ua Pes Peor RIERA semorg qui provider ClientBookingTableProvide dispose getColumnImage Object int getColumnText Object int addListener ILabelProviderListener isLabelProperty Object String o removeListener ILabelProviderListener to Class semorg gui ClientBookingWindow ClientBookingWindow Shell AssociationTabControl ClientBookingWindow Shell ClientBooking AssociationTabControl setlnput ClientBooking 0 00 setInputComponentsEnabled boolean 0 00 createButtonListener Composite AssociationTabControl checkInput confirmClose onSave top MMDC MCDC ModBl Class semorg gui list ClientListWindow 46 15 40 00 33 33 50 00 50 00 50 00 35 29 ClientListWindow ClientListWindow AssociationTabControl Ms A EN ClientListWindow ForeignKeyConstraintSelector AS createSpecializedMainTableListener createSpecializedMainTableListener AssociationTabControl createSpecializedMainTableListener ForeignKeyConstraintSelector createSpecializedToolBarListeners 50 00 createChooseButto
57. ent Coverage CO For the statement coverage it is necessary that every statement in the source code was executed Since each statement is only listed in the log file after its execution when the source code has been instrumented according to instrumentation level 3 the statement coverage is usually determined from the logged key data of the control flow after the program test e CO covered statements statements Note In the view CFG not all nodes correspond to statements and not every statement corresponds to a node Therefore the CO coverage cannot be computed from the covered nodes of the control flow graph it is rather based on the value Statements from the view Metrics 7 2 3 Branch Coverage C1 The full branch coverage was reached if all branches of the control flow graph are covered Computing the percentage coverage is done differently in practice to simplyfy matters SOTA computes this on the basis of the branches cf 7 1 5 as follows e Cl covered branches branches 7 2 4 Simple Condition Coverage C2 The simple condition coverage excusively tests whether all logical atoms of the conditions were evaluated true as well as false However this does not mean that branch coverage was reached as a minimum goal therefore it is hardly possible to draw any conclusions from the simple condition coverage For computing the percentage coverage SOTA counts all evaluations of each atom and compares them with the target value
58. f source code external compilation of with P SOTA external external source code using a script testin RE external 9 with start of SOTA external external program i using script restoration of original source SOTA SOTA SOTA ATM code reading of log files and calculation of SOTA SOTA SOTA ATM coverage metrics visualization of nois SOIA SOTA SOTA after the results EE report evaluation phase SOTA User s Guide Version 1 0 1 Introduction 2 Overview 3 Installation and Start of Program 4 User Interface and Functionality 5 Files 6 Tutorials 7 Appendix 3 Installation und Start of Program SOTA was developed as a standalone Eclipse RCP application The only requirement a system must meet to run SOTA 1s an installation of Java 6 0 or higher A simple calling of SOTA exe starts the program Starting SOTA for the first time configures the Eclipse Rich Client Platform EH sota configuration manual plugins workspace EE eclipseproduct a 45 Logger jar E langspec dtd ES language spec E project dtd E soTA exe E sota log La SOTA ATM jar fig SOTA directory structure Depending on the exploitation method it is necessary to install other systems e g Eclipse or ATOS SOTA User s Guide Version 1 0 1 Introduction 2 Overview 3 Installation and Start of Program 4 User Interface and Functionality 4 1 Menus und Toolbar 4 1 1 Menu Project 4 1 2 Menu Ta
59. h can assign a different instrumentation level to each structure file class function of the test project The overall instrumentation of the project according to one instrumentation level is available via a GloballScheme Sample Implementation of the Manual Test of HUSemOrg The following java program code reads the given project file starts a test with instrumentation level 2 compiles and starts the test program After completing the test all source files are restored to their original state the project is recompiled and an html report is generated from the imported test String projectFile D Development eclipse husemorg project String reportFile report_test_1 html String testName husemorg_test_1 String testDesc Test 087 1573 TreeSet lt String gt testSet new TreeSet lt String gt testSet add testName SotaATM atm new SotaATM fileName atm startTest testName testDesc new GlobalIScheme Level2 2 true atm stopTest testSet true atm createReport reportFile SOTA User s Guide Version 1 0 1 Introduction 2 Overview 3 Installation and Start of Program 4 User Interface and Functionality 5 Files 6 Tutorials 7 Appendix 7 1 Static Metrics 7 2 Coverage Metrics 7 3 Level of Instrumentation 7 4 More Terms and Definitions Glossary 7 Appendix 7 1 Static Metrics In SOTA static metrics sums up all metrics of the project that are obtained by static analysis of the source code
60. he following list to be included in the report the applied tests with Schemes and descriptions an overview of the coverage of all classes including or excluding inner classes and or an overview of all functions sorted according to their classes Finally the font size for the report file can be specified If Use Colors is selected the coverage metrics in the report file will be highlighted in color according to the degree of coverage similar to the presentation in the view Coverage The values for the bounds are adopted from the preferences item view Coverage but here the colors of each bound can be defined separately for the report file 4 3 5 Preferences View Source The highlighting of syntax in the view Source can be adjusted in the correspondent preferences item The colors for the ii Preferences type Filter text SourceYiew CPaview Preferences for the Sourceview Coverageview Candia Keywords keywords of the Comments language comments no strings and those comments added 45C comments by Indent size instrumentation through SOTA visited source lines can be chosen arbitrarily and subsequently the font size for the presentation of the source code Restore Defaults Apply can be defined Not visited source lines To view the coverage of the source code in the view source 1t 1s possible to select the background color for covered and not covered source code lines here The colored highlighting
61. in the program flow always has a cyclomatic complexity of 1 and each branch e g an if statement increases the cyclomatic complexity by 1 7 1 2 Essential Complexity The definition of the of the essential complexity is closely related to the cyclomatic complexity After recursively deleting all primitive control structures from a given control flow graph G the cyclomatic complexity of the resulting graph G 1s defined as the essential complexity e G of the graph G e G 7 G All simple structures which contain no jumps with the exception of break instructions in switch statements are considered primitive structures The existence of jumps out of control structures makes these structures and all structures including them irreducible thus increasing the value of the essential complexity 7 1 3 Lines of code LOC The number of the lines of code 19 listed here as one of the most primitive metrics of the source code encompassing the appropriate structure In contrast to all other metrics SOTA computes LOC strongly depends on the structure of the source code and also the commentary Therefore it should be regarded with care 7 1 4 Number of Statements Statements Unlike the Lines of code metric the number of statements offers an objective format independant metric for the extent of the project In order to compute this metric all executable statements are summed up for all structures The test of the coverage of statements consists of
62. ions If instructions for instrumentation are detected on parsing the file e g during a test run on instrumented sources the Icon will be displayed with a red exclamation mark e g De Via the menu selectable on the TS Project LEE El Diop main String E Digit evaluateDigitSeguence String top right corner of the view the presentation of A lt Hierarchical Presentation 2 Flat Presentation Show PackageMames the project T Show Files structures can be v 3 show Classes configured The v Show Inner Classes EWO presentation i options Fig view menu s Flat Presentation and ES Fig view Project Hierarchical Presentation are available The hierarchical flat presentation without files presentation matches the listing in a tree structure as described above which is the default setting In the flat presentation all structure units are at root level 1 e file classes and functions are listed coequally The additional options allow the following adjustments of the presentation Show PackageNames adds the packet name to the name of the structure default off Show Files additionally lists files in the view default on 3 Show Classes lists classes in the view This only affects the flat presentation whereas the hierarchical presentation always lists classes default on a Show Inner Classes lists inner classes in the view default on 4 2 2 View TestLogs If log fi
63. ise is Digit Then the program starts For the following program starts Eclipse should always choose the selected start configuration so itis sufficient to click the button G now labeled with Run Digit Select a way to run Digit Java Applet Java Application Digit default package Fig dialog Start Test The parameters for the program start in Eclipse could be written into the start configurations of the project but since this is quite intricate for the simple test it is recommended to change the string in the source code line System out println evaluateDigitSequence and then start the program without parameters The console in Eclipse should now show the succesful initialisation output of the ASCLogger as well as the result of the evaluation of the string Additionally the corresponding log file with the name of the test should appear in the project overview of Eclipse after the first program test Repeating the test adds the new log data to this file Remark It is also possible and often more convenient to execute the test cases in an automated way so that you don t have to enter the test data manually over and over again Eclipse supports the creation execution and evaluation of JUnit test cases The procedure is the following e Add test file to Eclipse project in a way that Eclipse will consider it a test file i e use option New JUnit Test Case e Define a name for a file as DigitTest If
64. it with a main function and the function evaluateDigitSequence which evaluates the string The string can either be passed as a parameter of the program or provided in the source code The hardwired string in the source code will be evaluated once the program is called without parameters The output of the program is either the number in case the program was able to evaluate the string or 1 in case of an error Le the string did not contain such a number public class Digit 4 public static void main String args if args length 0 System out println evaluateDigitSequence else System out println evaluateDigitSeguence args 0 private static double evaluateDigitSequence String inDigitString double value 0 0 double accuracy 1 0 String where prePoint boolean errorfree true int position 1 while position lt inDigitString length amp errorfree String chr inDigitString substring position 1 position if chr matches 0 9 if where equals postPoint accuracy accuracy 10 0 value 10 0 value Double parseDouble chr else if chr equals where equals prePoint where postPoint else errorfree false position 1f errorfree inDigitString length 0 where equals postPoint amp inDigitString length 1 return 1 0 else return value accuracy 6 1 2 SOTA and Eclipse General procedure The program Digit is supposed
65. ithout having to instrument and compile it again Build Project causes SOTA to compile the source code after the instrumentation For that purpose it is necessary to embed an Ant version by providing the path to the file ant bat under the point General of the preferences Furthermore the user has to include an appropriate xml file enabling SOTA to compile using Ant This option is only available if a script exists and the option Rerun configuration has not been selected Finally the option Run project actually causes the project to start after compilation as long as a startup script has been specified This will be possible only if one of the other two options are selected If a test is running already or SOTA finds the source code instrumented during parsing the option S Start Test will be exchanged with JE Restart Test It 1s not possible to re instumentate the sources without having stopped the current test and restored the original files However The instrumented test program may be started again with a different name for the test This only updates the initialization file of the logging component with the new name for the test but no source files or binaries are changed Restarting the test is equivalent to the rerun option of the normal Start Test dialog Stop Test The Stop Test menu item terminates the current test run Then the original source files are restored and re read If a corresponding Ant buildfile is available the origi
66. jo 0 00 0 00 000 0 00 000 0 00 Ha na createWindow Shell EUR 00 00 SS 33 33 MEEN confirmclose 80 00 76 92 66 67 MEE SE RS e Ed fa _ Eds 0 ea Smee NO ones o Doom 0 00 000 000 0007 Coco 0007 ModBl EH aeaaee o a HET E meewerk ow oo 000 000 000 000 000 000 000 000 4516 3333 2308 2508 S pete Le 5 LL ns mer coan o TR a So OIE EE GEN E 00 25 32 18 52 7 50 500 000 7 50 405 3 connectInDB 2 82 3 50 25 00 16 67 0 00 25 00 0 24 0 24 50 00 5000 sooo E 50 00 5000 50 00 ENE ENE IE setData String String addltemListener ItemListener OO removeltemListener ItemListener fireltemChanged 000 0 00 000 addUpdateListener 58 95 to FEEC ST ai ole moc mene o spa 3077 2092 2092 I 2597 1990 9807 sai EE es Pegco ae Mao 1 o b b b b 0 Class semorg sql tables Booking 52 38 58 95 Booking int Date Date Date Date Date i D n ct O O D un ct O 3 TD tableOK createBookingTable Statement getConfirmed setConfirmed Date getEnrolled setEnrolled Date getMessaged setMessaged Date getSignedOff setSignedOff Date getld Feste NE n equa ione 6667 6667 50 00 5000 50 00 EER 50 00 5
67. le value 0 0 double accuracy 1 0 Sting where preP oint boolean errorfras true int position 1 alulerposton lt UO SES ins lengdio Seervorfiaa 1 Sting echo inDipit tving substringipositton 1 position PLS clin nateliaso 0 9 4 fah aig quale poste ont accuracy accuracy 10 0 value 10 0 value Double parseD oublefzchn else iflzcline quale dr where agualsi preP oint where portPoint alza avoifree false position fieros inlimitftung engivi 0 were equal post oint rand ist ming lengdiu ss lnn return 1 0 alza return value accuracy I L Fig view Source 4 2 5 View CFG While the view Source only shows line coverage and is primarily intended to give an overview of the source code coverage the view CFG Control Flow Graph provides detailed information on the coverage of source code structures On selecting a function in the view Project the upper part of the view CFG shows the corresponding control flow graph For every function appears at least one node for entering and one for exiting the function with the latter being the joint of all edges leaving the function Each branching structure is represented by a node labeled as follows e conditional statement short variant if true 1f end e conditional statement long variant if true false 1f end mon mn e pre test iteration while and for loop iteration iter body Iter end e post test
68. les are imported into the project they will be listed in the view TestLogs In case a log file already exists the 23 Testlogs gt Ischeme X GE amp new log data are appended to its end so that a test log can ales contain a variety of test case data As a default the a i test 2 individual test cases are hidden from the user and only the gt B test logs are listed In order to examine the content of the D 1 Level 2 instrumentation test logs in detail the user has to unlock the button O 2 Level 1 instrumentation 3 main ell wertezifFernfolge us Lock TestLogs on the toolbar of the view which lists all test D test 1 1 cases as well as Schemes of every test log right beneath D test empty each one I husemorg test Log invalid log Since not every log file may have been created by SOTA and not all the test data belongs to the current project the view TESTLOG uses the following icons to represent compatibility Abb View TestLogs fully compatible test log and test case data unlocked U partially compatible test log also contains invalid test case data invalid log file or incompatible test case data If valid test logs and test cases are selected the coverage information in the views Source CFG and Coverage will be updated automatically Clicking the buttons and changes the entire selection To select several test logs at once the keys lt Shift gt or lt Ctrl gt respectively have to be held
69. les of the report file will only be created if the user selected the corresponding options in the preferences The standard settings cause the output of all tables If Show Testlogs is selected a list with all test files used for this report follows The corresponding ISchemes as well as their desciptions are also included Activating Use all tests for report causes all imported test logs to be used for the report and to be listed here The option Show Classes generated a table which lists all classes of the project in addition to the overall project and their individual coverage metrics which may be highlighted in color according to their values cf Preferences Show Functions causes a table for each class to appear after the item Detailed Coverage The tables contain the coverage metrics of the classes themselves and all their functions A link from each class in the table Coverage of Classes refers to the corresponding list of their functions in the section Detailed Coverage A comprehensive sample report for the project HUSemOrg is included in the user documentation SOTA Coverage Report Project Digit created 2009 03 23 13 54 35 Function Entry Exit Coverage FEEC 100 00 Files 1 Statement Coverage CO 100 00 e 1 1 0 Classes Decision Coverage C1 100 00 Functions 2 Condition Coverage C2 95 00 Lines 35 Minimal Multiple Decision Coverage MMDC 93 75 Statements 19 Modified Condition Decision 50 00
70. lso a setting for preventing SOTA from overwriting report files In that case a new file is generated for each report according to the format report lt date gt lt index gt html These and other settings for creating reports can be made in the preferences menu under the item Report Build Project The menu item Build Project enables compiling the project in SOTA on the basis of the current source files regardless of the test status This menu item will only be activated if an xml buildscript is specified for the current project and an Ant version is included in the preferences Run Project 3 The menu item Run Project will be available if a startup script is specified for the current project After clicking this item the project will be executed Restore Sources E In order to translate the project into its original state it is also possible to select the menu item Restore Sources Unlike the Stop Test menu item the source files are only restored and read They are not be re compiled assuming a buildfile exists nor does the dialog for reading the test logs appear Read Logs With the menu item Read Logs the log files can be read manually regardless of the test status of the project For that purpose the same dialog as for the Stop Test menu item appears allowing the import of log files After selecting the log files from the run directory of the project SOTA reads the files and processes the test results Show Coverage
71. m when executed Therefore it merely has to contain a typical Java call Java cp classname in way specified for the project If the start script 19 included in SOTA the test program will be executable in manual program testing with SOTA Contrary to the dynamic program test the static program analysis 19 done without executing the project The information needed for the static analysis 1s determined only by parsing the program This way SOTA identifies ten different static metrics which provide information about the structure and the complexity of the program and their components respectively lt ELEMENT Project Name Language Prefix BackupExtension ProjectDir ExecDir SourceFiles ISchemes AntLocation AntBuildFile RunScript gt lt ELEMENT SourceFiles File gt lt ELEMENT Schemes IScheme gt lt ELEMENT IScheme Name Description Level0 Levell Level2 Level3 gt lt ELEMENT Level0 File Class Function gt lt ELEMENT Levell File Class Function gt lt ELEMENT Level2 File Class Function gt lt ELEMENT Level3 File Class Function gt lt ELEMENT Description PCDATA gt lt ELEMENT Name PCDATA gt lt ELEMENT Language PCDATA gt lt ELEMENT File PCDATA gt lt ELEMENT Class PCDATA gt lt ELEMENT Function PCDATA gt lt ELEMENT Prefix PCDATA lt ELEMENT BackupExtension PCDATA gt lt ELEMENT ProjectDir PCDATA gt lt ELEMENT Exec
72. mentation scheme from 0 to 3 can be assigned to each of these structures by clicking the matching button for the level after IScheme name mein_Ivli_evaluateDigit Sequence 2 choosing a certain structure Description Chosen levels of instrumentation Consequently each structure is main Level 1 colored according to their respective AA alaihi levels Furthermore the substructures associated with them receive the Files in project Expand all Collapse al same level of instrumentation The New Scheme individual levels have the following E Digit java effects I E Digit Cl main Strina e da evaluateDigitSeguence String Level 0 no instrumentation of the source code ES Level D No instrumentation leaves sourcecode as it is Level 1 only instruments blocks Gee to dE Level 1 Instruments cfg relewant blocks standard coverage analysis determining the control flow relevant coverage metrics Level Instruments also conditions enables analysis of condition com FEEC CO C1 MBI BD e 3 Level 3 Pull instrumentation every single statement is instrumented Level 2 additionally instruments all blocks enabling condition coverage analysis C2 MMCC MCDC C3 Level 3 instruments the complete source code 1 e evaluates all instructions and conditions Fig dialog New IScheme For the program test instrumentation level 2 19 recommended as it allows the identification of all coverage metrics If
73. mented depending on the state of the source files report html rot ne dsd The report files generated by SOTA lt antbuildfilename gt xml An possibly existant Ant buildfile allowing the compilation of the test program in SOTA For Eclipse projects 1t can be exported via File gt Export gt Ant Buildfile lt runscriptname gt bat A batch file for booting the test program which allows in case it is included in SOTA the manual program test in SOTA in correspondance with the Ant build script Execution directory of the test program Execution directory The test program is booted in the execution directory In most cases this directory corresponds to the root directory of the program An exception may e g be the testing of an Eclipse RCP application in Eclipse since then the execution direction is the root directory of the Rich Client Platform 1 e in general eclipse ASCLogger ini On starting the test this initialization file for the logging component is copied into this directory where it is accessed by the class ASCLogger jar in order to create the log file lt testname gt log The log files created by ASCLogger ini 5 2 Project File To use SOTA it is not necessary to adjust the project files However should the user wish to adopt SOTA ATM as an automatic test module it may be beneficial to create or change the project files manually or with a script in order to gain comprehensive control over the test Th
74. mnProperty int o a Je wee wese ever ai CN CE C ST RT BT ET ea mus ppa String getColumnType 00 0 a apo wee wese e we 5 oi getValue FEEC Class semorg sal util DBTableChangedListener 00 0 D OI DBTableChangedListener int getType CSI iN CI amp oe coe es meas Er Class semorg gui util DelayedPaymentltem DelayedPaymentltem int Date getPresentationld getBilled getDebt A Jo eo woe s oat a Class semorg gui provider DelayedPaymentitemTableProvider getColumnI mage Object int EEN EEN EEN EN NN getColumnText Object int addListener lLabelProviderListener dispose isLabelProperty Object String removeListener ILabelProviderListener to Class semorg qui util DelayedPaymentViewerControl 66 67 76 74 50 00 62 50 62 50 50 00 62 50 DelayedPaymentViewerControl Composite Shell addUpdateListener int Update top Class semorg sql util DistinctVector DistinctVector DistinctVector Collection ST Lo a Le wee wee OE se Jo Lo wee wee upa S oi insertSalutationStringInDB String insertTitleStringInDB String 0 00 insertcountryStringlnDB String 0 00 insertAnnexStringlnDB String 0 00 ModBl ELE EEN KE E EEN EEN EEN KR onEdit Enumeration i 00 100 0
75. nListener AssociationTabControl createChooseButtonListener ForeignKeyConstraintSelector openList getTablelnputFromDB createWindow Shell Je a me meoc cs mes 8 getColumnImage Object int EEN EEN EN EN addListener ILabelProviderListener EEN EEN EN EN isLabelProperty Object String removeListener ILabelProviderListener Ze rec co a ca woe wese cs vo Class semorg gui ClientWindow 55227 ClientWindow Shell ClientWindow Shell Client AssociationTabControl ClientWindow Shell Client ForeignKeyConstraintSelector 59 57 setInputComponentsEnabled boolean EEN EEN EN EN CreateButtonListener AssociationTabControl createButtonListener ForeignKeyConstraintSelector commitinputToDB confirmClose ea A EA Company int String String String String String String String String String String String String String String String String String String String String String String String Date String String String Float Date Timestamp Timestamp tableOK createCompanyTable Statement insertintoDB 50 00 updateDB 50 00 removeFromDB Iterable 50 00 getVector ResultSet 75 00 getVectorFromDB Vector String 00 EEN EEN EN EN getNext int getPrevious int getCompany int ee E hashCode em EN ECC e OC E Comer EC eel 0000 10000
76. nal sources Read TestLogs will also be compiled again Please select testlogs to import Finally a dialog appears allowing the user to select log files from the run directory which SOTA will then read analyze and list in the Directory Di Development workspace Digit view TestLogs Subsequently the coverage O test log metrics for individual test logs or O test 2 log combinations thereof can be computed and 0 test 1 1 log displayed O test 1 log test empty log Fig dialog Read Logs Create Report By clicking the menu item Create Report it 1s possible to create a coverage report based on the selected test logs for the current project The user can adjust the Select file name for report Suchen in C Digit wi E test 2 Jog W E test log Zuletzt L E test empty log verwendete D S classpath project a5CLoager ini P report html Desktop 18 report_2009 03 05 html report_2009 03 05_1 html report_2009 03 05_2 html report_2009 03 05_3 html Eigene Dateien preferences report_2009 03 05_4 htrnl so that for report_2009 03 05_5 html each report PE E test 1 1 log a prompt El test 1 log Arbeitsplatz appears for setting the file name a Dateiname and path Netawerkumgeb Dateityp e Abbrechen Otherwise the report is Fig dialog Create Report stored in the root directory of the project under the name report html In the preferences there is a
77. ncelFee setCancelFee Float getNext int getPrevious int getPublicPresentation int getCurrentEntrantsCount ModBl emorg gui list PublicPresentationListWindow 38 46 37 14 33 33 50 00 50 00 PublicPresentationListWindow 29 41 PublicPresentationListWindow AssociationTabControl PublicPresentationListWindow ForeignKeyConstraintSelector createSpecializedMainTableListener createSpecializedMainTableListener AssociationTabControl createSpecializedMainTableListener ForeignKeyConstraintSelector createSpecializedToolBarListeners 50 00 createChooseButtonListener AssociationTabControl createChooseButtonListener ME Ee elector openList getTablelnputFromDB 33 33 50 00 50 00 createWindow Shell EE 50 00 16 67 16 67 Class S ModBl i mmoc f meoc To Class getColumnText Object int semorg gui provider PublicPresentationTableProvider FCIS ModBl PublicPresentationWindow Shell PublicPresentationWindow Shell PublicPresentation AssociationTabControl PublicPresentationWindow Shell PublicPresentation ForeignKeyConstraintSelector setl nput PublicPresentation 0 00 setlnputComponentsEnabled boolean 0 00 createButtonListener AssociationTabControl 0 00 createButtonListener nO elector checkInput commitinputToDB confirmClose onSave ModBl
78. ng code is an exemplary project file for the project Digit It contains the defintion for an IScheme which instruments the only source code file according to level and the method evalutateDigitSequence according to level 2 lt Project gt lt Name gt Digit lt Name gt lt Language gt Java lt Language gt lt Prefix gt asc lt Prefix gt lt BackupExtension gt backup lt BackupExtension gt lt ProjectDir gt D Development workspace Digit lt ProjectDir gt lt ExecDir gt D Development workspace Digit lt ExecDir gt lt SourceFiles gt lt File gt D Development workspace Ziffer src Digit java lt File gt lt SourceFiles gt lt ISchemes gt lt IScheme gt lt Name gt Scheme F lt Name gt lt Description gt Digit java Lvll evaluateDigitSegeuence Lv12 lt Description gt lt Levell gt lt Item gt Digit java lt Item gt lt Levell gt lt Level2 gt lt Item gt Digit java Digit evaluateDigitSequence String lt Item gt lt Level2 gt lt IScheme gt lt ISchemes gt lt Project gt 5 3 Report File The following report is a sample report for the project Digit for the input values 2 1 1 1 and without input value for more information on the program see 6 1 1 Each report file begins with the name of the project and the date of creation Following that 1s a list of all coverage metrics including the values for this project obtained during the test as well as individual static metrics The other tab
79. ng the type of instrumentation and instrumentating the source files For a manual program test or the use of an external testing system this 1s done via the graphical user interface When using SOTA ATM with an automatic testing system the desired behaviour is evoked by calling the appropriate library routines or starting SOTA ATM via command line parameters During the testing phase the compilation start of program and program test are executed Compiling the instrumented source files 19 outside the remit of SOTA However the compilation can be initiated by integrating a corresponding Ant build script If a fitting batch file is available it is even possible to start the program with SOTA Given that these two files are included in SOTA the manual program test can be carried out without using an external development environment During the program test the instrumentations that were added to the source code produce a log file containing the data necessary for a complete reconstruction of the program Finally during the evaluation phase the original source code is restored the log files are read and the coverage metrics calculated SOTA enables the user to evaluate the results visually and export them as an HTML report Program Test with SOTA reading and parsing source SOTA SOTA SOTA ATM code SOTA ATM configuration y of SOTA SOTA instrumentation preparatory phase configuration file instrumentation SOTA SOTA SOTA ATM o
80. nt insertIntoDB updateDB getColumns getBio setBio String getDailyFee setDailyFee Float getHourlyFee setHourlyFee Float getNext int getPrevious int getLectI nstructing int getLectSupervising int getLectConducting int Ze mec co ca c moc meoc cs Mode e Class semorg gui list LecturerListWindow 50 00 9 4 LecturerListWindow LecturerListWindow AssociationTabControl LecturerListWindow ForeignKeyConstraintSelector createSpecializedMainTableListener createSpecializedMainTableListener AssociationTabControl createSpecializedMainTableListener ForeignKeyConstraintSelector createSpecializedToolBarListeners createChooseButtonListener AssociationTabControl createChooseButtonListener ForeignKeyConstraintSelector openList getTablelnputFromDB createWindow Shell ee o 3 3 3 3 3 3 3 3 FEEE co ci co Mme meoc c3 Model Bi Class semorg gui provider LecturerTableProvider Em EEGENEN EE de de CI oe CORE E EU EEN EER LecturerWindow Shell Lecturer AssociationTabControl setlnput Lecturer createButtonListener Composite AssociationTabControl w ee e A EIA EN Em ListWindow Shell Vector AssociationTabControl ListWindow Shell Vector ForeignKeyConstraintSelector EET ST RT ea ea mee wese ewer a MainWindow Display
81. o restoring the original source files on stopping the test Otherwise the binaries of the test program would remain instrumented and continue creating logs 4 3 4 Preferences Report Under the preferences item Report the output of the report in a html file can be configured ii Preferences type Filter text Report Fiview Preferences for the Report Coverageview cere Prompt for file name Activating the Overwrite existing report option Prompt SourceView a file nam E Use all tests for report the first block Show Tests causes a file Show Classes selection dialog Ignore inner classes to ap H SSES Show Functions selecting la Create Report in Use Colors the menu The ora dialog asks for ee the name of the Shige Ber report file which will be created Second Bar Otherwise Third Bar SOTA creates the file Fourth Bar report html in the root directory of the test program Above 0 overwriting it every time a new report IS Restore Defaults Apply generated or SOTA saves OK Cancel ed acu according to the name scheme Fifth Bar Cs Fig Preferences Report report_ lt date gt _ lt index gt html This behavior is determined by the option Overwrite existing report In the second block the user can define the content and the presentation of the report It 1s possible to choose between using all test logs for the report or only the currently selected ones Also the user can select elements from t
82. ocks standard coverage analysis for a selected IScheme The list Level O Mo instrumentation leaves sourcecode as it is ae Level 2 Instruments also conditions enables analysis of condition coverage can be fully expanded or i i collapsed by clicking the 2 Level 3 Full instrumentation every single statement is instrumented buttons Expand All or Collapse All respectively Rerun configuration Don t change sources or binaries Once a name for the test and an IScheme have been selected the program test can be started by confirming the dialog SOTA stores the information Fig dialog Start Test relevant for the logging component in the file ASCLogger ini by adjusting the execution path of the project Subsequently a backup of all original source files is produced by changing the file extensions to backup and the source code is instrumented Then the project is parsed again Finally the user can view the instrumented source code in the SourceView Now the instrumented version of the source code can be compiled and e g tested systematically with an external testing system Instruments the proiect with the chosen IScheme Three more options can be selected by ticking the check boxes at the bottom of the dialog Rerun configuration merely leads to a change of the name of the test and its description in the ASCLogger ini leaving the source code unchanged This enables re testing the code with the same configurations but w
83. onal ISchemes which Ee Level 3 instrumentation will then appear in this view via the menu item New Fry main lvil_werteziffernfolgedus_lvi2 IScheme Double clicking on an IScheme opens a dialog listing analogous to the menu item New IScheme stored instrumentation settings of the IScheme and enabling the Fig view Schemes user to change all information The button Delete IScheme removes an IScheme from the project The standard Schemes across levels 1 2 and 3 are exempt from these changes and cannot be deleted 4 2 4 View Source In the view Source the source code of the file selected in the view Project is shown If individual classes or methods are selected only the corresponding source code lines will appear In case the coverage indicator eni how Coverage in the toolbar is enabled the coverage of the source code by the test logs selected in the view TestLogs is indicated with colors Green lines in the source code were covered by the tests whereas red ones were not traversed A line containing several statements is marked green as soon as one of the instructions was covered The corresponding colors for the line coverage and the syntax highlighting can be set in the menu item Preferences Source K FG Coverage Ed Metrics public class Digit 1 public state void muni Sting wed system ont printinievaluate Digits e genee Wi private static double evaluateDisit 9 quencertStine inliait ing q doub
84. org sgl util UtilitySQL convertTosQLDate java util Date Ele E convertToTimestamp java util Date A AA parseFloat String 50 00 50 00 50 00 parselnteger String isCorrectlnteger String CastArray Object Class
85. overage in type Filter text Coverage view the view ii Preferences Oa Coverage can be i Preferences For the CFt View Coverage view defined as well General First Bar bounds Source vie Third Bar Two elemental Fourth Bar complete 100 coverage The First Bar EI user can add percentage values Second Bar for five WW Third Bar additional bounds with the Fourth Bar EI result that each Fifth Bar cell is assigned a eee color according above 0 EI to the respective EI values The default limits are 25 50 75 90 and 99 If Apply the user specifies a bound with a value larger than Cancel one of the Fig Preferences view Coverage bounds above 1t this bound will be ignored Subsequently the user can assign a color to each bound In the table of the view Coverage all cells where the percentage of the coverage reaches the specified bounds will be colored as defined 4 3 3 Preferences General The user can change the general settings of SOTA under General the preferences imi Preferences CF ie General Preferences item General In Ee H Ant erge to Report Create log file compile a SOUFCE VIE Overwrite existing log File project automatically _ Parse instrumented sourcecode the Automatically rebuild project on test stop corresponding Ant file bin ant bat of an installed version of Ant Restore Defaults Apply has to be embedded here Then
86. position inDigitString lengthy erroriree additional sets of coverage False False False related data to be False False true true true true displayed For more details about the coverage of individual nodes with or without a Fig node information with MCDC pair condition a dialog box which can be opened via double clicking the node is available Firstly all information from the tooltip and all relevant coverage data of this node are listed both by percentage and numerically If the node contains a non trivial condition the logical structure of the condition as well as all assignments to their elements are also shown in a table The first column represents the truth vector for the entire condition as read from the test log file The second column contains the evaluation of the entire condition and the subsequent columns show the evaluation for each element For each elementary condition the appropriate MCDC pair if present will be highlighted by clicking the corresponding table head 4 2 6 View Coverage The view Coverage offers a means for essentially analyzing the coverage data of the project Analogous to the view Project all structures of the project are listed hierachically in a tree which can be configured through the menu of the view to adopt a flat representation For each project structure the percentage of the degree of coverage by various coverage metrics is specified in the corresponding columns If the option
87. prehensive explanation of the variuos configuration options can be found under 4 3 Preferences 4 1 4 Menu Help Manual Project Tasks Configuration LE LE Manual Selecting the menu item Manual opens the user About SOTA documentation on hand in the system s standard browser Fig menu Help About The menu item About opens a dialog containing information about the current SOTA version as well as the plug in status of the application 4 2 Views 4 2 1 View Project In the view Project all source code files of the test program and their subordinate structures as well as classes and Ta Project BE HET methods are listed im a tree structure The top nodes S D pat java represent the source code files imported on creating the EA Digit project The top level classes and methods and their respective internal classes and methods are represented as children in hierarchical order up to an arbitrary nesting level Using the two buttons and an the user can fully expand or collapse the tree The button Fe alternately causes the tree to be displayed in ascending and Fig view Project descending order On selecting an element of the tree the hierarchical presentation views Source and CFG are updated automatically main string E yaluateDigizequencer String The individual structures listed in the tree are E files G classes a internal classes a public functions o protected functions E private funct
88. procedure 1 Eclipse Create Program The test program should be written in Eclipse In case Eclipse is not installed yet the user can do this following these instructions http wiki eclipse org Eclipse Installation Firstly the user has to create a new project for the test program This is done via the menu File gt New gt Java Project A dialog opens where the name for the project Digit has to be entered for all other options the default values can be used Therefore the user can close the wizard on the first page clicking Finish To create a class Digit the user has to select File gt New gt G Class which calls the appropriate wizard Here the name of the class Digit has to be entered before closing the dialog Then a Java file is opened into which the source code above has to be copied New Java Project Create a Java Project Create a Java project in the workspace or in an external location Contents Gei Create new project in workspace O Create project from existing source Diipevelopmentimerkspaceipiat Fig Eclipse New Java Project New Java Class Java Class s The use of the default package is discouraged n Dame ad tre Enclosing type E SC Name Dat L Modifiers 2 public default abstract final Superclass java lang Object Interfaces _ _________ ___ z bA __z z gt z gt gt gt _ _ _ _z lt
89. program execution such as branch coverage or multiple condition coverage and provides a graphical representation of the results Before the analysis it 1s necessary to instrumentate the program that will be tested There are three basic ways to employ SOTA in a program test the manual program test in combination with an external testing system or integrated into an automatic testing system Durng the manual program test the program is tested manually i e started by hand functions are executed etc This can be done by using a development environment like Eclipse in addition to SOTA or with SOTA itself by integrating an Ant build file and a startup script The only difference when working with an external testing system 19 the method of testing As with the manual test SOTA controls pre and postprocessing However for the program test itself a separate testing system 1s used such as ATOS This results in a program sequence SOTA ATOS SOTA without any internal linking When working with an automatic test system SOTA can be embedded as a library SOTA ATM allowing the system to use the non GUI functionality of SOTA Core functions of SOTA can either be called through command line parameters or a SotaATM instance based on a class included in the library The work with SOTA 19 divided into the preparatory testing and evaluation phases Basically the preparatory phase for the program test consists of reading the source code determini
90. rTypeListWindow ForeignKeyConstraintSelector i createSpecializedMainTableListener createSpecializedMainTableListener AssociationTabControl createSpecializedMainTableListener ForeignKeyConstraintSelector i createChooseButtonListener AssociationTabControl createChooseButtonListener ForeignKeyConstraintSelector openList getTablelnputFromDB createWindow Shell 100 00 Se 71 43 t Class 73 53 82 00 75 86 50 00 50 00 50 00 71 43 semorg qui provider SeminarTypeTableProvider h d i 50 00 1 33 383 1 maia lea n 7083 az 75 86 5000 5000 RAN 3000 697 6957 BET IT Sino M 1 o R w ee e ae o wese mapa ss NON FSemnartypewindom She Pa j Ema SeminarTypeWindow Shell SeminarType Association TabControl SeminarTypeWindow Shell SeminarType ForeignKeyConstraintSelector setlnput SeminarType setinputComponentsEnabled boolean IE A o createButtonListener AssociationTabcontro WESSEN createButtonListener Fm E z checkinput __ 0 00 0 00 0 00 0 00 0 00 ooo ooo 000 000 commitinputToDB Li i i e oo oo ano confirmetose o o 0 00 9 00 000 000 0 09 0 00 000 000 onsaven_ ____ EE w RT Re moe e TS mapa asas gt EN EN EEN Ee EN EN EN se Jo messe e sessies sue NO gt E Supervisor ne int Timestamp em Aa ae LL a Cerco pa o ES EE EE BR CELL Ea
91. required allow Eclipse to add build path for JUnit e Ina created Java file copy source code of a test file After that the final project content should look like the following Refactor Navigate Search Project Run Window Help ci O Q dee ee POMS 4 5 ve eror ES 5 Debug E Java Pr s N Hi c O Ju tTestjava 3 a ese 1 package hu demo a E Digit lt 2 G src 3 import junit framework TestCase 83 hu demo 3 Digit java 5 public class DigitTest extends TestCase J Zifferjava FloatingPointNumberSca public void testScanDigitSequence Ziffer java backup feec test sequence to obtain full FEEC coverage CS testsrc co test sequence to obtain full C coverage E Di ee c1 test sequence to obtain full C1 coverage D Digit Testjava c2 test sequence to obtain full C2 coverage D SE AR mmec test sequence to obtain full MMCC coverage DE Ad mcdc test sequence to obtain full MCDC coverage DP I c3 test sequence to obtain max cB coverage Ei JRE System Library jre6 7 q 8 mA JUnit 3 ag ASCLogger jar E d ase 7 17 public void feec 1 E ASCLoggerini 18 System out println z Digit evaluateDigitSequence z E blalog 19 System out printin Digit evaluateDigitSequence 9 build xml 20 d log 21 E T_c0 log 225 public void c E T_Cl log 23 System out println 9 Digit evaluateDigitSequence 9 E T_C2 log 24 System out println
92. s sota log Here SOTA logs all its activities sota_ lt date gt _ lt index gt log language spec The specification file for all supported languages ASCLogger jar The logging component for Java test programs must be included in the test programm and demands an initalization file named ASCLogger ini see below during the test lt projectname gt project For each project the general project data are noted here SOTA ATM jar The automatic test module of SOTA enabling the testing funcionality to be applicable without GUI SOTA ATM jar is an executable jar file which can also be imported as a program library This allows access via command line scripts or software See the tutorial for more information Root directory of the test program Project directory The root directory of the project contains possibly in a subdirectory the sources of the test program and at the same location their backups that are generated by SOTA on creating the project SOTA puts all automatically generated report files at this location unless the user activated an inquiry via data selection dialog in the preferences src lt xyz gt java The source files of the test projects are partially instrumented after starting the test src lt xyz gt java backup The backup of the original source files is created before starting the test for all source files that are not instrumented bin lt xyz gt class The compiled class files which may be instru
93. s 71 43 85 71 77 78 66 67 66 67 33 33 66 67 66 67 66 67 semorg qui provider Company nternalPresentationTableProvider y i i i j g LL Jo Lo mee mapa Class semorg qui CompanylnternalPresentation Window CompanyInternalPresentationWindow Shel IA 1 ES CompanylnternalPresentationWindow Shell CompanylnternalPresentation AssociationTabControl CompanyInternalPresentationWindow Shell CompanylnternalPresentation ForeignKeyConstraintSelector TL DEAN setinputComponentsEnabled boolean O Mee createButtonListener AssociationTabContro _ HO on o SONO SONO oer ooo iscas MA ForeignKeyConstraintS elector checkinput 000 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 commitinputToDB O DDD EE A Goo 00 amer 1 as 0 00 000 000 oso 000 0 99 000 op G Model Bt Class semorg gui list CompanyListWindow 33 33 50 00 CompanyListWindow CompanyListWindow AssociationTabControl CompanyListWindow ForeignKeyConstraintSelector createSpecializedMainTableListener createSpecializedMainTableListener AssociationTabControl createSpecializedMainTableListener ForeignKeyConstraintSelector createSpecializedToolBarListeners createChooseButtonListener AssociationTabControl createChooseButtonListener ForeignKeyConstraintSelector openList getTablelnputFromDB 00 0 30 20 33 33 50 00 50 00 50 00 16 67
94. s is irrelevant java jar SOTA ATM jar p Ziffer project z c With this call the original state of the sources is restored java jar SOTA ATM jar p Ziffer project t testl r This call of the module creates a report for the project Digit The test test1 is read and the coverage metrics computed there are written into the standard report file report html If the source files are instrumented they will be restored to their original state first 6 2 2 SOTA ATM API Overview SOTA ATM can be imported as a library into a program thus enabling the usage of the non gui functionality of SOTA for the static analysis and the coverage test The javadoc documentation is available here The central class for the program test is SotaATM Each of its objects represent a test instance of SOTA A test instance can be configured by either loading a project file or passing an instance of ProjectConfiguration which contains all relevant data for the program test similar to a project file For this instance the full functionality of SOTA from instrumentation to report creation is available see javadoc Right after opening the project a Metrics object containing all metrics of the static analysis can be returned from the test instance The coverage metrics are available in the very same object after testing the program and evaluating the generated log files To configure the instrumentation for a test it is possible to define variable Schemes whic
95. s under Windows 2000 and upwards For use in automatic testing systems the non GUI functionality of SOTA is provided by the library SOTA ATM jar which also serves as an API for integrating SOTA into other programs Chapter 2 covers the main features of the program and ways to use it The operation of the program is then described in detail in Chapters 4 6 adopting In Chapters 4 and 5 a systematic as well as in Chapter 6 a process oriented approach The latter includes typical application scenarios in the form of tutorials Lastly all dimensions and other terms used in this manual are specified to provide a better understanding The program was developed for exclusive usage in teaching SOTA User s Guide Version 1 0 1 Introduction 2 Overview 3 Installation and Start of Program 4 User Interface and Functionality 5 Files 6 Tutorials 7 Appendix 2 Overview SOTA supports static program analysis and dynamic program testing During the static program analysis the source code of the program 19 analyzed and as a result SOTA determines ten software metrics like the Cyclomatic complexity or the number of modified boundary interior paths Before the analysis SOTA needs to parse the source code The dynamic program test from now on referred to as program test 19 a structure oriented and control flow related program test forming the main part of SOTA Depending on the specific test case SOTA determines nine code coverage metrics during
96. se execution within Eclipse 6 SOTA Evaluation Phase Reconstruction When the tests in Eclipse are completed the user has to inform SOTA of this by selecting the menu item Stop Test The sources are then reconstructed into their original state Alternatively the sources can also be rebuilt via the menu item GI Restore Sources Read TestLogs Please select testlogs to import Directory DiDevelopmentiworkspace Digit D test og D test 2 log test 1 1 log test Log D test empty log Fig dialog Read Logs 7 SOTA Evaluation Phase Evaluating the Test In order to read the log files a dialog opens immediately after selecting the menu item E Stop Test Here all files in the execution directory ending on log are listed for the import The log files selected there are read analysed for further evaluation and finally appear in the view TestLogs The TestLogs marked there will now be used for computing the soverage metrics and determine the presentation of the coverage in the views Source CFG and Coverage If the user wishes to import the log files when SOTA is not in testing mode the menu item Read Logs evokes the same behaviour 8 SOTA Evaluation Phase Create Report The program test is completed by creating a report This is done via the menu item Create Report which depending on the settings in the preferences either opens a dialog for entering a file name or automatically generates a name
97. sks 4 1 3 Menu Configuration 4 1 4 Menu Help 4 2 Views 4 2 1 View Project 4 2 2 View Testlogs 4 2 3 View Scheme 4 2 4 View Source 4 2 5 View CFG 4 2 6 View Coverage 4 2 7 View Metrics 4 3 Preferences 4 3 1 Preferences View CFG 4 3 2 Preferences View Coverage 4 3 1 Preferences General 4 3 1 Preferences Report 4 3 1 Preferences View Source 4 4 Delete Project 5 Files 6 Tutorials 7 Appendix 4 User Interface and Functionality Next to the compulsory menu bar and the toolbar offering quick access to the most common actions the user interface consists of different views available in three areas by clicking the corresponding tabs The upper left area offers the view with the project outline Arranged below the project outline are two views listing the test cases and the instrumentation schemes of the project respectively The right area takes up the largest part of the window consisting of different views supplying detailed information about the source code control flow graphs as well as coverage and metrics reports The views are linked so that e g selecting a file in the ProjectView causes the corresponding source code to appear in the SourceView Selecting a test in the TestView updates the coverage metrics for the entire project and the representation of the coverage for the selected file Wizards and dialogs are at the user s disposal in order to guide larger actions such as creating a new project Lastly a sta
98. so that the program will be started However the start option of the current version of SOTA does not offer parameter passing to the test program which strongly limits testing options for our program Digit In order to test several strings it is necessary to readjust the start script Start test Press OK to augment the project Test description Select Scheme Level 2 instrumentation w SE Digit java Digit ES main Strinal evaluateDigitSeguence Strings Expand all Collapse all Level 0 Mo instrumentation leaves sourcecode as it is ds Level 1 Instruments chg relevant blocks standard coverage analysis ds Level z Instruments also conditions enables analysis of condition coverage A Level 3 Full instrumentation every single statement is instrumented Rerun configuration Don t change sources or binaries Build project Run project Instruments the project with the chosen Scheme compiles and runs project with Ane Cancel Fig dialog Start Test with compilation and start options 6 Evaluation Phase Reconstruction 7 Evaluation Phase Evaluating the Test 8 Evaluation Phase Create Report The remaining steps of the manual program test are consistent with the description in 6 1 2 6 2 Test using an External Testing System ATOSj HU Seminar Organisation 6 2 1 ATOS and HUSemOrg To use ATOS as an external testing system and of HUSemOrg as a test program they need to be installed
99. subsequently the usage of the parameters r to create a report and z to rebuild the sources n lt name gt lt lang gt lt project dir gt L lt exec dir gt lt src dir gt The parameter n creates a new project on the basis of the passed values A project name the employed programming language and the reference to the project directory have to be indicated at the least The execution directory as well as the source directory are optional values and are set to the project directory if not specified otherwise The project file will be saved as lt name gt project Using the paramter n prohibits the usage of parameter p p lt name gt project With this parameter the user can pass a project file which contains the data to characterize the project Such a project file can be generated by using the graphical user interface by calling SOTA ATM with the parameter n see above or manually Using the paramter p prohibits the usage of parameter n r lt report file gt This parameter evokes the creation of a report containing the computed coverage metrics on the basis of the test s log files that have to be included by calling the parameter t If a reference to an html file follows the parameter the report will be written into it Otherwise a file report html will be created or overwritten The parameter r requires at least one log file passed via t and finally the paramter i evoking the instrumentation of the source files If
100. t evaluable with regard to the coverage test for conditions and have no influence on the control flow 7 1 10 Number of Logical Conditions Conditions The number of logical conditions contains the sum of all atomic and compound conditions This values is Important for computing the minimum multiple condition coverage Die Anzahl der logischen Bedingungen enthilt die Summe aller atomaren und zusammengesetzten Bedingungen Dieser Wert ist f r die Berechnung der minimal Mehrfach Bedingungsiiberdeckung wichtig 7 2 Coverage Metrics The actual aim of SOTA is to evaluate program tests by computing coverage metrics By including instrumentations a log file 1s created with the necessary data allowing SOTA to reconstruct the program flow and the evaluation of the conditions in retrospect From these data the most common coverage metrics are determined for the individual tests These are then listed in the View Coverage 7 2 1 Function Entry Exit Coverage FEEC The test for Function Entry Exit Coverage requires all inputs and outputs of each function to be regarded for full coverage It is computed as follows e FEEC visited function inputs visited function outputs function inputs function outputs In Java there exists only one input for each funtion Counted as possible outputs are the normal function ending in case it was reached as well as all return statements and all throw statements outside of try structures 7 2 2 Statem
101. tWindow Class semorg gui provider ClientBookingTableProvider Class semorg gui ClientBookingWindow Class semorg gui list ClientListWindow Class semorg qui provider ClientTableProvider 87 50 91 94 oi 18 80 00 80 00 60 00 80 00 88 24 88 24 Class semorg gui ClientWindow 62 96 84 50 51 25 55 88 57 69 55 22 Class semorg sql tables Company 84 64 Class semorg sql tables CompanyBooking Class semorg gui list CompanyBookingListWindow 50 00 50 00 50 00 Class semorg qui provider CompanyBookingTableProvider 65 52 75 00 68 00 50 00 50 00 50 00 60 87 60 87 Class semorg gui CompanyBookingWindow 73 91 86 23 55 56 66 67 68 42 65 52 Class semorg sql tables Company nternalPresentation Class semorg gui list CompanyInternalPresentationListWindow 61 54 55 56 33 33 50 00 50 00 0 00 50 00 47 06 47 06 Class 71 43 85 71 77 78 66 67 66 67 33 33 66 67 66 67 66 67 semord qui provider CompanvinternalPresentationTableProvider Class semorg qui CompanvyinternalPresentation Window 0 00 0 00 Class semorg gui list CompanyListWindow 50 00 0 00 50 00 35 29 35 29 Class semorg gui provider CompanyTableProvider 50 00 0 00 50 00 83 72 83 72 Class semorg gui CompanyWindow 59 62 17 02 58 06 0 00 Class semorg sgl tables Conduct ooo ooo H Class semorg qui util Configuration 40 00 19 23 11 11 0 00 0 00 0 00 20 00 20 00
102. the evaluation of log files with other programs There are two control options offered for the module On the one hand it is possiblet to start SOTA ATM as an executable Jar which can be controlled and started via command line parameters enabling its usage in simple scripts On the other hand SOTA ATM is a library which can be integrated making it an interface between functions for testing the project and other programs 6 3 1 SOTA ATM via command line call Parameters Overview SOTA ATM is an executable Jar and can be called via command line with several paramters The necessary values about the project can either be passed to SOTA ATM by importing a project file p or by passing them on calling SOTA ATM n A project file can be generated via the graphical user interface of SOTA by using the option n of SOTA ATM or manually The other options call the different components of the module The main aspects are the instrumentation of source files i the reconstruction of the original sources z and the evaluation of tests by reading the appropriate log files f as well as creating a report thereafter r Additional functions can be combined with the ones above Those are the compilation c and start of the test program s provided the necessary files are available The order of options is irrelevant Usage java jar SOTA ATM jar options options include c lt ant buildfile gt compile sourefiles only if
103. the source files are already instrumented they will be restored to their original state before the test is read S lt runscript gt The parameter s starts the test program It is necessary to reference a run script either as a parameter or in the project file t lt log file gt log lt log file gt log This parameter is used in two contexts When an instrumentation via i is called it defines the name of the test which is also the name of the log file that will be created For creating a report with the parameter r all log files to be imported are listed after the parameter t Z The parameter z restores all source files of the project from the backup files It cannot be used together with the parameter i for the instrumentation Sample Usage java jar SOTA ATM jar n Digit Java workspace Digit A project Digit of the programming language Java is created Its project directory is workspace Digit From there all java files are imported into the project After successful completion all project information is saved in the file Digit project Various settings e g build files that should be used can be changed in that file java jar SOTA ATM jar p Ziffer project i Level2 t testl c s This call of the module has the effect that the project Digit is parsed fully instrumented according to level 2 and then compiled and started Additionally the value testl is passed as a test name The order of the option
104. tus bar can be found at the bottom of the window providing information about the status of the original source files as well as the parsed source code SOTA is displaying The original state 1s designated as CLEAN whereas the instrumented state 1s marked as DIRTY ES SOTA Project Digit Project ek mm Ts Project d E E E source CFG evaluateDigitSeguence String IR iteration le O A LU Digit java SA Dig main string evaluateDigitSegue O 15cheme No OI 23 TestLogs test C test empty 4 1 Menus und Toolbar 4 1 1 Menu Project Tasks Configuration Help Project Pe New Project Open Project E Save Project Close Project ES Recover Project Exit Fig menu Project Tasks Configuration Help G bat Bege P m EI E 5 5 E E toe Metrics oi 5 6 iter body d Er d 2 e private static double evalusteDigtseruenceistring inDigt String nooo double wert 0 0 Status of source code CLEAN Status of parsed code CLEAN Fig main window of SOTA blg E New Project ar LA Clicking the menu item New Project opens the two page wizard for creating a project On the first page the user has to indicate a name for the project as well as the project directory and run directory From the project name SOTA generates the file lt project name gt project placing it in the SOTA directory The project directory is the root directory of the test program
105. will be applied as soon as test logs are read and if the option Fig Preferences view Source Es Show Coverage is activated 4 4 Delete Project A function Delete Project was not implemented This means that SOTA files corresponding to a project with the exception of the log files see View Testlogs have to be deleted by hand All SOTA files are described in 5 1 The files created by SOTA that have to be deleted in order to clear the system are in detail e SOTA root directory o lt projectname gt project e root directory of test program Project directory o lt xyz gt java backup o report html o report_ lt date gt lt index gt html o lib ASCLogger Jar e execution directory of test program Execution directory o lt testname gt log o ASCLogger ini It is absolutely necessary to restore any instrumented source files to their original state by choosing the menu item Restore Sources before deleting the backup files as this 19 hardly possible without them SOTA User s Guide Version 1 0 1 Introduction 2 Overview 3 Installation and Start of Program 4 User Interface and Functionality 5 Files 6 1 Overview 6 2 Project File 6 3 Report Files 6 Tutorials 7 Appendix 5 Files 5 1 Overview SOTA root directory The SOTA root directory contains the SOTA system as e g the boot file SOTA exe and the Eclipse Rich Client Platform installed with SOTA as well as further special SOTA files and the project file

User's Guide SOTA 1.0 - 1 Introduction

Contents

Download Pdf Manuals

Related Search

Related Contents