readme file - DevChakraborty.com
Contents
1. 26 Operating Characteristics This is a convenient way to visualize ROC AFROC and FROC plots for specific search model parameters to observe the differences between treatments and calculate ROC and JAFROC figures of merit One modality E to mF Number of Modalities Oe C wo Number of Lesons per image 2 Lesions image Search Model Parameters Mu Lambda Two modalities st characteres EATE Number of Mociaities C oe wo Number of Lesions per image Lesions Image Net Parameters MiM re Larosa LLF S Mins AH Parameters aot 20 Ahi Lambdo Ady We ROC AUC 0 763399 NH ROC AUC 0 007299 ROC EFFECT SIZE 0 0430998 Ne JAFROC AUC 0 63237 AM JAFROC AUC 0 707132 JAFROC EFFECT SIZE 0 0747417 Cee L_e 27 If warnings or errors during the analysis error message is displayed on console window If you cannot fix the problem send me the data file Please bring errors on screen to my attention Good luck Dev P Chakraborty Last revision 1 18 2011 28 References 1 Chakraborty DP Berbaum KS Observer studies involving detection and localization Modeling analysis and validation Med Phys 2004 31 8 2313 30 2 Metz CE Basic principles of ROC analysis Seminars in Nuclear Medicine 1978 8 4 283 98 3 Metz CE ROC Methodology in Radiolo2. Analysis Method C JAFROC1 JAFROC C ROC m il of NORMAL CASES 100 of ABNORMAL CASES 10 NA number of lesions pe Average number of lesions pe Total number of lesions 250 Se fault analysis method m Options I Pseudovalue files for SAS I Create ROC Irc file for DBM MRMC 2 r Sample Size Estimation I Perform Sample Size Estimation Analyze Data 1 First select the analysis option The simplest way is to keep the defaults 2 Then click on Analyze Data button Upon successful completion the following will show DataFile xls Datafile r Reader Training Show RAW FROC plots Analysis Method C JAFROC1 JAFROC C ROC r Options I Create ROC Irc file for DBM MRMC I Pseudovalue files for SAS r Sample Size Estimation I Perform Sample Size Estimation 1 11 2011 15 01 39 Ready Datafile C Users Administrator Documents JAFROC Data DataFile xls Number of TREATMENTS 2 Number of READERS 5 Number of NORMAL CASES 100 Number of ABNORMAL CASES 100 Maximum number of lesions per case 4 Average number of lesions per case 2 5 Total number of lesions 250 Set default analysis method JAFROC Analyzing Reporting to file C Users Administrator Documents JAFROC Data DataFile_JAFROC txt Done 11 Open the JAFROC Data folder ee Organize i Open Ft Favorites Documents library EE
3. Desktop JAFROC Data J Downloads Name S Date modified Type Size E Recent Places Arrange by Folder 4 3 2009 3 17 PM XLS File fi Libraries Documents a Music E Pictures J Computer amp Local Disk C G iDisk on psf V S hongjuny on pst V amp work on psf X G Home on psf Y G Host on psf Z DataFile JAFROC Date modified 1 11 2011 3 01 PM Date created 1 11 2011 2 18 PM Text Document Size 4 49 KB The output file is DataFile_JAFROC txt It will contain the following for the sample data set 12 Output file for the sample data set JAFROC SOFTWARE IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM DAMAGES OR OTHER LIABILITY WHETHER IN AN ACTION OF CONTRACT TORT OR OTHERWISE ARISING FROM OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE JAFROC Version 4 0 Last Compilation Date JAFROC_CORE DLL Jan 10 2011 12 34 00 JAFROC EXE Jan 11 2011 12 27 18 This run date 1 11 2011 15 01 46 Input Data Filename DataFile xls Output Data Filename DataFile_JAFROC txt Analysis method H JAFROC FOM DBM MRMC SIGNIFICANCE TESTING Number of Readers nes Number of Treatmen
4. a false positive and not a lesion See my 2004 Med Phys paper If ILF is large then one expects a large power advantage of JAFROC over ROC analysis If it is zero one does not expect a power advantage and you might as well analyze your data with DBM_MRMC software Results RANDOM_ALL F statistic and p value Significance level of test alpha 0 05 F statistie 0 382 p value 5 469E 001 DF Num Den 1 1 370E 001 NB Tf the p value is Less than 0 05 there is a significant difference between at least one pair of TREATMENTS If p lt 0 05 you have a significant difference READER averaged FOMs for TREATMENTS and 95 confidence intervals 1 G 690L 062867 073415 18 2 0 7207 0 6506 0 7708 Lists the reader averaged figures of merit for the 2 treatments modalities and 95 confidence intervals The confidence intervals have not been corrected for multiple comparisons INTER TREATMENT differences and 95 confidence intervals FOM 1 FOM 2 0 0206 0 0923 0 0571 Lists the inter treatment differences between reader averaged figures of merit for all pairings of the treatments and 95 confidence intervals for the differences If the 95 CI does not include 0 then the corresponding modality pairs are significantly different The confidence intervals have not been corrected for multiple comparisons more than one pair If they had been the corrected Cl s would be larger and you wou
5. you had selected Perform Sample Size Estimation in the main screen repeated below the program will walk you through one more step You will not have to re enter the variance components manually PUNTA Total number of lesions 250 ROC Checked Set default analysis method JAFROC Select Sample Size Estimation Analyze Data Upon the Sample Size Estimation screen you will enter appropriate parameters as for your request then click on Recalculate button Required number of cases per number of readers will be shown Press OK to report the result F socn i NH ROC FOM 0 777212 AH ROC FOM 0 827212 ROC Effect Size 0 0500001 NH JAFROC FOM 0 545253 AH JAFROC FOM 0 659475 JAFROC Effect Size 0 114222 0 05 Cases 188 Cases 126 Desired Power 1 beta Cases 105 3 Cases 97 0 8 Cases 88 Cases 86 Cases 82 Cases 80 Recalcul Cases 79 Cases 78 14 15 Cases 74 16 20 Cases 72 Readers 21 23 Cases 71 21 The rest of the output is for random cases and random readers analyses as per the DBM MRMC documentation 9 Options l Analysis Method C JAFROC1 Max JAFROC ahd C ROC i z Set Optio Create ROC Irc file for DBM MRMC l Pseudovalue files for SAS 2 Analyze Data 1 Convert FROC data into ROC and save it to the file compatible to DBM MRMC softw
6. 0923 0 0511 Variance components needed for sample size estimation var_tr 1 2143E 003 15 v r to 58 1149E 002 var tro D 7859E 002 Results RANDOM_CASES F statistic and p value Significance level of test alpha 0 05 F statistic 0 677 p value 4 116E 001 DF Num Den 1 1 990E 002 N B If the p value is less than 0 05 there is a significant difference between at least one pair of TREATMENTS READER averaged FOMs for TREATMENTS and 95 confidence intervals 0 6901 0 6390 0 7411 0 7107 0 6649 0 7565 INTER TREATMENT differences and 95 confidence intervals FOM 1 FOM 2 0 0206 0 0700 0 0288 Results RANDOM_RDRS 16 F statistic and p value Significance level of test alpha 0 05 F statistic 0 706 p value 4 480E 001 DF Num Den 1 4 000E 000 N B If the p value is less than 0 05 there is a significant difference between at least one pair of TREATMENTS READER averaged FOMs for TREATMENTS and 95 confidence intervals 0 6901 0 6403 0 7398 0 7107 0 6570 0 7644 INTER TREATMENT differences and 95 confidence intervals FOM 1 FOM 2 0 0206 0 0887 0 0475 17 Interpretation of the output file ine Loc Frac 2 Q 08T This is the incorrect localization fraction ILF the fraction of abnormal images on which the highest rated location was
7. AFROC Data Organize v R Open v Share with E mail Burn New folder fi Fe Favorites Documents library E Desktop JAFROC Data Downloads Name K Date modified Type Size Arrange by Folder T Recent Places f F DataFile 4 3 20093 17PM _XLSFile 294 KB E Libraries E DataFile_JAFROC 1 11 20113 15PM_ Text Document 5KB E DataFile ROC_PLOTS 1 11 2011 3 55PM _ Text Document 15 KB Pre plot template 7 29 2010 11 33 AM _ XLS File 1 226 KB i Pictures Type XLS File E Videos Size 1 19 MB Date modified 7 29 2010 11 33 AM jE Computer amp Local Disk C G iDisk on psf V G hongjuny on psf V amp work on psf X GA Home on psf Y P Host on psf Z Gia Network plot_template Date modified 7 29 2010 11 33 AM Size 1 19 MB XLS File Authors Dev Chakraborty Date created 7 29 2010 11 33 AM The file plot_template xls file includes script to read the search model fitted plots data file DataFile ROC_PLOTS txt Double click on plot_template xls and press Enable Macros to allow the script to run reading plots Security Warning xj C Wwork JAFROC_CUAFROC Test Cases plot_template xls contains macros Macros may contain viruses It is usually safe to disable macros but if the Select DataFile ROC_PLOTS txt Search model fitted ROC curves will be shown on the worksheet along with the fitted search model parameters 25
8. Downloads Name 5 Date modified Type E Recent Places i DataFile 4 3 2009 3 17 PM XLS File l E Libraries plot_template 7 29 2010 11 33AM XLSFile E Documents d Music E Pictures a E Videos JE Computer amp Local Disk C l G3 iDiskon psf V 4 m r File name DataFile v JAFROC Data Files xls A A If there are errors e g the selected file is not a valid input file or the data is inconsistent the program will display appropriate error messages on the console Otherwise the screen will show the following DataFilexls 1 11 2011 14 12 37 Ready Reader Training Datafile C Users Administrator Documents JAFROC Data DataFile xls Show RAW FROC plots Number of TREATMENTS 2 Number of READERS 5 E Number of NORMAL CASES 100 Number of ABNORMAL CASES 100 C JAFROC1 Maximum number of lesions per case 4 Average number of lesions per case 2 5 ORES Total number of lesions 250 ROC Set default analysis method JAFROC r Options I Create ROC Irc file for DBM MRMC I Pseudovalue files for SAS r Sample Size Estimation I Perform Sample Size Estimation Analyze Data ROC plots AFROC plots FROC plots ROC AFROC FROC Excel Plots 10 DataFilexis 1 11 2011 14 12 37 Ready Reader Training Show RAW FROC plots Datafile C Users Administri Number of TREATMENTS 2 Number of READERS 5
9. JAFROC 4 0 User Manual Dev P Chakraborty PhD Hong Jun Yoon MSEE Windows XP Vista and Windows 7 integrated software to perform JAFROC analysis and sample size estimation for FROC and ROC studies Dev Chakraborty www devchakraborty com Contents Fete Sinani gather ba earth De aL Ee eee apo bate at 3 Installing thesoftwares lt 3 tise Sana eee 3 Datta formatsion aen anin iine 6 Truth osheeee 66 6 TP lesion localization Worksheet ccdcccvdssecvesecieds cosac dels r ATE ERE do busad dea vecededsvsaicevevaces da svsadce svasedsauvedidevssaces EEA RA 7 FP non lesion localization Worksheet 8 RUNNING the SOMWaAMe e245 E acces hace 66666 9 Outputtile for the sample datarset ssseeeee 13 Interpretation of the OUt PUT TE a 18 ROG sample size calcul atOnssci sss wasc so eles Se ET 66 20 JAFROG Samplesize calculator ranee a eles 21 OPTIONS 2ccccccc EA 22 PIOS riie 23 A E E E A A E E 29 Features JAFROC analysis is described in Ref 1 Basic ROC references are 2 4 1 Standalone windows compatible software XP Vista 7 2 Convenient FROC raw data plotting this is a quick way to check the quality of the FROC data and train readers in optimal usage of the ratings scale 5 3 Integrated plotting routine uses maximum likelihood method to fit the search model 6 7 to ROC FROC and AFROC data 4 Integrated sample size analysis using an adaptation of the Hillis Berbaum method 8 5 Ability to perform use
10. are Irc Sample Size Perform Sample Size 2 Save pseudo values to a file in plain text format consists of four columns of numbers For each row there are Reader Index Treatment Index Case Index and the corresponding pseudo value 22 Plots Press Show RAW FROC plots button in the Reader Training section DataFile xis i r Reader Training Show RAW FROC plots Number Number z Number Analysis Method C JAFROC1 Average JAFROC C ROC Shown is the raw FROC plot for each reader and modality Press lt lt and gt gt button to browse other reader and modality Or you can save the raw plot as CSV file Raw FROC Plot i Press ROC plots AFROC plots and or FROC plots button in the Create ROC AFROC FROC Excel Plots section 23 Analyze Data reate ROC AFROC FROC Excel Plots C ROC plots AFROC plots FROC plots Shown is the Search Model fitted curve for each reader and modality Press lt lt and gt gt button to browse other reader and modality Or you can save the raw plot as CSV file emne E lt lt gt gt Reader 1 Modality 1 Also the search model fitted curves for every reader and modality pairs can be shown in Excel worksheet Open the JAFROC Data folder 24 gt Libraries Documents JAFROC Data Search J
11. e reader modality and case then they should appear consecutively and not be separated by data for other cases FP_Rating This positive integer floating point value is the rating assigned to a particular FP mark Multiple false positive are indicated on multiple lines Running the software Click Start menu and select JAFROC icon internet Explorer L Windows DVD Maker E Windows Fax and Scan Administrator Windows Media Center Documents Pictures Music Google Chrome Computer JAFROC Control Panel Sal pbc senna Devices and Printers E plot_template xls a a Maintenance J Microsoft Office Help and Support J Microsoft Silverlight a Nexon 4 Back Default Programs Search programs and files Shutdown O o Dathe 2 22 2002 18 12 37 Ready m Reader Training Analysis Method C JAFROK C JAFROC C ROC r Options I Create ROC Irc file for DBM MRMC F Pseudovalue files for r Sample Size Estimation F Perform Sample Size Estimation Create ROC AFROC FROC Excel Plots ROC plots AFROC plots FROC plots Locate your data file in Documents JAFROC Data folder Select your data file and double click or click Open E a Ji Documents JAFROC Data v Search JAFROC Data P e Otel a tA i hub Organize v New folder ev fl p library Arrange by Folder v esktop
12. ed by copyright law and international treaties Unauthorized duplication or distribution of this program or any portion of it may result in severe civil or criminal penalties and will be prosecuted to the maximum extent possible under the law User Account Control Program name C Users Public FROCWebSite archive JAFROC msi Publisher Unknown File origin Hard drive on this computer Show deta a E Change when these notifications appear Click on Yes After installation click on Start menu and check that the JAFROC folder was created with four icons in Ee Internet Explorer Windows DVD Maker 3 Windows Fax and Scan Administrator e Windows Media Center Documents Pictures Music Computer Control Panel Devices and Printers Default Programs a Maintenance ny Microsoft Office Help and Support J Microsoft Silverlight Shut down If you get the error message shown below when you start JAFROC software try installing Microsoft Visual C 2010 Redistributable Package x86 available at http www microsoft com downloads en details aspx familyid A7B7A05E 6DE6 4D3A A423 37BF0912DB84 amp displaylang en and restart JAFROC The program can t start because mfc100 dll is missing from your computer Try reinstalling the program to fix this problem Data format For a quick understanding of the format see the sample input file Data3 xls inc
13. gic Imaging Investigative Radiology 1986 21 9 720 33 4 Metz CE Some Practical Issues of Experimental Design and Data Analysis in Radiological ROC studies Investigative Radiology 1989 24 234 45 5 Bunch PC Hamilton JF Sanderson GK Simmons AH A Free Response Approach to the Measurement and Characterization of Radiographic Observer Performance J of Appl Photogr Eng 1978 4 4 166 71 6 Chakraborty DP A search model and figure of merit for observer data acquired according to the free response paradigm Phys Med Biol 2006 51 3449 62 7 Chakraborty DP ROC Curves predicted by a model of visual search Phys Med Biol 2006 51 3463 82 8 Hillis SL Berbaum KS Power Estimation for the Dorfman Berbaum Metz Method Acad Radiol 2004 11 11 1260 73 9 Hillis SL Berbaum KS MRMC Sample Size Program User Guide available from http perception radiology uiowa edu last accessed Dec 28 2009 Book MRMC Sample Size Program User Guide available from http perception radiology uiowa edu last accessed Dec 28 2009 City 29
14. ld see fewer significant differences But if p lt 0 05 at least one of the differences is significant Variance components needed for sample size estimation Var ctr I Z133sb 003 var te 0 var tra 60 19 ROC sample size calculator This could be used for sample size estimation for JAFROC but you don t know the JAFROC effect size corresponding to a given ROC effect size so it is best to use the built in capability in JAFROC described after this section This is an implementation of the Hillis Berbaum method 8 Click Start menu and select the Sample Size Calculator icon E Internet Explorer Windows DVD Maker iE Windows Fax and Scan Administrator e Windows Media Center Windows Media Player Windows Update A XPS Viewer Pictures J 7 Zip J Accessories a Games a Google Chrome JAFROC Music Computer Control Panel Devices and Printers A plot_template xls Default Programs a Maintenance a Microsoft Office Help and Support J Microsoft Silverlight Nexon 4 Back Search programs and files The following screen will show Enter the three variance components set the appropriate parameters and press Recalculate m Generalization To Random ALL Random CASES Random READERS Needed Input Number of Readers Effect Size Desired Power 1 beta 0 8 Alpha jos me Close 20 JAFROC sample size calculator If
15. luded in the zip file The file plot_template xls is not an input file It is intended for creating FROC and AFROC plots Note text input format is no longer supported Truth worksheet HR Ble gi Yew poet mat Joos pto Window f teed 20 si B U ESIH Sho 44 Label Meaning This integer field identifies the case or patient or image Cases must be CaselD uniquely labeled with integers the integers do not have to be sequentially numbered This integer field tells the program how many lesions are in a particular image and uniquely identifies them If this number is zero the case is considered to be normal If a case has multiple lesions then these must be uniquely identified with integers they do not need to be sequential For weighted analysis the lesion IDs in the Truth and TP worksheets must refer to the same lesions For un weighted analysis matching is not necessary E g 11 2 31 and 4 for an abnormal case with 4 lesions 1 and 2 for an abnormal case with 2 lesions O for a normal case LesionID This positive floating point value is the weight corresponding to the lesion ID For each abnormal case there must be as many weights as there are lesions e g 0 1 0 2 0 3 and 0 4 for a 4 lesion case 0 5 and 0 5 for a 2 lesion case Weien equally weighted Note that the sum of weights for a case must be 1 0 to within 1 Fora normal case the weight field must be zero TP lesion localization Wo
16. r defined FROC analysis by accessing the built in DLLs See JAFROC Development Kit Guide doc for further details Installing the software Install Microsoft Visual C 2010 Redistributable Package x86 before installing JAFROC software It is available at http www microsoft com downloads en details aspx familyid A7B7A05E 6DE6 4D3A A423 37BF0912DB84 amp displaylang en Search for Microsoft Visual C 2010 Redistributable Package x86 on Google to get to this site Double click JAFROC MSI file OA L gt JAFROC Organize v jp Install Share with Burn New folder Fe Favorites EE Desktop r Downloads E Recent Places Pe Libraries Documents a Music E Pictures E Videos Computer amp Local Disk C amp iDisk on psf V E hongjuny on psf V G work on psf X GA Home on psf Y G Host on psf Z Gia Network JAFROC Date modified 1 11 2011 12 27 PM Date created 1 11 2011 1 45 PM Windows Installer Package Size 3 63 MB Installation creates a JAFROC folder in the Program Files folder and JAFROC Data folder in Documents folder and copies a sample JAFROC data file DataFile xls and plot_template xls file to this folder Click on Next a JAFROC a Welcome to the JAFROC Setup Wizard The installer will quide you through the steps required to install JAFROC on your computer WARNING This computer program is protect
17. rksheet Hl Bile Edit view Insert Format Tools Data Window StatsDirect Help Adobe PDF Type a Arial z010 z BZ ee Sih 38 dS e c D E CaselD LesioniD TP_Rating Label Meaning ReaderID This integer field identifies the readers The entries must be unique integers e g 1 or 2 or 3 for 3 readers ModalityID This integer field identifies the modalities e g 1 or 2 for 2 modalities CaselD As in Table 1 Lesion ID As in Table 1 TP_Rating This positive integer floating point value is the rating assigned to a particular TP mark Higher numbers represent greater confidence that the location is actually a lesion For images with multiple lesions the ratings must correspond to the LesionIDs in the truth table unless the lesions are equally weighted Otherwise the rating associated with a particular lesion will not match its weight FP non lesion localization Worksheet iS file Edit View Insert Format Tools Data Window StatsDirect Help Adobe PDF PrE z210 zB ZU E ERS E fe ReaderlD BENER D ReaderlD IModalityID CaselD FP_Rating 1 1 107 1 109 a o gt WTP EP TuthZ i Draw 3 AutoShapesy a O 4d A oo a Lal Oy eh Label Meaning ReaderID As in Table 2 ModalityID As in Table 2 CaselD As in Table 1 but on this sheet the cases must be grouped if two or more false positives occur for the sam
18. ts ro Number of Normal Cases 100 13 Number of Abnormal Cases 100 fraction normal cases 0 500 min LESIONS_PER_IMAGE 1 max LESIONS_PER_IMAGE 4 mean LESIONS_PER_IMAGE 2 5 Total LESIONS 20 Inc Loc Frac gt 0 031 For TRT 1 and RDR 1 max FPF 0 990 For TRT 1 and RDR 2 Max FPF 0 990 For TRT 1 and RDR 3 max FPF 1 000 For TRT 1 and RDR 4 Max FPF 0 990 For TRT 1 and RDR 5 max FPF 0 990 For TRT 2 and RDR 1 max FPF 1 000 For TRT 2 and RDR 2 max FPF 1 000 For TRT 2 and RDR 3 max FPF 1 000 For TRT 2 and RDR 4 max FPF 1 000 For TRT 2 and RDR 5 max FPF 1 000 FOM Trapezoidal area under AFROC curve ignoring NLs on abnormal cases FOMs for TREATMENTS columns and READERS rows 14 0 6654 0 6486 0 6867 0 6963 0 7447 0 7067 0 7118 0 7602 0 6416 0 7416 Results RANDOM_ALL F statistic and p value Significance level of test alpha 0 05 F statistic 0 382 p value 5 469E 001 DF Num Den 1 1 370E 001 N B If the p value is less than 0 05 there is a significant difference between at least one pair of TREATMENTS READER averaged FOMs for TREATMENTS and 95 confidence intervals 0 6901 0 6286 0 7515 0 7107 0 6506 0 7708 INTER TREATMENT differences and 95 confidence intervals FOM 1 FOM 2 0 0206 0
Download Pdf Manuals
Related Search