Validate Workflow User Manual
Contents
1. 16 5 Visualizing the Validation 20 5 1 How to Use the Demonstrate R Package 20 5 2 How to Use Demonstrate on the DE 20 5 3 How to Use Demonstrate on Atmosphere 20 6 Acknowledgements 20 7 How to Interpret Performance Measures 20 Tels ZAC ties tute ee x Ghett soe ek eth Aya pay ie ee NAA 20 MAD A ee he obs Sadek df De hee ac lla ee Bd Sa ale ee HES ae 20 P Spies tae Ne BS ANS NO Ne th WRN a Gates ty datas Jat di 20 TA TPR ig ass uh etd Be oe NESS Raped a la abi BB 20 RIEBER o sadre dep dete della ec Bd Ba alee oe Nh amp eh ete 20 TO ACCUraCy lt Z woe bt Bae Sot eae eb ALY 20 Te MAE eoin ee Be Sn dy the Pe te ERA NR tae Ee Rais cit 20 T8 RMSE we goi a bak RAS ee his a i i ee ed 20 Validate A Workflow for Evaluating GWAS QTL Tools ONE LAZING MORNING AT AN IPLANT LABORATORY FOR ESTIMATING GENOTYPE TO PHENOTYPE ASSOCIATIONS 1 Introduction 1 JUST INVENTED A NEw METHOD SECONDS PASS YEARS OF RESEARCH NICE f ENDLESS HOURS OF K Z oF FLAWLESS ENGINEERING T 9 EOUN PPS COMPARES WITH OTHER THE GwASENATOR GWASENATOR LIKE TOOLS UMM NO a 1 1 About the Developers Project Lead and Principal Investigator Statistical Analyst and Developer 1 2 Why Validate Dr Ann Stapleton works at the interfaces such as the junctions between research and teaching individual research projects and large collabora tive projects the organizatio2. The second major step is accomplished once all the runs have finished In our example depending on the resources available PLINK can finish in a range from ten seconds to two and half minutes per run not including the queuing process This is non linear as well because many will be running simultaneously on separate nodes while some remain in queues Its hard to gauge exactly how long your tool will take but it will almost definitely run faster on multiple nodes this way Step 2 A Log in to the DE and make sure all runs are complete Let s say that we have already run all of our runs Then we would want to log in to the iPlant Discovery Environment to make sure all the folders are in our analyses folder As can be seen each run of PLINK with each distinct known truth data set has been put in to each folder If we were to open up each folder we would see a set of files of which only one we will want to put through Validate In the case of this PLINK run you need to put each file ending in qassoc the standard PLINK output filetype in to a single folder so that we can run Validate on that folder 10 Validate A Workflow for Evaluating GWAS QTL Tools OO Dashboard iPlant Collab x Discovery Environment x ay eis ET Apps fi D preview iplantcollaborative org de Q hittp emx io Dustin Landers Da J Discovery Environment Data Z Upload lt lt New Folder Navigat
3. Validate includes classification measures such as AUC and so you must include known SNPs file However the known effects file is optional Step 3 C Upload your known SNPs file and the known effects file using the DE Upload both of these text files using the Discovery Environment This is extremely easy to do It also does not matter where you upload them to Step 3 D Run Validate It s time to run Validate finally Open up the Apps icon on the DE and select Validate Then you need to just fill in the form Select a name and a description of your choice Then fill in the inputs section For the folder containing GWAS QTL you input the folder that contains all of your tool outputs In our case our PLINK outputs are contained in the folder dalanders analy ses my_Validate_folder The truth file is what we have been referring to as the known SNPs file and the effects file is what we have been referring to as the known effects file Again they need to be in that format On the column names and additional options tab input the names of columns that we discussed earlier For us it was P SNP and BETA 18 Validate A Workflow for Evaluating GWAS QTL Tools D Validate Oooo Analysis Name Validate_analysisi 7 Inputs a Column Names and Additional Options a Column name of variable measuring SNP importance P Column name of variable containing SNP names SNP Column name of variable containing estimated SNP ef
4. wish to just select the whole list you can select all between two files by using Shift Click We also provide a method selecting many at us by using the Select All Containing logic button at the bottom of the listbox For example our analyses folders all contain the words GenotypeData or even Trait We can type in the box just to the right of Select All Containing Trait and then click the button This should select just the folders we are interesting obtaining the contents of We can scroll up and down to make sure that we have only selected the ones that we need If we have selected a few that we didn t intend to We can select those using the mouse and then click the Delete button to remove those from the list eoo Aggregate Aggregate An Application for iPlant Collaborative Quit _ Delete Folders After Moving Files Username dalanders View Contents of Selected Folders Password Yaara Select File Type View Folders in Directory dalanders analyses Move Files Select Folders to View Select Folders and or Files to Move Log dalanders analyses PheNPStruct_086_Trait_H2_04_GenotypeD dalanders analyses PheNPStruct_087_Trait_H2_03_GenotypeD dalanders analyses PheNPStruct_088_Trait_H2_04_GenotypeD dalanders analyses PheNPStruct_089_Trait_H2_03_GenotypeD dalanders analyses PheNPStruct_089_Trait_H2_03_GenotypeD dalanders analyses PheNPStruct_090_Trait_H_06_GenotypeDat dalanders an
5. all the tool out puts from Step 1 that you want to be analyzed using the same known truth metadata in to aggregate folders For example say we are using simulations that are generated using varying levels of heritability Since the varying heritability values in essence produce different SNP effects we need to essentially run Validate three sep arate times Validate requires an input on an entire folder and then iterates over those tool outputs So the first step here is to decide how many different runs we need to do and then create that many folders We provide a GUI tool Aggregate that allows you to select files from multiple folders on your iPlant data store multiple runs of a single tool and move those or aggregate them in to a single folder Section 4 3 Running Validate on the Aggregated Folder Once you have all your outputs in aggregated folders You can simply log in to the iPlant Discovery Environment and run Validate on that folder You must have at least two columns with header names in the outputs The name of the SNP column and the name of threshold column such as P value Further if you wish the get back effect size estimation errors you must also know the column on the estimated effect size column for example PLINK s is BETA Text files with known SNPs and effect sizes must also be included Once you submit Validate you will receive a notification when it is completed The Validate output will be columns o
6. data sets looked like you would see that phenotype heritability information and population structure are all in the naming format for each data set This is an important piece of information because without an appropriate naming structure there would be no way for us to keep metadata on the known truth data sets throughout this testing process We break up the names to grab the pieces about phenotype information in order to essentially stratify sample our data sets by genotype We then have six data sets for any one genotype three missing values types and two population structure types and then we just pick two at random This insures that we include all of our genotypes in our runs but there are simpler ways to do this that basically just random sample from all of our data sets The following code again assumes you are using PEDMAPs to run PLINK but that you just want to sample a random 300 as opposed to a stratified random sample set seed require rPlant Validate username password mydir lt ListDir simulations mydir lt mydir 1 oddsamp lt seq 1 599 2 totest lt sample oddsamp 300 FALSE for i in l length totest SubmitJob job name NULL application plink 1 07ul file list list mydir totest i mydir totest i 1 file path simulation VVVVVVVV 3 Aggregating with Aggregate 3 1 How to Use the GUI Aggregate Tool
7. function x y 2 broken lt unlist strsplit x _ TRUE return broken y VVVVVV mydir lt mydir 1 for i in 1l length mydir genos i lt break data files i 2 gens lt na omit unique genos gt oddsamp lt seq 1 11 2 gt totest lt list gt for i in 1 length genoa myfiles lt list for j in 1 length files 1 if breakdat files j 2 genos i myfiles lt append myfiles files j gt myfiles lt myfiles sample oddsamp 2 FALSE gt myfiles lt unlist myfiles gt for k in l length files for p in 1l length myfiles if files k myfiles p totest lt append totest k gt totest lt unlist totest gt for i in 1 length totest SubmitJob job name NULL application plink 1 07ul file list list files totest i files totest i 1 file path simulations print i Every situation is unique in this case which is part of the reason we have yet to develop a graphical user interface method for submitting jobs All Validate A Workflow for Evaluating GWAS QTL Tools 9 that we did in the code above is demonstrate how rPlant may be used to break up some files that had naming conventions that were separated by underscores If you look back to the earlier image that showed what our known truth
8. 3 738 2 373 0 004842 1 575 0 1158 gt i peip 1 AWAW4 4 512 3 738 2 373 0 004842 1 575 0 1158 1 AWAWS 5 512 2 264 0 9708 0 01056 2 333 0 02006 gt J PheHa 1 AWAW6 6 512 0 9807 1 067 0 001653 0 919 0 3585 1 AWAW7 7 512 3 003 1 678 0 006239 1 789 0 07415 gt i Phehas 1 AWAWS 8 512 1 607 0 796 0 007929 2 019 0 04401 1 AWAW9 9 512 1 411 1 191 0 002744 1 185 0 2367 gt L PheHa 1 AWAWLO 10 512 0 7191 1 068 0 0008889 0 6736 0 5009 gt i Phettas 1 AWAW11 11 512 1 04 0 6927 0 004404 1 502 0 1337 1 AWAW12 12 512 999 1 066 0 002997 1 238 0 2162 gt i Phelta 1 AWAW13 13 512 0 9997 0 3982 0 01221 2 511 0 01236 1 AWAW14 14 512 2 853 2 375 0 00282 1 201 0 2303 gt i Pheas 1 AWAW15 15 512 0 2873 0 9043 0 0001978 0 3176 0 7509 E 1 AWAWL6 16 512 3 042 1 185 0 01275 2 566 0 01056 gt I PheHa 1 AWAW17 17 512 1 809 0 9726 0 006735 1 86 0 06352 gt C Pheltas 1 AWAW18 18 512 1 704 0 973 0 005978 1 751 0 08048 1 AWAW19 19 512 0 9714 0 6426 0 004461 1 512 0 1312 gt I PheHas 1 AWAW20 20 512 2 045 2 377 0 001449 0 8604 0 39 1 AWAW21 21 512 0 04449 0 3365 3 427e 05 0 1322 0 8949 s 1 AWAW22 22 512 2 245 2 377 0 001746 0 9445 0 3454 1 AWAW23 23 512 0 494 0 7989 0 000749 0 6183 0 5367 1 AWAW24 24 512 0 1613 2 379 9 02e 06 0 06783 0 946 1 E19 A ananens A FAD Step 3 B Gather known SNP and known effect text files The known SNP file contains essentially just text with a list of all the SNPs with known effects in our case its a list
9. 3_Trait_H_06_GenotypeData_3pctMissing_ assoc qassoc dalanders analyses PheHasStruct_021_Trait_H_06_GenotypeDaj003_Trait_H2_04_GenotypeData_NoMissing_ assoc qassoc dalanders analyses PheHasStruct_021_Trait_H2_03_GenotypeQ 005_Trait_H2_03_GenotypeData_NoMissing_ assoc qassoc dalanders analyses PheHasStruct_022_Trait_H2_03_GenotypeDt_005_Trait_H2_04_GenotypeData_3pctMissing_ assoc qasso dalanders analyses PheHasStruct_022_Trait_H2_04_GenotypeDt_006_Trait_H2_04_GenotypeData_3pctMissing_ assoc qasso dalanders analyses PheHasStruct_023_Trait_H2_03_GenotypeQ 006_Trait_H2_04_GenotypeData_NoMissing_ assoc qassoc dalanders analyses PheHasStruct_024_Trait_H_06_GenotypeDaj07_Trait_H_06_GenotypeData_NoMissing_ assoc qassoc dalanders analyses PheHasStruct_025_Trait_H2_03_GenotypeDt_007_Trait_H2_03_GenotypeData_3pctMissing_ assoc qasso dalanders analyses PheHasStruct_025_Trait_H2_03_GenotypeDt_007_Trait_H2_03_GenotypeData_8pctMissing_ assoc qasso dalanders analyses PheHasStruct_025_Trait_H2_04_GenotypeDt_008 _Trait_H2_03_GenotypeData_3pctMissing_ assoc qasso dalanders analyses PheHasStruct_026_Trait_H2_03_GenotypeDt_009_Trait_H2_03_GenotypeData_8pctMissing_ assoc qasso dalanders analyses PheHasStruct_027_Trait_H2_04_GenotypeQ 009_Trait_H2_03_GenotypeData_NoMissing_ assoc qassoc dalanders analyses PheHasStruct_028_Trait_H_06_GenotypeDaft_010_Trait_H2_03_GenotypeData_8pctMissing_ assoc qasso dalanders analyses PheHasStruct_028_Trait_H2_03_GenotypeDt_011_Trait_H2_04_G
10. HasStruct_001 CJ PheHasStruct_002 PheHasStruct_003 C PheHasStruct_003_Trait_H2_04_G J PheHasStruct 003 C PheHasStruct_003_Trait_H_06_Ge L7 PheHasStruct_005 C PheHasStruct_005_Trait_H2_03_G G my_Validate_analysis G PheHasStruct_001_Trait_H2_03_G C PheHasStruct_002_Trait_H2_03_G y Search by Name Last Modified 2014 Jan 24 13 21 13 2014 Jan 24 13 55 29 2014 Jan 24 13 55 42 2014 Jan 24 13 56 00 2014 Jan 24 13 55 56 2014 Jan 24 13 56 24 Step 2 E Enter the folder location where all your tool analyses are located and click View Folders in Directory We will enter dalanders analyses and click the button View Folders in Directory This is provide us with a list of all the files and folders in that directory on the iPlant data store in the left most listbox Aggregate Aggregate An Application for iPlant Collaborative Username dalanders Password prere View Folders in Directory dalanders analyses Select Folders to View dalanders analyses dalanders analyses ExtractR_2013 12 04_14 11 30 608 dalanders analyses fast lmm dalanders analyses my_Validate_analysis dalanders analyses PheHasStruct_001_Trait_H2_03_GenotypeD dalanders analyses PheHasStruct_002_Trait_H2_03_GenotypeD dalanders analyses PheHasStruct_003_Trait_H_06_GenotypeDa dalanders analyses PheHasStruct_003_Trait_H2_04_GenotypeD dalanders analyses PheHasStruct_005_Trait_H2_03_GenotypeD dalander
11. Share Search by Name Ta madhe e Name Last Modified Size Details E pPheHasstruct_001_Trait_H2_03_G 2014 Jan 24 13 55 29 Saas fe cr e 1o C PheHasStruct_002_Trait_H2_03_G C PheHasStruct_003_Trait_H2_04_G PheHasStruct_003_Trait_H_06_Ge 2014 Jan 24 13 55 42 2014 Jan 24 13 56 00 2014 Jan 24 13 55 56 PheHasStruct_005_Trait_H2_03_G C PheHasStruct_005_Trait_H2_04_G C PheHasStruct_006_Trait_H2_04_G PheHasStruct_006_Trait_H2_04_G G PheHasStruct_007_Trait_H2_03_G PheHasStruct_007_Trait_H2_03_G PheHasStruct_007_Trait_H_06_Ge gt PheHasStruct_008_Trait_H2_03_G 1 C PheHasStruct_009_Trait_H2_03_G C PheHasStruct_009_Trait_H2_03_G PheHasStruct_010_Trait_H2_03_G C PheHasStruct_011_Trait_H2_04_G C PheHasStruct_012_Trait_H2_04_G G PheHasStruct_013_Trait_H2_04_G G PheHasStruct_013_Trait_H_06_Ge C PheHasStruct_014_Trait_H2_04_G 2014 Jan 24 13 56 24 2014 Jan 24 13 56 28 2014 Jan 24 13 56 43 2014 Jan 24 13 56 47 2014 Jan 24 13 57 01 2014 Jan 24 13 57 06 2014 Jan 24 13 56 56 2014 Jan 24 13 57 11 2014 Jan 24 13 57 26 2014 Jan 24 13 57 30 2014 Jan 24 13 57 40 2014 Jan 24 13 57 54 2014 Jan 24 13 58 08 2014 Jan 24 13 58 27 2014 Jan 24 13 58 22 2014 Jan 24 13 58 41 o Feedback Step 2 B I
12. Trait H 06 GenotypeData_3pctMissing map K 2 PheHasStruct_001_Trait_H_ 06 GenotypeData_3pctMissing ped 3 PheHasStruct_001_ Trait H 06 GenotypeData_8pctMissing map 4 PheHasStruct_001 Trait H 06 GenotypeData_8pctMissing ped 5 PheHasStruct_001 Trait H 06 GenotypeData_NoMissing map 6 PheHasStruct_001_ Trait H 06 GenotypeData_NoMissing ped As can be seen for our particular structure we would need to submit two files for each run So let s walk through it We first can list all the apps in order to find ours which we have already installed so that we can get the exact name that we need to use for the SubmitJob function gt ListApps Validate A Workflow for Evaluating GWAS QTL Tools 7 e098 R Console w ORMA W h e ak 9 Desktop SyngentaReportData Q7 Help Search 34 35 s 36 q 37 is 38 39 40 3P 41 a 42 43 44 h fas 46 47 48 49 50 5 a 52 53 54 55 56 57 9 58 T 59 60 ig 62 62 63 yr 64 65 66 67 68 69 70 71 U 0 Luz GMAP_stampede 121212u1 GSNAP_lonestar 121212u1 GSNAP_stampede 121212u2 head stampede 5 97u2 idbaUD 1 0 0u2 interproscan 5 44 0u1 macs ranger 1 4 1 4u1 mafft 7 113u1 mafft lonestar 6 864u1 mafftDispatcher 1 0 13100u1 MergeG2P 0 0 1u1 metagenemark 1 00u3 metaphlan lonestar 1 6 0u4 MLMM 0 0 1u1 MrBayesmpi_ba
13. Validate A Workflow for Evaluating the Performance of GWAS QTL Tools Using Known Truth Datasets Contents Ann Stapleton Kurt Michels and Dustin Landers University of North Carolina Wilmington Abstract Understanding the effectiveness of Genome Wide Association GWAS and Quantitative Trait Loci QTL analytical tools under various situations is crucial to deciding which tools are best given a particular problem Validate provides a way to re turn classification and regression performance measures for large amounts of tool outputs generated using known truth simulations We also provide solutions for aggregating hundreds or thousands of outputs in to a single folder on the iPlant data store so that Validate can be used Introduction 3 1 1 About the Developers 2 3 1 2 Why Validate g t t RR amp Be TREE a Bae a eee 9 3 1 3 Getting iPlant Credentials 00 4 1 4 Getting Your Application on an API 4 1 5 Where Can I Get These Softwares oaoa 4 1 6 How to Use This Manual 4 Running Simulations 6 2 1 How To Run Simulations with rPlant 6 2 2 Using rPlant to Sample From Your Known Truth Data Sets 9 Aggregating with Aggregate 10 3 1 How to Use the GUI Aggregate Tool 10 3 2 How to Use the Aggregate Function in rPlant 16 3 3 How to Use Aggregate on Atmosphere 16 Validating with Validate 16 4 1 How to Use Validate on the DE
14. alyses PheNPStruct_090_Trait_H2_03_GenotypeD dalanders analyses PheNPStruct_091_Trait_H_06_GenotypeDat dalanders analyses PheNPStruct_091_Trait_H2_03_GenotypeD dalanders analyses PheNPStruct_091_Trait_H2_04_GenotypeD dalanders analyses PheNPStruct_092_Trait_H_06_GenotypeDat dalanders analyses PheNPStruct_092_Trait_H2_04_GenotypeD dalanders analyses PheNPStruct_094_Trait_H_06_GenotypeDat dalanders analyses PheNPStruct_094_Trait_H2_03_GenotypeD dalanders analyses PheNPStruct_094_Trait_H2_04_GenotypeD dalanders analyses PheNPStruct_096_Trait_H_06_GenotypeDat dalanders analyses PheNPStruct_096_Trait_H_06_GenotypeDa dalanders analyses PheNPStruct_096_Trait_H2_04_GenotypeD dalanders analyses PheNPStruct_097_Trait_H_06_GenotypeDa dalanders analyses PheNPStruct_097_Trait_H2_03_GenotypeD dalanders analyses PheNPStruct_098_Trait_H_06_GenotypeDa dalanders analyses PheNPStruct_098_Trait_H2_03_GenotypeD dalanders analyses PheNPStruct_098_Trait_H2_04_GenotypeD dalanders analyses PheNPStruct_099_Trait_H2_03_GenotypeD dalanders analyses PheNPStruct_100_Trait_H2_03_GenotypeD Select Delete Select Delete Clear Select All Containing Trait Select All Containing Step 2 G Click View Contents of Selected Folders Once you ve got all your folders in the first listbox click the button on the Validate A Workflow for Evaluating GWAS QTL Tools 13 right call View Contents of Selected Folders and this will iterate over the fold
15. ce for distribution details Natural language support but running in an English locale R is a collaborative project with many contributors Type contributors for more information and citation on how to cite R or R packages in publications Type demo for some demos help for on line help or help start for an HTML browser interface to help Type q to quit R R app GUI 1 62 6558 x86_64 apple darwinl0 8 0 Workspace restored from Users dustin Desktop SyngentaReportData RData History restored from Users dustin Desktop SyngentaReportData Rapp history gt If you haven t already installed rPlant we recommend that you do that After that you need to validate your iPlant credentials so that you can access your simulation files Then you will need to get a list of the names of your simulations files and iterate over them in a for loop submitting jobs to your installed application Step 1 B Consider the strategy you want to use Do you want to use all your known truth data sets or just a sample And if a sample should you take a stratified sample Validate A Workflow for Evaluating GWAS QTL Tools These are mainly just exam ple codes Although you are welcome to copy ours The following code is to just an example of how the submission may go assuming your known truth data sets are in something like a raw format where each known truth data set is stored in a single file rPlant a
16. earlier or if they all have the same file extension as ours do with gassoc then we can use another option Above the right most listbox is a button labelled Select File Type Next to that we type gassoc in and then click the button in order to just select the file types that we want to move 14 Validate A Workflow for Evaluating GWAS QTL Tools Aggregate Username Password View Folders in Directory Aggregate An Application for iPlant Collaborative Quit i _ Delete Folders After Moving Files dalanders View Contents of Selected Folders dalanders analyses Move Files Select Folders to View Select Folders and or Files to Move Log dalanders analyses PheNPStruct_086_Trait_H2_04_GenotypeDg4001_Trait_H2_03_GenotypeData_NoMissing_ assoc qassoc iew success dalanders analyses PheNPStruct_087_Trait_H2_03_GenotypeDgt_002_Trait_H2_03_GenotypeData_8pctMissing_ assoc qassoqView success dalanders analyses PheNPStruct_088_Trait_H2_04_GenotypeDq4003_Trait_H_06_GenotypeData_3pctMissing_ assoc qassoc iew success dalanders analyses PheNPStruct_089_Trait_H2_03_GenotypeDq003_Trait_H2_04_GenotypeData_NoMissing_ assoc qassoc iew success dalanders analyses PheNPStruct_089_Trait_H2_03_GenotypeDqg005_Trait_H2_03_GenotypeData_NoMissing_ assoc qassoc jew success dalanders analyses PheNPStruct_090_Trait_H_06_GenotypeDatft_005_Trait_H2_04_GenotypeData_3pctMissing_ assoc qassoqdView success dalanders analyses PheNPStr
17. enotypeData_3pctMissing_ assoc qasso dalanders analyses PheHasStruct_029_Trait_H_06_GenotypeDalt_012_Trait_H2_04_GenotypeData_3pctMissing_ assoc qasso dalanders analyses PheHasStruct_030_Trait_H_06_GenotypeDaj13_Trait_H_06_GenotypeData_NoMissing_ assoc qassoc dalanders analyses PheHasStruct_030_Trait_H2_04_GenotypeQ 013_Trait_H2_04_GenotypeData_NoMissing_ assoc qassoc dalanders analyses PheHasStruct_032_Trait_H_06_GenotypeDal014_Trait_H_06_GenotypeData_3pctMissing_ assoc qassoc dalanders analyses PheHasStruct_032_Trait_H_06_GenotypeDalt_014_Trait_H2_04_GenotypeData_3pctMissing_ assoc qasso dalanders analyses PheHasStruct_033_Trait_H_06_GenotypeDal015_Trait_H_06_GenotypeData_8pctMissing_ assoc qassoc dalanders analyses PheHasStruct_033_Trait_H2_03_GenotypeDt_015_Trait_H2_04_GenotypeData_8pctMissing_ assoc qasso dalanders analyses PheHasStruct_034_Trait_H_06_GenotypeDaft_016_Trait_H2_04_GenotypeData_8pctMissing_ assoc qasso dalanders analyses PheHasStruct_036_Trait_H2_03_GenotypeDt_017_Trait_H2_03_GenotypeData_8pctMissing_ assoc qasso Select Delete Select Delete Clear Select All Containing Trait Select All Containing 3 2 How to Use the Aggregate Function in rPlant 3 3 How to Use Aggregate on Atmosphere 4 Validating with Validate 4 1 How to Use Validate on the DE Step 3 A First take a look at one of your analyses documents on the DE Check for four things The nature of the column delimitation the name of the SNP column t
18. ers in the left most listbox and add the contents of all those folders to the right most listbox It should take a few minutes so give it some time Once its done see if you can spot the files that you were interested in moving In our case they are the files ending in gassoc e098 Aggregate Aggregate An Application for iPlant Collaborative Quit _ Delete Folders After Moving Files Username dalanders View Contents of Selected Folders Password seiahilaichaie adalah Select File Type View Folders in Directory dalanders analyses Move Files E l Select Folders to View Select Folders and or Files to Move Log dalanders analyses PheNPStruct_086_Trait_H2_04_GenotypeDadalanders analyses PheHasStruct_001_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_087_Trait_H2_03_GenotypeDadalanders analyses PheHasStruct_001_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_088_Trait_H2_04_GenotypeDadalanders analyses PheHasStruct_001_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_089_Trait_H2_03_GenotypeDadalanders analyses PheHasStruct_001_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_089_Trait_H2_03_GenotypeDddalanders analyses PheHasStruct_001_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_090_Trait_H_06_GenotypeDatjdalanders analyses PheHasStruct_001_Trait_H2_03_GenotypeD View success dalanders analyses PheNPStruct_090_Trait_H2_03_Genot
19. f performance measures for each tool output Section 5 4 Running Demonstrate on the Validate Output Finally once you have outputs from Validate which may be more than one you 4 Validate A Workflow for Evaluating GWAS QTL Tools need to combine them and test the differences in your measures be tween your simulation s parameters Demonstrate is an R package to be installed on the iPlant DE that quickly combines all your re sults files in a folder you specify and returns sciplot factorial graphs for parameters you specify It also returns the combined results file so that you can perform your own analyses Validate A Workflow for Evaluating GWAS QTL Tools 5 2 Running Simulations 2 1 How To Run Simulations with rPlant Step 1 A Install R and rPlant We will first show you how to batch run simulations with rPlant First we are using the latest R version 3 0 2 When you open the R console it should look something like this we are using a Mac so yours may look slightly different but it will function essentially the same e00 R Console S iel Se gt m ORwo Bo s B Desktop SyngentaReportData Q7 Help Search R version 3 0 2 2013 09 25 Frisbee Sailing Copyright C 2013 The R Foundation for Statistical Computing Platform x86 64 apple darwinl0 8 0 64 bit R is free software and comes with ABSOLUTELY NO WARRANTY You are welcome to redistribute it under certain conditions Type license or licen
20. fect BETA Severity Ratio 0 00004486472 Severity Ratio 2 1 Severity Ratio 3 Enter text We also input severity ratios here as well for the H measure We advise that you always do at least two the first one should be the proportion of positives divided by the proportion of negatives In our case that would To see more about how these be 35 780000 The second one should be just 1 severity ratio priors influence the H meastire see seetion 72 Validate A Workflow for Evaluating GWAS QTL Tools 19 5 Visualizing the Validation 5 1 5 2 5 3 How to Use the Demonstrate R Package How to Use Demonstrate on the DE How to Use Demonstrate on Atmosphere 6 Acknowledgements 7 How to Interpret Performance Measures 7 1 AUC 7 2 H 7 3 KS 7 4 TPR 7 5 FPR 7 6 Accuracy 7 7 MAE 7 8 RMSE hmeasure net 20 Validate A Workflow for Evaluating GWAS QTL Tools
21. he name of P value or other importance scoring column and the name of an effect size column if there is one For example our SNP column is SNP our importance column is P and our effect size column is BETA You are looking for those four things mentioned The goal is to assess how well PLINK was at identifying those known SNPs as compared to the ones where there are no effects We are also going to include a SNP effects text file so that we can measure the effectiveness for PLINK at determining those 16 Validate A Workflow for Evaluating GWAS QTL Tools O Data oeoo L Upload Eg New Folder Refresh Download Edit gf Share v Search by Name S Trash Navigation v LastModi Details gt gt 4 i dalanders rait_H2_03_GenotypeData_NoMissing map 2014 Jan Last Modified 2014 Jan 24 4 analyses rait_H2_03_GenotypeData_NoMissing ped 2014 Jan Date Submitted 2014 Jen 24 C fast imm tat HLI woel NoMissin joa 2014 B issi raj 7 5 enoty ata NoMissing ASSOC 014 Jan Issions gt mr va PheHasStru 001 Trait H2_03_GenotypeData_NoM nahii aiii C pies PheHasStruct_001_Trait_H2_03_GenotypeData_NoMissing_ assoc qassoc gt G PheHa P C Pheha Page Size KB P j4 1 of 8762 gt pi gt L PheHas CHR SNP BP NMISS BETA SE R2 P gt J PheHa 1 AWAWL 1 512 0 5428 0 321 0 005573 1 691 0 09151 l 7 AWAW2 2 512 1 373 2 378 0 0006536 0 5776 0 5638 p C Phera 1 AWAW3 3 512
22. ion 4 i dalanders 4 2 analyses fast mm 1 7 PheHasStruct_001 CO PheHasStruct_002 CI PheHasStruct_003 L7 PheHasStruct_003 PheHasStruct_005 PheHasStruct_005 Co PheHasStruct_006 PheHasStruct_006 Co PheHasStruct_007 CI PheHasStruct_007 PheHasStruct_007 Cl PheHasStruct_008 CO PheHasStruct_009 PheHasStruct_009 L7 PheHasStruct_010 PheHasStruct_011 PheHasStruct_012 Co PheHasStruct_013 PheHasStruct_013 CO PheHasStruct_014 KJ PheHasStruct_ 014 If you are testing for effect sizes that are different for dif ferent levels of heritability you may want to create more fold ers in this step Actually for the rest of this process you would have to everything mul tiple times for each folder you create here Say for example I m testing three different her itability levels which will have three different groups of actual effects Then I would need to run Validate three different times and thus need to make three folders in this step and separate them that way Note that in rPlant your don t have to type your user name and you begin iPlant data store file locations imme diately For example in rPlant the same location would just be written as analyses Wolfram Alpha Dev S Wolfram Alpha Con Data Science Kit C Function Reference y PostgreSQL add orc PostgreSQL Permiss Notifications oeoo Refresh Download v Edit g
23. ir i 1 1 print i This should iterate over each simulation in our data set and submit a PLINK job to the iPlant hardware Since iPlant uses a queuing system Validate A Workflow for Evaluating GWAS QTL Tools this effectively distributes your workload over many computer nodes as needed This is obviously helpful in achieving a high volume of runs even outside of known truth testing Now you know how 2 2 Using rPlant to Sample From Your Known Truth Data Sets Let s say for example that we had 600 total known truth data sets Our known truth datasets are generated from hundred different genotypes with three different types of simulated missing values such as 0 3 and 8 In addition we have data generated with and without popula tion structure meaning the stochastic processes generated the data have either a single population mean or varying group means Say that we didn t want to run all of them but maybe just half of them In many cases you may have substantially more than 600 and thus sam pling makes much more sense then it does in this example We can use some scripting in R and rPlant to accomplish the same thing we did earlier but with some random sampling Here is the code we used to sample 300 known truth PEDMAPs for PLINK set seed require rPlant Validate username password mydir lt ListDir simulations genos lt matrix nrow 1200 ncol 1 break data lt
24. n of international meetings and high school teacher inquiry labs Most recently she has worked at the interface be tween plant biology and software engineering leading the way to broad methods applicable to evaluating genotype to phenotype analytical meth ods Dustin Landers received a BS from Appalachian State where he learned a lot about survey research and using statistics to solve problems and answer questions He continued his education at UNCW by studying Applied Statistics Moving from world of polling and survey statistics to the that of Big Data he has an interest in bridging the gap between statistics and software engineering and seeing the combined discipline brought to bear on problems once considered impossible To test known truth data sets we originally weren t sure of the best way to approach the problem We had the idea of using ROC plots We wanted to look at true positive rates false positive rates and accuracy Of course this was never really feasible because any one simulation run could be atypical and all of these methods could only analyze a single simulation run We asked ourselves how could we run lots of simulations through a tool and test the outputs in a way that gave us insight in to the performance of these GWAS and QTL tools We decided that in order to allow lots of simulations to be tested we needed a method that that allowed iterations over hundreds or even thou sands of simulation runs This birthed
25. n the DE create a new folder for your Validate analysis The gassoc files in this case are the files containing the actual analysis that we will use for evaluating the effectiveness of PLINK Particularly those are the predicted values for each SNP based on the PLINK algo rithm Now it s time to compare those predicted values with what we know to be the actual values You need to create a folder that we will use for our Validate analysis In the DE create a folder called my_Validate_analysis The goal is to look inside each of the 300 folders we created and pull out the gassoc file and move it to the folder marked my_Validate_analysis Since this would take hours we suggest you launch the application we created called Aggregate that provides a fairly easy way to do this at least comparatively easier Step 2 C Launch Aggregate Once you launched Aggregate you will see there is a place to enter your username and password Go ahead and enter those Step 2 D Enter username and password Now you need to type the out the full location of the folder starting with username where your analyses are located In this case all of our PLINK analyses folders are located in the folder dalanders analyses Validate A Workflow for Evaluating GWAS QTL Tools 11 Data Z Upload 3 New Folder 2 Refresh lt 2 Navigation e Name 4 dalanders G fast lmm 4 J analyses 1 7 fast imm CJ Phe
26. of 35 SNPs For us these are called syntruth txt and synbetas03 txt respectively and we have included screenshots so you can get an idea of what they need to look like ae WAW4335 CWAW11957 CWAW49771 EWAW3698 EWAW11242 EWAW11527 EWAW26390 EWAW31724 EWAW43902 EWAW44879 EWAW56550 EWAW66544 EWAW89186 FWAW29987 FWAW37815 FWAW47509 FWAW52609 MMAW21533 MMAW42211 PWAW58082 RRAW6823 RRAW19403 RRAW44761 RRAW49606 RRAW50636 WWAW32296 WWAW55109 WWAW64744 ZWAW5647 ZWAW13135 ZWAW23310 ZWAW35801 ZWAW48419 ZWAW58428 ZWAW68497 Validate A Workflow for Evaluating GWAS QTL Tools 17 The known effects text file is the same way and it must be in the same order as the SNPs This means looking at the above screenshots that SNP CWAW1335 has effect 0 0324 and so on This information ordering information is used by Validate to assign true effect sizes to all the SNPs in your outputs and in doing so labels all others as zero Also Validate uses the information from the known SNPs file to label all SNPs in your output as either TRUE or FALSE based on whether or not they are included in that file eoo synbetas03 0324 0 0243 0 0004 0 0003 0 0014 0 0554 0 0020 0 0019 0 0019 0 0153 0 0016 0 0037 0 0016 0 0136 0 0016 0 0026 0 0061 0 0004 0 0058 0 0412 0 0056 0 0001 0 0002 0 0020 0 0020 0 0020 0 0034 0 0017 0 0108 0 0454 0 0058 0 0035 0 0003 0 0019 0 0018 This is how performance measures are inherently generated so this must be included By default
27. s analyses PheHasStruct_005_Trait_H2_04_GenotypeD dalanders analyses PheHasStruct_006_Trait_H2_04_GenotypeD dalanders analyses PheHasStruct_006_Trait_H2_04_GenotypeD dalanders analyses PheHasStruct_007_Trait_H_06_GenotypeDa dalanders analyses PheHasStruct_007_Trait_H2_03_GenotypeD dalanders analyses PheHasStruct_007_Trait_H2_03_GenotypeD dalanders analyses PheHasStruct_008_Trait_H2_03_GenotypeD dalanders analyses PheHasStruct_009_Trait_H2_03_GenotypeD dalanders analyses PheHasStruct_009_Trait_H2_03_GenotypeD dalanders analyses PheHasStruct_010_Trait_H2_03_GenotypeD dalanders analyses PheHasStruct_011_Trait_H2_04_GenotypeD dalanders analyses PheHasStruct_012_Trait_H2_04_GenotypeD dalanders analyses PheHasStruct_013_Trait_H_06_GenotypeDa dalanders analyses PheHasStruct_013_Trait_H2_04_GenotypeD dalanders analyses PheHasStruct_014_Trait_H_06_GenotypeDa dalanders analyses PheHasStruct_014 Trait_H2_04 GenotypeD _ Delete Folders After Moving Files View Contents of Selected Folders Select File Type Move Files Select Folders and or Files to Move Quit j Log Select Delete Select All Containing Select Delete Clear Step 2 F Select only folders from the known truth data sets runs 12 Validate A Workflow for Evaluating GWAS QTL Tools Now we want to just select the folders that we just created You can select them all using Control Click to select individual ones or if you
28. s analyses PheNPStruct_097_Trait_H_06_GenotypeDatidalanders analyses PheHasStruct_003_Trait_H_06_GenotypeDalView success dalanders analyses PheNPStruct_097_Trait_H2_03_GenotypeDadalanders analyses PheHasStruct_003_Trait_H_06_GenotypeDalView success dalanders analyses PheNPStruct_098_Trait_H_06_GenotypeDatidalanders analyses PheHasStruct_003_Trait_H_06_GenotypeDalView success dalanders analyses PheNPStruct_098_Trait_H2_03_GenotypeDadalanders analyses PheHasStruct_003_Trait_H_06_GenotypeDajView success dalanders analyses PheNPStruct_098_Trait_H2_04_GenotypeDddalanders analyses PheHasStruct_003_Trait_H_06_GenotypeDalView success dalanders analyses PheNPStruct_099_Trait_H2_03_GenotypeDadalanders analyses PheHasStruct_003_Trait_H_06_GenotypeDalView success dalanders analyses PheNPStruct_100_Trait_H2_03_GenotypeDadalanders analyses PheHasStruct_003_Trait_H_06_GenotypeDalView success Select Delete Select Delete Clear Select All Containing Trait Select All Containing As you can see every single item in the right most listbox is a file or folder contained with one of the folders in the left most listbox Now we need to select all the files from the right most listbox that we wish to move to the folder we created in order to run Validate Step 2 H Select just the tool outputs from the right most list box In order to select just the right files that you intend to move you can use the same Select All Containing logic button we used
29. sic 3 2 1u1 Muscle 3 8 32u4 mascle lonestar 3 8 31u2 newbler 2 6 0u1 NPUTE 0 0 2u1 NumericalTransform 0 0 1u1 oases 0 2 08u1 phylip dna parsimony lonestar 3 69u2 phylip protein parsimony lonestar 3 69u2 plink 1 07u1 prodigal 1 0 0u1 quicktree dm lonestar 1 1u2 quicktree tree lonestar 1 1u2 raxml lonestar 7 2 8u1 ray 2 2 0u1 scarf 1 00u1 soapdenovo trans 1 0ul soapdenovo 1 05u1 soapdenovo 2 04u1 STRUCTURE 2 3 4u2 STRUCTURE 2 3 5u1 STRUCTURE2TASSEL 0 0 1u1 TASSEL GLM 0 0 1u1 TASSEL MLM 0 0 1u1 TASSEL4 GLM 0 0 1u1 tasselDispatcher 1 0 13350u1 TNRS4GWAS 0 0 2u1 trinity_lonestar4 20130814u1 s trinity_normalize_by kmer_coverage_lonestar4 20130814u1 ili velvetg 1 2 07u2 velveth 1 2 07u1 XYPlot 0 0 1u1 We can see that the exact name of PLINK on the Foundation API is plink 1 07ul We will revisit the earlier explanation to see how we would submit a PLINK job using the PEDMAP format Step 1 D Use the short R script to select known truth data sets and submit a job run for each one using your application install packages rPlant require rPlant mydir lt ListDir simulations for gt gt gt Validate username password gt gt i in 1 nrow mydir SubmitJob job name NULL application plink 1 07ul file path simulations file list list mydir i 1 myd
30. ssumes you already include your iPlant data store username in the files path so that iplant home username simulations is just written as simula tions or iplant home username analyses is just written as analyses This is of course because validating your credentials allows you to not have to keep typing you user information over and over again gt install packages rPlant gt require rPlant gt Validate username password gt mydir lt List Dir LU simulations gt for i in 1 nrow mydir Submit Job job name NULL application myGWaASapplication file path simulations file list list mydir i 1 print i Step 1 C Find out the name of your application on the API and the inputs required You need to make sure of the specific application name assigned to your tool in the API and of any special parameters or inputs required in order torun For example PLINK requires a PEDMAP format The PEDMAP format is essentially two files the PED and the MAP file for each known truth data set For example our simulation files are stored in a shared folder where each odd file number is a PED and each even is a MAP Each odd is a PED file and the subsequent file number is a MAP like so 000 R Console w T gt l ea deed Re gt E o ama Ho e o Desktop SyngentaReportData Q7 Help Search gt head files 1 PheHasStruct_001_
31. t_H_06_GenotypeData_NoMissing_ assoc qassoc jew success dalanders analyses PheNPStruct_097_Trait_H_06_GenotypeDat 013_Trait_H2_04_GenotypeData_NoMissing_ assoc qassoc iew success dalanders analyses PheNPStruct_097_Trait_H2_03_GenotypeDa dalanders analyses PheNPStruct_098_Trait_H_06_GenotypeDa dalanders analyses PheNPStruct_098_Trait_H2_03_GenotypeDa dalanders analyses PheNPStruct_098_Trait_H2_04_GenotypeDa dalanders analyses PheNPStruct_099_Trait_H2_03_GenotypeDa dalanders analyses PheNPStruct_100_Trait_H2_03_GenotypeDa Select Select All Containing 014_Trait_H_06_GenotypeData_3pctMissing_ assoc qassoc iew success 014_Trait_H2_04_GenotypeData_3pctMissing_ assoc qassoqView success 015_Trait_H_06_GenotypeData_8pctMissing_ assoc qassoc jew success 015_Trait_H2_04_GenotypeData_8pctMissing_ assoc qassoqView success 016_Trait_H2_04_GenotypeData_8pctMissing_ assoc qassoqView success 017_Trait_H2_03_GenotypeData_8pctMissing_ assoc qassoqView success Delete Select Delete Clear Trait Select All Containing Step 2 I Move the files to your Validate analysis folder Now we can move the files In the text box next to the Move Files button enter the full path of the folder we created just for this purpose in Step 2 B In our case we named it my_Validate_analysis so we would enter in to the text box dalanders analyses my_Validate_analysis Then click Move Files in order to begin iterating over
32. that list and sending the qassoc files to the folder you created This process can take about as long or longer than it did to actually view the files Keep in mind that this process uses the iPlant Foundation API system so for each item in the right most list a request is send to the API to move that file to the folder you specified It may also be worth noting that by the time you read this manual this could be a fully integrated step in the iPlant Discovery Environment as there was talk about this functionality being available given more time However we found it necessary even though this part of the problem is almost entirely logistical to be able to select files from multiple files and move them in to a single aggregate folder Validate A Workflow for Evaluating GWAS QTL Tools 15 e098 Aggregate Aggregate An Application for iPlant Collaborative Quit _ Delete Folders After Moving Files Username dalanders View Contents of Selected Folders Password nnani Select File Type qassoc View Folders in Directory dalanders analyses Move Files ses my_Validate_analysis Select Folders to View Select Folders and or Files to Move Log dalanders analyses PheHasStruct_019_Trait_H_06_GenotypeDaj001_Trait_H2_03_GenotypeData_NoMissing_ assoc qassoc dalanders analyses PheHasStruct_019_Trait_H2_04_GenotypeDt_002_Trait_H2_03_GenotypeData_8pctMissing_ assoc qasso dalanders analyses PheHasStruct_020_Trait_H2_03_GenotypeQ 00
33. the idea of Validate which is a tool that returns these sorts of measures of tool success for massive amounts of outputs Validate A Workflow for Evaluating GWAS QTL Tools 3 But how do we run all these simulations How do we store them all in a single location so we can calculate these measures How do we visualize them These are the problems we sought to solve For each problem we have our own solution We also left them divided This way you can decide whether to use our solution or to use your own if you have a unique scenario 1 3 Getting iPlant Credentials 1 4 Getting Your Application on an API 1 5 Where Can I Get These Softwares 1 6 How to Use This Manual Evaluating a GWAS QTL tool with Validate is essentially four major steps after your application is installed on either the iPlant Foundation API or the Agave API Section 2 1 Running the Tool With Simulations As Inputs This step involves deciding what simulations to use and then iterating over those simulations and submitting them as job requests through the API We recommend using some sort of scripting method that you are comfortable with At this point we don t have a standalone application for submitting jobs We use rPlant which is freely available R package that allows you to connect with iPlant s API layer to submit job requests Section 3 2 Aggregating the Outputs into a Single Folder This part is a bit of a logistical exercise You need to put
34. uct_090_Trait_H2_03_GenotypeDat_006_Trait_H2_04_GenotypeData_3pctMissing_ assoc qassoqView success dalanders analyses PheNPStruct_091_Trait_H_06_GenotypeDat 006_Trait_H2_04_GenotypeData_NoMissing_ assoc qassoc iew success dalanders analyses PheNPStruct_091_Trait_H2_03_GenotypeDg07_Trait_H_06_GenotypeData_NoMissing_ assoc qassoc iew success dalanders analyses PheNPStruct_091_Trait_H2_04_GenotypeDat_007_Trait_H2_03_GenotypeData_3pctMissing_ assoc qassoqView success dalanders analyses PheNPStruct_092_Trait_H_06_GenotypeDa 007_Trait_H2_03_GenotypeData_8pctMissing_ assoc qassoqView success dalanders analyses PheNPStruct_092_Trait_H2_04_GenotypeDat_008_Trait_H2_03_GenotypeData_3pctMissing_ assoc qassoqdView success dalanders analyses PheNPStruct_094_Trait_H_06_GenotypeDa t 009_Trait_H2_03_GenotypeData_8pctMissing_ assoc qassoqView success dalanders analyses PheNPStruct_094_Trait_H2_03_GenotypeDqg009_Trait_H2_03_GenotypeData_NoMissing_ assoc qassoc jiew success dalanders analyses PheNPStruct_094_Trait_H2_04_GenotypeDat_010_Trait_H2_03_GenotypeData_8pctMissing_ assoc qassoqView success dalanders analyses PheNPStruct_096_Trait_H_06_GenotypeDat t_011_Trait_H2_04_GenotypeData_3pctMissing_ assoc qassoqView success dalanders analyses PheNPStruct_096_Trait_H_06_GenotypeDa _012_Trait_H2_04_GenotypeData_3pctMissing_ assoc qassoqdView success dalanders analyses PheNPStruct_096_Trait_H2_04_GenotypeDa13_Trai
35. ypeDadalanders analyses PheHasStruct_001_Trait_H2_03_CGenotypeDView success dalanders analyses PheNPStruct_091_Trait_H_06_GenotypeDatidalanders analyses PheHasStruct_001_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_091_Trait_H2_03_GenotypeDadalanders analyses PheHasStruct_001_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_091_Trait_H2_04_GenotypeDadalanders analyses PheHasStruct_002_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_092_Trait_H_06_GenotypeDatidalanders analyses PheHasStruct_002_Trait_H2_03_CGenotypeDView success dalanders analyses PheNPStruct_092_Trait_H2_04_GenotypeDadalanders analyses PheHasStruct_002_Trait_H2_03_CenotypeDView success dalanders analyses PheNPStruct_094_Trait_H_06_GenotypeDatidalanders analyses PheHasStruct_002_Trait_H2_03_CGenotypeDView success dalanders analyses PheNPStruct_094_Trait_H2_03_GenotypeDadalanders analyses PheHasStruct_002_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_094_Trait_H2_04_GenotypeDadalanders analyses PheHasStruct_002_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_096_Trait_H_06_GenotypeDatjdalanders analyses PheHasStruct_002_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_096_Trait_H_06_GenotypeDatidalanders analyses PheHasStruct_002_Trait_H2_03_GenotypeDView success dalanders analyses PheNPStruct_096_Trait_H2_04_GenotypeDadalanders analyses PheHasStruct_002_Trait_H2_03_GenotypeDView success dalander
Download Pdf Manuals
Related Search