ESO Phase 3 User Guide to the Data Submission Process
Contents
1. ccccecceseeeaeeeeeeeeaes T 43 2 3G0 IHVESUGALOR sicir nea ee ee Ee rSn 7 4 3 3 Principal investigator of normal ESO programme 202220022s0nennennnennnennnnnnnnenenennnnn nennen 8 4 3 4 Data products resulting from archival research 202220022000220snnnennnennnn anne nenennnnn nennen 9 4 4 Delegating the Phase 3 process u2zuusuuesnnnnnnnnnnnnonnnnnnnnnnonnnnnnnnnnn anne nun nenn anne nnnnnnnnnnnnnne nennen 9 4 5 Defining the ESO data release u024u0222022n0onnnenennnennennne nenn nennennne nenn nenne nnnenennnnnnennnenennn nee 10 4 5 1 Data collection and data release uu0220220002s0nnno non nenn anno nnnnnnnnnennnnnnnnnnn nennen nnne nennen 10 19 2 CENO e ee esse re A EE ct escredectossetsecteet 11 45 3 ni aldata relase erinnere ee ee une 12 4 5 4 Subsequent data release 20022022002susennnnnennnennnsnnnnnnennnnnnnnennnennnennn nennen ennnennenen 13 4 6 Adding data products to the release u2zuu22002s0nenenenennnnennnnnennnnnnnnenennnennnnenn nennen ennnenenenn 14 4 6 1 Validating the data using the Phase 3 Release Validator u 002002002s0senenne 14 292 FIP O2 UPIOA nee ee anne nee rinnen 14 4 7 Submitting the data release description u2220224002400Ben0nennenenenennnennenenenennnennenenenennnenn 15 4 8 Closing the data release 2u022402220022nenennnennonnne nenn nennennne nenn n2. 3 BATCH SUPERSEDING Show ERROR Phase 3 Release Manager 1 03 ESO Data Flow Infrastructure Send us your comments Subscribe to Newsletter Privacy Statement k Figure 10 Failed data release validation results in ERROR status 4 8 Closing the data release The PI concludes the data submission by closing the data release an action that cannot be delegated but has to be taken by the principal investigator in person Normally thereafter the release and its content cannot be changed anymore By closing the release the principal investigator approves the completeness and correctness of the submitted data and its description Furthermore the principal investigator agrees to the publication of the data being currently stored in the release directory of the staging area and to the publication of the associated data release description ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 1 7 of 21 Before of closing a data release its completeness should be carefully checked because the automatic validation procedures cannot detect if data is missing The PI can close the data release with the Phase 3 Release Manager by clicking the CLOSE button on data release page Figure 9 Closing the data release turns the data on the FTP area into read only state i e you cannot modify existing files or ad
3. Collection Name SGC em Figure 5 Creating a new collection ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 12 of 21 European a ES SE ee E N S Phase 3 Release Manager 0 Observatory Si ee ESO Reaching New Heights in Astronomy I Se ESOHome UserPortal Contact SiteMap Search A Go Science User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout ESO User Portal Programmes VHS DES Phase 3 Release Mgr Releases Properties DES Collection DES created Tag Type Mod Type Comment URL Validation Report Status Action Create new pe release Phase 3 Release Manager 1 03 ESO Data Flow Infrastructure Send us your comments Subscribe to Newsletter Privacy Statement h Figure 6 Newly created data collection without releases P 4 5 3 Initial data release If you are submitting data products for your programme for the first time the list of data collections is empty and the name of the data release under which the data submission takes place must be defined in the Phase 3 Release Manager by clicking the menu item Add collection see Figure 4 This system prompts you for the name of the data collection Figure 5 Note that the collection name should not contain any number indicating
4. Contents tab on the release information screen allows displaying the list of uploaded files on the FTP server as long as the release is in the open state Once the data has been archived this tab displays the contents organized by data sets Figure 15 ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany
5. cmoins eso org Revoked 22 02 2011 08 35 UT Go to the Phase 3 Release Manager ACE 1_0 ESO Data Flow Infrastructure Send us your comments Subscribe to Newsletter Privacy Statement r Figure 3 Page for managing permissions for Phase 3 delegation European FIES sounem a ne Phase 3 Release Manager O Observatory BF od ESO Reaching New Heights in Astronomy Se Zu LL DE ESO Home User Portal Contact SICH NET Search i Go Science User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout ESO User Portal Programmes VHS Phase 3 Release Mgr Programmes Collections Properties VHS Collection Name URL DES NGC SGC Add collection E Phase 3 Release Manager 1 03 ESO Data Flow Infrastructure Send us your comments Subscribe to Newsletter Privacy Statement Figure 4 List of data collections resulting from the selected programme 4 5 Defining the ESO data release After programme selection the Release Manager takes you to the page displaying the list of data collections for this programme Figure 4 In the last preparatory step before you can actually start to submit data products the ESO data collection and data release have to be defined 4 5 1 Data collection and data release Data collection and data release provide the framework that supports the data submission process and facilitates data access through the ESO archive The data
6. Reaching New Heights in Astronomy En i m ESO Home User Portal Contact Site Map Search esas Go Science User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout ESO User Portal Programs test collection 2 Phase 3 Release Mgr Release Tog 2 Contents Release 2 created You can upload your files here phase3ftp eso org under the directory test1 collection2 batch_2 Please download the previous release content description file using the link below and upload it with your release files Release Tag gt Release Type BATCH Modification Type UPDATING Download previous release content description file Status OPEN Comment Add any information regarding this data release Maximum characters 255 You have 255 characters left URL Contact Information cc List Close Date Archive Date sens Phase 3 Release Manager 1 03 ESO Data Plow infrastructure Send us your comments Subscribe to Newsletter Privacy Statement Figure 12 After having created an updating data release the content description file named CONTENT ESO must be downloaded to enable local validation of the data release European ES Southern zp i a si ES l Phase 3 Release Manager O Observatory y 5 ESO Reaching New Heights in Astronomy mH is I ESO Home User Portal Contact Site Map Search OO Go Science User Information gt ESO User Porta
7. collection allows organising the data products from a given programme according to the principal investigator s high level criteria into self consistent groups which subsequently the archive user can browse and access The data release is primarily a container for the data products to be submitted and released together ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 11 of 21 The concepts of data collection and data release are closely related to each other The two form a simple hierarchical structure where data collection is at the top and any data release must be associated to one and only one data collection The other way around any data collection must have at least one data release to be functional The data collection consists of a series of data releases tagged by incremental numbers starting from one All the data releases belonging to a given data collection inherit the data collection s name The individual data release may be considered a version of the data collection Initially any collection consists of the first data release Data Release 1 Subsequent data releases for the same collection may follow to add more data according to the observing programme s progress It is also possible to submit a revised version of a previously released data product 4 5 2 Guidelines Each d
8. links to the archived data products ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 16 0f21 European Ne Be Phase 3 Release Manager Q Observatory AT a at ESO Reaching New Heights in Astronomy J 3 m Zu I LOE E ESO Home User Portal Contact Site Map Search Go Science User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout ESO User Portal Programmes test1 collection2 Phase 3 Release Mgr i Releases Properties collection2 Tag Type Mod Type Comment URL Validation Report Status Action 1 BATCH NEW 2 BATCH UPDATING ARCHIVED 3 BATCH SUPERSEDING one E Phase 3 Release Manager 1 03 ESO Data Flow Infrastructure Send us your comments Subscribe to Newsletter Privacy Statement Figure 9 Closing a data release European a PESE soutnem en Fe Phase 3 Release Manager Observatory ESO Reaching New Heights in Astronomy 3m Hm I LL E ESO Home User Portal Contact EICH NET Search DZ Go Science User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout el ige Programmes test1 collection2 Phase 3 Release Mgr Tag Type Mod Type Comment URL Validation Report Status Action 1 BATCH NEW ARCHIVED 2 BATCH UPDATING goscesosossesesg
9. release from the Release Manager to your local disk and place it in the base directory that holds the data products of this release Figure 12 3 Create the special file CHANGES USER to indicate the deletion or replacement of products if needed 4 Validate the data release using the Phase 3 Release Validator with the command line option m UPDATE 5 Transfer the data including the CHANGES USER file to the phase3ftp server provided that the release passed the validation 6 CLOSE the release using the Release Manager Figure 13 CHANGES USER is a plain text format file written by the user It contains the instructions to remove or replace files of the previous release using one line per file following the scheme DELETE REPLACE lt filename gt 3 Note that replacing and removing of files applies to the current and possible future data release for this collection The replaced and removed files remain in the ESO archive and can be retrieved on demand by explicitly specifying the number of the previous data release ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 20of 21 Example of the CHANGES USER file REPLACE v20091023 00353 st tl fits REPLACE v20091023 00365 st tl fits REPLACE v20091023 00377 st tl fits REPLACE v20091023 00395 st tl fits DELETE v20091023 00407 st tl fits 4 11 2 2 To
10. this screen the user can commence the next release either superseding or updating the previous one Send us your comments Subscribe to Newsletter Privacy Statement 4 6 Adding data products to the release 4 6 1 Validating the data using the Phase 3 Release Validator The local validation of the data release is an essential preparatory step before uploading the data to ESO In this process the internal consistency and the metadata content of the data products will be checked against the ESO EDP standard For this purpose ESO provides a dedicated Java tool the Phase 3 Release Validator The validator is run on the local host from the command line using the calling sequence java jar validator jar r lt dir gt m create update The executable code including a comprehensive user manual is available online at http www eso org sci observing phase3 validator 4 6 2 FTP data upload Having defined and successfully validated the data release you are ready to transmit your data products to ESO for which a dedicated FTP server has been installed You may use any FTP client of your choice that supports secure authentication TLS see below for a list of compatible clients You may consider the powerful and free FileZilla http filezilla project org as FTP client which provides a graphical user interface and supports abort resume functionality among other useful features or the command line client ftp http Iftp yar ru which come
11. 3ftp server provided that the release passed the validation 5 CLOSE the release using the Release Manager Figure 13 As a result the data collection contains the data products of the previous release plus the new products of this release submission The incremental fashion of building up a data collection is suitable for large observing programmes especially if a large number of targets or a large sky area is aimed for 4 11 2 Revising the data collection 4 11 2 1 Updating and complementing The general procedure described below allows adding replacing and removing of individual data products There are two primary reasons for replacing data products i For large observing programmes that build up depth with time e g the UltraVISTA survey the latest data product resulting from the combination of the whole set of observations that have been acquired so far supersedes the previous product of this type ii A data product may be submitted a second time because its quality has been improved due to advanced data calibration or upgraded data reduction procedures Removing a previously released data product from a data collection is a rare event that happens only if a substantial quality problem has been identified for which there is no fix 1 Create the data release in the Release Manager by clicking Update or complement a previous batch release Figure 8 and Figure 11 2 Download the file named CONTENT ESO of the preceding
12. European Organisation Organisation Europ enne Europ ische Organisation for Astronomical pour des Recherches f r astronomische Research in the Astronomiques Forschung in der Southern Hemisphere dans I Hemisphere Austral sudlichen Hemisphare Data Management and Operations Division ESO Phase 3 User Guide to the Data Submission Process Doc No GEN SPE ESO 33000 xxxx Issue 2 Date 09 03 2011 Prepared J Retzlaff Name Date Signature Approved M Arnaboldi Name Date Signature Released Name Date Signature ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 2of21 CHANGE RECORD ISSUE DATE SECTION PARA REASON INITIATION AFFECTED DOCUMENTS REMARKS 27 11 2010 Initial issue of the Phase 3 User Guide to support the ESO Public Surveys Phase 3 Workshop ESO Garching 30 11 2010 07 03 2011 3 4 11 4 12 Added All Figures updated ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 30f21 TABLE OF CONTENTS 1 Purpose and SCOPE ccceccceccseccecessceceeceeecceeeceeeee ees eeseeeeuecseeeaeeseeeseeseeeseeeauesseeteesueeseeteeeteeesneeneetes 4 2 List of Applicable Documenis 0222002002suesnesnnennnennnnnnnnnnennnnnnnnnnenn
13. I and the contacts given in the CC list will be contacted by email to discuss the issue and to agree on corrective actions Depending on the nature of the problem the release state may have to be reverted to OPEN to allow the PI to modify the submitted data 4 10 Data release becomes public The data release will be archive and becomes public after the validation process has been concluded Figure 10 Each data release is announced using the appropriate ESO channels e g through the ESO science news and the ESO archive news feed The PI and the contacts given in the CC list will be notified about the actual date of public release Phase 3 Release Manager ESO Reaching New Heights in Astronomy I Science User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout ESO User Portal Create Release Create a new release for Program test1 Collection collection2 Release Type BATCH Modification Type UPDATING Comment Add any information regarding this data release here You ha characters left Contact Information cc List e mail is Figure 11 Creating an updating data release Send us your comments Subscribe to Newsletter Privacy Statement d ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 18 of 21 Phase 3 Release Manager ESO
14. ase 4 Note that the procedure for updating the collection 4 11 2 1 using the CHANGE USER file with REPLACE DELETE actions for every single file of the previous release would achieve exactly the same result ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 21 of 21 European Southern 2 ee Phase 3 Release Manager Observatory ESO Reaching New Heights in Astronomy _ EI p ESO Home User Portal Contact Site Map Search Go Science User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout ESO User Portal Programs test1 collection2 2 Phase 3 Release Mgr Download release content description file n253_tl_j fits 2 File Category Size Archive Id n253_tl_j fits science image 803 390 400 ADP 2011 02 25T11 37 02 297 n253_tl_j_conf fits ancillary weightmap 401 705 280 ADP 2011 02 25T11 37 02 313 n253_tl_z fits 2 Show Hide files File Category Size Archive Id n253_tl_z_conf fits ancillary weightmap 401 336 640 ADP 2011 02 24T20 49 29 317 n253_tl_z fits science image 802 653 120 ADP 2011 02 24T20 49 29 327 Phase 3 Release Manager 1 03 ESO Data Flow Infrastructure Send us your comments Subscribe to Newsletter Privacy Statement k Figure 15 Displaying the contents of an archived release 4 12 Reviewing the release content The
15. ata collection is defined by its name and by the set of data products being members of the associated data releases The collection name has to be defined at the time of starting the data submission and cannot be revised later by the user Each data release has to be accompanied by a description of its content data properties and format to enable the archive user to employ the data for their own scientific research the release description see below For ESO large programmes it is common practice to organize the data products according to the scientific publications that have been generated thereof One data collection may correspond to one target object Otherwise for programmes that consist of independent observations with multiple instruments data collections might be defined by instrument The type or number of data products are not useful criteria for the definition of data collections In practice one ESO programme may result in a single data collection like e g the ZCOSMOS galaxy redshift survey European gt TO le Phase 3 Release Manager O Observatory ar S BF a ESO Reaching New Heights in Astronomy _ 1 1 1 Sk im ESO Home User Portal Contact Site Map Search FE Go Science User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout ESO User Portal Programmes VHS Phase 3 Release Mgr Create Collection Create a new collection associated to the VHS programme
16. brings you to the list of data collections of this programme as shown in Figure 4 The final action of closing a data release cannot be delegated but the principal investigator in person has to take this action to approve the completeness and quality of the data and the accuracy of the associated description Closing a data release represents a crucial step in the course of the Phase 3 process because the operations following thereafter and leading to the public release of the data through the ESO Archive are being executed by the system in a largely automatic fashion ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 10of 21 European PESH sounem a a ACE Access Control ESO Q Observatory 5 mY ESO Reaching New Heights in Astronomy Eii p 1 Se ESO Home User Portal Contact Site Map Search Ps Go Science User Information gt ESO User Portal gt Access Control J Retzlaff Logout ESO User Portal Grant Revoke Phase 3 Rights for Programmes Access Control Admin Data Access Program Name Comment Delegated Revoked Users Delegate to Magda Arnaboldi rsliikhu eso org mamabol eso org 22 02 2011 08 15 UT Joerg Retzlaff jretziaf eso org 22 02 2011 08 38 UT View Data Access Phase 3 Access test1 Science Archive Archive Requests FAQ christophe moins
17. ci observing phase3 rm a dedicated web application for controlling the Phase 3 process to define a new data release As the principal investigator of an ESO programme you organise your data products according to appropriate high level criteria into self consistent groups called data collections which subsequently the archive user can browse and access Later on you may issue further releases to add more data or to update already released data products More about the concepts of data collection and release see 84 5 3 2 Delegate Phase 3 permissions As the principal investigator you can delegate the Phase 3 process for your ESO programme to one or more persons of your choice to distribute the effort of data submission and release preparation How to set up Phase 3 delegation see 4 4 3 3 Prepare the data The data should be prepared according to the ESO External Data Products Standard 1 which specifies data product types and their formats Download the ESO External Data Products standard from http www eso org sci observing phase3 p3edpstd pdf 3 4 Validate the data You validate the data before uploading to ESO using the Phase 3 Release Validator a java based tool to be run locally on your system Repeat the validation until you obtain as result the OK for data upload Download the software and its documentation from http www eso org sci observing phase3 validator 3 5 Upload the data to ESO You upload the data via se
18. cure FTP to the dedicated server phase3ftp eso org For this purpose a Suitable client like ftp or FileZilla has to be used Find more information about uploading your data in 4 6 3 6 Create and upload the release description You create and upload the data release description which provides an account of the release content the originating observations the calibration and data reduction procedures the data quality the data format and possibly the scientific context of the programme More about the preparation and submission of the data release description see 84 7 3 7 Closing the release By closing the release you approve the completeness and correctness of the submitted data and its description Furthermore you agree to the publication of the data and the associated data release description You close the data release using the Phase 3 Release Manager Closing the data release turns the data on the FTP area into read only state i e you cannot modify existing files or add new data anymore More about closing the data release see 4 8 ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 6 0f21 3 8 ESO validates the submitted data This is a two step process consisting of an automatic validation of the data format followed by a manual validation of the content and its description If the va
19. d new data anymore The operations following thereafter and leading to the public release of the data through the ESO Archive are being executed by the system in a largely automatic fashion without further interacting with the user assuming that the submitted data pass the validation process After successful validation ESO automatically archives and publishes the data release 4 9 Data release validation The validation process carried out at ESO consists of two stages First there is a fully automatic procedure checking the internal consistency and metadata content of the submitted data products which is basically identical to the user side validation If this validation fails the release state will change from to IE you will receive a notification by e mail which prompts you to correct the problems and to the release again Figure 10 You can avoid this type of error by validating all products on your side before of submission If the release hasn t passed the automatic validation you can click the respective link to Show the Validation Report The second level of validation is performed by the ESO Phase 3 support scientist and consists of reviewing the release content and description at large in order to spot possible issues The problems typically found in the course of this process lie in the following areas adequacy and consistency of the data release description data quality issues and metadata content In case of a problem the P
20. e owner and for your delegates In the course of preparing the data release you can modify files on the FTP server as needed including file removal As a data provider you also control the data flow in time You may transfer the files for a data release in one go or you may accumulate them on the FTP server over a period of time before you close the release This flexibility allows adapting the submission process to the particular needs due to data processing and verification activities of each programme If no more data is being added the data will be kept on the FTP server for a period of three months unless the release is closed before 4 7 Submitting the data release description The principal investigator is responsible for the content of the data release description and for its delivery to ESO along with the data submission The release description consists of formatted text typically supplemented by images and tabular information as needed The following file formats are accepted plain text ASCII HTML Microsoft Word doc for images JPEG TIFF GIF and PNG The files belonging to one release description should be bundled into a single file using the tar utility The release description with the filename release description tar has to be placed into the release directory of the FTP area ESO EDP is in charge of editing and formatting the release description and of publishing the information on the ESO web site including appropriate
21. enne nano nenn nenne nnnenennnnnnennnenennnenne 16 4 9 Data release Validation u220020002000200 anno ann nenn anno anne nun nenn anne nun nenn nenn anne nun nenn nme nun nnnnnnnennne 17 4 10 Data release becomes public 22022002002002n0nnnnnnennennnnnnnnnennennnnnnnnnennennnnnnnnnennennnnennene 17 4 11 Updating data Colleclions s sisien ntladeenils nennen nun nen hehe 18 4 11 1 Adding new data products ccccccccccccecceeeeeeeceeeeeeeeeeseeeseeeeeeeeeeseeeseeeseesaeeseeeseeeeeseeeaaes 19 4 11 2 Revising the data collection us42002400ennenennn nenn nnnenennnnnnne nenn nnnnnnn nennen nnnn nenn 19 4 11 2 1 Updating and complementing u0 s0224004000n0 nenn nen nnn nee nenennnnnnnnenenennnnennnnen 19 4 11 2 2 Totally replacing the previous content u0 402240Bennnn nen nenennnnnnn nennen nnnennenen 20 4 12 Reviewing the release content u04400000ennnene nen nnn nenn nennen esse eesseeeeeseeeseeeaeeeaeeseeesenes 21 ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 4of21 1 Purpose and Scope Phase 3 denotes the process in which principal investigators of ESO observing programmes return their reduced data products to ESO for storage in the ESO archive and subsequent data publication to the scient
22. from the list please contact usd help eso org Subject Phase 3 via email 4 3 2 Co Investigator If you act as a Phase 3 delegate on behalf of the principal investigator of an ESO public survey or large programme typically being Co Investigator of the programme you can pick the programme to submit data for from the list of delegated programmes by clicking on the respective name If the programme is missing from the list please contact the respective principal investigator to request Phase 3 delegation for this programme ESO user support is not able to effect Phase 3 delegation because authorization by the programme s principal investigator is strictly required To setup Phase 3 delegation as principal investigator see 4 4 page 9 ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 8of21 40 European a ESO User Portal ES Southem 0 Observatory t ESO Home User Portal Contact EICH NET Search ESO User Portal Welcome to the ESO User Portal Science Users For security reasons please Log Out and Exit your web browser when you have finished accessing services that require authentication General Information and FAQ Public Enter your ESO Username and Password Intranet As of 25 August 2009 ESO User Portal usernames and passwords are both case sensitive Troubles lo
23. gging in Try using the I forgot my links below Username JRETZLAFF Password eeeeeee osin cuen e forgot my ESO User Portal password e forgot my ESO User Portal username e would like to create a new account Last Update 02 07 07 GESO Send us your comments Subscribe to Newsletter Privacy Statement Figure 1 The ESO User Portal login page European PESE soutem a ei Phase 3 Release Manager O Observatory x Rie s AY ee M ESO Reaching New Heights in Astronomy sk Zu LEB ESOHome UserPortal Contact SiteMap Search co Science User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout ESO User Portal Phase 3 Release Mgr INWIEFERN ce Phase 3 Programmes Delegated to me Programme name Owner Comment test J Retzlaff VHS J Retzlaff rk Grant Revoke Phase 3 Permissions for your Programmes E Phase 3 Release Manager 1 03 ESO Data Flow Infrastructure Send us your comments Subscribe to Newsletter Privacy Statement Figure 2 Entry page of the Release Manager 4 3 3 Principal investigator of normal ESO programme If you are the principal investigators of a normal ESO programme i e neither ESO public survey nor large programme and you intend to submit data resulting from this programme for the first time you need to request the registration of your progamme for Phase 3 submission by sending an email message to usd he
24. hed data first choose the collection by clicking the collection name then choose the menu item Create superseding batch release Figure 8 The system automatically increments and assigns the data release number to the newly created data release Alternatively if the new data does not logically belong to the existing data collection you should create a new data collection by clicking the menu item Add collection in the Phase 3 Release Manager s data collection page Figure 4 Please take into account that the name of the data collection must be unique within the programme and that the collection name does not contain any number indicating the version or alike ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 14 of 21 y European ar ES ae me re gata Br I Phase 3 Release Manager O Observatory ESO Reaching New Heights in Astronomy bE EG ESOHome User Portal Contact SiteMap Search NN co e User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout ESO User Portal Programs test1 collection2 Tag Type Mod Type Comment URL Validation Report Status Action 1 BATCH NEW ARCHIVED Create superseding batch release Update or complement previous batch a Figure 8 The initial data release of this collection has been archived From
25. ific community To ensure the successful integration of External Data Products EDPs into the archive ESO supports the users in carrying out the Phase 3 process by defining ESO EDP data standards by devising procedures and providing the infrastructure for the delivery of EDPs and by supplying tools for the data preparation This document provides a user guide to the ESO Phase 3 data submission process It is intended to carry the user through each step of the submission process once the data has been prepared according to the ESO External Data Products Standard defined in 1 see below The target audience consists of principal investigators and their collaborators who return reduced data products resulting from ESO observations for public release to the astronomical community through the ESO archive 2 List of Applicable Documents 1 ESO External Data Products Standard Issue 2 Date 09 03 2011 http www eso org sci observing phase3 p3edpstd pdf and documents referenced therein 1 See http www eso org sci observing phase3 for the main entry point to the ESO Phase 3 ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 5of21 3 Overview of the Phase 3 Process The Phase 3 process consists of the following steps 3 1 Define new data release Go to the Phase 3 Release Manager http www eso org s
26. l gt Phase 3 Release Manager J Retzlaff Logout ESC Unori Eoria Programs test1 collection2 Phase 3 Release Mgr Tag Type Mod Type Comment URL Validation Report Status Action 1 Batch NEW er 2 BATCH UPDATING e Phase 3 Release Manager 1 03 ESO Data Flow Infrastructure Send us your comments Subscribe to Newsletter Privacy Statement Figure 13 Closing an updating data release 4 11 Updating data collections This section ties in with the data release definition in case you already have a previous data release for your programme as outlined in 4 5 4 ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 19 of 21 4 11 1 Adding new data products lf you want to extend the existing collection by adding new data products without modifying the previous content please go through the following steps 1 Create the data release in the Release Manager by clicking Update or complement a previous batch release Figure 8 and Figure 11 2 Download the file named CONTENT ESO of the preceding release from the Release Manager to your local disk and place it in the base directory that holds the data products of this release Figure 12 3 Validate the data release using the Phase 3 Release Validator with the command line option m UPDATE 4 Transfer the data to the phase
27. lidation fails the release state will change from CLOSED to ERROR and you will receive a notification by e mail which prompts you to correct the problems and to CLOSE the release again 3 9 ESO archives and publishes the data The data release will be archive and becomes public after the validation process has been concluded The community will be able to access the data via specifc query interfaces in the ESO archive web pages You will be notified about the actual date of public release ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page of21 4 Step by step user quide to the Phase 3 data submission process 4 1 Enter the Phase 3 Release Manager Navigate your web browser to http www eso org sci observing phase3 rm to access the ESO Phase 3 Release Manager a web application to manage the process of data submission and release through the ESO science archive facility The Release Manager allows ESO users to manage their ESO Phase 3 programmes to create new data collections and releases to manage the work flow of data releases and to view the contents of archived data releases 4 2 Log in to the ESO User Portal Unless you already have logged in to the ESO User Portal with your web browser you will be asked by the system to identify yourself with your personal ESO User Portal credentials Figu
28. lp eso org subject REQUEST FOR PHASE 3 PROGRAMME lt PPP C NNNN gt where lt PPP C NNNN gt denotes your ESO programme identifier The body of the message should be empty Thereafter usually within one working day you will receive by email the confirmation that the ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 90f21 programme has been registered and you can proceed by picking the respective programme from your list of programmes being displayed by the Phase 3 Release Manager 4 3 4 Data products resulting from archival research If you have carried out research being primarily based on ESO archival data rather than on ESO data that has been obtained in the course of your own ESO observing programme you are encouraged to submit the resulting data products to ESO for public release through the ESO science archive facility To this end you need to request the registration of a dedicated Phase 3 programme by sending an email message to usd help eso org Subject Phase 3 indicating the name of your research project and the type of data products to be submitted Thereafter usually within one working day you will receive by email the confirmation that the programme has been registered and you can proceed by picking the respective programme from your list of programmes being displayed by the Phase 3 Release Ma
29. nager 4 4 Delegating the Phase 3 process As the principal investigator of an ESO programme you can delegate the Phase 3 process to one or more persons of your choice to distribute the effort of data submission and release preparation The principal investigator can delegate the Phase 3 per programme to any ESO user i e a person being registered with the ESO User Portal A delegate can create new collections and data releases and can submit data products in the same way as the principal investigator can do The principal investigator can revoke at any time any delegation rights he might have granted Multiple delegates can in principle work on the same data release but it is in the sole responsibility of the principal investigator to ensure the overall consistency From the top level page of the Phase 3 Release Manager showing the overview of programmes choose the link Manage programme permissions below the table of your programmes near to the bottom of the screen to view and administrate delegation settings Figure 2 This brings you to the page for managing programme permission from where the following actions can be taken Figure 3 e Delegate a programme to another user registered in the ESO user portal by entering his her e mail address and clicking delegatel e Revoke the permissions from an existing delegated programme by clicking on lrevoke To return to the Release Manager click on the name of one of your programmes which
30. nennnnnnnnnnennnennn nenn nnnennnnnneen 4 3 Overview of the Phase 3 Process ccccccccscccseecenceceeeceueceeeceeeceueceeeeseeessueeseeseeessusesaeeseeessnsegeeess 5 3 1 Define new datarelease uu u nnnn a a aaa 5 3 2 Delegate Phase 3 permissions z 0224002402000000 nenn anno ann nenn anne nnno nun nenn anne nnnnnnnnnnennnennnnnnnnnennne 5 3293 Piepare DE Old ee O en ee 5 84 Validate the dala seen ee ee ee ee enge ee ee 5 39 Upload Nedala To ESO ee ne nen ee ee 5 3 6 Create and upload the release description u2200220022002002n0nnnnnnennnennnennnnnnn nenne nennen 5 1 Closing NETTE ee a ee ee ee ee 5 3 8 ESO validates the submitted data 2220222002000020000n0 nenne nen nenn nano nnne nenn nenne nenne nenne nnnenenenenne 6 3 9 ESO archives and publishes the data 2 u022002s0ssneesnennnennennnnnnnennnennnnnnennnennennnnnnenen 6 4 Step by step user guide to the Phase 3 data submission process uu2202ssussnensnennennnnnnnennnenneen 7 4 1 Enter the Phase 3 Release Manager u2 u0200020002000nnonnno non nenn anne nun nenn anne anne nnnnnnn anne anne T 4 2 Login tothe ESO User Portal eriraicsriscenreirnrerisinnerii nin in iien ninna einer 7 4 3 CHOOSING your Programme cneiccscisicscdencisecanendeas sendinsadigucesmecetacaswndesed sasnduiescewsetudenaedschsextieedeiveseeaeuecess 7 4 3 1 Principal investigator of ESO public survey or large programme
31. re 1 As a principal investigator of an ESO programme you are familiar with the ESO User Portal from the proposal submission and observing preparation procedures If you are not a registered user of the ESO User Portal first you will need to register yourself before proceeding To this end please click on the link named I would like to create a new account near the bottom of the ESO User Portal welcome page which carries you to the registration dialogue After having created a new User Portal account you can log in and proceed with the following step of the Phase 3 submission process 4 3 Choosing your programme The first administrative step of the data submission process is the identification of the ESO programme for which you are going to submit data products The entry point of the Phase 3 Release Manager application has two tabs listing the Phase 3 programmes that you own and the list of programmes delegated to you like shown in Figure 2 4 3 1 Principal investigator of ESO public survey or large programme If you are the principal investigator of an ESO public survey or large programme your programme is already registered and listed in the Phase 3 release manager If you own multiple programmes all of them are listed In order to choose the programme to submit data for just click on its name cf Fig Figure 2 which navigates you to the overview of data collections for this programme If by mistake your programme is missing
32. s with many Linux distributions by default First connect to the FTP server at phase3ftp eso org and login with your ESO User Portal credentials Then change to the directory 2 Find a list of compatible FTP clients at http www eso org sci observing phase3 ftp upload ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Issue 2 to the Data Submission Process Date 09 03 2011 Page 15 0f21 lt Phase 3 programme gt lt Data collection name gt batch_ lt Data release number gt For example for the VISTA Hemisphere Survey with Programme Name VHS Data Collection Name SGP Data Release 1 the complete path would read VHS SGP batch 1 Thereafter you can start transferring your set of products Since FITS checksums are employed to safeguard the data transfer against transmission errors it is essential that you properly update the respective checksum keywords beforehand see 1 83 6 8 For non FITS format ancillary files there is a special mechanism in place to catch possible transmission errors employing their MD5 hashes You have the option to specify the MD5 hash using the dedicated keyword ASSOMi in the science file defining the data set see 1 83 6 2 Then the Phase 3 validator detects any data inconsistency before the files will be archived The directory and the data deposited therein are visible and accessible just for you as the programm
33. sue 2 to the Data Submission Process Date 09 03 2011 Page 13 o0f 21 z ae aaa un E A TNN T Phase 3 Release Manager Observatory ESO Reaching New Heights in Astronomy u nn Li ESO Home User Portal Contact Site Map Search SSS Leto Science User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout D Last Update 03 ESO User Portal Phase 3 Release Mgr Create Release Programs Create a new release for Program VHS Collection SGC Release Type BATCH Modification Type NEW Comment Add any remarks for ESO EDP regarding this data release here Maximum characters 255 You have 192 characters left Contact Information cc List Add e mail eglez ast cam ac uk rgm ast cam ac uk sae 11 2010 ESO Send us your comments Subscribe to Newsletter Privacy Statement 4 Figure 7 Creating a new data release 4 5 4 Subsequent data release If you are submitting data products for your programme for which you have a previous data release there are the following options 1 If the new data extend or revise the existing data collection first choose the collection by clicking the collection name then choose the menu item Update or complement previous batch release Figure 8 The system automatically increments and assigns the data release number to the newly created data release See 4 11 for more details If the new data should replace previously publis
34. tally replacing the previous content This procedure serves the particular purpose of removing the previous content of the data collection altogether and substituting it with new content This operation should be executed with caution and only if new versions of all the previously released data products will be submitted again The steps for replacing the previous content of the data collection are as follows 1 Create the data release in the Release Manager by clicking Create superseding batch release Figure 8 and Figure 14 2 Validate the data release using the Phase 3 Release Validator with the command line option m CREATE 3 CLOSE the release using the Release Manager as usual European ES er Be st ON Phase 3 Release Manager O Observatory TEE 78 ESO Reaching New Heights in Astronomy 3 m ota I LE S ESO Home User Portal Contact SCA NET Search E Go Science User Information gt ESO User Portal gt Phase 3 Release Manager J Retzlaff Logout ESO User Portal Create Release Create a new release for Programme test1 Collection collection2 Release Type BATCH Modification Type SUPERSEDING Comment Maximum characters 255 You have 255 characters left Contact Information cc List Add e mail S Phase 3 Release Manager 1 03 ESO Data Flow Infrastructure Send us your comments Subscribe to Newsletter Privacy Statement hm A Figure 14 Creating a superseding data rele
35. the version or alike because the data releases will be automatically assigned an integer numbers incrementing with each release The collection name is subject to the following conditions it must start with a letter and should not contain any white space use the underscore character _ instead To proceed click on the name of the newly created data collection which brings you to the empty list of releases for this collection Figure 6 That followed the initial data release for this data collection must be created be clicking the menu item Create a new batch release The system automatically assigns the data release number 1 to the newly created initial data release Now you can enter a comment in the respective field and a list of email recipients to be notified about the progress of the Phase 3 process for this release Figure 7 The ESO EDP user support scientist while reviewing the content and validating the overall consistency of the data release will take the comment into account Because the comment itself will not be published it is primarily intended to pass information regarding the actual Phase 3 process rather than information about the data The later should always go into the release description which will be made publically available Eventually click the button to create the new data release ESO Karl Schwarzschild Str 2 85748 Garching bei M nchen Germany Doc GEN SPE ESO 33000 xxxx ESO Phase 3 User Guide Is
Download Pdf Manuals
Related Search