Home

ESO NG/AMS - User`s Manual

1. 20 EXPERT The Wake Up Plug In The Wake Up Plug In is used by an NG AMS Server that has been requested to wake up an NGAS Host that has suspended itself The Suspension Wake Up Service is described in Section 4 3 The actions to be carried out depends on the HW and on the system configuration Usually a Wake Up Plug In will send a message to a device connected to the network to indicate it to start up an NGAS Host this device could e g be the network card of the host 201 EXPERT Interface of a Wake Up Plug In The plug in must be contained in a Python module which has a function of the same name as the module The latter is the actual plug in which is invoked by NG AMS A Wake Up Plug In has an interface as shown in Figure 54 srvObj Instance of the ngamsServer class used by this session See also Section 12 1 hostId hostId Name of NGAS Host to wake up Wake Up Plug In Function Name The name of the plug in must be identical to that of the Python module hosting the plug in function Figure 54 Function interface of a Wake Up Plug In A Wake Up Plug In does not return any value to the NG AMS Server after execution 20 2 EXPERT Example Wake Up Plug In Under development Figure 55 Example Wake Up Plug In FILE VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 93 of 126 21 EXPERT The Filter Plug in The purpose of the Filter Plug In is to classify data when data is
2. As a result of the REMDISK command a report is send indicating which disk that was removed from the system 27 10 REMFILE Command Remove Files from the System The REMFILE command is used to remove information about files and the files themselves from NGAS Great caution should therefore be applied when using this command NG AMS will not accept to remove a file from the system unless there are at least three 3 independent copies of the file Three independent copies refers to three copies of the file stored on three independent storage media In order for the REMFILE command to be accepted the system must be configured to allow remove requests CFG NgamsCfg Ngams AllowRemoveReg If the command is executed without the execute parameter the information about the file s is not deleted but a report is generated indicating what will be deleted if the execution is requested execute 1 The selection of the files to be scheduled for deletion is done based on the parameters Disk ID File ID and File Version The rules for this are shown in Table 34 DiskID File Specified Unspecified Unspecified No files selected All files with the given File ID pattern on the disk with the given ID will be selected No specific File Version will be taken into account given ID is selected if this exists All files matching the given File ID pattern on the contacted NGAS Host are selected selected without taking the Disk ID into account
3. IngamsSubscriber py Table 24 Python modules in the ngamsLib sub module 23 2 EXPERT Online Browsing of NG AMS Inline Python Documentation It is possible to browse online the Python documentation contained in the NG AMS Python source code files This provides an acurate and comprehensive description of all classes methods and functions The following notation has been used to document the interfaces of methods and functions def notify ngamsCfgObj type subject msg Send a notification e mail to a subscriber about an event happening ngamsCfgObj Reference to object containing NG AMS Configuration file ngamsConfig type Type of Notification See NGAMS_NOTIF_ in ngams subject Subject of message string msg Message to send string Returns Void lt code gt Figure 61 Example of NG AMS inline documentation First in the description of a method function a small description of the task performed by the method is provided After that the input parameters are listed After the description of each parameter the type of the parameter is indicated in paranthesis The return value is also given in connection with the Returns tag VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 5 08 01 2003 101 of 126 The documentation can be browsed in an easy manner by using the documentation generator provided together with the Python package pydoc This can also be used as an HTTP server
4. InstallationDate Type LogicalName MainDisk HostId SlotId Mounted MountPoint Unique ID for the HDD ID of the archive to which this disk belongs Date ISO 8601 the disk was prepared Type of the HDD Logical human readable memorable name of the HDD Indicates if the HDD is the Main Disk or the Replication Disk 0 1 Name of the host where the HDD is installed Slot ID number in which the HDD is installed Indicates if the HDD is mounted in this NGAS System 011 VLT MAN ESO 19400 2739 2 08 01 2003 96 of 126 Indicates the mount point for the HDD path NumberOfFiles Number of files stored on the disk AvailableMb Indicates the capacity of the HDD MB BytesStored Bytes stored on the HDD Indicates if the disk is full no more data should be stored on this disk Completed Checksum Checksum for the data on the disk TotalDiskWriteTime Total time in seconds used for writing the bytes stored on this disk s LastCheck o lt ELEMENT DiskStatus FileStatus gt lt ATTLIST DiskStatus DiskId CDATA REQUIRED Archive CDATA REQUIRED InstallationDate CDATA REQUIRED Type CDATA REQUIRED LogicalName CDATA REQUIRED MainDisk 011 REQUIRED Hostid CDATA REQUIRED SlotId CDATA REOUIRED Mounted 011 REOUIRED MountPoint CDATA REQUIRED NumberOfFiles CDATA REOUIRED AvailableMb CDATA REQUIRED BytesStored CDATA REQUIRED Completed 011 REQUIRED Completi
5. 7 3 EXPERT Format of the NG AMS Redirection HTTP Response If an NG AMS Server is not configured to always act as a proxy when data is being requested by a client HTTP redirection response messages may be generated and send back to the requestor The format of such redirection responses is VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 50 of 126 HTTP 1 0 303 Method Serveri sgerver IDE Dates Keke Expires lt date gt Location lt URL pointing to actual location of file gt Content Type text xml Content Length lt length gt lt NG AMS status document gt Figure 27 Structure of NG AMS HTTP Redirection Response An example of such a redirection HTTP response is HTTP 1 0 303 Method Server v1 5 2002 02 12T10 52 10 Date Tue Jan 01 34 40 2 GMT Expires Tue Jan 01 34 40 2 GMT Location http jewe164 7777 RETRIEVE file id WFI 2001 09 25T21 19 17 508 Content Type text xml Content Length 339 lt xml version 1 0 gt lt NgamsStatus gt lt Status Date 2001 01 02T01 34 40 656 HostId jewel68 Message NGAMS INFO RETRIEVE REDIRECT 4024 INFO Redirection URL http jewel64 7777 RETRIEVE file_id WFI 2001 09 25T21 19 17 508 State ONLINE Status SUCCESS SubState BUSY Version v1 5 2002 02 12T10 52 10 gt lt NgamsStatus gt Figure 28 Example of NG AMS HTTP Redirection Response The client must then re issue the Retrieve Request to the alternative location given
6. A simple scheme has been implemented to avoid that the same data file is delivered several times to a client This scheme is based on recording the ingestion date for the last file delivered Le only files with a more recent ingestion date will be 4 http www apache org VLT MAN ESO 19400 2739 ESO Noms User s Manual Is 08 01 2003 33 of 126 taken into account This remembered last ingestion date for each client will be reset if a start date for the subscription older than this date is specified by a client A Data Subscriber unsubscribes itself by sending an UNSUBSCRIBE command The client remains subscribed as soon as it has sent the SUBSCRIBE command until an UNSUBSCRIBE is submitted When a client issues the UNSUBSCRIBE command the Subscription Back Log for that client will be reset and thus possible Back Log Buffered data will not be delivered Great care should be taken to avoid circular subscriptions i e that two clients subscribe to each other for the same type of data In such a case the two serves would continue to deliver the file to each other ending up saturating the system A Subscriber cannot subscribe to itself It is possible to switch off the Subscription Service globally via the configuration CFG NgamsCfg SubscriptionDef Enable The subscription service is handled by an internal thread Data Subscription Thread running within the NG AMS Server It is possible to specify
7. The following Python module is the implementation of a simple Filter Plug In which filters the file specified according to a requested mime type given in the Plug In Parameter string mime types can contain a list of I separated mime types FRR IRR ARR IRR AOR KK KK RR RR KEKE KK KK KK RK RR RK RK RK RR ARK ESO DFS Id ngamsMimeTypeFilterPI py v 1 2 2003 01 02 13 26 10 arcsw Exp jknudstr 21 11 2002 Created import exceptions from ngams import import ngamsPlugInApi def ngamsMimeTypeFilterPI srvObj filterPiPars filename VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 94 of 126 Eiere fileVersion 1 nu Example Filter Plug In used to filter on a given mime type In case the file referenced has the mime type as specified in the plug in parameters the file being tested is selected srvObj Reference to NG AMS Server Object ngamsServer filterPiPars Filter Plug In Parameters string fileld File ID for file to test string filename Filename of complete string fileVersion Version of file to test integer Returns 0 if the file does not match 1 if it matches the conditions integer 0 1 nu match 0 parDic ngamsPlugInApi parseRawPlugInPars filterPiPars if not parDic has key mime types errMsg ngamsMimeTypeFilterPI Missing Plug In Parameter mime_types raise exceptions Exception errMsg refMimeTyp
8. Unspecified Unspecified Specified No files are selected Unspecified Unspecified Unspecified No files are selected Table 34 Selection rules applied for the REMFILE command The REMFILE command takes the following input parameters TG disk_id lt disk ID gt See Table 34 Disk ID for the disk to remove execute 011 No If execute is not specified or specified as 0 no information will be deleted but a report will be send back to indicate what will be deleted if the command is executed If execute is specified as 1 the information in the DB and on the disk itself is deleted file_id lt file ID gt See Table 34 JID of files to take into account file_version lt file version gt See Table 34 Version of files to take into account Table 35 Parameters for the REMFILE command As a result of the REMFILE command a report is send back indicating which disk that were moved from the system or alternatively if the execute is 0 or unspecified a list of files that will be deleted if the command is executed is returned 2711 RETRIEVE Command Retrieve amp Process Files The RETRIEVE command is used to retrieve archived data files from an NGAS Node The RETRIEVE command accepts the following parameters file_id lt file ID gt ID of file to retrieve file_version lt file version gt Version of the file to retrieve internal lt filename gt No Retrieve the contents of an internal file Could e g be the a source file or
9. arcFile keyDic ARCFILE 0 els Seung pil eaten Aes TER dpId els 0 els 1 els 2 date string split els 1 T 0 Make sure that the files are stored according to JD one night is 12am gt 12am isoTime els 1 tsl PccUtTime TimeStamp isoTime ts2 PccUtTime TimeStamp tsl getMjd 0 5 dateDirName string split ts2 getTimeStamp T 0 return arcFile dpId dateDirName excepti err Did not find keyword ARCFILE in FITS file errMsg genLog NGAMS ER BAD FILE os path basename filename ngamsFitsPlugIn err raise exceptions Exception errMsg checkFitsFileSize filename wen Check if the size of the FITS file is a multiple of 2880 If this is not the case we through an exception filename FITS file to check string Returns Void won Seng pln ae rae KW Mes size ngamsPlugInApi getFileSize filename testVal size 2880 0 if testVal int testVal errMsg The size of the FITS file issued is not a multiple of 2880 Rejecting file errMsg genLog NGAMS ER BAD FILE os path basename filename ngamsFitsPlugIn errMsg raise exceptions Exception errMsg def checkChecksum parDic filename VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 81 of 126 Check that the checksum of the file is correct parDic Dictionary with disk information ngamsPhysDiskInfo objects dictionary filename Name of FITS file str
10. Log ID NGAMS ER START HTTP SERV Log Number 3012 Log Type ERROR Log Text Problems starting HTTP serving s SH The HTTP server could not be started This could because by conflicting port numbers or other Description i problems in connection with the network set up Log ID INGAMS_AL_NO_DISKS_AVAIL Log Number 3013 Log Type ALERT ALERT NO DISKS AVAILABLE IN THIS NGAS SYSTEM NGAS ID s Host ID s Check HW and try again Deo NG AMS didnt find any disks installed in this system Normally disks should be available for proper pay sete operation of the system Check the disk configuration and try again Log Text LlogID NGAMS AL NO STO SETS o O Log Number 3014 Log Type ALERT Log Text No Storage Sets found for mime type Gs Benoa No available Storage Set disk set was found for storing the data with the Mime type as referred escripit0n above in the error message Check the disk configuration and the NG AMS Configuration LogID NGAMS_AL NO TARGET DISK see Log Type An Log Text NO SUITABLE TARGET DISK FOUND FOR DATA WITH MIME TYPE s PROBABLY VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 109 of 126 NEED TO CHANGE DISKS Ecc Reon No suitable Storage Set disks was found for storing the data with the Mime type as referred above in P the error message Check the disk configuration and the NG AMS Configuration Log ID NGAMS_INFO_STARTING_SRV
11. User s Manual Is 08 01 2003 67 of 126 11 EXPERT The NG AMS Plug in API The NG AMS Plug In API provides convenience functions to facilitate the implementation of the various types of plug ins used within the context of NG AMS The actual thorough documentation is contained as inline Python documentation strings in the code itself For further information about this issue consult Section 23 2 It is recommended to restrict the usage of functions from NG AMS modules to only the one ones contained in the NG AMS Plug In API Python Module ngamsPlugInApi py It should be mentioned that for the moment the amount of convenience functions provided is limited Basically only the functions needed for implementing the plug ins provided so far have been considered in this context If new functions are needed requests for such can be issued to ngast eso org Here is an overview of the convenience functions provided by the NG AMS Plug In API for the moment determineMimeType From the filename of a data file determine the mime type of the file execCmd Execute a command on the shell and return a tuple with the following information lt exit code gt lt stdout gt lt stderr gt getDppiPars Get the parameters specified for a DPPI in the configuration in raw format as given in the configuration Parse the plug in parameters These are supposed to be defined in the following format lt parameter gt lt value gt lt param
12. enna physical slot in the NGAS Host in which the disk is installed The mount point is the absolute path Device Name A n setu PE BIT The device name is the directory Disk Dictionary device name name used to control the atus The status is used to indicate if there might be something wrong with a disk If thi OK th h DEON es o A pie n aaa ana The type of disk indicates the kind of storage media Could e g be ATA MAGNETIC DISK The name ONP anufa er Model The manufacturer is the name of the The model is the model identification producer of the disk Could be e g given by the manufacturer IBM or Seagate Figure 35 The NG AMS Physical Disk Dictionary An exception must be thrown in case errors occur during the process of bringing the system to Online State 12 2 EXPERT Example System Online Plug In In the following an example System Online Plug In which is used for the moment for the NGAS installation for WFI at the La Silla 2 2m telescope It is perhaps not a very good example of such a plug in since most of the code is distributed in other modules Please check the Python source files ngams ngamsPlugIns ngamsEscalada6800Utils py and ngams ngamsPlugIns ngamsLinuxSystemPlugInApi py for further information SEK KKKKKKK KK KK KK KK KK KK KK KK KK KK KK KK KEKE KK KEK KEK KK KK KK KEKE KK KK KK KK KK KK KK KEKE KK KK ESO DMD Id ngamsLinuxOnlinePlugIn py v 1 16 2002 07 10 08 34
13. 302200 Log Type ERROR 000 Log Text Not enough disk space for creating file s of size d bytes Description NG AMS could not create a file with the given name of the given size VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Isu 08 01 2003 110 of 126 Log ID INGAMS ER RETRIEVE KEYS Log Number 14000 Log Type ERROR Log Text Problem occurred retrieving key s s from data file s The keyword cards listed above could not be found in the file referred This means that the handling of the file could not be carried out as expected Description LogID NGAMS_ER_DAPI Log Number 4001 Log Type ERROR ker S En An error occurred in the Data Handling Plug In the file has not been handled successfully and has Description not been archived Log ID NGAMS ER REQ HANDLING Log Number 4002 oe Type ERROR Log Text Text Error Error occurred handling request Error exception s Error occurred handling request Error exception s request Error exception s A problem occurred while handling a request issued to NG AMS The request could not be carried out Description successfully LogiD NGAMSER ARCHIVE PUSH REQ SOSOSt CSsti S S SCS Log Type ERROR A problem occurred while handling an Archive Push Request The archive request has not been carried Description out successfully It should be investigated what caused the problem and the file should possi
14. Compression compress f lt DiskStatus lt NgamsStatus gt FileId WFI 2001 09 15T22 49 07 652 FileName saf 2001 09 15 1 WF1 2001 09 15T22 49 07 652 fits Z FileSize 142074506 FileStatus 00000000 FileVersion 1 Format application x cfits Ignore 0 IngestionDate 2003 01 02T13 48 10 000 Tag UncompressedFileSize 141546240 gt 5 Figure 60 Example File Info Status VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 99 of 126 23 EXPERT The NG AMS Python Modules In this chapter an overview of the NG AMS Python modules classes functions and constants is given It is not the intention to provide the complete and detailed documentation for all this This is contained as inline Python documentation in the Python code and can be browsed online See Section 23 2 for a description of how to do this The purpose of this chapter is merely to give an overview of the NG AMS Package For the basic usage of NG AMS it is normally not necessary to have a deep knowledge about the internals of the SW However when developing the different types of plug ins which must be provided to adapt NG AMS to various specific contexts it is an advantage and in some cases crucial to have some insight in and overview of the SW and the classes and features available 231 EXPERT NG AMS Module Structure Although this manual is not meant as a maintenance manual for NG AMS the structure of the NG AMS modules is brie
15. DCPI A Data Checksum Plug In must return the calculated checksum value as a string 182 EXPERT Example Data Checksum Plug In In the following the source code of a small example Data Checksum Plug In is shown It generates the checksum based on routines built into Python HARK KK A A RRR AAI KK KK RR RRR IA ARR II k k k k k k k k k k k k k k k k k A k k k k k k k k k k ESO DFS Id ngamsGenCrc32 py v 1 12 2002 07 10 08 34 33 arcsw Exp jknudstr 23 01 2002 Created Checksum Plug In to generate the checksum stored in the ngas files tabres in connection with each file archived into NGAS nun import binascii from ngams import def ngamsGencrc32 srvObj filename Plug in to generate CRO 32 Checksum for an arohivedtdata fale srvObj Reference to instance of NG AMS Server class ngamsServer filename Name of file to generate checksum for string Returns CRC I SZ Che CkKsUunstowstate MSN gk VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 90 of 126 fo open filename r but 0 read 65536 cree 0 while buf GEO pinas erie CoL lous Ciel put fo read 655316 POSS LOOSELY return str crc Figure 51 Example Data Checksum Plug In FILE ngams ngamsPlugIns ngamsGenCrc32 py VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 91 of 126 19 EXPERT The Suspension Plug In When an NG AMS Server is s
16. Indicates the free disk space limit before changing disk MB Za lt ELEMENT Monitor EMPTY gt lt ATTLIST Monitor MinFreeSpaceWarningMb CDATA REQUIRED FreeSpaceDiskChangeMb CDATA REQUIRED gt SESS The Log Element defines properties for the logging performed by the NG AMS Server Attributes SysLog Switch on UNIX syslog logging 011 SysLogPrefix Prefix tag written first in the syslog entries in the data part LocLogFile Indicates a name of a local log file Should be complete path LocalLogLevel Log level for producing logs into the local log file Must be in the interval from 0 to 5 LogBufferSize Size of the internal log buffer in terms of log entries This indicates how many entries are buffered before the logs are purged out to the log file lt ELEMENT Log EMPTY gt lt ATTLIST Log SysLog SysLogPrefix LocalLogFile LocalLogLevel LogBufferSize 011 REQUIRED CDATA REQUIRED CDATA REQUIRED 01112131415 REQUIRED CDATA REQUIRED gt A The Notification Element is used to define subscribers e mail recipients that will receive an e mail when certain events occur For the moment the following events can trigger e mails to one or more recipients Alert Logs Error Logs Disk Running Full Min Free Disk Space Disk Change Disk Change Log o No more disk space lt ELEMENT Notification AlertNotification ErrorNotification DiskSpaceNotification DiskChangeNotification NoDiskS
17. Log Number 3016 Log Type INFO Log Text Starting initializing NG AMS Server Version s Host s Port d Description The NG AMS Server is initializing and preparing for execution Log ID NGAMS_INFO_TERM_SRV ILog Type o S E LlogID NGAMS_ER_EMAIL NOTIF 0 Log Type ERROR Log Text Problem sending email notification message to s sender Yos using SMTP host Yos Error Yos The system could not send an Email Notification Message to the destination specified using the SMTP Description host specified Check if the parameters are correct in particular if the SMTP host is correctly specified and accessible ILogID INGAMS INFO DATA CHK STAT gt gt gt Log Number 8020 logTye NFO SOS C lt C lt C Log Text Number of files checked d Amount of data checked 3f MB Time for checking 3fs DRON After one Data Checking cycle has been executed it is possible to confi gure NG AMS to generate a p log entry in the log outputs CFG NgamsCfg FileHandling DataCheckLogSummary ILogID NGAMS ER ARCH BACK LOG BUF OW Log Number 30200 ES Error _ f k Log Text An error occurred while archiving data found in NG AMS Back Log Buffer Error s An error was encountered when the NG AMS Janitor Thread tried to archive a file found in the poe NG AMS Back Log Buffer The symptom type of the error is indicated in the error message LogID NGAMSERNO DISKSPACE gt Log Number
18. NG AMS does not implement any means whatsoever to synchronize distributed DBs nor to prevent the various NGAS DB sites from being corrupted It has been decided to keep NG AMS completely decoupled from any DBMS specific feature and it is entirely up to the designers of the NGAS infrastructure to ensure data consistency in the various DB holdings As an example of this it can be mentioned that if disks are prepared in the Archive Facility Site and distributed to various remote Data Production Sites it is desirable that 1 The DB at the Archive Facility Site is kept up to date 2 The NGAS installation at the Production Site recognizes that the disk is already registered as an NGAS Storage Media see also Section 2 7 and 3 At the same time after a Storage Media has been completed has left the Production Site and is located in an NGAS Host at the Archive Facility Site that it is no changes introduced to the record for that disk at the remote site are replicated to the Archive Facility Site NGAS DB Unidirectional Conditional Replication NGAS Disks Figure 29 Example of a Distributed NGAS installation using unidirectional conditional DB replication A way to implement this is as follows When NG AMS registers a new disk it creates an entry for this in the NGAS DB at the Archive Facility Site NG AMS also generates an NgasDiskInfo document see Section 22 2 on the disk In this way the disk is marked as a known N
19. The DPPI must be contained in a Python module file which has a function of the same name as the module The latter is the actual DPPI which is invoked by NG AMS A DPPI has an interface as shown in Figure 46 srvObj Instance of the ngamsServer class used by this session See also Section 12 1 reqPropsObj filename regPropsObj Object of class ngamsRegProps Contains information in connection with the handling of one request filename Name of file to be processed pes h Laga ye Ft PPT mn DPPI Function Name 4 Or ne identical to that of the Python module hosting th T function Figure 46 Function interface of a DPPI VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 86 of 126 A DPPI must return an object of the type ngamsDppiStatus This again contains one or more objects of the type ngamsDppiResult which each refer to result data or contains the result of the processing This means that it is possible to produce several results in a DPPI and to have these send back to the requestor The concept of the DPPI return object is shown in Figure 47 Data Type FILE Mime Type Ref Filename Proc Directory Data Type DATA Mim e Ref Filename Proc Directory 15 APP ee Result File Data Type DATA Mime Type Ref Filename Proc Directory Figure 47 DPPI structure of return data As shown in Figure 47 the ngamsDppiStatus object can co
20. User s Manual Is 5 08 01 2003 54 of 126 OA Pa Requests varchar 20 Indicates the State of the server Set to Tif the server is suspended varchar 32 null IName of an NG AMS Server which is requested to wake srv_data_checking tinyint null Indicates if this server is carrying out Data Consistency Checking see Section 3 9 up this host server that has suspended itself datetime Time when the host server would like to be woken up Table 15 Contents of the NGAS Hosts DB Table The ngas_subscr_back_log and ngas_subscribers tables are used by NG AMS to keep track of the status in connection with Subscribers and of data that could not be delivered These tables are maintained entirely by NG AMS and usually the user does not have to be concerned with these 8 5 EXPERT Table ngas subscr back log host id varchar 32 not null The ID of the host where the Data Provider NG AMS Server is running srv port int not null The port number used by the Data Provider NG AMS Server int fuser td ars froc OOO fine ID of the Subscriber subscr_url varchar 255 not null The Delivery URL submitted by the Subscriber The Data Provider will POST the data on this URL Te ame The format mime 1ypo of the fe Table 16 Contents of the NGAS Subscription Back Log DB Table varchar 64 NGAS ID of file that could not be delivered 8 6 EXPERT Table ngas subscribers Type The ID of the host where the Data Prov
21. c nst int pot Case Tioat timeoutSecs const char fileUri const char STONUL PIPAT IG const int size const char mimeType const int noVersioning const int wait ngamsSTATUS status Archive a file which contents have been loaded into a buffer in memory fileUri mimeType noVersioning See ngamsArchive IQUE C 16 9 Pointer to buffer containing the contents of the file VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 60 of 126 Size i bytes of the data loaded into memory ngamsSTAT ngamsClone const char Ne Sitar CON por const float timeoutSecs const char xfileId const int fileVersion const char Cli Sele CONS CEA wait ngamsSTATUS Status Execute a CLONE command For the exact interpretation of various combinations of fileId fileVersion and diskId consult the man page for the NG AMS Python module ngamsCloneCmd function clone fileId TD oye EILS Eo CLtONGs fileVersion Versron of IriLes tobe taken into account or ehe cloning diskId Disk ID for the files to be taken into account ngamsSTAT ngamsCmd2No const ngamsSMALL BUF cmdStr ngamsCMD cmdCode Convert a command given as string into the corresponding code integer cmdStr Command name as string cmdCode Command code as defined by the enumerated type ngamsCMD ngamsSTAT ngamsCmd2Str const ngamsCMD cmdCode ngamsSMALL_BUF cmdStr Convert an NG AMS command given as a code integer to a s
22. connecting to the DB should be created e g ngas The NGAS tables must also be created This should be done using the SQL script contained in ngams ngamsSql The script is called ngamsCreateTables sql The script can be executed using isgl Prepare NG AMS Configuration Use possibly as a template configuration the configuration example file provided within the NG AMS SW package ngams ngamsData ngamsServer xml Go carefully through the list of parameters and configure these according to the description provided in Chapter 6 Prepare the necessary plug ins needed for operating NG AMS The plug ins to consider first are the System Online and Offline Plug Ins see the Chapters 12 and 13 In addition the DAPI for each type of data to be handled archived see Chapter 15 If it is desirable to calculate a checksum for the data files being archived a Data Checksum Plug In must be provided see Chapter 18 If data should be processed a DPPI should be provided for each type of processing offered by the system see Chapter 17 If labels for the disk cases should be generated a Label Printer Plug In must be provided as well see Chapter 14 Example implementations of all of these types of plug in are provided within the NG AMS package ngams ngamsPlugIns Note that all plug ins provided should be made available in a path pointed to by the PYTHON_PATH variable or in one of the search paths compiled into the Python interpreter Launch
23. e g arcdevl jknudstr dev ngams 65 gt pydoc p 7878 amp 2 15578 arcdevl jknudstr dev ngams 66 gt pydoc server ready at http localhost 7878 Figure 62 Starting the pydoc utility as an HTTP server Afterwards the NG AMS documentation can be accessed online via the URL e g http arcdev1 hq eso org 7878 ngams html The pydoc utility provides a convenient way of browsing the documentation and generates the documentation online It locates the NG AMS module if installed within the search paths compiled into the Python interpreter Alternatively it should be located in a path defined in the PYTHON_PATH environment variable VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual 1 08 01 2003 102 of 126 24 EXPERT Installation The installation of an NG AMS Server can be a relatively straightforward and simple procedure if it is not necessary to create an adapted installation for a specific context Under normal circumstances the most complex action in this connection might be to configure the server properly The steps to carry out to obtain a running NG AMS installation are as follows Step Aciton i i Verify that a Sybase server is available Sybase ASE 12 Also check that it is possible to connect to the server from the NGAS Host with isgl If this is not possible an entry should be added in the SYBASE interfaces file It should also be verified that the libraries libct a libct sl so are av
24. 33 5 THE NG AMS SERVER AND UTILITIES oo mo Woman 34 5 1 NG AMS Server Command Line Interface ooocooooocoooooocoooooooooooooooooooooooooooooooooooo o 34 5 2 Python and C Command Utilities oocooocooooooooooooooooooooooooooooooooocooooooooocoooooooooo 35 6 EXPERT CONFIGURING NG AMS oo om oom ea ana aan 38 6 1 EXPERT NG AMS Configuration DTD ngamsCfg dtd ooooooocooooooooooooooooooooooooooooo 38 6 2 EXPERT NG AMS Base DTD ngamsiInternal dtd cooooooooocoooooooooooooooooooooooooooooooooo 38 6 3 EXPERT NG AMS Configuration Example oocooooooooooooooooooooooooooooooooooooooo 45 7 EXPERT NG AMS SERVER COMMUNICATION PROTOCOL oo mmm 48 71 EXPERT Format of NG AMS HTTP Command Messages oooooocooooooooooooooooooooooo 48 7 2 EXPERT Format of the NG AMS HTTP Reply cooooocoooooooooooooooooooooooooooooooooooooooooo 49 7 3 EXPERT Format of the NG AMS Redirection HTTP Response ocoocoocooooooooooooooooooooooooo 49 8 Seng THE NGAS DB om nona kaan 51 8 1 EXPERT Table ngas disks oooooooocooooooooooooooooooooooooooooooooooooooococooooooooooooooooooooooo 51 8 2 EXPERT Table ngas disks hist ocoooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 52 8 3 Angan Table mas AS E E E nan 52 8 4 Beng Table ngaS hosts sssssosssssossscssscsosseosessoosesocssososseossssoossssosssoosssososooossssoessosssesosssossssssessssssessssese 53 8 5 BER
25. Contents of the NGAS Disks DB Table cooooooocoooooooooo ooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 52 Table 13 Contents of the NGAS Disks History DB Table coooooooooocooooooooooooooooooooooooooooo 52 Table 14 Contents of the NGAS Files DB Table ocoocooooooo oo ooooooooooooooooooococoroooooooooooooooooooooio io 53 Table 15 Contents of the NGAS Hosts DB Table cooooooocooooooooo ooooooooooooooooooooooooooooooooooooooooooooooooooooo 54 Table 16 Contents of the NGAS Subscription Back Log DB Table oooooococoocooooooooooooooooooooocoo 54 Table 17 Contents of the NGAS Subscribers DB Table ocoooooooo o oooooooooooooo oooooooooooooooooooooooooooooooo o 54 Table 18 Source files in the C API module ooooooooooooooooooooooooooooooooooooooooooooooocooco oooooooooooooooooooo oo o co 57 Table 19 Files generated compiling the C APL cocoooooooooooooooooooooooooooooooooooooooooooooooooooo 57 Table 20 Methods functions in the Python API cooooooooooocoooooooooooooooooooooocoooooooooooooooo 65 Table 21 Functions in the NG AMS Plug In APL ocoooooooooooooooooooooooooooooooocoocoooooooooooooooco 67 Table 22 Return parameters of a DAPI o ocoocooooooooooooooooooooooooooooooocoooooooooooooooooio 78 Table 23 Files and modules in the NG AMS project oooooooooocooooooooooooooooooooooooooooooooooooooooooo 99 Table 24 Python modules in the ngamsLib sub modu
26. EXPERT NG AMS Server Communication Protocol The NG AMS command interface is based on the HTTP protocol which is a widely used standard protocol This makes it easy to interface various client applications with NG AMS In this chapter the details of the NG AMS command interface are described Using the NG AMS Python and C APIs the client applications do not need to worry about the format of the requests sent and replies generated by NG AMS It is therefore recommended whenever possible to use the APIs provided with the NG AMS package 71 EXPERT Format of NG AMS HTTP Command Messages The format of the NG AMS messages is defined as follows Archive Push Request POST ARCHIVE HTTP 1 0 User Agent user agent Content Type ngas archive request lt mime type gt Content Length lt length gt Content Disposition attachment filename lt file uri gt wait 0 1 no versioning 0 1 lt data gt Figure 20 Format of an Archive Push HTTP request Example POST ARCHIVE HTTP 1 0 User Agent NG AMS C API Content Type ngas archive reguest Content Length 69120 Content Disposition attachment filename tmp TestFile fits wait 1 SEURE yayasan ea KE Figure 21 Example of Archive Push HTTP reguest Archive Pull Request Other Commands GET command lt parameter gt lt value gt HTTP 1 0 Figure 22 Structure of NG AMS GET method HTTP request Example Archive Pull Request GET ARCHIVE file
27. For each Data Stream Defined in the configuration file a target disk must be available in order for the system to go Online for an archiving unit If such is not found the system will not go Online LogID NGAMS ER DISK STATUS o Log Number 3006 Log Type ERROR Log Text Error querying information for disk with ID s cannot generate disk status on disk An error occurred while trying to query information about a disk from the NGAS DB This means that it is not possible to generate the NGAS Disk Info Status File on the disk This file is normally Description generated when the system goes Online Offline The status file for the disk in question may not be up to date Note maybe the problem is caused by incorrect information in connection with the disk Could also be caused by a general problem with the communication with the DB server Log ID NGAMS ER OFFLINE PLUGIN Log Number 3007 ici Type 7 b Log Text Text O A problem occurred while executing the Offline Plug In The system could not be brought properly to Description Offline State or some actions may have been skipped LogID INGAMS ER ONLINE PLUGIN Log Number 8008 Log Type ERROR Log Text Text Log Text s A problem occurred while executing the Online Plug In The system could not be brought properly to Description 5 Offline State or some actions may have been skipped Description Log ID INGAMS ER INIT SERVER Log Numbe
28. Local Log File is generated it should be considered that this will continuously grow in size The speed with which it will be growing depends on the Log Level selected If it is desirable to keep the log files a DAPI to handle this could be provided for NG AMS and a cron job launched periodically to archive the log file into NG AMS and subsequently to delete it If it is not desirable to preserve the information the file could be deleted periodically This however is up to the people responsible for the individual installation to decide how to handle this Since no security mechanisms are provided at the level of NG AMS to prevent intruders to connect to the server such mechanisms should be put in place at the level of the operating system or network It is up to the people responsible for the security in connection with IT services to decide how to implement this If an organization is running a distributed NGAS system whereby data e g are produced on several remote sites and are made available in an Archive Facility considerations should be done as how to set up the DB infrastructure It might be most logical to have the central reference DB in connection with the Archive Facility and to set up replication from the various remote sites to the Archive Facility DB See also Section 8 7 for more information about this issue Table 25 Steps needed to install NG AMS As can bee seen above the installation in the worst case may be a qu
29. NG AMS Memory may be allocated dynamically and needs to be freed EXAMPLES To archive a file using the API the following must be called from the application include ngams h VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 64 of 126 ngamsSTATUS status if ngamsArchive wfinau 7171 10 home data MyFile fits 1 0 amp status ngamsSTAT SUCCESS ngamsDumpErrStdout amp status error handling To retrieve a file into the directory home data target_dir The name will be the same as the File ID include ngams h ngamsSTATUS status if ngamsRetrieve2File wfinau 7171 30 Uta AOO MIA WAS LEO QS Bu home data banged amp status ngamsSTAT SUCCESS ngamsDumpErrStdout amp status error handling CAUTIONS If data is returned by the NG AMS Server the member replyData will have a pointer pointing to each block of data It is the responsibility of the calling application to free these data chunks The function ngamsereestatus t can be used for thls Last change 24 12 02 12 24 Figure 30 The functions and macros of the NG AMS C API VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 65 of 126 10 EXPERT The Python API The NG AMS Python API can be used by Python applications to interface with the NG AMS Server in an easy and straightforward manner The API hides most of the technical details of the NG AMS com
30. Read error while reading data Invalid answer from data server Invalid filename specified Cannot allocate memory status code command issued target filename specified parameters given Unknown Unknown Invalid Invalid Request successfully handled by server The reply is an KUN redirection tesponse Invalid query ors defined by NG AMS can be returned In case of All functions return ngamsSTAT_SUCCESS in case of success an error a termination status within the set of status codes given above The following macros are defined for referring to NG AMS commands Command Macros 2 Description ngamsCMD_ARCHIVE ngamsCMD_CLONE ngamsCMD_EXIT ngamsCMD_LABEL ngamsCMD_ONLINE Archive file Clone files Make NG AMS Server exit Make NG AMS print out a disk label Bring NG AMS Server Online VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 59 of 126 ngamsCMD_OFFLINE Bring NG AMS Server Offline ngamsCMD_REGISTER Register files on a disk ngamsCMD_REMDISK Remove a disk from NGAS ngamsCMD_REMFILE Remove a file from NGAS ngamsCMD_RETRIEVE Retrieve a file ngamsCMD_STATUS Query status information from NG AMS ngamsCMD_SUBSCRIBE Subscribe to a NG AMS Server ngamsCMD_UNSUBSCRIBE Un subscribe cancel a previous subscription 2 All command macros exist also as a string which carries the name of the enumerated macro name with a _STR appended In the following the functions pr
31. S O NG AMS User s Manual LABEL Command 117 Label Printer Plug In 73 Label Printer Plug In Example 73 Label Printer Plug In Interface 73 Label Printing 28 label Python API 65 Label Example 28 libngams a C API 57 Local Host 22 Local Log File 23 Local Log Files Handling of 101 Location of a Local Log File 23 Log Level 24 Logging 23 Logical Name 13 Main Data File 13 Main Storage Area 13 Makefile C API 57 mime type 19 Mime types 20 Modules 97 MS Windows 16 multipart mixed 49 Multisite DB 101 Multithreading 16 N Next Generation Archive System 12 NG AMS 12 NG AMS Base DTD 38 NG AMS C API 100 NG AMS Commands 116 NG AMS Configuration 38 NG AMS Configuration DTD 38 NG AMS Configuration Example 45 NG AMS Disk Infrastructure 18 NG AMS Executables 34 NG AMS HTTP Command Messages 48 NG AMS HTTP Reply 49 NG AMS Modules 97 NG AMS Plug In API 67 NG AMS Python Modules 97 NG AMS Redirection HTTP Response 49 NG AMS Server 13 34 NG AMS Server Command Interface 27 NG AMS Server Communication Protocol 48 NG AMS Server Command Line Interface 34 NG AMS Status DTD 93 NG AMS Status XML Document 93 ngams h C API 57 ngamsC Client C API 57 ngamsC Client Module 57 ngamsC Client c C API 57 ngamsC ClientLib c C API 57 ngamsCfg dtd 38 ngamsConfig 97 ngamsConfig Class 67 ngamsConfig py 97 ngamsDb Class 67 ngamsDb py 97 ngamsDhp
32. Table ngas subscr back 10g coocooooomo ooo oo o o o o o o oooo oo o co oo cooco oo Kooooo oocooco oo 54 8 6 BABERIR Table ngas subscribers ooocoooooooo oo o oo o ro o ooro coocoocoo o o oo ooco coo o oo onconcoooo ooio 54 87 EXPERT Synchronizing Distributed NGAS DBs ooooooooooooooooooooooooooooooooooooooo 55 9 Wa THE CAP en ta laa eee 57 91 EXPERT NG AMS C API Header File ngams h ooooo oooooooooooooooooooooooooooomnn nee nnennannnnnan 57 9 2 10 11 12 12 1 12 2 13 13 1 13 2 14 14 1 14 2 15 15 1 15 2 15 3 16 16 1 16 2 17 17 1 17 2 18 18 1 18 2 19 VLT MAN ESO 19400 2739 ESO Noms User s Manual Is 08 01 2003 5 of 126 EXPERT NG AMS C API Man Page oooooooooooooooooooooooooooooooooooooooooooooooooooooco 57 Bee THE PYTHON API mmm ea mean nan aan 65 EXPERT THE NG AMS PLUG IN API mm mna 67 EXPERT THE SYSTEM ONLINE PLUG IN o ooo om mo Woman enam 68 EXPERT Interface of a System Online Plug In oooooocoooooooooooooooooooooooooooooooooooooooooo 68 EXPERT Example System Online Plug In ooooooooocooooocoooooooooooooooooooooooooooooooooooooooooooooco 69 EXPERT THE SYSTEM OFFLINE PLUG IN oo oom mantan 71 EXPERT Interface of a System Offline Plug In ooocoooooocooooooooooooooooooooocoooooooooooo 71 EXPERT Example System Offline Plug In ooooocoocooooocoooooooooooooooooooooooooooc
33. The bytes in the status are counted from left to right 1 8 The File Checksum is incorrect Zn File being checked 3 8 Not used The bytes 3 8 may be used at a later stage Table 14 Contents of the NGAS Files DB Table 8 4 EXPERT Table ngas hosts host_id varchar 32 not null ID of the NGAS Host e g jewel65 Should be given only as the name i e without the domain name varchar 30 Domain name of the NGAS Host e g hq eso org paddies varchar frorn he IP address of the NGAS Host ngas_type varchar 10 not null The type of NGAS Host i e which role it has The value of this is not used by the NG AMS SW Suggested values could be NAU NGAS Archiving Unit NBU NGAS Buffering Unit NCU NGAS Central Unit NMU INGAS Master Unit and AHU Archive Handling Unit mac_address varchar 20 null The MAC address coded into the network card used for communication by the NGAS Host This is an address like 05 4E 14 8A 11 2B Kumber of slots inthe NOAS Node varchar 10 IName of the NGAS Cluster this system belongs to installation_date datetime null Date the OS and NG AMS running on the NGAS Host have been installed varchar 20 Version of the NG AMS Server Port used by the NGTAMIS Server Archive Requests 1 accept Archive Requests Retrieve Requests 1 accept Retrieve Requests gt All columns initiated with srv_ are set by NG AMS VLT MAN ESO 19400 2739 ESO NGIAMS
34. before archiving it Such a check could e g be to calculate the checksum of the file or to check that certain parameters are properly set in the data file If inconsistencies are found the file should be moved to the Bad Files Directory on the Target Disk This however is done by NG AMS If a file is found to be bad an exception should be thrown which contains the error mnemonic NGAMS ER BAD FILE see the example DAPI Section 15 3 for clarification on this topic FUNCTION ngams ngamsPlugIns ngamsFitsPlugIn checkCheckSum e Data Processing Before archiving a file it is often necessary required to do some processing It could be something as simple as compressing the file but in principle there are no limits to the kind of data processing that can be carried out If the processing changes the mime type of the file it is important that the DAPI returns the new mime type to NG AMS e Generating Final Target Filename The target filename of a data file may be generated from parameters in the header The filename is composed by the Mount Point of the disk plus the Path Prefix from the configuration How the rest of the filename is generated is up to the plug in implementation Generating Standard DAPI Return Value A number of parameters like File ID File Version file size Disk ID and more for the file archived must be returned to NG AMS in order to update the NGAS DB accordingly A convenience function provided in ngamsPlugInApi shoul
35. being selected for delivery to a requestor for instance in connection with the Data Subscription Service see Section 4 2 The Filter Plug In is a function which understands the data it is applied to and which based on the contents of the data file perhaps the filename and other pertinent information can determine if the file matches the requirements 21 1 EXPERT interface of a Filter Plug In The plug in must be contained in a Python module which has a function of the same name as the module The latter is the actual plug in which is invoked by NG AMS A Data Checksum Plug In has an interface as shown in Figure 56 bj Instance of the ngamsServer class used by this session See also Section 12 1 filterPiPars Plug in parameters specified in connection wit the request involving the Filter Plug In filename Complete filename of file to test fileld ID of the file on which to apply the Filter Plug In should be capable this is unspecified 1 filename fileld fileVersion 1 reqPropsObj None req psObj bbject of ISS nga egrrops Contains infor the handling of one reguest See chapter 23 Beware the plug should be capable of handle the situation when this is undefined None be identi in function Figure 56 Function interface of a Filter Plug In A Filter Plug In must return either 1 or 0 depending on whether the filter conditions are met or not 21 2 EXPERT Example Filter Plug In
36. by the C API The source and the header files for the C API is contained in the module ngams ngamsCClient This CVS module contains the files Makefile Make that generates compiles the C API Can be invoked with the parameters clean and all e g make clean all ngams h Header file for the NG AMS C API module This contains the definition of the function prototypes and the definition of various macros that can be used in the clients built using the ING AMS C API ngamsCClient c The source file for the NG AMS C based command line utility See also Section 5 2 for more information about this tool ngamsCClientLib c The source file for the library functions provided by the NG AMS C API Table 18 Source files in the C API module Compiling the ngamsCClient module the following header files and binaries are generated The binary executable utility which can be used to communicate with the NG AMS Server from the command line Refer to Section 5 2 for further information ngamsMAN_PAGE h Header file containing the text of the man page for the NG AMS C API command line utility see Section 9 2 ngams VERSION h Header file containing the version information for the given distribution of NG AMS Table 19 Files generated compiling the C API In the following sections the header file for the NG AMS C API is listed In addition the man page for the C API library functions is shown 9 1 EXPERT NG AMS C API Header File nga
37. can either subscribe to all data being archived on an NGAS Host or to part of it Latter is done by means of Filter Plug Ins that are applied on the data to determine whether to export it or not In this way it is e g possible to synchronize data holdings between different NGAS Nodes see Section 4 2 Removing File and Disk Information Two commands are provided to remove single files or set of files Another command is provide to remove an entire disk from the system see the Sections 27 9 and 27 10 Suspension Wake Up Service An NG AMS Server can be configured to suspend the NGAS Host where it is running A service is provided so that another NGAS Server can be requested to wake up an NG AMS Server suspending itself see Section 4 3 Generation of Checksum NG AMS generates a checksum value for each file generated This is based on a plug in concept so that context data specific checksum calculation can be applied see Chapter 18 Extendable for Usage with Various DBMS NG AMS is prepared for usage with various DBMS For now only Sybase is supported but this can easily be expanded Simulation Mode NG AMS provides a Simulation Mode which makes it possible to operate the system without the availability of the actual HW like the disk controller disks etc Running in Simulation Mode a simulated NG AMS environment is generated on a single disk This is useful for test and development The Simulation Mode however could also be us
38. clean up and terminate execution ngamsSTAT ngamsLabel const char host GONSE ANE POrt const float timeoutSecs const char iko IG ngamsSTATUS Stachel sy Send a LABEL command to the NG AMS Server Silet ID of slot hosting the disk for which to generate the label char ngamsLicense void Return pointer to buffer containing the NG AMS License Agreement ngamsSTAT ngamsOffline const char SAOS Const ANE po const float tim outseds const int FOLES Const AAE wait ngamsSTATUS Saus Send an OFFLINE command to the NG AMS Server to bring it to Offline State force Force the server to go Offline immediately even though it is busy ngamsSTAT ngamsOnline const char host const int port const float timeoutSecs const int wait ngamsSTATUS SS LEIS Send an ONLINE command to the NG AMS Server to bring it to Online State ngamsSTAT ngamsRegister const char sosis const int port const float timeoutSecs const char path const int wait ngamsSTATUS status Send an REGISTER command to the NG AMS Server to make it register a file or set of files path Path starting point from which the scanning for files to register will be initiated Only files with a known mime type ig taken into account ngamsSTAT ngamsRemDisk const char host const int port const float timeoutSecs gonst ehar ZS krad const int execute ngamsSTATUS See Send a REMDISK command to the NG AMS Server If execute is 0 the dis
39. eso org NGAST Figure 16 The NG AMS C Client and Python Client online help output on stdout VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 38 of 126 6 EXPERT Configuring NG AMS The NG AMS SW is implemented in a very flexible way to be able to adjust the system for various scenarios In order to obtain this a wide range of parameters can be adjusted in the NG AMS Configuration This chapter contains a description of these parameters The format for the NG AMS Configuration also includes the Header Element which is a generic standard header for XML documents This header is not described here An example of an NG AMS Configuration can be found in Section 6 3 The DTDs defining the format of the NG AMS Configuration can be found in the Sections 6 1 and 6 2 6 1 EXPERT NG AMS Configuration DTD ngamsCfg dtd The DTD for the NG AMS Configuration is based on the ngamsInternal dtd see Section 6 2 which defines the NG AMS specific elements used in the NG AMS Configuration The contents is as follows lt xml version 1 0 encoding UTF 8 gt lt ENTITY XmlStd SYSTEM http www eso org projects esoxml Xm1Std dtd gt SXm1Std lt ENTITY NgamsInternal SYSTEM ngamsInternal dtd gt sNgamsInternal 2 sagem ea eee Ww Is 7 200271202 jieOjsI2 Gea mx Si When What KAR KKK KK KAR KK KKKKK KARR KK KKK KKK KKK KKK KKK KKK KKK KK KK KKK KKK KK KK KKK kak kak kk jknudstr 04 04 2001 Cr
40. file with URE S sss Description The file referenced by its URI in the log message was successfully handled archived by NG AMS LlogID NGAMS_ER MIS PAR gt Log Number MoA Log Type ERROR eessesssss s s s s Log Text Missing parameter s in connection with command Gs O oa Together with the command issued the parameter listed above should have been issued Since this is Description not the case the request cannot be handled Log ID NGAMS_INFO_FILE_AVAIL Log Number 4022 Log Type INFO Log Text File with File ID s is available on NGAS Host with Host ID s Description The file with the given File ID is available and accessible on the NGAS Node contacted Log ID NGAMS INFO FILE NOT AVAIL Log Number 4023 Log Type INFO Log Text File with File ID s is not available on NGAS Host with Host ID s Description The file with the given File ID is not available accessible on the NGAS Node contacted Log Type NFO The file requested is not available on the NGAS Node contacted and the NG AMS Server on this Description system cannot act as proxy to query the file and send it back The file in question can be found at the URL given in the message Log ID NGAMS_INFO_SERVICE_UNAVAIL Log Number 4025 VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 114 of 126 Log Type INFO Log Text The service s is not available on this system D The service
41. g if a location for the Processing Area is chosen which has a poor I O performance this may slow down the processing considerably It is possible to make NG AMS carry out replication of the files being archived This feature can also be disabled CFG lt NgamsCfg Ngams Replication gt The data files archived must be stored under a single directory referred to as Storage Area in Figure 3 in the mount directory on the target disks The name of this area is configurable CFG lt NgamsCfg FileHandling PathPrefix gt It is up to the DAPI implementation to define the structure of the directories and files within the Storage Area On the data disks there is also a Staging Area used by NG AMS when receiving data files Data is received directly onto the Main Target Disk for efficiency reasons The name of this directory is lt NgamsCfg Ngams MountRootDirectory gt lt disk mount directory gt staging There is only one such Staging Area on the Main Disk In case a file is identified as bad by the DAPI it is stored in the Local Bad Files Area The exact path of this is lt NgamsCfg Ngams MountRootDirectory gt lt disk mount directory gt bad files A file could be considered as bad e g if a checksum value for the file is found to be inconsistent There is only a Bad Files Area on the Main Disks Also located on the Data Disks is a file named NgasDiskInfo This file is an XML document that contains a summary of the information about the disk
42. in the redirection response and will be able to get access to the data directly from that location if the system permits It should be mentioned that it is normally more efficient to request the data directly from the location where it is actually located rather than using NG AMS as a proxy server Using the NG AMS APIs this is all handled transparently for the client application VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 51 of 126 8 EXPERT The NGAS DB The NG AMS SW is based on four tables in the NGAS DB These are e ngas disks Contains information about the disks which have been registered in an NGAS installation e ngas disks hist Contains a log about major events that has occurred in the life time of a disk The information in the ngas disks hist is newer removed automatically by NG AMS and will continue to grow with time whereby the amount of events recorded are kept to a minimum i e only the most essential events in the life time of a disk are logged in this table The entries of this table can be used to keep analyze how a disk have been used in the NGAS system e g how many times it has been registered after re cycling and when it was registered the first time e ngas files Contains information about each files which have been archived into NGAS e ngas hosts Contains information about the hosts in an NGAS installation In addition two internal tables are used by NG
43. lt host gt port Sport cmd lt command gt v lt level gt version gt d noWait status license timeOut lt secs gt lt cmd spec pars gt hostoLhost gt i Host where the NG AMS Server is running SPOTL PO Port number used by the NG AMS Server cmd lt command gt Command to issue The commands accepted by the tool args ARCHIVE Archive a file LONE Clone a file or a set of file Make the server exit Re initialize the server Print a disk label Bring server CO OnLine Sicenes Bring server to OFFLIN State REG sie aA FILO Oie a SSE Cie las Remove information about a disk from the NGAS DB and delete the files stored on the disk Remove info about a file or a set of files from the NGAS DB and delete the files from the disks hosting them RETRIEVE Retrieve a data file from NGAS STATUS guery Statuss SUBSCRIBE Subscribe to data and act as HTTP daemon that is ready to receive the datas UNSUBSCRIBE Cancel a subscription to data v lt level gt Verbose output leve version Print version and exit Run in Debug Mode Don t wait for the NG AMS Server to terminate the handling of the command Dump the status message sent by the NG AMS perver To Siecloiwie license Print out NG AMS license information in stdout timeOut lt secs gt Timeout in seconds to apply during the communication with the server Note This is only supported for the NG AMS C Client For the P Clie
44. making use of the multipart mixed mime type as known from e g emails This is foreseen to be supported soon VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 87 of 126 from ngasUtils import printhead def ngamsExtractFitsHdrDppi srvObj regPropsObj filename wen This DPPI extracts the main header from a FITS file requested from the ESO Archive Note This DPPI works directly on the archived file since it is read only access srvObj Reference to instance of the NG AMS Server class ngamsServer regPropsObj NG AMS request properties object ngamsRegProps filename Name of file to process string Returns DPPI return status object ngamsDppiStatus statusObj ngamsDppiStatus ngamsDppiStatus pH printhead PrintHead filename pos filename rfind fits file_id filename pos resFilename file id hdr mimeType ngamsPlugInApi determineMimeType srvObj getCfg resFilename resObj ngamsDppiStatus ngamsDppiResult NGAMS_PROC_DATA mimeType pH HEAD resFilename statusObj addResult resObj return statusObj Figure 48 Example Data Processing Plug In FILE ngams ngamsPlugIns ngamsExtractFitsHdrDppi py Another example of a trivial DPPI is shown in xxx This DPPI is used to decompress files which have been archived in compressed format SEK KKKKKKK KK KEK KEK KK KK KK KK KK KK KK KK KEKE KK KK KK KK KK KK KK KEKE KK KK KK KK KK KK KK KK KK KK KK
45. must be identical to that of the Python module hosting the DAPI function Figure 41 Function interface of a DAPI A DAPI must perform the following return when finishing execution return ngamsPlugInApi genDhpiSuccessStat diskId relFilename fileld fileVersion format fileSize uncomprSize compression relPath Slo tidy fileExists completeFilename Figure 42 DAPI return statement The return parameters of a DAPI are as follows format String Format or mime type of the file Only mime types defined in the NG AMS Configuration are accepted Size ofthe file as it is archived uncomprSize Integer Uncompressed size of the file I e if the file was compressed this is the original size before archiving compression compression String Compression method used to compress file Should be the command invoked to compress the file e g compress Path relative to the mount point of the target disk Slot ID of slot in which the Main Disk is installed fileExists Integer Indicates if the file already existed on the target disk In case yes this should be 1 otherwise 0 completeFilename String The complete name of the file as it should be The complete name must be generated by the DAPI Table 22 Return parameters of a DAPI E S O NG AMS User s Manual VLT MAN ESO 19400 2739 2 08 01 2003 79 of 126 15 2 EXPERT Overall Structure amp Algorithm of a DAPI The overall structure of a DAPI Py
46. not formatted as expected Log ID INGAMS_ER_MISSING_DISK_ID Log Number 2000 VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 106 of 126 Log Type ERROR Log Text Error Disk ID s not found in DB Description Could not find the Disk ID referred in the DB LogID INGAMS WA DB CON Log Number 2001 Log Type WARNING Log Text DB Connection not open trying to reconnect eni A problem occurred interacting with the DB It will be attempted to connect again to see if problem Description 3 may be rectified LlogID NGAMSER_DB_COM gt gt Log Number 2002 Log Type ERROR 0 Log Text Problems communicating withtheDB s A problem occurred interacting with the DB It may be temporarily impossible to communicate with Description the DB server When this situation occurs the system can buffer frames if configured to do this It will be attempted to archive these buffered frames at a later stage ILogID NGAMS_NOTICE_NGAS_HOSTS gt Log Number 2003 Log Type NOTICE Log Text Table ngas_hosts in the NGAS DB could not be accessed 0000 Desioa A problem occurred while trying to access the ngas_hosts table in the NGAS DB This in some cases p is accepted as it possible to operate NG AMS without the availability of ngas_hosts Log ID NGAMS_ER_MAIN_DISK_WRONGLY_USED Log Number 3000 Log Type ERROR Disk in slot s with Logical Name s is previously registered as
47. of NG AMS within ESO ngamsServer Contains the source code used to build the NG AMS Server ngamsSql Contains the SQL scripts used to build the NGAS DB Table 23 Files and modules in the NG AMS project The ngamsLib module is the one which a plug in developer mostly will be concerned with although some knowledge about the NG AMS Server Class and Python module is also needed In the following some components of potential interest for plug in development from the module ngamsLib are briefly described Python Module ngamsConfig py Contains the code for the ngamsConfig class together with other classes used in connection with the NG AMS Configuration This is all used to handle the configuration programmatically ngamsDppiDef Contains the definition of one DPPI from the configuration ngamsStorageSet Class used to manage the information in connection with one Storage Set from the NG AMS Configuration ngamsStream Class used to manage the information in connection with a Stream Definition from the ING AMS Configuration ngamsConfig Class used to handle the information in the NG AMS Configuration It is possible to load and save the configuration file as well as to setting and getting all properties of the configuration It is also possible to generate an XML document of the configuration contained in the object ngamsDb py The module provides the class ngamsDb which is used to access the NGAS DB All DB access s
48. retrieved Used interchangeably with Storage Disk disk as HDDs are the media used presently to store data A storage unit which consists of either one or two disks on which data is archived See Data Subscriber A buffer in which data files that could not be delivered to Subscribers are stored until a delivery can be carried out successfully Media selected to receive a file being archived Table 4 Glossary used in the NG AMS User s Manual VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 15 of 126 2 Overview In this chapter the basic concepts of NGAS and NG AMS are described An overview of the NG AMS is given as well as a description of the various fundamental features and services provided by NG AMS This chapter provides a somewhat high level description of the most important features and services More in depths descriptions can be found in the subsequent chapters Data Provider SSS N Archive Push R rchive Push Request Seas Archive Pull Request HTTP POST Request Data Vaz Requestor NG AMS Server NG AMS Pa Stdout Info Requestor HTTP POST KOK Reguest Data Subscriber Figure 1 Example operational environment of the NG AMS Server 21 The Concept of NGAS amp NG AMS The concept of NGAS is to use random access Storage Media to obtain high I O performance However any device that can be mounted under UNIX LIN
49. s Manual or the online NG AMS help for further information about the command LogID INGAMS ER CMD BXEC gt gt A Log Number 4029 Log Type ERROR O OSOS Log Text Problem s encountered executing command 9es Error Gs A problem or problems was were encountered while executing the command indicated in the error Description message This may be due to an illegal combination of parameters or due to that prerequisites for executing the command are not fulfilled Log ID NGAMS ER DEL FILE DISK Log Number 4030 Log Type ERROR Problem s encountered deleting file from disk Disk ID s File ID s File Version d Log Text Yak Exception Yos Description A problem or problems was were encountered while trying to delete the referenced file from an NGAS VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 115 of 126 disk Could be due to e g that the disk is read only mounted or that the file itself is read only Log ID NGAMS ER DEL FILE DB Log Number 4031 Log Type ERROR Problem s encountered deleting file info from DB Disk ID s File ID s File Version d Log Text Sten Exception s A problem or problems was were encountered while trying to delete the information for the referenced Description file from an NGAS DB Could be due to that the DB user has not permission to carry out DELETE SQL statements LogID INGAMS INFO DEL FILE Log Number MO32 EY Log Type NO Z
50. the life time of the disk varchar 64 mot null The originator of the event i e identification of the ING AMS hosting the disk ist synopsis varchar 255 Short headline indicating the type of event that occurred ime_ varchar 64 null Mime type of the data stored in the history description column text null Additional information in connection with the event This will typically be a snap shot of the NgasDiskInfo file at the time the event occurred This is e g the case when new disks are registered and when disks are removed from the system Table 13 Contents of the NGAS Disks History DB Table er za izy zg ea EN gt a e n n n n Tes ae ES TEE on On 2 2 D aj a 6 O ga 3 oO z oO 8 3 EXPERT Table ngas_files EE Tipe isk_id varchar 128 ID of the disk where the file is stored varchar 255 mot null Name of the file This must be given relative to the mount point of the disk varchar 64 mot null File ID allocated to the file by the DAPI The set of File ID Disk ID and File Version uniquely defines a file ile version default 1 Version of the file The first version is number 1 varchar 32 not null Format of the file This is generated by the DAPI Should be the mime type of the file as stored on the disk numeric 20 0 not null Size of the file This must be given in bytes If the file is compressed the compressed file size must be given as value for this column Sy o H
51. the must essential methods of this class are 1 getCfg to get the reference to the NG AMS Configuration 2 getDb to get the reference to the NG AMS DB object used to interact with the NGAS DB For more information about the ngamsServer class consult the online documentation see Section 23 2 TETS None L in ee iin mie fends of one request er chapter 23 Beware the plug should be capable of handle the situation when this is undefined None s wees ane pi be identical to that of the Python module hosting the plug in function Figure 34 Function interface of a System Online Plug In The return value of a System Online Plug In is the Physical Disk Dictionary This must be generated by the plug in Itis a standard Python dictionary with ngamsPhysDiskInfo objects stored in it The Slot IDs of the disks are used as keys in the dictionary The Disk Dictionary is very essential for the proper operation of NG AMS It is therefore crucial that the plug in extracts and generates this information correctly for NG AMS The contents of the Physical Disk Dictionary is depicted in Figure 35 VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 69 of 126 This a very essential piece of Port No n n information It is used to uniquely The La si cant ie P At identify the disk The SONPI must por on econo ler boar controlling ensure that a unique name is Slot ID 3 The Slot ID is the identification of the
52. v 1 1 2003 01 03 10 36 45 arcsw Exp gt lt Description gt This XML document contains the configuration for the jewel64 machine running as a buffering unit NCU lt Description gt lt Header gt lt Ngams ArchiveName ESO ARCHIVE PortNo 7777 SwVersion v2 0 Beta2 OnlinePlugIn ngamsLinuxOnlinePlugIn OnlinePlugInPars uri http localhost 1080 technical html module 3w xxxx OfflinePlugIn ngamsLinuxOfflinePplugIn OfflinePlugInPars unmount 0 LabelPrinterPlugIn ngamsBrotherPT9200DxPlugIn LabelPrinterPlugInPars dev dev ttySO font file opsw NGAS ngams ngamsData ngamsBrotherPT9200DxFonts fnt Replication 1 BlockSize 65536 Simulation 0 MountRootDirectory NGAS AllowArchiveReq 1 AllowRet rieveReq AllowProcessingReq AllowRemoveReq 1 ForceProxyMode 1 JanitorSuspendTime 0T00 03 00 BackLogBuffering 1 BackLogBuf ferDirectory NGAS ngams_staging gt Server LUXSRV Name ngas User ngas Passworda ex ex ERM gt lt MimeTypes gt lt MimeTypeMap MimeType image x fits Extension fits gt lt MimeTypeMap MimeType ngas nglog Extension nglog gt lt MimeTypeMap MimeType ngas paf Extension paf gt lt MimeTypeMap MimeType ngas log Extension log gt lt MimeTypeMap MimeType application x gfits Extension fits gz gt lt MimeTypeMap MimeType application x cfits Extension fits Z gt lt MimeTypeMap MimeType application x hfits Extension hfits gt lt Mi
53. 002 ERROR Problems communicating with the DB Error connection is not open State ONLINE Status FAILURE SubState IDLE Version v2 0 Beta2 2002 12 04T09 22 53 gt lt NgamsStatus gt Figure 7 Example reply when Back Log Buffering is applied The NG AMS Server has an internal thread Janitor Thread which runs periodically and tries to clean up the NG AMS environment One of the tasks performed is to archive Back Log Buffered data If such an attempt fails due to one of the reasons justifying for Back Log Buffering the data will be kept in the Back Log Buffer and a new attempt to archive it repeated later If the attempt fails for another reason the data will be moved to the Global Bad Files Area shown in Figure 3 In this case a Notification Message will be sent out to the subscribers of Error Notification Messages and the appropriate information logged in the log output targets specified see the Sections 3 3 and 3 4 In the NG AMS Configuration it can be specified if Back Log Buffering should be performed as well as the parent directory for the Back Log Buffer Directory For further information about this specific property consult Chapter 6 VLT MAN ESO 19400 2739 ESO Noms User s Manual Is 08 01 2003 27 of 126 3 8 The NG AMS Server Command Interface The NG AMS Server command interface is based on the standard HTTP protocol This makes it possible to interface to the NG AMS Server from different kinds of clie
54. 1 and Chapter 15 e Logging A quite substantial set of information can be logged according to different levels 1 On stdout 2 In the UNIX syslog and 3 In a log file see Section 3 3 e Data Consistency Checking If enabled an NG AMS Server will run a periodic data consistency check of the data stored on the disks under its control Via a number of parameters it is possible to adjust quite accurately how much load and how long time this task should take up see Section 3 9 e Production of Disk Labels NG AMS can produce labels for the disk cases on request The actual SW to operate the printer must be provided in the form of a plug in see Section 3 10 e Email Notification Service A service is provided for notifying subscribers about various events occurring during operation Examples of such events are errors disk change requests and data inconsistency reports see the Sections 3 4 and 3 5 VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual 1 5 08 01 2003 17 of 126 e Information Query A set of various types of information can be queried via the STATUS command This information is such as the state of the system or information about files and disks see Section 27 12 e Data File Retrieval amp Processing NG AMS provides a scheme for transparent access to the data Based on the information in the NGAS DB a contacted NG AMS Server can locate the data reguested by the user and provide this to the user by acting as a p
55. 2 Data Handling Plug In 67 76 Data Inconsistency Notification Message 28 Data Processing 76 Data Processing Plug In DPPI 83 Data Stream 19 DCPI 87 Debugging Trouble Shooting 24 DHPI 12 76 DHPI Example 80 DHPI Interface 78 DHPI Structure amp Algorithm 79 Disk Change Notification 25 Disk Dictionary 68 Disk Handling 21 Disk Life Cycle 21 Disk Space Monitoring 25 Disk Space Notification 25 DPPI 12 83 DPPI Example 84 DPPI Interface 83 DPPI Return Value 84 E Email Notification 24 Enable Disable Data Consistency Checking Service 28 Error Notification 24 Example Application Python API 66 Example of Local Log File 23 Example of syslog 23 Example of Verbose Log 24 EXIT Command 117 exit Python API 65 EXPERT 11 Extensible Markup Language 12 External Application amp NGAS DB 51 F Features 16 Final Filename 76 Format of NG AMS HTTP Command Messages 48 Format of the NG AMS HTTP Reply 49 Format of the NG AMS Redirection HTTP Response 49 Format of the Verbose Logs 24 G Generating Labels 117 Get Help 11 Global Bad Files Directory 19 H Handling of Local Log Files 101 HTTP protocol 27 HTTP Protocol 16 HTTP Reply 49 Idle Sub State 18 INIT Command 117 init Python API 65 Inline Python Documentation 98 Install NG AMS SW 100 Install Python 100 Installation 100 J Janitor Thread 26 L LABEL command 28 E
56. 2003 112 of 126 Log Number 4013 Log Type ERROR Pa Could not retrieve information about disk with Disk ID s Disk is probably unknown to this KA NG AMS Rejecting reguest bean NG AMS was reguested by a client to return information about a specific disk This disk however is escripit0n hot known to NG AMS Check the Disk ID or other information given with the guery and try again Log ID NGAMS ER ILL RETRIEVE REO Log Number 4014 Log Type ERROR Log Text Illegal Disk ID found s for file with ID Yos ea The Disk ID registered in connection with the file with the given ID is not know to NG AMS Check Description 5 DB for inconsistent entries and try again Log ID NGAMS_WA_BUF_DATA Log Number 4015 Log Type WARNING Problems occurred while handling file with URI s Data will be buffered and attempted archived at Log Text 5 a later stage Previous error stack s A problem occurred while handling an Archive Request The kind of error that occurred may be Description recovered later The data has therefore been buffered by NG AMS and will be attempted archived by an internal procedure at a later stage Log ID NGAMS_ER_PROB_STAGING_AREA Log Number 4016 Log Type ERROR Log Text Problem encountered while storing data in Staging Area File s Exception s A problem occurred while storing the data on the HTTP connection into the Staging Area on the target Description disk This might be caused by that th
57. 22 3 file_version lt file version gt Used to specify specific version of file to query information for configuration_file No Query the configuration used by an NG AMS server The result is a complete ING AMS Configuration XML document as shown in Section 6 3 file_access lt file ID gt No This parameter is used to make an NG AMS Server probe if it can physically access a file The body of this request must contain an NG AMS File Status XML document with the detailed information about the file It is therefore necessary to issue also the content length and content type HTTP headers followed by the file status in the request This command is thus similar to an Archive Push request refer to Section 7 1 for further information about this issue flush_log No Used to make the NG AMS Server flush the logs it may have cached internally into the Local Log File if such is specified Table 37 Parameters for the STATUS command It is only possible to specify one of the parameters at a time 27 13 SUBSCRIBE Command Subscribe to Data from NGAS Host The SUBCRIBE command is used by a Data Subscriber to subscribe to a certain kind of data becoming available and which is available at the time of the subscription The issue of Data Subscription is described in detail in Section 4 2 The parameters of the SUBSCRIBE command are listed in Table 38 filter_plug_in lt plug in gt Name of a Filter Plug In see Chapter 21 to invoke on the data
58. 2s i 3 ical E Dn Sg 2 a oO oO 5 VLT MAN ESO 19400 2739 2 F S O NG AMS User s Manual O 53 of 126 uncompressed_file_size Inumeric 20 0 not null If the file was compressed this indicates the size of the uncompressed file If the file is not compressed this will be equal to the file size compression varchar 32 null The compression method applied on the file Could be e g gzip This should indicate clearly how the file has been compressed to make it possible to decompress it at a later stage ingestion_date datetime Date the file was ingested archived ignore tinyint null Used to indicate that this file should be ignored 1 ignore If set to one this entry for this file will not be taken into account by NG AMS when files or information about files is queried checksum varchar 64 null Checksum of the file This value is generated by the checksum plug in specified in the configuration varchar 64 null IName of the checksum plug in used to generate the checksum for the file This is used by NG AMS when performing the Data Consistency Checking of data files ING AMS in this way invokes the same plug in as was used to generate the checksum originally file_status char 8 default 00000000 Current status of the file The status should be seen as a sequence of bytes each with a certain signification what concerns the condition and status of the file These bytes are used to indicate the following when set to 1
59. 33 arcsw Exp jknudstr 10 05 2001 Created Module that contains a System Online Plug In used by the ESO NGAS installations un from ngams import import ngamsPlugInApi import ngamsLinuxSystemPlugInApi ngamsEscalada6800Utils def ngamsLinuxOnlinePlugIn srvObj reqPropsObj None wen Function mounts all NGAMS disks and loads the kernel module for the IDE controller card It returns the NGAMS specific disk info dictionary srvObj Reference to instance of the NG AMS Server class ngamsServer regPropsObj NG AMS reguest properties object ngamsRegProps Returns Disk info dictionary dictionary un rootMtPr srvObj getCfg getMountRootDirectory parDic ngamsPlugInApi N parseRawPlugInPars srvObj getCfg getOnlinePlugInPars stat ngamsLinuxSystemPlugInApi insMod parDic module if Sea 0 8 VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 70 of 126 msg Kernel module parDic module loaded info 1 msg diskDic ngamsEscalada6800Utils parseHtmlinfo parDic uri rootMtPr ngamsLinuxSystemPlugInApi removeFstabEntries diskDic ngamsLinuxSystemPlugInApi ngamsMount srvObj getDb diskDic return diskDic elses errMsg Problem executing ngamsLinuxOnlinePlugIn errMsg genLog NGAMS_ER_ONLINE_PLUGIN errMsg error errMsg raise exceptions Exception errMsg name _ main iba Main function nu import sys import ngamsConfig n
60. 739 ESO Noms User s Manual Is 08 01 2003 22 of 126 3 Basic Features In this chapter the basic features from the list in Section 2 2 of NG AMS are described in detail 31 Data File Archiving Archiving of files in NG AMS is done via the ARCHIVE command Data files can be archived either by using the Archive Push Technigue or the Archive Pull Technigue Using the former the application archiving the file reads in the file and sends the contents of the file in the HTTP message body When making use of the latter technigue the client sends a URL indicating a location where the file can be picked up by NG AMS This URL must therefore be accessible for NG AMS NG AMS maintains a scheme of file versioning whereby if a file with the same File ID is issued several times the version number in the NGAS DB DB ngas_files file_version is incremented by one The first file with a given ID has version number 1 In short the handling of an Archive Request is done as follows e NG AMS receives the Archive Push or Pull Request e It determines the mime type of the data file From the configuration and the current disk status which it reads from the NGAS DB it finds a suitable Target Disk for the data e The data is subsequently received into the Staging Area of the Main Target Disk e Afterwards the DAPI corresponding to the type of data in question is invoked to do the specific handling of the data e After the DAPI has finished
61. AMS e ngas subscr back log Used by NG AMS to keep track of files that should have been delivered to a Subscriber whereby the delivery failed e ngas_subscribers Contains a persistent snap shot of the Subscribers that are subscribed to a given NG AMS Server The present release of NG AMS is based on Sybase ASE but it is foreseen to be able to make the SW work with other DBMS by using ODBC The system is capable of operating without the availability of the ngas_hosts table In addition it is possible to operate with a DB where the ngas_hosts is available but is empty This however only under certain conditions It is however recommended to create and populate this table at least when operating an NGAS Cluster Usually it is not foreseen that external applications perform queries directly into the NGAS DB I e all information needed should be retrieved via the NG AMS Server Apart from saving external applications from knowing technical details about the NGAS DB this has the advantage of making such external applications independent of the DBMS used by an NG AMS installation For this reason it is not guaranteed that 100 backwards compatibility is maintained when it comes to the format of the NGAS DB In the following sections the exact contents of the NGAS tables is described 81 EXPERT Table ngas_disks Type disk_id varchar 128 mot null The ID of the disk This information is extracted by the Online Plug In from the BIOS o
62. Afterwards the information for the Replication File is updated in the DB 9 NG AMS can either return an immediate reply to the client issuing the Archive Request or it can return a reply when the file has been successfully or unsuccessfully handled pee NGAS DB 7 DB Update Pa Main File 9 DB Update Replication File 3 DAPI Invocation ae a gt 5 DAPI Return Status 1 Archive Request A 4 Data Checking Processing Parameter Extraction Data File 2 Reception in Staging Area Target Storage Set 6 Storage of Main File in Final Location 8 Replication of File Main Disk Figure 40 Handling of an Archive Request Note that the DAPI is a function running within the same Python interpreter as the NG AMS Server process VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is i 08 01 2003 78 of 126 15 1 EXPERT Interface of a DAPI The DAPI must be contained in a Python module file which has a function of the same name as the module The latter is the actual DAPI which is invoked by NG AMS A DAPI has an interface as shown in Figure 41 aa aa F srvObj Instance of the ngamsServer class used by this def lt DAPI name gt srvobj session See also Section 12 1 regPropsObj regPropsObj Object of class ngamsRegProps Contains information in connection with the handling of one reguest See m rr M5 chapter 23 DAPI Function Name The name of the DAPI
63. DA Eno Taimer 0040185 Table 8 The supported log output formats The Log Level is a number in the range from 1 to 5 whereby 1 is the high level logs and 5 is the lowest deepest level providing the most thorough information The interpretation of the various Log Levels is as follows The lowest Log Level which only provides a brief summary of the actions performed Errors and warnings are always logged This level provides more thorough information of the actions performed 3 This level performs a quite extensive set of logs describing in details the various actions carried out by NG AMS and the plug ins invoked by this For logging in the log file there should normally not be logged with a higher level than 3 This level provides a very profound set of information It is usually only used for debugging and test purposes and for locating errors occurring in the system aa The deepest level provides a quite extensive set of logs Some of the log will be quite repetitive and logs may be produced cyclically from e g the Janitor and the Data Consistency Check Services The quantity of log information produced is quite big and if logging into a log file with this level care should be taken that it may grow in size very rapidly Table 9 Interpretation of Log Levels The level intensity with which there should be logged as well as name of Local Log File and a prefix for the syslog entries can be specified in the NG AMS Configura
64. ESO DFS Id ngamsEsoArchDppi py v 1 8 2002 11 04 12 28 47 arcsw Exp jknudstr 08 01 2002 Created Contains a DDPI which is used by the ESO Archive Facility to perform the processing in connection with a standard data request handling from ngams import import ngamsPlugInApi ngamsDppiStatus def ngamsEsoArchDppi srvObj reqPropsObj filename nu This DPPI performs the processing neccessary for the files reguested from the ESO Archive by the Data Reguestor srvObj Reference to instance of the NG AMS Server class ngamsServer regPropsObj NG AMS reguest properties object ngamsRegProps filename Name of file to process string Returns DPPI return status object ngamsDppiStatus statusObj ngamsDppiStatus ngamsDppiStatus Decompress the file if the last extension is Z ir filename Splitin aa procFilename procDir ngamsPlugInApi prepProcFile srvObj getCfg filename exitCode stdOut stdErr ngamsPlugInApi N VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 88 of 126 execCmd uncompress procFilename if exitCode 0 errMsg ngamsEsoArchDppi Problems during data handling Decompressing the file filename failed Error message str stdErr raise exceptions Exception errMsg resFilename procFilename 0 2 elses resFilename filename procDIr mi mimeType ngamsPlugInApi determineMimeType srvObj getCfg resFilename res
65. EUROPEAN SOUTHERN OBSERVATORY Organisation Europ enne pour des Recherches Astronomiques dans I H misph re Austral Europ ische Organisation fiir astronomische Forschung in der s dlichen Hemisphire Data Management Division DFS Software NG AMS Next Generation Archive Management System User s Manual Doc No VLT MAN ESO 19400 2739 Issue 2 Date 08 01 2003 DRAFT FOR INTERNAL USE ONLY Name Date Signature Prepared J Knudstrup 08 01 2003 e Name Date Signature Approved A Wicenec 12003 e Name Date Signature Released M Peron P Quinn 1 12003 ESO TELEPHONE 089 3 20 06 0 http www eso org VLT MAN ESO 19400 2739 NG AMS User s Manual 2 08 01 2003 2 of 126 CHANGE RECORD SECTION PAGE REASON INITIA TION ESE ee AFFECTED DOCUMENTS REMARKS Preperation 1 29 01 2002 1 Preparation 2 05 03 2002 Added comment after internal review 1 Preparation 1 08 01 2003 Major update with new features VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is i 08 01 2003 3 of 126 TABLE OF CONTENTS T ABOUT THIS GUIDE ose na am an aan en aa aa aan 11 1 1 Purpose amp SCOP oooocooooocoooooocoooooooooooooooooooocooooooooocoaeann atasa eissai saoirsi eh 11 12 How to Read this Manual ooooooocoooocoooooooooooooocoooooooooooooooooooooooooooco 11 13 How to Get Help or Report Problems with NG AMS or this Manual coooooooooooooooooooooc 11 1 DISCIAT MET
66. GAS Disk and wherever it appears NG AMS will recognize the disk and take the information in the NgasDiskInfo document and write this to the NGAS DB connected if the disk is not already registered in that DB This means that after having prepared the disk at the Archive Facility Site and after this has been received and installed at the Production Site the disk has now been registered in the NGAS DB at the Production Site based on the NgasDiskInfo document The Production Site NGAS System now archives data on the disk During this phase all changes in connection with the disk are replicated from the Production Site NGAS DB to the Archive Facility Site NGAS DB disk information information about new files When the disk is completed it will be marked as such by NG AMS at the Production Site and this information replicated to the Archive Facility Site The disk is subsequently send to the Archive Facility Site where NG AMS recognizes the disk and updates the information about the disk in the DB at that site In order to prevent that changes introduced from this The scenario described here explains how the NGAS DB replication is handled in connection with the ESO NGAS System VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 56 of 126 point on at the Production Site where the disk was residing are replicated to the Archive Facility Site DB the DB replication could implement some conditions indicati
67. I module and a DAPI function The exact sequence of the actions performed and the actions themselves may vary from DAPI to DAPI I e maybe the data processing is done before the generation of the final target filename In Section 15 3 an example DAPI module is shown VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 80 of 126 15 3 EXPERT Example DAPI WFI FITS File DAPI In the following an example DAPI which is used for archiving FITS files at the 2 2m telescope at La Silla is shown FRR RRR RR RRR II RRR ARR KK KK KK KK KK KK OR KK KK RRR RK RO RR RK ESO DMD Id ngamsFitsPlugIn py v 1 50 2002 07 10 08 34 33 arcsw Exp jknudstr 10 05 2001 Created This Data Archiving Plug In is used to handle reception and processing of FITS files Note that the plug in is implemented for the usage at ESO If used in other contexts a dedicated plug in matching the individual context should be implemented and NG AMS configured to use it import os exceptions string import PccUtTime from ngams import import ngamsPlugInApi ngamsDiskUtils ngamsDiskInfo def getDpIdInfo filename Generate the File ID here DP ID for the file filename Name of FITS file string Returns Tuple containing the value of ARCFILE the DP ID of the file and the JD date The two latter deducted from the ARCFILE keyword tuple nu ieee keyDic ngamsPlugInApi getFitsKeys filename ARCFILE
68. ID gt See Table 28 The Disk ID of the files to consider Table 29 Parameters for the CLONE command As a result of the CLONE command a File Cloning Status Report is send back indicating which files were cloned and the target names of these 27 3 EXIT Command Terminate Server The EXIT command is used to make the NG AMS Server exit The EXIT command does not accept any parameters 27 4 INIT Command Re Initialize the System The INIT command is used make the NG AMS Server re initialize This means that it will first go Offline load the configuration and subsequently go Online The INIT command does not accept any parameters 27 5 LABEL Command Generating Disk Labels The LABEL command is used to print out labels to be put on the disk cases The label is the Logical Name of a disk The LABEL command accepts the following parameters VLT MAN ESO 19400 2739 NG AMS User s Manual 2 08 01 2003 120 of 126 slot_id lt slot ID The ID of the slot in which the disk is installed es i i iskisi host_id lt host ID gt e host in which the disk is installed If this is not specified the local host is assumed Table 30 Parameters for the LABEL command An example of a label generated by NG AMS by means of the Label Plug In can be found in Section 3 10 27 6 OFFLINE Command Bring System to Offline State The OFFLINE command is used to make the NG AMS Server go Offline The OFFLINE command accepts the following parame
69. K KK KK EK KK KK KK KK KK KK KK KK KEKE KKR ESO DMD A Id ngamsBrotherPT9200DxPlugIn py v 1 18 2002 07 10 17 11 41 arcsw Exp awicenec jknudstr 10 05 2001 Created This module contains a plug in driver for printing labels on the Brother PT 9200DX label printer import sys time from ngams import import ngamsPlugInApi ngamsConfig def genFontsDictionary fnm Function reads the contents of a bitmap character file lt fnm gt The character contents of this file has to be compliant with the VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 74 of 126 keys Header 01 1 2 3 4 5 6 7 8 91 ei CAR Ten 0 n tpi On Tei Ten Wao tro Tg Wer tint N O TPI 19 gt Tr Tuo We OR Tet DNAN 2 at bt c dr e gt tht it 3 Met Im n o p ar ttt tut twt tw txt tyt Uva Vataeelal letra These keys are used to fill a dictionary with the bitmaps and can then be used to print strings on the Brother pTouch 9200DX printer Synopsis charDict ngamsGetCharDict lt fnm gt fnm Filename of font definition file string Returns Return value is a dictionary with the keys given above dictionary nu keys Header 0 1 2 13 TAN 15 161 7 tgr ror igh TA Tt er Opt Tan Int Teo TN ao Tag Tt Tao Tmt G0 WO Ten Tew Tiki vgn Tilt Tun Ty TR Ose OSA Uo Vet Gat tel git
70. Kandis nd sccssccsssissocasecsssecssnsosoccsscssossnscoeassvenasesesaasoensessnaseoesdssooecesonssdeedsdonmssesenssenaboooossasensesenadssssoasenecoces 12 1 5 Reference Documents ooooocooooocoooooooooooooooooooocoooooooooocooooooooooooooooooooooiooocococo 12 EE ACKONY IMS an ba B RN Ea ANN E E E AN NENEK ana nu antan 12 1 7 Glossary Se ansunanngn abuse nang BEE densedongtse BBB ES 13 2 OVERVIEW ena en masuk 15 2 1 The Concept of NGAS amp NG AMS ococooooooooooooooooooooooooooooocoocoooooooocoooccooc 15 2 2 Services amp FeatureS ocoooooooooocooooooooooocoooocoooooooooooocoooooooooooooooooooooooocooocooooooooooooooooooooooo o 16 2 3 Starting amp Stopping the NG AMS Server cooococoooooooooooooooooooooooooooooooooooooooooocooi c 17 2 4 The NG AMS Server States amp Sub States oooooooooocooooooooooooooooooooooooooooooooooooooooooooooooo 18 2 5 The NG AMS Storage Media Infrastructure oooooocoooooooooooooooooooooooooooooooooooooooooooo 18 2 6 Data Classification amp Handling oococoocooooocoooooooooooooooooooooooooooooooooo 19 2 7 Disk Handling Life Cycle of an NGAS Storage Media ooooooooocooooooooooooooooooooooooooooooo 21 3 IBASIG FEATURES 2s dana Mk laa na anang 22 3 1 Data File Archiving ooooooooooocoooooooooooooooooocoooooooooooooooooooooocooonceanntennnnnn 22 3 2 Data File Retrieval amp Processing ocoocooooocooooooooooooooooooooooooooooooooooooooooooooo
71. LastCheck LogicalName FITS M 000001 Manufacturer IBM MountPoint NGAS datal Mounted 1 NumberOfFiles 164 SlotId 1 TotalDiskWriteTime 896 20280099 Type MAGNETIC DISK ATA gt lt FileStatus Checksum 1246906309 ChecksumPlugIn ngamsGenCrc32 Compression compress f ala kake Nee 2001 A05 Ati y3 A50 Ooo ak 2D FileName saf 2001 05 08 3 TEST 2001 05 08T15 25 00 123 fits Z FileSize 53546 FileStatus 00000000 FileVersion 3 Format application x cfits Ignore 0 Ingest ionDate 2002 12 23T16 10 42 000 Tag UncompressedFileSize 69120 gt lt DiskStatus gt lt NgamsStatus gt Figure 25 Example of NG AMS HTTP response Archive Request In a reply to a Retrieve Request the data returned will be contained in the message rather than the NG AMS Status XML document shown above Such a reply thus looks like this e g HTTP 1 0 200 OK Server NGAMS v2 0 Beta2 2002 12 04T09 22 53 Content type application x cfits Expires Mon 23 Dec 2002 16 15 22 GMT Date Mon 23 Dec 2002 16 15 22 GMT Content disposition attachment filename TEST 2001 05 08T15 25 00 123 fits Z Content length 53546 data Figure 26 Example of NG AMS HTTP response Retrieve Reguest It is foreseen at a later stage to make it possible to query several files simultaneously with one query This means that the mime type multipart mixed will be used as the overall mime type of the reply and that each part has its proper mime type defined
72. MS Server maintains a State Sub State scheme to make it possible to restrict the services provided according to the condition of the server see Section 2 4 e XML Information Exchange All information sent back from the server status messages are based on XML see e g Chapter 22 e APIs for C and Python APIs for communicating with the server are provided for applications written in C and Python see the Chapters 9 and 10 e Command Line Utilities Two command line utilities for communicating with the NG AMS Server are provided These are based on the NG AMS C and Python APIs see Section 5 2 e SW Modularity Re usage The NG AMS SW is implemented as a number of classes and library functions which can be used to build dedicated servers and other applications if needed see Chapter 23 e Data File Archiving via Push Pull Technique Efficient archiving of data files is provided based on an Archive Pull Technique whereby NG AMS picks up files given by a URI and on an Archive Push Technique where the data provider writes pushes the data to the server see Section 3 1 e Canalization of Data Streams Via the configuration file it is possible to define how NG AMS should stream data onto the various Storage Disks available in an NGAS Host see Section 2 6 e Data Replication NG AMS can handle replication of data files if requested Also the information for such replicated files is updated automatically in the NGAS DB see the Section 3
73. NACCESSIBLE 3004 ERROR Disk with ID Slot ID 3 Disk ID IC35L080AVVA07 0 VNC400A4C1G8RA is not accessible writable Feb 20 12 58 04 w2p2nbu python DFSLog 2002 02 20T12 58 04 410 Error w2p2nbu NGAMS ER DISK INACCESSIBLE 3004 ERROR Disk with ID Slot ID 4 Disk ID IC35L080AVVA07 0 VNC400A4GOKZ8A is not accessible writable Feb 21 23 43 56 w2p2nbu python DFSLog 2002 02 21T23 43 56 800 Notice w2p2nbu Disk with ID IC35L080AVVA07 0 VNC400A4C1607A Name LS FitsStorage3 M 000027 Slot No 5 running low in available space 4938 MB Feb 22 09 29 40 w2p2nbu python DFSLog 2002 02 22T09 29 40 740 Notice w2p2nbu Marked Main Disk with ID IC35L080AVVA07 0 VNC400A4C1607A Name LS FitsStorage3 M 000027 Slot No 5 as completed PLEASE CHANGE Feb 22 09 29 40 w2p2nbu python DFSLog 2002 02 22T09 29 40 770 Notice w2p2nbu Marked A mechanism for this may be provided later within NG AMS Characters in bold are part of the contents VLT MAN ESO 19400 2739 NG AMS User s Manual ANN 24 of 126 Replication Disk with ID IC35L080AVVA07 0 VNC400A4C1622A Name LS FitsStorage3 R 000027 Slot No 6 as completed PLEASE CHANGE Verbose Log The Verbose Logs are written to stdout They contain more detailed information than the two other types of logs This type of log is usually used for debugging trouble shooting and test purposes The Verbose Log Level is adjusted via a command line parameter v lt level If this
74. NG AMS Server to subscribe as client to a certain kind of data Send an UNSUBSCRIBE command to the NG AMS Server to unsubscribe a client Table 20 Methods functions in the Python API VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 66 of 126 A small example application based on the NG AMS Python API is listed in the following It is used to archive a file SEK KKKKKK KEK KK KK KK KK KK KEK KK KEK KK KK E KK KK E E EEEE E E EE KK KK RRR RR RO KK KK ESO DFS Id ngamsPClientEx py v 1 2 2002 02 26 17 25 41 safcvs Exp jknudstr 26 02 2002 Created Small example application archiving a file import sys from ngams import import ngamsPClient Check the input parameters if len sys argv 4 print Correct usage is n print ngamsPClientEx lt host gt lt port gt lt file URI gt n sys exit 1 Get the parameters for handling the archiving host sys argv 1 port sys argv 2 fileUri sys argv 3 Create instance of NG AMS Python API client ngamsPClient ngamsPClient host port Execute the command status client archive fileUri Handle result here we simply print the XML status message to stdout print status genXml 0 1 1 1 toprettyxml ig SN Tau LOS 1b Figure 32 Small example program using the Python API FILE ngams ngamsPClient ngamsPClientEx This small test program will generate an output as the following on stdout while archiv
75. NSUBSCRIBE command is listed in Table 39 5 VLT MAN ESO 19400 2739 2 E S O NG AMS User s Manual R 123 of 126 Mandatory url lt delivery URL gt Yes The URL which was submitted when the client subscribed itself The Delivery URL is used by NG AMS to identify each Data Subscriber Table 39 Parameters for the UNSUBSCRIBE command If the client was not subscribed the UNSUBSCRIBE command has no effect E S O NG AMS User s Manual VLT MAN ESO 19400 2739 2 08 01 2003 124 of 126 28 Index A Alert Notification 24 ARCHIVE Command 116 Archive Pull Request Other Commands 48 Archive Pull Technique 22 116 Archive Push Request 48 Archive Push Technique 22 116 Archive Request 13 Archive Request Handling of 76 archive Python API 65 Archiving 22 Archiving Data Files 116 Back Log Buffering 13 26 Back Log Directory 19 Bad File 13 Bad Files Directory 13 Bad Files Storage Area 13 Busy Sub State 18 C C API 57 Command Interface 27 Commands 116 Communication Protocol 48 Configure 100 Configuring NG AMS 38 Configuring of Security Mechanisms 101 CPU Consumption 28 D Data Checksum Plug In 87 Data Checksum Plug In Example 87 89 90 Data Checksum Plug In Interface 87 89 90 91 Data Classification and Handling 19 Data Consistency Check Service 25 Data Consistency Checking 27 76 Data Error Notification 25 Data File Archiving 22 Data File Retrieval 2
76. Obj ngamsDppiStatus ngamsDppiResult NGAMS PROC FILE mimeType resFilename resFilename procDir statusObj addResult resObj return statusObj Figure 49 Example Data Processing Plug In FILE ngams ngamsPlugIns ngamsEsoArchDppi py VLT MAN ESO 19400 2739 ESO NGAMS User s Manual NAN 89 of 126 18 EXPERT The Data Checksum Plug In The Data Checksum Plug In is a simple plug in used to generate the checksum value for a data file being archived This value is written in the record for the file in the NGAS DB and used later on to check periodically if the file is in a good condition Le that it is not damaged or corrupted in any way The Data Checksum Plug In is invoked by NG AMS after the DAPI has finished the data type specific processing 181 EXPERT Interface of a Data Checksum Plug In The plug in must be contained in a Python module which has a function of the same name as the module The latter is the actual plug in which is invoked by NG AMS A Data Checksum Plug In has an interface as shown in Figure 50 srvObj Instance of the ngamsServer class used by this session See also Section 12 1 filename filename Name of file for which to generate the checksum def lt plug in name gt srvOobj Data Checksum Plug In Function Name The name of the plug in must be identical to that of the Python module hosting the plug in function Figure 50 Function interface of a Data Checksum Plug In
77. S Server on the super node determines that the file cannot be accessed directly since the target node is located within a private network From the ngas hosts table it finds the NGAS Main Node for that sub cluster NGAS Main Node ngas hosts cluster name and it forwards the Retrieve Request to the NGAS Main Node 3 2 This in turn figures out that the file is located on a disk hosted in a node within its private network It therefore retrieves the file via the private network 3 and 4 If processing was requested this VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 31 of 126 is carried out on the sub node Subseguently the NGAS Main Node 3 sends back the final result to the NGAS Super Node 5 The NGAS Super Node in turn returns the result of the Retrieve Reguest to the external reguestor 6 The scenario in Figure 13 shows arather complex environment The example serves mainly to illustrate the capabilities of the NG AMS SW It is probably always an advantage to have one single entry point to an NGAS Archive to make it easy for external clients to access the data In addition for security reasons it is an advantage to have only one such entry point to the archive cluster A disadvantage of this scenario is that each reguest for data will have to pass through two NG AMS Server acting as proxies before arriving to the client This of course means an extra overhead If the Suspension Wake Up Servi
78. Server in Simulation Mode The first time when the NG AMS Server is started after doing all the necessary configuring it may be convenient to start it manually in an xterm in the Verbose Mode Verbose Level 3 or 4 see also Section 5 1 This could be done in Simulation Mode to first get the basic things In the present release only Sybase is supported Other DBMS will be supported in the future by means of ODBC ESO noams Launch server in real mode Decide how to Start the Server Handling of Local Log Files Configuring of Security Mechanisms Setting Up of Multi Site DB Environment VLT MAN ESO 19400 2739 2 08 01 2003 103 of 126 User s Manual straightened out If the server encounters problems it will bail out and report these on stdout The switching on off of the Simulation Mode Normal Mode must be done in the NG AMS Configuration It could be tried to issue some commands like ARCHIVE and RETRIEVE to verify the proper functioning When the server is running properly in Simulation Mode it could be tried to switch to the Normal Mode in the configuration and try the same as described in the previous step When the server can be executed and is operating correctly it should be decided how it should be started Under normal circumstances it should be started when the host on which it is running is booting and run as a daemon I e the start up scripts on the host should be configured accordingly If a
79. StorageSet Awan MimeType Mime type identifying this type of data PlugIn A plug in command that will be executed to 1 Check the consistency of the data file of the given mime type 2 Generate the necessary information for the NGAS DB PlugInPars Parameters which will be transferred to the plug in function Z lt ELEMENT Stream StorageSetRef gt lt ATTLIST Stream MimeType CDATA PlugIn CDATA PlugInPars CDATA lt ELEMENT StorageSetRef EMPTY gt lt ATTLIST StorageSetRef StorageSetId CDATA REQUIRED gt REQUIRED REQUIRED IMPLIED gt E S O NG AMS User s Manual The Processing Element defines the set of DPPIs that are available on an NGAS system The element contains a number of PlugIn Elements which each define the properties of the DPPIs In addition the PlugIn Element defines which types of data can be handled a lt ELEMENT Processing PlugIn gt lt ELEMENT PlugIn MimeType gt lt ATTLIST PlugIn Name PlugInPars lt ELEMENT MimeType EMPTY gt lt ATTLIST MimeType Name CDATA CDATA REQUIRED REQUIRED gt CDATA REQUIRED gt ole The Register Element defines the of DRPIs needed for registering various file types gt lt ELEMENT Register PlugIn gt lt a The Monitor Element defines properties for ingestion handling Sr data Tries Attributes MinFreeSpaceWarning Indicates the free disk space limit before issuing an Warning Log Message MB FreeSpaceDiskChangeMb
80. Tel Wel Ug Wal Tao 840 Tee ai Im n ot p qt txt st TE tut twt twl txt ty Wye geni VI f open fnm charArr f read f close except exceptions Exception e Gror ae tE errMsg Problems opening CharDict file str e raise exceptions Exception errMsg charArr charArr split ZG charDict i 0 if len charArr len keys errMsg Wrong number of characters in CharDict file error str e raise exceptions Exception errMsg k in keys if k Header or k Trailer charDict update k charArr i else charDict update k G charArr i put the G back CharDicl update Y V 2227222272220 J add a blank gi tab tar ah return charDict ngamsBrotherPT9200DxPlugIn srvObj label Driver for printing labels on the label printer Brother PT 9200DX srvObj Reference to instance of the NG AMS Server class ngamsServer label Label text to print string Returns Void wen plugInPars srvObj getCfg getLabelPrinterPlugInPars info 2 Executing plug in ngamsBrotherPT9200DxPlugIn with parameters plugin bass Nita be 1 ab Nan 5 parDic ngamsPlugInApi parseRawPlugInPars plugInPars Get the font bit pattern dictionary fontDic genFontsDictionary parDic font file Generate the printer control code printerCode fontDic Header for i in range len label if not fontDic has_key label i errMsg No font defintion for charac
81. UX and on which a file system can be created can become an NGAS Storage Media Storing data on HDDs has several advantages over the present scheme used e g by ESO whereby CD ROMs and DVDs are used to store the data Some advantages are e The archiving of data files can be carried out very fast e Data is online as soon as it has been archived e It is not necessary to store data in an intermediate location and to generate later the final media The final media is create real time e The processing power of the computers hosting the disks can be used to process the data both during archiving and while retrieving data e In general an archive system based on NGAS requires much less manual intervention than existing systems based on CD ROM DVD disks or tapes e The price per storage unit is relatively low compared to other solutions The NGAS is based on standard PC HW running Linux They are normally equipped with a set of HDD sliders in which HDDs can be inserted and removed easily It is foreseen to have one or more NGAS Hosts at the telescope sites one in VLT MAN ESO 19400 2739 ESO Noms User s Manual Is 5 08 01 2003 16 of 126 connection with each major producer of data As soon as a disk is full it will be send to the Archive Facility Site where it will be installed in a free slot in an NGAS Host in the archive NGAS Cluster The data is immediately online as soon as the NG AMS has recognized the disk NG AMS c
82. a Main Disk but is now installed Log Text eae in a Replication Disk slot When a disk has been registered by the system as being a Main Disk it should not be attempted to use Description it later as a Replication Disk The way to recover from this problem is to install the disk in a Main Disk Slot together with the Replication Disk with which it was originally registered Log ID INGAMS_ER_REP_DISK_WRONGLY_USED Log Number 3001 Log Type ERROR Disk in slot s with Logical Name Yos has previously been registered as a Replication Disk but is Log Text now installed in a Main Disk slot When a disk has been registered by the system as being a Replication Disk it should not be attempted Description to use it later as a Main Disk The way to recover from this problem is to install the disk in a Replication Disk Slot together with the Main Disk with which it was originally registered VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 107 of 126 Log ID INGAMS_ER_DISK_INACCESSIBLE Log Number 3004 Log Type ERROR Log Text Disk with ID s is not accessible writable NG AMS probes for each disk installed and configured for an archiving system if it is possible to write Pesanan on the disk If not an error is returned Log ID INGAMS ER NO TARG DISKS Log Number 8005 Log Type ERROR Log Text No target disks found for the Stream s with mime type s s
83. account Le all nodes are accessible externally In addition the file access is no longer transparent since the client has to support the re direct scheme defined by the HTTP protocol Using the C or Python APIs or command utilities however this is handled transparently for the client see the Chapters 9 and 10 and Section 5 1 As a last example scenario an architecture similar to the one in Figure 14 could be used whereby there is one NGAS Main Node acting as main entry point All nodes however in the cluster are connected to the switch via a private network which is not accessible externally This topology has the advantage of still having only one contact point for external clients At the same time access to the data is handled transparently as the NGAS Main Node will always act as proxy on behalf of VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 32 of 126 the client There is only one intermediate copy created due to the proxy mode using this architecture as opposed to two proxy copies in the scenario in Figure 13 The processing is carried out on the individual NGAS Nodes Many other architectures can be designed using the NG AMS Server and NGAS Nodes in various configurations The three architectures discussed in this section merely serve to give an impression of what possibilities are provided 42 EXPERT Data Subscription Service The Data Subscription Service of NG AMS makes it possible to sy
84. ailable in SYBASE lib Install Python Check Existing The present version of Python required for NG AMS is 2 1 2 1 1 Check that the proper Python Installation version is installed If the wrong version is installed or if Python is not installed at all it should be downloaded from http www python org and installed according to the instructions Check in particular that the Sybase Python module is available if Sybase is used Get the NG AMS Python SW Get the sources of the NG AMS SW This can be requested by contacting ngast eso org Install NG AMS SW Configure Install the sources simply by copying the NG AMS root module directory ngams to a path the Environment contained within the PYTHON PATH list of paths or add the new location of ngams in the PY THON_PATH variable The NG AMS C API should also be compiled and installed This is done by entering in the directory ngams ngamsCClient and typing make clean all The binary ngamsCClient should be installed in a bin directory for global access The ngams h and libngams a files should be copied into an area which is globally accessible if needed for application development It could also be chosen to make the NGAMS Server source file ngams ngamsServer ngamsServer py executable and globally accessible The same goes for the NG AMS Python API ngams ngamsPClient ngamsPClhient py Prepare the NGAS DB in the Sybase DB server A user to be used by the NG AMS when
85. amsLinuxOfflinePlugIn py 72 Figure 38 Function interface of a Label Printer Plug In oocooocooooooooooooooooooooooo 73 Figure 39 Example Label Printer Plug In FILE ngams ngamsPlugIns ngamsBrotherPT9200DxPlugIn py 75 Figure 40 Handling of an Archive Request ecooocooooooooooooooooooooooooooooooooooooooooooooooooooooo 77 Figure 41 Function interface of a DAPI ocoocooooooooooooooooooooooooooooooooooooooooooco 78 Figure 42 DAPI return statement Figure 43 Typical structure of a DAPI module and a DAPI function ooooocooooooooooooooooo 79 Figure 44 Example Data Archiving Plug In FILE ngams ngamsPlugIns ngamsFitsPlugIn py oooooooooooo 82 Figure 45 Example Register Plug In FILE ngams ngamsPlugIns ngamsFitsRegPlugIn py oooooooooooooooo 84 Figure 46 Function interface of a DPPL ocoocooooocoooooooooooooooooooooooooooooooooooooooooooco 85 Figure 47 DPPI structure of return data oooooooooooooooooooooooooooooooooocoooooooooooooooooooo 86 Figure 48 Example Data Processing Plug In FILE ngams ngamsPlugIns ngamsExtractFitsHdrDppi py 87 Figure 49 Example Data Processing Plug In FILE ngams ngamsPlugIns ngamsEsoArchDppi py woo 88 Figure 50 Function interface of a Data Checksum Plug In DCPI ooooooocoocooooooooooooooooooooooooooooo 89 Figure 51 Example Data Checksum Plug In FILE ngams ngam
86. an produce a Replication Disk so that a back up of the data is available For more information about NGAS check out 2 and the links found at this site Although the system has been developed on UNIX it may be possible to port the SW relatively easily to other platforms supporting Python like e g MS Windows or MAC OS The philosophy behind the NG AMS SW is to provide an open architecture that can be extended and adapted to be used as a generic archive facility in many different contexts Therefore the NG AMS SW in itself does not have any specific functionality built in to handle specific types of data or specific HW All this knowledge must be implemented and made available for an NG AMS Server in order to make it carry out the requested tasks This is done by providing a lot of specific services in the form of plug ins which are simple Python functions Due to this scheme it should be possible to adapt NG AMS with a minimum amount of effort to handle e g many different kind of data The heart of the NG AMS is the NG AMS Server This is a multithreaded server based on the standard HTTP protocol It can be seen as a dedicated WEB server Since the server is multithreaded it is possible to issue several requests simultaneously A number of commands are provided by NG AMS For more detailed information about these commands consult Chapter 25 For more information about the technical details of the command interface consult Chapter 7 2 2 Service
87. anual Ae 73 of 126 14 EXPERT The Label Printer Plug in The purpose of the Label Printer Plug In is to print a label on request from NG AMS on the label printer installed on the NGAS Host The plug in must generate the appropriate control sequence of characters in order to request the printer to produce the label Also other actions needed to control the printer should be taken care of by the plug in Le the plug in could be seen as a high level intelligent printer driver 141 EXPERT Interface of a Label Printer Plug In A Label Printer Plug In must be contained in a Python module file which has a function of the same name as the module The latter is the actual plug in which is invoked by NG AMS A Label Printer Plug In has an interface as shown in Figure 38 srvObj Instance of the ngamsServer class used by this session See also Section 12 1 regPropsObj None chapter 23 Beware the plug should be capable of handle the situation when this is undefined None the function Figure 38 Function interface of a Label Printer Plug In A Label Printer Plug In does not return any data to NG AMS An exception must be thrown in case errors occur during the printing process 142 EXPERT Example of a Label Printer Plug In In the following the source code of an example is shown This is used to control a Brother label printer Brother P Touch 9200 DX SEK KKKKKK KK KK KK KEK KK KK KK KK KK KK KK KK KEKE KK KK KK KK K
88. apter 6 3 7 Back Log Buffering Back Log Buffering is used to temporarily buffer data which for some reason not necessarily related to the quality of the data prevents NG AMS from performing a proper archiving of the data file An example of such an event is e g if the DB connection is lost temporarily As shown in Figure 3 the Back Log Buffer Area could be located in the NG AMS Root Mount Directory as it is practical to collect the data of NG AMS under a single point If a problem occurs during the handling of an Archive Request a file with a unique name will be created in this area and the data of the request buffered in this file The reply to the Archive Request will indicate the problem i e that the data was Back Log Buffered No further actions are needed from the client that issued the Archive Request Figure 7 shows an example of a reply from NG AMS when back Log Buffering was done ngasmgr acngast1 opsw NGAS ngams ngamsData gt ngamsCClient port 7777 host acngast1l status cmd ARCHIVE fileUri tmp SmallFile fits lt xml version 1 0 gt lt DOCTYPE NgamsStatus SYSTEM http acngast1 hq eso org 7777 RETRIEVE internal ngamsStatus dtd gt lt NgamsStatus gt Status Date 2003 01 08T16 44 40 562 HostId acngast1 Message NGAMS_WA_BUF_DATA 4015 WARNING Problems occurred while handling file with URI SmallFile fits Data will be buffered and attempted archived at a later stage Previous error stack NGAMS_ER_DB_COM 2
89. at contains a sub result from a DPPI execution ngamsDppiStatus Class that contains the resulting data from a DPPI execution The module provides the class ngamsFileInfo which is used to handle all the information in connection with a file which has been archived in an NGAS Host It is possible to generate an XML document from the contents of the object Used to manage list of file information objects ngamsFileInfo e g to dump the information into XML documents Contains various basic convenience functions used throughout the NG AMS SW ngamsFileList ngamsPhysDiskInfo py Provides the class ngamsPhysDiskInfo which is used to manage the physical information about a disk extracted by the System Online Plug In see Chapter 12 Modules that provides various utility functions to be used for implementing plug ins It is recommended only to use the functions contained in this module for implementing the plug in apart from varios classes like ngamServer ngamsDb and ngamsConfig IngamsReqProps py Module that provides the object ngamsReqProps which is used to keep a record of actions carried out during the handling of a request IngamsStatus py Provides the class ngamsStatus which is used to handle the information in connection with a status generated for NG AMS Used to handle information about one Subscriber Modules that provides a small class ngamsURLopener which is used to access URLs in a transparent manner
90. be woken up by external intervention This means that either the host must be switched on manually or the server must request to receive a wake up call from another NGAS Host An NGAS Host suspending itself signals by which other NGAS Host it would like to be woken up CFG NgamsCfg HostSuspension WakeUpServerHost This means that in an NGAS Cluster where host suspension is used one host should be kept switched on with an NG AMS Server running in Online State An NG AMS Server suspending itself will calculate when it should be woken up at latest This is determined by the time for scheduling the next data checking batch if Data Consistency Checking is active A suspended server will also be woken up if a request for data located on the suspended host is received In order for this to work all Retrieve Requests must pass through one node in the NGAS Cluster main node which is never suspended The main node will identify that the requested data is stored on a suspended host and will wake up this node as described above Handling a Retrieve Request of data stored on a suspended host may therefore take some time depending on how long time it takes the host to become operational Online A proper time out must therefore be applied when retrieving data from an NGAS Cluster where host suspension is used Once the suspended server is Online requests will be handled rapidly until it is suspended again after the specified period o
91. bly be archived again LogID NGAMS_ER_ARCHIVE PULL REG o Log Number 4004 Log Type ERROR 0 Log Text Problems occurred handling Archive Pull Request URI s 000 A problem occurred while handling an Archive Pull Request The archive request has not been carried Description out successfully It should be investigated what caused the problem and the file should possibly be archived again Log ID INGAMS ER MISSING URI Log Number 14005 Log Type ERROR Log Text Must specify a URI for the data file to archive When issuing an archive reguest a URI must always be specified This is used as reference to the file Basa and in the case of an Archive Pull Reguest the URI is used to actually pick up the file VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Isu 08 01 2003 111 of 126 Log ID INGAMS ER UNKNOWN MIME TYPEI Log Number 4006 Log Type ERROR Log Text Could not determine mime type for data file with URI s Check NG AMS Configuration From the extension of the file URI given the mime type could not be determined i e is not among Description the mime types defined in the configuration file Check the configuration file to see if support for this type of data file should be added Log ID NGAMS_ER_IMPROPER_STATE Log Number 4007 Log Type ERROR Log Text s not allowed when in State Sub State s s Allowed State s Sub State s are s s The request issued canno
92. ce is used see Section 4 3 it is important that each suspended host is accessible by one other NGAS Host which is never suspended and therefore can be reguested to wake up such a suspended host In the example in Figure 13 the sub nodes could be suspended whereas the main nodes will have to be kept running with an NG AMS Server in Online State running on them A more simple example of an NGAS Cluster is shown in Figure 14 Retrieve Reguest 1 NGAS Main Node Network Switch Figure 14 Example of a simple NGAS Cluster The architecture in Figure 14 is based on a flat structure providing external access to the individual node in the cluster The client has still one contact point as in the scenario in Figure 13 namely the NGAS Main Node and it sends all Retrieve Requests to this node 1 In this case however the NGAS Main Node does not act as proxy and after identifying on which NGAS Node the file is located it returns an HTTP Redirection Response to the requestor 2 The client now issues the same Retrieve Request directly to the NGAS Node where the file is located 3 The NG AMS Server on that host handles the request and possibly processes the file and sends this back directly to the requestor 4 It goes without saying that the structure shown in Figure 14 makes the handling of requests far more efficient compared to the structure used in Figure 13 There is however still the issue of security to take into
93. ceDiskChangeMb 300 gt lt Log SysLog 1 SysLogPrefix DFSLog LocalLogFile NGAS ngams_staging log LogFile nglog LocalLogLevel 1 LogBufferSize 100 gt lt Notification SmtpHost smtphost hq eso org Sender ngast eso org INCE AL SSO lt AlertNotification gt lt EmailRecipient Address jknudstr eso lt AlertNotification gt lt ErrorNotification gt lt EmailRecipient Address jknudstr eso lt ErrorNotification gt lt DiskSpaceNotification gt lt EmailRecipient Address jknudstr eso lt DiskSpaceNotification gt lt DiskChangeNotification gt lt EmailRecipient Address jknudstr eso lt DiskChangeNotification gt lt NoDiskSpaceNotification gt lt EmailRecipient Address jknudstr eso lt NoDiskSpaceNotification gt lt DataErrorNotification gt lt EmailRecipient Address jknudstr eso lt DataErrorNotification gt lt Notification gt lt HostSuspension IdleSuspension 0 IdleSuspensionTime 10 WakeUpServerHost dmdarcl SuspensionPlugIn ngamsTestSuspensionPlugIn WakeUpPlugIn ngamsTestWakeUpPlugIn WakeUpCallTimeOut 10 gt lt SubscriptionDef AutoUnsubscribe 1 SuspensionTim OOO MOS OUT BackLogExpTime 28T00 00 00 Enable 0 gt lt SubscriptionDef gt lt NgamsCfg gt S5 doro a Figure 19 Example NG AMS Configuration file FILE ngams ngamsData ngamsServer xml VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 48 of 126 7
94. contained in the DB An example of such a file can be found in Section 22 2 2 6 Data Classification amp Handling One of the fundamental concepts behind NG AMS is the way data is classified and handled This is based on the same concept as used by many WEB browsers and mail tools namely on the mime type of the data which again is derived from the extension of the data files It is also possible to explicitly specify a mime type for a data file when issuing it for archiving In NG AMS no mime types for the data files handled are hard coded into the SW By means of the NG AMS Configuration mime types for new types of data files to be handled can be added Note that for new types of data a corresponding DAPI must be provided see Chapter 15 If NG AMS encounters a data file with an unknown mime type not defined in the configuration while handling an Archive Request the request will be rejected Such data files will not be buffered on the NGAS Host handling the request It is also possible to define an arbitrary number of Data Streams normally one per each type of data to be handled In the data stream the following information must be defined 1 VLT MAN ESO 19400 2739 2 NG AMS User s Manual Ana Pada 20 of 126 ime Type The mime type indicating for which data the stream is defined DAPI Parameters The DAPI that should be used to handle the processing and archiving of the data file In addition parameters for the DAPI can be
95. d be used for this FUNCTION ngams ngamsPlugInApi genDapiSuccessStat After the DAPI has finished execution NG AMS will move the processed file to its final destination which was decided by the DAPI Also the NGAS DB is updated by NG AMS with the information about the new file If replication is requested the file is replicated and the DB updated also with the information for the Replication File The DAPI is only concerned with the Main File If a Replication File should be produced this is entirely handled by NG AMS The DAPI can indicate to NG AMS that no replication should be carried out by setting the flag ngamsRegProps setNoReplication 1 Note that if no Replication Disk is specified in the Storage Set no replication is performed automatically If replication is switched off via the configuration file by the DAPI or if no Replication Disk is specified in the configuration the information about the Replication File is not updated in the DB File Versioning can be switched off by the client issuing the file for archiving using the parameter no_versioning 1 see 27 1 The DAPI must use the value of ngamsRegProps getNoVersioning to check if File Versioning is active This is handled automatically by the convenience function ngamsPluglInApi genFileInfo and the DAPI does not need to worry about this The diagram in Figure 40 shows the actions carried out by NG AMS and the DAPI while handling an Archive Requ
96. e It is possible to operate the NG AMS Server in Simulation Mode whereby a number of features are disabled or are working slightly different than in Normal Mode One of the major differences is that it is possible to run without the availability of real storage disks Simulated storage disks are created as directories in the Mount Root Point These are of the format lt mount root point gt lt storage set ID gt Main Rep lt simulation slot ID gt VLT MAN ESO 19400 2739 ESO Noms User s Manual Is 08 01 2003 26 of 126 Another difference compared to running in real mode is that the Online and Offline Plug Ins are not executed since no disks need to be mounted or unmounted The disk information about the disks is generated simulated and written in the DB For the clients of NG AMS there is no visible difference between running in Normal Mode or in Simulation Mode Also the internal aspects are the same so that e g the DB is updated in the same manner in Simulation Mode as in Normal Mode The Simulation Mode can be quite useful for developing and testing e g DAPIs and DPPIs It should be mentioned that it is possible to have a fully operational NG AMS installation running in Simulation Mode on a normal workstation archiving and retrieving data to from one of the system disks of the workstation To enable disable the Simulation Mode the attribute NgamsCfg Ngams Simulation in the configuration is used See also Ch
97. e 11 NG AMS High Level Services that can be enabled disabled See also Section 6 2 Ngams Element Apart from disabling handling of REMFILE REMDISK commands Table 11 it might be advisable to implement additional schemes for preventing data from being deleted from an archive data server NGAS system This could e g be done within the System Online Plug In see Chapter 12 One of the responsibilities of this plug in is to mount the Storage Media available in the NGAS Node In case a Storage Media is marked as completed in the NGAS DB the media could be mounted read only to prevent data from being accidentally removed NG AMS does not provide any features to prevent data from being as such and it is up to the designers of the NGAS environment to define how to provide this VLT MAN ESO 19400 2739 ESO Noms User s Manual Is 08 01 2003 30 of 126 4 EXPERT Advanced Features In this chapter the advanced features of NG AMS listed in Section 2 2 are described in detail 41 EXPERT Operation in Cluster Mode For larger data holdings it will normally be necessary to have a maybe large number of NGAS Nodes to provide online access to the data in an Archive Facility It is therefore important to make a proper design of the architecture of such a Archive Facility Cluster NG AMS provides a few services to support such operation in cluster mode Among this is the capability of NG AMS to act a proxy while handling a Re
98. e Staging Area directory is inaccessible or the target disk as such or that that there is no more space in the Staging Area LogID NGAMS_ER PROB BACKLOG BUF gt Log Number 407 ILog Type ERROR O Z OSOS Log Text Problem encountered while storing data of file with URI s in Back Log Buffer s Exception s A problem occurred while storing data in the Back Log Buffer Area This might be caused by that the Description Back Log Buffer directory is inaccessible or the target disk as such or that that there is no more space in the Staging Area LlogID NGAMS_NOTICE_FILE_REINGESTED gt O Log Number MoI Log Type NOTICE o o Log Text Note File issued with URI s was already archived and has been re ingested PAn The file issued was already archived and has been re ingested into NGAS This means in practice that P an additional copy of the file is available in the NGAS data repository VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Isu 08 01 2003 113 of 126 Log ID INGAMS_INFO_ARCHIVING_FILE Log Number 4019 Log Type INFO Log Text Archiving file with URL s NG AMS is about to handle archive the file referenced in the log message An additional message additional messages will follow this one indicating if the archiving was successful or not Description Log ID NGAMS_INFO_FILE_ARCHIVED Log Number 4020 Log Type INFO Log Text Successfully archived
99. e type parameter es mime_type lt mime type gt No If the File URI of an Archive Request does not reveal the mime type of the file to be archived the mime type should be specified in the Archive Request This makes the handling of the request more efficient An example using this parameter is given in Example 3 below No Used to switch the automatic versioning on off If File Versioning is on a file archived with a File ID already registered in the NGAS DB will get a new version number previous number 1 No With this parameter it is possible to specify if the NG AMS Server should send back an immediate reply wait 0 when handling an Archive Request or if a reply should be sent after the request has been handled wait 1 In the former case the client will not know if the Archive Request was handled successfully The default behavior of the server is to send the reply after the Archive Request has been handled Table 27 Parameters for the ARCHIVE command Example 1 Archiving using Archive Push Technique An example of an Archive Push Request can be found in Section 7 1 Figure 20 Example 2 Archiving using Archive Pull Technique The URL for the NG AMS Server could be something like this http hostx 7878 A R CHIVE filename ftp hosty data 2002 02 11 Fits1 fits In this case the NG AMS Server will pick up the file from the location given Le the client need not to issue the data in the HTTP request In the exam
100. eated KKK KKK KKK KK KKK KK KKK KKK KK KK KK KKK KKK KKK KK KKK KKK KKK KKK KKK KKK hehehehe kk kk kk ka ngamsCfgNau dtd defines the contents and lay out of the configuration file loaded by the NG AMS Server at start up Consult the DTD ngamsInternal dtd for further information It contains the actual definition of the elements of the NG AMS Configuration gt The NgamsCfg element is the root element of the NG AMS Configuration for NG AMS NAU Systems gt lt ELEMENT NgamsCfg Header Ngams Db MimeTypes StorageSet Stream Processing Register FileHandling Monitor Log Notification HostSuspension gt lt Ole gt Figure 17 NG AMS Configuration DTD FILE ngams ngamsData ngamsCfg dtd 6 2 EXPERT NG AMS Base DTD ngamsInternal dtd The base DTD is used to define various XML elements which can be re used in various deducted DTD XML documents The contents is as follows lt xml version 1 0 encoding UTF 8 gt aa E S O who When What KKK KK KK RRR RR RRR RR RK RK KKK KK KK KK KK KK KK KKK KEKE kk KEKE jknudstr 04 04 2001 Created Te RR RR RR RR RR RK RK RR kk kk kk RK EK KR KKK kk kk KEKE kk kk kk The ngamsInternal dtd defines various common elements to be used in the NG AMS XML documents gt VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 39 of 126 El The Ngams Element defines properties for the individual installation of NG AMS A
101. ed to run an NG AMS on a normal workstation for archiving data in a production system see Section 3 6 Thorough Documentation Apart from this manual thorough and accurate documentation contained in the Python source code of NG AMS is provided This makes it possible to browse the documentation online e g using pydoc see Section 23 2 The services and features listed above and described shortly are explained in more detail in this and the following chapters 2 3 Starting amp Stopping the NG AMS Server The NG AMS Server can be invoked with a number of different command line parameters These are described in Section 5 1 It is mandatory to specify an NG AMS Configuration to be used by the NG AMS session How to configure the NG AMS environment is described in Chapter 6 The server can be started with the v option to produce output on stdout Normally in a production environment it will be started as a back ground process which only produces log output to the UNIX syslog and or a Local Log File see also Section 3 3 The server can be stopped either by sending a SIGTERM signal 15 or by sending an EXIT command which can be issued when the server is in Offline State see also 2 4 If the server is killed with a SIGTERM signal it will invoke internally a signal handler that cleans up the environment and shuts it down in a proper manner whereby also the System Offline Plug In Chapter 13 is invoked Als
102. ency Checking Report oocoooooooooooooooooooooooooooooooooooo 28 Figure 11 Example of a Data Consistency Checking Status Log oooooocoocoocooooooooooooooooooocoo 28 Figure 12 Example Disk Label as generated by NG AMG o cooooco ooooooooooooooooooocooooooooooooooooooo 28 Figure 13 Example of hierarchical NGAS Cluster Figure 14 Example of a simple NGAS Cluster Figure 15 The NG AMS Server online help output as written on StdOut ooooooooooooooooooooooooooooooooo 34 Figure 16 The NG AMS C Client and Python Client online help output on stdout ooooooooooooooooooooo 37 Figure 17 NG AMS Configuration DTD FILE ngams ngamsData ngamsCfg dtd oooooooooooooooooo 38 Figure 18 NG AMS Configuration generic DTD FILE ngams ngamsData ngamslnternal dtd ooooooo 44 Figure 19 Example NG AMS Configuration file FILE ngams ngamsData ngamsServer xml ooooooooooooooo 47 Figure 20 Format of an Archive Push HTTP request ocoocoooooooooooooooooooooooooooooooooooooooooooo 48 Figure 21 Example of Archive Push HTTP request Figure 22 Structure of NG AMS GET method HTTP request coooooocoooocoooooooooooooooooooooooooooooo 48 Figure 23 Example of NG AMS GET method HTTP request Archive Pull Reguest ooooooooooooooooooo 48 Figure 24 Format of NG AMS HTTP response ooooooooooocooooooo
103. eport is generated indicating which files where registered 27 9 REMDISK Command Remove Information about Disks The REMDISK command is used to remove information about entire disks from NGAS Great caution should therefore be applied when using this command Both the information about the Storage Media and the files stored on will be removed NG AMS will not accept to remove a file from the system unless there are at least three 3 independent copies of the file Three independent copies refers to three copies of the file stored on three independent Storage Media In order for the REMDISK command to be accepted the system must be configured to allow remove requests CFG NgamsCfg Ngams AllowRemoveReg If the command is executed without the execute parameter the information about the disk is not deleted but a report is generated indicating what will be deleted if the execution is requested execute 1 The REMDISK command takes the following input parameters disk_id lt disk ID gt Yes Disk ID for the disk to remove execute 011 No If execute is not specified or specified as 0 no information will be deleted but a report will be send back to indicate what will be deleted if the command is executed If execute is specified as 1 the information in the DB and on the disk itself is deleted Table 33 Parameters for the REMDISK command VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Isu 5 08 01 2003 121 of 126
104. er 6 3 10 Label Printing A label to stick on the disk cases can be produced by NG AMS by means of the LABEL command The text on the label is the Logical Name allocated to the disk In addition printed on the label is the host ID and the Slot ID An example of a label is as follows generated by the Brother P Touch 9200 DX label printer LS FitsStorage4 R 000028 jewel64 7 Figure 12 Example Disk Label as generated by NG AMS The part with the Host ID Slot ID should be removed from the label before sticking it on to the disk case The LABEL command takes as input the Slot ID for the disk in question In addition the Host ID of the host in which the disk is installed The label is printed by the Label Printer Plug In see Chapter 14 311 Security It is important to keep in mind that the NG AMS SW does not come with any security mechanisms built in when it comes to preventing undesirable intruders hackers from connecting to the server and invoking services This must all be handled VLT MAN ESO 19400 2739 ESO Noms User s Manual Is 08 01 2003 29 of 126 at the level of the operating system and network firewalls etc What is supported are checks to disable certain services and to ensure that only a limited set of plug ins can be invoked by clients The high level NG AMS services that can be enabled disabled via the NG AMS Configuration are It is possible to enable disable acceptance of Archive Requests This i
105. ery Several Files Simultaneously 49 Query State 18 R Redirection HTTP Response Example 50 Redirection Response 50 Re Initializing 117 Remote Location 22 Replication 76 Replication Data File 14 Reply Archive Reguest Example 49 Reply Retrieve Reguest Example 49 Report Problems 11 Retrieval 22 RETRIEVE Command 118 119 retrieve2File Python API 65 Retrieving 119 Retrieving and Processing Files 119 Return Value System Online Plug 68 S Security 28 Services 16 Simulation Mode 25 Stages in life cycle NGAS disks 21 Staging Area 14 Standard DHPI Return Value 76 Starting the NG AMS Server 17 States amp Sub States 18 STATUS Command 120 Status DTD 93 Status XML Document 93 status Python API 65 VLT MAN ESO 19400 2739 2 08 01 2003 126 of 126 Stopping the NG AMS Server 17 Storage Set 14 Sub States 18 Sybase DB 100 Syslog 23 System Offline Plug In 71 System Offline Plug In Example 71 System Offline Plug In Interface 71 System Online 118 System Online Plug In 68 System Online Plug In Example 69 System Online Plug In Interface 68 System Status 120 T telnet 27 Terminating Server 117 U UNIX Syslog 23 Utilities 34 V Verbose Log 24 Verbose Log Level 24 W Windows 16 X XML 12
106. es parDic mime_types split Perform the matching actMimeType ngamsPlugInApi determineMimeType srvObj getCfg filename for mt in refMimeTypes if actMimeType mt strip match 1 return match Figure 57 Example Filter Plug In FILE ngams ngamsPlugIns ngamsMimeTypeFilterPL py VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 95 of 126 22 The NG AMS Status XML Document The NG AMS Status Document is used in various contexts either as the complete status or as partial status for a specific context For instance in the reply of most commands a small status is given indicating if the command was executed successfully and in case not indicating the error that occurred 22 1 EXPERT NG AMS Status DTD ngamsStatus dtd The NG AMS Status document is based on the NG AMS base DTD described in Section 6 2 lt xml version 1 0 encoding UTF 8 gt lt ENTITY XmlStd SYSTEM http www eso org projects esoxml XmlStd dtd XmlStd lt ENTITY NgamsInternal SYSTEM ngamsInternal dtd sNgams Internal al E S O Who When What KKK KK KK OR RR RR RR RR RK RK KR RE KR KE KK EK KK KK kk KEKE kk kakak jknudstr 04 04 2001 Created FOI III III KK KK KK KEK KEKE KEKE EK E E E e E E IEE IE K E ICICI k k e ngamsStatus dtd defines the contents and lay out of the NG AMS Status Report Consult the DID ngamsInternal dtd which contains the actual definition of the eleme
107. est Only the main actions are shown in the figure Behind the scenes a number of other tasks are performed in order to archive a file properly As seen in Figure 40 the handling of an Archive Request is initiated by a data provider sending an Archive Pull or an Archive Push Request to the NG AMS Server 1 NG AMS determines the type of data mime type and from this the Target Storage Set is determined Subsequently the data is received into the Staging Area on the Target Main Disk 2 Subsequently the DAPI is invoked 3 which does the necessary data consistency checking processing and extraction of 7 If the mime type is not specified explicitly in the Archive Request NG AMS will attempt to determine the mime type from the extension specified in the URI of the data file issued for archiving In this case the value of the mime type should be set to the generic mime type ngas archive request to signal to NG AMS to figure out the mime type VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual 1 5 08 01 2003 77 of 126 information from the file 4 The DAPI returns control to NG AMS and delivers back a set of information needed by NG AMS for the further processing of the file 5 NG AMS stores the Main File in its final location on the Main Disk 6 Then the information about the new Main File is updated in the NGAS DB 7 If replication is enabled and a Replication Disk is defined NG AMS creates the Replication File 8
108. eter gt lt value gt The parameters and values are returned in a dictionary whereby the keys of the dictionary are the parameter names jparseRegPlugInPars Get the parameters for a Register Plug In referred to by its mime type from the configuration parse these and return them in a dictionary prepProcFile The function is used to create a copy of a file to be processed in the Processing Directory Table 21 Functions in the NG AMS Plug In API Examples of plug in can be found in the Chapters 12 System Online Plug In 13 System Offline Plug In 14 The Label Printer Plug In 15 The Data Handling Plug In 17 The Data Processing Plug In and 18 The Data Checksum Plug In Apart from the functions contained in the module ngamsPlugInApi py the following classes are used for implementing the plug ins ngamsServer ngamsConfig ngamsReqProps ngamsDppiStatus ngamsDb ngamsPhysDiskInfo NG AMS Disk Dictionary These classes are all described in more details in Chapter 23 Frequently needed in plug in is access to the NG AMS Configuration and to the NGAS DB Access to these can be obtained by means of the methods ngamsServer getCfg and ngamsServer getDb A reference to the ngamsServer object is handed over to all types of NG AMS plug in functions To be able to write efficiently plug ins for NG AMS it is required to have a more a less profound overview of the NG AMS SW or at least this
109. exit 1 cfg ngamsConfig ngamsConfig cfg load sys argv 1 ngamsBrotherPT9200DxPlugIn cfg sys argv 2 Figure 39 Example Label Printer Plug In FILE ngams ngamsPlugIns ngamsBrotherPT9200DxPlugIn py VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 76 of 126 15 EXPERT The Data Archiving Plug In DAPI The purpose of the DAPI is to handle the archiving of data files There are often specific aspects to take into account while archiving various kinds of data The DAPIs make it possible to adapt NG AMS for handling new user specific types of data Le nothing is hard coded in the SW in connection with the data handling When the NG AMS Server receives an Archive Request a thread is spawned to handling the request It first classifies the data and finds the appropriate Storage Set on which to store the file Subsequently it receives the data into an intermediate file with a unique name in the Staging Area on the Main Disk of the target Storage Set The target Storage Set is determined from the NG AMS Configuration From the mime type of the data a suitable Stream is found and afterwards a suitable Storage Set After having received the file the DAPI configured for handling that type of data is invoked to carry out specific tasks to be done during the archiving The main tasks of a DAPI are as follows e Data Consistency Checking Usually it is advisable to carry out a check of the data
110. f disks currently installed If the amount of data on a Storage Set reaches a certain limit defined by a configuration parameter a Notification Message can be send out to a list of subscribers for this event see 3 4 This event is a pre warning that this Storage Set is going to be completed full within a limited time The latter depends on the threshold defined in the configuration file When a Storage Set is considered as completed another type of Notification Message can be broadcast to a number of subscribers This message will indicate that the Storage Set is full and needs to be replaced The appearance of such an email message is as follows example Subject NGAS w2p2nau 7777 CHANGE DISKS Date Fri 25 Jan 2002 01 06 26 0100 MET From ngasmgr w2p2nau eso org Notification Message PLEASE CHANGE DISKS Main Disk Logical Name FitsStorage M 000024 r STOL IDS 5 Replication Disk Logical Name FitsStorage R 000024 grob IDs 6 Figure 6 Disk Change Email Notification Message The Logical Name s Disk Label s as well as the Slot in which the disk s are hosted are indicated in the mail When such a message is received by the NGAS responsible operators it is advisable to carry out the suggested changes as soon as possible to avoid saturation If only a single disk in a set is completed the Email Notification will only indicate the name of this completed disk see the Sections 2 5 and 2 7 3 6 Simulation Mod
111. f idle time An NG AMS Server which is requested to wake up a suspended NGAS Host will invoke a Wake Up Plug In when the time for waking up the suspended host has arrived CFG NgamsCfg HostSuspension WakeUpPlugIn see also Chapter 20 for more information about the Wake Up Plug In The Wake Up Plug In will usually inform some device connected to the network to switch on the suspended NGAS Host After having launched the Wake Up Plug In the NG AMS Server will wait for the suspended NGAS Host to become active If this does not occur within a certain time out CFG NgamsCfg HostSuspension WakeUpCallTimeOut an error message is logged and an Error Email Notification Message send to the subscribers of this VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 34 of 126 5 The NG AMS Server and Utilities Three executables are provided within the NG AMS package These are 1 The NG AMS Server ngamsServer py 2 The NG AMS Python Client ngamsPClient py and 3 The NG AMS C Client ngamsCClient These executables are described in the following sections 5 1 NG AMS Server Command Line Interface By calling the NG AMS Server without command line parameters or illegal ones the following online help is printed on stdout ngasmgr acngast1 opsw NGAS ngams ngamsServer gt python ngamsServer py h Correct usage is ngamsServer cfg lt cfg file gt v lt level gt version license
112. f the disk drive This is the unique identifier of the disk archive varchar 64 mot null The name of the archive to which the disk belongs The value for this is taken from the NG AMS Configuration installation_date datetime not null The date for registering the disk the first time Subsequent re registering do not change the value of this column type varchar 64 not null Describe the type of the media e g MAGNETIC IDISK ATA The value for this is generated by the Online Plug In manufacturer varchar 64 null The manufacturer of the disk Could e g be IBM or Seagate This value is generated by the Online Plug In logical_name varchar 128 not null The Logical Name of the disk is a human readable unique ID for the disk It is generated by NG AMS when the disk is registered the first time host id varchar 32 null The ID of the host where a disk is currently registered If the disk is not registered in any NGAS Host this will be set to slot_id varchar 32 null The ID of the slot in the NGAS Host in which the disk is currently registered If the disk is not registered this will be mounted tinyint null Used to indicate if a disk is mounted or not 1 mounted 0 not mounted VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 52 of 126 varchar 128 null Used to give the complete name of the mount point where the disk is mounted If the disk is not mounted this will be i
113. fly mentioned here This information may be useful in case of troubleshooting or in general for obtaining a deeper insight into the system The main NG AMS project module contains the following files and modules only items of interest in this context are listed Module FWe ngams The main module containing all the NG AMS source code This is managed in a CVS repository located within the ESO network init__ iti i i ee The main Python module containing definitions of basic functions and definition of various constants variables File that contains the license and distribution conditions for the NG AMS SW File that provides a small installation guide for the NG AMS SW Contains the version information for NG AMS This is the information that is printed on stdout when issuing the version parameter to the NG AMS Server or the command line utilities This is also used in the XML status Imessages sent back as response to requests to the server ngamsCClient The NG AMS C based API Also provides the C based command line utility Contains the definition of the various NG AMS XML data formats In addition various example files are provided ngamsLib The base module provides various Python modules with fundamental functions classes and methods used throughout the NG AMS SW ngamsPClient The NG AMS Python based API Also provides the Python based command line utility ngamsPlugIns Contains various example plug ins implemented for the usage
114. gIn PlugInPpars lt StorageSetRef StorageSetId FitsStoragel lt StorageSetRef StorageSetId FitsStorage2 gt lt StorageSetRef StorageSetId FitsStorage3 gt lt StorageSetRef StorageSetId FitsStorage4 gt lt Stream gt lt Processing gt SPlugIn Name ngamsEsoArchDppi PlugInPars gt lt MimeType Nam image x fits lt MimeType Name application x gfits gt lt MimeType Name application x cfits gt lt PlugiIn gt SPlugIn Name ngamsExtractFitsHdrDppi PlugInPars lt MimeType Name image x fits MimeType Nam application x gfits lt MimeType Name application x cfits gt lt PlugiIn gt SPlugIn Name ngasWfiPreview PlugInPars gt image x fits gt application x gfits lt MimeType Name application x cfits gt PlugIn lt Processing gt lt Register gt SPlugIn Name ngamsFitsRegPlugIn PlugInPars gt lt MimeType Nam image x fits lt MimeType Name application x gfits gt lt MimeType Name application x cfits gt lt PlugiIn gt lt Register gt lt FileHandling ProcessingDirectory NGAS VLT MAN ESO 19400 2739 ESO NGAMS User s Manual AAA 4T of 126 PathPrefix saf ChecksumP lugIn ngamsGenCrc32 ChecksumPlugInPars DataCheckActive 0 DataCheckPrio 1 DataCheckMinCycle 00T00 02 00 DataCheckDiskSeq SEQUENTIAL DataCheckFileSeq SEQUENTIAL DataCheckLogSummary 1 gt lt Monitor MinFreeSpaceWarningMb 5000 FreeSpa
115. gInPars This is only unmounting the NGAMS disks and may lead to problems if someone mounts other disks off line if parDicOffline has_key unmount unmount int parDicOffline unmount elses unmount 1 if unmount ngamsLinuxSystemPlugInApi ngamsUmount diskDic stat ngamsLinuxSystemPlugInApi rmMod parDicOnline module ala MA errMsg Problem executing ngamsLinuxOfflinePlugIn The system is in not in a safe state errMsg genLog NGAMS ER OFFLINE PLUGIN errMsg error errMsg raise exceptions Exception errMsg msg Kernel module parDicOnline module unloaded VLT MAN ESO 19400 2739 E S O NG AMS User s Manual Ae 72 of 126 info 1 msg alir name main Main function uu import sys import ngamsConfig ngamsDb Scenog ne Ol MUA doa Tua ah if len sys argv 2 print nCorrect usage is n print python ngamsLinuxOfflinePlugIn lt NGAMS cfg gt n sys exit 0 ngamsCfgObj ngamsConfig ngamsConfig ngamsCfgO0bj load sys argv 1 dbCon0Obj ngamsDb ngamsDb ngamsCfgObj getDbServer ngamsCfgObj getDbName ngamsCfgObj getDbUser ngamsCfgObj getDbPassword dbConobj guery use ngamsCfgobj getDbName ngamsLinuxOfflinePplugIn dbConObj ngamsCcfgobj ij 60g Figure 37 Example System Offline Plug In FILE ngams ngamsPlugIns ngamsLinuxOfflinePlugIn py VLT MAN ESO 19400 2739 ESO NGAMS User s M
116. gamsDb SEloeCorc O VW Op We in if len sys argv 2 print nCorrect usage is n print python ngamsLinuxOnlinePlugIn lt NGAMS cfg gt n sys exit 0 ngamsCfgObj ngamsConfig ngamsConfig ngamsCfgO0bj load sys argv 1 dbConobj ngamsDb ngamsDb ngamsCfgObj getDbServer ngamsCfgObj getDbName ngamsCfgObj getDbUser ngamsCfgObj getDbPassword dbConobj guery use ngamsCfgobj getDbName diskDic ngamsLinuxOnlinePlugIn dbConObj ngamsCcfgobj print DS Date baron SE dnskeDute ip ee Figure 36 Example System Online Plug In FILE ngams ngamsPlugIns ngamsLinuxOnlinePlugIn py VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 71 of 126 13 EXPERT The System Offline Plug In The purpose of the System Offline Plug In is to prepare the system for the Offline State where it should be put to its standby condition During this procedure the disks could be unmounted and other actions performed like e g unloading of SW modules used for accessing the Storage Media 13 1 EXPERT Interface of a System Offline Plug In The function interface of a System Offline Plug In is the same as for the System Online Plug In see 12 1 A System Offline Plug In does not return any information to NG AMS An exception must be thrown in case errors occur during the process of bringing the system to the Offline State 13 2 EXPERT Example System Offline Plug In In the f
117. h URI s Rejecting request The mime type specified in the Archive Request for the file with the given URI is unknown to this NG AMS installation Check the configuration Description Table 26 NG AMS log definition FILE ngams ngamsData ngamsLogDef xml VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 117 of 126 26 NG AMS License Conditions The license conditions provided for NG AMS is based on the BSD License more information about this can be obtained following the link http opensource org licenses bsd license php The NG AMS License text can be obtained either directly in the ngams module as ngams LICENSE It can also be obtained by invoking the NG AMS Server with the parameter license ngamsServer license Likewise it can be obtained using the same parameter for the NG AMS C and Python Clients i e ngamsCClient license and ngamsPClient license Finally it can be obtained from the C API using the function ngamsLicense and from the Python API using the function ngamsLicense In the following the content of the NG AMS License Condition is listed EO HEHH HHRHH EO Li Se te te HHHH HHH xxx LICENSE CONDITIONS Copyright c 2001 2002 European Southern Observatory All rights reserved Redistribution and use in source and binary for
118. hanging the configuration will find the necessary information in this document Finally support for the very advanced user is provided The latter type of user is the user who adds or changes functionality of the system by providing new plug ins or changing existing ones 1 2 How to Read this Manual The intention of this manual is not to provide a book that can be read sequentially chapter by chapter For the user unknown to NG AMS it is recommended to read this chapter and Chapter 2 to get an overview of the manual and of NG AMS and its features For more specific issues it is suggested to check the index or the table of contents and read the referenced sections in connection with these issues The following conventions are used in this manual the object referred by the name is used for commands as they must be types on the shell Same lt element lt element Used to refer to a specific attribute in an XML document e g NgamsCfg Ngams CentralUnit CRC an abon OES Reference to an element attribute in the NG AMS Configuration For detailed information Sh ees p about the NG AMS Configuration consult Chapter 6 DB lt DB column gt Refers to a DB column The reference may also be given as lt db gt lt table gt lt column FILE lt filename gt Reference to a file within the NG AMS SW Package FUNCTION METHOD lt name gt Reference to a specified function or method provided within the NG AMS Package Table 1 Co
119. hen one of the disks is completed also the other disk will be marked in the DB as completed CFG NgamsCfg StorageSet Synchronize Synchronization should normally be used when disks of the same size are used together to avoid that one disk remains un completed whereas in reality hardly any space is available on the media As seen in Figure 1 the NG AMS Storage Media infrastructure is based on a single root directory under which the Storage Disks are mounted Under this area NG AMS is also storing some files for internal purposes Among these is a file containing the PID of the NG AMS Server process FILE lt NgamsCfg Ngams MountRootDirectory gt NGAS_ lt NGAS ID gt e g NGAS NGAS acngast1 7777 The Back Log and Global Bad Files Directories can be placed in a location of choice This is done via the NG AMS Configuration The names of these directories are lt NgamsCfg Ngams BackLogBufferDirectory gt bad files and lt NgamsCfg Ngams BackLogBufferDirectory gt back log The Processing Area Directory shown in Figure 3 is used by NG AMS for storing temporary files while doing file processing The files stored in this directory will be removed by NG AMS after the processing has finished The name of this directory is lt NgamsCfg FileHandling ProcessingDirectory gt processing Some care should be applied when determining the location of these directories since it may have an influence on the performance of the system E
120. hould be performed through this class This therefore contains all the necessary SOL queries used by the NG AMS SW Many methods are provided to perform various specific queries into the NGAS DB A native SQL query can be performed using the method ngamsDb guery IngamsDapiStatus py Po he module provides the class ngamsDapiStatus which is used to handle the status VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 100 of 126 information from the execution of a Data Handling Plug In see Chapter 15 An instance of this class is returned by a DAPI to the NG AMS Server Provides the class ngamsDiskInfo which is used to handle all the information in connection with an NGAS disk The object can also contain information about the files on the disk This is stored internally as ngamsFileInfo objects It is possible to generate an NG AMS XML Status document from the contents of the object IngamsDisk Utils py Functions used to carry out the handling management of the disks installed Among this are function to extract the information about the disk configuration and a function to check the accessibility of the disks installed IngamsDppiStatus py The module provides the class ngamsDppiStatus which is used to handle the status information from the execution of a Data Processing Plug In see Chapter 17 An instance of this class is returned by a DPPI to the NG AMS Server ngamsDppiResult Class th
121. how often this thread should be scheduled in the configuration CFG NgamsCfg SubscriptionDef SuspensionTime This suspension time determines how often the server will try to deliver Subscription Back Log data The Data Subscription Thread is scheduled explicitly when new data become available on an NGAS Host The suspension time defines how frequently the thread should try to deliver Subscription Back Log Buffered data 43 EXPERT Server Suspension Wake Up Service Since an NGAS Host may be idling for longer period of times it is relevant to suspend such a host This is relevant in particular in case of clusters of NGAS nodes which consume a non negligible amount of power A feature is provided by NG AMS whereby it is possible to configure an NG AMS host to suspend itself after a certain period of idle time CFG NgamsCfg HostSuspension IdleSuspensionTime Host suspension can be enabled disabled globally via the configuration CFG NgamsCfg HostSuspension IdleSuspension When an NG AMS Server identifies that it should suspend itself it invokes the so called Suspension Plug In see Chapter 19 which actually takes care of suspending the system Apart from various possible clean up of the system this usually simply means to shut down the NGAS Host The host should normally be configured such that when a shut down is performed the NG AMS Server is terminated in a clean manner After suspending itself an NGAS Host can only
122. iStatus py 97 ngamsDiskInfo py 98 VLT MAN ESO 19400 2739 2 08 01 2003 125 of 126 ngamsDiskUtils py 98 ngamsDppiResult 98 ngamsDppiStatus 84 98 ngamsDppiStatus Class 67 ngamsDppiStatus py 98 ngamsFileInfo py 98 ngamsInternal dtd 38 ngamsLib py 98 ngamsPClient Python API 65 ngamsPClient py Python API 65 ngamsPhysDiskInfo Class 67 ngamsPhysDiskInfo py 98 ngamsPlugInApi py 67 98 ngamsRegProps Class 67 ngamsRegProps py 98 ngamsServer Class 67 ngamsStatus py 98 ngamsStorageSet 97 ngamsStream 97 ngamsUrlLib py 98 NGAS 12 15 NGAS Archiving Unit 20 NGAS Concept 15 NGAS DB 51 NGAS Disk Info Status Example 95 NGAS File Info Status Example 95 NGAS Node to NGAS Node Archiving 116 ngas disks DB table 51 52 54 ngas files DB table 52 57 ngas hosts Availability 51 ngas hosts Db table 53 NgasDiskInfo 19 NGAST 12 No Disk Space Notification 25 Notification Setup 25 O of Security Mechanisms 101 OFFLINE Command 118 offline Python API 65 Offline State 18 Online Browsing of NG AMS SW 98 ONLINE Command 118 online Python API 65 Online State 18 P platforms 16 Plug In API 67 Plug In Concept 16 Private Network 22 Processing Area 14 Processing Area Directory 19 Processing Files 119 Proxy Mode 23 pydoc 99 Python API 65 Python Documentation 98 Python Modules 97 PYTHON_PATH 99 100 E S O NG AMS User s Manual Q Qu
123. ia Storage Set Subscriber Subscription Back Log Target Disk Target Storage Media Providers invokes the appropriate DAPI to handle the data and ingests the information about the data in the NGAS DB Refers to a set of NGAS Hosts which are used together as an archive unit The XML based document used as input to the NG AMS Server when this is started Contains all the information needed for NG AMS to operate in a given context Refers to a WS which has the NG AMS SW installed and which is used as an archiving or data server unit Dictionary that contains information about each NGAS Storage Media available at a certain NGAS Node Directory used to store temporary copies of files to be processed and other temporary files created during processing Refers to the site where data is produced and archived in this case into the NGAS System The site e g at the telescope site where data is produced and being archived into an NGAS System The copy of a data file which is stored in the Replication Area The array of HDDs that contains the replicas of the data on the disks in the Main Storage Area A storage location directory used to temporarily store data files being handled NG AMS e g uses a Staging Area on each Target Disk for receiving data files before moving the files to their final location Refers to a storage unit used in the context of NGAS for receiving data being archived and from which archived data is
124. ider NG AMS Server is running The port number used by the Data Provider NG AMS Server subscr prio tinyint not null The priority of the Subscriber as indicated by the Subscriber itself The priority indicates how fast the data will be delivered to this Subscriber i e how much CPU is allocated to deliver the file varchar 255 See ngas_subscr_back_log susbcr_id varchar 255 See ngas_subscr_back_log subscr_url subscr_start_date datetime null Date from which data should be considered for delivery for that Subscriber subscr_filter_plugin varchar 64 null A Filter Plug In see Chapter 21 which will be applied on the data to determine whether to deliver it or not to a Subscriber subscr_filter_plugin_pars varchar 128 Plug In Parameters to hand over to the Filter Plug In last_file_ingestion_date datetime null The Ingestion Date of the last file delivered to the Subscriber Used to avoid delivering the same data files in multiple copies to the same Subscriber Table 17 Contents of the NGAS Subscribers DB Table VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 55 of 126 87 EXPERT Synchronizing Distributed NGAS DBs Operating an NGAS system which makes use of several independent DBMS constitutes a delicate problem in order to ensure that all sites are kept up to date and to avoid corruption of the data holdings due to unforeseen unwanted replication between the sites
125. ies 1 Problem Description File ID Version Figure 10 Example of a Data Consistency Checking Report If files are found which do not have the checksum properly set NG AMS will calculate the checksum using the DCPI specified in the configuration and send a Data Inconsistency Notification Message to the subscribers of this type of message It is possible to enable and disable the Data Consistency Checking Service CFG NgamsCfg FileHandling DataCheckActive In addition it is possible to allocate a priority to the data checking thread to calibrate the CPU consumption CFG NgamsCfg FileHandling DataCheckPrio It is also possible to specify how disks and files are checked whereby this can either be done sequentially or randomly CFG NgamsCfg FileHandling DataCheckDiskSeq NgamsC fg FileHandling DataCheckFileSeq A minimum cycle time for one iteration of the service can also be defined CFG NgamsCfg FileHandling DataCheckMinCycle If the checking is carried out in less then the specified minimum cycle time the service will be suspended for a while A parameter is used to configure the service to produce a log entry after each iteration with summary information about the check carried out This log entry has the following contents example log entry taken from the Local Log File Figure 11 Example of a Data Consistency Checking Status Log The configuration parameters mentioned above are described in more detail in Chapt
126. igure 63 The NG AMS License Conditions FILE ngams LICENSE VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 118 of 126 27 NG AMS Commands This chapter contains a detailed reference to the commands supported by the NG AMS Server All the commands are listed and explained and the command parameters in connection with these are described Using the NG AMS APIs Chapters 9 and 10 or the NG AMS Python or C based command line utilities Section 5 2 the user is assisted in applying the proper parameters It is recommended to use these when communicating with the NG AMS Server 271 ARCHIVE Command Archive Data Files The ARCHIVE command is used to archive data files The ARCHIVE command accepts the following parameters Mandatory filename lt file URI gt Y The parameter is used to specify the location of the file In case of an Archive Push Request NG AMS may use the given URI to determine the mime type of the file In addition the temporary filename in the Staging Area is based on the filename without the path given in the URI For an Archive Push Request the URI is the location URL where the file can be picked up by the NG AMS Server The location must then be accessible from the NG AMS Server either via HTTP http FTP ftp or directly as file file Also in this case the mime type pf the data may be determined from the path if not specified directly by means of the mim
127. ing Ret rns Void uu Only do check if the checksum util parameter is set if not parDic has key checksum util return Execute the checksum routine and evaluate result info 2 Invoking checksum test utility parDic checksum_util on file filename res ngamsPlugInApi execCmd parDic checksum_util filename if int res 0 0 errMsg Problem occurred invoking checksum check utility parDic checksum_util errMsg genLog NGAMS_ER_DAPI errMsg error errMsg raise exceptions Exception errMsg res 1 parDic checksum_result errMsg Executing checksum utility parDic checksum_util gave unexpected result Result res 1 AN Expected Result parDic checksum_result errMsg genLog NGAMS ER BAD FILE filename ngamsFitsPlugIn errMsg error errMsg raise exceptions Exception errMsg DAPI function def ngamsFitsPlugIn srvobj regPropsObj Data Archiving Plug In to handle archiving of FITS files srvObj Reference to NG AMS Server Object ngamsServer regPropsObj NG AMS request properties object ngamsRegProps Returns Standard NG AMS Data Archiving Plug In Status as generated by ngamsPlugInApi genDapiSuccessStat ngamsDapiStatus wen stagingFilename trgFilename mountPoint An tol Ping hending data FOr LiVe yu MURI ap os path basename regPropsObj getFileUri diskInfo reqPropsObj getTa
128. ing the file example ngasmgr acngast1 opsw NGAS ngams gt python ngamsPClient ngamsPClientEx py acngast1 7777 home ngasmgr tmp WFI 2001 09 15T22 49 07 652 fits lt xml version 1 0 gt lt NgamsStatus gt lt Status Date 2002 12 31T09 28 09 251 HostId acngast1 Message Successfully handled Archive Push Request for data file with URI WFI 2001 09 15T22 49 07 652 fits State ONLINE Status SUCCESS SubState IDLE Version v2 0 Beta2 2002 12 04T09 22 53 gt lt DiskStatus Archive ESO ARCHIVE AvailableMb 32300 BytesStored 8709834319 Checksum Completed 0 CompletionDate DiskId IC35L040AVERO7 0 SXPTX093675 HostId acngastl InstallationDate 2002 11 25T09 48 25 000 LastCheck LogicalName FITS M 000001 Manufacturer IBM MountPoint NGAS datal Mounted 1 NumberOfFiles 163 SlotId TotalDiskWriteTime 905 324898006 Type MAGNETIC DISK ATA gt lt FileStatus Checksum 1810827525 ChecksumPlugIn ngamsGenCrc32 Compression compress f FileId WFI 2001 09 15T22 49 07 652 FileName saf 2001 09 15 1 WF1 2001 09 15T22 49 07 652 fits Z FileSize 142074506 FileStatus 00000000 FileVersion 1 Format application x cfits Ignore 0 Ingest ionDate 2002 12 31T09 28 08 000 Tag UncompressedFileSize 141546240 gt lt DiskStatus gt lt NgamsStatus gt ngasmgr acngast1 opsw NGAS ngams gt Figure 33 Output on stdout from example program using the Python API VLT MAN ESO 19400 2739 ESO NGIAMS
129. ion closed by foreign host ngasmgr acngast1 opsw NGAS ngams gt Figure 9 Example of interaction with NG AMS using telnet In general the NG AMS Python or C based command interface tools should be used when interacting with NG AMS from the shell See Section 5 2 for more information about these tools For more in depth information about the NG AMS command interface consult the Chapters 27 and 7 3 9 Data Consistency Checking The NG AMS Server can be configured to carry out a periodic consistency check of the data files which are stored on the disks installed on that NGAS Node The following checks are carried out e It is checked if files are registered in the DB but are not found on the disk e Checksum value for each file is checked according to the value registered in the NGAS DB for the file e It is checked if files are found in the Storage Area of the storage disks which are not registered in the DB VLT MAN ESO 19400 2739 ESO Noms User s Manual Isu 08 01 2003 28 of 126 In case discrepancies are found in the data holding on the disks in connection with an NGAS Host a Data Inconsistency Notification Message is send out This has the format e g Subject NGAS arcus2 7778 DATA INCONSISTENCY IES FOUND Date Fri 25 Jan 2002 01 06 26 0100 MET From jknudstr eso org Error Message DATA INCONSISTENY IES FOUND IN DATA HOLDING Date 2002 02 12T15 32 05 424 NGAS Host arcus2 Inconsistenc
130. ite complex procedure It is therefore not feasible to provide a complete and detailed information in the NG AMS User s Manual about this In case of problems or questions it is suggested to contact ngast eso org for advice and help on how to approach this matter VLT MAN ESO 19400 2739 ESO Noms User s Manual Is 5 08 01 2003 104 of 126 25 NG AMS Log and Error Messages Definition Many important log messages information and error messages are defined in a formal way in the XML document ngams ngamsData ngamsLogDef xml Defining these log messages in this way makes it possible for a client application to better parse analyze a reply sent back from NG AMS since the log definition is based on tags and error code which will appear in the message The complete log definition is contained in Table 26 Context NGAMS Release 1 0 Source jknudstr eso org Revision Id ngamsLogDef xml v 1 10 2002 12 19 09 14 35 arcsw Exp Description This XML document contains the definition of the error logs used within the NG AMS project Log ID NGAMS ER NO STORAGE SET Log Number 1000 a Type ERROR Log Text Text No No Storage Set matching the Slot ID s Check NG AMS Configuration Yos Set matching the Slot ID s Check NG AMS Configuration s DESH BGOR NG AMS could not find a Storage Set which matches the given Slot ID There seems to be a problem P in the NG AMS Configuration Log ID INGAMS_ER_NO_MIME_TYPES Log Numbe
131. its processing it returns control to NG AMS e The DAPI has then extracted produced the necessary information for NG AMS to be able to update its status in the DB and to be able to move the file to its final destination e If replication is enabled NG AMS also carries out this action and updates the information for the Replication File in the DB The ARCHIVE command is described in Section 27 1 A more detailed description of the procedure executed by NG AMS while handling an Archive Request can be found in Chapter 15 describing the DAPI 3 2 Data File Retrieval amp Processing Archived files can be retrieved from NG AMS using the RETRIEVE command In the present implementation it is only possible to retrieve one file at a time It is possible to request to have the data processed by a DPPI before NG AMS returns it The concept of the DPPIs is described in detail in Chapter 17 When NG AMS receives a Retrieve Request it checks the NGAS DB for the various instances of a data file with the given ID which are online Several versions may be available The decision of which file to choose is done as follows e A list of all files with the given File ID which are registered as being online is retrieved from the NGAS DB In addition files marked to be ignored are not considered e The files are ordered according to their File Version whereby the latest file highest version number gets the highest priority e Itis subsequently inves
132. k information will not be deleted from the NGAS DB and from the the cligis sesal Otherwise alse il as Soge sel ENIS 1nEOrmat LON yal ILL Will pe deleted VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 62 of 126 diskId ID of disk to remove execuiber If set to 1 the command will be executed and the disk removed from the system if possible Otherwise a report will be send back indicating if it is possible to remove the disk referenced ngamsSTAT ngamsRemFile const char hosty const int port const float timeoutSecs const har diskId const har filerd const int fileVersion const int execute ngamsSTATUS status Send a REMFILE command to the NG AMS Server If execute is 0 the disk information will not be deleted from the NGAS DB and from the the disk itself Otherwise if 1 is specified this information will will be deleted For the interpretation of various combinations of the pa ra neters diskIid fileIld fileVersion and execute Cansu the man page of the Python module ngamsRemoveCmd function remFile Cat ALCIR ID of disk hosting the file s to be removed fileld ID of file s to be removed fileVersion Version of file s to be removed execute If set to 1 the files will be removed if possible otherwise a report will be send back indicating what would be removed if the command is executed ngamsSTAT ngamsRetrieve2File const char INOS const int port const float timeo
133. le ooooooooooooooooooooooooooooooooooooooooooooi 100 Table 25 Steps needed to install NG AMS coocoooocoooooooooooooooooooooooooooooooooocoooooooooocooooooi 103 Table 26 NG AMS log definition FILE ngams ngamsData ngamsLogDef xmP ooooooooooooooooooooooooooco 116 Table 27 Parameters for the ARCHIVE command sssssssscescesceeccecccccecccccccccccsecssssssscescescecescesceecccsccasscssecsses 118 Table 28 Rules applied when selecting files for cloning oocoooocoooocoooooooooooooooooooooooooooooooooco 119 Table 29 Parameters for the CLONE command oooooooooooooooooo o ooooooooooooooooooooooooooooooooooooooooooooooooo 119 Table 30 Parameters for the LABEL command oocooooo co ooooooooooooooooooooroocomoomoooooooooo conannnnnann 120 Table 31 Parameters for the OFFLINE command ooooooooooooooooooo o ooooooooooooooo ooooooooooooooooooooooooooooo 120 Table 32 Parameters for the REGISTER Command sssssssscessesceeceeceeccecccccccccecsscssssssscescesceececeeceecccsscessessecsess 120 Table 33 Parameters for the REMDISK command ooooooooooocooooo o o oooooooooooooooo ooooooooooooooooooooooooooooooooo 120 Table 34 Selection rules applied for the REMFILE command coooooocoooocooooooooooooooooooooooooco 121 Table 35 Parameters for the REMFILE commmand sssssssscsscesceecceccscceccccccccccssscssssssscescesceeccecececcccsscessecsecsees 121 Table 36 Paramete
134. le of handling the data delivery request The handling could be implemented as a CGI script An NG AMS Server can be configured to subscribe itself as a Data Subscriber to another NG AMS Server In this case the Subscriber URL should be the URL used when performing an Archive Push Request i e http lt host gt lt port gt ARCHIVE and the corresponding DAPI should be made available within the context of the Data Subscriber NG AMS Server to handle the possible types of data that can be delivered In order to make an NG AMS Server subscribe itself the configuration must be adjusted accordingly CFG NgamsCfg SubscriptionDef see Chapter 6 It is possible to instruct an NG AMS Server to un subscribe itself automatically when it goes Offline CFG NgamsCfg SubscriptionDef AutoUnsubscribe When a client subscribes it can allocate a priority to itself This priority determines how much CPU time the delivery of data files to that client may consume A client that subscribes itself with a lower priority than other Subscribers will receive the files later than these other Subscribers It should be evaluated carefully for each client how soon the data should be delivered The default priority is 10 The lower the priority number the more CPU time the client is allocated Le in principle 0 would be the highest available priority It is advisable to allocate such a priority with great care since the data delivery might consume a lo
135. les on the disk VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Isu 08 01 2003 116 of 126 successfully deleted Log ID NGAMS INFO DEL DISK SEL Log Number 4037 Log Type INFO Log Text Selected info for disk for deletion Disk ID s mae The REMDISK command given would delete the disk referenced to in the log message if execute 1 Description 5 would be specified LogID INGAMS WA DEL DISK2 Log Number 4038 Log Type WARNING Log Text No disk found on NGAS Host s with Disk ID s No disk selected Description The REMDISK command given did not match any disk installed in the contacted NGAS Host Log ID NGAMS_WA_FILE_COPIES Log Number 4039 Log Type Type Log Type WARNING gt gt _ or more files requested to be deleted are not available in at least 3 independent copies within this Log Text NGAS system For security reasons it is enforced by NG AMS to have at least two independent copies of each file defined by a File ID File Version In case of a REMDISK or REMFILE command it will be Description checked for each file that would be deleted by the request if it is available within the NGAS system in at least 3 independent copies An independent copies is defined as instances of one file stored on different storage media Log ID NGAMS ER UNKNOWN MIME TYPE2 Log Number 4040 Log Type ERROR Log Text Illegal mime type s issued in Archive Request for file wit
136. lities are introduced 1 5 Reference Documents The following documents contain additional information and are referenced in the text Reference Document Issue Date Title Number VLT SPE ESO DES Software Next Generation Archive Management VEEMAN ti leet 19400 2534 ae ee System Design Description J Knudstrup http archive 5 VLT MAN ESO 19400 2739 leso ors NGAS Next Generation Archive Systems Technologies oa a IA Wicenec Table 2 Reference documents 1 6 Acronyms The following abbreviations and acronyms are used in this document DB Database DAPI Data Handling Plug In DPPI Data Processing Plug In DTD Document Type Definition HDD Hard Disk Drive HW Hardware N A Not Applicable NGAS Next Generation Archive System NGAS DB NGAS Data Holding DB NGAST Next Generation Archive System Technologies NG AMS NGAS Archive Management System OS Operating System SW Software XML Extensible Markup Language Table 3 Acronyms used in the NG AMS User s Manual VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Isu 08 01 2003 13 of 126 1 7 Glossary Archive Facility Site Archive Cluster Archive Reguest Back Log Buffering Bad File Bad Files Directory Bad Files Area Data Provider Data Subscriber Disk Hard Disk Drive HDD Disk Dictionary Disk Set Dynamic Disk Set Logical Name Main Data File Main Disk Main Storage Area NG AMS Server The f
137. ll and should be Disk Space Notification Disk Change Notification removed from the archiving unit and normally replaced with a new Storage Set See also Section 3 5 If no more free Storage Sets are available a No Disk Space Notification Message is send out No Disk Space Notification to the subscribers of this event Since this is a severe problem a special Notification Message is dedicated for referring to this specific problem If the Data Consistency Check Service encounters errors problems with data files a Data Error Notification Message is send to the subscribers of this event The files that were found to be bad in some way should be analyzed to find out what is causing the problem It could be caused by physical problems of the disk or that due to long storage on the disk failures Data Inconsistency Notification start to occur Problems with a problematic file are normally only reported once I e if the problem is not solved there will be no more notification about the problematic file until the NG AMS Server is re started Table 10 The different types of Notification Messages The Notification Setup is configured in the NG AMS Configuration CFG NgamsCfg Notification For further information about this specific property see Chapter 6 An example Disk Change Notification Message can be found in Section 3 5 3 5 Disk Space Monitoring During the archiving process NG AMS monitors constantly the state of the set o
138. locLogFile lt log file gt locLogLevel lt level gt sysLog lt level gt sysLogPrefix lt prefix gt force autoOnline d seirg seng files NG AMS Configuration File ay Kya Verbose Mode Level version Print out version of server license Print out license information locLogFile lt file gt Name of Local Log File locLogLevel level Level for logging in Local Log File sysLog Switch syslog logging on sysLogPrefix lt prefix gt Prefix for syslog logging A Debugging Mode force Force execution eventhough PID File found autoOnline Bring the server to Online State automatically after initialization noAutoExit If autoOnline is specified and an error occurs preventing the system from going Online it will not auto matically exit Note The values given on the command line overwrites the ones given in the NG AMS Configuration File c ESO DMD 2001 2002 NGAS Project http archive eso org NGAST ngasmgr acngast1 opsw NGAS ngams ngamsServer gt Figure 15 The NG AMS Server online help output as written on stdout VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 35 of 126 5 2 Python and C Command Utilities By invoking the NG AMS Python and C command utilities without input parameters or with illegal ones the following online help is written on stdout example help page generated invoking the C Command Utility gt ngamsCClient host
139. lt NgamsStatus gt Figure 59 Example NGAS Disk Info file FILE xmount root point gt lt disk mount point gt NgasDiskInfo The NGAS Disk Info files are stored at the following location for each disk lt mount root point gt lt disk mount point gt NgasDiskInfo 22 3 NGAS File Info Status Example The following is an example of a File Info Status document which is generated e g when archiving a file or when issuing a STATUS file_id lt file ID gt amp file_version lt file version gt request lt xml version 1 0 gt lt DOCTYPE NgamsStatus SYSTEM http acngast1 hq eso org 7777 RETRIEVE internal ngamsStatus dtd gt lt NgamsStatus gt ESO VLT MAN ESO 19400 2739 NG AMS User s Manual 2 08 01 2003 98 of 126 lt Status Date 2003 01 02T13 48 49 758 HostId acngast1 Mess age Successfully handled command STATUS State ONLINE Status SUCCESS SubState IDLE Version v2 0 Beta2 2002 12 04T09 22 53 lt DiskStatus Archive ESO ARCHIVE AvailableMb 32374 BytesStored 8851908825 Checksum Completed 0 CompletionDate DiskId IBM DTLA 305040 YJ0YJ070913 HostId acngast1 InstallationDate 2002 11 25T09 48 25 000 LastCheck LogicalName FITS R 000001 Manufacturer IBM MountPoint NGAS data2 Mounted 1 NumberOfFiles 164 SlotId 2 TotalDiskWriteTime 350 22437346 Type MAGNETIC DISK ATA gt lt FileStatus Checksum 1810827525 ChecksumPlugIn ngamsGenCrc32
140. lterPlugIn A Filter Plug In applied on the data files to decide whether to deliver a file to a Subscriber VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 44 of 126 Or NOC FilterPlugInPars Parameters to issue to the Filter Plug In lt ELEMENT Subscription EMPTY gt lt ATTLIST Subscription HostId CDATA REQUIRED PortNo CDATA REQUIRED Priority CDATA 10 SubscriberUrl CDATA REQUIRED FilterPlugIn CDATA IMPLIED FilterPlugInPars CDATA IMPLIED gt Zll e De Figure 18 NG AMS Configuration generic DTD FILE ngams ngamsData ngamsInternal dtd VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 45 of 126 6 3 EXPERT NG AMS Configuration Example In the following an example NG AMS Configuration is listed lt xml version 1 0 encoding UTF 8 gt lt DOCTYPE NgamsCfg SYSTEM ngamsCfg dtd gt aan Eno Who When What KKK KK KEK OR RK RR RR KEK KE KK RK RK KKK KK KK KKK KKK RK KK kk kk KEKE jknudstr 04 04 2001 Created KR RK RK RK RR KK RK KE RK RK RK KK KK KKK kk KK RK KK KKK RK kk kk KKK KKK EK KK EK This is the NG AMS Configuration for the WFI NGAS NCU System Consult the DTDs ngamsCfgNbuNcu dtd and ngamsInternal dtd for further information lt NgamsCfg gt lt Header Name NgamsCfg ESOECF GarNcul xml Type NGAMS CONF IGURAT ION NBU Context NGAMS RETSassa il AE Source jknudstr eso org Revision Id ngamsServer xml
141. me of DB server Name Name of the DB to use User The DB user name to connect as Password The password for the DB user lt ELEMENT Db EMPTY gt lt ATTLIST Db Server REQUIRED Name REQUIRED User REQUIRED Password REQUIRED gt SN The MimeTypes Element contains a mapping between the mime types used by NG AMS and the extension names of the data files The element MimeTypeMap contains the mapping between each mime type and the corresponding extension Attributes MimeType Mime type e g ngas fits Extension Extension of data file e g fits without leading dot lt ELEMENT MimeTypes MimeTypeMap gt lt ELEMENT MimeTypeMap EMPTY gt lt ATTLIST MimeTypeMap MimeType CDATA REQUIRED Extension CDATA REQUIRED gt Aa The StorageSet Element defines properties for each storage unit consisting of two disks Main Disk and Replication Disk Attributes StorageSetId ID used to refer to this particular storage unit string DiskLabel Name of label prefix to put on the disk Optional string MainDiskSlotId Slot ID for Main Disk string RepDiskSlotId Slot ID for Replication Disk string Mutex Indicates if mutual exclusion access should be enforced to the disks If this is enabled only one data reception session will write data to that slot disk and not simultaneous as otherwise done if several files for the same disk is received at the same time 1 mutual exclusion integer S
142. meTypeMap MimeType image x fitshdr Extension hdr gt lt MimeTypes gt VLT MAN ESO 19400 2739 ESO NGAMS User s Manual aes 46 of 126 lt StorageSet StorageSetId FitsStoragel DiskLabel FITS MainDiskSlotId 1 RepDiskSlotId 2 Mutex 0 Synchronize 1 gt lt StorageSet StorageSetId FitsStorage2 MainDiskSlotId 3 RepDiskSlotId Mutex 0 Synchronize 1 gt lt StorageSet StorageSetId FitsStorage3 DiskLabel FITS PAF MainDiskSlotId 5 RepDiskSlotId Mutex 0 Synchronize 1 gt lt StorageSet StorageSetId FitsStorage4 MainDiskSlotId 7 Mutex 0 Synchronize 1 gt Stream MimeType image x fits PlugIn ngamsFitsPlugIn PlugInPars compression compress f checksum util utilFitsChecksum checksum_result 0 0000000000000000 gt lt StorageSetRef StorageSetId FitsStoragel gt lt StorageSetRef StorageSetId FitsStorage2 gt lt StorageSetRef StorageSetId FitsStorage3 gt lt StorageSetRef StorageSetId FitsStorage4 gt lt Stream gt Stream MimeType application x cfits PlugIn ngamsFitsPlugIn PlugInPars compression compress f checksum util utilFitsChecksum checksum_result 0 0000000000000000 gt lt StorageSetRef StorageSetId FitsStoragel lt StorageSetRef StorageSetId FitsStorage2 gt lt StorageSetRef StorageSetId FitsStorage3 gt lt StorageSetRef StorageSetId FitsStorage4 gt lt Stream gt lt Stream MimeType ngas nglog PlugIn ngamsNgLogPlu
143. ms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer Redistributions in binary form must reproduce the above copyright notice this list of conditions and the following disclaimer in the documentation and or other materials provided with the distribution Neither the name of the European Southern Observatory nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT INDIRECT INCIDENTAL SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES INCLUDING BUT NOT LIMITED TO PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE For further information and in case of questions please contact ngast eso org F
144. ms ForceProxyMode 1 Proxy Mode If a file is located on a host within the private network or on a remote host possible processing requested will be carried out on the host on which the file resides Methods for retrieving files in an easy manner are provided by the C and Python APIs See the Chapters 9 and 10 3 3 Logging A number of different types of log output can be produced by NG AMS These and their properties are Local Log File The location of a Local Log File is defined in the NG AMS Configuration file CFG NgamsCfg Log LocalLogFile I e the user can decide himself where to put this file The level intensity with which there is logged can be adjusted as well CFG NgamsCfg Log LocalLogLevel Note that NG AMS will continue to write append logs in the same log file Le it should be considered to implement means to purge the log file periodically The format of the Local Log file log entries is as follows ISO 8601 time stamp lt type gt lt log message lt source file gt lt method gt lt line lt thread gt where lt type gt is defined as lt type gt EMERGENCY ALERT CRITICAL ERROR WARNING NOTICE INFO DEBUG Examples of some entries in a Local Log File are 2002 12 19T17 00 20 640 INFO Handling HTTP request client_address 134 171 21 30 34094 method POST path ARCHIVE content disposition attachmentj filename SmallFile fits wait 1 no_versioning 0 content ty
145. ms h The source of the NG AMS C API header file can be found in the NG AMS module as follows ngams ngamsCClient ngams h It contains the prototype definitions for the various functions provided by the API and also the definition of various macros 92 EXPERT NG AMS C API Man Page The man page for the NG AMS C API contains the following information NAME ngamsArchive ngamsArchiveFromMem ngamsClone ngamsCmd2No ngamsCmd2Str ngamsDumpErrStdout ngamsEncodeUrlVal ngamsExit ngamsFreeStatus ngamsInitStatus ngamsIsDir ngamsLabel ngamsLicense ngamsOffline ngamsOnline ngamsRegister ngamsRemDisk ngamsRemFile ngamsRetrieve2File ngamsStat2Str ngamsStatus ngamsSubscribe ngamsToUpper ngamsUnsubscribe ngamsVersion C functions to interface to NG AMS SYNOPSIS include ngams h In general for the NG AMS interface functions listed below the host parameter is the name of the host where the NG AMS Server is running E g arcdevl hg eso org The port parameter is the socket port which the NG AMS Server is waiting on E S O NG AMS User s Manual Tepe hes param ter Twait request will be generate The parameter status i Data Type Member _ BU AL date TAT ngamsS ngamsS ngamsSMALL host1d ngamsHUGE ngamsS ngamsS ngamsS ngamsS char message status state subState version replyDat ul ul L L 1 The f
146. munication interface To use the Python API the following import statements must be contained in the client application from ngams import import ngamsPClient Figure 31 Using the NG AMS Python API The API provides a class ngamsPClient which is contained in the Python module ngamsPClient py The complete documentation for the API is contained as in line Python documentation strings in the source file It is suggested to browse this documentation online using a WEB browser and the pydoc utility See also Section 23 2 Here only a summary of the API is given The most significant methods provided by this class are archive clone exit getHost setHost getPort setPort handleCmd init abel offline online ushFile register remDisk remFile retrieve2File status subscribe unsubscribe Archive a file to NG AMS This can be either done by the Archive Pull Technique or by the Archive Push Technique The technique applied depends on the File URI given Send a CLONE command to NG AMS Used to issue an EXIT command to a running NG AMS Server communicated is running there is communicated Python based command line utility based on the Python API Send a REMDISK command to NG AMS to remove a disk from the system Handle the sending of a command and reception of the reply to this command Send a STATUS command to the NG AMS Server Send a SUBSCRIBE command to the
147. n int t not null Indicates how many data files that have been archived on the disk t not null Used to indicate the amount of available storage capacity still free on the disk given in MBs numeric 20 0 not null Used to indicate the amount of data stored on the disk given in bytes tinyint not null Used to indicate that the disk is completed i e NG AMS has been archiving files on the disk and has reached the threshold specified in the configuration file Lt datetime null Set by NG AMS when the disk reached the threshold for completion varchar 64 null The global checksum value for the disk Note this is not set for the moment _write_ti Total time spent on writing data on the disk datetime null Timestamp for when the last check was carried out This is used to schedule the checking of the data holdings on the disks so that the disks that were not checked or the ones that was checked the longest time ago are checked first varchar 32 null The ID of the host were the disk was registered the last time This is used in order to identify where a file disk is located even though the host has been suspended Table 12 Contents of the NGAS Disks DB Table number of files available mb total disk write time oC Nn n E Ti O 5 S Z 8 lhe E 8 2 EXPERT Table ngas disks hist Tire varchar 128 See disk id in ngas disks table datetime not null Timestamp indicating when the event happened in
148. n a dynamic manner so that disks are associated with each other when NG AMS goes Online and as long as it remains Online This dynamic association is done based on the definition of Storage Sets in the NG AMS Configuration see Section 6 2 XML element StorageSet As soon as the server goes Offline the association does not exist anymore This means that in the context of NGAS data is seen file wise and not disk wise and no attempts are made in order to maintain identical sets of files on different disks once associated during operation Note that it is possible to have a Dynamic Disk Set consisting of only a Main Disk It is recommended however always to use the Replication Service of NG AMS to increase data safety to avoid loss of data A Storage Set is considered to be completed when either of the two disks in it are considered as full see also Section 3 5 If a Storage Set consists only on one Main Disk it is considered completed when the Main Disk is full Beware that since the association between disks only exists during operation it may be that only one of the disks in a set is actually marked as completed in the DB whereas the other remains un completed and can be used together with another disk In this way it is e g possible to use Main Disks of size 80 GB together with Replication Disks of size 200 GB In the definition of the Storage Set in the NG AMS Configuration it is possible to lock two disks together so that w
149. name file tmp SmallFile fits amp wait 1 HTTP 1 0 Figure 23 Example of NG AMS GET method HTTP request Archive Pull Request The exact list of parameters for each command are described in Chapter 27 VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 49 of 126 7 2 EXPERT Format of the NG AMS HTTP Reply The format of replies from NG AMS is defined as follows HTTP lt HTTP version lt HTTP response code lt message gt Server lt server ID gt Date lt date for generating reply gt Expires lt expiration date Date gt Content Type lt mime type gt Content Length lt data length gt lt data gt Figure 24 Format of NG AMS HTTP response An example of a reply to an Archive Request is HTTP 1 0 200 OK Server NGAMS v2 0 Beta2 2002 12 04T09 22 53 Content type text xml Expires Mon 23 Dec 2002 16 10 43 GMT Content length 1188 Date Mon 23 Dec 2002 16 10 43 GMT lt xml version 1 0 gt lt NgamsStatus gt lt Status Date 2002 12 23T16 10 43 079 HostId acngast1 Message Successfully handled Archive Push Request for data file with URI SmallFile fits State ONLINE Status SUCCESS SubState IDLE Version v2 0 Beta2 2002 12 04T09 22 53 gt lt DiskStatus Archive ESO ARCHIVE AvailableMb 32300 BytesStored 8567866905 Checksum Completed 0 CompletionDate DiskId IC35LO40AVERO7 0 SXPTX093675 HostId acngastl InstallationDate 2002 11 25T09 48 25 000
150. nchronize data holdings of different NGAS Nodes partially or completely and to export data on the fly from one NGAS archive to remote sites that need part of the data or all data becoming available on an NGAS Host A client subscribing for data is referred to as a Data Subscriber An NG AMS Server which delivers data to such a Subscriber is referred to as a Data Provider When a client subscribes it can specify to receive data from a certain point in time In this way it is possible for a client to receive older files Otherwise the time for subscription is taken as starting point and only new files archived from the time of the subscription are taken into account for that client It is also possible to specify a Filter Plug In see Chapter 21 which is applied on the data files to determine whether or not to deliver the file to a specific Data Subscriber A client subscribes itself by issuing a SUBSCRIBE command see also Section 27 13 The client subscribes itself giving a so called Subscriber URL to the Data Provider NG AMS NG AMS delivers data to the client by performing an HTTP POST on the Subscriber URL It is up to the client to specify a proper Subscriber URL On the client side a corresponding HTTP server must be ready to handle the data delivery POST requests from the Data Provider Any WEB server can be used from a simple customized implementation to an existing and widely used server like e g Apache The server must of course be capab
151. ne word string e g PROCESSED_FILES A comment can be added to a File List to remark special conditions Status A status allocated to the File List lt ELEMENT FileList FileStatust FileList gt lt ATTLIST FileList Id CDATA REOUIRED Comment CDATA IMPLIED Status CDATA IMPLIED gt GS 10 Figure 58 NG AMS Status DTD FILE ngams ngamsData ngamsStatus dtd 22 2 NGAS Disk Info Status Example Apart from keeping the information about an NGAS Disk in the NGAS DB DB ngas disks NG AMS maintains a snap host of this information in an XML document on each disk This document is referred to as the NgasDisklnfo file It is created when the disk is first registered and subsequently updated each time the NG AMS Server goes Online Offline and when the disk is completed The following is an example of an NgasDiskInfo file Such XML status documents are stored on each NGAS disk lt xml version 1 0 gt lt NgamsStatus gt lt Status Date 2003 01 02T08 40 23 350 HostId acngast1 Message Disk status file Version v2 0 Beta2 2002 12 04T09 22 53 gt lt DiskStatus Archive ESO ARCHIVE AvailableMb 32300 BytesStored 8709834319 Checksum Completed 0 CompletionDate DiskId IC35L040AVERO7 0 SXPTX093675 InstallationDate 2002 11 25T09 48 25 000 LogicalName FITS M 000001 Manufacturer IBM NumberOfFiles 163 TotalDiskwriteTime 905 324898006 Type MAGNETIC DISK ATA gt
152. needed for fulfilling the request is not available on this NG AMS System Could e g be Description p 4 File Retrieval Log ID NGAMS_ER_BAD_FILE Log Number 4027 Log Type ERROR Log Text Error occurred handling file s in DAPI s Error os ine A DAPI detected a problem inconsistency in the file being handled The file is considered as bad and Description cannot be archived properly Log ID NGAMS ER ILL DPPI Log Number 4028 Log Type ERROR EOSTI Ilegal DPPI s specified in connection with Retrieve Request Given DPPI is not defined in the 08 1eXt ING AMS Configuration An NG AMS installation only accepts to execute DPPIs which are explicitly defined in the NG AMS Configuration If a DPPI is specified which is not defined in the configuration this will be rejected Description even if this DPPI might be available as such on the NGAS Host handling the request The problem can be solved by adding the name of the DPPI in the configuration file provided that this DPPI in fact is available on the system and that it is desirable to make this DPPI available for external users Log ID INGAMS ER CMD SYNTAX Log Number 4028 Log Type ERROR Log Text The combination of parameters given together with the command s is illegal Parameter s Yos The combinations of parameters given together with the command referred to in the error message is Description illegal syntactic or semantically wrong Check the NG AMS User
153. ng of files is that when registering files the files stay in the location where they are and it is not necessary to create a new target filename for these and to move these around A Register Plug In must be defined for each type of data that is going to be registered using the REGISTER command See also Section 6 2 Register Element 16 1 EXPERT Interface of a Register Plug In The interface of a Register Plug In is identical to that of the Data Archiving Plug In see Section 15 1 for further information 16 2 EXPERT Example Register Plug In In the following an example Register Plug IN which is used for archiving FITS files is shown SEE KK KKKK KK KK KK KK KK KK KK KK KK KK KK KK KK K KE KK KK KK KK KK KK KK kk kk KK Kk kk Kk kk KKK kk Kk KK KK ESO DMD SId ngamsFitsRegPlugIn py v 1 5 2002 07 10 08 34 33 arcsw Exp jknudstr 10 05 2001 Created This Data Register Plug In is used to handle the registration of FITS files already stored on an NGAS disk which just need to be registered in the DB Note that the plug in is implemented for the usage at ESO If used in other contexts a dedicated plug in matching the individual context should be implemented and NG AMS configured to use it import os exceptions string from ngams import import ngamsPlugInApi ngamsDiskUtils ngamsDiskInfo ngamsFitsPlugIn Data Registration Function def ngamsFitsRegPlugIn srvObj regPropsObj Data Registra
154. ng the configuration file at the position in the document as indicated by Pesanan the error message Check correct configuration file and try again Log ID NGAMS ER ILL MOUNT ROOT DIR Log Number 1007 Log Type ERROR Illegal path specified for Mount Root Directory in configuration file s Parameter Log Text FileHandling MountRootDirectory Path given s The directory specified as Mount Root Directory is not writable or not existing Create this and try Description again LogID INGAMS ER LOAD CFG Log Number 1008 Be es NA Log Text Problem encountered attempting to load the NG AMS Configuration 9s Error message s Desa The specified configuration file could not be loaded This could e g be due to that the file is not P readable has not read permissions for the user running NG AMS or that the file is not available Log ID NGAMS ER ILL PROC DIR Log Number 1009 Log Type ERROR ar Illegal path specified for Processing Directory in configuration file s Parameter 8 FileHandling ProcessingDirectory Path given s The directory specified as Processing Directory is not writable or not existing Create this and try Description again Log ID INGAMS_ER_PLUGIN_PAR Log Number 1010 Log Type ERROR Log Text Plug In Parameters are improperly formatted Yos Correct format is par value par value Description The Plug IN Parameters defined in connection with a plug in are
155. ng when to actually update a record in the Archive Facility DB This could e g be done by using the information in the ngas hosts table If the disk for which information is received for update is located in a host at the Archive Facility Site such an update is discarded The replication engine can determine where the disk is located from the columns ngas hosts host id and ngas hosts domain The complete association to do implement is as follows ngas disks host id ngas hosts host id ngas hosts domain Note that using bi directional DB replication may not be an optimal solution either as unforeseen unwanted changes in the Production Site DBs are propagated to the Archive Facility Site s Using bi directional replication it would still be necessary to implement a conditional replication as explained previously The scenario described in this section should be seen mainly as an example and could be used as inspiration when designing the DB system for a distributed NGAS installation VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 5 08 01 2003 57 of 126 9 EXPERT The C API Together with the NG AMS package an API to be used for interfacing C applications with the NG AMS Server is provided This is provided in the form of a small library with a set of functions making it easy to communicate from client applications to the NG AMS Server Also a number of various macros are provided
156. ns FILE ngams LICENSE ooooocoocoocooooooooooooooooooooooooooooo 117 VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 5 08 01 2003 10 of 126 LIST OF TABLES Table 1 Conventions styles used in the NG AMS User s Manual ocoooocoooocoocooooooooooooooooooooooooooooo 11 Table 2 Reference dOcUuMEtS 5n NN delece Soe bancaueed Soei oee stees Seose s oeeo seos Sais 12 Table 3 Acronyms used in the NG AMS User s Manual oooooocooooooooooooooooooooooooooooooooooooo 12 Table 4 Glossary used in the NG AMS User s Manual coooocoooooooooooooooooooocoocoooooooooooooooooooo 14 Table5 NG AMS State Sub S tatess 2200112015 NUNUKAN Na Ges deesdevascaveedeasenecveencodesess 18 Table 6 Parameters for data classification ccccccccsccsscsssssssccsssssscescesceeccececcesccecccccceccecsscssssssscsceeceecescescescccssccsseeses 20 Table 7 Reserved mime typeS ooooooocoooooooooocooooooooooooooooooooooooooooooooooocooocooooooooooooco 20 Table 8 The supported log output formats oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 24 Table 9 Interpretation of Log Levels ooocoocooooooooooooooooooooooooooooooooooooooooooooooooooo 24 Table 10 The different types of Notification Messages oooooocoooooooooooooooooooooooooooooooooooooooooooooo 25 Table 11 NG AMS High Level Services that can be enabled disabled ocoocooocooooooooooooooooooooo 29 Table 12
157. nt this parameter is ignored somad Spes era The command specific parameters are Arska lt sdisk LD Refers to a specific disk Sexe UB Execute an action fileId lt id gt Refers to a specific file ee ee aliasing URI pointing to the file to be archived file home data Imagel fits This will result in an Archive Pull Request It can also be given directly as a filename e g home data Imagel fits E S O NG AMS User s Manual 2 fileVersion lt version gt filterPlugIn iOWESS internal lt filename gt mimeType lt mt gt ngLog outputFile path path plugInPars priority lt prio gt SESSA Ae SDDS 1s processingPars lt pars gt SLOC ac Sieger Denes Swell surts RCHIVE LONE Heli RETRIEVE VLT MAN ESO 19400 2739 08 01 2003 36 of 126 This will result in an Archive Push Request Refers to a specific File Version of a file Reference to a Filter Plug In to apply in connection with handling the request Force an action Refer to an NG AMS internal file If it is not possible to determine the mime type of a data file from the filename the mime type must be explicitly given with the ARCHIVE command Refers to the Local Log File used by NG AMS Directory or file In hich to dump a tile when retrieving files Specifies a path which should be taken into account when carrying out the request Parameters to provide to the pl
158. ntain an arbitrary number of ngamsDppiResult objects each containing the information of one sub result As can be seen in the figure the data of a sub result can either be contained directly in the ngamsDppiResult object or the data can be stored in an external file which is referred to by the object Whether to use the one or the other depends on the nature of the data If the result consists of a smaller amount of non binary data it is more convenient to store the data internally to avoid having to create access and delete the result files For larger amounts of result data and for binary data it is recommended to use an external result file See Chapter 23 for more information about these classes External temporary files Result Files will be deleted automatically by NG AMS after the result data has been returned to the requestor 17 2 EXPERT Example DPPIs In the following a very trivial example of a DPPI is shown It is used to extract the header information of a FITS file Pe ea a ee ee ee KK KK KK KK KK KK KK KEKE KK KK KK KK KK KK KK EK KK KK KK ESO DFS SId ngamsExtractFitsHdrDppi py v 1 9 2002 09 27 12 13 34 arcsw Exp awicenec 26 09 2002 Created Contains a DDPI which is used to extract the main header from FITS files un from ngams import import ngamsPlugInApi ngamsDppiStatus 8 For actually supporting completely latter NG AMS needs to be extended to be able to return replies
159. nts in a simple and straightforward manner E g from a WEB browser better if XML enabled it is possible to query the status of an NG AMS Server 3 http facngasti hg eso org 7777 STATUS M File Edit View Favorites Tools Help 14 Z 2 Q 9 x 2 Address E http facngast1 hg eso org 7777 STATUS Go Links a lt xml version 1 0 gt lt DOCTYPE NgamsStatus View Source for full doctype gt lt NgamsStatus gt Status Date 2003 01 02T08 41 33 976 HostId acngast1 Message Successfully handled command STATUS State ONLINE Status SUCCESS SubState IDLE Yersion 2 0 Beta2 2002 12 04TO9 22 53 gt lt NgamsStatus gt 9 SPA Favori gt Search pag Favo ites Figure 8 Interaction with an NG AMS Server from a WEB browser Also a standard utility like telnet can be used to interact with NG AMS e g to issue a command like OFFLINE ngasmgr acngast1 opsw NGAS ngams gt telnet acngastl 7777 Atsewaliove LIANT ARII 9 Connected to acngastl Escape character is GET STATUS HTTP 1 0 200 OK lt xml version 1 0 gt lt DOCTYPE NgamsStatus SYSTEM http acngast1 hq eso org 7777 RETRIEVE internal ngamsStatus dtd gt lt NgamsStatus gt lt Status Date 2002 12 23T14 59 42 724 HostId acngast1 Message Successfully handled command STATUS State ONLINE Status SUCCESS SubState IDLE Version v2 0 Beta2 2002 12 04T09 22 53 lt NgamsStatus gt Connect
160. nts of the NGAMS Status NA The NgamsStatus element is the root element of the NGAMS Status Document 5 lt ELEMENT NgamsStatus Header Status NgamsCfg DiskStatus FileList gt SI The Status Element is used to generate a status with log information It can contain an arbitrary number of log elements defined in the LogMlBase DTD Attributes Date Date this Status Element was generated Version Version of NG AMS generating the status HostId Name of host where the NG AMS Server is running Status Overall status of the status information Can be used to signal if errors are contained in the Status Elements Message Message generated by the NG AMS Server State State of the NG AMS Server SubState Sub State of the NG AMS Server 5 lt ELEMENT Status Trace Debug Info Warning Error Alarm Archive gt lt ATTLIST Status Date CDATA REQUIRED Version CDATA REQUIRED HostId CDATA REQUIRED Status OK FAILURE Wein Message CDATA IMPLIED State ONLINE OFFLINE OFFLINE SubState IDLE BUSY IDLE gt SI The NgamsCfg Element contains the configuration used by NG AMS lt ELEMENT NgamsCfg Ngams Server Db StorageSet FileHandling Stream Monitor Log Notification HostSuspension gt MESS The DiskStatus Element contains the status for each disk and the status for the files stored on an HDD E S O NG AMS User s Manual ACErIDUtSSI DiskId Archive
161. nventions styles used in the NG AMS User s Manual Some sections are dedicated to the more advanced users of NG AMS These sections are marked with EXPERT A normal user may want to skip these sections lt element gt lt element gt lt attribute gt The last chapter 27 contains a quick reference to the commands supported by NG AMS 13 How to Get Help or Report Problems with NG AMS or this Manual In case problems are encountered using NG AMS bug problem reports can be submitted via email to ngast eso org This also goes for guestions and other assistance needed in connection with the usage and enhancement of the system VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 12 of 126 1 4 Disclaimer Although great efforts have been invested in designing robust interfaces for the NG AMS SW e g when it comes to the HTTP communication protocol various XML document formats and the interfaces of the APIs provided it should be mentioned that NG AMS is still in a relatively early phase and minor changes may have to be introduced in the various interfaces It will however be attempted to limit the amount of such changes to an absolute minimum See also Chapter 26 NG AMS License Conditions In general it is attempted to keep NG AMS compatible with previous versions However in cases where it is necessary to change interfaces etc to comply with standards it may be that some backwards incompatibi
162. o when issuing an EXIT command the server invokes the proper clean up procedure If the server is killed by a SIGKILL 9 signal the signal handler is not invoked and the server leaves its environment in an undefined state This also happens if the computer on which the server is running is shut down abruptly If this happens it will be necessary to start the server subsequently with the force parameter to force it to start up It is possible to clean up the environment by bringing the server Online Offline in the proper manner http www python org doc current lib module pydoc html VLT MAN ESO 19400 2739 ESO NGAMS User s Manual A 18 of 126 2 4 The NG AMS Server States amp Sub States The NG AMS Server is maintaining a scheme of a State and a Sub State that determine which services the server can handle at a given point in time and which indicate the condition of the server The States and Sub States and the corresponding conditions are as follows C Offline This is the condition in which the NG AMS Server enters In this state the server is performing the transition from Offline after starting up and when the OFFLINE command haslto Online or is preparing to exit from execution No been issued In this state only the STATUS command is commands are accepted accepted I e no Archive or Retrieval Requests are handled The EXIT command is also accepted Latter makes the server clean up and te
163. oO Ol Log Text Successfully deleted info for file Disk ID s File ID s File Version d oe The information for a file was successfully removed from the NGAS DB and from the NGAS disk Description 4 hosting the file Log ID NGAMS INFO FILE DEL STAT Log Number 4033 Log Type INFO Log Text File deletion status Files Selected d Files Deleted d Failed File Deletions d Berasa Status over a REMFILE command indicating 1 How many files were selected for deletion 2 How P many files were deleted 3 How many attempts to delete file that failed LogID NGAMS_ER_ DEL DISK Aa Log Number o3 Log Type ERROR 00 Log Text Problem encountered deleting files on disk Disk ID 9es Exception s o A problem was encountered while trying to delete files on the disk with the Disk ID given in the error Description SA message Maybe the disk is mounted read only Log ID NGAMS ER DEL DISK DB Log Number 4035 Log Type ERROR Log Text Problem encountered deleting disk info from DB Disk ID s Exception os A problem was encountered while trying to delete the information for a disk from the NGAS DB Maybe the DB user used does not have permission to execute DELETE SQL statements Description Log ID INGAMS_INFO_DEL_DISK Log Number 4036 Log Type INFO Log Text Successfully deleted info for disk Disk ID s Description The information for a disk was successfully removed from the NGAS DB and the fi
164. oco 89 EXPERT THE SUSPENSION PLUG IN oo mm Woman naa 91 19 1 19 2 20 20 1 20 2 21 21 1 21 2 22 22 1 22 2 22 3 23 23 1 23 2 24 25 26 27 27 1 27 2 27 3 27 4 27 5 27 6 27 7 27 8 VLT MAN ESO 19400 2739 ESO Noms User s Manual Isu 08 01 2003 6 of 126 EXPERT Interface of a Suspension Plug In o oooocoooooooooooooooooooooooooooooooooooooooco 91 EXPERT Example Data Checksum Plug In ooooooooocoocoocooooooooooooooooooooocoooooooo 91 EXRERIE THE WAKE UP PLUG IN oooo oom en anna an 92 EXPERT Interface of a Wake Up Plug In oooooooooocoooooooooooooooooooooooooooooooooooooooooooooooco 92 EXPERT Example Wake Up Plug In ooocooooooooooooooooooooooooooooooooooooocoocooooco 92 EXPERT THE FILTER PLUG IN oom en nan en ana an 93 EXPERT Interface of a Filter Plug In oooooooocooooocoooooooooooooooooocooocoooooooooooooooooooooio 93 EXPERT Example Filter Plug In ocoooooooooooooooooroooooooooooooooocoooooooooooooooooooooooooo 93 THE NG AMS STATUS XML DOCUMENT oo nana 95 EXPERT NG AMS Status DTD ngamsStatus dtd oocoooocoooocooooooooooooooooooooooooooooooooio 95 NGAS Disk Info Status Example oocoooocoooooooooooooooooooooooooooooooooooooooocoooooooo o 97 NGAS File Info Status Example ooooooooocooooooooooooooooooooooooooooocoooooooo 97 EXPERT THE NG AMS PYTHON MODULES oo oom mann nana 99 EXPERT NG AMS Module Str
165. ollowing an example System Offline Plug In which is used for the moment for the NGAS installation for WFI at the La Silla 2 2m telescope It is perhaps not a very good example of such a plug in since most of the code is distributed in other modules Check the Python source files ngams ngamsPlugIns ngamsEscalada6800Utils py and ngams ngamsPlugIns ngamsLinuxSystemPlugInApi py for further information SEK KKKKKKK KK KK KK KK KK KK KEKE KK KK KK KK KEKE KK KK KK KK KK KK KK EK KK KK KK KK KK KK KK KK KEKE KKR ESO DMD Id ngamsLinuxOfflinePlugIn py v 1 10 2002 07 10 08 34 33 arcsw Exp jknudstr 10 05 2001 Created Module that contains a System Offline Plug In used by the ESO NGAS installations wen from ngams import import ngamsPlugInApi import ngamsLinuxSystemPlugInApi ngamsEscalada6800Utils def ngamsLinuxOfflinePlugIn srvObj reqPropsObj None wen Function unmounts all NGAMS disks and removes the kernel module for the IDE controller card srvObj Reference to instance of the NG AMS Server class ngamsServer regPropsObj NG AMS request properties object ngamsRegProps Returns Void won rootMtPr srvObj getCfg getMountRootDirectory parDicOnline ngamsPlugInApi N parseRawPlugInPars srvObj getCfg getOnlinePlugInPars diskDic ngamsEscalada6800Utils parseHtmliInfo parDicOnline uri rootMtPr parDicOffline ngamsPlugInApi parseRawPlugInPars srvObj getCfg getOfflinePlu
166. ollowing error are defined data ty Error Macro ngamsSTAT SUCCESS ngamsERR HOST ngamsERR SOCK ngamsERR CON ngamsERR COM ngamsERR_TIMEOUT ngamsERR_WR_HD ngamsERR_WR_DATA ngamsERR_RD_DATA ngamsERR_INV_REPLY ngamsERR_F ngamsERR_A LE LLOC_MEM KNOW KNOW V_TA V_PA ngamsERR_U ngamsERR_U ngamsERR_I ngamsERR_I STAT _CMD RG_FILE RS OK REDIRECT ngamsS ngamsS RV RV ngamsSRV INV OUERY Apart trom that Ce isis errorcode VLT MAN ESO 19400 2739 2 08 01 2003 58 of 126 ie SS ico Or CP alos an immediate reply to the before the request has been handled s a structure containing the following members Description Date for handling query Error code giving status for the query See 1 Host ID for host where the NG AMS Server is running Message from the NG AMS Server Status One query TOKY MByVAIMEUIRI 9 State of the NG AMS Server Sub State of the NG AMS Server Version of the NG AMS Server Pointer array Ole pointers pointing to allocated buffers containing the reply data from the request a codes internal to the NG AMS C API pe ngamsSTAT Description Ouery successfully executed No such host Cannot Create Socket Cannot connect to host server Problem occurred during socket connection Timeout encountered while communication with server Write error on socket while writing header Write error on socket while writing data
167. ollowing glossary is used in this document Refers to an NGAS based archive system within an organization where all data produced are being managed and from where online access to the data and processing facilities are provided There might be several such Archive Facilities within an organization Reguest from a client of the NG AMS Server to have a file archived Back Log Buffering can be carried out by the NG AMS Server if an error occurs which makes it impossible to archive the file at that moment The file will thus be stored temporarily in the Back Log Buffer Area The NG AMS Server Janitor Thread will attempt at a later stage to handle the file A Bad File is a file that could not be accepted for archiving by NG AMS Le it was rejected by the DAPI handling this file type This could e g be due to a wrong expected size of a FITS file Area on the disk where files which are mal formed are stored There is a Global Bad File Directory on one of the system disks on each NGAS Node Apart from that there is a Bad Files Directory on each archive disk installed An NG AMS Server to which one or more Subscribers have submitted a request for data Each time a new file becomes available on this NG AMS system the Data Provider will check if it should be delivered to one or more of its Subscribers Client that has subscribed itself to receive a certain kind of data from an NG AMS Server In order for a Subscriber to subscribe itself it mus
168. onDate CDATA REQUIRED Checksum CDATA REQUIRED TotalDiskWriteTime CDATA REQUIRED LastCheck CDATA IMPLIED gt Date for last check Sa The FileStatus Element contains the status of each file stored on the individual HDD Attributes FileName Name of the file relative to the disk mount point Fileld ID of the file e g DP ID for ESO FITS files FileVersion File Version Format Mime type of file FileSize Size of the file bytes UncompressedFileSize Size of file uncompressed bytes Same as FileSize if uncompressed Compression Type of compression applied on the file IngestionDate 5 lt ELEMENT FileStatus EMPTY gt Date the file was ingested ISO 8601 VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 97 of 126 lt ATTLIST FileStatus FileName IMPLIED File1d IMPLIED FileVersion IMPLIED Format IMPLIED FileSize IMPLIED UncompressedFileSize IMPLIED Compression IMPLIED IngestionDate IMPLIED Ignore IMPLIED Checksum IMPLIED ChecksumPlugIn IMPLIED FileStatus IMPLIED gt The FileList Element is used to contain a list of files This can be used for various purposes is e g used for the CLONE command to indicate which files were attempted to be cloned but which couldn t be properly cloned The element contain 1 or more FileStatus Elements Attrib tes Name Name allocated to the list Tas A short ID allocated to the file list Should normally be a o
169. oooocoocooooooocooooooooooooooooooooooooco 121 27 11 RETRIEVE Command Retrieve amp Process Files oocooooooooooooooooooooooooooooooooooooooooooooo 121 27 12 STATUS Command Query System Status amp Other Information oocoooocoooooooooooooooooic 122 2713 SUBSCRIBE Command Subscribe to Data from NGAS Host ooococoocooooooooooooooooio 122 27 14 UNSUBSCRIBE Command Unsubscribe a Previous Data Subscription cooooooooooooooooooooooooc 122 28 INDEX aan denna anakan ma binal mask Eaa aae Eaa 124 LIST OF FIGURES Figure 1 Example operational environment of the NG AMS Server ooooooocooocoooooooooooooooooooooooooo 15 Figure 2 Response to STATUS command ocoooooooooooooooooooooooooooooooococoooooooooooooocoooi 18 Figure 3 The NG AMS Storage Media Infrastructure ooooooocoooooooooooooooooooooooooooooooooooooo 18 Figure 4 Data channeling Figure 5 Life cycle of an NGAS disk ooocococooooooo Figure 6 Disk Change Email Notification Message oooooocooooooooooooooooooooooooooooooooooooooooooooo 25 Figure 7 Example reply when Back Log Buffering is applied cooooooooocoooooooooooooooooooooooooooo 26 Figure 8 Interaction with an NG AMS Server from a WEB browser oooocoocooooooooooooooooooooooo 27 Figure 9 Example of interaction with NG AMS using telnet oooooooooocooooooooooooooooooooooooooooooooooooo 27 Figure 10 Example of a Data Consist
170. oooooi 71 EXPERT THE LABEL PRINTER PLUG IN woo mm Woman 73 EXPERT Interface of a Label Printer Plug In ooocoocooooooooooooooooooooooorooooooooooco 73 EXPERT Example of a Label Printer Plug In oooooocoooooooooooooooooooooooooooooooooooooooco 73 EXPERT THE DATA ARCHIVING PLUG IN DAPI mo mom mann 76 AMBER Interface of a DAPL ocoocoococooooocooccocoo Wco c co m oo o o coo wooo o oco o oo o o o o Wocoo o oo mo 78 EXPERT Overall Structure amp Algorithm of a DAPI ooocoooococoocooooooooooooooooocooooooooooco 79 EXPERT Example DAPI WFI FITS File DAPI o ocoocooocooooooooooooooooooooooooooooooooooooooooco 80 EXPERT THE REGISTER PLUG IN oo om en teen men ena naa 83 EXPERT Interface of a Register Plug In ooooooooocooocooooooooooooooooooooooooocoooooooooooooo 83 EXPERT Example Register Plug In oooooooooooooocoooooooooooooooooooooooooooooooooooooooooooooco 83 EXPERT THE DATA PROCESSING PLUG IN DPPI ooo oom momo 85 EXPERT Interface of a DPPI oocooooocooooooooooooooooooooooooooooooooocooocoooooooooooooooooaca 85 EXPERT Example DPPIS oocoocoocoooocooooooooooooooooooooooooooocooocoooooooooooooocoocoannan 86 EXPERT THE DATA CHECKSUM PLUG IN oo nana 89 EXPERT Interface of a Data Checksum Plug In oooooooooooooocooooooooooooooooooooocooooooooio 89 EXPERT Example Data Checksum Plug In ooooooooocooooocoooooooooooooooooooooooooooooooooooo
171. oooooooooo 22 3 3 VOB SUING vsesecscnadctenstssticcescesetensesstecsesaastessecooasessascesanasesssessnseedstacesdsseususdeustesdneseauesdesenssdenasensadsscusssdenasessadsstenes nseses 23 34 Email Notification ooooooooooooooooooooooooooooooooooooooooooooooocooooooooooooocooi 24 3 5 Disk Space Monitoring ooooooocoooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo o 25 3 6 Simulation Mode oooocoooooooocoooooooooooooooooooooooooooooooooooooooooocoocoooooooooooooooooooo o 25 3 7 Back Log Buffering ooooooooocooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 26 3 8 The NG AMS Server Command Interface coooooooooooooooooooooooooooooooooooooooooooooooo 27 3 9 Data Consistency Checking ooooooocoooooooooooooooooooooooocoooooooooooooooooooooooooooo 27 VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 4 of 126 3 10 Label Printing oocooooooooooooooooooooooooooocoooooooooooooooooooooc 28 SLD SECARA RABBA BNN csoastandes cousdssssesesassesoiesennedsestesevekssoanedsdsesseetteoonest 28 4 EXRERIE ADVANCED FEATURES ooWoo oom mon anna aan 30 41 EXPERT Operation in Cluster Mode ooocoooooooooooooooooooooooooooooooooooooooooooooooo 30 42 EXPERT Data Subscription Service ocooooooooooooooooooooooooooooooooooooocooooooooooooooo 32 43 EXPERT Server Suspension Wake Up Service oocooooooooooooooooooooooooooooooooooooooooooooooooooooo
172. oooooooooooooooooooooooooooooooooocoooio 49 Figure 25 Example of NG AMS HTTP response Archive Request oooooocoooocoooooooooooooooooooooooooooo 49 Figure 26 Example of NG AMS HTTP response Retrieve Reguest ooooocoooocooooooooooooooooooooocoo 49 Figure 27 Structure of NG AMS HTTP Redirection Response oocoooococooooooooooooooooooooooooooo 50 Figure 28 Example of NG AMS HTTP Redirection Response ooooooooocooooooooooooooooooooooooooooo 50 Figure 29 Example of a Distributed NGAS installation using unidirectional conditional DB replication 55 Figure 30 The functions and macros of the NG AMS C API ooooocoooooooooooooooooooooooooooooooooooooo 64 Figure 31 Using the NG AMS Python API ocoocooooocooooooooooooooooooooooooooooooococooooooooooooooocooio 65 Figure 32 Small example program using the Python API FILE ngams ngamsPClient ngamsPClientEx 66 Figure 33 Output on stdout from example program using the Python APL oocooooooooooooooooooooooooo 66 Figure 34 Function interface of a System Online Plug In cooooooooooocooooooooooooooooooooooo 68 Figure 35 The NG AMS Physical Disk Dictionary ooooooooooocoooooooooooooooooooooooooooooooooooooooooooo 69 Figure 36 Example System Online Plug In FILE ngams ngamsPlugIns ngamsLinuxOnlinePlugiln py 70 Figure 37 Example System Offline Plug In FILE ngams ngamsPlugIns ng
173. ovided for interacting with NG AMS are listed The specific parameters for each function are listed in connection with the function The parameters used by several functions are as follows ROSE Host name of NG AMS Server to contact port Port number used by NG AMS Server to contact timeoutSecs Timeout in seconds to apply while executing the request wait Wait for the NG AMS Server to finish the request 1 completely or return an immediate response 0 gt status Pointer to ngamsSTATUS structure containing the status of the query The functions provided by the NG AMS C API ngamsSTAT ngamsArchive const char nose const int po const float timeoutSecs const char fileUri const char mimeType const int noVersioning Cons TE wait ngamsSTATUS status Archive a file into NGAS fileUri Reference name for the file to archive mimeType The mime type of the file to archive In some cases it is not possible for NG AMS to determine the mime type of a data file to be archived e g when the file being is archived is RETRIEVEd from another NGAS Host For efficiency it is thus better to indicate the mime type to enable NG AMS to store the file directly on the target disk If ot use this c n be p t to noVersioning If set to 1 no new File Version will be generated for the file lbeing archived even though a Litre with that File ID was already existing ngamsSTAT ngamsArchiveFromMem const char Shes
174. paceNotification DataErrorNotification gt SmtpHost CDATA REQUIRED Active 011 REQUIRED gt lt ELEMENT EmailRecipient EMPTY gt lt ATTLIST Notification VLT MAN ESO 19400 2739 2 08 01 2003 42 of 126 VLT MAN ESO 19400 2739 ESO NGAMS User s Manual ANA 43 of 126 lt ATTLIST EmailRecipient Address CDATA REQUIRED gt lt ELEMENT AlertNotification EmailRecipientt gt lt ELEMENT ErrorNotification EmailRecipientt gt lt ELEMENT DiskSpaceNotification EmailRecipient gt lt ELEMENT DiskChangeNotification EmailRecipientt gt lt ELEMENT NoDiskSpaceNotification EmailRecipientt gt lt ELEMENT DataErrorNotification EmailRecipientt gt SN The Host Suspension Element is used to define the conditions for suspending an NGAS Host and for waking up a suspended NGAS Host Attributes IdleSuspension Switches on off host idle suspension 0 Off 1 On IdleSuspensionTime The idle time that must elapse for an NGAS system to suspending itself given in seconds WakeUpServerHost Name of server that should be requested to wake up a suspended NGAS Host SuspensionPlugIn Name of plug in to be invoked to suspend the NGAS Host WakeUpPlugIn Name of plug in to be invoked to wake up a suspended NGAS Host WakeUpCallTimeOut Time out for waiting for an NGAS Host being woken to wake up lt ELEMENT HostSuspension EMPTY gt lt ATTLIST HostSuspension IdleSuspension 011 OM IdleSu
175. parDic compression stagingFilename if exitCode 0 errMsg ngamsFitsPlugIn Problems during data handling Compressing the file failed raise exceptions Exception errMsg stagingFilename stagingFilename comprExt Remember to update the Temporary Filename in the Request Properties Object regPropsObj setStagingFilename stagingFilename info 2 File compressed Generate status info 4 Generating status format application x cfits fileSize ngamsPlugInApi getFileSize stagingFilename info 3 DAPI finished processing of file return ngamsPlugInApi genDapiSuccessStat diskInfo getDiskId relFilename dpId fileVersion format fileSize uncomprSize parDic compression relPath diskInfo getSlotId fileExists complFilename Figure 44 Example Data Archiving Plug In FILE ngams ngamsPlugIns ngamsFitsPlugIn py VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 83 of 126 16 EXPERT The Register Plug In The Register Plug In is used when executing a REGISTER command see Section 27 8 to handle the processing and extraction of information from a data file which is being registered The plug in is very similar to the Data Archiving Plug In Chapter 15 but due to a few minor differences it has been chosen to define explicitly a new type of plug in for the purpose of registering files The main difference between registering and archivi
176. parameter is not specified no Verbose Log output is produced The format of the Verbose Logs is as follows lt ISO 8601 time stamp gt lt module gt lt method gt lt line no gt lt log type gt gt lt log message gt The various values for lt type gt are defined in connection with the Local Log File An example of Verbose Log output is lowest Log Level 1 2002 12 23T14 38 53 380 ngamsServer py handleHttpRequest 950 Thread 1 INFO gt Handling HTTP client_address 134 171 21 30 34398 method POST path ARCHIVE user content disposition attachment filename home ngasmgr tmp WFI 2001 09 15T22 3A49 3A07 652 fits wait 1 no_versioning 0 content type ngas archive request content length 141546240 2002 12 23T14 38 53 380 ngamsCmdHandling py cmdHandler 893 Thread 1 INFO Received command ARCHIVE 2002 12 23T14 38 53 390 ngamsCmdHandling py handleCmdArchive 580 Thread 1 INFO gt Handling Archive Push Reguest 2002 12 23T14 38 53 390 ngamsArchiveUtils py dataHandler 586 Thread 1 INFO Archiving file WFI 2001 09 15T22 49 07 652 fits with mime type image x fits 2002 12 23T14 38 59 230 ngamsFitsPlugIn py ngamsFitsPlugIn 129 Thread 1 INFO Plug In handling data for file with URI WFI 2001 09 15T22 49 07 652 fits 2002 12 23T14 39 48 410 ngamsArchiveUtils py dataHandler 659 Thread 1 INFO NGAMS INFO FILE ARCHIVED 4020 INFO Successfully archived file with WIRE 8 WFI 2001 09 TITL AO U
177. pe ngas archive reguest host acngastl content length 69120 ngamsServer py handleHttpRequest 950 Thread 2 ZOO2Z BSAA S Mele OO TAG 150 INFO Received command ARCHIVE ngamsCmdHandling py cmdHandler 889 Thread 2 2092 AAL NA 3002205 GSO INFO Handling Archive Push Request ngamsCmdHandling py handleCmdArchive 580 Thread 2 2002 12 19T17 00 20 670 INFO Archiving file SmallFile fits with mime type image x fits ngamsArchiveUtils py dataHandler 586 Thread 2 2002 LAMPS OOSAO VEN T FINFO T Baktrj inn handling data fon PRLS wrt UMK SMAL IFITEV lie ngamsFitsPlugIn py ngamsFitsPlugIn 129 Thread 2 2002 12 19T17 00 22 540 INFO NGAMS_INFO_FILE_ARCHIVED 4020 INFO Successfully archived file with URI SmallFile fits Time 1 874s ngamsArchiveUtils py dataHandler 659 Thread 2 UNIX Syslog It is possible to instruct NG AMS to produce log entries into the UNIX syslog This is only done when certain important events occur Such events are error conditions and handling of archive requests Whether or not to log into syslog is specified in the configuration file CFG NgamsCfg Log SysLog It is possible to specify an ID which is written in each log entry in the syslog CFG NgamsCfg Log SysLogPrefix This makes it possible to filter out logs for a certain context at a later stage An example of some syslog entries produced by NG AMS is Feb 20 12 58 04 w2p2nbu python DFSLog 2002 02 20T12 58 04 200 Error w2p2nbu NGAMS ER DISK I
178. perform processing requests on this NG AMS installation 011 Allow clients to carry out REMFILE and REMDISK commands 0 1 If a server which is contacted with a Retrieve Request finds a file to be retrieved on another NGAS Host it will act as proxy and send back the file to the requestor if possible Period of time the Janitor Thread is suspended after each iteration Should be given as lt days gt T lt hours gt lt minutes gt lt seconds gt Enable disable Back Log Data Buffering 0 1 Directory that will host the back log directory where data files are buffered for later handling gt lt ELEMENT Ngams EMPTY gt lt ATTLIST Ngams ArchiveName CDATA REQUIRED PortNo CDATA REOUIRED SwVersion CDATA uy OnlinePlugIn CDATA REOUIRED OnlinePlugInPars CDATA REOUIRED OfflinePlugIn CDATA REOUIRED OfflinePlugInPars CDATA REOUIRED LabelPrinterPlugIn CDATA IMPLIED LabelPrinterPlugInPars CDATA IMPLIED Replication CDATA IMPLIED BlockSize CDATA REOUIRED Simulation 011 WN MountRootDirectory CDATA REOUIRED AllowArchiveReg 011 REOUIRED AllowRetrieveReg 011 REOUIRED E S O NG AMS User s Manual AllowProcessingReq COTES REQUIRED AllowRemoveReq 011 REQUIRED ForceProxyMode 011 IMPLIED JanitorSuspendTime CDATA REQUIRED BackLogBuffering COLEN IMPLIED BackLogBufferDirectory CDATA IMPLIED gt Mai The Db Element defines properties for the interaction with the NGAS DB Attributes Servers Na
179. plained in Table 28 DiskID File Unspecified Specified Unspecified Clone one file with the given ID Latest version of the file is taken Specified Specified Unspecified Clone one file stored on the given disk Latest version on that disk is taken Unspecified Specified Specified Clone all files found with the given File Version Storage location Disk ID is not taken into account Specified Specified Specified Clone one file on the given disk with the given File Version Specified Unspecified Unspecified Clone all files from the disk with the given ID Specified Unspecified Specified Clone all files with the given File Version from the disk with the ID given Unspecified Unspecified Specified Illegal Not accepted to clone arbitrarily files given by only the File Version Unspecified Unspecified Unspecified Illegal No files specified Table 28 Rules applied when selecting files for cloning The only way in the present version to abort a Clone Request in progress is to send an OFFLINE force to the server a CLONE abort might be provided at a later stage Note in this version of NG AMS it is not possible to specify a target disk for the cloning This will be provided at a later stage The parameters for the CLONE command are PEN file_id lt file ID gt See Table 28 The ID of the files to consider file_version lt file version gt See Table 28 The File Version of the files to consider disk_id lt disk
180. ple shown the NG AMS Server will generate a reply after having handled the Archive Request Example 3 NGAS Node to NGAS Node Archiving As a small curiosity this example shows an Archive Pull Request whereby the file URI specified is referring to a file located on another NGAS Node http ngas1 7878 ARCHIVE mime_type application fits amp file_uri http ngas2 7878 RETRIE VE file_id X YZ 2002 02 01T02 23 41 342 In this example the NG AMS Server handling the Archive Request will pick up the file from the remote NGAS Host using the file URI which in this case is a Retrieve Request and will archive it 10 The HTTP query string must be encoded according to the specification of the HTTP protocol Here they are shown un encoded VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 5 08 01 2003 119 of 126 27 2 CLONE Command Copy Files The CLONE command is used to create copies of single files or sets of files In order for the CLONE command to be accepted by an NG AMS Server the system must be configured to accept Archive Reguests Also enough free disk space must be available in the NGAS Host handling the request If the files to be cloned are located on other NGAS Hosts these will be reguested automatically during the cloning if possible The files to be cloned are selected based on the parameters File ID Disk ID and File Version The interpretations of the various combinations of these parameters are ex
181. r 1001 Log Type ERROR Log Text No mime type extension mappings defined in configuration file s Element MimeTypes There are no mime types Data Handling Plug In mappings defined in the NG AMS Configuration Pa These are necessary in order to have each type of data file properly handled by NG AMS Log ID INGAMS_ER_MISSING_ELEMENT Log Number 1002 Log Type ERROR Log Text Could not find element s in NG AMS Configuration Yos Must be specified The element referred to in the log text was not found in the configuration file as expected Must be specified in order to run the NG AMS Server Description Log ID NGAMS ER CRE GLOB BAD FILE DIR Log Number 1004 Log Type ERROR eres Problem creating Global Bad Files Directory s specified in configuration file s Parameter OS TAU FileHandling GlobalBadDirLoc Decius The Global Bad Files Directory could not be created Make the parent directory writable for the p NG AMS Server host account and try again Log ID NGAMS_ER_CONF_PROP VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 105 of 126 Log Number 1005 Log Type ERROR Log Text Ps SES The value defined for the property referred is not properly defined Must define a proper value Check Description 5 5 configuration file and try again LogID NGAMS_ER_CONF FILE Log Number 1006 Log Type ERROR Log Text s An error occurred while parsi
182. r 3009 Log Type ERROR Log Text Problems occurred initializing NG AMS Server Error message s An error occurred while initializing the NG AMS Server The server could not be prepared for Besa execution and was terminated Consult the NG AMS Logs for further information VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Isu 08 01 2003 108 of 126 Log ID INGAMS_ER_MULT_INST Log Number 3010 Log Type ERROR Apparently an instance of the NG AMS Server is running or the server was not shut down properly Log Text 5 Te previously If it is desirable to force the server to start use the force command line parameter Another instance of the NG AMS Server may be running already as it was intended to start a session Having several NG AMS Servers running in parallel may cause conflicts If it is the intention to star a Description new session the previous server should be terminated Otherwise if it is the intention to run two servers another communication port number could be specified A new session can be forced started using the force command line option LogID_ INGAMS ER INIT LOG Log Number BOLD Log Type ERROR TA Log Text Problem setting up logging properties Check if log file s can be created Exception s Ea A problem occurred setting up the properties for the logging Check the configuration file and or the Description ON command line input parameters and try again
183. rate status info 4 Generating status fileSize ngamsPlugInApi getFileSize stageFile if stageFile find zZ 1 format application x cfits compresion compress elif stageFile find gz 1 format application x gfits compresion gzip elses format image x fits compresion uncomprSize ngamsPlugInApi getFileSize workingFile info 3 Register Plug In finished processing of file return ngamsPlugInApi genRegPiSuccessStat diskInfo getDiskId relFilename dpId fileVersion format fileSize uncomprSize compresion relPath diskInfo getSlotId fileExists complFilename Figure 45 Example Register Plug In FILE ngams ngamsPlugIns ngamsFitsRegPlugIn py VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 85 of 126 17 EXPERT The Data Processing Plug in DPPI The purpose of the Data Processing Plug In DPPI is to provide a specific type of processing to be applied on a specific type of data when data is being retrieved from an NGAS Host Processing could be as trivial as simply uncompressing a data file which is stored in compressed format It could also be far more complex and involve advanced image processing and parameter extraction How the DPPI actually processes the data is left up to the DPPI implementation The DPPI only has to obey the set of rules for interfacing as for any other plug in defined for NG AMS 171 EXPERT Interface of a DPPI
184. rgDiskInfo parDic ngamsPlugInApi parseDapiPlugInPars srvObj getCfg regPropsObj getMimeType If the file is already compressed we have to decompress it tmpFn regPropsObj getStagingFilename EE feimoimarsaaaicvel Aar eS aly fowe EPEN EnaA gA ln ngamsPlugInApi execCmd gunzip tmpFn regPropsObj setStagingFilename os path splitext tmpFn 0 stagingFilename reqPropsObj getStagingFilename comprExt if parDic has_key compression if string split parDic compression 0 compress comprExt Z elif string split parDic compression 0 gzip comprExt gz Check file size checksum checkFitsFileSize stagingFilename checkChecksum parDic stagingFilename Get various information about the file being handled dpIdInfo getDpIdInfo stagingFilename dpId dpIdInfo 1 dateDirName dpIdInfo 2 fileVersion relPath relFilename complFilename fileExists ngamsPlugInApi genFileInfo srvObj getDb srvObj getCfg regPropsObj diskInfo stagingFilename dpId dpid dateDirName comprExt If a compression application is specified apply this uncomprSize ngamsPlugInApi getFileSize stagingFilename if parDic compression VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 82 of 126 info 2 Compressing file using parDic compression exitCode stdOut stdErr ngamsPlugInApi execCmd
185. rminate In this state the server is ready to handle commands like In this state the server is busy handling one or more Archiving ARCHIVE and RETRIEVE In addition the OFFLINE fjor Data Retrieval Requests Also the STATUS command is command is accepted accepted An OFFLINE command will be rejected Table 5 NG AMS State Sub States It is possible to query the state of the server by issuing a STATUS command without parameters The reply toa STATUS command is an XML document with the following contents lt xml version 1 0 gt lt DOCTYPE NgamsStatus SYSTEM http acngastl hq eso org 7777 RETRIEVE internal ngamsStatus dtd gt lt NgamsStatus gt lt Status Date 2002 12 23T13 15 52 194 HostId acngast1 Message Successfully handled command STATUS State ONLINE Status SUCCESS SubState IDLE Version v2 0 Beta2 2002 12 04T09 22 53 gt lt NgamsStatus gt Figure 2 Response to STATUS command 2 5 The NG AMS Storage Media Infrastructure The Storage Media infrastructure used by NG AMS is depicted in Figure 3 AS Host Storage Media Storage Set 1 Admin Files 7 S Root Directory System Replication Disk NgasDiskInfo Main Disk Figure 3 The NG AMS Storage Media Infrastructure VLT MAN ESO 19400 2739 ESO Noms User s Manual Is 5 08 01 2003 19 of 126 During operation archiving disks are usually used in pairs as shown in Figure 3 This association of disks however is handled i
186. roxy transparent data access It can also send back HTTP redirection messages to indicate to the data requestor where to find the data The C and Python APIs handle the data access completely transparent for the client see Section 3 2 Access Service Restriction It is possible to enable disable some basic services via the configuration The services in questions are for the moment 1 Handling of Archive Requests 2 Handling of Retrieve Requests 3 Data Processing and 4 Remove Requests removing of disk and file information from the system see Section 3 11 Back Log Buffering of Data In case problems occur preventing NG AMS from archiving data NG AMS will Back Log Buffer data and try to handle this at a later stage see Section 3 7 Disk Registration amp Supervision When a disk first has been registered by NG AMS the movements of the disk will be monitored by NG AMS so that when it appears inan NGAS Host the NGAS DB will be updated to indicate the latest status of the disk see Section 2 7 File Registration A number of parameters are registered for the files archived in the NGAS DB making it possible to locate retrieve and process these files see the Sections 8 3 and 15 1 File Cloning A service is provided with which it is possible to clone single files sets of files or entire disks see Section 27 2 Data Subscription Service A service is provided to export data being archived to Subscribers interested A Subscriber
187. rs for the RETRIEVE command oooooooocoooooooo oooooooooooooooooooooooooooooooooooooooooooooo 122 Table 37 Parameters for the STATUS command ooooooooooooocooooo o o o oooooooooooooooooooooooooooooooooooooooooooooo 122 Table 38 Parameters for the SUBSCRIBE command ocoooocoooooooo o oooooooooooooooooooooooooooooooooooooooooooooooooooo 122 Table 39 Parameters for the UNSUBSCRIBE command ooocooooooo o oooooooooo oooo oooooooooooooooooooooooooooooooooo 123 VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 5 08 01 2003 11 of 126 1 About this Guide 11 Purpose amp Scope This document is the user s manual for the Next Generation Archive Management System NG AMS NG AMS is the SW for the Next Generation Archive System 2 It is in charge of the handling of storage media and of archiving and retrieving data files to from an NGAS Archive Numerous other services are provided for carrying out the daily operation of an NGAS Archive System This manual contains the information needed for configuring and operating NG AMS It is also described how to enhance the system with new features by adding various types of plug ins These plug ins are small Python functions with a specific interface and a specific set of tasks The audience of this document is high level users who wish to perform archiving and retrieval of data files into from NGAS However also more advanced users who need to tune and adapt the system by c
188. rst ting a dis nsta i L In a typical scenario the disks When the disk is inst in the jiibivine unit ter ition the disk is been registered by NG AMS marked as unayaila ble not mounted in the disk is now mark and will be recognized from that point on NGAS DB NGAS system 10 Recycling After the disk has served for a pile as storage media it may t cle i 4 Data Storagel zchivine Data is now being ved on the disk m e first on disks n them and disk reformatt system as a new disk 9 Data Retrieval amp Processing grr and can 5 Data Retrieval In some sce data from This is fet completed configured 8 Identification R om astering 7 Transport 6 Completion 1 Dis is received in the Archive The Main Disk si transported to the the disk NG AMS free slot in an k lable capacity late its status in the DB to online data at all times Notification Message can be pent soit The disk is marked as completed in the DB Figure 5 Life cycle of an NGAS disk Empty NGAS disks used for archiving purposes are usually kept together in pairs NG AMS however does not impose a Static association between disks and the association of non completed disks provided by the Logical Name serves mainly to make the operation of an NGAS based archive system more convenient For more information about the handling of disk association consult Section 2 5 VLT MAN ESO 19400 2
189. s amp Features Some of the main services and features provided by the NG AMS SW are e Multithreaded Server The NG AMS Server is using threads when handling requests from clients This means that it is capable of handling several requests simultaneously e HTTP Protocol The communication interface of NG AMS is based on the standard HTTP protocol This makes it easy to access the server from various clients It is even possible to interact with the server using a WEB browser see Section 3 8 e Flexible Adaptation via Configuration File The NG AMS Server is configuring itself at start up based on a large number of configuration parameters defined in the NG AMS Configuration which is an XML document This makes it possible to adapt the server for specific contexts in a flexible way see Chapter 6 e Cluster Mode NG AMS is prepared for operation of a set of NGAS Nodes in a cluster that constitutes an archive data server and processing facility see Section 4 1 e Adding of Specific Behavior Based on Plug In Concept NG AMS is implemented in a way so that only the kernel general functionality is implemented hard coded into the server SW All the context specific features are provided based on a plug in scheme making it possible to adapt the server in a very flexible way As an example of this the specific handling of data during archiving is done by a plug in provided for each type of data see the Chapters 12 21 e State Management The NG A
190. s e g relevant in an archive data server cluster configuration where no files are being archived Note that if archiving is disabled apart from the ARCHIVE Command also the CLONE Command will not be accepted In addition it will no be possible for NG AMS to handled back logged buffered data nor will it be possible to act as Data Subscriber CFG NgamsCfg Ngams AllowArchiveReg Disabling of handling of Retrieve Reguests may be applied for NGAS Node used as Retrieve Reguest Handling archiving units where it is not desirable that handling of external data retrieval disturbs loads the system CFG NgamsCfg Ngams AllowRetrieveReg In connection with a Retrieve Reguest it is possible to specify that data processing should be applied on the data before replying to the reguestor This may be relevant to avoid to load an Processing Reguest Handling NGAS Host too much if handling of the Retrieve Reguests themselves is high priority and where processing would load the system too much to get access to the data within a limited period of time CFG NgamsCfg Ngams AllowProcessingReg If this feature is disabled no REMFILE and REMDISK commands will be accepted by the NGAS Host and it is thus not possible to delete any information in the NGAS system This should usually be applied e g for NGAS Nodes operating in an NGAS data server cluster CFG NgamsCfg Ngams AllowRemoveReg Archive Request Handling Remove Request Handling Tabl
191. sPlugIns ngamsGenCrc32 py ooooooooooooo 90 Figure 52 Function interface of a Suspension Plug In oooooooocoooooooooooooooooooooooooooooooooooooooooo 91 Figure 53 Example Suspension Plug In FILE o oocooooooooooooooooooooooooooocoocoooooooo man cntntnnan 91 Figure 54 Function interface of a Wake Up Plug In oooooooooooocoooooooooooooooooooooooooooooooooooooooo 92 Figure 55 Example Wake Up Plug In FILE 7 cssccsssssssssscssscecsscsscsscesssscesscesescsssnsssessscsecsssesssssesnscsecssesssosee 92 Figure 56 Function interface of a Filter Plug In ooooooooooooooocooooooooooooooooooooooooooooooooooooo 93 Figure 57 Example Filter Plug In FILE ngams ngamsPlugIns ngamsMimeTypeFilterP1 py Figure 58 NG AMS Status DTD FILE ngams ngamsData nganssStatus dtd ooooocooooooooooooooooooo Figure 59 Example NGAS Disk Info file FILE lt mount root point gt lt disk mount point gt NgasDiskInfo 97 VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 9 of 126 Figure 60 Example File Info Status ooococoooooooooooooooooooooooooooooooooooooooooooio 98 Figure 61 Example of NG AMS inline documentation oooooooooooooooooooooooooooooooooooooooooooooooo 100 Figure 62 Starting the pydoc utility as an HTTP server oooocoooooooooooooooooooooooooooooooooooooooco 101 Figure 63 The NG AMS License Conditio
192. scribe const char host const int porty Gange aelleeiic timeoutSecs const char Monel Onsite priority const char secret const char filterPlugIn const char saaerbaugn beras ngamsSTATUS SI Saus Send a SUBSCRIBE to NG AMS to subscribe for data or a specific type Or claic el rL Subscriber URL to where data is pushed Prony Priority of the Subscriber low number high priority Default vatu 1110 5 startDate Start date defining which data the subscription should take into account filterPlugIn Optional Filter Plug In to apply when selecting WALC data files to deliver to the clients filterPlugInPars Optional ErIter PLug In Parameters Oom mans mer CO ENS lgs PLUS IN void ngamsToupper char str Convert a string to upper characters Sis Pointer to Stringi tO CONVSrt char ngamsVersion void Return pointer to buffer with the NG AMS version information ngamsSTAT ngamsUnsubscribe const char host const int p rt const float timeoutSecs const char eneg ngamsSTATUS FICUS Send an UNSUBSCRIBE to NG AMS to subscribe for data or a specific type gf data Skaal Subscriber URL to where data is pushed CAUTIONS This is a first implementation of the module Changes may be introduced in order to improve the usability of the API Remember to perform a call to ngamsInitStatus right after declaring this and a call to ngamsFreeStatus after each call to one of the commands used to issue commands to
193. specified in the configuration file Target Storage Set s One or more Storage Sets on which the data can be stored Table 6 Parameters for data classification See Chapter 6 for more information about the NG AMS Configuration Storage Set Data Streams Main Disk Replication Di Archive Request Storage Set p m NG AMS Server Data File Main Disk Replication Di Data Reception amp Classification Storage Set Main Disk Replication Di Figure 4 Data channeling Two standard mime type are used by NG AMS These are text xml Used by NG AMS to indicate that a reply contains an XML document ngas archive request Generic mime type used to indicate for NG AMS that it should determine the mime type from the file URI It is also possible to specify the mime type explicitly in an Archive Request Table 7 Reserved mime types Using the NG AMS APIs see the Chapters 9 and 10 the user client normally does not have to worry about this aspect VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 21 of 126 2 7 Disk Handling Life Cycle of an NGAS Storage Media In this section the various stages in the life cycle of an NGAS disk are described In the diagram in Figure 5 a typical life cycle for an NGAS Storage Media is shown There might be differences for the various contexts to how the actual disk handling is implemented 1 Initial Registering 2 Transport 3 Recognition amp Re registering The fi
194. spensionTime CDATA 0 WakeUpServerHost CDATA ut SuspensionPlugIn CDATA WakeUpPlugIn CDATA WakeUpCallTimeOut CDATA 0O gt SI The SubscriptionDef Element is used to define the properties for Data Subscription It may contain a number of Subscription Elements indicating to which server this NG AMS should subscribe to Attributes AutoUnsubscribe Indicates if the server should un subscribe automatically when going Offline SuspensionTime Time the thread should suspend itself between attempting to deliver back logged buffered files ISO8601 BackLogExpTime Expiration time for entries in the Subscription Back Log string 1ISO8601 5 lt ELEMENT SubscriptionDef Subscription gt lt ATTLIST SubscriptionDef AutoUnsubscribe 011 uii oa SuspensionTime CDATA REQUIRED BackLogExpTime CDATA REQUIRED Enable Oa LAS Sl The Subscription Element define a data subscription It will trigger the NG AMS Server to subscribe itself to data from another NG AMS Server Attributes HostId Host ID of the Data Provider PortNo Port number used by the Data Provider PFIORICYS Priority requested for this Subscription The lower the number the higher the priority Default is 10 If other Subscribers subscribe with a higher priority then this client they will be served faster SubscriberUrl Subscriber URL which is issued when subscribing The Data Provider will use this URL when delivering the data to the Selociciewesie Fi
195. sser CPU the check will consume At the same time it will take more time to carry out the check of the data holding Indicates the minimum allowed time for carrying out one check cycle ddThh mm ss If this is specified e g to 24 hours and one check cycle would only take 11 hours the check would be suspended for 13 hours before running again Used to indicate if disks should be checked sequentially ordered according to the Slot ID or randomly SEQUENTIAL RANDOM Used to indicate if files on a disk should be checked VLT MAN ESO 19400 2739 2 08 01 2003 41 of 126 sequentially ordered according to the ingestion date or randomly SEQUENTIAL RANDOM DataCheckLogSummary If set to 1 a summary log info will be generated each time a complete check of the data holding of one NGAS Host has been carried out lt ELEMENT FileHandling EMPTY gt lt ATTLIST FileHandling ProcessingDirectory PathPrefix ChecksumPplugIn ChecksumP lugInPars DataCheckActive DataCheckPrio DataCheckMinCycle DataCheckDiskSeq DataCheckFileSeq DataCheckLogSummary CDATA CDATA CDATA CDATA CHED CDATA CDATA SEQUENTIAL RANDOM SEQUENTIAL RANDOM 011 REQUIRED REQUIRED REQUIRED REQUIRED REQUIRED REQUIRED REQUIRED REQUIRED REQUIRED REQUIRED gt ese The Stream Element defines properties for the handling of data streams into the NGAS System The StorageSetRef contains an attribute with a reference to a
196. t be handled by NG AMS when in the State Sub State as indicated in the Description error message Bring the system to one of the allowed States Sub States listed in the error message and try again Log ID NGAMS_ER_ILL_CMD Log Number 4009 Log Type ERROR Log Text Illegal command Yos received Rejecting reguest Description The command issued is not known accepted by the NG AMS Server Check context and try again ILogID INGAMS ERILL REQ Log Number 4010 Log Type ERROR 000 Log Text This NG AMS is not configured for accepting s Requests Rejecting request DEER This installation of NG AMS is not configured for accepting the request type as specified in the error P message Re consider the request issued or to configure the server to handle the given type of request ILogID INGAMS ER RETRIEVE CMD A A Log Number 4010 Log Type ERROR eess COsSS S sS Log Text Incorrect parameter given for RETRIEVE command 22222 Description The syntax specified for the RETRIEVE command is illegal LlogID NGAMS_ER_UNKNOWN_FILE gt Log Number 4012 Log Type ERROR 00 Log Text File with ID s is not available Rejecting request DEER It was requested by a client to retrieve information about a certain file or to return the file This file p however is not available Check the File ID and try again Log ID NGAMS_ER_UNKNOWN_DISK VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01
197. t of CPU time and may interfere with more urgent Archive or Retrieve Requests When a client has first subscribed itself to a certain type of data NG AMS guarantees that all files of that type and matching the time constraint will be delivered to the client If it is impossible to deliver a file if e g the client has terminated execution or due to interruption of the network connection NG AMS will back log buffer the data in the Subscription Back Log and try periodically to deliver the data to the client Even if the Storage Media hosting the files to deliver to the client are removed from the Data Provider NGAS Host the files will be delivered since the Subscription Back Log is located in a separate area Note that normally the Back Log Area should be located on one of the permanent disks of the NGAS Host to facilitate this scheme Beware that if files cannot be delivered during a longer period of time the back log storage area may fill up with Back Log Buffered files It is possible to specify an expiration period of time indicating for how long time data should be kept in the Subscription Back Log CFG NgamsCfg SubscriptionDef BackLogExpTime Data residing longer than the expiration time will be deleted and thus never delivered The name of the Subscription Back Log is as follows lt NgamsCfg Ngams BackLogBufferDirectory gt subscr back log A table in the NGAS DB is used to keep track of this Subscription Back Log see Section 8 5
198. t send the SUBSCRIBE command In order to un subscribe it must send the UNSUBSCRIBE command In the context of NG AMS the term disk refers to a random access storage device which can be mounted under UNIX and which has a file system created on it A dictionary containing information for the Storage Media available in an NGAS Host A consisting of one or two Storage Media see also Storage Set An association between two non completed Storage Media This association exists as long as NG AMS is Online Le there is no static link between NGAS Storage Media A human readable name that is used when referring to disks The disks will typically be labeled with the Logical Name Should be unique although this may not be guaranteed as for the Disk ID The copy of the data file stored on the Main Storage Area Data is archived onto Storage Sets The Main Disk is the primary Storage Media of the set and must also be present The array of HDDs in an NGAS Node which when filled with data are send to the Archive Facility Site The central process of NGAS It receives the data file from the Data VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual 1 i 08 01 2003 14 of 126 NGAS Cluster NGAS Configuration NGAS Host NGAS Node Physical Disk Dictionary Processing Area Production Site Production Site Data Production Site Replication Data File Replication Storage Area Staging Area Storage Med
199. ter label i AN meei tone derinition riss MN par tone le MN cannot generate disk label label error errMsg ngamsP lugInApi notify srvObj getCfg NGAMS NOTIF ERROR ngamsBrotherPT9200DxPlugIn ILLEGAL CHARACTER REQ FOR PRINTING errMsg VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 75 of 126 raise exceptions Exception errMsg printerCode printerCode fontDic label i printerCode printerCode fontDic Trailer Generate printer file write printer control code printerFilename tmp ngamsLabel_ ngamsPlugInApi genNgasId srvObj getCfg prn fo open printerFilename w fo write printerCode fo close Write the printer code file to the device res ngamsPlugInApi execCmd cat tprinterFilename gt parDic dev if not srvObj _getUnitTest os system rm f printerFilename ata ires 0 tE errMsg Problem occurred printing label error errMsg ngamsPlugInApi notify srvObj getCfg NGAMS_NOTIF_ERROR ngamsBrotherPT9200DxPlugIn PROBLEM PRINTING LABEL errMsg raise exceptions Exception errMsg info 2 Executed plug in ngamsBrotherPT9200DxPlugIn with parameters pingInPars an ra bea p Aabel E Iya name _ main ea Main function wen See OG COME Or tu any if len sys argv 3 print nCorrect usage is n print python ngamsBrotherPT9200DxPlugIn lt NGAMS CFG gt lt text gt n sys
200. ter AEE Force the system to Offline State even though an action is in progress like file cloning Table 31 Parameters for the OFFLINE command Usage of the force option should be done with great care as operations may be interrupted before termination leaving the system in an undefined condition 27 7 ONLINE Command Bring System to Online State The ONLINE command is used to make the NG AMS Server go Online The ONLINE command does not accept any parameters 27 8 REGISTER Command Register Existing Files on a Disk The REGISTER command is used to register files already stored on an NGAS disk It is possible to register single files or entire sets of files by specifying a starting path from which NG AMS will look for files Only files that are known to NG AMS with a mime type defined in the configuration will be taking into account It is also possible to explicitly specify a comma separated list of mime types that will be registered Files with other mime types than specified in this list will be ignored Mandatory mime_type No Comma separated list of mime types to take into account A single mime type can also be specified path Yes The starting path under which NG AMS will look for candidate files to register It is also possible to specify a complete path whereby only a single file will be registered Table 32 Parameters for the REGISTER command As a response to the REGISTER command and Registration R
201. the log definition file This is mostly intended for maintenance trouble shooting purposes Retrieve the contents of the NG AMS Local Log File see 3 3 processing_pars lt DPPI gt With this parameter it is possible to specify a DPPI which is invoked to process the data before sending it back NG AMS will send back the result of the processing and not the original file VLT MAN ESO 19400 2739 ESO Noms User s Manual Is 5 08 01 2003 122 of 126 Table 36 Parameters for the RETRIEVE command It is possible to receive an HTTP redirection response as response to the Retrieve Request In this case the client must re send the Retrieve Request to the alternative URL given in the redirection response See also Section 7 3 27 12 STATUS Command Query System Status amp Other Information The STATUS command is used to query various status information from the NG AMS Server The STATUS command accepts the following parameters Mandatory lt no parameters gt No In this case a reply is returned which contains an NG AMS Status document An example of such a status document can be found in Section 3 8 N disk_id lt disk ID gt o Query information about a disk referred to by its Disk ID The reply is an NG AMS Disk Status XML document An example of this can be found in Section 22 2 file_id lt file ID gt No Query information about a file with a given File ID The reply is an NG AMS XML Status document as shown in Section
202. thon source and in particular the DAPI function itself is shown in Figure 43 Module Structure style documentation string with an overall description of the module It is then possible to browse the documentation e g with pydoc on he DAPI a sually following should be imported from ngams import import ngamsPlugInApi ngamsDiskUtils ngamsDiskInfo understand it may be tried to decrease the length of the code of the DAPI function by making a number of utility functions to be used within the DAPI This is up to the DAPI implementation however with a predefined interface NG AMS imports the DAPI module and invokes the DAPI of the same name as the DAPI module with the objects needed for the data handling Plug In Function Structure parameters which are objects providing the necessary information about the file See also 15 1 Python documentation string for the DAPI further processing it is often necessary to extract a number of features about the file TA advisable to check the data is consistent which the file will be stored on the Target Storage Set This could be generated based on information in the file itself determine which kind of processing should be carried out on the file before archiving it parameters to NG AMS This should be done with the convenience function ngamsPlugInApi genDhpiSuccessStat Figure 43 Typical structure of a DAP
203. tigated where the instances of the given file are located Three cases are considered 1 Local Host The file is stored on the NGAS Host handling the request 2 Private Network The file is stored on an NGAS Host within the private network IP address of the format 10 X X X 3 Remote Location The file is stored on a remote NGAS Host not contained within private network Le if a file with a given ID and the same version is found on several NGAS Hosts the selection criteria for which file to take is done according to the priority list described in this paragraph Le if a file is found on the local host and on a remote host the instance on the local host is taken e When a file is found to be residing on the local host or on an NGAS Host within the private network NG AMS always gets the file processes it if requested and sends it back to the requestor If the file is located on a remote host NG AMS will either send back an HTTP redirection response HTTP Status Code 303 or it can act as VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 23 of 126 proxy In the former case it is up to the requestor to re issue the request to the NGAS Host referenced in the redirection HTTP response see also Section 7 3 In the latter case it retrieves the file from the remote location and subsequently sends back this file to the requestor Whether an NG AMS Server should act as a proxy or not is configurable CFG NgamsCfg Nga
204. tion CFG NgamsCfg Log For further information about this specific properties see Chapter 6 3 4 Email Notification Apart from the various types of logging described in Section 3 3 it is also possible to instruct NG AMS to notify various recipients about important events occurring via email The various types of Notification Message are An Alert Message is generated as a result of a serious problem encountered Such a problem Alert Notification may not be recoverable and it is likely necessary to do some manual intervention Normally preventative actions should be undertaken immediately z Fi An Error is the result of a problem encountered which is not of a very severe character Error Notification Saat een oh ors AA Often an error situation is provoked by an external request which is illegal for some reason VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual 1 08 01 2003 25 of 126 Could e g be that it is tried to archive a file when the system is in Offline State Depending on the type of error intervention should be undertaken ASAP A Disk Space Notification is sent out when a certain threshold of minimum free disk capacity is reached This message is meant only as a warning indicating that the Storage Set is about to be full No actions are needed apart from maybe verifying that Storage Sets with free disk space are available A Disk Change Notification is send out to indicate that a Storage Set is fu
205. tion Plug In to handle registration of FITS files srvObj Reference to NG AMS Server Object ngamsServer regPropsObj NG AMS request properties object ngamsRegProps Returns Standard NG AMS Data Archiving Plug In Status as generated by ngamsPlugInApi genDapiSuccessStat ngamsDapiStatus wen info 1 Plug In registering file with URI regPropsObj getFileUri diskInfo reqPropsObj getTargDiskInfo parDic ngamsPlugInApi parseRegPlugInPars srvObj getCfg reqPropsObj getMimeType stageFile reqPropsObj getStagingFilename If the file is already compressed we have to decompress it IE Stage ane an oa Wes ai eng Stage riren PINa AEN a aly na workingFile procDir ngamsPlugInApi prepProcFile srvObj getCfg stageFile ngamsPlugInApi execCmd gunzip workingFile if working ae na IS Tys workingFile workingFile 2 CAR workingFile workingFile 3 elses workingFile stageFile VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 84 of 126 Check file size checksum ngamsFitsPlugIn checkFitsFileSize workingFile ngamsFitsPlugIn checkChecksum parDic workingFile Get various information about the file being handled arcFile dpId dateDirName ngamsFitsPlugIn getDpIdInfo workingFile fileVersion relPath relFilename complFilename fileExists ngamsPlugInApi genFileInfoReg srvObj getDb srvObj getCfg regPropsObj diskInfo stageFile dpId Gene
206. to determine whether to deliver this to the client or not plug_in_pars lt pars gt A set of parameters to transfer to the Filter Plug In when it is invoked priority lt prio gt Priority for delivering data to this Data Subscriber The lower the number the higher es data delivery the priority Clients with a higher priority get more CPU time in connection with the start_date lt ISO8601 gt Date from which the data to deliver is taken into account If not specified the time when the SUBSCRIBE command was received is taken as start date url lt delivery URL gt ig The URL to which the data will be delivered On the client side a corresponding HTTP server must be ready to receive requests data via the given URL Table 38 Parameters for the SUBSCRIBE command Note a Data Subscriber that has subscribed remains subscribed also if the HTTP server which handles the receiving of data on the client side terminates NG AMS will back log data for that client and when it re subscribes the back logged data will be delivered If it not desirable the client should un subscribe it self by submitting an UNSUBSCRIBE command 27 14 UNSUBSCRIBE Command Unsubscribe a Previous Data Subscription Used by Data Subscribers to un subscribe a previously established subscription for data If NG AMS holds back logged buffered subscription data for the client from a previous subscription the Subscription Back Log will be reset The parameter of the U
207. trieve Reguest see also Section 3 2 In addition NG AMS distinguishes between NGAS Nodes globally accessible and nodes within a private network With this simple scheme it is possible to build up e g a hierarchical cluster as the one shown in Figure 13 Retrieve Request Private Network Cluster Back Bone Network 6 NGAS Super Node Proxy Mode 5 2 Network Switch NGAS Main Node 2 NGAS NGAS NGAS Sub Node Sub Node Sub Node 10 X X X 10 X X X 10 X X X NGAS Sub Node 10 X X X NGAS NGAS Main Node 1 9 9 q 9 9 4 q q d 4 d Main Node 3 ASS SSS SSs SSS Network Network Switch Switch 4 13 NGAS NGAS NGAS NGAS NGAS NGAS NGAS Sub Node Sub Node Sub Node Sub Node Sub Node Sub Node Sub Node 10 X X X 10 X X X 10 X X X 10 X X X 10 X X X 10 X X X 10 X X X 223 5 AAA FEE Figure 13 Example of hierarchical NGAS Cluster In the cluster shown in Figure 13 the main entry point of the NGAS Cluster and the only one for that matter is the NGAS Super Node All reguests must pass through this node When a Retrieve Reguest is received by the super node 1 it will identify that the file reguested is located on the NGAS Node high lighted in the figure This it finds out from the ngas files and ngas disks tables in the NGAS DB DB ngas files disk id ngas disks host id From the IP address of the host of interest DB ngas hosts ip address the NG AM
208. tring cmdCode Camel S8 See ngamsCmd2No void ngamsDumpErrStdout const ngamsSTATUS status Dump an error message La Ld SEAL Gicwibiercwugs on Selo status Pointing to instance of the ngamsSTATUS structure containing the information to print out char ngamsEncodeUrlVal const char urlVal const int skipScheme Encode the value given as input parameter to replace special characters to make the value suitable for usage in a URL urlVal Value to be encoded skipScheme If the value is initiated with an HTTP scheme ftp http file this will not be encoded ie Ul ag 19 Be GO te void ngamsFreeStatus ngamsSTATUS iS techie sp Free the memory occupied by the status object status Pointing to instance of the ngamsSTATUS structure containing the information to print out void ngamsInitStatus ngamsSTATUS Stachel sy Initialize the ngamsSTATUS structure making it ready for usage within the NG AMS C API functions status Pointing to instance of the ngamsSTATUS structure containing the information to print out VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 61 of 126 int ngamsIsDir const char filename Return 1 if the filename given is a directory otherwise 0 is returned filename Name of directory to probe ngamsSTAT ngamsExit const char sae COasw INE port const Float timeoutsegs const int wait ngamsSTATUS Sie lie Send an EXIT command to the NG AMS Server to make it
209. ttributes ArchiveName PortNo SwVersion OnlinePlugIn OnlinePlugInPars OfflinePlugIn OfflinePlugInPars Replication BlockSize Simulation MountRootDirectory AllowArchiveReg AllowRetrieveReg AllowProcessingReg AllowRemoveReg ForceProxyMode JanitorSuspendTime BackLogBuf fering BackLogBufferDirectory Name of the archive e g ESO VLT Port number to use for the NG AMS HTTP server If defined NG AMS will compare the value of this attribute with the version ID of NG AMS If these are not completely identical an error is reported and the NG AMS Server will perform an exit If the SwVersion is not specified set to this check is not done Plug In utility invoked by NG AMS when going Online to prepare the system and to obtain the information about the current disk configuration and status of the disks Input parameters to the Online Plug In Plug In utility invoked by NG AMS when going Offline to prepare the system for standby mode Input parameters to the Online Plug In Indicates if file replication should be carried out by this NG AMS 0 1 Block size applied when receiving and sending data via HTTP bytes Simulation system 1 otherwise 0 Base directory used as root directory when mounting the disks Allow clients to issue Archive Requests to the system 011 Allow clients to retrieve data from this NG AMS installation 0 1 Allow clients to
210. ucture ooooooooooocooooooooooooooooooooooooooooocooooco 99 EXPERT Online Browsing of NG AMS Inline Python Documentation oooooooooooooooooooooo 100 eee INSTALLATION oo momen an an anna 102 NG AMS LOG AND ERROR MESSAGES DEFINITION o oom 104 NG AMS LICENSE CONDITIONS o oooo mna nana 117 NG AMS COMMAND isa saman anna maa 118 ARCHIVE Command Archive Data Files ooooooooooooooooooooooooooooooooooooooooooooooooooooo 118 CLONE Command Copy Files o ocoocooocoooooooooooooooooooooooooooooooooooooooooooooooooooo 119 EXIT Command Terminate Server oooooooooocooooooooooooooooooooooooooooooooooocoocooooooooco 119 INIT Command Re Initialize the System o oooooooooocoooooooooooooooooooooooooooooooooooooooocooocoo 119 LABEL Command Generating Disk Labels oooocoooooocoooocooooooooooooooooocooooooooooocoo 119 OFFLINE Command Bring System to Offline State oocoooocoooooooooooooooooooooooooooooooooo 120 ONLINE Command Bring System to Online State coocoooocoooooooooooooooooocooooooooooooo 120 REGISTER Command Register Existing Files on a Disk ooooooooooooooooooooooooooooooooooocooocoo 120 VLT MAN ESO 19400 2739 ESO NGIAMS User s Manual Is 08 01 2003 7 of 126 27 9 REMDISK Command Remove Information about Disks ooooooocooooooooooooooooooic 120 27 10 REMFILE Command Remove Files from the System oooo
211. ug in Priority Used by Subscribers to give another priority than the default one 10 A low number means a high priority Name of Data Processing Plug In to execute on the d ta betore sending Lt back to the requestor Optional parameters to be handed over to the DPPI These should normally be given on the format parl val par2 val2 although it is up to the DPPI to interpret these Slot ID for which to generate label Date in ISO8601 format indicating a lower limit in time URL Used for the SUBSCRIBE command to indicate to where NG AMS should deliver data subscribed for he parameter combinations valid for each command are fileUri lt uri gt mimeType lt mt gt noVersioning fileId lt id gt v diskId lt id gt v fileVersion ver None None SGiloie cl Kale force None path path diskId id execute diskId lt id gt execute fileId lt id gt fileVersion lt ver gt fileId lt id gt fileVersion lt ver gt outputFile lt of gt processing lt DDPI gt processingPars lt pars gt internal lt filename gt ngLog 5 VLT MAN ESO 19400 2739 2 E S O NG AMS User s Manual A 37 of 126 STATUS None SUBSCRIBE Sona uiel foeni Gore pas vameDateksstantndate TEOJ G0I I filterPlugIn lt filter plug in I plugInPars lt filter plug in pars gt UNSUBSCRIBE Shell suris c ESO DMD 2001 2002 NGAS Project http archive
212. uspending itself after the specified suspension time out has elapsed CFG NgamsCfg HostSuspension IdleSuspensionTime it invokes the specified Suspension Plug In CFG NgamsCfg HostSuspension SuspensionPlugIn which actually carries out the actions needed to suspend the host see also 4 3 In the simplest case the Suspension Plug In might simply invoke a shutdown command on UNIX as root to shut down the host but in principle there are no limitations to which kind of actions that are performed 19 1 EXPERT Interface of a Suspension Plug In The plug in must be contained in a Python module which has a function of the same name as the module The latter is the actual plug in which is invoked by NG AMS A Suspension Plug In has an interface as shown in Figure 52 srvObj Instance of the ngamsServer class used by this session See also Section 12 1 def cplug in name srvobj e Suspension Plug In Function Name The name of the plug in must be identical to that of the Python module hosting the plug in function Figure 52 Function interface of a Suspension Plug In A Suspension Plug In does not return control to the NG AMS Server but will in fact terminate this after having done various clean up that might be necessary 19 2 EXPERT Example Data Checksum Plug In Under development Figure 53 Example Suspension Plug In FILE VLT MAN ESO 19400 2739 ESO NGAMS User s Manual AA 92 of 126
213. utSecs const har xfileId CONSE ane fileVersion const char processing CONSE CRAT processingPars const char targetFile ngamsSTATUS SS Send a RETRIEVE command to the NG AMS Server to retrieve a data file and store this din el sesLILS Cin ERE Toca disk fileId ID of the file to retrieve fileVersion Jose version of file tee retrieve INE SSE O latest version will be retrieved processing Name of DPPI to be invoked by NG AMS when data is retrieved processingPars Optional list of parameters to hand over to the DPPI targetFile If a valid filename is specified the data retrieved will be stored in a file with that name If a directory is given the data file retrieved will be stored in that directory with the name under which it is stored in NGAS If this parameter is an empty string it will be tried to stored the file retrieved under the NGAS archive name in the current working directory ngamsSTAT ngamsStat2Str const ngamsSTAT statNo ngamsMED BUF statStr Convert a status code ngamsSTAT to a readable string statNo Siselcus as Code Siecle siesta Status as string VLT MAN ESO 19400 2739 E S O NG AMS User s Manual 2 08 01 2003 63 of 126 ngamsSTAT ngamsStatus const char lO Site const int port const float timeoutSecs ngamsSTATUS 5 Sie Sle Ss Send a STATUS command to NG AMS to query the current status of the system No parameters are defined at present ngamsSTAT ngamsSub
214. will be of major advantage depending on the complexity of the tasks performed by the plug ins An overview of the NG AMS SW is given in Chapter 23 VLT MAN ESO 19400 2739 ESO NGAMS User s Manual Re 68 of 126 12 EXPERT The System Online Plug In The purpose of the System Online Plug In is to prepare the system for the Online State where it must be fully operational according to the configuration During this phase the storage disks are usually mounted and possibly checked for proper functioning and accessibility A very essential task of a System Online Plug In is to generate the so called Physical Disk Dictionary This contains the physical information about the disks installed in an NGAS Host The plug in is invoked by NG AMS when it is going Online i e either when it has received an ONLINE command or when it has been started with the autoOnline command line parameter The actual implementation depends highly on the context HW in use and other specific requirements in connection with an NGAS Node The System Online Plug In is not executed when running in Simulation Mode see Section 3 6 121 EXPERT Interface of a System Online Plug In The System Online Plug In must be contained in a Python module file which has a function of the same name as the module The latter is the actual plug in which is invoked by NG AMS A System Online Plug In has an interface as shown in Figure 34 session For the context of a plug in
215. ynchronize The synchronize flag is used to indicate if the two disks of a set should be considerd as locked together so that when one is completed also the other is marked as completed although it may not be integer 0 1 lt ELEMENT StorageSet EMPTY gt lt ATTLIST StorageSet StorageSetId REQUIRED DiskLabel we MainDiskSlotId REOUIRED RepDiskSlotId IMPLIED Mutex OW Synchronize Was VLT MAN ESO 19400 2739 2 08 01 2003 40 of 126 E S O NG AMS User s Manual The FileHandling Element defines properties for the file handling Attributes ProcessingDirectory PathPrefix ChecksumPlugIn ChecksumPlugInPars DataCheckActive DataCheckPrio DataCheckMinCycle DataCheckDiskSeq DataCheckFileSeq Base directory in which temporary files will be stored during processing A prefix which will be used when building up the target filename Could e g be used as to build a filename like lt mount point gt lt PathPrefix gt lt add path gt lt filename gt lt but it is up to the Data Handling Plug In to use it in a way appropriate for the context DCPI Data Checksum Plug In checksum of a data file which generates the Input parameters to the DCPI Enable disable Data Check Service 1 activate Priority of Data Check Thread 1 highest priority with this parameter it is possible to adjust the amount of CPU power the Data Check Thread should consume The higher the number the le

ESO NG/AMS - User`s Manual

Contents

Download Pdf Manuals

Related Search

Related Contents