COMPSs User Manual: Application Execution
Contents
1. 0 Figure 10 resources log generated by the execution of the Simple java application COMPSs 4 0K simple Simple 03 4 0K jobs 0 job1i_NEW err 380 job1_NEW out 612 resources log 70K runtime log 4 0K tmpFiles Figure 11 Structure of the logs folder for the Simple java application in debug mode Furthermore when running runcompss with additional flags such as monitoring or tracing additional folders will appear inside the base log folder The meaning of the files inside these folders is explained in Section 4 18 4 COMPSs Tools 4 1 Application graph At the end of the application execution a dependency graph can be generated representing the order of execution of each type of task and their dependencies To allow the final graph generation the g flag has to be passed to the runcompss command the graph file is written in the base log folder monitor complete graph dot at the end of the execution Figure shows a dependency graph example of a SparseLU java application The graph can be visualized by running the following command compss bsc gengraph COMPSs sparseLU arrays SparseLU_01 monitor complete_graph dot Figure 12 The dependency graph of the SparseLU application 4 2 COMPSs Monitor The COMPSs Framework includes a Web graphical interface that can be used to monitor the execution of COMPSs applications COMPSs Monitor is installed as a service and can be easily managed by2. 2dp runtime state cfg 2D plot of runtime state 2dp_tasks cfg 2D plot of tasks duration 3dh_duration runtime cfg 3D Histogram of runtime execution 3dh duration tashs cfg 3D Histogram of tasks duration compss_runtime cfg Shows COMPSs Runtime events at master and workers compss_tasks_and_runtime cf g Shows COMPSs Runtime events at master and workers and tasks execution compss_tasks cf g Shows tasks execution compss_tasks numbers cfg Shows tasks execution by task id compss trans fers cfg Shows transfer time spent by each worker Table 7 Available paraver configurations for COMPSs Applications 4 4 COMPSs IDE COMPSs IDE is an Integrated Development Environment to develop compile deploy and execute COMPSs applications It is available through the Eclipse Market as a plugin and provides an even easier way to work with COMPSs For further information please check the COMPSs IDE User Guide available at compss bsc es 25 O E O OD MN dun UE ee MA A A LU AA man A A dl ee ee MS A RA A MN i ll O O O O ROO O N DS Li A A O A DN EE A A A LE T A n O LE LL IS EE O AS dl au O DOLO A il i AM NO MO DO II OO O Li IO Y DL OO A AA A RA A a E A A A A A A O Figure 17 Examples of complex traces 26 5 Common Issues This section provides answers for the most common issues of the execution of COMPSs applications For specific issues not covered in this section
3. COMP SUPERSCALAR User Manual Application execution guide VERSION 1 3 November 3 2015 Barcelona Supercomputing Center Centro Nacional de Supercomputaci n This manual provides information about how to execute COMPSs applications how to retrieve the results and the logs of an execution and it provides an overview of the COMPSs tools usage It is highly recommended to test the examples described in this manual with a working COMPSs installation For this purpose we provide a COMPSs Virtual Machine available at http compss bsc es For information about the installation process please refer to the COMPSs Installation Guide available at http compss bsc es For further information about the application development please refer to the COMPSs User Manual Application development guide available at http compss bsc es For an extensive list of COMPSs application examples codes execution commands results logs etc please refer to the COMPSs Sample Applications guide at compss bsc es Contents 1 COMP Superscalar COMPSs 2 Executing COMPSs applications 4 5 2 5 4 1 Cloud connectors Amazon E 21 2 5 4 2 Cloud connectors rOCCI 3 1 Results LL LL LL LL LL pu BONS pe SE de Dan ee ee eee ee ee COMPSs Tools 4 1 Application graph 21 4 4 cts po ada eas 4 2 COMPSs Monitorl pes des at NA A abia 4 3 Application tracing
4. 0 lt TaskCount gt lt Queue gt short lt Queue gt lt Queue gt lt Host gt lt Processor gt lt Architecture gt IA32 lt Architecture gt lt Speed gt 3 0 lt Speed gt lt CoreCount gt 4 lt CoreCount gt lt Processor gt LOS lt OSType gt Linux lt OSType gt lt MaxProcessesPerUser gt 32 lt MaxProcessesPerUser gt lt 0S gt lt StorageElement gt lt Size gt 8 lt Size gt lt StorageElement gt lt Memory gt lt PhysicalSize gt 4 lt PhysicalSize gt lt VirtualSize gt 8 lt VirtualSize gt lt Memory gt lt ApplicationSoftware gt lt Software gt Java lt Software gt lt ApplicationSoftware gt lt Service gt lt V0 gt lt Cluster gt lt FileSystem gt lt NetworkAdaptor gt lt JobPolicy gt lt AccessControlPolicy gt lt Capabilities gt lt Requirements gt lt Adaptors gt lt Adaptor name gt lt MinPort gt 43001 lt MinPort gt lt MaxPort gt 43002 lt MaxPort gt lt Adaptor gt lt Adaptors gt lt Resource gt lt ResourceList gt 2 4 2 Project file The project file provides information about the resources used in a specific execution Consequently the resources that appear in this file are a subset of the resources described in the resources cml file This file that contains one entry per worker is usually edited by the users and changes from execution to execution Its full definition schema can be found at opt COMPSs Runtime configuration iml projects proje
5. Python COMPSs application 2 3 3 Running C C applications To launch a COMPSs C C application users have to compile the C C application by means of the buildapp command For further information please refer to the COMPSs User Manual Application development guide document available at http compss bsc es Once complied the lang c option must be provided to the runcompss command compss bsc cc o vee A somp e runcompss lang c simple lt initial_number gt compss compss bsc runcompss lang c home compss workspace_c simple master simple 1 Executing simple JVM OPTION FILE tmp tmp EIQA3NX3Wr IT HOME opt COMPSs Runtime scripts user Args 1 WARNING IT Properties file is null Setting default values API Starting COMPSs Runtime v1 3 build 20150820 1617 rnu1l Initial counter value is 1 Final counter value is 2 API No more tasks for app 0 API Getting Result Files 0 API Execution Finished Figure 3 Execution of a C COMPSs application 2 4 Additional configurations The COMPSs runtime has two configuration files resources zml and project cml These files contain information about the execution environment and are completely independent from the application For each execution users can load the default configuration files or specify their custom configurations by using respectively the resources lt absolute path to resources cml
6. please do not hesitate to contact us at support compss bsc es 5 1 How to debug When the application does not behave as expected the first thing users must do is to run it in debug mode executing the runcompss command withthe d flag to enable the debug log level In this case the application execution will produce the following files e runtime log e resources log e jobs folder First users should check the last lines of the runtime log If the file transfers or the tasks are failing an error message will appear in this file If the file transfers are successfully and the jobs are submitted users should check the jobs folder and look at the error messages produced inside each job Users should notice that if there are RESUBMITTED files something inside the job is failing 5 2 Tasks are not executed If the tasks remain in Blocked state probably there are no existing resources matching the specific task constraints This error can be potentially caused by two facts the resources are not correctly loaded into the runtime or the task constraints do not match with any resource In the first case users should take a look at the resouces log and check that all the resources defined in the project ml file are available to the runtime In the second case users should re define the task constraints taking into account the resources capabilities defined into the resources xml and project xml files 5 3 Jobs fail If all the application
7. running any of the following commands 19 4 2 1 Service configuration The COMPSs Monitor service can be configured by editing the opt COMPSs Tools monitor apache tomcat conf compss monitor conf file which contains one line per prop erty e IT MONITOR Default directory to retrieve monitored applications defaults to the COMPSs folder inside the root user e COMPSs_MONITOR_PORT Port where to run the compss monitor web service defaults to 8080 e COMPSs_MONITOR_TIM EOUT Web page timeout between browser and server defaults to 20s 4 2 2 Usage In order to use the COMPSs Monitor users need to start the service as shown in Figure compss bsc etc init d compss monitor start Starting COMPs Monitor Checking JAVA Installation Warning JRE_HOME not defined Info JAVA_HOME found Loading JRE_HOME from JAVA_HOME Success Checking IT_HOME WARNING IT_HOME not defined Trying default location opt COMPSs Success Checking IT_MONITOR IT_MONITOR home compss COMPSs Success Checking COMPSs Monitor Port Warning COMPSs_MONITOR_PORT not defined Loading from configuration file COMPSs_MONITOR_PORT 8080 Success Checking COMPSs Monitor Timeout Warning COMPSs_MONITOR_TIMEOUT not defined Loading from configuration file COMPSs_MONITOR_TIMEOUT 20000 Success Configuring COMPSs Monitor service Success Using CATALINA_BASE opt COMPSs Tools monitor apache tomcat Using CATALINA_HOME opt COMP
8. s tasks fail because all the submitted jobs fail it is probably due to the fact that there is a resource miss configuration In most of the cases the resource that the application is trying to access has no passwordless access through the configured user This can be checked by e Open the project xml The default file is stored under opt COMPSs Runtime configuration xml projects project cml 27 e For each resource annotate its name and the value inside the User tag Remember that if there is no User tag COMPSs will try to connect this resource with the same username than the one that launches the main application e For each annotated resourceName user please try ssh user resourceName If the connection asks for a password then there is an error in the configuration of the ssh access in the resource The problem can be solved running the following commands compss bsc scp ssh id_dsa pub user resourceName mydsa pub compss bsc ssh user resourceName These commands are a quick solution for further details please check the Additional Configuration section inside the COMPSs Installation Manual available at our website http compss bsc es 5 4 Compilation error AMethod not found When trying to compile Java applications users can get some of the following compilation errors error package integratedtoolkit types annotations does not exist import integratedtoolkit types annotations Constraints err
9. trace file may contain different events to determine the COMPSs master state the task execution state or the file transfers The current release does not support file transfers informations During the execution of the application an XML file is created in the worker nodes to keep track of these events At the end of the execution all the XML files are merged to get a final trace file In this manual we only provide information about how to obtain a trace and about the available Paraver the tool used to analyze the traces configurations For further information about the application instrumentation or the trace visualization please check the COMPSs Tracing Manual available at http compss bsc es 4 3 1 Trace Command In order to obtain a post execution trace file the option t must be added to the runcompss command Next we provide an example of the command execution with the tracing option enabled for the Hmmer java application compss bsc runcompss t classpath home compss workspace_java hmmerobj jar hmmerobj jar hmmerobj HMMPfam sharedDisk Hmmer smart HMMs bin sharedDisk Hmmer 256seq home compss out txt 2 8 A 222 a pi rai a ES CUE e o o ND A o a SSIS SS WARNING IT Properties file is null Setting default values Welcome to Extrae 3 1 1rc revision 3360 based on extrae trunk Extrae Warning EXTRAE_HOME has not been defined Extrae Generating intermediate files Paraver traces Extrae Intermediate files wi
10. v1 3 build 20150820 1617 rnull Initial counter value is 1 Final counter value is 2 API No more tasks for app 1 API Getting Result Files 1 API Execution Finished Figure 1 Execution of a Java COMPSs application compss bsc export CLASSPATH CLASSPATH home compss workspace_java simple jar simple jar compss bsc runcompss simple Simple lt initial_number gt compss bsc runcompss classpath home compss workspace_java simple jar simple jar simple Simple lt initial_number gt 2 3 2 Running Python applications To launch a COMPSs Python application users have to provide the lang python option to the runcompss command compss bsc cd workspace_python simple compss bsc workspace_python simple runcompss lang python simple py lt initial_number gt compss bsc workspace_python simple runcompss lang python simple py 1 Using default location for project file opt COMPSs Runtime scripts user configuration xml projects project xml Using default location for resources file opt COMPSs Runtime scripts user configuration xml resources resources xml Executing simple py WARNING IT Properties file is null Setting default values API Starting COMPSs Runtime v1 3 build 20150820 1617 rnull Initial counter value is 1 Final counter value is 2 API No more tasks for app 0 API Getting Result Files 0 API Execution Finished Figure 2 Execution of a
11. with this flag some of the COMPSs Monitor features will be disabled compss bsc workspace_java simple jar runcompss dm simple Simple 1 Using default location for project file opt COMPSs Runtime scripts user configuration xml projects project xml Using default location for resources file opt COMPSs Runtime scripts user configuration xml resources resources xml Executing simple Simple IT Properties file is null Setting default values Deploying COMPSs Runtime v1 3 build 20150820 1617 rnull Starting COMPSs Runtime v1 3 build 20150820 1617 rnull Initializing components Ready to process tasks Opening file home compss workspace_java simple jar counter in mode WRITE File URI any bsc home compss workspace_java simple jar counter File target Location home compss workspace_java simple jar counter nitial counter value is 1 API Creating task from method increment in simple SimpleImpl API There is 1 parameter API Parameter 1 has type FILE_T inal counter value is 2 API No more tasks for app 1 API Getting Result Files 1 API Stop IT reached API Stopping AP API Stopping TD API Stopping Comm Execution Finished Figure 15 Execution of the Simple Java application with the monitoring flag enabled 21 compss bsc cd COMPSs compss bsc COMPSsS tree gt 4 l job1_NEW err job1_NEW out complete_graph dot COMPSs_state xml current_graph dot resources log
12. C2 connector 13 2 5 4 2 Cloud connectors rOCCI The connector uses the rOCCI binary client version newer or equal than 4 2 5 which has to be installed in the node where the COMPSs main application is executed This connector needs additional files providing details about the resource templates available on each provider This file is located under lt COMPSs_INSTALL_DIR gt configuration xml templates path Additionally the user must define the virtual im ages flavors and instance types offered by each provider thus when the runtime decides the creation of a VM the connector selects the appropriate image and resource template according to the requirements in terms of CPU memory disk etc by invoking the rOCCI client through Mixins heritable classes that override and extend the base templates Table 5 contains the rOCCI specific properties that must be defined under the Provider tag in the project ml file and Table 5 contains the specific properties that must be defined under the Instance tag Provider ca path Path to CA certificates directory user cred Path of the VOMS proxy auth Authentication method x509 only supported sl Optional Used by the VENUS C Job Manager PMES jobname Table 5 rOCCI extensions in the project xml file Instance Multiple entries of resource templates Type Name of the resource template It has to be the same name than in the previous files CPU Numb
13. SparseLU application 19 13 COMPSs Monitor start commandl 20 14 COMPSs monitoring interfacel Q Q LL LL LL LL LL 21 15 Execution of the Simple Java application with the monitoring flag enabled 21 16 Logs generated by the Simple java application with the monitoring flag XE OPUN dh eds ds Ae RED AA eva 22 17 Examples of complex traces oa a a ee 26 111 List of Tables Peake ea 11 A a et 12 ied hy aire es 12 ware rere 13 aia ada 14 indio ia 14 CA 25 lv 1 COMP Superscalar COMPSs COMP Superscalar COMPSs is a programming model which aims to ease the develop ment of applications for distributed infrastructures such as Clusters Grids and Clouds COMP Superscalar also features a runtime system that exploits the inherent parallelism of applications at execution time For the sake of programming productivity the COMPSs model has four key charac teristics e Sequential programming COMPSs programmers do not need to deal with the typical duties of parallelization and distribution such as thread creation and syn chronization data distribution messaging or fault tolerance Instead the model is based on sequential programming which makes it appealing to users that either lack parallel programming expertise or are looking for better programmability A task is a method or a service called from the application code that is intended t
14. Ss Tools monitor apache tomcat Using CATALINA_TMPDIR opt COMPSs Tools monitor apache tomcat temp Using JRE_HOME Jusr lib jvm java 7 openjdk amd64 jre Using CLASSPATH opt COMPSs Tools monitor apache tomcat bin bootstrap jar opt COMPSs Tools monitor apache tomcat bin tomcat juli jar Tomcat started Figure 13 COMPSs Monitor start command And use a web browser to open the specific URL 20 COMPSs Monitor x localhost MS Reg LARA Barcelona FAQ Configuration Logout Supercomputing COMPSS MONITOR Center Centro Nacional de Supercomputaci n Applications nescurcss vtorata a ic a a o ca hello Hello_01 e E Status Resource Name CPU Count Core count Memory Size Disk Size Provider Image Running Tasks ry gi g O localhost 1 4 4068 8068 Refresh Figure 14 COMPSs monitoring interface The COMPSs Monitor allows to monitor applications from different users and thus users need to first login to access their applications As shown in Figure 14 the users can select any of their executed or running COMPSs applications and display it To enable all the COMPSs Monitor features applications must run the runcompss command with the m flag This flag allows the COMPSs Runtime to store special information inside inside the log base folder under the monitor folder see Figures 15 and 16 Only advanced users should modify or delete any of these files If the application that a user is trying to monitor has not been executed
15. ansferred object is not serializable Users should check that all the object parameters in the task are either implementing the serializable interface or following the java beans model by implementing an empty constructor and getters and setters for each attribute Please find more details on the COMPSs framework at http compss bsc es 31
16. ct_schema tsd We emphasize the importance of correctly defining the following entries installDir Indicates the path of the COMPSs installation inside the resource not necessarily the same than in the local machine User Indicates the username used to connect via ssh to the resource This user must have passwordless access to the resource for more information check the COMPSs Installation Manual available at our website http compss bsc es If left empty COMPSs will automatically try to access the resource with the same username than the one that lauches the COMPSs main application LimitOfTasks The maximum number of tasks that can be simultaneously scheduled to a resource Considering that a task can use more than one core of a node this value must be lower or equal to the number of available cores in the resource 2 5 Configuration examples In the next subsections we provide specific information about the services shared disks cluster and cloud configurations and several project zml and resources cml examples 2 5 1 Services configuration To allow COMPSs applications to use WebServices as tasks the resources zml can include a special type of resource called Service For each WebService it is necessary to specify its wsdl its name its namespace and its port When configuring the project zml file it is necessary to include the service as a worker by adding an special entry indicating only the name and the limit of task
17. de bd se ood ee me ERS 4 3 1 Trace Command poema pias MEA 44 COMPSs IDE Common Issues 5 1 How to debug esa io ae Da a dE Pas de ya 5 2 Tasks are not executedl 5 3 Jobs fail LL LL LL LL LL 5 4 Compilation error Method not found and Pa a a a s a a a ra de a a e dr En 2 2 Runcompss command acs no 4 ae de Se we ee Eh eee ee A e AAA a 2 3 1 Running Java applications a o o o LL LL E A 2D ee ie Ga de trail Be qr daa ae e is ea ESA e OA Es dl 2 4 1 Resources file 2 aa pa da Po der do qe ke we ee E A ee eee ca Of wt DE acne ES EO a Ga dd El cd eae ra ea der id as EN e 2 5 2 Cluster and grid configuration static resources 2 5 3 Shared Disks configuration example 2 5 4 Cloud configuration dynamic resources 5 6 Jobs failed on reflect target invocation null pointer 15 15 16 19 19 19 20 20 22 23 23 24 24 25 List of Figures pre a ig Seg gat 4 2 Execution of a Python COMPSs application 4 3 o Execution of a C COMPSs applicationJ 5 dd di dE OS db A Bea qe a A men a Era qae ee tea EE ay ee 15 o 15 o as 16 E O o 17 saa 17 11 Structure of the logs folder for the Simple java application in debug mode 18 ee E 00 12 The dependency graph of the
18. duler esourcesCreated Resource http bscgrid05 bsc es 20390 hmmerobj hmmerobj wsdl created 753 2015 08 20 3 DataInfoProvider lt init gt Initialization finished 787 2015 08 20 TaskAnalyser processTask New method task increment ID 1 791 2015 08 20 E TaskScheduler scheduleTask Blocked Task 1 increment 1479 2015 08 20 A Communication etWorkerIsReady Notifying that worker is ready localhost 1892 2015 08 20 34 31 TaskScheduler esourcesCreated Resource localhost created 1893 2015 08 20 34 31 TaskScheduler asksForResource Available Resource localhost Task 1 score 0 1894 2015 08 20 2 34 31 JobManager processJob New Job 1 Task 1 1894 2015 08 20 K JobManager processJob Method name increment 1895 2015 08 20 gt JobManager processJob Target host localhost 1899 2015 08 20 Communication submit Submit NIOJob with ID 1 1944 2015 08 20 JobManager fcompletedJob Received a notification for job 1 with state OR 1945 2015 08 20 g TaskProcessor notifyTaskEnd Notification received for task 1 with end status FINISHED 1946 2015 08 20 131 94 TaskProcessor waitForTask End of waited task for data 1 1955 2015 08 20 HE hs TaskProcessor noMoreTasks All tasks finished 1962 2015 08 20 16 34 31 TaskProcessor run AccessProcessor shutdown 1965 2015 08 20 34 31 96 Communication stop Shutting down localhost 43001 F
19. er of cores Memory Size in GB of the available RAM Disk Size in GB of the storage Price Cost per hour of the instance Table 6 Configuration of the lt provider gt cml tem plates file https appdb egi eu store software rocci cli 14 3 Results and logs 3 1 Results When executing a COMPSs application we consider different type of results e Application Output Output generated by the application e Application Files Files used or generated by the application e Tasks Output Output generated by the tasks invoked from the application Regarding the application output COMPSs will preserve the application output but will add some pre and post output to indicate the COMPSs Runtime state Figure shows the standard output generated by the execution of the Simple Java application The green box highlights the application stdout while the rest of the output is produced by COMPSs compss bsc workspace_java simple jar runcompss simple Simple 1 Using default location for project file opt COMPSs Runtime scripts user configuration xml projects project xml Using default location for resources file opt COMPSs Runtime scripts user configuration xml resources resources xml Executing simple Simple WARNING IT Properties file is null Setting default values API Starting COMPSs Runtime v1 3 build 20150820 1617 rnull Initial counter value is 1 Final counter value is 2 API No more tasks for ap
20. g lt CloudProvider gt The project xml complements the information about a provider listed in the resources zml file This file can contain a lt Cloud gt tag where to specify a list of providers each with a lt Provider gt tag whose name attribute must match one of the providers in the re sources cml file Thus the project zml file must contain a subset of the providers specified in the resources cml file Table 3 summarizes the information that users need to specify inside the lt Cloud gt tag and Table 2 summarizes the information that users need to specify inside the lt Provider gt tag of the project zml file 11 Initial V Ms Number of VM to be created at the be ginning of the application minVMCount Minimum number of VMs available in the computation maxVMCount Maximum number of VMs available in the computation Provider Multiple entries of Cloud providers Table 2 Configuration of project xml file tag lt Cloud gt LimitOfVMs Maximum number of VMs allowed by the provider Property Multiple entries of provider specific properties e Name e Name of the property e Value e Value of the property ImageList Multiple entries of VM images available at the provider e Image e VM image InstallDir Path of the COMPSs worker scripts in the image WorkingDir COMPSs working directory in the de ployed instances User Account username Package Multiple entr
21. gt and the project lt absolute_path_to_project ml gt in the runcompss command The default files are located in the opt COMPSs Runtime configuration aml path Users can manually edit these files or can use the Eclipse IDE tool developed for COMPSs For further information about the Eclipse IDE please refer to Section Next sections describe in detail the resources zml and the project zml files explaining the available options 2 4 1 Resources file The resources file provides information about all the available resources that can be used for an execution This file should normally be managed by the system administrators Its full definition schema can be found at opt COMPSs Runtime configuration aml resources resource_schema tsd It contains one entry per available resource defining its name its capabilities and its requirements Administrators can define several resource capabilities see example in the next listing but we would like to underline the importance of Processor CoreCount This capability represents the number of available cores in the described resource and it is used to schedule the correct number of tasks Thus it becomes essential to define it accordingly to the number of cores in the physical resource compss bsc cat opt COMPSs Runtime configuration xml resources resources xml lt encoding Ge lt ResourceList gt lt Resource Name gt lt Capabilities gt lt Host gt lt TaskCount gt
22. he available resources such as the number of processors of each resource slots the information about running or pending tasks in the resource queue and the created and destroyed resources The jobs folder will be empty unless there has been a failed job In this case it will store for each failed job one file for the stdout and another for the stderr As an example Figure 8 shows the logs generated by the same execution than the previous case but with info mode resources log runtime log Figure 8 Structure of the logs folder for the Simple java application in info mode The runtime log and resources log are quite large files thus they should be only checked by advanced users For an easier interpretation of these files the COMPSs Framework includes a monitor tool For further information about the COMPSs Monitor please check Section Figures 9 and 10 provide the content of these two files generated by the execution of the Simple java application compss bsc COMPSs simple Simple_02 cat runtime log 732 2015 08 20 16 34 30 731 TaskScheduler lt init gt Initialization finished 738 2015 08 20 30 TaskScheduler lt init gt Initialization finished 742 2015 08 20 730 JobManager lt init gt Initialization finished 742 2015 08 20 a TaskDispatcher lt init gt Initialization finished 748 2015 08 20 730 TaskAnalyser lt init gt Initialization finished 753 2015 08 20 30 TaskSche
23. ies of local packages that have to be deployed in new instances Source Local path of the package Target Path where to deploy the package in x InstalledSoftware the new instance x List of software included in the package InstanceTypes e Resource List of resource types that are available in the provider e Resource description Table 3 Configuration of project xml file tag lt Provider gt The next sections provide a description of each of the currently available connectors 2 5 4 1 Cloud connectors Amazon EC2 The COMPSs runtime features a connector to interact with the Amazon Elastic Compute Cloud EC2 Amazon EC2 offers a well defined pricing system for VM rental A total of 8 pricing zones are established corresponding to 8 different locations of Amazon datacenters around the globe Besides inside each zone several per hour prices exist for VM instances with different capabilities The EC2 connector stores the prices of standard on demand VM instance types tl micro m1 small m1 medium ml large and m1l xlarge for each zone Spot instances are not currently supported by the connector When the COMPSs runtime chooses to create a VM in the Amazon public Cloud the EC2 connector receives the information about the requested characteristics of the new VM namely the number of cores memory disk and architecture 32 64 bits According to that information the connector tries to find the VM ins
24. igure 9 runtime log generated by the execution of the Simple java application Running COMPSs with log level debug generates the same files as the info log level but with more detailed information Additionally the jobs folder contains two files per submitted job one for the stdout and another for the stderr In the other hand the COMPSs Runtime state is printed out on the stdout Figure 11 shows the logs generated by the same execution than the previous cases but with debug mode The runtime log and the resources log files generated in this mode can be extremely large Consequently the users should take care of their quota and manually erase these files if needed When running Python applications a pycompss log file is written inside the base log folder containing debug information about the specific calls to PyCOMPSs 17 compss bsc COMPSs simple Simple_02 cat resources log TIMESTAMP 1440081270727 New resource available in the pool Name http bscgrid05 bsc es 20390 hmmerobj hmmerobj wsd1l s 1440081270752 CORE INFO COREID 0 NO_RESOURCE TO_RESCHEDULE ORDINARY 0 MIN 100 MEAN 100 MAX 100 0 0 1 TIMESTAMP 1440081271891 INFO_MSG New resource available in the pool Name localhost TIMESTAMP 1440081271962 INFO_MSG Stopping all workers TIMESTAMP 1440081271962 LOAD_INFO CORE_INFO COREID O NO_RESOURCE TO_RESCHEDULE ORDINARY 0 MIN 56 MEAN 56 MAX 56 19
25. irectory lt encoding gt lt ResourceList gt lt Disk Name gt lt MountPoint gt sharedDisk lt MountPoint gt lt Disk gt lt ResourceList gt On the other side to declare that a worker has a shared disk mounted the resources xml file must include a Disk tag inside the specific worker indicating its name defined in the master Disk tag and its mount point inside the worker In the following example the sharedDisk0 is mounted on the resource hostname1 domain es under the path home user mySharedDisk lt Resource Name gt lt Capabilities gt The Disks tag can have multiple disk children nodes 2 5 4 Cloud configuration dynamic resources In order to use cloud resources to execute the applications the following steps have to be followed 1 Prepare cloud images with the COMPSs Worker package or the full COMPSs Framework package installed 2 The application will be deployed automatically during execution but the users need to set up the configuration files to specify the application files that must be deployed The COMPSs runtime communicates with a cloud manager by means of connectors Each connector implements the interaction of the runtime with a given provider s API supporting four basic operations ask for the price of a certain VM in the provider get the time needed to create a VM create a new VM and terminate a VM This design allows connectors to abstract the runtime fro
26. ject file opt COMPSs Runtime scripts user configuration xml projects project xml Using default location for resources file opt COMPSs Runtime scripts user configuration xml resources resources xml Executing hello Hello WARNING IT Properties file is null Setting default values API Deploying COMPSs Runtime v1 3 build 20150820 1617 rnull API Starting COMPSs Runtime v1 3 build 20150820 1617 rnull API Initializing components APII Ready to process tasks Hello World from main application API Creating task from method sayHello in hello HelloImpl API There is parameter API No more tasks for app 1 API Getting Result Files 1 API Stop IT reached API Stopping AP API Stopping TD API Stopping Comm Execution Finished compssQbsc more COMPSs hello Hello_01 jobs job1_NEW out 2015 08 20 16 47 31 615 DEBUG integratedtoolkit Worker Executor Creating sandbox for job 1 WORKER Parameters of execution Method class hello HelloImpl Method name sayHello Parameter types Parameter values Hello World from a task 2015 08 20 16 47 31 616 DEBUG integratedtoolkit Worker Executor Removing sandbox for job 1 Figure 6 COMPSs execution of the Hello java application 3 2 Logs COMPSs includes three log levels for running applications but users can modify them or add more levels by editing the logger files under the opt COMPSs Runtime configura tion log folder Any of these
27. ll be stored in home compss uorEspace java hmmerobj jar Extrae Tracing buffer can hold 500000 events Extrae Tracing mode is to Detail Extrae Successfully initiated with 1 tasks Extrae Warning API tries to initialize more than once Extrae Previous initialization was by API APT Starting COMPSs Runtime v1 3 build 20151016 1931 rnull API No more tasks app 1 API Getting Result Files 1 API Execution Finished 23 At the end of the execution the trace will be stored inside the trace folder under the application log directory 4 3 2 Trace Configurations The traces generated by an application execution are ready to be visualized with Paraver Paraver is a powerful tool developed by BSC that allows users to show many views of the trace data by means of different configuration files Users can manually load edit or create configuration files to obtain different trace data views In Table 7 we provide information about the different pre build configurations that we distribute with COMPSs and that can be found under the opt COMPSs Dependen cies paraver cfgs folder For further information about Paraver please visit the following site http www bsc es computer sciences performance tools paraver 4 3 3 Trace examples COMPSs traces can be very complex as the number of workers or tasks grows Just to illustrate this the following pictures show traces with a greater number of workers and tasks
28. log levels can be selected by adding the log_level lt debuglin folof f gt flag to the runcompss command The default value is off The logs generated by the NUM EX EC execution of the application APP by the user USER are stored under home USER COM PSs SAPP SEX EC NUMBER folder from this point on base log folder The EX EC NUMBER execution number is automatically used by COMPSs to prevent mixing the logs of data of different executions VVhen running COMPSs vvith log level off only the errors are reported This means that the base log folder will contain two empty files runtime log and resources log and one empty folder jobs If somehow the application has failed the runtime log and or the resources log will not be empty and a new file per failed job will appear inside the jobs folder to store the stdout and the stderr Figure 7 shows the logs generated by the execution of the Simple java application without errors in off mode L_ 4 6K 4 0K 0 resources log 0 runtime log 4 0K Figure 7 Structure of the logs folder for the Simple java application in off mode When running COMPSs with log level info the base log folder will contain two files runtime log and resources log and one folder jobs The runtime log file contains the execution information retrieved from the master resource including the file 16 transfers and the job submission details The resources log file contains information about t
29. lt tracing lt bool gt tracing t Generation of traces When no value is provided it is to Default monitoring lt int gt monitoring m Period between monitoring samples milliseconds When no value is provided it is to 2000 Default O Advanced options comm lt path gt Class that implements the adaptor communications Default integratedtoolkit nio master NIOAdaptor library_path lt path gt Non standard directories to search libraries e g Java JVM library Python library C binding library Default classpath lt path gt Path the application classes modules Default current working directory task_count lt int gt Only C Python Bindings Maximum number of different functions methods invoked from the application that have been selected as tasks Default 50 uuid lt int gt Preset an application UUID Default Automatic random generation PyObject_serialize lt bool gt Only Python Binding Enable the object serialization to string when possible Ye Default 2 3 Running a COMPSs application Before running COMPSs applications the application files must be in the CLASSPATH Thus when launching a COMPSs application users can manually pre set the CLASS PATH environment variable or can add the classpath option to the runcompss command The next three sections provide specific information for launching COMPSs applica tions developed in different code languages Java Py
30. m the particular API of each provider and facilitates the addition of new connectors for other providers The resources zml file must contain one or more lt CloudProvider gt tags that include the information about a particular provider associated to a given connector The tag must have an attribute name to uniquely identify the provider Table 1 summarizes the information to be specified by the user inside this tag Server Endpoint of the provider s server Connector Class that implements the connector 10 ImageList Multiple entries of VM templates e Image e VM image Architecture Architeture of the VM image OSType Operative System installed in the VM image ApplicationSoftware Multiple entries of software installed in the VM image Software x Software installed in the VM image SharedDisks Multiple entries of shared disks mounted in the VM im age Disk Disk description Adaptors Multiple entries of adaptors available at the VM image Adaptor Adaptor name and spe cific properties InstanceTypes Multiple entries of resource templates e Resource e Instance type offered by the provider Capabilities Hardware details of instance type Processor x Architecture and number of available cores StorageElement Size in GB of the storage x PhysicalSize in GB of x Memory the available RAM Table 1 Configuration of resources xml file ta
31. o be spawned asynchronously and possibly run in parallel with other tasks on a set of resources instead of locally and sequentially e Infrastructure unaware COMPSs offers a model that abstracts the application from the underlying distributed infrastructure Hence COMPSs programs do not include any detail that could tie them to a particular platform like deployment or resource management This makes applications portable between infrastructures with diverse characteristics e Standard programming languages COMPSs is based on the popular program ming language Java but also offers language bindings for Python and C C ap plications This facilitates the learning of the model since programmers can reuse most of their previous knowledge e No APIs In the case of COMPSs applications in Java the model does not require to use any special API call pragma or construct in the application everything is pure standard Java syntax and libraries With regard the Python and C C bindings a small set of API calls should be used on the COMPSs applications 2 Executing COMPSs applications 2 1 Prerequisites Prerequisites vary depending on the application s code language for Java applications the users need to have a jar archive containing all the application classes for Python applications there are no requirements and for C C applications the code must have been previously compiled by using the buzldapp command For further information ab
32. or package integratedtoolkit types annotations does not exist import integratedtoolkit types annotations Method error package integratedtoolkit types annotations does not exist import integratedtoolkit types annotations Parameter error package integratedtoolkit types annotations Parameter does not exist import integratedtoolkit types annotations Parameter Direction error package integratedtoolkit types annotations Parameter does not exist import integratedtoolkit types annotations Parameter Type error cannot find symbol Parameter Type FILE direction Direction INOUT symbol class Parameter location interface APPLICATION_Itf error cannot find symbol Constraints processorCoreCount 2 symbol class Constraints location interface APPLICATION_Itf error cannot find symbol Method declaringClass symbol class Method location interface APPLICATION_Itf 28 All these errors are raised because the compss engine jar is not listed in the CLASS PATH The default COMPSs installation automatically inserts this package into the CLASSPATH but it may have been overwritten or deleted Please check that your envi ronment variable CLASSPATH containts the compss engine jar location by running the following command If the result of the previous command is empty it means that you are missing the compss engine jar package in your classpath The easiest solution is to manually export the CLASSPATH variable into
33. out how to develop COMPSs applications please refer to the COMPSs User Manual Application development guide available at the http compss bsc es webpage 2 2 Runcompss command COMPSs applications are executed using the runcompss command compss bsc runcompss options application_name application_arguments The application name must be the fully qualified name of the application in Java the path to the py file containing the main program in Python and the path to the master binary in C C The application arguments are the ones passed as command line to main application This parameter can be empty The runcompss command allows the users to customize a COMPSs execution by spec ifying different options For clarity purposes parameters are grouped in Runtime config uration Tools enablers and Advanced options compss bsc runcompss h Runtime configuration options project lt path gt Path to the project XML file Default opt COMPSs Runtime configuration xml projects project xml resources lt path gt Path to the resources XML file Default opt COMPSs Runtime configuration xml resources resources xml lang lt name gt Language of the application java c python Default java log_level lt level gt debug d Set the debug level off info debug Default off Tools enablers graph lt bool gt graph g Generation of the graph When no value is provided it is to Defau
34. p 1 API Getting Result Files 1 API Execution Finished Figure 4 Output generated by the execution of the Simple Java application with COMPSs Regarding the application files COMPSs does not modify any of them and thus the results obtained by executing the application with COMPSs are the same than the ones generated by the sequential execution of the application Regarding the tasks output COMPSs introduces some modifications due to the fact that tasks can be executed in remote machines After the execution COMPSs stores the stdout and the stderr of each job a task execution inside the home USER COMPSs APPNAME EX EC_NUMBER jobs directory of the main application node Figures 5 and 6 show an example of the results obtained from the execution of the Hello Java application While Figure 5 provides the output of the sequential execution of the application without COMPSs Figure 6 provides the output of the equivalent COMPSs execution Please note that the sequential execution produces the Hello World from a task message in the stdout while the COMPSs execution stores the message inside the jobl_N EW out file compssQibsc workspace_java hello jar java cp hello jar hello Hello Hello World from main application Hello World from a task Figure 5 Sequential execution of the Hello java application 15 compss bsc workspace_java hello jar runcompss d hello Hello Using default location for pro
35. runtime log Figure 16 Logs generated by the Simple java application with the monitoring flag enabled 4 2 3 Graphical Interface features In this section we provide a summary of the COMPSs Monitor supported features available through the graphical interface Resources information Provides information about the resources used by the application Tasks information Provides information about the tasks definition used by the application Current tasks graph Shows the tasks dependency graph currently stored into the COMPSs Runtime Complete tasks graph Shows the complete tasks dependecy graph of the application Load chart Shows different dynamic charts representing the evolution over time of the resources load and the tasks load Runtime log Shows the runtime log Execution Information Shows specific job information allowing users to easily select failed or uncompleted jobs Statistics Shows application statistics such as the accumulated cloud cost The webpage also allows users to configure some performance parameters of the mon itoring service by accessing the Configuration button at the top right corner of the web page For specific COMPSs Monitor feature configuration please check our FAQ section at the top right corner of the web page 22 4 3 Application tracing COMPSs Runtime can generate a post execution trace of the execution of the application This trace is useful for performance analysis and diagnosis A
36. s as shown in the following example lt encoding RS lt Project gt lt Description for any physical node gt lt Worker Name gt lt Worker gt lt Worker Name gt lt LimitOfTasks gt 3 lt LimitOfTasks gt lt Worker gt lt Project gt 2 5 2 Cluster and grid configuration static resources In order to use external resources to execute the applications the following steps have to be followed 1 Install the COMPSs Worker package or the full COMPSs Framework package on all the new resources following the Installation manual available at http compss bsc es 2 Set SSH passwordless access to the rest of the remote resources 3 Create the WorkingDir directory in the resource remember this path because it is needed for the project rml configuration 4 Manually deploy the application on each node The resources rml and the project cml files must be configured accordingly Here we provide examples about configuration files for Grid and Cluster environments lt encoding Ge lt ResourceList gt lt Resource Name gt lt Capabilities gt lt Host gt lt TaskCount gt 0 lt TaskCount gt lt Queue gt Short lt Queue gt lt Host gt lt Processor gt lt Architecture gt x86_64 lt Architecture gt lt Speed gt 2 5 lt Speed gt lt CoreCount gt 4 lt CoreCount gt lt Processor gt lt 0S gt lt 0SType gt Linux lt 0SType gt lt 0S gt lt StorageElement g
37. t lt Size gt 250 0 lt Size gt lt StorageElement gt lt Memory gt lt PhysicalSize gt 4 0 lt PhysicalSize gt lt Memory gt lt ApplicationSoftware gt lt Software gt BLAST lt Software gt lt ApplicationSoftware gt lt Capabilities gt lt Requirements gt lt Adaptors gt lt Adaptor name gt lt MinPort gt 43001 lt MinPort gt lt MaxPort gt 43002 lt MaxPort gt lt Adaptor gt lt Adaptors gt lt Resource gt lt Resource Name gt lt Resource gt lt ResourceList gt lt encoding gt lt Project gt lt Worker Name gt lt InstallDir gt opt COMPSs Runtime scripts system lt InstallDir gt lt WorkingDir gt home user lt WorkingDir gt lt User gt user lt User gt lt LimitOfTasks gt 2 lt LimitOfTasks gt lt Worker gt lt Worker Name gt lt Worker gt lt Project gt 2 5 3 Shared Disks configuration example Configuring shared disks might reduce the amount of data transfers improving the ap plication performance To configure a shared disk the users must edit the resources zml indicating how the shared disk is hosted in the master node and how the shared disk is mounted in each worker To indicate a shared disk hosted in the master node the resources xml file must include a Disk tag describing the disk and the mount point The following example states that in the master node there is a shared disk labelled sharedDisk0 mounted on the sharedDisk d
38. tance type in Amazon that better matches those characteristics and then requests the creation of a new VM instance of that type Once an EC2 VM is created a whole hour slot is paid in advance for that reason the connector keeps the VM alive at least during such period saving it for later use if necessary When the task load decreases and a VM is no longer used the connector puts it aside if the hour slot has not expired yet instead of terminating it After that if the task load increases again and the EC2 connector requests a VM first the set of saved VMs is examined in order to find a VM that is compatible with the requested characteristics If one is found the VM is reused and becomes eligible again for the execution of tasks hence the cost and time to create a new VM are not paid A VM is only destroyed when the end of its hour slot is approaching and it is still in saved state Table 4 summarizes the provider specific properties that must be defined in the project xml file for the Amazon EC2 connector Placement Location of the amazon datacentre to use Access Key Id Identifier of the access key of the Amazon EC2 ac count Secret Key Id Identifier of the secret key of the Amazon EC2 account Key host location Path to the SSH key in the local host used to connect to the VMs KeyPair name Name of the key pair to use SecurityGroup name Name of the security group to use Table 4 Properties of the Amazon E
39. the user session However you will need to remember to export this variable every time you log out and back in again Consequently we recommend to add this export to the bashrc file 5 5 Jobs failed on method reflection When executing an application the main code gets stuck executing a task Taking a look at the runtime log users can check that the job associated to the task has failed and all its resubmissions too Then opening the jobX_N EW out or the jobX_N EW err files users find the following error 29 This error is due to the fact that COMPSs cannot find one of the tasks declared in the Java Interface Commonly this is triggered by one of the following errors e The declaringClass of the tasks in the Java Interface has not been correctly defined e The parameters of the tasks in the Java Interface do not match the task call e The tasks have not been defined as public 5 6 Jobs failed on reflect target invocation null pointer When executing an application the main code gets stuck executing a task Taking a look at the runtime log users can check that the job associated to the task has failed and all its resubmissions too Then opening the jobX_N EW out or the jobX_N EW err files users find the following error This cause of this error is that the Java object accessed by the task has not been correctly transferred and one or more of its fields is null The transfer failure is normally caused because the tr
40. thon and C C For clarity pur poses we will use the Simple application developed in Java Python and C available in the COMPSs Virtual Machine or at https compss bsc es projects bar webpage This application takes an integer as input parameter and increases it by one unit using a task For further details about the codes please refer to the Sample Applications document available at http compss bsc es 2 3 1 Running Java applications A Java COMPSs application can be launched through the folloving command compss bsc workspace_java simple jar compss bsc workspace_java simple jar runcompss simple Simple lt initial_number gt In this first execution we use the default value of the classpath option to automatically add the jar file to the classpath by executing runcompss in the directory which contains the jar file However we can explicitly do this by exporting the CLASSPATH variable or by providing the classpath value Next we provide two more ways to perform the same execution compss bsc workspace_java simple jar runcompss simple Simple 1 Using default location for project file opt COMPSs Runtime scripts user configuration xml projects project xml Using default location for resources file opt COMPSs Runtime scripts user configuration xml resources resources xml Executing simple Simple WARNING IT Properties file is null Setting default values API Starting COMPSs Runtime
Download Pdf Manuals
Related Search