Home

Chapter 1. ProActive Scheduler

1. 1 4 2 2 Create a TaskFlow job using Java API To make a new instance of a TaskFlow job just create it as shown below TaskFlowJob job new TaskFlowJob To parameterize your TaskFlow Job just go to Section 1 4 1 2 Create a job using Java API 1 4 3 Create a ProActive job To create a non specialized job please first read Section 1 4 1 Create a job 1 4 3 1 Create a ProActive job using XML descriptor To specify that the job is a ProActive Job just add the proActive tag Here s an example of how to go on to a ProActive Job using the previous job descriptor lt xml versionz 1 0 encoding UTF 8 gt job xmins urn proactive jobdescriptor 0 91 xmins xsi http www w3 org 2001 XMLSchema instance xsi schemaLocation urn proactive jobdescriptor 0 91 http proactive inria fr schemas jobdescriptor 0 91 schedulerjob xsd id job_name priority normal projectName project_name cancelOnError true logFile path to a log file log gt lt description gt Job description lt description gt lt variables gt variable name vali value toto gt lt variables gt lt genericinformation gt Part I ProActive Scheduler ProActive a Chapter 1 ProActive Scheduler info name vari value val1 gt info name var2 value val2 gt lt genericinformation gt proActive neededNodes 10 gt lt Job will be completed here later gt lt proActive gt lt job
2. 1 Finished john Normal 3 Finished i Normal Job 5 has 8 tasks Finished time Not yet Not yet Not yet Start time Not yet Not yet Not yet Re ru Description 0 1 This task wil sleep 6s 072 This task wil sleep 6s 0 1 Thistask wil sleep 8s Not yet Not yet Om This task wil sleep 4s Not yet Not yet on This task will sleep 5s 15 46 21 11 22 07 15 46 45 11 22 07 0 1 This task will sleep 10s Not yet Not yet 0 1 This task wil sleep 2s 15 46 22 11 22 07 15 46 45 11 22 07 0 2 This task wil sleep 6s 5 Stalled job with dep Normal Pending tasks number 6 Running tasks number 0 Finished tasks number 2 Total tasks number 8 Submitted time 15 46 20 11 22 07 Started tima 35 45 21 11 223 Figure 2 1 The Scheduler Perspective 26 Part I ProActive Scheduler ProActive e Chapter 2 ProActive Scheduler Ecplise Plugin 2 2 Views composing the perspective obs 3 gt 4 Boe H D a 2 0 gt Pending 7 Running 11 Finished 8 State User Priority Name State Progress Finish Id State User Priority Nam 3 4 j 7 8 j 50 Finished chri Normal job 0 8 j 354 Finished admin Normal Matlab 1 4 350 Finished jl Normal job na 1 4 363 Finished jo Normal Matlab 1 9 365 Finished admin Normal job pre Pending chri Low job proAc Running Pending admin Normal Job with Running Pending jo Normal job with Running Pending j Normal job with Running Pending i Normal job P Running Pending j Low
3. task id rooi4 preciousResult true gt lt description gt Calculates the root of a polynomial lt description gt lt selection gt lt script gt file urlz http proactive inria fr userfiles file scripts checkMatlab js gt lt script gt lt selection gt javaExecutable class 0rg ow2 proactive scheduler ext matlab SimpleMatlab gt lt parameters gt E Example 5 1 imple Matlab Job descriptor Example parameter name script value out roots in gt lt parameters gt lt javaExecutable gt lt task gt lt taskFlow gt lt job gt Part III ProActive Scheduler s Matlab ProActive 4 Chapter 5 ProActive Scheduler s Matlab extension Extension 5 3 A More Complex Example a Matlab task flow Now we will get through a more complex example This example will use an interesting feature of the Matlab extension the ability to pass results of one task as inputs of another task This exemple on the contrary of the previous one is not a simple parallel batch processing it s a flow of tasks which depends from each others This example will compute the sum of a big randomly generated array values taken from 50 to 50 The example contains 3 steps 1 It splits the big array into several smaller arrays 2 It computes the sum of each array in parallel 3 It merges the results from each parallel sub total to compute the final sum This is not of course a real case example as computing the sum of a b
4. lt javaExecutable class org ow2 proactive scheduler ext scilab SimpleScilab gt lt parameters gt lt parameter name scriptFile value SCILAB_SCRIPTS intsin sci gt Example 64pSeitateJobdescriptordixample 5 gt lt parameter name outputs value out gt lt parameters gt lt javaExecutable gt lt task gt task id t3 gt lt description gt Extension Part IV ProActive Scheduler s Scilab ProActive 4 Chapter 6 ProActive Scheduler s Scilab extension Extension Here is the Scilab script that calculates individual integrals out integrate sin x x i 9epi n i 1 9epi n Example 6 2 Integral script Here is the Scilab script that merges the individual results and computes the final answer out outi out2 out3 out4 out5 Example 6 3 Merging script This descriptor is very similar to the descriptor Example 5 2 Complex Matlab Job descriptor Example We ll go through the similarities and differences of these two descriptors 6 2 2 1 Similarities with Matlab job descriptor Concepts of job tasks and dependences are common to all ProActive Scheduler jobs so we find here the same concepts than in Matlab s Definitions of task s main and input scripts are done through the same task parameters script scriptFile scriptUrl and input 6 2 2 2 Differences with Matlab job descriptor For example in the following task javaExecutable class org objectweb proactive extensions scheduler
5. lt parameters gt lt parameter name numberToFind value 200 gt lt parameters gt lt proActiveExecutable gt lt post gt lt script gt file path SCRIPT_DIR unset js gt lt script gt lt post gt lt task gt To have an exhaustive list of which options are available and what they are suppose to do just go to the task explanation section at Section 1 4 4 6 Tasks options and explanations 1 4 4 5 Create and Add a ProActive task using Java API To create a ProActive task use the ProActiveTask class In this type you must specify the class you want to start with by mentioning a Class of extending ProActiveExecutable To make your own executable see the proper section Section 1 4 4 3 1 Define your own ProActive executable In addition you can add arguments with which the task will be launched These launching arguments will be given to the ProActive executable as a Map Just take a look at the example below to see how to use the task creation Java API see also Java DOCumentation of the Scheduler to learn more create a ProActive Task using the default constructor that we ll call aTask ProActiveTask aTask new ProActiveTask then set the desired options for example aTask setName task 1 aTask setDescription T his task will do something aTask addGenericlnformation key value aTask setResultPreview UserDefinedResultPreview class add arguments optional
6. 17 Part I ProActive Scheduler ProActive e Chapter 1 ProActive Scheduler proActiveExecutable class org ow2 proactive scheduler examples ProActiveExample gt lt parameters gt lt parameter name numberToFind value 200 gt lt parameters gt lt proActiveExecutable gt lt task gt The ProActive Task is composed of one proActiveExecutable that specified the ProActiveExecutable Java class to use A set of parameters has also be defined to provide this executable some informations These parameters will be available into the HashMap of the init HashMap method into your ProActiveExecutable To put this task inside your ProActive job just put it between the ProActive tags On this job it is necessarily to set the number of node you desired for your ProActiveExecutable Instead of deploying resources as it must be done in ProActive Suite the resources are provides by the ProActive Scheduler Here s how a complete ready to be scheduled ProActive Job seems like lt xml version 1 0 encoding UTF 8 gt job xmins urn proactive jobdescriptor 0 91 xmins xsi http www ws8 org 2001 XMLSchema instance xsi schemaLocation urn proactive jobdescriptor 0 91 http proactive inria fr schemas jobdescriptor 0 91 schedulerjob xsd id job name priority normal projectName project name cancelOnError true logFile path to a log file log gt lt description gt Job description lt description gt lt variables gt lt
7. just go to the task explanation section at Section 1 4 4 6 Tasks options and explanations 1 4 4 2 2 Create and Add a native task using Java API To create a native task use the NativeTask class In this type you must specify the executable you want to start by mentioning a command line In addition you can add arguments with which the task will be launched These launching arguments will be append to the command line Just take a look at the example below to see how to use the task creation Java API see also Java DOCumentation of the ProActive Scheduler to learn more create a native task with the default constructor that we ll call aTask NativeTask aTask new NativeTask then set the desired options for example aTask setName task 1 aTask setDescription This task will do something aTask addGenericlnformation key value aTask setPreciousResult true aTask setRerunnable 2 aTask setRestartOnError RestartMode ELSEWHERE aTask setResultPreview UserDefinedResultPreview class set the command line with the parameter append to the process name aTask setCommandLine path to command cmd parami param2 SCRIPTS EXAMPLE If the script to use is in a file or URL String args new String foo bar File scriptFile 2 new File path to script file URL scriptURL new URL url to script file Script script new SimpleScript scriptFile args 15 Part I Pro
8. 91 http proactive inria fr schemas jobdescriptor 0 91 schedulerjob xsd xmlnsz urn proactive jobdescriptor 0 91 id Scilab_job priority normal logFile HOME scilab_job log gt lt description gt A simple Scilab batch job which computes the numerical integration of sin x between 0 and PI lt description gt lt variables gt variable namez PROACTIVE HOME valuez home user ProActive gt variable name SCILAB SCRIPTS value PROACTIVE HOME scripts unix scilab examples gt lt variables gt lt taskFlow gt task id i1 gt lt description gt Calculates the numerical integration of sin x between i Pl n and i 1 Pl n lt description gt lt selection gt lt script gt lt file url http proactive inria fr userfiles file scripts checkScilab js gt lt script gt lt selection gt lt javaExecutable class 0rg ow2 proactive scheduler ext scilab SimpleScilab gt lt parameters gt lt parameter name scriptFile value SCILAB_SCRIPTS intsin sci gt parameter name input value i 0 n 5 gt lt parameter name outputs value out gt lt parameters gt lt javaExecutable gt lt task gt task id t2 gt lt description gt Calculates the numerical integration of sin x between i Pl n and i 1 Pl n lt description gt lt selection gt lt script gt lt file url http proactive inria fr userfiles file scripts checkScilab js gt lt script gt lt selection gt
9. Display the Connect to scheduler dialog in order to establish a connection to a remote ProActive scheduler j Figure 2 9 Disconnect from the scheduler Figure 2 10 Change view from Vertical to Horizontal mode Allows to switch the job s display to Horizontal from Vertical mode Figure 2 11 Change view from Horizontal to Vertical mode Allows to switch the job s display from Horizontal to Vertical mode p Figure 2 12 Submit a job Display the Choose file dialog in order to submit a XML defined job to the scheduler ae A Figure 2 13 Pause Resume a job Pause or resume the selected job only if you are the job owner Figure 2 14 Change job priority Change job priority only if you are the job owner Priority allowed are Lowest Low Normal Figure 2 15 Display job output To display the selected job s standard and error output only if you are the job owner 32 Part I ProActive Scheduler ProActive 4 Chapter 2 ProActive Scheduler Ecplise Plugin Figure 2 16 Kill Job To kill the selected job only if you are the job owner 2 4 2 The Jobs view buttons in Administrator Mode All buttons allowed in user mode are also allowed in Administrator mode Moreover you can execute any action even you aren t the job owner In Administrator mode 3 other choices for job priority are available Idle lower priority than the 3 User Mode priorities High higher priority than the 3 User Mo
10. behavior Walltime optional Task Walltime is a maximum allowed execution time of a task It can be specified for any task irrespectively of its type If a task does not finish before its walltime it is terminated by the ProActive Scheduler An example has been given above with the walltime specified Note that the walltime is defined in a task thus it can be used for any type of a task The general format of the walltime attribute is hh mm ss where h is hour m is minute and s is second The format still allows for more flexibility We can define the walltime simply as 5 which corresponds to 5 seconds 10 is 10 seconds 4 10 is 4 minutes and 10 seconds and so on The walltime mechanism is started just before a task is launched If a task does finish before its walltime the mechanism is canceled Otherwise the task is terminated Note that the tasks are terminated without any prior notice If the walltime is specified for a Java task as in the example it enforces the creation of a forked Java task instead fork and forkEnvironment optional only for Java Task The purpose of a new type of a task Forked Java Task is to gain more flexibility with respect to the execution environment of a task A new JVM is started with an inherited classpath and possibly redefined Java home path and JVM properties It allows to use a JVM from a different provider and specify options to be passed to JVM like memory usage A Forked Java Task
11. cccscssu cecissssaceestonchsasneateponbdnateposenbeapenoachanGa EEE E EAE EEE EE E EEEE 44 25 2 Complex Matlab Job descriptor Example vices iie ns ererureseorturi ve sikian r CEs ree VE ECE o ple ln ed ERU ea D ER Eie 46 6 1 Scdab Job descriptor Example erre reete bee eem sete pea pev et EEEE EEE EPO EESE EEEE EEEE ESERE S 51 6 2 Integral SCrpt OE E 52 6 3 Merenge SCi sporre 52 Part I ProActive Scheduler ProActive 4 Part ProActive Scheduler Table of Contents Chapter 1 ProActive Scheduler 4uissecnoeiee nr ERE ORE EPEIE SERRE RUE EE IE EPE FEN IPS QM END M XO EIER OIN PEE INIM a 1 1 IMPORTANT INOWE 8e roost revenge ete e ee put eest ese muette meta E fete or vor t EN siens E pet veni rv proi iaces 2 1 2 OVeEVIOW secet sandrine etaient nie A Dane Tied bono e I E DEP de ete cl O 2 1 3 Scheduler Concept mr 2 1 3 3 What 18 4 Job l aa tester ut yd diner tree tates dies nn nine dein tae tte none ni 2 13 23 What isa Task Rr Em 2 1 3 3 Dependenci s between Tasks 52 mener co dededi geetareaed spend uve De cd Dess i eee teo didier Pr RP se ent 3 1 34 Scheduling POL CY sise Sorteo rera nee e a a E o i e Aie E E Ee E E Ei 4 1 4 User Manual es en E E a E E E E ques d c tien 4 14 1 GE LC 4 1 42 Createa FasKFlOW Ep rU 5 1 4 3 Create a ProActive Job 1 ort cocer bae eo t qr tov eater red re
12. false is an option that define if a task has to be restarted if an error occurred Error can be both exception for Java task or error code 1 255 for native task It is a way to managed user error If not defined the task will never restart This option can be set to anywhere that means the task will restart on the first available node It can also be set to elsewhere meaning that the task will restart on a different node that the last used In these 2 last cases the job will be failed if the maximum number of retries retries option is reached This option is not available for proActive Task Combined with the job cancelOnError option it can be useful to know the behavior of your job Here s a table that explains what can be done with tasks and job 20 Part I ProActive Scheduler ProActive 4 Chapter 1 ProActive Scheduler CancelOnError amp RestartOnError mechanism Native Task Result 1 255 Java Task Object Legend Tast is restarted without condition Task resut is returned at the end of the first execution Task is retried on the first ready node until the define number of retries default is one Task is retried on the first ready node that is different from the previous one until the define number of retries default is one Job will be canceled atthe end of the first execution Used for internal management only Job will be failed at the end of retries Figure 1 2 CancelOnError and RestartOnError
13. gt To create and add tasks to your Job just go to Section 1 4 4 Create and Add a task to a job 1 4 3 2 Create a ProActive job using Java API To make a new instance of a ProActive job just create it as shown below ProActiveJob job new ProActiveJob To parameterize your ProActive Job just go to Section 1 4 1 2 Create a job using Java API 1 4 4 Create and Add a task to a job As it has been said it is possible to create 3 types of tasks Native and Java tasks can be add to TaskFlow Job and one ProActive Task to one ProActive Job 1 4 4 1 Create and Add a Java task Note It is only possible to add a Java task in a TaskFlow Job To learn how to create a TaskFlow Job just go to Section 1 4 2 Create a TaskFlow job Once your TaskFlow Job created you can add as many Java tasks as needed to perform an application 1 4 4 1 1 Define your own Java executable First of all you must know that you can create your own java executable by implementing scheduler executable interfaces What is called executable is in fact the executed process that is a Java class in this case Here s an example to create your own Java executable public class WaitAndPrint extends JavaExecutable Override public Object execute TaskResult results throws Throwable String message try System err printin D marrage de la tache WaitAndPrint System out printin Parameters are for TaskResult
14. is defined as a Java Task with a forkEnvironment element The aim of a forkEnvironment element is providing javaHome and jvmParameters attributes For any undefined attribute a default environment value will be applied Note that the javaHome attribute points only to the Java installation directory and not the Java application itself If the javaHome is not specified then the ProActive Scheduler will execute simply a Java command assuming that it is defined in the user path The jvmParameters attribute is a string composed of a sequence of Java options divided by a space parameters optional only for Java and ProActive Task is a way to define some parameters to be transfered to the executable This is best explained in Section 1 4 4 1 1 Define your own Java executable Each parameters is define with a name and a value and will be passed to the Java Executable as an HashMap arguments optional only for native Task is a way to define arguments for your native process Each arguments is define by a value that will be append to the process name to create a command line e resultPreview optional allows to specify how the result of a task should be displayed in the Scheduler graphical client The user should implement a result preview class that extends 21 Part I ProActive Scheduler ProActive 4 Chapter 1 ProActive Scheduler org objectweb proactive extensions scheduler common task ResultPreview abstract class which specifies
15. job_proAc Running Pending j Low job proAc Running 0 9 370 Finished chri Normal job pre Running 2 8 j 372 Finished admin Normal job pre 1 4 2 8 Running Running COLLATE 3 Figure 2 2 The Jobs view All buttons on upper right are describe in the This view is composed of 3 tables which represents pending running and finished jobs In each table you can watch many different informations about jobs as their state their name their id 27 Part I ProActive Scheduler ProActive 4 Chapter 2 ProActive Scheduler Ecplise Plugin El Console 23 Ex rt Da Job 11 task7 13 47 56 D marrage de La tache numero 7 A Parameters are INIT 4 sleepTime 5 INIT 4 number 4 D marrage de la tache numero 4 Parameters are INIT 3 sleepTime 8 INIT 3 number 3 D marrage de la tache numero 3 task3 13 47 57 Parameters are task1 13 47 57 INIT 1 sleepTime 8 task7813 47 56 task16813 47 57 INIT 1 number 1l task4 13 47 56 task4813 47 56 task4 13 47 56 task4813 47 56 task3813 47 57 task3813 47 57 task3813 47 57 task1813 47 57 D marrage de la tache numero 1 task16813 47 57 Parameters are task7 13 48 00 Terminaison de la tache numero task6 13 48 68 Terminaison de la tache numero task4 13 48 01 Terminaison de la tache numero task5 13 48 62 Terminaison
16. only after all these tasks have completed and the results of all these tasks will be the inputs of the Collector you ll see on 5 3 4 New parameter in SimpleMatlab tasks index lt task name sumi gt lt description gt Calculates the sum of the elements in the array lt description gt lt depends gt 47 Part III ProActive Scheduler s Matlab ProActive a Chapter 5 ProActive Scheduler s Matlab extension Extension task ref splitter gt lt depends gt lt selection gt lt script gt file url http proactive inria fr userfiles file scripts checkMatlab js gt lt script gt lt selection gt lt javaExecutable class org objectweb proactive extensions scheduler ext matlab SimpleMatlab lt parameters gt lt parameter name index value 0 gt parameter name scriptFile value MATLAB_SCRIPTS summer m gt lt parameters gt lt javaExecutable gt lt task gt A new parameter appears in this descriptor for the SimpleMatlab task the index The parameter is related to the splitting mechanism It can be defined only inside a SimpleMatlab task and has sense only if the Simple task has a Splitter task as parent The Splitter sends an output in the form of a list of results to each child task The same list will be sent to every children Therefore each one needs to specify at which index of the list it will look at For example a splitter task splits the array 1 2 3 4 into two arrays 1 2 and 3 4
17. parameters gt parameter name scriptFile value MATLAB_SCRIPTS splitter m gt lt parameter name input value in round rand 1 1000000 100 50 gt lt parameter name number_of_children value 6 gt lt parameters gt lt javaExecutable gt lt task gt task id sum1 gt lt description gt Calculates the sum of the elements in the array lt description gt lt depends gt lt task ref splitter gt lt depends gt lt selection gt lt script gt file url http proactive inria fr userfiles file scripts checkMatlab js 7 lt script gt lt selection gt javaExecutable class 0rg ow2 proactive scheduler ext matlab SimpleMatlab gt lt parameters gt lt parameter name index value 0 gt parameter name scriptFile value MATLAB_SCRIPTS summer m gt lt parameters gt lt javaExecutable gt lt task gt lt task id sum2 gt lt description gt Calculates the sum of the elements in the array lt description gt lt depends gt lt task ref splitter gt lt depends gt lt selection gt lt script gt Example 5 2ii omplex Matlab Job d scriptor Example check Viailad js gt lt script gt lt selection gt lt javaExecutable class 0rg ow2 proactive scheduler ext matlab SimpleMatlab gt lt parameters gt parameter name index value 1 gt Part III ProActive Scheduler s Matlab ProActive 4 Chapter 5 ProActive Scheduler s Matlab extension Extension 5 3 1 Des
18. post script The pre script is always executed on the node that has been selected by the resource manager after the execution of the task itself selection script The selection script is always executed before the task itself on any candidate node the execution of a selection script must set the boolean variable selected that indicates if the candidate node is suitable for the execution of the associated task Now that your have your job created next step is to submit it to the ProActive Scheduler 1 4 5 Submit a job to the ProActive Scheduler The submission will perform some checking to ensure that a job is correctly formed Then the job is inserted in the pending list and wait for executions until free resources become available Once done the job will be started on the resources deployed by the resource manager Finally once finished the job goes in a finish queue and will wait until user to retrieve its result Their are three ways to submit a job to The Scheduler 1 4 5 1 Submit a job using the Graphical User Interface Scheduler Eclipse Plugin To submit a job using the graphical tools you must have first created a job XML Descriptor Then refer toChapter 2 ProActive Scheduler Ecplise Plugin documentation to submit it 1 4 5 2 Submit a job using shell command Use the provided shell script jobLauncher sh to submit a job using command line This script bin unix jobLauncher sh has 1 mandatory option and 3 optional The pat
19. proactive inria fr usertiles file scripts checkMatlab js gt lt script gt lt selection gt lt javaExecutable class 0rg ow2 proactive scheduler ext matlab SimpleMatlab gt lt parameters gt lt parameter name input value in 1 0 3 2 5 1 gt lt parameter name script value out roots in gt lt parameters gt lt javaExecutable gt lt task gt task id root2 preciousResult true gt lt description gt Calculates the root of a polynomial lt description gt lt selection gt lt script gt file url http proactive inria fr usertiles file scripts checkMatlab js gt lt script gt lt selection gt javaExecutable class 0rg ow2 proactive scheduler ext matlab SimpleMatlab gt lt parameters gt lt parameter name input value in 1 0 3 2 5 2 gt parameter name script valuez out roots in lt parameters gt lt javaExecutable gt lt task gt lt task id root3 preciousResult true gt lt description gt Calculates the root of a polynomial lt description gt lt selection gt lt script gt file url http proactive inria fr usertiles file scripts checkMatlab js gt lt script gt lt selection gt javaExecutable class 0rg ow2 proactive scheduler ext matlab SimpleMatlab gt lt parameters gt lt parameter name input value in 1 0 3 2 5 3 gt lt parameter name script value out roots in gt lt parameters gt lt javaExecutable gt lt task gt
20. run this task This script can for example test that Matlab is installed and has the right version that specific Toolboxes are installed We provide a generic script which simply tests if Matlab is installed The script is retrieved from the URL http proactive inria fr userfiles file scripts checkMatlab js task namez root1 preciousResult irue gt description Calculates the root of a polynomial lt description gt selection script file url http proactive inria fr userfiles file scripts checkMatlab js gt lt script gt lt selection gt lt task gt 5 2 3 3 The Matlab script definition Now we finally write the script that will be executed on the remote machine The javaExecutable tag is a container for our Matlab script it s a java program that will connect to the Matlab engine and launch the given script inside it lt javaExecutable class 0rg objectweb proactive extensions scheduler ext matlab SimpleMatlab gt lt parameters gt lt parameter name input value in 1 0 3 2 5 1 gt parameter name scripi value out roots in gt lt parameters gt lt javaExecutable gt The javaExecutable tag contains an attribute class which tells which type of Matlab task will be used here we ll describe only the task called SimpleMatlab In Section 5 3 2 New Tasks MatlabSplitter and MatlabCollector we describe more advanced tasks The javaExecutable tag contains a child tag called parameters
21. s Eclipse Plugin 39 Part III ProActive Scheduler s Matlab ProActive 4 Part Ill ProActive Scheduler s Matlab extension Table of Contents Chapter 5 ProActive Scheduler s Matlab Extension 4 cesse eere ee eere eee eee eee ee eese se seesseeeesseeeess 4 Sells Presertatloni sro opes eere sere oo anses anne So tubas eve TeA Ee E E sie qrasasnagaodevtonse rien ne een one nantes noeud entente 41 5 2 Quick Start with the Matlab Extension sise EE EEE EEE EEE GE EE EE EEE EEG EE EE EEE GEESE EE 41 2 241 INSTA ATION M 41 5 2 2 Writing a simple example the Matlab Script ss 41 5 2 3 Writing a simple example the Scheduler job descriptor sss 41 5 3 A More Complex Example a Matlab task flow seen 45 9 3 1 Descriptor variables orco iet roris tuetur or uve Passo repas ud ved estu ee usse eros Bl espe ui ie edite diese dinde 47 5 3 2 New Tasks MatlabSplitter and MatlabCollector ss 47 9 3 3 Task dependencies eese tita reo doe tete e eroe ege tonnes preset biens ve pede sc Sat OMNI diese EEE E vea Po Adis 47 5 3 4 New parameter in SimpleMatlab tasks index 5 3 5 Matlab Scripts for this example sise 48 Part III ProActive Scheduler s Matlab ProActive a Chapter 5 ProActive Scheduler s Matlab extension Extension Chapter 5 ProActive Scheduler s Matlab Extension 5 1 Presentation MATLAB is a numerical computing environment and programmin
22. the first child needs to specify index 0 and second index 1 note that the indexes range from 0 to number_of_children 1 By specifying these indexes the first child will get as input the array 1 2 and the second child will get 3 4 5 3 5 Matlab Scripts for this example 5 3 5 1 Script of the Splitter Task n fix length in nout for i 1 nout 1 out i in 1 n i 1 n i end out nout in 1 n nout 1 end The Splitter script contains two important aspects It contains two inputs the variable in which is fed by the input script of the splitter task and the variable nout which contains the value of the number_of_children parameter The out variable which is the output of the script must be a cell array of size nout 5 3 5 2 Script of the Summing Task out sum in 5 3 5 3 Script of the Collector Task out 0 for i 1 length in out out in i end The important aspect of the The Collector script is that the input parameter in is a cell array 48 Part IV ProActive Scheduler s Scilab ProActive 4 Part IV ProActive Scheduler s Scilab extension Table of Contents Chapter 6 ProActive Scheduler s Scilab Extension 4 eee e ee ee eee eee eee eee eese to seen setae setae seta 6 1 Presenta NON ocio rer rotor vertere agre iod eta voor vett ees nine rat e timide ent i nues ut om Ee qe dbotr enter TE eee Ede ge 6 2 Quick Start with the Scilab Extension ss
23. type static gt file path SCRIPT_DIR host_selection js gt arguments argument value EXCLUSION_STRING gt lt arguments gt lt file gt lt script gt lt selection gt lt pre gt lt script gt file path SCRIPT_DIR set js gt lt script gt lt pre gt javaExecutable class org ow2 proactive scheduler examples Property Task gt lt post gt lt script gt file path SCRIPT_DIR unset js gt lt script gt lt post gt lt task gt task id Pl_Computation walltime 00 10 gt lt genericinformation gt 10 Part I ProActive Scheduler ProActive SY Chapter 1 ProActive Scheduler info name name value val1 gt lt genericinformation gt javaExecutable fork true class org objectweb proactive extensions scheduler examples MonteCarlo gt lt forkEnvironment javaHome jvmParameters d32 gt lt parameters gt lt parameter name steps value 20 gt parameter name iterations value 100000000 gt lt parameters gt lt javaExecutable gt lt task gt To have an exhaustive list of which options are available and what they are suppose to do just go to the task explanation section at Section 1 4 4 6 Tasks options and explanations 1 4 4 1 3 Create and Add a Java task using Java API To create a Java task use the JavaTask class In this type you must specify the class you want to start with by mentioning a Class of your executable To
24. variable name WORK_DIR value path to your working dir gt lt variable name SCRIPT_DIR value path to your script dir gt lt variables gt lt genericinformation gt info name vari value WORK _DIR gt info name var2 value val2 gt lt genericinformation gt proActive neededNodes 10 gt task id Controller gt description Will control the workers in order to find the prime number lt description gt proActiveExecutable class 0rg ow2 proactive scheduler examples ProActiveExample gt lt parameters gt lt parameter name numberT oFind value 200 gt lt parameters gt lt proActiveExecutable gt lt task gt lt proActive gt lt job gt Some other stuffs and options can be set onto a ProActive task here s a new example of what can be done with task XML descriptors task id Controller gt description Will control the workers in order to find the prime number lt description gt selection script type static gt 18 Part I ProActive Scheduler ProActive e Chapter 1 ProActive Scheduler file path SCRIPT_DIR host_selection js gt arguments argument value EXCLUSION_STRING gt lt arguments gt lt file gt lt script gt lt selection gt lt pre gt lt script gt file path SCRIPT_DIR set js gt lt script gt lt pre gt lt proActiveExecutable class 0rg ow2 proactive scheduler examples ProActiveExample gt
25. 1 Compute Pi and return it Computation4 transO4 inria fr 13 52 38 11 23 07 13 53 09 11 23 07 0 1 Compute Pi and return it The host name which execute the task The task started time The task finished time Figure 2 4 The Tasks view In the column Re runnable the first number represents how many times the task was re executed and the second number how many times the task can be re executed The description of the task 29 Part I ProActive Scheduler ProActive 4 Chapter 2 ProActive Scheduler poet i Job Info 3 Property Value Id 67 State Stalled Name job nativ Priority Normal Pending tasks number 1 Running tasks number 0 Finished tasks number 3 Total tasks number 4 Submitted time Started time Finished time Pending duration Execution duration Total duration Description om o 13 53 42 11 23 07 13 53 48 11 23 07 Not yet 6s 288ms Not yet Not yet Will execute 3 native C ta gt Figure 2 5 The Job Info view This view provides many informations on the selected job as The job id The job state The job name The job priority The number of pending task The number of running task The number of finished task The number of task composing the job The job submitted time The job started time The job finished time The description of the job 30 Part I ProActive Scheduler ProActive a Chapter 2 ProActive Scheduler Ecplise Plugin E Result Prev
26. Active Scheduler ProActive Chapter 1 ProActive Scheduler Script script new SimpleScript scriptURL args aTask setPreScript script If the script to use is in a Java string for example Script script new SimpleScript Script content type of language where type of language can be any language supported by the underlying JRE aTask setPreScript script same construction for the post script aTask setPostScript script same construction for the selection script the last parameter is still not used in the current implementation SelectionScript selScript new SelectionScript script true aTask setSelectionScript selScript To complete your job by adding the task inside the job just add it as followed add the task to the job job addTask aTask Here s some other features than can be performed on tasks such as dependencies or wallTime admitting task 2 and task 3 has been create just before we have to create task 5 create a new task NativeTask task5 new NativeTask fill task5 as describe above then specify dependencies by using the addDependence Task method task5 addDependence task2 task5 addDependence task3 or use the addDependences list lt Task gt method as shown task5 addDependences new ArrayList lt Task gt task2 task3 set a walltime to stop the process after the given time even it is not finish aTask setWallTime 10000 Here s a last example t
27. Client Platform available for any platform Windows Linux Mac OSX Solaris Asetof Eclipse http www eclipse org plugins with all the functionalities within the stand alone application enhanced with a tool that makes easier the scheduling and monitoring of jobs and applications 2 1 The Scheduler perspective The Scheduler plugin provides the Scheduler perspective http help eclipse org help31 index jsp topic org eclipse platform doc user gettingStarted qs 43 htm displayed in the Figure 2 1 The Scheduler Perspective This perspective defines the following set of views http wiki eclipse org index php FAQ_What_is_a_view 3F The Jobs view shows pending running and finished jobs in the scheduler The Console view shows jobs standard and error output on demand The Tasks view displays detailled informations on tasks contained in the selected job The Jobs Info view displays all informations of the selected job The Result Preview view displays a textual or graphical preview of the result of the selected task 25 Part I ProActive Scheduler Scheduler Eclipse SUR file Edt Navigate Project Window Help jra ii ProActive amp Chapter 2 ProActive Scheduler Ecplise Plugin c E S Jobs 92s 68 exit Pending 1 Finished 2 M State User Priority Name N Id State User Priority 9 Pending chi Normal Matlab job on jo Low job pi
28. Core Main component selects and gives nodes to scheduler maintains different nodes states for each node and receive new available nodes acquired by Node Sources Monitoring component Resource manager can have monitors connected to it Monitors are external programs such as monitor GUI that want to be informed about RM current activity numbers of nodes and their availability for example The Monitoring component is in charge of throwing RM information to its monitors Node Sources Resource manager can handle nodes coming from heterogeneous environments a Node source component is in charge of nodes acquisition deployment and monitoring for a dedicated infrastructure It means we can have a Node source which manages nodes deployed by a ProActive descriptor one for nodes acquired from a Peer to peer infrastructure and another for nodes acquired from a cluster 36 Part II ProActive Resource Manager ProActive 4 Chapter 3 ProActive Resource Manager RESOURCE MANAGER Figure 3 1 resource Manager architecture 3 4 Static Node Source and Dynamic Node Source Node Sources objects are aimed to acquire nodes for the RM there are two kinds of Node Sources Static Node Source defined for deploying and acquiring nodes by a ProActive descriptor All nodes handled by the source are kept permanently by the Resource Manager i e until the termination of the RM or if Administrator ask to remove some of them Dynamic Node Source designe
29. ET SUED P 4 12 1 2 Created JO Drm etos tp taints tune edto at catwaronentenveneatdaceventiatusk cntaecnnentey Aere Pete Er cii oe cerca sede 4 TAD Create a T skElow JOD iret dr tenter tv etes elo rentes EVEN NES Rex EEES tense tite aene EEE etes 5 1A 3 Create a ProActive JOD eere rir POP e creas RN EE REF EEE ees pente er copa ex beet sed eae REPE en canes oan 6 1 4 4 Create and Add a task to a job sise 7 1 4 5 Submit a job to the ProActive Scheduler ssssssssssssssssssee eme me ee 22 146 er a Job RS Seeecig cd aeeie ea EE EEE EEEE AE EEEE 23 1 4 7 Register to ProActive Scheduler events cccceccceeeceeeceeece cece eceeeeeeeeeeceeeceeeeeeeeeeeaeeeeeeeeeeges 23 1 5 Administrator Manual e imite 24 Chapter 2 ProActive Scheduler Ecplise Plugin sense 25 2 1 Th Scheduler Perspective 55e e ree ne rette te cea quads ove e ocu e ge e Ree Yo tn een de esee sU ge dent E ree E t patent 25 2 2 Views composing the perspective ses 27 2 3 Connect to the started ProActive Scheduler sisi 31 2 4 The Scheduler perspective buttons sise 32 2 4 1 The Jobs view buttons in User Mode ssssssssssssssse m mme me e rennen nre 32 2 4 2 The Jobs view buttons in Administrator Mode ssssssssssssssseeene nee eene nennen 33 3 3 Resource Manager ar litectUte ss sie e tiersen Euer De ane esi ptus t
30. Job description lt description gt lt variables gt variable name val1 value toto gt lt variables gt lt genericinformation gt info name vari value val1 gt info name var2 value val2 gt lt genericinformation gt lt taskFlow gt task id task1 retries 2 gt lt description gt human description lt description gt lt javaExecutable class org ow2 proactive scheduler examples WaitAndPrint gt Part I ProActive Scheduler ProActive a Chapter 1 ProActive Scheduler parameters parameter name foo value true gt parameter name bar value 1 gt lt parameter name test value toto gt lt parameters gt lt javaExecutable gt lt task gt task id task2 gt lt depends gt task ref iask1 gt lt depends gt javaExecutable class 0rg ow2 proactive scheduler examples WaitAndPrint gt lt parameters gt parameter name foo value false gt parameter name bar value 12 gt parameter name tesi value titi gt lt parameters gt lt javaExecutable gt lt task gt lt taskFlow gt lt job gt It is obviously possible to mix Java and Native task inside a taskFlow Job Some other stuffs and options can be set onto a Java task here s two examples of what can be done with task XML descriptors task id iaskName preciousResult true gt description Testing the pre and post scripts lt description gt selection lt script
31. Node is busy and administrator or its dynamic Node Source has asked to remove the node So the node will be removed from RM after task s end Down Node has a problem unreachable fallen and can t execute tasks anymore 3 6 Starting the Resource Manager To start the Resource Manager run the RMlauncher sh script in scripts scheduler directory Without arguments Resource Manager will start and create four ProActive nodes on the local host RMlauncher sh can be started with 1 optional argument Path of a ProActive descriptor file for example ProActive descriptors Workers xml Descriptor is deployed and nodes added to the RM by a static Node Source at the RM s startup You can also start Resource Manager using the java API Resource Manager can be started with static functions of RMFactory class Here a short sample of RM instantiation with deployment of a ProActive descriptor Nodes deployment is asked to RMA dmin object RMFactory startLocal creates Resource Manager components RMAdmin admin RMFactory getAdmin get RMAdmin object creates ProActive Descriptor object from an xml file ProActiveDescriptor pad PADeployment getProactiveDescriptor myDescriptor xml Ask to RMAdmin component to deploy the ProActive Descriptor and add deployed nodes to the RM admin addNodes pad 38 Part II ProActive Resource Manager ProActive 4 Chapter 4 Resource Manager s Eclipse Plugin Chapter 4 Resource Manager
32. ProActive Programming Composing Deploying on the Grid ProActive Scheduler OASIS Research Team and ActiveEon Team CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE Version 2008 07 16 10 47 41 2008 07 16 10 47 41 Copyright 1997 2008 INRIA ProActive Scheduler v2008 07 16 10 47 41 Documentation ProActive Scheduler OASIS Research Team and ActiveEon Team Legal Notice The ProActive Scheduler is being distributed under the GPL2 license Copyright INRIA 1997 2008 ProActive Contributors and Contact Information Contributors and Contact Information Team Leader ActiveEon Team Denis Caromel Christian Delb INRIA 2004 Route des Lucioles Arnaud Contes BP 93 Vladimir Bodnartchouk 06902 Sophia Antipolis Cedex Emil Salageanu France phone 33 492 387 631 fax 433 492 387 971 Denis Caromel inria fr OASIS Team Past And External Important Contributors Guillaume Laurent Baptiste De Stefano Lionel Mestre Laurent Baduel Robert Lovas Nicolas Dodelin Matthieu Morel Alexandre di Costanzo Jonathan Martin Yu Feng Guillaume Chazarain Romain Quilici Elton Mathias Imen Filiali Nadia Ranaldo Maxime Menant Johann Fradj Julien Vayssiere Guilherme Perretti Pezzi Abhijeet Gaikwad Franca Perrina Regis Gascon Kamran Qadir Jean Michael Guillamume Bastien Sauvan Abhishek Rajeev Gupta Germain Sigety Elaine Isnard Etienne Vallette De Osia Vasile Jureschi Laurent Vanni Francoise Baude Yulai Yuan Antonio Cansado Sylvain Cussat Blanc Ma
33. Scheduler RCP Client directory unzip the java5 jsr223 patch zip file and execute java jar java5 jsr223 patch jar NOTE A RCP Eclipse graphical client is available but not yet documented for monitoring and controlling a Resource Manager You can find it in the ProActive Download page 3 2 Role As Scheduler manages pool of jobs to execute Resource manager is in charge of supplying Scheduler in resources ProActive nodes Resource Manager RM takes benefits of the ProActive library so it can handle resources from LAN on cluster on P2P desktop Grids or on Internet Grids ResourceManager provides scheduler in nodes according to criteria of the task to execute on it operating system dynamic libraries memory Its main functions are 3 3 Creation acquisition and removal of ProActive nodes Supplying nodes to scheduler for tasks executions Scheduler can ask nodes that verify criteria these criteria are defined in a selection script Maintaining and monitoring its list of node resources and manage states of its handled nodes free busy down Resource Manager architecture Resource Manager is made of five components User Resource Manager s frontend for the Scheduler this component provides for scheduler an entry to get and give back nodes Admin Frontend for RM s administrator provides administrator actions add and remove nodes add and remove different node sources shutting down the Resource manager
34. Task To have an exhaustive list of which options are available and what they are for just go to Section 1 4 4 6 Tasks options and explanations 1 4 4 6 Tasks options and explanations As it has been shown in the different examples it is possible to create 3 types of tasks These 3 types have some common features like name description scripts etc Here s the details of each of these common features id is the name assigned to the task It can be whatever you want as a String This name must be unique for each task description optional is a human readable description of the task It is for human use only This field is optional but it is better to set it generic informations optional is a way to define some informations inside your task This informations could be read inside the policy similar to job s one It can be useful to add new complex scheduling behavior preciousResult optional default is false is the way to define that a result of a task is important or not For example in a job result you could have to retrieve only some task results that are important for you By setting the precious result to true you ll be able to retrieve easily these results retries optional default is 1 is a way to define how many times a task will be reran if a network problems occur Set this value to n if you want the task to be restarted n times and so started a maximum of n 1 times restartOnError optional default is
35. This tag contains a list of parameter tags which define the task parameters Each parameter tag has name value couple attributes The SimpleMatlab task accepts the following parameters script defines which matlab script will be launched The value attributes will contain the matlab script code useful for one line scripts only scriptFile defines which matlab script will be launched The file at the given path will be loaded scriptUrl defines which matlab script will be launched The file at the given remote url will be loaded 42 Part III ProActive Scheduler s Matlab ProActive 4 Chapter 5 ProActive Scheduler s Matlab extension Extension input defines an input script which will be launched before the actual matlab script The value attribute needs to contain the script code which must be single line only 43 xml version 1 0 encoding UTF 8 gt job xmlns xsiz http www w3 org 2001 XMLSchema instance xsi schemaLocation urn proactive jobdescriptor 0 91 Par http proactive inria fr schemas jobdescriptor 0 91 schedulerjob xsd ext xmins urn proactive jobdescriptor 0 91 id Matlab job _simplest gt lt description gt A simple Matlab job which computes the roots of several polynomials lt description gt lt taskFlow gt task id rooti preciousResult true gt lt description gt Calculates the root of a polynomial lt description gt lt selection gt lt script gt file url http
36. aTask addArgument 5o0 new Boolean true aTask addArgument bar new Integer 12 aTask addArgument test test add executable class or instance pat setExecutableClassName org ow2 proactive scheduler examples ProActiveExample add number of nodes needed for the application pat setNumberOfNodesNeeded 10 SCRIPTS EXAMPLE If the script to use is in a file or URL String args new String foo bar 19 Part I ProActive Scheduler ProActive Chapter 1 ProActive Scheduler File scriptFile new File path to script file URL scriptURL new URL url to script file Script script new SimpleScript scriptFile args Script script new SimpleScript scriptURL args aTask setPreScript script If the script to use is in a Java string for example Script script new SimpleScript Script content type of language where type of language can be any language supported by the underlying JRE aTask setPreScript script same construction for the post script aTask setPostScript script same construction for the selection script the last parameter is still not used in the current implementation SelectionScript selScript new SelectionScript script true aTask setSelectionScript selScript To complete your job by adding the task inside the job just add it as followed note that you can only add ONE ProActive task in a ProActive Job add the task to the job job addTask a
37. al features can be set on this job name is a way to identify your job or just simply name it If this value is left to the empty string the Scheduler will set it by a default one projectName optional can be define in your job This information also goes to the policy in order to group different job by project name for example priority optional is the scheduling priority level for your job A user can only set its job priority to lowest low or normal CancelOnError optional is a way to define if your job will continue if a user exception or error occurs during the whole job process It means that if the value of this property is true the job will stop immediately every running task if one error occurs in one of the task of this job It will have the consequence to failed the job but free resources for other jobs It is useful when it is no need to go further after a task failure logFile optional is the path of an optional log file Set it if you want to save the job generated log in a file description optional is a human readable description of the job for human use only This field is optional but it s better to set it genericInformation optional is a way to define some informations inside your job These informations could be read by the policy of the Scheduler It can be useful to modify the scheduling behavior Contact your administrator if you want an information to be interpreted by the policy To create
38. and add tasks to your Job just go to Section 1 4 4 Create and Add a task to a job 1 4 2 Create a TaskFlow job The TaskFlowJob or data flow job is a job that can contain one or more task s with the dependencies you want To start with the job creation please first read Section 1 4 1 Create a job 1 4 2 1 Create a TaskFlow job using XML descriptor To specify that the job is a TaskFlow Job just add the taskFlow tag Here s an example of how to go on to a TaskFlow Job using the previous job descriptor Part I ProActive Scheduler ProActive a Chapter 1 ProActive Scheduler lt xml versionz 1 0 encoding UTF 8 gt job xmins urn proactive jobdescriptor 0 91 xmins xsiz http www w3 org 2001 XML Schema instance xsi schemaLocation urn proactive jobdescriptor 0 91 http proactive inria fr schemas jobdescriptor 0 91 schedulerjob xsd id job_name priority normal projectName project_name cancelOnError true logFile path to a log file log gt lt description gt Job description lt description gt lt variables gt variable name vali value toto gt lt variables gt lt genericinformation gt info name var1 value val1 gt info name var2 value val2 gt lt genericinformation gt lt taskFlow gt lt Job will be completed here later gt lt taskFlow gt lt job gt To create and add tasks to your Job just go to Section 1 4 4 Create and Add a task to a job
39. can see submitting a job will return a Job ID This is the identification code of the submitted Job It is useful to save it in order to retrieve future informations on this job 1 4 6 Get a Job result Once a Job terminated it is possible to get its result You can only get the result of the job that you own 1 4 6 1 Get a Job result using the Graphical User Interface Scheduler Eclipse Plugin To get a job result using the graphical tools please refer to Chapter 2 ProActive Scheduler Ecplise Plugin documentation 1 4 6 2 Get a Job result using shell command To get the result of a job using a command line use the getResult sh script in the bin unix getResult sh directory This script has 2 optional options The URL of a started scheduler using the u URL option If you don t use this it will try to connect to a started scheduler on local host Your login using the 1 LOGIN option If you use this option only your password will be requested Otherwise both will be requested It will print the result on the screen as the toString Java method could have done it For more informations use h or help option i e jobLauncher sh h 1 4 6 3 Get a Job result using Java API To do it in Java use the getJobResult JobId method in the UserSchedulerInterface and the job ID you got when you submitted it It is also possible to create a new ID based on the integer id you got A job result is in fact a list of task resul
40. cation This kind of Job requires the usage of the ProActive API in order to be able to build ProActive application A finished Job contains a result that is provided by the scheduler once the job terminated which in term contains all of its tasks results However it is possible to mark some task as precious in order to retrieve their result easily in the job result In the event of a failure the finished Job contains the causes of the exception Further details on how to create a Job and the different options can be found in Section 1 4 1 Create a job 1 3 2 What is a Task The Task is the smallest schedulable entity It is included in a Job see Section 1 3 1 What is a Job and will be executed in accordance with the scheduling policy see Section 1 3 4 Scheduling Policy on the available resources Part I ProActive Scheduler ProActive Chapter 1 ProActive Scheduler There are three types of Tasks JAVA its execution is defined by a Java class extending the org ow2 proactive scheduler common task executable JavaExecutable class e NATIVE its execution can be any user program specified by a simple command line or by a generation script that can dynamically generates the command line to be executed PROACTIVE its execution is defined by a Java class extending the org ow2 proactive scheduler common task executable ProActiveExecutable class which defines a ProActive application Coding this
41. criptor variables The variables declaration allows a user to define a variable which can be used as a pattern in other parts of the descriptor This helps writing more generic descriptors and replacing only the variables values to adapt the descriptor to many contexts Here is the variables declaration in the preceding descriptor variables variable namez HOME valuez user fviale home variable namez MATLAB SCRIPTS value HOME matlab gt lt variables gt the variables tag contains a list of variable tags which each defines a variable through a name and a value attribute The variable can then be used by writing the pattern name_of_the_variable Variable can be reused inside the variable declaration itself but the variables are processed sequentially from top to bottom Therefore in this example In this example the MATLAB_SCRIPTS variable could not be used before the HOME variable 5 3 2 New Tasks MatlabSplitter and MatlabCollector lt javaExecutable class org objectweb proactive extensions scheduler ext matlab MatlabSplitter lt parameters gt parameter name scriptFile value MATLAB_SCRIPTS splitter m gt parameter name input value in round rand 1 1000000 100 50 gt lt parameter name number_of_children value 6 gt lt parameters gt lt javaExecutable gt lt javaExecutable class org objectweb proactive extensions scheduler ext matlab MatlabCollector gt lt parameters gt para
42. d by a description tag which gives textual description of the job Finally the next tag will be the type of job to schedule In our case it will be a taskFlow job a job containing several tasks 41 Part III ProActive Scheduler s Matlab ProActive 4 Chapter 5 ProActive Scheduler s Matlab extension Extension job xmIns xsi http www w3 org 2001 XMLSchema instance xsi schemaLocation urn proactive jobdescriptor 0 91 http proactive inria fr schemas jobdescriptor 0 91 schedulerjob xsd xmins urn proactive jobdescriptor 0 91 name Matlab_job_simplest gt lt description gt A simple Matlab job which computes the roots of several polynomials lt description gt lt taskFlow gt lt taskFlow gt lt job gt 5 2 3 2 The task definition The task tag contains all the information for a single task executed on a single machine In our example this task will be the matlab script calculating the root of a polynomial The task tag must contain a name attribute like the job tag Here it contains as well the attribute preciousResult which tells the scheduler that we need the result of this task as final output for our job The task tag is immediately followed by a description tag containing a textual description of this task The description tag is followed by a selection tag This tag describes a script which will select among all the machine resources that the Scheduler controls the specific resources machine that can effectively
43. d to acquire nodes from an infrastructure which can provide nodes just for a limited time So this Node Source get a node from a specific infrastructure keep it during a defined time When this keeping time is elapsed dynamic node source remove the node from the RM and give back node to its infrastructure A dynamic node source have three main parameters Max number of nodes number of nodes that dynamic Node Source has to get from its corresponding infrastructure The dynamic node source tries to acquire this number of node source but its infrastructure may not be able to provide as many nodes So this is the number of acquired nodes that Dynamic Node Source tries to reach Time to release TTR keeping duration of an acquired node when this keeping duration is reached by a node dynamic node source releases it Nice time After a node release time to wait for the dynamic node source before trying to get a new node from its infrastructure After each node release dynamic node source waits nice time and after tries to acquire a new node 3 5 Nodes states Resource Manager has to maintain states of its handled nodes here the different nodes states Free Node is available and there is no task launched on it Node can be supplied to a scheduler 37 Part II ProActive Resource Manager ProActive 4 Chapter 3 ProActive Resource Manager Busy Node has been given to scheduler and a task is executed on it To be released
44. de la tache numero task2 13 48 02 Terminaison de la tache numero task3 13 48 05 Terminaison de la tache numero taskl 13 48 65 Terminaison de la tache numero task8 13 48 26 Terminaison de la tache numero CO p UJ ut oi h_a Figure 2 3 The Console view This view displays all jobs standard and error output only on demand 28 Part I ProActive Scheduler Tasks x Id 50007 Pending 50005 Pending 50003 Pending 50001 Running 50008 Finished 50009 Finished 50006 Finished 50004 Finished 50002 Finished This view provides many informations on tasks composing a job as The task id The task state The task name ProActive amp Job 50 has 9 tasks Chapter 2 ProActive Scheduler Ecplise Plugin Name Host name Start time Finished time Re run Description Averagel n a Not yet Not yet 0 1 Do the average of 1 2 3 and return it LastAverage n a Not yet Not yet 0 1 Do the average of average 1 2 and return it Average2 n a Not yet Not yet 0A Do the average of 4 5 6 and return it Computation6 nahuel nria fr 13 52 39 11 23 07 Not yet 0 1 Compute Pi and return it Computation2 amda inria fr 13 52 36 11 23 07 13 53 14 11 23 07 0 1 Compute Pi and return it Computation5 puravida inria fr 13 52 24 11 23 07 13 53 09 11 23 07 0 1 Compute Pi and return it Computation3 pincoya inria fr 13 52 36 11 23 07 13 53 08 11 23 07 0A Compute Pi and return it Computationl macyavel inria fr 13 52 37 11 23 07 13 53 09 11 23 07 0
45. de priorities Highest higher priority than the 3 User Mode priorities and than High b Figure 2 17 Start the scheduler Figure 2 18 Stop the scheduler Zzz Figure 2 19 Freeze the scheduler This freezes the scheduler When the scheduler receives this event it pauses all running jobs and no other pending jobs will be scheduled Hd Figure 2 20 Pause the scheduler This pauses the scheduler When the scheduler receives this event no pending jobs will be scheduled but all running jobs complete Ol Figure 2 21 Resume the scheduler Lim Figure 2 22 Shutdown the scheduler This shutdowns the scheduler When the scheduler receives this event job submission is no more allowed but all running jobs complete When all jobs are finished the scheduler is shutdown En ko Figure 2 23 Kill scheduler 33 Part I ProActive Scheduler ProActive 4 Chapter 2 ProActive Scheduler Ecplise Plugin This shutdown immediately the scheduler without waiting for any job completion 34 Part II ProActive Resource Manager ProActive Part Il ProActive Resource Manager Table of Contents Chapter 3 ProActive Resource Manager sascsssssssscsssesescsasassicsissssasesssssaacnaateuesasasidassasesassteaaocnaisiacenncasvaneead 36 31 IMPORTANT NOTE sesceusscshestnesspescenssarveesstoewonneie guae eee tetuomay or vawtyiouewiests ord a do S E E E E EE 36 Did NJ RE 36 3 3 Resource Manager archite Chute sacs ss ssieicarsenv
46. der in which you specify that Task 5 depends of Task 2 and 3 is very important Indeed if you set the list of dependencies for Task 5 as 2 then 3 the result of these two task will be given to Task 5 in this order 1 3 4 Scheduling Policy By default the scheduler will schedule tasks according to the default FIFO First In First Out with job priority policy So if you want a job to be scheduled quickly increase its priority or ask your administrator for an other policy 1 4 User Manual 1 4 1 Create a job A job is the entity that will be submitted to the ProActive Scheduler As it has been explained in the Section 1 3 1 What is a Job it s possible to create more than one type of job A job can also be created using an XML descriptor or the provided ProActive Scheduler Java API 1 4 1 1 Create a job using XML descriptor Just follow the example below in order to create your Job with XML description lt xml versionz 1 0 encoding UTF 8 gt job xmins urn proactive jobdescriptor 0 91 xmins xsiz http www w3 0rg 2001 XML Schema instance xsi schemaLocation urn proactive jobdescriptor 0 91 http proactive inria fr schemas jobdescriptor 0 91 schedulerjob xsd id job name priority normal projectName project name cancelOnError true logFile path to a log file log gt lt description gt Job description lt description gt lt variables gt variable name val1 value toto gt lt variables gt lt g
47. e amp Chapter 1 ProActive Scheduler make your listener SchedulerEventListener mySchedulerEventListener new chedulerEventListener public void jobRunningToFinishedEvent JobEvent event if my job is finished if event getJobld equals myJobld get its result JobResult myResult scheduler getJobResult myJobld Implement other methods D add the listener to the scheduler specified which events you want to receive scheduler addSchedulerEventListener MySchedulerEventListener SchedulerEvent JOB RUNNING TO FINISHE This example shows you how to listen to the scheduler events here the finished job event only But you can listen for every events you want containing in this interface For more details and features on the user scheduler interface please refer to the java Documentation TODO 1 5 Administrator Manual 24 Part I ProActive Scheduler ProActive a Chapter 2 ProActive Scheduler Ecplise Plugin Chapter 2 ProActive Scheduler Ecplise Plugin The Scheduler Eclipse Plugin is a graphical client for remote monitoring and control of the ProActive Scheduler see Chapter 1 ProActive Scheduler including remote submission of XML defined jobs see Section 1 4 1 1 Create a job using XML descriptor The Scheduler Eclipse Plugin is available in two forms e A Java stand alone application based on Eclipse Rich Client Platform RCP http wiki eclipse org index php Rich
48. e d eU parbVR IER DUREE S 31 2 0 Connect TO scheduler eter nent dens ute tuos xaeenidrer e En a des nup E dencre ess date x rtp us e PERO VR Cres 32 2 9 Disconnect from the scheduler sssrini eaer repr ve dete oro rU ee ee IE Yea a dosette tente ie ERNEA tienne 32 2 10 Change view from Vertical to Horizontal mode sise 32 2 11 Change view from Horizontal to Vertical mode sise 32 212 SUDMUGA ojo 32 2 13 Pause R sume d Job 32 2 14 Change Job priority lt 2 sich Athens D MO lente 32 2 15 Display JOD Output ME m 32 PATH SB Re E E EENE e Ee AE ER EE E E E e S 33 2 17 Start the schedulet PE ENSEN EEREN EEEE EENES 33 2 18 Stop Xni sm na EERE EELEE ED EREE EET CENENE EEEE sed AEE tin ele EEEREN EEEE EEDEN tutos 33 219 Freeze the sched let saiisine resena re EE EN EEE E Mes erT RAE OE E EE seeds 33 220 Pause th scheduleiz iuc rere repe ree ete rie OOE eSEE e dy rE EESE EE OEEO SEEE 33 2 21 Resume The scheduler oec eee eee EE EENE a E o E e a detecte 33 2 22 Shutdown the scheduler eot IEEE nE EEE E EE EAEE SEEE 33 2 235 Kill schedu let rotor oe e trs rh S e E a EE eE ne a E Eea eE EES 33 3 1 resource Mana per architecture eet pete eese dates ed beer edere Dk bogus EEEE OEE EEES EAEE EE REEERE TEE Ve OPERE Ee DA 37 ProActive amp List of Examples 5 1 Simple Matlab Job descriptor Example
49. e eee eee e nee mee eese heres e esr er e ese eren 6 2 PM ebur ete 6 2 2 The Scilab Job descriptor sirere iiie Ee ter Pro dense tannins EE TO EK ENS TERES E PENNE SML te Part IV ProActive Scheduler s Scilab ProActive a Chapter 6 ProActive Scheduler s Scilab extension Extension Chapter 6 ProActive Scheduler s Scilab Extension 6 1 Presentation Scilab is a scientific software for numerical computations Developed since 1990 by researchers from INRIA and ENPC it is now maintained and developed by Scilab Consortium since its creation in May 2003 Scilab includes hundreds of mathematical functions with the possibility to add interactively programs from various languages C Fortran It has sophisticated data structures including lists polynomials rational functions linear systems an interpreter and a high level programming language Scilab works on most Unix systems including GNU Linux and Windows 9X 2000 XP Similarly to ProActive Scheduler s Matlab extension the goal of the Scilab Extension is to allow users to easily launch Scilab scripts over an heterogeneous grid describe complex task flows in Scilab using human readable XML descriptors Ability to communicate the result of one task as input of another task users won t have to write any other code than Scilab script code support the following Scilab types Double Complex and String Arrays the extension currently supports fewer ty
50. enericinformation gt info name vari value val1 gt info name var2 value val2 gt lt genericinformation gt lt Job will be completed here later gt lt job gt As shown several features can be set on this job id is a way to identify your job or just simply name it If this value is left to the empty string the Scheduler will set it by a default one projectName optional can be define in your job This information also goes to the policy in order to group different job by project name for example priority optional is the scheduling priority level for your job A user can only set its job priority to lowest low or normal CancelOnError optional is a way to define if your job will continue if a user exception or error occurs during the whole job process It means that if the value of this property is true the job will stop immediately every running task if one error occurs in one of the task of this job It will have the consequence to failed the job but free resources for other jobs It is useful when it is no need to go further after a task failure logFile optional is the path of an optional log file Set it if you want to save the job generated log in a file description optional is a human readable description of the job for human use only This field is optional but it s better to set it variables optional is a way to define variables which can be reused throughout thi
51. ers gt lt javaExecutable gt lt task gt lt task id task2 gt lt depends gt task ref iask1 gt lt depends gt javaExecutable class org ow2 proactive scheduler examples WaitAndPrint gt lt parameters gt parameter name foo value false gt parameter name bar value 12 gt lt parameter name test value titi gt lt parameters gt lt javaExecutable gt lt task gt The Java Task is composed of one javaExecutable that specified the executable Java class to use A set of parameters has also be defined to provide the executable some informations These parameters will be available into the HashMap of the init HashMap method into your JavaExecutable This example also shows the definition of two tasks with dependencies We can easily see that task 2 depends on task 1 So task 2 will be executed when task 1 has finished To put these two tasks inside your TaskFlow job just put it between the taskFlow tags Here s how a complete ready to be scheduled TaskFlow Job seems like lt xml versionz 1 0 encoding UTF 8 gt job xmins urn proactive jobdescriptor 0 91 xmins xsiz http www w3 0rg 2001 XML Schema instance xsi schemaLocation urn proactive jobdescriptor 0 91 http proactive inria fr schemas jobdescriptor 0 91 schedulerjob xsd id job_name priority normal projectName project name cancelOnError true logFile path to a log file log gt lt description gt
52. ex starts from 1 and the order matches the depends list order 52
53. exi EEEE te tenue Ce EEEE EEEE EEEE EEs 36 3 4 Static Node Source and Dynamic Node Source essieu 37 3 53 NOdES SALES A EPEEPT E 37 3 6 Starting the Resource Manager sise 38 Chapter 4 Resource Manager s Eclipse Plugin sssccsssscssssscssssscesssccssssccssssccssssscssssscssssscsssses 39 Part III ProActive Scheduler s Matlab extension ccccccssscsccsssssscccssssccccccsccceseees Chapter 5 ProActive Scheduler s Matlab Extension eee eee esee eere ee eren eee ee eese seeeesseees FL Dells Presentation cereo stories tere Eoo a chere ds cabanes tee gero tes testo eee ace does dote a A ii ProActive amp 5 2 Quick Start with the Matlab Extension si se eeeeeeeeeeae eee eeeeeeeaeeneeneee 41 p LAB ES PU AT 1 m 41 5 2 2 Writing a simple example the Matlab Script ssssssssssssssee mmm 41 5 2 3 Writing a simple example the Scheduler job descriptor sssssssseem e 41 5 3 A More Complex Example a Matlab task flow ssssssssssssssssesee e meme re rene 45 5 3 1 Descriptor variables oti teeth ee petto sn UR bat vac cinder den nube emos eo arae UA denim E eT VR e Ei 47 5 3 2 New Tasks MatlabSplitter and MatlabCollector sess 47 5 3 3 Task dependenCles iie ori nee E er rentre RESO en duae Pa RERPUI eee EE RE AERRUN FR ERES MER EE 47 5 3 4 New parameter in S
54. ext scilab SimpleScilab gt lt parameters gt lt parameter name scriptFile value SCILAB_SCRIPTS intsin sci gt lt parameter name input value i 3 n 5 gt parameter name outputs value out gt lt parameters gt lt javaExecutable gt The main Scilab task is now called SimpleScilab There exists no splitting mechanism yet but there is a merging mechanism explained below A new important task parameter appears outputs This parameter is used to specify which variables will be extracted from the Scilab environment at task s end You can specify multiple output variables by separating them with commas If you don t specify an output parameter the variable called out will be extracted leading to an error if it doesn t exist The merging mechanism is different than for Matlab s If a task depends from a bunch of other tasks An automatic environment merging will be done For example if we have 3 tasks A B C and C depends of A and B if task A outputs a variable a and task B outputs a variable b task C will get as input both variable a and b Now a problem arise when several tasks output the same variable name In order to avoid overlapping and to allow merging of results this variable will be renamed by appending index at the end of the conflicting variable name In the current example each tasks t1 t5 output the same variable out task t6 will accordingly get as input variable out out5 The ind
55. g language Created by The MathWorks MATLAB allows easy matrix manipulation plotting of functions and data implementation of algorithms creation of user interfaces and interfacing with programs in other languages The Goal of ProActive Scheduler s Matlab Extension is to allow users to easily launch Matlab scripts over an heterogeneous grid describe complex task flows in Matlab using human readable XML descriptors Ability to communicate the result of one task as input of another task users won t have to write any other code than Matlab script code support the following Matlab types Double Complex Int or String Arrays Cells Records A good way to start manipulating and scheduling Matlab scripts is to have a look at the standalone and simple Matlab Scilab GUI described in the ProActive documentation If you want to directly through more complex Matlab job scheduling go on with the following tutorial 5 2 Quick Start with the Matlab Extension To get quickly our hands in we ll write a very simple Matlab job example This simple example will compute the roots of several polynomials 5 2 1 Installation Before starting to use the Matlab interface you need to install the Matlab interface to Java You ll find all the instructions on PROACTIVE scripts unix matlab README This interface will build the native libraries of the Java Interface to Matlab As this library is native it is important that you build it for each cou
56. gt task id task2_native gt lt description gt Will display 10 dots every 2s lt description gt lt depends gt lt task ref task1_native gt lt depends gt lt nativeExecutable gt lt staticCommand value SCRIPT_DIR launcher sh gt lt arguments gt argument value WORK_DIR native_exec gt lt argument value 2 gt lt arguments gt lt staticCommand gt lt nativeExecutable gt lt task gt The native Task is composed of one nativeExecutable that specified the executable process to use A set of parameters has also be defined to provide the executable some informations These parameters will be append to the command line starting by your native executable This example also shows the definition of two tasks with dependencies We can easily see that task2 native depends on task1 native So task2 native will be executed when task1 native has finished To put these two tasks inside your TaskFlow job just put it between the taskFlow tags Here s how a complete ready to be scheduled TaskFlow Job seems like lt xml versionz 1 0 encoding UTF 8 gt job xmins urn proactive jobdescriptor 0 91 xmins xsiz http www w3 org 2001 XML Schema instance xsi schemaLocation urn proactive jobdescriptor 0 91 http proactive inria fr schemas jobdescriptor 0 91 schedulerjob xsd id job_name priority normal projectName project_name cancelOnError true logFile path to a log file log gt lt desc
57. h to the job file descriptor is mandatory using the j PATH option The URL of a started scheduler using the u URL option If not mentioned the script will connect an existing localhost Scheduler e Your login using the 1 LOGIN option If you use this option only your password will be requested Otherwise both will be The number of jobs to submit by default only 1 will be submitted using the n A NUMBER option For example jobLauncher sh j jobs descriptors Job with dep xml l login n 12 u localhost will submit 12 times the Job with dep job to a local ProActive Scheduler and only your password will be requested Authorized username and password are defined by the administrator For more informations use h or help option i e jobLauncher sh h 1 4 5 3 Submit a job using Java API To connect the ProActive Scheduler and submit a Job using Java API just proceed as following 22 Part I ProActive Scheduler ProActive Chapter 1 ProActive Scheduler join an existing ProActive Scheduler retrieving an authentication interface SchedulerAuthenticationInterface auth SchedulerConnection join host SCHEDULER OBJECT NAME connect and log to the Scheduler Valid username and password are define by the administrator UserSchedulerlnterface scheduler auth logAsUser username password submitting a new job and get the associated id Jobld myJobld scheduler submit job As you
58. hat describe how to create a native task with a dynamic command i e generated by a script called a generation script The generation script can only be associated to a native task the execution of a generation script must set the string variable command The value of this variable is the command line that will be executed by the ProActive Scheduler as task execution create a new native task NativeTask task2 new NativeTask create a generation script with a script as shown above GenerationScript gscript new GenerationScript script set the command to execute as a string task2 setGenerationScript gscript To have an exhaustive list of which options are available and what they are for just go to Section 1 4 4 6 Tasks options and explanations 16 Part I ProActive Scheduler ProActive e Chapter 1 ProActive Scheduler 1 4 4 3 Create and Add a ProActive task Note It is only possible to add a ProActive task only in a ProActive Job To learn how to create a ProActive Job just go to Section 1 4 3 Create a ProActive job Once your ProActive Job created it is possible to just add ONE ProActive task inside your job 1 4 4 3 1 Define your own ProActive executable First of all you must know that you can create your own ProActive executable by implementing scheduler executable interfaces What is called executable is in fact the executed process that is a Java class in this case Here s an example to create you
59. her e a 1 6 or greater Java Runtime Environment without any modifications e or with a 1 5 JRE the JSR 223 jar files http jcp org aboutJava communityprocess final jsr223 index html First the script api jar script js jar and js jar files must be added in the ProActive dist lib directory if you are using the bin release or ProActive or in the ProActive lib directory if you build ProActive from the source release Then the javab5 jsr223 patch jar patch released with the Scheduler RCP Client should be executed in the Scheduler RCP Client directory unzip the java5 jsr223 patch zip file and execute java jar java5 jsr223 patch jar 1 2 Overview The execution of parallel tasks on a pool of distributed resources what we call nodes such as network of desktops or clusters requires a main system for managing resources and handling task execution a batch scheduler A batch scheduler provides an abstraction of resources to users Users submit jobs containing tasks to the scheduler who is in charge of executing these tasks on the resources A scheduler allows several users to share a same pool of resources and also to manage all issues related to distributed environment such as faulted resources The ProActive Scheduler is connected to a resource manager that will do the resource abstraction see Chapter 3 ProActive Resource Manager In this chapter we present a ProActive based Scheduler accessible either from a Java program
60. iew 28 ka DEFAULT DESCRIPTION 3 14162024 Figure 2 6 The Result Preview view This view display the result of the selected task in task view according to the ResultPreview field see Section 1 4 4 Create and Add a task to a job 2 3 Connect to the started ProActive Scheduler In order to establish a connection to the ProActive Scheduler 1 open the Scheduler Perspective Window gt Open Perspective gt Other gt Scheduler it could be already opened as it is the default perspective 2 select Connect to the ProActive scheduler in the Scheduler menu or in the contextual menu right click it opens the Connect to the ProActive scheduler dialog displayed in the Figure 2 7 Connect to scheduler 3 enter informations required about the remote scheduler and click OK note If you check log as admin in the previous dialogue if the ProActive scheduler accepts your connection you ll be able to do more actions than an simple user see Section 2 4 2 The Jobs view buttons in Administrator Mode se 6 c rose 6 Ce e e ciel ti sf Url irmi fiacre inria fr 6622 5CHEDULER v login johann v password log as admin Figure 2 7 Connect to scheduler 31 Part I ProActive Scheduler ProActive e Chapter 2 ProActive Scheduler Ecplise Plugin 2 4 The Scheduler perspective buttons 2 4 1 The Jobs view buttons in User Mode dr Figure 2 8 Connect to scheduler
61. ig array will be much faster on a single machine due to the overhead of launching Java and a Matlab engine and the network latency but it is meant to illustrate a simple task flow in Matlab We ll go through the new concepts introduced in this example compared to the previous one Have a look at the new job descriptor first 45 Part III ProActive Scheduler s Matlab ProActive 4 Chapter 5 ProActive Scheduler s Matlab extension Extension lt xml versionz 1 0 encoding UTF 8 gt job xmlns xsiz http www w3 org 2001 XMLSchema instance xsi schemaLocation urn proactive jobdescriptor 0 91 http proactive inria fr schemas jobdescriptor 0 91 schedulerjob xsd xmins urn proactive jobdescriptor 0 91 id Matlab_job priority normal logFile HOME matlab_job log gt lt description gt A simple Matlab taskflow which computes the sum of a randomly generated array lt description gt lt variables gt variable namez PROACTIVE HOME value home user ProActive gt variable name MATLAB SCRIPTS value PROACTIVE_HOME scripts unix matlab examples gt lt variables gt lt taskFlow gt lt task id splitter gt lt description gt Splits a big array lt description gt lt selection gt lt script gt file urlz http proactive inria fr userfiles file scripts checkMatlab js gt lt script gt lt selection gt javaExecutable class 0rg ow2 proactive scheduler ext matlab MatlabSplitter gt lt
62. impleMatlab tasks index sss rrr 47 5 3 5 Matlab Scripts for this example eerte eter eek neki Ere eR eR iane eie 48 Part IV ProActive Scheduler s Scilab extension eee ecce eee eee ee eee ee esee eeeno Chapter 6 ProActive Scheduler s Scilab Extension eere ecce eee eene eren eese esee eeeseeeeesse DU 6 1 Presentation m 50 6 2 Quick Start with the Scilab Extension sssssssssssssssessee enne ee enne ener serre nennen ener nens 50 6 2 PECES 50 6 2 2 The Scilab Job descriptor sise eee ter tea nere cast Poe ante Ye eder vea VE Re en eine ee eesi 50 ProActive amp List of Figures 1 1 Task flow job example s morei cuttin vats a ext eiotuu ss Oe vote lates cauergenbs dnthionacuengeaws aeS E yes ceptacemenet 3 1 2 CancelOnError and RestartOnError behavior ii ea ease seen ease e ese ese sese enone 21 2 1 The Sch dul r Perspective D 26 22 WG VOWS VIEW aeeoa REP UMEN Hd tame MIRI UM 27 23x The Console View auior arin EU TU D LE RIKUEIIMMM 28 2s MEP RTS UL ARCET ccm 29 2 9 Whe JOD AMES VIEW RTT 30 2 6 The Result Preview VIEW ois teu mirer Sentit Miet DURUM nes dees tetes RIVE IER seen unin QUIE E NOTA E 31 2 7 Connect to schedulet series rtr aeta te este votes sen UR v Ee desires ke d rb E x bod Ter
63. ipt to use is in a Java string for example Script script new SimpleScript Script content type of language where type of language can be any language supported by the underlying JRE aTask setPreScript script same construction for the post script aTask setPostScript script 11 Part I ProActive Scheduler ProActive Chapter 1 ProActive Scheduler same construction for the selection script the last parameter is still not used in the current implementation SelectionScript selScript new SelectionScript script true aTask setSelectionScript selScript To complete your job by adding the task inside the job just add it as followed add the task to the job job addTask aTask Here s some other features than can be performed on tasks such as dependencies or wallTime admitting task 2 and task 3 has been create just before we have to create task 5 create a new task JavaTask task5 new JavaTask fill task5 as describe above then specify dependencies by using the addDependence Task method task5 addDependence task2 task5 addDependence task3 or use the addDependences list Task method as shown task5 addDependences new ArrayList Task task2 task3 set this task as forked aTask setFork true or set a walltime aTask setWallTime 10000 you can also define a fork environment for example ForkEnvironment env new ForkEnvironment env setJavaHome Your ja
64. last one requires a knowledge base on the use of ProActive Needed resources are provided it is no need to learn about the deployment During its execution a Task can crash due to host or code failure It s good to know that a Task can be re started a parameterizable number of time see re runnable in section Section 1 4 4 Create and Add a task to a job A Task may optionally be accompanied by 3 kinds of scripts pre script post script and selection script that allow to select the suitable resource for a given task and possibly configure it before and after task execution see Section 1 4 4 Create and Add a task to a job Dependencies between Tasks can also be defined this aspect is detailed in next section 1 3 3 Dependencies between Tasks Dependencies can be set between Tasks in a TaskFlow Job It provides a way to execute your tasks in a specified order but also to forward the results of an ancestor task to its children as parameter Dependency between task is then both a temporal dependency and a data dependency ee Figure 1 1 Task flow job example In this example we made an 8 Tasks Job where the Job s type is TaskFlow As you can see Task 4 depends on Task 1 Task 5 depends on Tasks 2 and 3 etc In other words Task 4 will wait for Task 1 to finish before starting Task 5 will wait for Task 2 Part I ProActive Scheduler ProActive a Chapter 1 ProActive Scheduler AND 3 etc In addition the or
65. make your own executable see the proper section Section 1 4 4 1 1 Define your own Java executable In addition you can add arguments with which the task will be launched These launching arguments will be given to the Java executable as a Map Just take a look at the example below to see how to use the task creation Java API see also Java DOCumentation of the Scheduler to learn more create a Java Task with the default constructor that we ll call aTask JavaTask aTask new JavaTask then set the desired options for example aTask setName task 1 aTask setDescription his task will do something aTask addGenericinformation key value aTask setPreciousResult true aTask setRerunnable 2 aTask setRestartOnError RestartMode ELSEWHERE aTask setResultPreview UserDefinedResultPreview class add arguments optional aTask addArgument foo new Boolean true aTask addArgument bar new Integer 12 aTask addArgument test test add executable class or instance pat setExecutableClassName org ow2 proactive scheduler examples WaitAndPrint SCRIPTS EXAMPLE If the script to use is in a file or URL String args new String foo bar File scriptFile 2 new File path to script file URL scriptURL new URL url to script file Script script new SimpleScript scriptFile args Script script new SimpleScript scriptURL args aTask setPreScript script If the scr
66. meter name scriptFile value MATLAB_SCRIPTS collector m gt lt parameters gt lt javaExecutable gt Two new tasks appear in this descriptor the MatlabSplitter and the MatlabCollector The Splitter task is used to split an input into a list of several chunks The Collector task is used to collect and merge the results from several parallel tasks Each of these tasks come with the same parameter as the SimpleMatlab tasks with an addition The Splitter expects an additional parameter called number_of_children This parameter tells the Matlab script responsible for splitting in how many parts the input should be divided 5 3 3 Task dependencies In order to do complex task flows it is necessary to add the concept of dependencies between tasks lt task name sumi gt lt description gt Calculates the sum of the elements in the array lt description gt lt depends gt lt task ref splitter gt lt depends gt lt task gt The depends tag in this task definition defines a dependency of the task named sum1 to the task named splitter This means that the task sum1 will be launched afer the task splitter is complete and that the outputs of splitter will be fed as inputs to sum1 You ll notice that in this example all the sumX SimpleMatlab tasks depend from the Splitter This means that the output from the Splitter will be fed to each sum task On the other hand the Collector depends on every sumX task It will be launched
67. ming API a command line based job submitter It is also recommended to use the graphical user or administration interface Eclipse RCP Plugin see Chapter 2 ProActive Scheduler Ecplise Plugin which can be plugged on the scheduler core application In the rest of this chapter we will expose how the scheduler works what policies govern the job management how to create a job and how to get the jobs and the nodes state using either the shell communicator or the GUI NOTE Additionally you can find here http proactive inria fr userfiles file tutorials ProActiveSchedulerTutorial pdf a fully documented example of the Scheduler and Resource Manager usage This tutorial does not require Java nor ProActive knowledge since it s is only based on graphical interface and command line actions 1 3 Scheduler Concept 1 3 1 What is a Job A Job is the entity to be submitted to the scheduler It is composed of one or more Tasks A Job can have one of the following types TASKSFLOW represents a Job that contains a bag of Tasks which can be executed in parallel or according to a dependency tree The Tasks inside this Job type can be either Java A task written in Java extending a given interface or Native Any native process PROACTIVE represents a Job that contains a ProActive application embedded in only one ProActive Task Its execution starts with a given predefined number of resources on which the user can start the ProActive appli
68. odvsseuesnsing oswewss deurdestessvoveesenenssatevintcuesgan Vllbst veros qug p sue bAPe Pe DET s tenue 36 3 4 Static Node Source and Dynamic Node Source sisi 37 3 25 Nodes States 58 user sun adectequlloatue eta bec dede de dot ose sevde open de reb E el pepe dibstadebtu o eren ete vendendi ne drt babes eue ovs 37 3 60 Starting the Resource Manager I or E E de TRE Vete EMI I ERRSER VENE e teen den R ER PERRA E o e eta deste 38 Chapter 4 Resource Manager s Eclipse Plugin ce eee eee eerte ee ette oe eese sese eese e aos ee eo see aue 39 Part II ProActive Resource Manager ProActive Chapter 3 ProActive Resource Manager Chapter 3 ProActive Resource Manager 3 1 IMPORTANT NOTE Some parts of the ProActive Scheduler and ProActive Resource Manager rely on Java Scripting capabilities JSR 223 http jcp org en jsr detail id 223 As a consequence it requires either a 1 6 or greater Java Runtime Environment without any modifications or with a 1 5 JRE the JSR 223 jar files http jcp org aboutJava communityprocess final jsr223 index html First the script api jar script js jar and js jar files must be added in the ProActive dist lib directory if you are using the bin release or ProActive or in the ProActive lib directory if you build ProActive from the source release Then the javab5 jsr223 patch jar patch released with the Scheduler RCP Client should be executed in the
69. pes than the Matlab s one A good way to start manipulating and scheduling Scilab scripts is to have a look at the standalone and simple Matlab Scilab GUI described in the ProActive documentation If you want to directly through more complex Scilab job scheduling go on with the following tutorial 6 2 Quick Start with the Scilab Extension We ll write a simple Scilab job example This example will assume that you are familiar with the example in Section 5 3 A More Complex Example a Matlab task flow This example will compute the numerical integration of sin x between 0 and PI 6 2 1 Installation Before starting to use the Scilab interface you need to install Scilab in your environment You ll find all the instructions on PROACTIVE scripts unix scilab README Scheduler Scilab section Once Scilab is installed you won t need at runtime to bother where Scilab is installed the Scheduler will determine it for you 6 2 2 The Scilab Job descriptor This is the descriptor of the scilab job which will be executed inside the scheduler 50 Part IV ProActive Scheduler s Scilab ProActive SY Chapter 6 ProActive Scheduler s Scilab extension lt xml version 1 0 encoding UTF 8 gt lt xml stylesheet type text xsl href _ src Extra org ow2 proactive scheduler common xml stylesheets variables xsl gt job xmins xsi http www w3 org 2001 XMLSchema instance xsi schemaLocation urn proactive jobdescriptor 0
70. ple Matlab version Architecture inside your grid infrastructure If you are using a ProActive installation on a centralized NFS folder this will be sufficient Otherwise you will have to build and install the native library inside your ProActive installation on each machine used The good news are if you successfully run the configuration script you won t have to bother where Matlab is installed at runtime the Scheduler will determine it for you A little drawback to this is that the scheduler will use the first Matlab installation found on the system so it might not do what you want when several Matlab installations are on the same machine Further releases of the extension will allow a finer control over that by specifying Matlab s minimum version requirement inside job descriptors 5 2 2 Writing a simple example the Matlab Script We write a very simple script which computes the roots of a single given polynomial out roots in The in and out variables are specific variables which describe the inputs and outputs of a Matlab script for the Scheduler in and out can contain anything supported by the extension Double Complex or String arrays Cells Records 5 2 3 Writing a simple example the Scheduler job descriptor This is a step by step guide to write this job descriptor 5 2 3 1 The job definition The job tag is the root tag of our descriptor it must have a name attribute which holds an id of the job It is generally followe
71. r own ProActive executable application public class ProActiveExample extends ProActiveExecutable private int numberToFind 5003 Override public Object execute ArrayList lt Node gt nodes System out printin ProActive job started create workers on local node Vector lt Worker gt workers new Vector lt Worker gt for Node node nodes try Worker w Worker PAActiveObject newActive Worker class getName new Object node workers add w catch ActiveObjectCreationException e e printStackTrace catch NodeException e e printStackTrace create controller Controller controller new Controller workers int result controller findNthPrimeNumber numberToFind System out printin last prime result return result As shown in a ProActive tutorial this example uses the given nodes with the ProActive API in order to start workers on them The execute nodes method shows what can be done inside this kind of task For more details about how to use the ProActive API see the appropriate documentation The complete example file can be found under jobs descriptors Job ProActive xml 1 4 4 4 Create and Add a ProActive task using XML descriptor Just take a look at the example below to understand the syntax of the ProActive task task id Controller gt description Will control the workers in order to find the prime number lt description gt
72. rcela Rivera Boutheina Bennour Ludovic Henrio Vincent Cave Fabrice Huet Guillaume Chazarain Virginie Contes Clement Mathieu Mario Leyton Eric Madelaine Paul Naoumenko Brian Amedro Viet Dong Doan Florin Bratu Fabien Viale Tomasz Dobek C dric Dalmasso Khan Muhammad Jean Luc Scheefer Julian Krzeminski Zhihui Dai Public questions comments or discussions can pe posted on the ProActive public mailing list proactive ow2 org The mailing list archive is placed at http www objectweb org wws arc proactive Bugs can be posted on the ProActive Jira bug tracking system https galpage exp inria fr 81 81 jira ProActive amp Table of Contents List of TOUTES sn nn drain AV ipaq qoc V Part L ProActive Scheduler 5 20 ot eas Saee epe eco serecpo even nao e eer eUceP aset sooo san UPS sue esae Chapter 1 ProActive Scheduler me JA IMPORTANT NOT E 2 eiie riter reinemhe ere Jae puna sya AETERNI OEEO EEE 2 IVO uS PEE c 2 1 3 Scheduler Concept M 2 13 1 Whatis JOD V 2 3 AA CAE LE MEET DE PR 2 1 3 3 Dependencies between Tasks ssssssssssssssssssssesee ee ee eee ene rene e emen nren nre EEEE Ee nEri 3 1 3 4 Scheduling Poley ss ue cers estere une eror eR ERE Y anne ERU antenne nette rest e NER EEAN KEEPER CR ERE SE sites 4 IE B
73. result rendrering in two different manners atextual manner by implementing public abstract String getTextualDescription TaskResult result This method similarily to String Object toString should return a String bject that describes the result a graphical manner by implementing public abstract JPanel getGraphicalDescription TaskResult result This method should return a Swing JPanel object that describes the result Some useful methods to create a specific preview class can be found in org objectweb proactive extensions scheduler common task util ResultPreviewTool such as automatic display of an image file or automatic translation between windows and unix path scripts optional The ProActive scheduler supports portable scripts execution through the JSR 223 Java Scripting capabilities scripts can be written in any language supported by the underlying Java Runtime Environment Scripts are used in the ProActive scheduler to Execute some simple pre and post processing optional pre script and post script Select among available resources the node that suitable for the execution optional selection script can be associated to a task Dynamic building of a command line for a native task optional generation script detailed in next section Here are some details and examples pre script The pre script is always executed on the node that has been selected by the resource manager before the execution of the task itself
74. ription gt Job description lt description gt lt variables gt lt variable name WORK_DIR value path to your working dir gt lt variable name SCRIPT_DIR value path to your script dir gt lt variables gt lt genericinformation gt info namez var1 value WORK _DIR gt info name var2 value val2 gt lt genericinformation gt lt taskFlow gt task idz task1 native retries 2 gt 13 Part I ProActive Scheduler ProActive e Chapter 1 ProActive Scheduler description Will display 10 dots every 1s lt description gt lt nativeExecutable gt lt Consider that the VAR_ NAME has been defined in the job description as describe in the job creation section gt lt static Command value WORK _DIR native exec gt lt arguments gt lt argument value 1 gt lt arguments gt lt staticCommand gt lt nativeExecutable gt lt task gt task id task2_native retries 2 gt lt description gt Will display 10 dots every 1s lt description gt lt nativeExecutable gt lt static Command value SCRIPT_DIR launcher sh gt lt arguments gt argument value WORK _DIR native exec lt argument value 1 gt lt arguments gt lt staticCommand gt lt nativeExecutable gt lt task gt lt taskFlow gt lt job gt It is obviously possible to mix Java and Native task inside a taskFlow Job Some other stuffs and options can be set onto a native task here s t
75. s descriptor Inside this tag each variable can be reused even in another following variable definition by using the syntax name of variable Part I ProActive Scheduler ProActive 4 Chapter 1 ProActive Scheduler genericInformations optional is a way to define some informations inside your job These informations could be read by the policy of the Scheduler It can be useful to modify the scheduling behavior Contact your administrator if you want an information to be interpreted by the policy To specialize your job to a taskFlow job go to Section 1 4 2 1 Create a TaskFlow job using XML descriptor To specialize your job to a ProActive job go to Section 1 4 3 1 Create a ProActive job using XML descriptor 1 4 1 2 Create a job using Java API To make a new instance of a TaskFlow job go to Section 1 4 2 2 Create a TaskFlow job using Java APT To make a new instance of a ProActive job go to Section 1 4 3 2 Create a ProActive job using Java APT Then just follow the example below in order to create your Job using the Java Scheduler API job has already been created under the job variable job setName job name job setProjectName project name job setPriority JobPriority NORMAL job setCancelOnError true job setLogFile path to a log file log job setDescription Job description job addGenericInformation var1 vali job addGenericInformation var2 val2 As shown sever
76. se R bates vr E TU RUE VEF ENEN 6 1 4 4 Create and Add a task to a Jobi o Ut rtr redes tante ed exe ecd dti d ex UE e bi ts 7 1 4 5 Submit a job to the ProActive Scheduler sise 22 1 4 6 Get gw 23 1 4 7 Register to ProActive Scheduler events sise 23 1 5 Administrator Manual oett eot nie nec etes ey viens enter esie s trentaine E vu essc p estare eost ESES 24 Chapter 2 ProActive Scheduler Ecplise Plugin e ee eeee ecce ee eese eere eese esee eoeeeeeeseeeseseeee 25 2 4 The Scheduler perspectives ss donnes Soregete eventos evo sua es deos esototietudeves fent eenrt gue doueswadnr EOE EE TEE es 25 2 2 Views composing The perspective vezi overs cdcversesccosesswece owes rent est se eco mane o sd epee Ve dese HE ERE AE do ed rue e lire hace eee 27 2 3 Connect to the started ProActive Scheduler seen 31 2 4 The Scheduler perspective buttons sise 32 24 1 The Jobs view buttons im User Mode oorr rrr torre ote tete reete ar hrneta od ete EEE eg ENE E S 32 2 4 2 The Jobs view buttons in Administrator Mode s eese es re enne ne se 33 Part I ProActive Scheduler ProActive 4 Chapter 1 ProActive Scheduler Chapter 1 ProActive Scheduler 1 1 IMPORTANT NOTE Some parts of the ProActive Scheduler and ProActive Resource Manager rely on Java Scripting capabilities JSR 223 http jcp org en jsr detail id 223 As a consequence it requires eit
77. t ordered in three lists A full list that contains every result or exception of every tasks A failed list that contains every result or exception returned by a task that failed Anda precious result list that contains every result or exception returned by the task marked precious This result will be given to you exactly like you returned it in your executable To know when a job that you have submitted has finished its execution you can subscribe to the scheduler to be notified of some events This will be explain in the next section get the user interface UserSchedulerlnterface scheduler auth logAsUser username password get the result of the job JobResult myResult scheduler getJobResult myJobld Nook at inside the JobResult to retrieve TaskResult 1 4 7 Register to ProActive Scheduler events If you are using the Java API it is possible to get events from the Scheduler In order to be notified about the scheduler activities you can add a Scheduler listener that will inform you of some events like job submitting job or task finished scheduling state changing etc To add a listener just make your listener by implementing the SchedulerEventListener interface and add it to the scheduler You will then receive the scheduler initial state containing some informations about the current scheduling state See the ProActive Scheduler JAV ADOC for more details 23 Part I ProActive Scheduler ProActiv
78. tRes results if tRes hadException System out printin i tRes getTaskld tRes getException getMessage else System out printin i tRes getTaskld tRes value Part I ProActive Scheduler ProActive e Chapter 1 ProActive Scheduler message UR Builder getLocalAddress toString Thread sleep 10000 catch Exception e message crashed e printStackTrace System out printin Terminaison de la tache return message i slept for 10 sec This executable will print an initial message then check if there are results from previous tasks and if so print the value of these parameters It will then return a message containing what the task did The return value will be store in the job result It is also possible to get a list of arguments that you can give to the executable at its start by overriding the init method on a Java executable How to give arguments to the task will be explain further We get back the foo bar and test arguments to illustrate the task creation example below private boolean foo private int bar private String test Override public void init Map String Object args foo Boolean args get foo bar Integer args get bar test args get tes To sum up create an executable is just extend the JavaExecutable abstract class and fill the execute method The given TaskResult resul
79. ts arguments permit to get the results from previous dependent tasks that have finished their execution As shown in the following lines the given array of TaskResults results will be an array of two results TaskResult 2 and 3 in this order if the dependences of Task 5 is Task 2 and Task 3 in this order Therefore you can use them to perform Task 5 process Override public Object execute TaskResult results throws Throwable TaskResult tResult2 results 0 TaskResult tResult3 results 1 Finally overriding the init method can be useful if you want to retrieve some parameters The task is the entity that will be scheduled by ProActive Scheduler As it has been explained in the Section 1 3 2 What is a Task it s possible to create and add Java tasks to your TaskFlow Job A Java task can also be created using an XML descriptor or the provided ProActive Scheduler Java API Part I ProActive Scheduler ProActive ae Chapter 1 ProActive Scheduler 1 4 4 1 2 Create and Add a Java task using XML descriptor Just take a look at the example below to understand the syntax of a task task id taski retries 2 gt lt description gt human description lt description gt javaExecutable class 0rg ow2 proactive scheduler examples WaitAndPrint gt lt parameters gt lt parameter name foo value true gt lt parameter name bar value 1 gt lt parameter name test value toto gt lt paramet
80. va home path env setJVMParameters d12 aTask setForkEnvironment env To have an exhaustive list of which options are available and what they are for just go to Section 1 4 4 6 Tasks options and explanations 1 4 4 2 Create and Add a native task Note It is only possible to add a native task in a TaskFlow Job To learn how to create a TaskFlow Job just go to Section 1 4 2 Create a TaskFlow job Once your TaskFlow Job created you can add as many native tasks as needed to perform an application A native task can be any native application such as programs scripts process etc 1 4 4 2 1 Create and Add a native task using XML descriptor Just take a look at the example below to understand the syntax of a native task lt This native task example shows a native executable directly started as a command gt task idz task1 native retries 2 gt 12 Part I ProActive Scheduler ProActive e Chapter 1 ProActive Scheduler description Will display 10 dots every 1s lt description gt lt nativeExecutable gt lt Consider that the VAR NAME has been defined in the job description as describe in the job creation section gt lt staticCommand value WORK_DIR native_exec gt lt arguments gt lt argument value 1 gt lt arguments gt lt staticCommand gt lt nativeExecutable gt lt task gt lt This native task example shows a native executable started by a shell script
81. wo examples of what can be done with task XML descriptors task id iaskName preciousResult true gt description Testing the pre and post scripts lt description gt selection script type static gt file path SCRIPT_DIR host_selection js gt arguments argument value ECLUSION STRING gt lt arguments gt lt file gt lt script gt lt selection gt lt pre gt lt script gt file path SCRIPT_DIR set js gt lt script gt lt pre gt lt nativeExecutable gt lt staticCommand value WORK_DIR native_exec gt lt arguments gt lt argument value 1 gt lt arguments gt lt staticCommand gt lt nativeExecutable gt 14 Part I ProActive Scheduler ProActive Chapter 1 ProActive Scheduler lt post gt lt script gt file path SCRIPT_DIR unset js gt lt script gt lt post gt lt task gt task idz Pl Computation walltime 00 10 gt lt genericinformation gt info name namel value val1 gt lt genericinformation gt lt nativeExecutable gt lt Consider that the VAR_NAME has been defined in the job description as describe in the job creation section gt lt staticCommand value WORK_DIR native_exec gt lt arguments gt argument value 1 gt lt arguments gt lt staticCommand gt lt nativeExecutable gt lt task gt To have an exhaustive list of which options are available and what they are suppose to do

Chapter 1. ProActive Scheduler

Contents

Download Pdf Manuals

Related Search

Related Contents