Grid Computing
Contents
1. Other configuration parameters location of certificates timeouts etc a 1666 UNIVERSITY Recall handling secure connections e arcproxy Client tool creates proxies Needs an extra file to keep addresses of Virtual Organisations e g voms vomses Needs your X509 keys p12 or pem files e All other client tools use proxies for secure communications arcsub triggers creation of a delegated proxy on the cluster Also uses proxies for authentication arccp arcstat and others use proxies for authentication Authorisation check is performed by the server side components e Other Grid client tools can use your proxy Ifitis in the default location and has a default name tmp x509up_ u lt UID gt Or if you define environment variable X509 USER_PROXY pointing to your proxy file ARC client tools can also find non default proxy location from the configuration file e All Grid tools need CA keys e g in etc grid security certificates 2014 Oxana Smirnova Dept of Physics 141 UNIVERSITY Hold on I can t remember all these defaults e No worry nobody can e There are three ways to define default locations Use default file names e g globus userkey pem Define environment variables in your logon scripts e g X509 USER KEY Use ARC client configuration file explained later in this lecture e All these three ways can be combined e There can be up to 11 such default locations for e g t2. Scheduling clients Client tools know everything Ca mny Certificate Application database NS S r ik _ Authorised users 3 gt L director SS JR y J Certificate Grid job a management service ace Data Policies Cany Certificate 1 Certificate Certificate Researcher Information services tools Grid client tools is your interface to the Grid e Mostly command line tools E Researcher Grid tools Certificate 135 UNIVERSITY 2014 Oxana Smirnova Dept of Physics One client tool or many e Historically most Grid tools are Command Line Interface because It is similar to batch system tools It is similar to many Unix Linux tools Unix Linux users like to write own scripts on top of generic tools There are too many parameters to make a useable graphical tool e Some graphical tools do exist focused on certain tasks Like e g the storage explorers e In general there are many client tools Each focussed on specific tasks Some use Grid libraries while others are wrapper scripts Some overlap in functionalities e We will focus on the set of ARC client tools 4 ga P a n 7 d 4 wa a 7 L nw m A A IYA pe a Y i lOY 2 IAA Y J T LSJINAIIA OAI IJ V Ci 7 N UNIVERSITY Grid workload management concepts e Alternative approach B e Original idea A s Set can submit jo
3. af Phueine IER UNIVERSITY f LJ T LJ ACaAILICGA O11 INJ V 1 5 bet Ly K fLs NJ L I y ollu ud J Exercises e Create your own client conf using the template from etc arc cp etc arc client conf arc client conf e Configure your grid security settings pathes for certificate files Create a new block common Use the keypath and certificatepath attributes to specify that your certificate files are on the USB key common certificatepath media your USB disk your dir your certfile keypath media your USB disk your dir your keyfile e Use arcproxy to generate VOMS proxy use the d VERBOSE option to check which defaults pathes are actually used arcproxy S nordugrid org d VERBOSE 2014 Oxana Smirnova Dept of Physics 157 UNIVERSITY Exercises Let s try to submit a job to the Grid and let the client find a suitable cluster for your job arcsub hello grid xrsl Most probably it will result in Job submission aborted e The entry point s to the Grid must be specified in client conf add a couple of information index blocks e g registry topindex1 url ldap indexl1 nordugrid org 2135 Mds Vo name NorduGrid o grid registryinterface org nordugrid ldapegiis default yes e Re run the arcsub hello grid xrsl and check which cluster was selected for your job fy f A A ZU 14 i NJ i i Use the d DEBUG to see the various steps the client is doing on the Grid T
4. bs Provided the relevant clusters are up and running e ARC keeps jobs list in a database arc jobs dat In older versions it is an XML file containing job IDs arc jobs xml e arcsync is the client tool that finds your jobs in the Grid information system and updates your jobs dat ond Garna mimoa Dente Physics 1AR UNIVERSITY L LJ LSJ NAIIA WITT IVJ V GO LAW N J I a WS y i y VJiuc T Checking job progress and getting results e arcstat shows job status taken from the information system Using A REX states as described in the 4 lecture e arccat prints out standard output error log of a job e arcget downloads the job results Only those files that you described in XRSL as Output files standard output standard error and Grid log Other files are removed when the job finishes Job results are stored in the folder named after job ID This can be changed in the configuration Once the results are downloaded the job is cleaned from the cluster Option k prevents clean up 2014 Oxana Smirnova Dept of Physics 147 UNIVERSITY Manipulating jobs e arckill triggers termination of job execution Cleans up the job unless option k is given e arcclean triggers clean up of a job Useful when you don t need to do arcget e arcrenew triggers a new delegated proxy generation Useful when the old proxy expires e arcresume triggers restart of the job from the stage where it failed Us
5. bs to any One central service to orchestrate the workload No single point of failure Queue on top of other queues gt Problems Problems Non optimal workload Limited scalability Rather complex clients Single point of failure Slow interaction with users 2014 Oxana Smirnova Dept of Physics 137 UNIVERSITY ARC architecture overview cones gt Registration Client gt Query Koto f Query and job actions Data transfer lt a 2014 Oxana Smirnova Dept of Physics 138 UNIVERSITY Tasks of Grid client tools Secu rity e Create proxy certificates Tavvelaaa ation e Discover Grid resources e Interpret job description and submit it to a matching resource Computing DEERE Copy files to from the Grid 2014 Oxana Smirnova Dept of Physics 139 UNIVERSITY What should client tools do for me e Handle secure connections proxies delegation e Matchmaking and brokering Find matching clusters actually queues for my job description and pick the best from the list of candidates e Keep the list of my jobs actually can submit jobs from different computers Aclient tool should discover them all and update my local list e Do job and data manipulations on my request Check status get outputs kill jobs move files etc e Do initial configuration Most importantly store starting points of the information system
6. he VO contact points file vomses The client tool e g arcproxy will try them all one by one e So check documentation and pick whichever way is best for you Some locations need superuser privileges UNIVERSITY Matchmaking and brokering arcsub tool User submits a job arcsub myjob xrsl e Even if option c is specified matchmaking and brokering still proceeds Client tool arcsub looks up the list of default clusters and information indices in the configuration file Matchmaking the tool then polls information system to discover all queues that e Match myjob xrs1l e Authorise the user Brokering from all matching queues the tool selects one e By default randomly e A user can request a specific ranking algorithm 2014 Oxana Smirnova Dept of Physics 143 UNIVERSITY Actual job submission e Once a target queue is selected the ARC client tool does the following signs the delegated proxy request using your own proxy Changes your XRSL job description to match the target Converts expressions like memory gt 10 to memory 20 Adds extra attributes like queue long It can even convert your job description to another language Uploads securely the following files to the target server Job description document Executable files if they are on your computer Other input files if they are on your computer Adds the submitted job ID to your local list database of
7. her client tools Use the arctest utility to easily launch simple test jobs and the other arc commands to manage jobs arctest J 1 arcstat lt jobID gt arccat lt jobID gt arcclean lt jobID gt Submit a longer job arctest J 1 is long enough and kill it when it starts running arctest J 1 arcstat lt jobID gt arckill lt jobID gt Submit a longer job and renew its proxy arctest J 1 arcrenew lt jobID gt p A fr A 1 f fN ZA nN A lt r a aA AA fr y A A AN BD P o Fa 4 AA 2014 Oxana omirnova De Dl OI Ph YSICS IOU
8. jobs e Actually you can simply upload files by hand it will trigger job submission anyway But the job is likely to fail in a wrong queue and without proxy leaving no trace G NIRMOS 2014 Gan eaim Deakali Civics 144 UNIVERSITY J i i LJIAAIIA OLITHIIIhUVG 5 et VL K Ea NJ i i y VDJihuc T T E Job submission in ARC summary Client tool must Query information Match it to the job description document Select the best site Convert to a server document deterministic Upload all the files A REX discovers uploaded job files and launches job processing Advance reservation is not possible Currently information and upload use different protocols arcsub https will be used in future for better consistency e All steps require authorisation LUND 2014 Oxana Smirnova Dept of Physics 145 UNIVERSITY Keeping the list of jobs arcsync e Keeping a list of jobs sent to the Grid is important You don t want to lose track of your jobs Useful reference when you want to operate on several jobs at once For example kill all jobs sent to a bad cluster e If you always use the same computer to send skill jobs it is easy to have the list on that computer arcsub will do it for you e f you use different computers jobs list needs to be synchronised But you cant synchronise with a notebook that is turned off Luckily Grid information system knows about the jo
9. ry to find bad clusters and exclude those from the submission attempts using the rejectdiscovery host name of the cluster in the common block of the configuration file Try to modify the registry blocks to use only a subset of the entire Grid e g specify only Swedish EGllSes Use the timeout parameter in the common block to speed up the job submission FSS aa Oxana Smirnova Dept of Physics 158 Exercises e Let s find all your jobs on the grid with arcsync pretend that you start with a clean system therefore remove your arc jobs dat file if it exists Start with a Known cluster arcsyne c arc iridium lunarc 1lu se Run arestat a to check the newly discovered jobs Extend the search for jobs synchronization for the entire Grid arcsync d VERBOSE e After synchronization use arcstat to list All your jobs on a cluster arestat c arc iridium lunarc lu se All the deleted jobs on the Grid arcstat s DELETED All the successfully completed jobs on the Grid arestat s FINISHED All the running jobs arestat s Running e Fetch the output of one of the FINISHED jobs with arcget edit client conf to change the default download directory jobdownloaddirectory in the common block Run areget lt jobid gt What happens if you run arcget against a not yet completed job or against a job you already i downloaded os O xana S Mm irn Ova Deo OT O f Ph VSICS 1590 U NIVERSITY Exercises e Try ot
10. s wrong e y prints out version number e h prints short help e t changes default timeout e z Changes default configuration file oO A 1 A C vana Q ry iIrtnn 1 9 D an A f D h IQCICOQC 11 u N IV E RS ITY VJ i LIACI iad ODII Vc A 5 bet Ly K JL UIA E y olL UU D ARC client configuration e ARC keeps relevant files in arc by default Command line options overwrite defaults as usual Default configuration file arc client conf Command line option z allows to specify some alternative configuration Template can be found in etc arc client conf e Most important is to configure the information system entry points Your client con must contain one or more blocks like this registry index2 url ldap index2 nordugrid org 2135 Mds Vo name NorduGrid o grid org nordugrid ldapegiis registryinterface default yes UNIVERSITY 2014 Oxana Smirnova Dept of Physics Configuration file details e Read ARC Client User Manual for a complete description htto www nordugrid org documents arc ul pdt e The file uses the INI format ANAA 2014 Plain text file Consists of blocks corresponding to different groups of configurable parameters Each block starts with a header in square brackets blockname Header names indicate hierarchy of blocks e g registry index1l registry index2 Each block defines a set of parameters as attribute value pairs Commented lines start
11. ually needed after arcrenew e arcresub resubmits the job a new Useful when you dont have the original XRSL or input files any more A 1A C yya na Q mM rnn 2 D An A f D h J2irnea 1 A Q u N IV E R S ITY LJ T LSJINAILICA OI il J VGO LJO K JE NJ E I y Wile hd Working with files e arcls prints out remote file or directory listing e arccp copies files to from Grid addresses e arcrm removes files and directories on the Grid Caution Can even remove a job Job ID is actually a valid URL e arcmkdir creates a new directory on the Grid e arcrename renames files and directories Not all protocols support it 2014 Oxana Smirnova Dept of Physics 149 UNIVERSITY Other ARC client commands e arcinfo prints out information about clusters and queues As taken from the information system e arctest submits some test jobs Very useful to check that everything works e Other commands found in the distribution arcslcs creates short living certificates using 3 party identity providers Not known to be used in practice arcmigrate moves jobs from one cluster to another Works only for a very specific interface Not known to be used in practice ONA fans eran Man ar Phuc IEN UNIVERSITY U T A ACI iad Dl i III VGAa WG K J P LJ i rIVoOlIUO J J Common command line options e d defines verbosity level of terminal output use d VERBOSE or d DEBUG when you cant understand what i
12. with Quotation marks are not allowed j by 4 1666 G Z 05 5118 Oxana Smirnova Dept of Physics 153 UNIVERSITY Configuration blocks e common used to define many common parameters such as e g timeout proxy location verbosity level etc e registry lt alias gt used to define information indices These are your entry points to the Grid e computing lt alias gt used to configure preferences for favourite computing clusters 2014 Oxana Smirnova Dept of Physics 154 UNIVERSITY Other configuration files e voms vomses Can be used by native VOMS clients too e arc srms conf useful when you use SRM protocol for data transfer ls created and populated automatically normally needs no user attention e Server has an own configuration file arc conf but we as users should never have to bother about it 2014 Oxana Smirnova Dept of Physics 155 UNIVERSITY Other client tools e Some 3 party ARC client tools exist ARC submission plug in of HI Condor uses basic file upload LUNARC s graphical tools use ARC libraries Simplify work with many jobs but dont implement all possible functions Several Web portals exist Some use ARC libraries Some call out to ARC command line tools Most are tailored for specific usage f by zZ lt T E U e9 ARNS Vi 7 z gt n 6 O OSAA gt 1666 gt Op 7 ONIA Ayana Smirnava Dant
Download Pdf Manuals
Related Search