Home

High Performance Computing Cluster Quick Reference User Guide

1. Stores the search path for library files that may be used by executed LD LIBRARY PATH applications SMANPATH Stores the search path for user manual pages describing different Utilities and tools SUSER Stores your current user name PWD Stores your current working directory 4 2 MODULES ENVIRONMENT SWITCHER The module environment switcher provides a convenient method of setting your environment variables for the different libraries installed on your cluster When invoked the utility modifies the running session setting the PATH LD LIBRARY PATH and MAN PATH variables as requested a DYNAMIC MODULE LOADING Modules are installed in a shared area e g opt gridware modules to make them available across the HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 13 of 40 cluster on all login and compute nodes All users logged into any cluster node can Use module commands with identical syntax To view the different module environments available on your cluster use the module avail command user loginOl module avail oe Oe anidar eee nos eS oo O aro a ool qero rape ck 3 11 blss 1 apps bedtools 2 16 2 g0c0c 14 4 6 appbs lat 9 gqgec 14 1 06 apps breakdencer L 3 5 1 gec 144 6 samtools 0 1 18 b008t 1 4540 apes Mars mp o Oc dd 0 Openmp a OL apps drs mpi 869 qee desd oO pen aoe apps hats
2. on them using the cluster A quick solution is to generate 1000 separate jobscripts and submit them all to the queue This is not efficient neither for you nor for the scheduler master node grid scheduler allows users to submit a single job with a number of separate tasks these are scheduled with a single job ID making it simple to track prioritize or cancel all the jobs with a number of separate tasks When the example jobscript below is submitted an array of 10 000 tasks will be generated and executed on the queue selected HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 33 of 40 bin bash Export our current environment V and current working directory cwd FS V cwd Tell SGE that this is an array Job with tasks to be numbered 1 to 10000 ve SEN When a single command in the array job is sent to a compute node 7 SeOked in tho variablo GE TASK ID sO We cal use tho value or that Variable to gert the re cults ve vant Its task number Ls programs program Se data Input SGE TASK TD oo results OME puto cl Taek ID The script can be submitted as normal with the qsub command and is displayed by grid scheduler as a single job with multiple parts user Glog e c culb Serial q arta Joel Wout JoD array 22 Ly array Jobn has beer cunmi ted userGlogin0l s g3tat Job ID prior na
3. IN TO THE CLUSTER VIA THE COMMAND LINE Users can log in to a cluster login node using secure shell SSH from your desktop system or another machine on the network To log in from a UNIX Linux or MacOS X system use the ssh command Password x xx x x username loginl username workstation ssh username loginl mycluster local To log in from a Microsoft Windows workstation users must download and run an SSH client application the open source PulTY package is a popular choice as shown below fee PuTTY Configu Specify the destination you want to connect to Host Name or IF address Port usemame login 1 mycluster local Connection type O Raw Telnet Rlogin SSH Serial Load save or delete a stored session Saved Sessions Delete HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 8 of 40 3 2 GRAPHICAL LOGIN ENVIRONMENT If your cluster login node s have been configured with graphical login service authorized users can obtain an interactive desktop session from a network connected workstation or laptop Your system administrator will let you know which services are configured and if your user account is enabled for graphical access a USING REMOTE DESKTOP CONNECTIONS When configured the XRDP service provides a quick and easy method to access your cluster system and run gra
4. Password REA The NIS password has been changed on headnodel cluster local Your user password can be reset by your system administrator if you have forgotten what it is a temporary password may be issued to you to allow you to log in use the yppasswd command to change the temporary password to one that you can remember b CHANGING PASSWORDS WITH LDAP For clusters that are configured to use site wide LDAP services access to login machines will be authorized using your standard LDAP password used on other site machines For clusters that incorporate a private LDAP server users can change their password using the passwd command user loginl S passwd Changing password for user test Enter Login LDAP passwords AREAS New UNIX password SERRA Retype new UNIX password Xxxx LDAP password information changed for test passwd all authentication tokens updated successfully msertlogsinl For clusters that are configured to authenticate to an existing LDAP service on your site contact your system administrator for assistance changing your LDAP password HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 12 of 40 4 CONFIGURING YOUR USER ENVIRONMENT 4 1 LINUX SHELL CUSTOMISATION As a cluster user you may need access to specific applications compilers libraries and utilities in order to run you
5. about the scheduler system also listing the status of each queue instance on every execution host available in your cluster Queues are listed with the following status Status code Queue state Description A queue instance has exceeded its pre configured maximum a alarm load q P J load threshold E configuration A queue instance has a configuration error contact your error system administrator for assistance A queue instance has been temporarily disabled by a system d disabled i aa s administrator The indicated queue instance has been de configured but jobs O orphaned l are still running using queve resources s suspended The queue instance has been suspended The scheduler has lost contact with the machine hosting the u unknown queue instance alarm l A The queue instance has exceeded its suspension threshold suspend The queue has been automatically suspended via the built in Calendar ES E C calendar facility Contact your system administrator for suspended information on the configured calendar policies for your site The queve has been automatically disabled via the built in D omen lendar facility Contact t dministrator f disabled calendar facility ontact your system administrator for information on the configured calendar policies for your site The scheduler was unable to contact the shepherd process on E Error the machine hosting this queue instance Contact your system administrator
6. for assistance This queue instance has been suspended via subordination to S Subordinated another queue 8 7 SUBMITTING A PARALLEL JOB Your cluster has also been configured with a parallel queue suitable for running MPI jobs across multiple nodes By default this queue has been configured with a parallel environment for the MPI environments available on your cluster To submit a parallel job via the cluster scheduler users can create a job script and submit it with the gsub command using the pe name lt slots gt grid scheduler directive HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 31 of 40 usertlogin01 cat jobscript sh bin bash FS V cwd pe mpi 4 mplirun np 8 benchmarkls bin user loginOl module load mpi intel openmpi 1 4 usert login 1 sub 2 7Jeobscriprs sh The mpirun command included in the jobscript above submits an OpenMP job with 8 processes the MPI machinefile is automatically generated by grid scheduler and passed to the MPI without needing further parameters The pe mpi 4 directive instructs the scheduler to submit the jobscript to the MPI parallel environment using 4 node slots using 2 processes per node There are 2 parallel environments set up on your cluster by default e mpi default MPI parallel environment e mpi verbose submits the job wrapped in job subm
7. queues configured on your cluster 8 3 GRID SCHEDULER PSEUDO ENVIRONMENT VARIABLES To assist users when writing job scripts grid scheduler automatically creates a number of environment variables that may be referenced in the script These are Variable name Description HOME The user s home directory on the execution machine SUSER The user ID of the job owner JOB_ID The current job ID JOB_NAME The current job name may be set via the N directive SHOSTNAME The name of the execution host STASK ID The array job task index number Other environment variables may be set manually by the user before submitting the job or by loading environment modules containing the variables required Users must remember to use the v directive to export their environment variables to submitted jobs if they have manually configured environment variables 8 4 JOB OUTPUT FILES By default grid scheduler will collect any information output to the stdouf or sfderr channels by your jobscript and write it to files named jobscript o and jobscript e where jobscript is the filename of the submitted jobscript The default location of these files is your home directory but this can be modified using the ewd o or e directives If the j y directive is also specified stdout and stderr streams are both written to a single file named jobscript o or to the location specified by the o directive 8 5 SUBMITTING NON INTERACT
8. s is suspended n never email for this job m beas M user host Specifies the email address to use to notify users of job status M_myuser NOtLEY Request that the scheduler sends warning signals SIGUSR1 and SIGUSR2 to running jobs prior to sending the actual SIGSTOP or SIGKILL messages at noOt1IEy HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 27 of 40 Directive Description qsub example termination time now yles n o Signifies that interactive jobs run using qsub qsh lem Cle cn SiOUlG ss se ecules a mun immediately or not at all MO y Allows users to set the name to be used to identify N name the job If this parameter is not specified the job is N BobVMDjob4 given the same name as the jobscript Indicates the path to be used for standard output o path Users Dob OUEDue streams p priority Allows a user to request that their jobs are run with lower than normal priority Valid priorities are 0 default to 1023 lowest priority p 1023 pe name nodes Requests the named parallel environment and number of nodes for the job to be run over Pe mpi verbose 8 q queue The scheduler queue to which the job should be submitted If omitted the scheduler automatically determines the correct queue to use based on
9. the job ype q serial g fee o yes Mm paremele causes le seneculer lo r yles inlo E Gos automatically re run the job if it fails during execution Specifies that jobs submitted with resource requirements can still be run even if the requested SoLtu resources are not available If this parameter is not soft specified the scheduler defaults to hard requirements Causes the qsub command to wait for the job to y mery les mod SS MIE complete before exiting Submits a job task array starting at task first and E first last t 1 L000 ended with task last Displays results of a dry run submission without actually verify verify submitting the job for execution V Exports all current environment variables to the job S 2 wd path Sets the working directory of the job wd users bob app2 grid scheduler directives may also be specified at job submission time as parameters to the qsub command for example uwsertlogin01l qsub J y V cwd my serial job sh Please note that grid scheduler automatically implements a walltime setting per job queue which controls the maximum amount of time a user job may be allowed to execute See your system administrator for HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 28 of 40 more information on the maximum time limits enforced for the
10. ALCES SOFTWARE L WALCES High Performance Computing Cluster Quick Reference User Guide Base Operating System Redhat TM CentOS Scientific Linux 6 3 with Alces HPC Software Stack HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 1 of 40 TABLE OF CONTENTS Vs CEN SLEW INCOR A TCD eat ea et il len o ee na ih 3 eI RODUC MON ad aaa enes 4 Ma O US pos REO PPP PEN A Re mauaaians 4 LaZr OES TING TOUR DAA INI IN CATION ica 4 ASEOS ERA Go E lt P ee Terre Cent oe NCTE eer te E CaN er eee ere ere teen ee ete reer eet ree ere te 4 2O LUSTER ARC HITEC MIR sass a 5 ZO REA TAN ME a AE O A SE ONE USE A GN dd NG article oline O SA o o Sacl ou eat aa rn aa atic 2A MARE DATA STORAGE IN GUANO 3 ACCE SINGO THE COMPUTE C LUSTER cererii ains 8 MaS ANEAN RE U TER VATRE C OAA PI INE eagat a iapaneemueieasers 8 e SRA PAIS AE EOIN ENVIRONMENT oa a iboi 9 a SING REMO TEDESCO CONNEC TION sustituta iii 9 MN AGO NES e OS 9 o AAN GUNS ADA OO N 12 a ERAN INAP WORDS tantas ias 12 ANS E SA O A eter mele maar T meme Nee nnn emery ameer eer Nene 12 4 CONFIGURING YOUR USER ENMIRONMEN Dic sista Qenasunie a iatesesusiuewends oeiat EEE yesaniee 13 A VEINS AMA AA e UnA IS A INN DUCE SERR ONAE E gt Ea A 13 ANAIS MODULE CODI C eruo eaa e ead hat A E E ias 13 oe EIR UR OU DEFRA YEF EIN IRORNME N es 19 EPLOADING MODUBES FROM SC HEDULER JO
11. BS CRIP US t pocscansccsansdocuitasiennstuiieduislisnel end E 10 ONLOAD ING VOD ULE PFRON SC RPTE ETE SS 10 EPMANUALEY CONFIGURING FOUR USER ENVIRONMENT creeerde AAEE 16 Y USING COMPILERS ANDEIBRARIE id 18 AAV AIEA BEE COMPILERS FOR SERIALZ BATEL ODO 18 OZ OAVAEABEE COMP IRE ROP OR FATALE OBO ta iaa 18 SAO AE ily A An PP 19 SA SOMPILER ENVIRONMENT VARA BLES FOR LIBRARIE O ci ia 20 MESSAGE PASSING ENVIRONMENT Ss las 21 OVE COR E PASSION ati 2 OAE T IRER ETONE G T taa dalla 2 VEER ETA E A TAO Z SIENA AO tediosa 22 Oreo LECTIN TAR Ve I UME VAIN TAT ON esnias nen a 22 OA EAEC IHN G A PARALELA AMO Ieier ahaa bcd lau alsin Se talancetaner dante 29 eNO SET SCHE OIE lo ee mtn 24 Pol SE CAEDUCER NID CIONES 24 AA VOF A O Doaa a a a n n a e 24 HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 2 of 40 8 OPEN GRID SCHEDULER CLUSTER SCHEDULER cm sasssssuesseeesesaenseneasannnacniauube veteesacsescnnnaenissemameonencons 26 Sl ING O ENGRASE ca 20 SAID CIS DIESE ct 20 6 0 Grs CAREDULESPOEUDIO ENVIRONMENT VARIABLES perniso Inesi Seas dad 29 AN A e e arae E 29 00 SUBMITTING INOINSINTER AC HYE JODO MAGO UD ito 29 OO VIEWING IHE STATUS OF A SUBMITED JO a sesepoigrarenisne ratiromunte rns EE EREE 30 wE SA A PP E e e E E T EET 31 0 SEIS ERA CUE DO A is 32 o OIE ASA IF O rie erences 33 SUSO DEPENDEN sae 34 Os DELETING Ac YBMTIED JOB aspirinas is
12. COPYING doc examples MANUAL scripts TUTORIAL bin lib 11b64 VERSION user loginOl 1 3 9 b SETTING UP YOUR DEFAULT ENVIRONMENT Users can request that certain modules are loaded every time they log into the cluster by using the module initadd and module initrm commands These commands manipulate a file held in the users home directory called modules causing the system to pre load the desired modules as the user logs in This is particularly useful for users who run a fixed set of applications and require the same environment to be configured each time they log into the cluster e Use the module initadd command to cause a module to be loaded on login e Use the module initprepend command to cause a module to be loaded on login before any other modules that are already being loaded on login e Use the module initlist command to list the modules that will be loaded on login e Use the module initrm command to remove a module listed to load on login e Use the module initclear command to clear all modules from being loaded on login HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 19 of 40 Your system administrator can also configure the system to automatically load one or more default modules for users logging in to the cluster Use the module list command to determine which modules are loaded wh
13. Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 24 of 40 the cluster for their jobs to run Scheduler systems provide a mechanism for collecting the information output by non interactive jobs making it available as files for users to query after the job has completed Non interactive jobs are usually submitted using a jobscripf which is used to direct the scheduler how to run the application The commands syntax used in the job script will depend on the type and version of scheduler installed on your HPC cluster Interactive jobs are commonly submitted by users who need to control their applications via a graphical or command line interface When submitted the scheduler attempts to execute an interactive job immediately if suitable resources are available if all nodes are busy users may choose to wait for resources to become free or to cancel their request and try again later As the inout and output data for interactive jobs is dynamically controlled by users via the application interface the scheduler system does not store output information on the shared filesystem unless specifically instructed by the application Interactive jobs only continue to run while the user is logged into the cluster they are terminated when a user ends the login session they were started from HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alce
14. IVE JOBS VIA QSUB To submit a batch or serial job use the qsub command Wser loginWih S cat my serial 060 5 bin bash echo I am a serial job sleep 10 isertlogin0l S qsub my serial jobwsh Your job will be held in the serial queue until a compute node is available to run it Some clusters have specific queues configured to run different types of jobs use the gstat g c command to view the queues configured on your cluster HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 29 of 40 serte login0l r gastat g E CLUSTER QUEUE CQLOAD USED RES AVAIL MOTA ekCDS cdsuE dev multiway q 0 06 0 0 32 48 16 0 dev parallel q G20 1 0 4 6 al 0 dev serial q 0 06 1 0 29 48 8 0 multiway q Jeta Sl 0 160 1000 760 8 parallel g Oat 86 0 20 125 16 6 serial g Gara aU 0 e 1000 T92 8 smp q sO 4 0 60 64 0 0 Use the q lt queuename gt directive to submit your job to a particular queue as directed by your local system administrator usertlogindl qsub g serial my sertal 30b sh 8 0 VIEWING THE STATUS OF A SUBMITTED JOB The grid scheduler scheduler allows users to view the status of the jobs they have submitted The qstat commana displays the status of all the jobs submitted by the user JTOB LD prior Se Sa 00 331 Us 22 000 Soe is 2000 234 0 12000 oor Ds ADO
15. OB NAME run job sh PE smp verbose QUEUE smp gqg MASTER smpl alces software com A machine file has been written to tmp sge machines 31 on empl alces software com If an output file was specified on job submission Job Output Follows Us Hello World Ly Hello world 7 Hello World 4 Hello World 2 Hello Wor kgl 2 Hello World 5 Hello World 6 Hello World HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 40 of 40
16. OpenMP API supports multi platform shared memory parallel programming in C C and Fortran OpenMP is a portable scalable model with a simple and flexible interface for developing parallel applications on platforms from the desktop to the supercomputer OpenMP jobs may be executed on standard compute nodes or used on dedicated SMP nodes often equipped with larger numbers of processing cores and more system memory The following example demonstrates how to write a basic OpenMP program and submit it to run on the cluster via the SGE scheduler Log in to your cluster as a user not the root account and make a new directory for your OpenMP test program u e rel agin mkdir openmp test iMseer loginGl e sed Cpenmp test Create a new file called openmp test hello c and enter the following C code to form your OpenMP program include lt omp h gt include lt stdio h gt int main int argc Char argv 4 ine ad pragma omp parallel private id 1d Oper thread mum printer sat Hello World n e mar teruern oO Compile your C code to make an executable binary called hello and use the 1dd command to confirm the shared libraries it will use on execution MeerGlogingl smpi qec fopenmp o hello helilowe usertlogin01 smo s ldd hello labgomp So l gt usr 11064 11bgo0mp s0 1 0x0000203c9ttFf 9000 libpthread so 0 gt 1ib64 libpthread so 0 0x0000003283e00000 Inbe S0 6 gt
17. Serials 1 8 0 qcc 1 1040 appe des serial L 8 gcc Lido apps 1ozone 3 108 9c0c0c1 4 6 apps memtester 4 3 0 3gcc 4 4 6 apps perl 5 16417 gec 4 4 6 apps picard 1 75 noarch appe 00 byonen2 222008403 oce 4 4 6 apps py Non 2 123 0ce 4 44 46 apbs semtools 0 1 18 qgee 1 4 6 apps stem IO ts tel oea apps trimgalore 0 2 5 noarch apps varscan 2 3 1 noarch compilers gcc 3 4 6 default Compilers gc Tolo compr lers qec 1 63 1103 at las 1007 gee 4 400 labs blas qe Lo Iibs ELE We vdoub lee cloaca Tepo TEENA double mpiy 2 1 57 geCC 4 4 07 Opennip LL Otal lies fitwe float 22i lt 5 gec 4 446 tibor triw Eloat MpT 2 Le geo tel 61 Opennpi Ol lies GSC E lo es aser t l a DS geo os labs lapack 3 4 1 7 gcc 434 46 ia bey numas Lo geo 4 2G On 21 ss aces gt a mon openne1y lao 1 q00 1 2 00 luser login0Ol 1 e Use the module load command to enable a new environment from the list displayed The load command may be used multiple times to include settings from multiple different profiles e Use the module list command fo list currently loaded modules e Use the module unload command to unload an already loaded module e Use the module purge command to clear all loaded modules The module command supports command line completion allowing users to type in the first part of the module name then use the TAB key to automatically fill in the remainder For example with
18. a binary instead of a script o Y mybinary bin nergejs enie se ecvle e execule e 99 Mom Me cwd cwd current working directory Allows grid scheduler to configure the remote X dis play display jmacts display for a graphical application Available for users allowed to submit deadline jobs to MEA ss indicate the final date and time there jobs can be run Scull 2 LLO doo Indicates the path to be used for standard error output e path e users bob output streams Specifies that jobs submitted with resource hard requirements must fully satisfy these requirements hard before they can run hold lt jobid gt that the submitted before Until all Defines job must wait executing jobs in the comma separated lt jobid gt list have completed O 232 pod lt file gt Specifies that the input stream to the job should be taken from the file lt file gt i uses Ol Mode 4 242 j yles n o Instructs the scheduler to merge the stdout and stderr streams from the job into a single file By default separate files will be created for each stream l resource value Specifies that the job requires a particular resource to be able to run The gconf sc command shows the configured resources available for selection exclusLyestrue m blelalsin Instructs the scheduler the send email to notify the user when the job b begins e ends a is aborted
19. age copying or moving files etc or calling a binary application to execute with parameters An example jobscript is shown below bin bash This is an example joo script grid scheduler directives are shown below Export my current environment variables ie V Merge stdout and stderr 5 j y echo Job starting at date test myapplication i data intputfile4 Lines preceded by a character are interoreted as comments and are ignored by both the shell and grid scheduler as the script is submitted and executed Lines preceded by the characters are interpreted as grid scheduler directives these options are parsed when the jobscript is submitted before the script itself is run on any system Directives can be used to control how grid scheduler schedules the job to be run on one or more compute nodes and how you can be notified of the status of your job The following common directives are supported by grid scheduler HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 26 of 40 Directive Description qsub example a LCC YY MMDDAAnm SS the date and time when a executed Defines job can be a 2011310213630 ar ano 10 Assigns the job to the advanced reservation with ID ar id a 412 b y es n o Allows a user to indicate that the command provided to qsub is
20. as submitted alternatively you can use the qstat command to retrieve the number of the job you want to delete HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 34 of 40 9 LUSTRE PARALLEL FILESYSTEM 9 1 LUSTRE BACKGROUND The Lustre parallel filesystem provides scalable high performance storage services to multiple client nodes via a range of interconnect technologies A file stored in a Lustre filesystem is available to all clients mounting the filesystem similar to filesystems shared with the NFS or CIFS file sharing protocols No special file handling is required to use the shared filesystem all POSIX compliant applications and utilities are compatible Files stored in a Lustre filesystem are separated into their metadata file names sizes locations permissions etc and a number of data blocks File metadata is stored by the Lustre metadata servers MDS and block data is stored by object storage servers OSS A single shared filesystem requires one MDS typically deployed as a pair of high availability servers and two or more OSS The usable size and performance of the resulting filesystem depends on the configuration of OSS machines supporting it The Lustre filesystem may be accessed via a range of network technologies including gigabit Ethernet 10 gigabit Ethernet and Infiniband the only requirement is that all servers a
21. ation available to all compute nodes users may add libraries into their environment by loading the module for the library set they wish to use See section 4 2 for more information about loading environment modules The following library packages are commonly available on HPC compute clusters Library name License type Library type Library location variable ATLAS Open source Auto tuned linear algebra software SATLASLIB BLAS Open source Basic linear algebra sub programs F77 implementation SBLASLIB CBLAS Open source Basic linear algebra sub programs C implementation SCBLASLIB FFTW2 double Open source Discrete Fourier transform double precision version 2 SFFTWDIR FFTW2 float Open source Discrete Fourier transform single precision version 2 SFETWDIR FFTWS rones eee Fourier transform latest oc uae 3 for Sauer single float double and long double precision LAPACK Open source Linear algebra package SLAPACKDIR Intel MKL Commercial Intel kernel math library suite SMKL ACML Commercial AMD core math library suite SACML Commercial libraries require a valid license for your site before users can compile and run applications using them If your cluster is properly licensed environment modules are provided to enable you to use the libraries listed above If your cluster is not licensed for commercial libraries or you have not loaded a compiler environment module the standard system librar
22. c Linux cluster installation lr shows basic examples of compiling and running compute jobs across the cluster and provides information about where you can go for more detail The primary audience for this guide is an HPC cluster user All software is configured to be as close to the requirements of the original community supported distribution as is possible more detailed documentation on specific packages are simple to find by following the links provided in this document 1 1 SYSTEM INFORMATION In order to access the cluster your system administrator will provide you with the following e The IP address and network information for the cluster e A username and password to allow you to log in to the cluster The software installed on your server nodes is designed to accept and run jobs from particular systems these systems are called the login nodes In a cluster configuration this may be the headnode system s or dedicated login nodes may be provided lt is not normally necessary to log in directly to compute or worker nodes and your system administrator may prohibit direct access to non interactive systems as part of your site information security policy 1 2 PROTECTING YOUR DATA AND INFORMATION The software installed on your HPC cluster is designed to assist multiple users to efficiency share a large pool of compute servers ensuring that resources are fairly available to everyone within certain configured parameters Security b
23. ccessing the filesystem have network access fo all the MDS and OSS machines Lustre also supports a gateway service which can be used to extend the filesystem to remote clients across a LAN or WAN link The gateway can also be used to allow filesystem access for remote Ethernet connected clients irrespective of the local Lustre interconnect type 9 2 QUERYING FILESYSTEM SPACE The lfs df command is used to determine available disk space on a Lustre filesystem If displays the amount of available disk space on the mounted Lustre filesystem and shows space consumption per OST If multiple Lustre filesystems are mounted a path may be specified but is not required Supported options includes h Human readable print sizes in an easily readable format i inodes Lists inodes instead of block usage The df i and lfs df i commands show the minimum number of inodes that can be created in the filesystem Depending on the configuration it may be possible to create more inodes than initially reported by df i Later df i operations will show the current estimated free inode count If the underlying filesystem has fewer free blocks than inodes then the total inode count for the filesystem reports only as many inodes as there are free blocks This is done because Lustre may need to store an external attribute for each new inode and it is better to report a free inode count that is the guaranteed minimum number of i
24. describe the connection and the hostname or IP address details of your cluster login HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 9 of 40 or master node machine Set the slider to match the network connection speed between your client machine and the HPC cluster NX Connection Wizard leales ES Insert name of the session Your configuration settings will be saved NOMAGCHINE gasson gt HPC Ousterlog nnode Insert server s name and port where you want to connect login cluster local Click the Next button and select a UNIX GNOME desktop the standard for Linux RedHat CentOS and Scientific Linux systems Configure the size of your desktop by selecting one of the pre defined options or select custom and enter the desired width and height in pixels A Using NX Client you can run RDP VNC and X desktops depending on what the service provider has made available MOMACHIME GNOME Settings Select size of your remote desktop Custom W 1280 H 1024 5 is established To enhance performance you can disable the encryption of the data traffic Disable encryption of all traffic Click the Next button and click on the Show Advanced Settings check box click Next again The advanced settings dialogue box allows users to enter the connection key used to connect to the NA server ru
25. dministration overhead Typically compute nodes have a minimal operating system installation to minimize unwanted services and helo reduce jitter across the cluster The configuration of the node operating system is controlled by the cluster provisioning software The operating systems on special purpose nodes are configured specifically for their purpose in the cluster Machines are categorized by the roles they perform for example Head node system provides cluster management and scheduler services interactive system provides software development environment Compute node system used solely for executing compute jobs Storage system provides data services to other nodes in the cluster Gateway system provide a secure pathway into the cluster Depending on your cluster size and configuration some physical servers may perform several different roles simultaneously for example the headnode of a small cluster may provide cluster management scheduler servers an interactive development environment and storage services to the compute nodes in your cluster All system administration is usually performed from the cluster headnode or administration machine HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 9 of 40 the goal of the installed cluster management utilities is to reduce the requirement to log in to each compute node in turn whe
26. en you log in c LOADING MODULES FROM SCHEDULER JOBSCRIPTS When submitting jobs through the scheduler system users may wish to use module commands to configure their environment variable settings before running jobs Three methods can be used to use environment variables in job submission scripts e Permanently add the required modules to be loaded for your login environment e Load modules before submitting jobs and use the V scheduler directive to export these variables to the running job e Use a module load command in your jobscript to load additional modules Users incorporating a module load command in their job script should remember to source the system profile environment e g etc profile in their jobscripts before loading modules to allow the environment to be properly configured e g bin bash This is an example job script Scheduler directives are shown below ho Y O myourouriitle Source the etc profile file to enable modules lete prolile Load modules modula load mp1 gce openmpi 1 5 2 echo Job starting at date test myapplication i data intputfile4 d LOADING MODULES FROM SCRIPT INTERPRETERS Environment modules may also be included from scripting languages including Python R and Perl allowing convenient inclusion of additional libraries or modules for these languages The following example shows an application module being loaded by a perl script usr b
27. ent interconnect fabric Use the module load command to load the appropriate environment module for the MPI required user loginOl S module avail a lal TOOL GY Ao O Ee a E Compilers ante eS IMOs IMtely rte 2 2 7 hetws compilers impel 111 072 04 Mp qee mpren2 1 3 7 mp veh iis dec atlas Ss 3 92 atlas mpl gec mvapien l 2rc1 mvapich 1158 9900 plas mol goe mvapaeh2 1 5 loL mvapich2 lbs qgee eblas 1 cbles mpi gcc openmpi 1 5 openmpi ins ges trw2 2 1 o frtw doub le mpel 1intel mpicn2 1 3 mprehZ libs gcc frCw2 2 1 3 frtw floart upi vel maple i rel myo pich libs qgcec fTEw 2 272 TTEW m9l lintel mv apienz2 1 gt 1p1 mvapieb2 Iibs gcc Lapack 2 3 07 lapack Moly intel openmpa 5 openmp Ms iare een 2 e oyi rora double Core 200 Sade raul bs Luce roba 2 Ll S Pitt dat wser lbogindl S module load mpi gce mpichZ usertlogin0L 16 If users are developing source code that integrates multiple compiler MPI and HPC libraries they must remember to load all the modules required for the compilation e g user login0l s module load compilters 1intel libs intel ifitw mpl antel mprenz HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 22 of 40 Users may prefer to include commonly used environment modules into their default environment to ensure that they are loaded every time they login See sect
28. ent module the open source GNU compilers appropriate to your Linux distribution will be available for you to use 5 2 AVAILABLE COMPILERS FOR PARALLEL JOBS When compiling source code for parallel jobs users may prefer fo use a compiler which is capable of creating binaries compatible with a message passing interface MPI The following compiler commands are automatically available after loading an MPI environment module e g OpenMPI MvAPICR Language C Cz Fortran77 Fortran90 Fortran95 Command mpicc mpicCcc mpif77 mpif77 mpiCC HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 18 of 40 9 9 ACCELERATED HPC LIBRARIES To assist developers of high performance applications and utilities a number of optimised libraries are available to provide commonly used functions including Fourier transforms linear algebra sparse solvers vector mathematics and other routines Often these libraries are tuned to take advantage of the latest CPU instruction sets to accelerate their performance on modern computer systems a well tuned library set can improve the performance of applications run across the cluster by Up to 30 Your HPC cluster has been installed with a set of high performance libraries that are appropriate for the hardware and software components that make up the system Each library is installed in a shared loc
29. etween users and groups is strictly maintained providing you with mechanisms to control which data is shared with collaborators or kept private When using the HPC cluster you have a responsibility fo adhere to the information security policy for your site which outlines acceptable behaviour and provides guidelines designed to permit maximum flexibility for users while maintaining a good level of service for all Your system administrator can advise you of any specific rules covering the HPC system but users are encouraged to e Change your system password as regularly as possible e Only access machines that you have a valid requirement to use e Adhere to your site information security policy e Make backup copies of important or valuable data e Remove temporary data after use 1 3 USING THIS GUIDE This quick start user manual is intended to provide users with basic information about their HPC cluster Your local cluster may be configured differently by your system administrator to suit the available hardware if in doubt contact your local system administrator for further assistance HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 4 of 40 2 CLUSTER ARCHITECTURE 2 1 LINUX OPERATING SYSTEM Your HPC cluster is installed with a 64 bit Linux operating system CentOS and Scientific Linux SL are freely available Linux distributions based
30. g with the qstat command and read the output file to view the results of the job execution user login 0 mow Estilo qsub sub lt sh Tour JO Ao test les been SUS e tea usertliogun0lL mp esto datat f queuename gtyre res ised rot load avg erch states parallel q node00 alces softwa IP 0711 0 06 1x24 amd64 lo 00 mpi ten E user ie O27 0272000 12 18 49 1 parallel q node0l alces softwa IP Oya al 0 04 1x24 amd64 E 0 5 S500 mol eset user Y 02 02 2009 12712749 1 serial q node00 alces software BI 070 8 0 06 1x24 amd64 S serial q node01 alces software BI 0 0 8 0 04 1x24 amd64 S fuser logue mp esse ses OUTEUT SGE job submitted on Wed May 27 04 14 15 BST 2009 2 hosts used JOB ID 43 JOB NAME mpl test PE mpi QUEUE parallel g MASTER node00 alces software org Nodes used node00 nodeQl Hello from node0042 8 on Wed May 27 04 14 30 2009 Hello from nodeQ0 0 8 on Wed May 27 04 14 30 2009 Hello from node0Q0 4 8 on Wed May 27 04 14 30 2009 Hello from nodeQ0 6 8 on Wed May 27 04 14 30 2009 Hello from node01 1 8 on Wed May 27 04 14 37 2009 Hello from node01 3 8 on Wed May 27 04 14 37 2009 Hello from node01 5 8 on Wed May 27 04 14 37 2009 Hello from node01 7 8 on Wed May 27 04 14 37 2009 HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 38 of 40 APPENDIX B EXAMPLE OPENMP JOB COMPILATION The
31. have one or more dedicated gigabit Ethernet networks for MPI traffic Clusters that process large amounts of parallel workload may be installed with a high bandwidth low latency interconnect such as Infiniband Myrinet or 10 gigabit Ethernet These specialist networks are especially designed to provide the fastest message passing systems available at bandwidths of multiple gigabytes per second Your system administrator will provide you with information on the different MPI networks installed on your cluster and can offer advice on how to get the best performance for your applications a ETHERNET NETWORKS The majority of HPC clusters incorporate one or more gigabit Ethernet networks into their design Popular for their low cost of ownership relatively high performance of 1Gbps per link and flexible CATOE cabling gigabit Ethernet networks are now commonly being deployed across large campuses and to users desktops Your cluster may have a dedicated MPI gigabit Ethernet network or might share a network with other tasks 10 gigabit Ethernet links are also becoming increasingly popular and are often used simply as a higher bandwidth version of gigabit Ethernet Suitable MPI implementations for Ethernet networks include e OpenMPI developed by merging the popular FI MPI LA MPI and LAM MPI implementations e MPICH an implementation of the MPI 1 1 standard e MPICH2 an implementation of the MPI 2 0 standard e Intel MPI a commercial MPI implementati
32. ies appropriate to your Linux distribution will be available for you to Use Some high performance libraries require pre compilation or tuning before they can be linked into your applications by the compiler you have chosen When several different compilers are available on your cluster libraries may have been prepared using multiple different compilers to provide greater flexibility for users who include them from their source code The modules environment may list some high performance libraries multiple times one for each compiler they have been prepared on Users should load the library environment module that matches the compiler they intend to use to compile their applications HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 19 of 40 5 4 COMPILER ENVIRONMENT VARIABLES FOR LIBRARIES When a module containing accelerated libraries is loaded the system automatically updates relevant system environment variables to assist users in writing portable code and makefiles which can be used with multiple different libraries The CPPFLAGS and LDFLAGS variables are dynamically updated to include the relevant include directives for compilers as library modules are loaded e g user loginOl module load libs gcc fftw3 3 2 2 user loginOl echo CPPFLAGS Opt gridare libs gcc trims 3 4 2 71 nelude user loginOl echo SLDFLAGS u o
33. in env perl SEN a beee MODULE O OUIETA true eval modulecmd perl load apps lifescope system lifescope sh e MANUALLY CONFIGURING YOUR USER ENVIRONMENT The modules environment switcher is installed for the convenience of users to assist in managing multiple different compilers libraries and software development environments If you prefer your user environment HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 16 of 40 can be set up manually bypassing the modules system for ultimate flexibility when configuring how you use the cluster system This can be achieved by manually setting your environment variables specifying the individual libraries to build against when compiling code or storing your desired libraries in your home directories and addressing them directly Although these methods are quite valid for advanced users please be aware that bypassing the global modules configuration may reduce the ability of your system administrator to assist you if your applications do not later execute as expected HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 17 of 40 9 USING COMPILERS AND LIBRARIES 9 1 AVAILABLE COMPILERS FOR SERIAL BATCH JOBS As well as pre compiled binary applications many HPC users also write their ow
34. invoked usually your local graphical terminal Although graphical applications are typically executed from an interactive graphical desktop session via Remote Desktop or NX advanced users can direct the graphical display to a workstation outside the cluster by HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 32 of 40 setting the DISPLAY variable before running qrsh The example below shows how the qsh command launches an xterm session on an available compute node When the graphical glxgears application is started the output window is automatically displayed in the originating Remote Desktop graphical desktop window ee Connectic te ystem Qs ia alceshencnode A o Compt File Edit View Terminal Tabs Help alces headnode qsh Your job 26 INTERACTIVE has been submitted waiting for interactive job to be scheduled Your interactive job 26 has been successfully scheduled alces headnode alces s H O alces node00 on node00 alces node00 glxgears 8 E raiceseneaanoae E aicesenodeoo ton nod aceseneacnose 1 O gugears O IE 8 9 SUBMITTING AN ARRAY OF JOBS A common problem is that you have a large number of jobs to run and they are largely identical in terms of the command to run For example you may have 1000 data sets and you want to run a single program
35. ion 4 2 for more information about loading environment modules Appendix A shows an example compilation of a parallel application and its execution using OpenMPl and the Open Grid Scheduler job management software 6 4 EXECUTING A PARALLEL APPLICATION After compiling your parallel application it can be submitted for execution via the cluster job scheduler system Most job schedulers use jobscripts to instruct the scheduler how to run your application see section for more information on how to create jobscripts Your cluster scheduler automatically determines which compute nodes are available fo run your parallel application taking into account the topology of the requested interconnect network and any existing jobs running on the cluster The cluster scheduler communicates this information to the selected MPI implementation via a machinefile which lists the hostnames of compute nodes to be used for the parallel execution In order to control how many compute cores to use on each multi CPU compute node users must instruct the scheduler to allocate the correct number of cores to the job Section 7 provides examples of the different options which can be used to pass this information to the cluster scheduler HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 23 of 40 7 CLUSTER SCHEDULERS 7 1 CLUSTER SCHEDULER INTRODUCTION Your HPC cluster i
36. is 34 9 LUSTRE Me BEE HLES OEM rodas 35 AO lio OA B seer eee o nen te ee ee eee 35 AO NS a AS 35 APPENDIA A EXAMPLE PARALLEL JOB EXECUTION sicario a E EE A 37 APPENDIX B EXAMPLE OPENMP JOB COMPILA TlON ooocccccccconnnnnnconcccnnnnncnnnnnnnnconcccnnnnnnnonnnnnncrnnnninnnnnos 39 0 1 LICENSING INFORMATION This work is Copyright 2007 2013 Alces Software Ltd All Rights Reserved Unauthorized re distribution is prohibited The Alces Software HPC Cluster Toolkit is free software you can redistribute it and or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation either version 3 of the License or at your option any later version These packages are distributed in the hope that they will be useful but WITHOUT ANY WARRANTY without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE See the GNU Affero General Public License for more details http www gnu org licenses A copy of the GNU Affero General Public License is distributed along with this product For more information on Alces Software please visit http www alces software com HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 3 of 40 1 INTRODUCTION This quick reference guide is intended to provide a base reference point for users of an Alces Software configured RedHat Centos or Scientifi
37. ission information such as date submitted and queue information 8 8 SUBMITTING INTERACTIVE JOBS grid scheduler also allows interactive jobs to be scheduled to run on the compute nodes of your cluster This method can be used to allow multiple users to run command line and graphical applications across cluster compute nodes fairly sharing resources between other interactive applications and batch jobs The qrsh lt binary gt command can be used to schedule and launch an application or when invoked without an application name to launch an interactive shell session on an available compute node The grsh session is scheduled on the next available compute node on the default interactive queue or a specific queue specified with the q lt queuename gt parameter user loginOl S uptime 03 99 23 Up oz days 19 users load average 2 32 5 437 2 50 usere logund 6 qrni user nodel2 S uptime USS 59525 Ao ao mS 1 user load average 0 00 0 00 0 00 user nodel2 The qrsh V xterm command can be used to schedule and launch an interactive xterm session on a compute node The grsh session is scheduled on the next available compute node on the default interactive queue or a specific queue specified with the q lt queuename gt parameter The graphical display for your application or xterm session will be displayed on the system identified by the DISPLAY environment variable setting when qrsh was
38. libedy libesso 6 Ux0000003 2383700000 Ibert son gt 711564 li brt so l 0200000032247200000 Mibod Tda linux x90 014 50 2 0x0000003282e00000 Next create a new file called openmp_test run job sh and edit it to contain the following lines to form a grid scheduler jobscript 1 SCE SUBMISSION SCRIPT Submit to the smp verbose parallel environment by default so we can see output messages Use 8 slots in the parallel environment corresponding to the number of threads to use one slot one thread one CPU core it fe 1 cmd pe Smp verbose 3 2 Y 0 Outs JOR 1D a7 hello HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 39 of 40 Submit a new job to the cluster scheduler that will execute your binary using the grid scheduler jobscript Confirm the job is running with the qstat command and read the output file to view the results of the job execution usere tocante Mei vessels suo tum Obs ll Your job 11 hello has been submitted usera logino l IL smpi o gastat f q smp g queuename atype resyv used tot load avg arch smp q smpl aalces software com IP 0 8 32 0 00 1x24 amd64 31 0255500 run job sh alces r 1270172009720 10 27 8 smp g smp2 alces software com IP 07 07 32 0 00 1x24 amd64 tseerGdloginUl smp cat outl SGE Job submitted on Tue Sep 8 10 10 19 GMT 2009 JOB ML J
39. me user state submit start at queue slots Ja task ID 32 Oro 00 array Job Users qw O37 2772010 12 08 02 1 I 10000 1 usert logino l gt 8 10 JOB DEPENDENCIES Users submitting jobs can indicate that grid scheduler should wait before starting their job until a previous job or list of jobs has been successfully completed If the proceeding job is an array job grid scheduler will wait until all array tasks have been completed The following example may be used to submit a new job called my_job sh that will run when the job with ID number 448 has been completed User leginOl ele gsub q serial q hold Ate my Job sh Your Job 212 my Ossa Das Deen gumi Cend user login0l gstat JoOp ID Prior name user state submit start at queue slots ja task ID O eee ole Wiser 4s qw 037 2972010 TO Tea 1 user login0Ol 8 11 DELETING A SUBMITTED JOB Users can delete their own jobs from the scheduler system using the qdel command Jobs which have not yet been scheduled to run will be removed from the queuing system without ever running Jobs which are already running or have partially run and been suspended will be sent SIGSTOP and SIGTERM signals to stop them executing Users may specify one or more jobs on the command line separated by spaces e g usertlogindOl TS del 374 325 user has deleted job 324 user has deleted job 325 user login0Ol The job ID was printed to your screen when the job w
40. n configuring and managing the system Non administrative cluster users typically access the cluster by submitting jobs to the scheduler system this may be performed remotely by using a scheduler submission client installed on their local workstation or by submitting jobs from the cluster login or interactive nodes lt is not normally necessary for users to log in directly to the compute nodes many cluster administrators disallow this to encourage the fair sharing of resources via the cluster scheduler system Your cluster may be configured with multiple different networks dedicated to different purposes These may include e Interactive user data and job scheduler information e Operating system provisioning e Lights out management IPMI or SNMP traffic e Storage networking e Message passing interconnect traffic e Graphics rendering visualization interconnect Depending on the size of the cluster some of these functions may be shared on a single network or multiple networks may be provided for the same function When using your cluster for the first time you should consider which network to use for message passing functions as these operations are sensitive to message latency and network bandwidth 2 3 CLUSTER SERVERS Your cluster is configured with a number of different server systems that perform different functions in the cluster Typical systems may include e Cluster login nodes e g o loginl university ac uk logi
41. n software to solve complex mathematical and scientific problems Users are encouraged to compile new applications on the login node systems testing them for correct behaviour before submitting them as jobs to run across the compute cluster Your compute cluster may be configured with a dedicated pool of resources for development and testing which is distinct from the main cluster A number of different open source and commercial Linux compilers are available providing different optimisations tuning options and feature sets The default open source GNU compiler suite is provided as part of your Linux distribution and is available by default on all compute clusters The following table summarises the compiler commands available on HPC clusters Language Package License type C Fortran77 Fortran90 Fortran95 GNU Open source ejeje g gfortran gfortran gfortran Open 4 Open source opencc openCC n a openf90 openf95 Intel Commercial Lee Ce ONT TOCE TEOL Portland Group Commercial pace pace pa pgf90 pgf95 Pathscale Commercial parhes Parece n a pathf90 joene lane SS Commercial compiler software requires a valid license for your site before users can compile and run applications If your cluster is properly licensed environment modules are provided to enable you to use the compiler commands listed above If your cluster is not licensed for commercial compilers or you have not loaded a compiler environm
42. n the dialogue boxes provided and press the Login button Connection messages will be displayed while your client negotiates with the NX server before your remote desktop session is displayed NOMACHINE ad login mycluster domain Login as a quest user Contact your system administrator for assistance if your remote desktop session is not started as expected they may ask for the diagnostic messages displayed during negotiation to assist in troubleshooting the connection problem HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 11 of 40 3 9 CHANGING YOUR PASSWORD Your system administrator will have provided you with a login password or SSH key access to the compute cluster allowing you to login and use the attached resources We recommend that you change your system password as often as possible and adhere to the recommendations explained in your site information security policy a CHANGING PASSWORDS WITH NIS If your cluster uses Network Information Services NIS you can change your user password using the yppasswd command on the command line when logged in to the cluster user loginOl S yppasswd Changing NIS account information for user on headnodel cluster local Please enter old password Changing NIS password for user on headnodel cluster local Please enter new password x x Please Eetveos Mew
43. nl2 business org gateway corp com e Compute node servers e o o comp00 comp23 standard compute nodes o himem00 himem03 high memory compute nodes Cluster service nodes e g o headnodel primary headnode system Q headnode2 secondary headnode system O mds1 Lustre filesystem primary metadata server O nfs1 Network file system primary server o ossl oss2 Lustre object storage servers As a cluster user you will mostly be running jobs on compute node servers via the cluster job scheduler HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page of 40 often also called cluster load balancing software or the batch system Each compute node is equipped with high performance multi core processors tuned memory systems and often have a small amount of local temporary scratch space assigned on their hard disks In contrast cluster service nodes often feature specialised hardware and slower low power processors to enable them to perform their intended purpose efficiently Your compute jobs will execute fastest on compute nodes and should not be started on cluster service nodes unless directed by your local administrator Each compute node is installed with a minimal operating system that allows it to boot and function as a Linux host to run compute jobs All application software compilers libraries and user data is held cen
44. nning on your HPC cluster This key is specific to your cluster and ensures that only clients that HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 10 of 40 identify themselves with the correct key are allowed to connect Your system administrator will provide you with a copy of the keyfile stored on your HPC cluster login or master node s Click on the Key button on the configuration dialogue and enter the key provided by your system administrator Press the Save button on the key entry box then press Save again on the main configuration box to save your changes im NX HPC Cluster login node NOMAGHINE im Key management he BEGIN DSA PRIVATE KEY allBwadjWukth2 bkawdku23 7 Ibdkasd2bcjklaawek Sy 3ujF CSlS bEW LdaWaglYAlcFlsMClkmWijeS y Tmeoyp iL THI aBsBGilsop9NiD imEpAgG nHHrhv T Az pUvirsrAcdMyME nx FdHZADWSKAV I r Qat EkqLAoGBAl4L1 TOG Fkq Tanl 5 2PeELdg OKO Jeo L381 NL Sulc4PoNy TRU TSRASAZTTs ex GNOME Settings bk 2YhRqoQTtXPFsl4quzBWet42WonWayyb 1bPBHk Jm gt ae Gkz3Ha TA0GANIgcCuA4wrC 3Cic9CFcqiwO En IvkBdve QU7TGVLk2Cz7Y4dasriad 1f6Fs TE8DHTNGreELPKRuLstGr mbX Sraid wi stit TnX 8541Kv29HJguPvOK 5 7cjz4 5qNFD MODEM ISDN ADSL WAN LAN SxAPsSKs6yZ61FNidfu END DSA PRIVATE KEY Display Custom r W 1280 H 1024 Settings Enter your cluster username and password i
45. nodes that can be created HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 39 of 40 HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 36 of 40 APPENDIX A EXAMPLE PARALLEL JOB EXECUTION The following example demonstrates how to write a basic MPI program and submit if fo run on the cluster via the grid scheduler scheduler Log on to your cluster as a user not the root account and make a new directory for your MPI test program useri loging a medie Mp test Miser logimOl a cas mol Test Create a new file called mpi_test test c and enter the following C code to form your MPI program include lt stdio h gt include lt mpi h gt include lt time h gt include lt string h gt ine main int argo Char arar char name MPI MAX PROCESSOR NAME IE Mprocs procno len MPL Inii sarge Gargv MPL Comm size MPL COMM WORLD proce y MPT Comm rank MPT COMM WORLD procno MPT Get processor name mame len name Men 0s tame Es de lt time NULL printt Rello 11 from SseSa sd on sn name procno aprocs erime clt MPI Barrier MPI COMM WORLD MET panal eo return 03 The source code for this short program is stored in the opt alces examples directory on you
46. on RedHat Enterprise Linux Both projects attempt to maintain 100 compatibility with the corresponding Enterprise distrioution by recompiling packages and updates from source as they are released by RedHat These open source distributions are supported by a thriving community of users and developers with many support sites wikis and forums to assist adopters with its configuration Redhat distribution version numbers have two parts a major number corresponding to the Enterprise Linux version and minor number which corresponds to the Enterprise update set Redhat 4 6 therefore corresponds to RedHat Enterprise Linux 4 update Since 64 bit extensions were first integrated into X80 compatible processors in 2003 64 bit capable operating systems have allowed users to take advantage of both 32 bit and 64 bit computing Each system in your Linux cluster is installed with either Redhat CentOS or Scientific Linux 64 bit edition 2 2 BEOWULF CLUSTER ARCHITECTURE The purpose of a Beowulf or loosely coupled Linux cluster is to provide a large pool of compute resource Which can be easily managed from a single administration point Typically worker nodes are identically configured with other special purpose servers to provide cluster services including storage management and remote user access Each node in a Linux cluster is installed with an operating system often personality less system configurations are adopted on compute nodes to help reduce a
47. on available for both X86 and IAO4 architectures HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 21 of 40 b INFINIBAND FABRICS Available as add on interfaces to most compute node system Infiniband host channel adapters allow compute nodes to communicate with others at speeds of up to 32Gbps and latencies of around 2 microseconds Infiniband also provides RDMA remote direct memory access capabilities to help reduce the CPU overhead on compute nodes during communication Infiniband can transmit and receive messages in a number of different formats including those made up of Infiniband verbs or CP messages similar to Ethernet networks Suitable MPI implementations for Infiniband include e OpenMPI developed by merging the popular FI MPI LA MPI and LAM MPI implementations e MVAPICH an implementation of the MPI 1 1 standard e MVAPICH2 an implementation of the MPI 2 0 standard e Intel MPI a commercial MPI implementation available for both X86 and IAO4 architectures 0 3 SELECTING AN MPI IMPLEMENTATION Each MPI available on your cluster has been pre built for the available hardware and different compiler options available on your site Each option is configured by default to automatically run over the best performing interconnect network available Contact your system administrator if you need assistance running a parallel job over a differ
48. openmpi sh sleepjob sh sleepjob sh sleepjob sh sleepjob sh alces user r alces user r alces user alces user alces user 01 06 2011 10 04 01 06 2011 11 44 20 serial qlunode00 lel gw 017 0672011 didas 0 gw Ol 067 2011 11 48 44 gw 01 06 2011 11 48 52 243 paerallel qunode0l gel The job state can be marked as one or more of the following Status code Job state Description d deleted A user or administrator has requested that the job should be deleted from the queuing system 5 ee The job is in error status Use the explain E option to qstat for more information h hold The job has been set to hold by a user or administrator r running The job is running R Restarted The job has been restarted s suspended The job has been suspended and is not currently running S Suspended The job is currently being suspended t transferring The job is being transferred to an execution host to be run q queued The job is queued for execution wW waiting The job is waiting for resources to be available HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 30 of 40 By default the qstat command only shows jobs belonging to the user executing the command Use the gstat u command to see the status of jobs submitted by all users The qstat f command provides more detail
49. phical applications using a Remofe Deskfop client You can connect to the cluster headnode from any Linux Solaris Mac OSX or Microsoft Windows client with a suitable client application installed By default the XRDP application runs on port 3389 the default for Windows Terminal Services or Remote Desktop clients For Linux or MacOS X clients use the rdeskfop client to connect to the headnode machine username workstation rdesktop headnode 3389 For Microsoft Windows clients start the Remote Desktop Connection application enter the hostname or IP address of your cluster login node system and press the connect button Be Remote Desktop Connection BA Remote Desktop 2 Connection Computer login mycluster domain Username None specified You will be asked for credentials when you connect E Options b NX CLIENT CONNECTIONS The NX series of display libraries provides a convenient high performance method of running a graphical desktop from the cluster master or login node on your local client system Open source client binaries are provided by the NoMachineNX project for the majority of client desktops including Microsoft Windows Mac OSX Solaris and Linux Visit the following URL to download a client package suitable for your client system ECO www nomachine com download ph After installation start the NX Connection Wizard to configure the settings for your HPC cluster enter a recognisable name to
50. pt Gridware Libo eco Trews 3 202 11 userflogin01 HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright O 2013 Alces Software Ltd All Rights Reserved Page 20 of 40 MESSAGE PASSING ENVIRONMENTS 0 1 MESSAGE PASSING INTRODUCTION A message passing interface MPI is an API specification that allows processes to communicate with one another by sending and receiving messages Typically used for parallel programs an MPI can allow processes On one node to communicate with processes running on an entirely separate node providing greater flexibility and large scale applications to be executed across an HPC cluster Communication messages between nodes can be transported via a number of common interconnect networks including Ethernet Myrinet or Infinibana A number of different commercial and open source MPI implementations are available for C C and Fortran code and there may be multiple different options installed on your compute cluster Applications written to use an MPI are often compiled to include the MPI libraries they require using the compilers and optionally high performance libraries described in section 5 0 2 MPI INTERCONNECTS The network transport used for MPI messages will depend on the hardware provided with your HPC cluster Most basic clusters have a gigabit Ethernet interconnect which is shared between a number of different tasks More complex clusters may
51. r HPC cluster for convenience Use the modules command to load the OpenMP environment and compile your C code to make an executable binary called mpi_test user loginOl module load mpi openmpi 1 2 6 gcc User eloganG kl p mpc O Mol Pest Os estao Next create a new file called mpi_test sub sh and edit if fo contain the following lines to form a grid scheduler jobscript SGE SUBMISSION SCRIPT Specify the parallel environment and number of nodes to use Submit to the mpi verbose parallel environment by default so we can see output messages fo pe Mpi verbose 2 Specify specific SGE options Po cmd a yy Specify the SGE job name HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 37 of 40 5 N mpi test Specify the output file name FS o OUTPUT F A Machine tile for the Job is available here if you need to modify from the SGE default and pass to mpirun The machine hostnames in use are listed in the file separated by new lines in e g node00 node01 MACHITNEBIDE tmp s9S machines JOB 1D The mpirun command specify the number of processes per node and the binary mpr on manel up e qenmachines J 06 1D np 2 7 il test mp test Submit a new job to the cluster scheduler that will execute your binary using the grid scheduler jobscript Confirm the job is runnin
52. r compute jobs on the machine When logging in the default command line access method is via the bash shell interpreter Bash can be configured to provide convenient access to the different software you use need to help simplify usage of the cluster In your home directory a file called bashre contains a list of customisations which are executed by the bash shell every time you log in As your home directory is shared across all machines in the compute cluster the bashrc file is also executed automatically when your compute jobs are executed by cluster compute nodes The bashrc file may be customised by users by simply editing it in a text editor e g nano emacs or vim After changing the file users must logout and login again causing the bash shell to re read the file and apply any changes made When running applications or developing and compiling code to run on an HPC cluster there are a number of different compiler and library options which users may need to build or link against Many different packages have similar names for commands and it can quickly become difficult to instruct the system exactly which tools and utilities should be used Linux shell interpreters e g bash csh tsh use a series of environment variables to provide convenient short cuts for running binaries and locating system libraries Common variables include Environment variable Usage SPATH Stores the search path for executable tools and utilities
53. ration server Cluster configuration information Local compute node disk or scratch or tmp High speed transient data storage storage servers Where data storage areas are shared via NFS from the central fileserver compute nodes are configured with both a unique scratch storage area mounted under tmp or scratch and a shared storage area mounted under users Any applications or services installed on compute nodes that use the tmp area to store temporary data can make use of individual compute node disks without affecting the performance or capacity of the centrally stored data Users should be aware that data stored on node hard disks may be removed after their job has finished executing always copy data to be retained to your home directory as part of your jobscript HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 7 of 40 3 ACCESSING THE COMPUTE CLUSTER Cluster users typically work from the login nodes of an HPC cluster using them to submit jobs compile code and monitor the status of running jobs Larger clusters may have one or more dedicated login nodes whereas smaller machines may have just one login node hosted by the headnode system Your system administrator will provide you with the host address of the cluster login node s as well as your user name and password to access the cluster 3 1 LOGGING
54. s Software Ltd All Rights Reserved Page 29 of 40 8 OPEN GRID SCHEDULER CLUSTER SCHEDULER 8 1 USING OPEN GRID SCHEDULER Open Grid Scheduler OGS is a cluster scheduler designed to manage the resources available on your cluster machines and allow multiple users to securely and efficiently submit work to a shared resource Grid scheduler provides a simple mechanism for users to submit batch and parallel jobs from interactive nodes into centralized queves and have job results delivered to a designated location A typical grid scheduler installation requires a gmaster server normally the headnode or a cluster service node one or more submit hosts from where users can submit jobs typically a login or headnode server and a number of execution hosts where jobs are run The process for running a job through grid scheduler IS e Prepare the application or binary file to run on a cluster node e Create a jobscript to run the application with the required parameters e Select the grid scheduler directives required to control how your job is run e Submit the jobscript to GE from the cluster login or master node The steps below indicate how to perform these steps to submit different types of job to the grid scheduler scheduler 8 2 JOB SCRIPTS AND QSUB DIRECTIVES A jobscript usually takes the form of a simple shell script containing a list of tasks to perform in order to run your job These may include simple commands e g printing a status mess
55. s managed by a cluster scheduler also known as the batch scheduler workload manager queuing system or load balancer This application allows multiple users to fairly share the managed compute nodes allowing system administrators to control how resources are made available to different groups of users A wide variety of different commercial and open source schedulers are available for compute clusters each providing different features for particular types of workload All schedulers are designed to perform the following functions e Allow users to submit new jobs to the cluster e Allow users to monitor the state of their queued and running jobs e Allow users and system administrators to control running jobs e Monitor the status of managed resources including system load memory available etc More advanced schedulers can be configured to implement policies that control how jobs are executed on the cluster ensuring fair sharing and optimal loading of the available resources Most schedulers are extendible with a variety of plug in options for monitoring different metrics reporting system usage and allowing job submission via different interfaces The scheduler system available on your compute cluster will depend on how your system administrator has configured the system they will be able to advise you on how your HPC cluster is set up When a new job is submitted by a user the cluster scheduler software assigns compute cores and memory to sa
56. the above list of modules installed the following two commands are equivalent juser login0l module load apps hdts mpi l 8 6 qce 4 4 670penmpi 1 6 1 luUser loginOl Ilr module load apps hdrs mp17 1 8 6 HPC Cluster Quick Reference User Guide for Redhat CentOS Scientific Linux 6 v2 6 Copyright 2013 Alces Software Ltd All Rights Reserved Page 14 of 40 The module display and module help commands can help users to determine which environment variables are managed by different modules lwserGlogindl 7 module help Jibs gsl il lt 14 a Module Spectiic Help tor lao ace ely ld Sa Adds gs1 1 14 to your environment variables da ENV after load da GSLDIR gt Base path of library GSGLLB gt lb path GSLBIN gt bin path GSLINCLUDE gt Include path Adde GoLLIB CO LD LUTBART PATH Adds GSLBIN to PATH Adds gsl ManPages to MANPATH Adds GonINCLUDE CO C INCLUDE PATH and Pus INCLUDE PATH Adds necessary flags to build link against the library to CFLAGS and LDFLAGS userflogin01 After loading an application module the base location of the application is set in an environment variable to provide quick access to any application help or example files bundled with the application For example user loginOl module load apps mrbayes user loginOl echo mrbayes opt gridware apps mrbayes 1 3 9 user loginOl cd Smrbayes user loginOl 1 3 9 1s AUTHORS
57. tisfy the job requirements If suitable resources are not available to run the job the scheduler adds the job to a queue until enough resources are available for the job to run Your system administrator can configure the scheduler to control how jobs are selected from the queue and executed on cluster nodes Once a job has finished running the scheduler returns the resources used by the job to the pool of free resources ready to run another user job 7 2 TYPES OF JOB Users can run a number of different tyoes of job via the cluster scheduler including e Batch jobs non interactive single threaded applications that run only on one compute core e Array jobs two or more similar batch jobs which are submitted together for convenience e SMP jobs non interactive multi threaded applications that run on two or more compute cores on the same compute node e Parallel jobs non interactive multi threaded applications making use of an MPI library to run on multiple cores soread over one or more compute nodes e Interactive jobs applications that users interact with via a command line or graphical interface Non interactive jobs are submitted by users to the batch scheduler to be queued for execution when suitable resources are next available Input and output data for non interactive jobs are usually in the form of files read from and written to shared storage systems the user does not need to remain logged into HPC Cluster Quick Reference User
58. trally on your cluster service nodes with very few software packages installed on individual compute nodes This helps to ensure that all nodes are identically configured allowing jobs submitted to the cluster scheduler to run equally well whichever node they are executed on The operating system and system software installed on compute nodes is controlled by your system administrator from a central location individual nodes may be automatically re installed between successive job executions Users should only store temporary data on compute node hard disks as scratch areas may be cleaned between job executions 2 4 SHARED DATA STORAGE CONFIGURATION Your cluster system may be configured with a number of different file storage areas mounted on different servers for administrative and data sharing purposes These areas are designed to provide user data system libraries and configuration information to the nodes from a single central location By using shared storage the management overhead of a large cluster can be significantly reduced as there is no need to copy data to every node in the cluster for running jobs By default the following shared storage areas are typically configured on the cluster Storage mount point on nodes File server Purpose headnode or users Shared storage area for users storage servers headnode or Optional software including opt gridware storage servers cluster scheduler opt alces headnode or configu

High Performance Computing Cluster Quick Reference User Guide

Contents

Download Pdf Manuals

Related Search

Related Contents