Introduction to HPC at MSU - Vice President for Research and
Contents
1. Maintain the university supercomputer provide software as a service Organize Training Provide 1 on 1 consulting Help with grant proposal Funding From rt tC o e The Vice President office for Research and Graduate Studies VPRGS e Engineering College College of Natural Science and College of Social Science e This allows us to provides services and resources for FREE Online Resources T Ea e icer msu edu iICER Home hpcc msu edu HPCC Home e wiki hpcc msu edu HPCC User Wiki When would use the HPC e Takes too long for computation e Runs out of memory e Needs licensed software e Needs advanced interface visualization database e Read write Lots of data What is a supercomputer MICHIGAN STATE UNIVERSITY What is a supercomputer e A computer at the frontline of contemporary processing Capacity Introduced in 1960s Seymour Cray e MSU s supercomputer is a collection of computers that feature High FLOPS Floating Point Operations Second Fast Network Fast Reliable File Services Agenda Introduction to HPC Introduction to iCER e How to Use the HPC Get an account Install needed Software SSH X11 Xming XQuariz Basic navigation commands Transfer files Compile Test programs on a developer node Write a submission script Submitting a job Accounts O D e Pis can request accou2. Z O a O gt Z 2 gt rr SeT tr Fs Advanced Tip 2 e Access previous command by pressing up arrows e See full history by typing history e Repeat previous command in history by using the exclamation sign e g choiyj dev intell0 history 8 Is lat 18 will call the command Is lat e Practicing navigation again s e Navigate to the data directory e Use the tab completion feature Try to find the name and size of the file in that directory Examining Files e e Print all contents of a file Using cat cat filename e When the file is really big use less less filename Use arrow keys to scroll by line Space to go to the next page b to go backwards g to go to the beginning G to go to the end q to quit MICHIGAN sTaTE TO seach UNIVERSITY Basic manipulation commands Command cp lt from gt lt to gt cp r lt from gt lt to gt mkdir directory mv lt from gt lt to gt rm filename rmdir directory rm r directory Meaning Copy files Copy recursively files and directories Make named directory Move a file can be used as a rename command Remove the file Remove an empty directory Remove a directory recursively i e include all subdirs and files bcs ICER Short Exercise i oo 1 Rename the hidden directory to not_hidden 2 Create a new
3. Submit a job to the queue e qdel job id Delete a job from the queue e showq u user id Show the current job queue of the user e checkjob job id Check the status of the current job e showstart e all job id Show the estimated start time of the job MICHIGAN STATE UNIVERSITY Submission Script ns e List of required resource All command line instructions needed to run the computation Typical Submission Script 1 bin bash login PBS 1 walltime 10 00 00 mem 3Gb nodes 190 ppn 1 PBS j oe PBS_0_WORKDIR myprogram my input arguments astat f PBS_JOBID Submit a job e e go to the helloworld directory cd helloworld e Create a simple submission script nano hello qsub e See next slide to edit the file hello qsub tT bin bash login PBS 1 walltime 00 01 00 PBS 1 nodes 1 ppn 1 cd PBS O WORKDIR hello qstat f PBS JOBID Details about job script ti tC E is normally a comment except I special system commands bin bash PBS instructions to the scheduler PBS I nodes 1 ppn 1 PBS I walltime hh mm ss PBS I mem 2gb Not per core but a whole e hito wiki Apcc msu edu x Np T Submit a job a D e Once job script created submit the file to the queue qsub hello qsub e Record job id number and wait around 30 seconds e Chec
4. getexample a eet ty e If you do not load powertools please do it now e Download the helloworld example using getexample e Check what you downloaded What is the biggest file getexample e You can obtain a lot of examples through getexample Take advantage ot it Possible example names abaqus_examp1Le cloudy_examp1le fortran_openmp_blas makeflow Molpro_exampLe qsub_arraydepend ADMB_exampLe condor_advanced FreeSurfer MATLAB_basic mothur R_example ADMB_example2 condor_basic GAMESS_example MATLAB_blcr mothur2 R_plot affinity condor_Python gmp_mpfr MATLAB_compi Ler mothur_exampLe SAS_example alLinea_map condor_R gromacs MATLAB_compiler2 MPI_pi simpleMatlab Amber_CUDA_example condor_simple Hadoop_wordcount MATLAB_many_jobs multi_variable SNPPipeline Amber_example cuda HEEDS_test MATLAB_movie myhadoop splitBam avida_blcr cuda_clock helloMPI MATLAB_parameter_sweep NAMD_CUDA_example STATA_example basic_array_job cuda_hybrid hel LoOpenMP MATLAB_par for NAMD_example stata_parallel blast CUDA_WORKSHOP_UIUC1208 helloworld MATLAB_parfor2 Octave_basic tbb_exampLe blender_farm DDT_examples HFSS_examp1le MATLAB_patternsearch OpenACC_exampLe TotaLView_MPI_example BOOST_exampLe econ_examp1Les ImageJ MIC_exampLes OpenCL_hello_world Valgrind_example bowtie espresso_benchmark intro_workshop MKL_benchmark OpenMP_profiling VASP_example brother_test fftw lammps_test MKL_c_eigenvalues paraview_basic velvet_blcr burn_heat fluent3D LAPACK_example MKL_Examp
5. directories F append indicator one of to entries h print sizes in human readable format e g 1k 2 5M 3 1G 1 list with a long listing format t sort by modification time Basic navigation commands cd Command Meaning cd directory name change to named directory cd change to home directory cd change to home directory cd change to parent directory cd change to the previous directory display the path of the current d pw directory MICHIGAN STATE UNIVERSITY Basic Commands e To find a command use a search engine e Complete options instructions for that command man command_name e An exhaustive list hitp ss64 com bash e A useful cheatsheet http fosswire com post 2007 08 unixlinux command cheat sheet e Explain a command given to you http explainshell com MICHIGAN STATE fca UNIVERSITY CE Exercise e e Please type in your terminal module load powertools getexample intro_workshop This will copy some example files for this workshop to your directory Exercise try to find the file youfoundit txt in the hidden directory Advanced Tip EE e Tab Completion short cut Typing out directory file program names is time consuming When you start typing out the name hit tab key The shell will try to fill in the rest of the directory name E g return to home directory cd type cd I then press tab
6. directory called new 3 Copy the file youfoundit txt into the new directory Creating Editing text files e You could create edit files on your local system and transfer them using filezilla Downside slow Need to run command dos2unix if you are running a windows system e Much faster in the long term to learn how to edit files in the Command line Editors eT e Choice of editor is really a religion emacs popular powerful vi vim gvim popular powerful Nano simple easy to learn e Today we will learn nano To start type nano newfile txt e Commands are in the bottom A means control key e Create a file that says this is a text file e MICHIGAN STATE fca ICER Nano ss e Nano does not leave any output on the screen after it exists e But Is now shows that we have a new file called newftile e Lets tidy up by deleting this new file rm newfile NOTE no undelete in Linux unlike windows Other useful commands e wget get something from the www wget htto e ps get processes i e running programs e top get cou memory utilization e find a way to locate files e grep search for patterns e we get number of words lines e be bash calculator Available Software Fc ec ke E e Compilers debuggers and profilers Intel compilers ooenmp openmpi mvapich totalview e Libraries ACML BLAS FFTW LaPAC
7. Introduction to HPC at MSU CYBERINFRASTRUCTURE DAYS 2014 Oct 23 2014 Yongjun Choi choiyj msu edu Research Specialist Institute for Cyber Enabled Research Agenda e Introduction to HPCC Introduction to iCER e How to Use the HPC Get an account Install needed Software SSH X11 Xming XQuariz Basic navigation commands Transfer files Compile Test programs on a developer node Write a submission script Submitting a job How this workshop works O o SS e We are going to cover some basics Lots of hands on examples e Exercises are denoted by the following icon in this oresents Green and Red Sticky e Use the provides sticky notes to helo me help you No Sticky am working Green am done and ready to move on yea Red am stuck and need more time and or some help Agenda ec etc tele o TT e Introduction to HPC Introduction to iCER e How to Use the HPC Get an account Install needed Software SSH X11 Xming XQuariz Basic navigation commands Transfer files Compile Test programs on a developer node Write a submission script Submitting a job ICER T E D e What does ICER stand forg Institute for Cyber Enabled Research e Mission Reducing the Mean time to Science ICER s mission to help researchers with computational components of their research ICER Overview SSS e ICER is a research unit at MSU We
8. K MKL e Commercial Softwares MATLAB Mathematica FLUENT Abaqus e Fora more up to date list see the documentation wiki http wiki hpcc msu edu Module System A E E e To maximize the different types of software and system configurations that are available to the users HPCC uses a Module system e Key Commands module list List currently loaded modules module load modulename Load a module module unload modulename Unload a module module spider keyword Search modules for a keyword module show modulename Show what is changed by a module module purge Unload all modules MICHIGAN STATE bcs UNIVERSITY OER Short Exercise a SSS SSsq e Unload all modules and load these modules GNU powertools e Check which modules are loaded e Several versions of MATLAB are installed on HPC Find what versions are available on HPC Load the latest version Powertools e Powertools is a collection of software tools and examples that allows researchers to better utilize High Performance Computing HPC systems e module load powertools Useful Powertools commands e getexample Download user examples e powertools Display a list of current powertools or a long description e licensecheck Check software licenses e avail Snow currently available node resources e For more information refer to this webpage e hitos wikihbocc msu edu x p4Bn
9. You can run a program in the background by typing an amp after the command e You can make a program keep running even after you log out of your ssh session by using nohup command e You can run an entire session in the background even if you log in and out of your ssh session by using the screen command e All three of these options are common to linux and tutorials can be found online CLI vs GUI Me CLI Command Line Interface e GUI Graphical User Interface What is X11 Lo e Method for running Graphical User Interface GUI across a network connection SSH X11 PC Cluster What is needed for X11 tT a e X11 server running on your personal computer e SSH connection with X11 enabled e Fast network connection Preferably on campus Graphical User Interface a oe e X11 Windows Install Xming from ICER thumb drive ssh X username hpc msu edu e Turn on x11 forwarding e Note Mac Users should use XQuariz an Test GUI using X11 o e run X11 e Try one of the following commands xeyes or and firefox Programs that can use GUI Fe ec te o e MATLAB e Mathematica e totalview C C fortran debugger especially for multiple processors DDD Data Display Debugger graphical front end for command line debugger e Etc etc and etc Remote Desktop Client LC ST e RDP allows users to connect to our systems wi
10. ares Directory 09 19 2014 10 5 2 ddd Directory Wallpaper Cl Directory 03 30 2008 02 3 felix Directory d780a803346 Directory 10 22 2014 06 0 2 fontconfig Directory 125474175a1 Directory 10 22 2014 06 0 aean Pilenekan 112 files and 14 directories Total size 755446863 bytes 39 files and 43 directories Total size 1595249 bytes Server Local file Direction Remote file Size Priority Status Queued files Failed transfers Successful transfers a Queue empty ee UNIVERSITY ICER Transferring Files SFTP or SCP e Hostname gateway hpcc msu edu e Username msu netid e Password netid password e Port 22 an Exercise download this presentation from mnt scratch cholyj intro_hpcc intro_hpcc pdf to your local system Agenda ec ke o EEE Introduction to HPC Introduction to ICER e How to Use the HPC Get an account Install needed Software SSH X11 Xming XQuartz Basic navigation commands Transfer files Compile Test programs on a developer node Write a submission script Submitting a job Running Jobs on the HPC O o D e The developer dev nodes are used to compile test and debug programs e Two ways to run jobs submission job scripts interactive way e Submission scripts are used to run heavy many jobs on the cluster We will be back here later Advantages of running Interactively e Yo do not need to write a submission scrip
11. d access is prohibited We recommend using dev intel14 or nodes with low usage For GPU development please use green nodes For MIC development please use underlined nodes Development Nodes usage Filesystem Information dev intel 7 dev intel1 n a HOME at 3 usage dev gfx1 Clow dev intel14 Clow Cused 1 5G of 5G dev intel14 phi Clow dev intelL14 k2 Clow Cluster Load utilization short jobs lt 4 hrs C 50 general jobs 93 nodes with gt 256 GB 93 accelerated nodes 8 gateway 01 MICHIGAN STATE UNIVERSITY 7 ICER Types of Nodes E gateway hpcc msu edu dev intelO7 dev intell0 dev gfx10 Sa dev intel14 dev intel 4 k20 Say dev intel1 4 phi Say eval hocc msu edu GATEWAY NODES MICHIGAN STATE UNIVERSITY DEVELOPER NODES AN en a le Can Stock Photo csp0072636 COMPUTE NODES Gateway Nodes e Shared accessible by anyone with an MSU HPCC account e Shared resource hundreds of users on the gateway nodes e Only means of accessing HPCC compute resources e Useful information status file soace messages e DO NOT RUN ANYTHING ON THESE NODES Developer eval nodes e a E O O e Shared accessible by anyone with an MSU HPCC account Meant for testing short jobs Currently up to 2 hours of CPU time Develooment nodes are identical to compute nodes in the cluster Evaluation nod
12. es are unique nodes Node name descriptive name feature year e hito wiki hocc msu edu x owNe Compute Nodes et e Dedicated you request cores memory walltime also for advanced users accelerators temporary file space licenses e Queuing system when those resources are available for your job those resources are assigned to you e Two modes batch mode generate a script that runs a series of commands Interactive mode What is a Shell O o eet e A command interpreter that turns text into instructions Text can be entered interactively in the command line Text can be contained within a file e On the HPCC default shell is called BASH aka Borne Again shell e Note use bash as search parameter in google secure shell a nt e If you re not connected to the HPCC please do so NOW e From gateway please connect to a development node ssh dev intel10 e ssh secure shell allows you to securely connect to other computers Agenda e Introduction to HPC Introduction to iCER e How to Use the HPC Get an account Install needed Software SSH X11 Xming XQuariz Basic navigation commands Transfer files Compile Test programs on a developer node Write a submission script Submitting a job Basic navigation commanas Is po EE Command Meaning ls list files and directories Some options for ls command a list all files and
13. k jobs in the queue with qstat u userid Delete a job in a queue qdel jobid Status of a job qstat f jobid MICHIGAN STATE lig UNIVERSITY ICER Monitoring ee El e Submit the file to the queue qstat f jobid e When will a job start showstart e all jobid Scheduling Priorities IES o OS e NOT First Comes First Serves e Jobs that use more resources get higher priority because these are hard to schedule e Smaller jobs are backfilled to fit in the holes created by the bigger jobs Eligible jobs acquire more priority as they sit in the queue e Jobs can be in three basic states Blocked eligible or running Cluster Resources 2007 2009 2010 2010 2011 2014 intelO7 amd09 gfx10 intel10 intel11 intel14 MICHIGAN STATE UNIVERS IT Y Quad core 2 3GHz Intel Xeon E5345 Sun Fire X4600 Fat Node AMD Opteron 8384 NVIDIA CUDA Node no IB Intel Xeon E5620 2 40 GHz Intel Xeon 2 66 GHz E7 8837 Intel Xeon E5 2670 v2 2 6 GHz 2 NVIDIA K20 GPUs 2 Xeon Phi 5110P Large Memory 32 32 32 64 20 20 20 20 48 or 96 256GB 18GB 24GB 512GB 1TB 2TB 64GB 256GB 128GB 128GB 1 6TB 32 191 128 24 40 28 1008 96 256 1528 64 32 128 2560 480 800 560 288 7 ICER Scheduling Tips T C e Requesting more resources does not make a job run faste
14. le pbdR_examples XSEDE_MPI_WORKSHOP Circuitscape_examples fluentMPI magma_example MKL_mic pcap_example Clang_example fortran_openmp makefile_exampLe MKL_parallLel Python_MPI ICER Standard in out err and piping O o e S e You can redirect the output of a program to a file using gt character e myprogram gt output txt e You can also cause the output of the program to be the input of another program using the pipe character e myprogram myotherprogram Redirection and Piping a E e Change to the helloworld directory e Redirect the manual of the Is command to Is txt Print all content of Is txt using cat and more using pipe Print all content of Is txt using cat and less using pipe e Can you see the difference between more and less Redirection and Piping a e You can redirect the output of a program to a file using gt gt characters instead of gt Redirect the output of the Is command to Is1 txt USING yn 2 Repeat 3 Redirect the output of the Is command to Is2 txt USING gt gt 4 Repeat 3 5 Check the difference between Is1 txt and Is2 txt Agenda Introduction to HPC Introduction to iCER e How to Use the HPC Get an account Install needed Software SSH X11 Xming XQuariz Basic navigation commands Transfer files Compile Test programs on a developer node Write a submission script Submitti
15. ng a job Transferring Files SFTP or SCP s e Windows ICER thumb drive Pre installed filezilla e OSX download and install filezilla http filezilla project org e Hostname gateway hpcc msu edu e Username msu netid e Password msu netid password e Port 22 Transferring Files SFTP or SCP 4 Al EE G S Bw ok HA Fa Host sftp gateway hr Username choiyj Password eecceccces Port Quickconnect Command Is Status Listing directory mnt home choiyj Status Calculating timezone offset of server Command mtime intel Response 1406924164 Status Timezone offsets Server 14400 seconds Local 14400 seconds Difference 0 seconds Status Directory listing successful Local site Users yongjunchoi Downloads Remote site mnt home choiyj ba gt j v Bg gt Library v mt gt Movies v home gt Music gt j LOCAL b Brei gt Public SYSTEM p l Sandbar MSU s Filename Filesize Filetype Last modified Filename Filesize Filetype H PCC BB B lt 1d96d1c1318 Directory 10 22 2014 06 0 2 Mathematica Directory 1 5cdc0d385a Directory 10 22 2014 06 0 1 backup Directory About Down Directory 07 01 2014 10 0 1 cache Directory CodePad app Directory 05 23 2014 09 4 1 codeblocks Directory 2 Draft Directory 09 30 2014 09 4 1 config Directory GraphicalHtt Directory 03 27 2013 08 0 2 dbus Directory Softw
16. nts for each group member at http contact icer msu edu account e Each account has access to 50 GB of replicated file soace mnt home userid 520 processing cores 360 TB of high speed scratch space mnt scratch userid e Also available shared group folder upon request O O gt Z gt rm Nuts amp Bolts a e Our supercomputer is a remote service gateway hocc msu edu Personal Computer a _ gt gt gt f X 5 http wiki Apcc msu edu x DYAT Can Stock Photo csp0072636 Agenda ec ke o EEE Introduction to HPC Introduction to ICER e How to Use the HPC Get an account Install needed Software SSH X11 Xming X Quartz Basic navigation commands Transfer files Compile Test programs on a developer node Write a submission script Submitting a job Connecting to the HPCC using ssh i e Windows we have putty pre installed on iCER thumb drive Insert the thumb drive Open appropriate folder StartPortableApps exe Locate putty in the menu double click on socc default connection Username your netid password your netid password e OSX Access spotlight from its menu bar icon or SPACE Type terminal MICHIGA UNIVERSITY In terminal type ssh netid gateway hpcc msu edu N STATE Connecting to the HPCC using ssh VT ROG Epa j eg eg A A a ITE I If Pe Vu ee JY f f S Unauthorize
17. r unless you a running a parallel program e The more resources you request the harder it is for the scheduler to reserve those resources e First time over estimate how much resources you need and then modify appropriately e qstat f S PBS_JOBID at the bottom of my scritps will give you resources Information when the job is done Advanced Scheduling Tips e Resources A large proportion of the cluster can only run jobs that are four hours or less Most nodes have at least 24 gb of memory Half have at least 64 gb of memory Few have more than 64 gb of memory Maximum running time of jobs 7 days 168 hours Maximum memory that can be requested 6tb e Scheduling 10 eligible jobs at atime 512 running jobs MICHIGAN SRO Submitted jobs UNIVERSITY Job completion e By default the job will automatically generate two files when it completes Standard Output e Ex jobname o594557 1 Standard Error Ex jooname e594557 You can combine these files if you add the join option in your submission script PBS j oe You can change the output file name PBS o mnt home netid myoutputfile txt Other Job Properties e resources l Walltime memory nodes processor network etc PBS I feature gpgpu gbe PBS I nodes 2 ppn 8 gpu 2 PBS mem 16gb e Email address M PBS M choiyj msu edu Email Options m PBS m abe Many others see the wiki hito
18. t e Yo do not need to wait in the queue e You can provide input to and get feedback from your programs as they are running Disadvantages of running Interactively D De e All the resources on developer nodes are shared between all uses e Any single process is limited to 2 hours of cpu time If a process runs longer than 2 hours if will be killed e Programs that overutilize the resources on a developer node preventing other to use the system can be killed without warning Developer Nodes _ SSS r Dev intel07 dev gfx10 4 18 2xM 1060 Nvidia Graphics Node dev intel0 8 24 dev intell4 8 64 dev intell4 phi 20 128 2xPhi Xeon Phi Node dev intell4 k20 20 128 2xK20 Nvidia Graphics Node MICHIGAN STATE ifa UNIVERSITY ICER Compiling e Most users use the developer nodes for developing their software e f you are using a makefile you can compile using more processors with the j option make j8 Will compile using 8 core thread Compilers O o D e by default we use the GNU compilers However lots of other compilers are available including Intel and Portland compilers e The module system always sets environment variables such that you can easily test with other compilers Compile Code e P l e change to helloworld directory e run the gcc Compilers gcc O3 o hello hello c run the program hello Running in the background D E e
19. th superior performance relative to X11 forwarding over SSH e RDP is available at rdp hocc msu edu from on campus or via the MSU VPN e Windows Microsoft Remote Desktop e Mac Need fo install the Microsoft Remote Desktop from the App Store e The easiest way to connect to the HPCC via RDP is to download amp run this template file ICER HPCC RDP httos wiki nocc msu edu x tABZAQ MICHIGAN au UNIVERSITY Remote Desktop Client e Run RDP client and connect with the remote desktop client using ICER HPCC RDP template file e Open a terminal Application Menu gt Terminal Emulator or click the terminal icon E on the upper left corner e Connect to dev intell10 and run xeyes Agenda Introduction to HPC Introduction to iCER e How to Use the HPC Get an account Install needed Software SSH X11 Xming XQuariz Basic navigation commands Transfer files Compile Test programs on a developer node Write a submission script Submitting a job Resource Manager and Scheduler Not First In First Out MICHIGAN STATE mn Scheduler vs Resource Manger e Scheduler Moab Tracks and assigns e Memory e CPUs e Disk soace e Software Licenses e Power environment e Network Resource Manager PBS Torque Hold jobs for execution Put the jobs on the nodes Monitor the jobs and nodes Common Commands e qsub submission script
20. wiki hocc msu edu Advanced Environment Variables ed e The scheduler adds a number of environment variables that you can use in your script PBS_JOBID e The job number for the current job PBS_O_WORKDIR e The original working directory which the job was submitted Example mkdir S PBS_O_WORKDIR S PBS_JOBID Softwares Modules again T i ii E e ICER has over 2500 software titles installed e Not all titles are available by default e We use modules to setup the software e Some modules are loaded by default module list e To see different version module spider MATLAB e To search also use the module spider command Getting Help Documentation and User Manual wiki nocc msu edu Contact HPCC and ICER Staff for Reporting System Problems HPC Program writing debugging Consultation Help with HPC grant writing System Requests Other General Questions Primary form of contact httpo contact icer msu edu HPCC Request tracking system rt hpcc msu edu HPCC Phone 517 353 9309 HPCC Office 1400 PBS nat Office Hours 1 2 om Monday Thursday PBS 1440 MICHIGAN STATE bcs UNIVERSITY OER
Download Pdf Manuals
Related Search