Home

ScaMPI User's guide

1. mpid mpimon mpisubmon and the libraries all have version variables that aretested at startup An incompatibility have normally one of two reasons A new version of ScaMPI is installed without restart of mpid or the environment variable MPI_H OM E is set wrongly Why does my application terminate abnormally Are you reasonable certain that your algorithms areMPI safe The program just hangs Try to start your program with init_comm_world specified if it doesn t start you have a buffer allocation problem read controlling shared memory allocation If you have large degree of asynchronicity try to increase the channel_size else are you really sure that your algorithms are MPI safe The program terminate without with error message If core was dumped look at the file else try again with ver bose Why does my application terminate abnormally SCI interconnect failures The program terminate with ICMS_ FAILURE The program terminate with ICMS_OUTBOUND_MAP_FULL This is a SCI problem and a reload of SCI drivers may be nec cessary Check the SCI documentation in the Scali system guide and your system administration about the cause of the problem Use the interconnect diagnostic in the Scali Desktop GUI Contact support scali com if there is a SCI problem need ing attention Problems and fixes will be included in a FAQ on http www scali com What monitor options are availa ble Run mpi
2. CHECK Use e g opt scali bin scaps A process holds SCI or shared memory resources Core dumping takes time CHECK 1 Use opt scali bin scidle CHECK 2 Use opt scali bin scish to check for SHM segments ipcs for Solaris amp Linux TBD for WinNT Why doesn t my program start to run advanced interconnect problems space overflow Y our application have required too much SCI or shared mem ory resources CHECK Y our mpimon size specifications are too large check with mpimon verbose CHECK Number of communicators in the program is higher than expected Why doesn t my program start to run advanced core dump The application core dumps CHECK Use a parallel debugger e g TotalView to find the point of violation after an appliacation recompile for inclusion of debug symbol information g for most compilers A man Ual debug session is possible with gdb dbx pgdbg or any available sequential debugger Table 5 2 Trouble shooting ScaMPI User s Guide Version 1 9 0 5 4 Trouble shooting Problem Description Solution Why does mpid not start mpid opens a socket with a fixed identification obtained from getservbyname with name mpid If mpid is terminated abnor mally then this socket will not be available before a timeout Use netstat a grep mpid to observe when socket is released and start mpid again How to resolve incompatible mpi versions
3. BASIC EE TEE 8 Typographic Copventions nennen nnnnn nn 9 Environment variables on Uni 16 Environment variables for Unix on Windows NT A 17 Basic options CO mpimmon nennen nn nnnen nn 20 Mpimon parameters sennnennnnennannnnnnnannnnnnnnnnnnnnnnnnnnnnnnnsnnnnnanannnnsnn nn 21 NuMetal osa 21 Complete list of mpimon OPti0NS E 22 Environment setup for TotalView En 28 Setup for VAMPIR and VAMPilHtrace AEN 3l L DRAG ES e ee ee die 35 Keele EE 36 Startup Phase 2 nennen 37 e EL EE 38 Stopping Steps EE 39 Namespace pollution niiata niini id diaii iaiia 48 roubl amp Sho8tiNng ee nn RS 50 ScaMPI User s Guide Version 1 9 0 65 ScaMPI User s Guide Version 1 9 0 66 c CCS resource management software unsnnersennnnnnnsnennnennnnnnsnnnnnnnnnnnnnnennnnnnnnnnnnnnnnnn na 57 compiler e EE 18 de E le EE 17 COMPUTING iiuii sasata bat TDi adde Na ed e denat 12 17 D e Le MEET 25 E error MESSAGES a ne na en RT 49 G e le EE 28 l Mii een ehe 59 install requirements iiinis diine eier bits 59 L ege 35 HAKINA WEE TEE 19 M MiPI Test programs cicatrices arta are Henne A enn ee 13 ul tee e GE 35 MP oros ahaa lod 35 O O 35 MPISUDIMOM O Eet ate 35 P POMO andara nea 29 R GUNNING nk dd tada 20 S Scali Desktop GU cocci n id ta tae Ai 20 SOUL WEE 16 17 T A A and ET 25 Total View Users s Guide neiepehednni a aien i i aeaaea ka daare irera 57 trouble Sho ii A A dci 50 VM Vocero tats 29 VAMPIRtrace Installation an
4. tion on a Scali system by using a performance analysis tool Recommended are the VAMPIR MPI profiling software for ScaM PI contact sales support com for further details It is also possible to use the freely available ScaMPE profiling library with ScaMPI How do optimise MPI perform ance uneffectiveisend irecv If communication and calculations does not overlap using MPI_Isend MPI_Irecv or variants of these are usually per formance ineffective Table 5 2 Trouble shooting ScaMPI User s Guide Version 1 9 0 5 4 Trouble shooting Problem Description Solution How do optimise MPI perform ance avoid starving processes fairness MPI programs may if not special careis taken be unfair and may starve processes eg by using MPI_Waitany as illus trated for a client server application in example3 15 3 16 in the MPI 1 1 standard Fairness can be enforced e g by use of several tags or separate communicators How do optimise MPI perform ance avoid using isend irecv creating many threads beyond number of processors When immediate send are used ScaMPI creates an additional thread for handling this The same applies for immediate receive so a total of two extra threads may be created when using immediate function calls Having morethan onethread on a multi processor usually improve performance while increasing the number of threads beyond thenumber of proces sors
5. Table 3 6 Complete list of mpimon options ScaMPI User s Guide Version 1 9 0 3 5 Debugging ScaMPI applications 3 5 Debugging ScaMPI applications Debugging ScaM PI applications is recommended using a parallel debugger Manual debugging with a separate debugging session for each MPI process requires no parallel debugger It may however become a time consuming and tedious task when debugging manually several processes Currently the only parallel debugger option is TotalView with support for Solaris and a subset of compilers Support for other OSes and compilers can be made available contact sales scali com for further information Starting a ScaMPI program with a parallel debugger you can stop the program in MPI_Init Here the parallel debugger give you a single point of control for the debugged MPI program s A manual debug session is started with mpimon options e g as debugger dbx debug 0 An xterm window is started for each of the specified MPI processes in this example process 0 A message containing the program run parameters needed by the debugger is displayed Then the debugger here dbx is started The user must manually insert therun parameters to the debugger for all debug processes before setting breakpoints or any other debugging actions 3 5 1 Debugging with TotalView This section describes TotalView briefly explains how to set up the user environment and gives the first getting started with ScaMPI and TotalView instruc
6. connection between a sender and a receiver There will be only one eagerbuffer for a sender receiver pair E agerbuffer Bluffers Sender ii Receiver Usedflags Figure 4 5 Eagerbuffer ScaMPI User s Guide Version 1 9 0 41 4 4 Communication resources 4 4 1 3 Transporter Transporter is a connection between a sender and a receiver for large datatransfers There will be only onetransporter for a sender receiver pair Transporter Ringbuffer Sender Y Receiver Figure 4 6 Transporter 4 4 2 Parameters mpimon options 4 4 2 1 Prefixes The parameters concerning resources have a prefix which is either inter meaning between two hosts or intra meaning within same host 4 4 2 2 Channel The size of channel ringbuffer can be set with inter_ intra_channel_size 4 4 2 3 Eagerbuffer The size of each eagerbuffer can be set with inter_ intra_eager_size The number of eagerbuffers can be set with inter_ intra_eager_count 4 4 2 4 Transporter The size of each ringbufferpart can be set with inter_ intra_transporter_size The number of ringbufferparts can be set with inter_ intra_transporter_count ScaMPI User s Guide Version 1 9 0 42 4 5 Communication protocols 4 4 2 5 Shared memory mpisubmon has two pools of shared memory one for inter host resources and the other for intra host resources The size of eac
7. may reduce performance dueto active waiting and con text switch time How do optimise MPI perform ance using sendrecv transforms to isend irecv In ScaMPI MPI_Sendrecv transforms into MPI_Isend amp MPI_Recvi e twothreads Using MPI_Sendrecv to communi cate on two processes on one node then transform into 4 threads assuming application is single threaded and may be slow on a dual processor node For small messages lt eager_size MPI_Isend is treated as MPI_Send and hence acceptable performance To shift the performance degradation to out of observation field set the intra_eager_size 1M or asizelarger than the largest message size Table 5 2 Trouble shooting ScaMPI User s Guide Version 1 9 0 Chapter 6 Support 6 1 Feedback We welcome any suggestions improvements feedback or bug reports on this User Guide and the software described herein Please send your comments to support scali com We also encourage the user of parallel tools software with ScaM PI on Scali clusters to comment any aspects of the softwaretothe NHSE National HPCC Software Exchange branch Parallel Tools Library 12 It is a site open tothe public giving review of parallel tools and allowing users software vendors and otherstorespond and express their views and frustrations Please help us improve the parallel software on our Scali clusters Other important mail groups are scali announce and scali user J oin the appro
8. program Run the program with mpimon environment export to collect performance data It is important to export the environment so that all participating processes have an appropriate license setup When MPI communication stops the trace is written from memory to a file and a message is sent to stdout It is also possible to instrument a program manually with VAM PIRtrace calls The extra overhead caused by profiling code can be turned off by linking with the dummy library IVTnull instead of the real IVT library The DIMEMAS performance prediction library can also be substituted for the performance analysis VAMPIR library contact sales scali com if you want this option Analyse the program performance with VAMPIR Start the VAMPIR analyse program e g vampir A UseFilemOpen Tracefileand browse and select the appropriate trace file for analysis The point and dick interface of VAMPIR should get you easily started F igure 3 3 and 3 4 show some global information displays offered by VAMPIR Look in the VAMPIRtrace Installation and User s Guide on ScaMPI 7 for further information 3 6 2 Profiling with the MPE library ScaMPE is a modified version of the freely available MPE Multi Processing Environment libraries from MPICH An executable linked with the ScaMPE library collect performance data during runtime with output to a file The main components of the MPE library are e A set of routines for creating logfiles for examination by e
9. pshot Start the upshot or any other analyse program that understands the clog format e g nupshot or jumpshot 9 upshot Browse and select the appropriate trace file for analysis MPICH documentation 14 includes a description of the MPE graphics routines The ScaM PE package is downloadable from http www scali com ScaMPI User s Guide Version 1 9 0 33 3 6 Profiling ScaMPI applications ScaMPI User s Guide Version 1 9 0 34 Chapter 4 Description of ScaMPI 4 1 General description ScaM PI consists of libraries to be loaded with the user application program and a set of executables which control the startup and execution of the user application program 4 1 1 Libraries Name Description libmpi Standard library containing C api libfmpi Library containing fortran api wrappers Table 4 1 Libraries 4 1 2 Mpimon Mpimon is a monitor program which is the user interface for running the application program 4 1 3 Mpisubmon M pisubmon is a submonitor program which controls execution of application program One mpisubmon is started on each host per run 4 1 4 Mpiboot M piboot is a bootstrap program used when running in manual debug mode 4 1 5 Mpid M pid is a daemon program running on all hosts that can run ScaM PI Mpid is used for starting mpisubmon programs to avoid using UNIX facilities like remote shell which has proven to be error prone M pid is started automatically when a hos
10. shooting Problem Description Solution How do control SCI and local shared memory usage advanced an example with ScaMPI 1 6 4 Running two processes on one node with channe size 256k mpimon intra_channel_size 256k intra_pool_size 4m opt scali examples bin bandwidth lt nodename gt 2 Would terminate without starting with the message uiError Intra_pool_size must be at least 4227072 bytes 2113536 bytes 2 processes for given set of parameters Calculation is left out as an exercise Using theminimum pool size mpimon intra_channel_size 256k intra_pool_size 4227072 Y opt scali examples bin bandwidth lt nodename gt 2 Would start with the following parameters intra_channel_size 256K intra_chunk_size 1M intra_eager_count 1 intra_eager_size 1K intra_pool_size 4227072 intra_transporter_count 4 intra_transporter_size 512 A more natural choice of parameters may be mpimon intra_channel_size 256k intra_pool_size 6m opt scali examples bin bandwidth lt nodename gt 2 Would start with the following parameters intra_channel_size 256K intra_chunk_size 1M intra_eager_count 2 intra_eager_size 64K intra_pool_size 6m intra_transporter_count 4 intra_transporter_size 64K Note channel_size 256k is an unusual high value How do optimise MPI perform ance perfomance analysis Learn about the performance behaviour of your MPI applica
11. Description lt program amp host spec gt lt program spec gt lt host spec gt Lost spec gt lt program spec gt lt userprogram gt lt programoptions gt lt userprogram gt Name of application program lt programoptions gt Program options to application program Separator signals end of user program options lt host spec gt lt hostname gt lt count gt lt hostname gt Name of host lt count gt Number of processes to run on host mpimon will start 1 process on host if lt count gt is omitted Table 3 4 Mpimon parameters Numeric values can be given as mpimon options in the following way Option Description lt numeric value gt lt decimal value gt lt decimal value gt lt postfix gt lt postfix gt lt K gt numeric value gt lt decimal value gt 1024 lt M gt numeric value gt lt decimal value gt 1024 1024 Table 3 5 Numeric input ScaMPI User s Guide Version 1 9 0 21 3 4 Running MPI programs Mpimon option automatic lt a ection gt Description Set automatic mode for process es Default none Legal n m o list or n m range or all barrier_fanin lt count gt Set number of barrier fanin reads barrier_fanout lt ount gt debug lt se ection gt Default Set number of barrier fanin reads Default 8 Set debug mode for process es Def
12. Pointer to log error facility structure mpipriv Common block for all global variables when linking with Fortran code defined in mpif h PMPI_Comml dent C routine which returns a unique identifier for each communi cator used in trace routines PMPI_PrintAllState C routine which can be used to print some extra state used when debugging optimization Table 5 1 Namespace pollution ScaMPI User s Guide Version 1 9 0 48 5 3 Error messages Name Description PMPI_PrintCommState C routine which can be used to print some extra state used when debugging optimization PMPI_PrintRecvChannelState C routine which can be used to print some extra state used when debugging optimization PMPI_PrintSendChannel State C routine which can be used to print some extra state used when debugging optimization Table 5 1 Namespace pollution 5 3 Error messages 5 3 1 User interface errors U ser interface errors are problems with the environment setup causing difficulties for mpimon when starting a ScaMPI program M pimon will not start before the environment is properly defined These problems are usually easy to fix by giving mpimon the correct location of some executable The error message give a stright forward indication of what to do Only particularly troublesome user interface errors will be listed here 5 3 2 Fatal errors At a fatal error ScaMPI writes an error message before starting MPI_Ab
13. ScaMPI User s guide Copyright 1999 2000 Scali AS All rights reserved Acknowledgement The development of ScaMPI has benefited greatly from the work of people not connected to Scali We wish especially to thank the developers of MPICH for their contributions to the first ScaMPI implementation The list of persons contributing to algorithmic ScaM PI improvements are impossible to compile here We apologise to those who remain unnamed and mention only those who certainly are responsible for a step forward Scali is thankful to Rolf Rabenseifner for the improved reduce algorithm used in ScaM PI Table of contents Chapter 11Introduction uuanesonnunnnnunnnunnnnunnununnunnunnnnnannnnnnnnnnnnnnnnnnnnnannnnannun rr 7 1 1 Purpose of the ScaMPI Usereouide AEN 7 1 2 Scope of the ScaMPI User s ouide NEEN 7 1 3 Who should read this ouide AEN 7 1 4 Acronyms and abbreviations ENN 8 1 5 Terms and Ee e CC del EEN 8 1 6 Typographic comventtons ENEE 9 Chapter 2 Getting started commons crecen rre anar 11 2 1 Thefirst example ci een aba anne rada 11 2 1 1 Setting up your BASH environment E 11 2 1 2 C source of a Hello world MPI prooram nn 12 2 1 3 Fortran source of a Hello world MPI program 12 2 1 4 ue O oia een 12 2 1 5 Dat d Ile DEE 13 21 6 RUNMING WEE 13 22M PIES PO AMOS ii er nei ih 13 2 2 1 A producer consumer MPI test program 13 2 2 2 A bandwidth MPI test porogram nn 14 2 2 3 A bidirectional MPI test program E 14 Chapter 3 Us
14. alView Multiprocessor Debugger User s Guide Dolphin Toolworks Version 3 8 0 1998 http www dolphinics com tw 8 TotalView Multiprocessor Debugger Installation Guide Dolphin Toolworks Version 3 8 0 1998 http www dolphinics com tw 9 VAMPIRtrace for Solarisx86 ScaMPI Installation and User s Guide Pallas GmbH Release 1 0 for VAMPIRtrace version 1 5 1998 http www pallas de 10 Review of Performance Analysis Tools for MPI Parallel Programs http www cs utk edu browne perftools review 11 High Performance Debugger Forum http Avww ptools org hpdf 12 NHSE National HPCC Software Exchange Parallel Tools Library http www nhse org ptlib 13 TFCC IEEE CS Task Force on Clustered Computing Parallel Tools Library http www dgs monash edu au rajkumar tfcc ScaMPI User s Guide Version 1 9 0 57 7 1 References 14 The MPICH implementation home page http www mcs anl gov mpi mpich index html 15 The Extreme Linux Organisation hitp www extremelinux org 16 The Scali parallel tools environment Draft 1999 Scali AS http www scali com ScaMPI User s Guide Version 1 9 0 58 Appendix A Install ScaMPI This appendix explains some details of theScaM PI installation process Scali software installation onto a Scali system with many cluster nodes is most easily done from the GUI Scali Desktop see the Scali system guide Contents summary If you are onl
15. and default values Italics User input Command prompt in shell with super user privileges Command promt in shell with normal user privileges s gt 2 Notation intended for simple display of Visual GUI setup is dick is double click is uncheck gt is a menu selection chain is a window open for multiple selections is a binary operator left operand is name or category of selection right operand is value of selection It is either a predefined value or a specified string user context requirements E g File gt New gt Project 3Win32 Console Application project name location or input according to These icons are used througout the text to refer to OS specific details respectively Solaris and Linux Table 1 3 Typographic conventions ScaMPI User s Guide Version 1 9 0 O 1 6 Typographic conventions ScaMPI User s Guide Version 1 9 0 10 Chapter 2 Getting started This chapter shows how to run your first MPI program in section 2 1 It is also possible to run some test programs that measure basic MPI performance in section 2 2 All examples are compiled with Gnu or Microsoft Visual compilers and run from a BASH shell Of course ScaMPI has to be installed and working on your system If not contact your system administrator or refer to ScaMPI install instructions in appendix A 1 Using other compilers or a Windows NT Visual GUI on these examples might cause some problems Please
16. are export MPI_LDLIBS CRT_BEGIN 1mpi SCRT_END CRT_BEGIN and CRT_END should be empty if no value is given for the compiler Details can befound in the release notes LD_LIBRARY_PATH TheLD_LIBRARY_PATH variable may be updated to includethe directory wherethe dynamic libraries can be found export LD_LIBRARY_PATH LD_LIBRARY_PATH MPI_HOME lib An alternative to use LD_LIBRARY_PATH is to give a flag to the linker which can includea specific path to the dynamic libraries see the release notes PATH The PATH variable may be updated to include the directory where the mpi binaries can be found export PATH PATH MPI_HOME bin Table 3 1 Environment variables on Unix ScaMPI User s Guide Version 1 9 0 16 3 2 Compiling 3 1 2 The Unix on Windows NT ScaMPI environment At present thereareno Unix on Windows NT e g Cygnus Interix available with MT safe POSIX interface therefore ScaMPI on Windows NT only supports Windows MPI programs It is however possible torun ScaMPI from a Cygnus BASH shell and use a Unix makefile system with the Visual C and Fortran compilers A few environment variables must be set Name Description MPI_HOME Installation directory For a standard installation this should be set as MPI_HOME c opt scali export MPI_HOME MPI_LDLIBS Library to be loaded defined as Id directives For a standard installation the necessary libraries are export MPI_LDLIBS scampi li
17. ault none Legal n m o list or n m range or all debugger lt debugger gt Set debugger to start in debug mode disable timeout Disable process timeout display lt display gt Set display to use in debug manual mode exact_match Set exact match mode execpath lt execpath gt Set path to internal executables environment lt value gt Define how to export environment Default none Legal export all or mpi MPI_ or none help Display this home lt directory gt Set installation directory immediate_handling sdection gt Handling of immediates Default lazy Legal lazy threaded automatic inherit_limits Inherit userdefinable limits to processes init_comm_world Init COMM_WORLD at startup all channels are created inter_adapters lt adapters gt Set list of sci adapters for inter communication Default all Legal n m o list or n m range or all inter_channel_inline_threshold lt size gt Set threshold for inlining in bytes per inter channel Default 560 Table 3 6 Complete list of mpimon options ScaMPI User s Guide Version 1 9 0 3 4 Running MPI programs Mpimon option inter_channel_size lt size gt Description Set buffer size in bytes per inter channel Default 4K Legal Powers of 2 inter_chunk_size lt size g
18. b scacom lib scabase lib scasci lib Details can be found in the release notes LD_LIBRARY_PATH TheLD_LIBRARY_PATH variable may be updated to include the directory wherethe dynamic libraries can be found export LD_LIBRARY_PATH S LD_LIBRARY_PATH MPI_HOME lib An alternative to use LD_LIBRARY_PATH is to give a flag to the linker which can include a specific path to the dynamic libraries see the release notes PATH The PATH variable may be updated to include the directory where the mpi binaries can be found export PATH PATH MPI_HOME bin Table 3 2 Environment variables for Unix on Windows NT 3 2 Compiling ScaM PI is an API Application Programming Interface and not an ABI Application Binary Interface hence all applications must be recompiled and linked with ScaMPI 3 2 1 Compiler support Unix ScaMPI is a C library built using GNU g hence the MPI libraries must be linked in differently depending on the users choice of compilers Check the release notes for details regarding support of your compiler Note the ScaFgcc package or a similar version of gcc must be installed on the system Solaris ScaMPI is supported for use with the listed compilers GNU and EGCS gcqg g77 for UltraSPARC and 86pc ScaMPI User s Guide Version 1 9 0 17 3 2 Compiling Apogee apcc apC C apf77 apf90 for UltraSPARC Portland Group pgcc pgf77 pgf90 86pc Sun SunPro CC f77 f90 for UltraSPARC and i86
19. d User s Guide AAA 57 ScaMPI User s Guide Version 1 9 0 67
20. e 4 10 Transporter ScaMPI User s Guide Version 1 9 0 45 4 5 Communication protocols ScaMPI User s Guide Version 1 9 0 46 Chapter 5 Getting help This chapter is the place to start when something goes wrong running your ScaMPI programs If you have any problems with ScaMPI first check the not yet complete list of common errors and their solutions Eventually a compiled and updated list will be available on a frequently asked questions FAQ page at http www scali com If you cannot get help about how to solve your problems here please get assistance from Scali Read the support chapter 6 first before you contact support scali com Contents summary Section 5 1 contains some application notes about why some programs run with MPICH and not ScaMPI Section 5 2 give a list of all the names to avoid when programming with ScaMPI Section 5 3 contains error messages given by ScaM PI and section 5 4 about trouble shooting give the solution to common problems These sections are by no means complete now but problems reported to Scali will eventually get intothisScaMPI troubleshooting chapter So please send your relevant complaints to Scali 5 1 Application notes 5 1 1 MPI_Probe and MPI_Recv During development and test of ScaMPI we have run into several application programs with the following code sequence while MPI_Probe MPI_ANY_SOURCE MPI_ANY_TAG comm sts if sts gt MPI_TAG SOME_VALUE MPI_Rec
21. ed pool size Thecommunicators parameter is dependent of the application assumed to betwo in the automatic approach If more com municators than expected by the buffer size cal culations are used the application may run out of shared memory The pool sizeis a limit for thetotal amount of shared memory Default pool sizeis set to 32M inter and 4M intra node How do control SCI and local shared memory usage advanced automatic buffer management The automatic buffer size computations is based on a full con nectivity i e all communicating with all others If all process communicate with all others they will communicate P_inter processes inter node and with P_intra processes intra node onesdf inclusive Each communication channd is therefore restricted to use only inter_part inter_pool_size P_inter P_intra intra_part intra_pool_size P_inter P_intra P_intra The automatic approach is to downsize all buffers associated with a communication channd until it fits in its part of the pool The chunk size sets the size of each individual allocated memory segment The automatic chunk sizeis calculated to wrap a complete communication channd How do control SCI and local shared memory usage advanced barrier buffer The barrier buffer is one page and up to barrier_fanout 1 default 8 1 buffermappings are created Table 5 2 Trouble shooting ScaMPI User s Guide Version 1 9 0 5 4 Trouble
22. ed or Project gt Settings Settings For Win32 Release C C gt Code Generation U se Runtime Library M ultithreaded Compiling Fortran code skip if only c c source ScaMPI User s Guide Version 1 9 0 18 3 3 Linking 6 Add a preprocessor definition and include path Project gt Settingsl Settings For All Configurations Fortran gt Preprocessor Preprocessor Definitions WinNT Custom INCLUDE c opt scali include JU se Path c opt scali include 7 Select multithreaded runtime libraries enable reentrancy support Project gt Settings Settings For All Configurations F ortran gt Libraries U se M ulti threaded Library E nable Reentrancy Support 8 Choose default external procedures calling interface Project gt Settings ISettings For All Configurations Fortran gt E xternal Proceduresl Default Calling C By Reference 3 3 Linking 3 3 1 Linking on Unix The following string give the setup for necessary link flags BASH syntax L opt scali lib W1 R opt scali lib CRT_BEGIN lmpi CRT_END Since it is required to link with the GNU runtime library the syntax is dependent on the compiler you use Please check therelease notes The runtime setup CRT_BEGIN and CRT_END libraries are defined for some compilers R is a flag asking the linker to include the path to the dynamic libraries so that the environment variable LD_LIBRARY_PATH is not needed see table 3 1 When linki
23. enn 42 LA ZP notando 42 A A e 42 4 4 2 3 Eagerb uffer iii 42 44 24 E ee e EE 42 4 4 2 5 SN A 43 4 5 Communication Droe 43 TO 43 4 5 2 E Ager buenO a ao od 44 BC RENE O 45 Chapter 5 Getting help uuusannenonnunnnnunnunnnnnonnunnnnunnunnnnnnnnnnnnnunnnnannnnnnnnnnnnnannnnnnnnnnn 47 5 1 Application nokes un ea area 47 5 1 1 MPI_Probe and MPI Be 47 5 1 2 Unsafe MPI programs ine aariin 48 5 2 Namespace pollution siseasi ieie air seasea eeii ia aia 48 SB EERO MESSAGES min a a e adapa arai a ais ahi 49 KEREN 49 E PA e EARS EE 49 5 4 Trouble Shooting a helle A a 50 Chapter 6 Support cnc REE 55 A u ke en uno hin a bir 55 6 2 Problem report Sa rin lonas aia 55 PER AEO Ana 2 2 aa an ae ee AE 55 ScaMPI User s Guide Version 1 9 0 4 Chapter 7 Related Documentation uussnnuonnnnonnunnnnunnunnnnnunnunnnnunnnnnnnnnnnnnannnnnnnnnnn 57 KR AA ET 57 Appendix A Install SCAMPI uanuuuonnunnnuonnunnnnunnnununnunnnuunnunannunnnnnannennunnnnnannnnannnn anna 59 ER Late ln Le EE 59 As1 REQUIFEMENtS nicas inneres 59 A 1 2 Distribution UR 59 A 1 3 Licensing in soon ia 60 A 1 4 Removing an earlier release Of GcaMbl A 61 A 1 5 Installing a new release 61 A 1 6 Verification of installation A 61 A 2 Scali packages file system Lovout EN 62 A 3 Useful 3rd party parallel software 62 ScaMPI User s Guide Version 1 9 0 ScaMPI User s Guide Version 1 9 0 Chapter 1 Introduction A Scali system is a cl
24. ense dat contains the host name and port number on the SERVER line as e g SERVER scali front end ANY 7788 Then install the ScaL M node package After an installation on the entire cluster with input on demand 7788 scali front end each node has a local opt scali etc ScaLM licserver file containing a reference to the license server ScaLM licserver reads LM_LICENSE_FILE portGhost export LM_LICENSE_FILE with 7788 scali front end as portGhost setting in this example ScaMPI User s Guide Version 1 9 0 60 More detailed information about end user license administration can be found with the ScaLM package in the online HTML version of the FLEXIm End User Manual To view this online documentation use your web browser to open the file file opt scali license doc html man index html FLEXImis a trademark of Globetrotter Software Inc A 1 4 Removing an earlier release of ScaMPI Only a single release of ScaMPI can be installed at a system If an earlier release of ScaM PI already is installed at a system this release must be removed before a new release can be installed Solaris pkgrm ScaMPI Linux rpm e ScaMPI WinNT Use Windows uninstall to remove the old ScaMPI release Start Settings Control Panel Add Remove Programso o ScaM PI o Add Remove All files and mpid deamon processes will be removed A 1 5 Installing a new release Only asinglerelease of this software can be installed on a system If an earlie
25. esktop GUI 4 It is also possible that ScaMPI programs are started from a 3rd party software workload management system on top of mpimon 6 16 3 4 1 Mpimon mpimon have many options which can be used for optimising ScaM Pl performance Normally it should not be necessary to use any of these However unsafe MPI programs 3 might need buffer adjustments to get rid of hangs it is of course not a recommended method of programming Trading performance by changing communication space is best avoided if there are no compelling reason to do so and you know what to do 3 4 1 1 Basic usage mpimon lt user program gt lt program options gt lt hostname gt lt count gt lt hostname gt lt count gt will be normal use Option Description lt userprogram gt Name of application program lt programoptions gt Program options to application program Separator signals end of user program options lt hostname gt lt count gt Pair of name of host and number of processes to run on that host Hosts can occur several times in the list Processes will be given ranks sequentially according the list of host number pairs Table 3 3 Basic options to mpimon 3 4 1 2 Advanced usage mpi mon lt mpimon option gt program amp host spec gt program amp host spec gt ScaMPI User s Guide Version 1 9 0 20 3 4 Running MPI programs is the complete syntax for using mpimon Parameter
26. following figure 3 3 and 3 4 ScaMPI User s Guide Version 1 9 0 29 3 6 Profiling ScaMPI applications VAMPIR 28 359 K 28 359 K 26 797 K 28 359 K 26 797 K 28 359 K 26 992 K 26 992 K Figure 3 3 VAMPIR snapshots ScaMPI User s Guide Version 1 9 0 30 3 6 Profiling ScaMPI applications Figure 3 4 VAMPIR profiled MPI program call tree window VAMPIR user environment setup To use VAMPIRtrace a user must have access to the VAMPIRtrace directories and the environment variables for the license scheme must be set correctly Most likely the VAMPIR environment is setup automatically by your system administration If not the license keys are stored in a plain ASCII file the pathname of which must be made known to VAMPIRtrace by setting one of the two environment variables in table 3 8 Name Description PAL_ROOT points to the root of the VAMPIRtrace installation PAL_LICENSEFILE specifies the complete pathname of the license key file Table 3 8 Setup for VAMPIR and VAMPIRtrace Create a VAMPIR trace file Link your MPI program with VAMPIRtrace by adding the VT library before your ScaM PI libraries on the link line BASH syntax LSVAMPIR_LIB_DIR lfmpi 1VT lmpi ScaMPI User s Guide Version 1 9 0 31 3 6 Profiling ScaMPI applications This example shows the placement of the VAMPI Rtrace library for a fortran program remove the reference to the fortran interface library libfmpi when linking a C main
27. g jumpshot upshot or nupshot e Trace or real time animation of MPI calls e A shared display parallel X graphics library Tolink with the MPE libraries include one of the following libraries before the Impi library e Itmpi ScaMPI User s Guide Version 1 9 0 32 3 6 Profiling ScaMPI applications Trace all MPI calls Each MPI call is preceded by a line that contains the rank in MPI_COMM_WORLD of the calling process and followed by another line indicating that the call has completed Most send and receive routines also indicate the values of count tag and partner destination for sends source for receives Output is to standard output e IImpi Im Generate an upshot style log file of all MPI calls The name of the output file is executablename_profile log For example if the program is sendrecv the generated log file is sendrecv_profile log e lampi Impe Im 1X11 Produces a real time animation of the program This requires the MPE graphics and uses X11 Window System operations Y ou may need to provide a specific path for the X11 libraries instead of IX11 In Fortran it is necessary to include the library Ifmpi ahead of the profiling libraries a part of ScaMPI This allows C routines to be used for implementing the profiling libraries for use by both C and Fortran programs F or example to generate files in a Fortran program the library list is Ifmpi IImpi Jm Analyse the program performance with U
28. h chunk and the number of chunks in each pool can be set by options inter_ intra_pool_size and inter intra_pool_count 4 5 Communication protocols ScaMPI employs different protocols for communication depending on thesize of the message Increasing message size Transporter Eager Buffering Inlining Figure 4 7 Thresholds for different communication protocols 4 5 1 Inlining Inlining means including data in the header and is used for small messages lt inter_ intra_channel_inlining_theshold bytes Sender Receiver Figure 4 8 Inlining ScaMPI User s Guide Version 1 9 0 43 4 5 Communication protocols 4 5 2 Eagerbuffering E agerbuffering uses a scheme where buffers are allocated by the sender of a message and are released by the receiver without any explicit communication between the two ele N TOTO Figure 4 9 Eagerbuffering ScaMPI User s Guide Version 1 9 0 44 4 5 Communication protocols 4 5 3 Transporter For large messages we use a rendez vous communication protocol called transporter The sender only sends a header phase 1 and when the receiver is ready to accept the data it communicates this to the sender phase 2 Data is then transferred using dedicated buffers phase 3 Phase 1 Phase 2 Transporter selection field BA Figur
29. icensing system FlexLM examples Example applications source and documentation contrib 3rd party software adapted to Scali software init d boot startup scripts are installed here Soft links are made to system startup catalogues rc d and init d under etc A 3 Useful 3rd party parallel software It is possible to integrate any kind of parallel MPI software with ScaMPI Contact sales scali com for latest information on availability or how to get your parallel software option Other good places for information about 3rd party parallel software are 12 13 14 15 and 16 ScaMPI User s Guide Version 1 9 0 62 List of figures 3 1 TotalView main window eersnensennnnnernnnnnnnnnnennnnnnnnn nn rr 26 3 2 TotalView MPI process window uenseensnnersnnnnnnnnnnennnnnnnnnnnen nennen nennen 27 3 3 VAMPIR snapshots s a ne ee a 30 3 4 VAMPIR profiled MPI program call tree window uneerseennsennnnennennnnnn 3l 4 1 Startup phase een dena na erlebe 36 4 2 Start p PHASE 2 iii aida baaa dida 37 4 3 Startup PNAS 3r dida reiia e i a dd 38 A RE 40 45 Ge EE 41 4 6 rans porte ernennen 42 4 7 Thresholds for different communication Droe 43 4 8 NINN OO 43 4 9 Eager b ffering iaa eier ee 44 4 10 ZTEL nSsp rter AT 45 ScaMPI User s Guide Version 1 9 0 63 ScaMPI User s Guide Version 1 9 0 64 List of tables 1 1 1 2 1 3 3 2 3 3 3 5 3 6 3 8 4 1 4 3 4 4 4 5 5 1 5 2 ell iaa il 8
30. ing SCAMPI uuuuannunonnunnnuunnunnnnuunnununnunnnunnnnunnnnannunnnnannnnnnnnnnnnnannnnnnnnnnn 15 3 1 Setting up a ScaMPI environment NA 15 3 1 1 The Unix ScaMPI enwviromment E 16 3 1 2 The Unix on Windows NT ScaMPI environment sssseeersesenen 17 SPAS ue AU DE 17 E e ue Te NEE 17 3 2 2 U NIX e ul E Ee CG 18 3 2 3 Windows NT compile Tags 18 CN nl Le aiii dao oriol 19 A OF Unix aaa Ense 19 3 3 2 Linking on Windows NT AEN 19 3 4 R nning MPI POS snie aaia i 20 SR Olli WEE 20 3 4 1 1 BASIC Te aan rar 20 3 4 1 2 Advanced USAGE enean anon aaaea ea Ea aAA Ea aTh 20 3 5 Debugging ScaMPI appltcattons En 25 3 5 1 Debugging with TotalView E 25 3 5 2 Deb ugging with GAD u arena nad 28 3 6 Profiling SCAM PI applications NA 29 ScaMPI User s Guide Version 1 9 0 3 3 6 1 Profiling with Vampir 29 3 6 2 Profiling with the MPE Ibrarw nn 32 Chapter 4 Description of SCAMPI uunnuesnnuunnununnunnnnunnnnunnunnnunnnnunnunnnnunnnnnunnnnnnnnnnn 35 4 1 General description seen bon 35 ALA A DN A ahnen 35 n R RPA PIMOM EE 35 41 3 M PISUDMON anne ea a nern 35 GE Wert 35 ALO M rte een eb nenne 35 4 2 Start OO 35 APD O A A E ee herren 36 4 2 2 Phase NAO 37 HE OOO 38 ASCO EE 39 4 4 Communication FESOUFCES EEN 40 4 4 1 E lge un ana ren ee 40 4 4 O eisen sr ae gees 40 4 4 1 2 Eagerb ffer u arena 41 Neie e EE 42 4 4 2 Parameters MpiMON options uunseeeeersseenennnnneennnennnnnnnnnnnnnnnnnnnsnnnn n
31. integer rank size ierr call mpi_init ierr call mpi_comm_rank MPI_COMM_WORLD rank ierr call mpi_comm_size MPI_COMM WORLD size ierr write A 13 A 13 Hello I m rank rank Size is size call mpi_finalize ierr end 2 1 4 Compiling Unix WinNT gcc c D_REENTRANT ISMPI_HOME include hello c 977 c D_REENTRANT I MPI_HOME include hello f T cl c D_REENTRANT D_CONSOLE DWIN32 DWinNT MT ISMPI_HOME include hello c 77 unix reentrancy threaded threads D_REENTRANT c nologo DWin32 DWinNT iface cref iface nomixed_str_len_arg I MPI_HOME include hello f ScaMPI User s Guide Version 1 9 0 12 2 2 MPI test programs 2 1 5 Linking Unix gcc hello o L SMPI_HOME lib W1 R opt scali lib SMPI_LDLIBS o hello 917 hello o L MPI_HOME lib W1 R opt scali lib MPI_LDLIBS o hello WinNT cl nologo MT hello obj o hello link LIBPATH MPI_HOME lib MPI_LDLIBS 77 unix nologo reentrancy threaded threads hello obj Fehello exe link LIBPATH MPI_HOME lib 2 1 6 Running Run the hello world example on 3 Unix nodes named hostA hostB hostC or a Windows NT node with name hostN Unix mpimon hello hostA 1 hostB 1 hostc 1 WinNT mpimon hello exe hostN 3 The hello example will produce the following output Hello I m rank 0 Size is 3 Hello I m rank 2 Size is 3 Hello I m rank 1 Size is 3 2 2 MPI test programs TheScaMPlts
32. is guide This guide is written for users which have a basic understanding of MPI 1 2 3 It also assumes basic knowledge of C programming The guide is not a tutorial in using MPI nor a guide how to write efficient MPI programs ScaMPI User s Guide Version 1 9 0 7 1 4 Acronyms and abbreviations 1 4 Acronyms and abbreviations Abbreviation Meaning SCI Scalable Coherent Interface MPI Message Passing Interface SSP Scali Software Platform is the generic name of all Scali software packages Table 1 1 Abbreviations 1 5 Terms and conventions In all examples default are gcc GNU C compiler and bash GNU Bourne Again SH ell Term Description Node A single computer in an interconnected system of one or more computers Cluster A duster is a set of interconnected nodes with the possibility to act as a sin gleunit Scali system A Scali HPC system Scali software compute nodes high speed inter connect Process Instance of application program with unique rank within MPI_COMM_WORLD Host A single node of a Scali system i e a multiprocessor workstation or PC Unix Refers to all UNIX and lookalike OSes supported by the SSP e g Solaris and Linux WinNT Refers to Microsoft Windows NT ScaMPI User s Guide Version 1 9 0 Table 1 2 Basic terms oo 1 6 Typographic conventions 1 6 Typographic conventions Term Description Bold Program names options
33. l c_info c_info gt r_buffer c_info gt rank size 0 c_info gt num_procs size 1 put n_sample i 2 amp defect 3 t2 MPI_ktimet r time it2 t1 gt n_sample 5 else Action Points line 105 in Alltoall 0x11c in _signotifywait 1 line 221 in main 0x354 in _swtch in _libc_read in _lwp_sema_post Figure 3 2 TotalView MPI process window TotalView user environment setup TotalView users must have a few environment variables defined seetable 3 7 giving the locations of binaries man pages libraries and licenses This environment should be set automatically by your system administration If you need more information check the TotalView Installation Guide 8 ScaMPI User s Guide Version 1 9 0 27 3 5 Debugging ScaMPI applications Name Description PATH points to directory wherethe TotalView binaries are installed e g opt totalview bin MANPATH points to directory wherethe TotalView man pages are installed e g opt totalview man LD_LIBRARY_PATH points to directory wherethe TotalView libraries are placed e g opt total view lib LM_LICENSE_FILE a separated list which points to the TotalView license file e g opt total view license dat Table 3 7 Environment setup for TotalView Run a ScaMPI program from within the TotalView debugger First of all compile your programs with the g flag to include debug symbols in the executables Depending on your compilie
34. l gt lt node2 gt lt nodeN gt 2 2 2 A bandwidth MPI test program Bandwidth is a program to measure bandwidth for various messages sizes between two processes First one way bandwidth and the latency for a zero byte message are measured then the ping pong two way bandwidth and latency Measure the bandwidth between any pair of nodes nodeX and nodeY by running mpimon bandwidth nodeX nodeY 2 2 3 A bidirectional MPI test program Bidirect tests uni and bi directional traffic between a given number of nodes The program may be run between two nodes nodeX and noer as run_bidirect nodeX nodeY or between a set of given nodes nodeX nodeY nodeZ as run_permutated_bidirect nodeX nodeY nodez Therun_permutated_bidirect script will test uni and bi directional trafficbetween all permutations of node combinations ScaMPI User s Guide Version 1 9 0 14 Chapter 3 Using ScaMPI This chapter describes the setup compile link run debug and profile of an MPI program using ScaMPI The control and start up of any MPI program using ScaMPI is monitored by mpimon Running ScaMPI on a Scali cluster with multi user and resource management software implies that the startup mechanism is built on top of mpimon Only mpimon is described in this document We advocate use of state of the art cluster management software it is a user friendly and effective way of utilising a Scali cluster Seethe documentation at your com
35. l to the entire cluster A 1 2 Distribution file ScaM PI is distributed as a single package file named ScaMP 1 os arch x y z package ScaMPI User s Guide Version 1 9 0 59 where X Y Z is the release number e g 1 0 2 os isthe operating system eg SunOS5 Linux2 or WinNT4 arch is the architecture e g sparc u or 86pc package is e g pkg rpm or exe depending on the operating system A 1 3 Licensing ScaM PI is licensed using the FLE XIm license manager system Y ou will need to install the license server software and obtain a valid demo or permanent license in order to run ScaMPI If you have not done so already you must also acquire the Scali license server package ScaLM This package contains the Scali vendor daemon FLE XIm end user utilities and the necessary information to install and run the license server Note that you will need this package to get the Scali vendor daemon scald even though you might have a working v5 or newer version of FLEXIm installed on your system ScaLM can be downloaded from the Scali Web site at http www scali com Requests for permanent or demo licenses can be made using the E mail address sales scali com For technical questions use support scali com Install the ScaLM license manager package preferably on a front end of the Scali cluster Add your license dat file to opt scali license on the chosen license server This file holds the licenses for all Scali software Lic
36. library but it is possible to use MPE a freely available MPI profiling library available from Scali as the ScaMPE package The performance trace libraries are linked with ScaMPI libraries through a profiling interface defined in the MPI standard Performance data is output to a file when the instrumented program reaches MPI_Finalize The VAMPIR softwareis currently only supported with ScaMPI on Solaris but can be made available on other platforms Contact sales scali com for more information 3 6 1 Profiling with Vampir This section contains a brief description of VAM PIR how to setup the user environment and basic instructions about how to collect performance data and how to analyse the performance Refer to the VAMPIR User s Guide documentation for VAM PIR specific explanations What is VAMPIR e Vampir MPI performance analysis help you organise the performance data understand application and communication behaviour evaluate load balancing and identify communication hots pots A timeline window display application and message passing activities and shows parallelism as the sum of participating processes Communication statistics can be displayed for selected intervals of time and message length e Execution statistics for selected call trees can be displayed and comparisons made on data from multiple runs Other features include source code display and animation A few of the common global information displays are shown in the
37. mon without any options to get thelist of all mpimon options with a short explanation Or check the mpimon description in chapter 4 How do control SCI and local shared memory usage simple adjusting ScaMPI buffer sizes Forcing size parameters to mpimon is usually not neccessary This is only a means of optimising ScaM PI to a particular application on knowledge of communication patterns F or unsafe MPI programs it may be required to adjust buffering to allow the program complete Table 5 2 Trouble shooting ScaMPI User s Guide Version 1 9 0 5 4 Trouble shooting Problem Description Solution How do control SCI and local shared memory usage advanced one communciation channe The eager buffers are used for small messages while the trans porter buffers are used for handling large messages larger than eager size Thefunction of the various buffers is detailed in the ScaM PI user s guide All buffers are created at start up when init_comm_world is specified or when needed tried used for the first time The channel buffers is a send queue where each entry is 64 bytes i e there is room for 64 outstanding requests in a Ak buffer The bufferspace required by a communication channel is approxi matey channel 2 channel_size communicators transporter_size transporter_count eager_size eager_count 512 How do control SCI and local shared memory usage advanc
38. ng a Fortran main program include the F ortran interface library Ifmpi before CRT_BEGIN 3 3 2 Linking on Windows NT Linking on a Windows NT GUI is explained in detail and continues the description of compiling on WinNT section 3 2 2 Check the release notes if you need additional information Linking both for fortran and c c source 9 Add ScaMPI library path and ScaM PI libraries Project gt Settings Settings For All Configurations Link gt nput JObject library modules ScaM PI lib ScaBaselib ScaCom lib ScaSci lib Additional library path c opt scali lib 10 It is recommended to enable the startup banner for compiling and linking Project gt Settingsl Settings F or All Configurations Link gt Customize Suppress startup banner ScaMPI User s Guide Version 1 9 0 19 3 4 Running MPI programs Project gt Settings ISettings For All Configurations C C gt Customize Suppress startup banner You should now be able to build your project 3 4 Running MPI programs Executables containing ScaM PI calls cannot be started directly from a shell prompt They must be started using mpimon the control startup program supplied with a ScaM PI distribution Remember it is absolutely necessary to run an ScaMPI program with a homogene file system image providing the same path and program name on all nodes of the cluster Mpimon can be started from a shell prompt or the Scali D
39. ort to shut down all MPI processes ScaMPI User s Guide Version 1 9 0 49 5 4 Trouble shooting Problem How do start my program use mpimon from shell prompt use mpimon from Scali Desktop GU 5 4 Trouble shooting Description Solution See Getting started in chapter 2 for practical examples If your program is reading from stdin remember to use the mpimon option stdin O to send the input to at least one MPI process Default is stdin none It is also possible to use the Scali Desktop GUI opt scali bin scadesktop amp Why doesn t my program start to run simple set the application and library paths mpimon command not found FIX Indude opt scali bin in you PATH environment variable mpimon can t find mpisubmon FIX Set MPI_HOME opt scali or use the execpath option The application has problems loading libraries libsca FIX 1 Update the LD_LIBRARY_PATH to include opt scali lib FIX 2 Link your application with path to Scali libraries gnu WI R opt scali lib Why doesn t my program start to run advanced interconnect problems routing The program hangs in MPI_Init and terminatein ICMS_NO_RESPONSE CHECK Check if routing is properly set with scaconftool It is easiest to use the Scali Desktop GUI Why doesn t my program start to run advanced interconnect problems bad clean up A previous ScaMPI run has not been properly terminated
40. pc WinNT ScaMPI is supported for use with following compilers MS Visual C Digital Visual Fortran for i86pc 3 2 2 Unix compile flags The following string must be included as compile flags BASH syntax D_REENTRANT I MPI_HOME include 3 2 3 Windows NT compile flags WinNT Building a ScaMPI application on Windows NT using a GUI is described in detail since this approach is very different from an ordinary compile session on unix Please check the release notes if you need additional information Microsoft Visual C Digital Fortran 1 Start MS Visual C or Digital Visual Fortran 2 Choose a project name and a project location It is recommended that the program location is at a shared file system availablefrom all the nodes wherethe program shall be run File gt N ew gt Project 1 1 Win32 Console Application project name Jlocation 3 Add c c or fortran files to the project Project gt Add To Project gt F iles Settings for compiling c or c code skip if only fortran source 4 Add a preprocessor definition and location of additional include files Project gt Settings ISettings F or All Configurations C C gt Preprocessor Preprocessor Definitions WinNT JAdditional include directories opt scali include 5 Select a multithreaded runtime library for debug or release code Project gt Settings Settings For Win32 Debug C C gt Code Generation Use Runtime Library Debug Multithread
41. priate mailing group on http www scali com 6 2 Problem reports Problem reports should include software version computer architecture an example and a record of the sequence of events causing the problem 6 3 Platforms ScaM PI is available on Scali SCI clustered Intel PC s with either Solaris Linux or Windows NT ScaMPI is available on Scali SCI clustered Sun UltraSPARC workstations with Solaris For further information contact sales scali com or visit Scali s website http www scali com ScaMPI User s Guide Version 1 9 0 55 6 3 Platforms ScaMPI User s Guide Version 1 9 0 56 Chapter 7 Related Documentation 7 1 References 1 MPI A Message Passing Interface Standard The Message Passing Interface Forum Version 1 1 June 12 1995 http www mpi forum org 2 MPI The complete Reference Volume 1 The MPI Core Marc Snir Steve W Otto Steven Huss Lederman David W Walker Jack Dongarra 2e 1998 The MIT Press 3 MPI The complete Reference Volume 2 The MPI Extension William Grop Steven Huss Lederman Ewing Lusk Bill Nitzberg William Saphir Marc Snir 1998 The MIT Press 4 Scali System Guide Scali AS http www scali com 5 ScaMPI release notes Scali AS http www scali com 6 CCS Computing Center Software resource management for networked high perform ance computers Paderborn Center of Parallel Computing http www uni paderborn de pc2 7 Tot
42. puter lab about cluster management Contents summary Section 3 1 1 explains how tosetup the ScaMPI environment in a BASH shell for Unix or Unix on Windows NT Compile and link instructions are given respectively in section 3 2 and 3 3 How torun a ScaMPI program with mpimon is explained in section 3 4 which contains a list of all mpi mon program parameters Debugging ScaMPI programs is explained in section 3 5 and performance profiling of ScaMPI programs is explained in section 3 6 3 1 Setting up a ScaMPI environment System administration propably has set up your ScaMPI shell environment in startup scripts Environment variables point tothe ScaMPI installation directory theScaMPl executables must be located in your path and ScaMPI libraries and dynamic link library paths has to be defined If you have a proper ScaMPI environment setup you may safely skip section 3 1 ScaMPI User s Guide Version 1 9 0 15 3 1 Setting up aScaMPI environment 3 1 1 The Unix ScaMPI environment The use of ScaM PI requires that some environment variables are defined These are usually set in the standard startup scrips e g bashrc when using BASH gnu Borne Again SHell but they can also be defined manually Name Description MPI_HOME Installation directory For a standard installation this should be set as export MPI_HOME opt scali MPI_LDLIBS Library to be loaded defined as Id directives For a standard installation the necessary libraries
43. r s Guide Version 1 9 0 3 4 Running MPI programs Mpimon option Description intra_transporter_count lt count gt Set number of buffers for transporter intra protocol Default 4 Legal Powers of 2 intra_transporter_size lt size gt Set buffer size in bytes for transporter intra protocol Default manual lt sd ection gt Set manual mode for process es Default none Legal n m o list or n m range or all separate_output lt sdection gt Enable separate output for process es F ilename ScaM Pl output_host_pid_rank Default none Legal n m o list or n m range or all sm_debug lt se ection gt Set debug mode for submonitor s Default none Legal n m o list or n m range or all sm_manual lt se ection gt Set manual mode for submonitor s Default none Legal n m o list or n m range or all sm_trace lt se ection gt Enable trace for submonitor es Default none Legal n m o list or n m range or all statistics Enable statistics stdin lt sdection gt Distribute standard in to process es Default none Legal n m o list or n m range or all verbose Display values for user options timeout lt timeout gt Set timeout elapsed time i seconds for run Legal
44. r it may be necessary to specify what kind of debug symbols to be generated see the ScaMPI release notes Use tvmpimon lt mpimon options gt tostart an MPI program with ScaMPI in a single TotalView debugger window See the TotalView User s Guide 7 for help on debugging with TotalView TotalView ask if you want to stop in MPI_Init A stop give you a single point of control of the debugging session Use the middle mouse button to get a menu of possible commands The point and click interface of TotalView should get you easily started Current TotalView support At the moment TotalView is only supported on Solaris The message queue feature of TotalView with MPI is not yet implemented and there may be some restrictions on debugging threads See the TotalView release notes and the ScaM PI release notes for further details 3 5 2 Debugging with gdb The default mpimon debugger is the GNU debugger gdb Specify the processes you want to debug with the debug option to mpimon Cut the xterm displayed run parameters and paste them into gdb when you want to start the debugged process ScaMPI User s Guide Version 1 9 0 28 3 6 Profiling ScaMPI applications 3 6 Profiling ScaMPI applications For MPI programs the most useful tool beside a parallel debugger is a MPI performance tracing tool showing the message passing performance of a program run There are two MPI profiling options with ScaMPI Recommended is VAMPIR a commercially supported
45. r release is already installed on the system please check the remove section A 1 4 before continuing Solaris pkgadd d ScaMPl os arch x y z pkg ScaMPI Linux rpm ivh ScaMPI os arch x y z rpm WinNT Use Windows install to locate the ScaMPI release or double click on the ScaMPI executable Start Settings Control Panel Add Remove Programso o Install or ScaMPI os arch x y z exe Normally the ScaMPI package will be installed in opt scali WinNT c opt scali A 1 6 Verification of installation After an attempted installation of ScaMPI you may check the installation Solaris pkginfo ScaMPI Linux rpm qi ScaMPI WinNT Check if ScaMPI is listed among the installed Windows programs Start Settings Control Panel Add Remove Programso o ScaMPI 2 ScaMPI User s Guide Version 1 9 0 61 A 2 Scali packages file system layout Almost all Scali software package files are installed under a common directory opt scali Under this directory files are organised as follows bin All normal executables to be run by users sbin All daemons and executables to berun by administrators libexec Executables which are hidden from normal invokation e g used by applications under bin lib Libraries used by our application and by end users include Incude files for libraries under lib doc All documentation etc Configuration files for our software license Files belonging to a 3rd party l
46. refer tothe detailed compile link and run instructions in chapter 3 to overcome any difficulties 2 1 The first example Example and test MPI programs are located under opt scali examples src and bin directories Asa first example you may use the hello world MPI program in hello c or the fortran version hello f Complete step by step instructions for a compile link and run of hello c and hello f with ScaMPI are included below 2 1 1 Setting up your BASH environment Set MPI_HOME to point to opt scali the installation directory of ScaMPI and set MPI_LDLIBS to include all the required ScaMPI link libraries Also put the ScaM PI executables in your path Unix MPI_HOME opt scali export MPI_HOME MPI_LDLIBS Impi export MPI_LDLIBS WinNT MPI_HOME c opt scali export MPI_HOME MPI_LDLIBS scampi lib scabase lib scacom lib scasci lib export MPI_LDLIBS PATH PATH MPI_HOME bin export PATH ScaMPI User s Guide Version 1 9 0 11 2 1 The first example 2 1 2 C source of a Hello world MPI program include lt stdio h gt include mpi h void main int argc char argv int rank int size MPI_Init amp argc amp argv MPI_Comm_rank MPI_COMM_ WORLD amp rank MPI_Comm_size MPI_COMM_WORLD amp size printf Hello I m rank d Size is d n rank size MPI_Finalize 2 1 3 Fortran source of a Hello world MPI program program hello implicit none include mpif h
47. t Set chunk size for inter communication Default 512k Legal Multiplum of pages inter_eager_count lt count gt Set number of buffers for eager inter protocol Default 2 inter_eager_size lt size gt Set buffer size in bytes for eager inter protocol Default 128K Legal Powers of 2 inter_pool_size lt size gt Set buffer pool size for inter communi cation Default 32M Legal Multiplum of pages inter_transporter_count lt count gt Set number of buffers for transporter inter protocol Default 4 Legal Powers of 2 inter_transporter_size lt size gt Set buffer size in bytes for transporter inter protocol Default intra_channel_inline_threshold lt size gt Set threshold for inlining in bytes per intra channel Default 560 intra_channel_size lt size gt Set buffer size in bytes per intra channel intra_chunk_size lt size gt Default 4K Legal Powers of 2 Set chunk size for intra communication Default 1M Legal Multiplum of pages intra_eager_count lt count gt Set number of buffers for eager intra protocol Default 2 intra_eager_size lt size gt Set buffer size in bytes for eager intra protocol Default 128K intra_pool_size lt size gt Legal Powers of 2 Set buffer pool size for intra communication Default 4M Legal Multiplum of pages Table 3 6 Complete list of mpimon options ScaMPI Use
48. t boots and must run at all times 4 2 Startup ScaM PI uses sockets for control purposes Schematically a startup of a mpi cluster is done as follows ScaMPI User s Guide Version 1 9 0 35 4 2 1 Phase 1 4 2 Startup Step Description Parameter control mpimon does as much control of options and parameters as possible user program names are checked to be ok and hosts are contacted through sock ets to see if they are alive and that mpid is running Connecting to mpids mpii mon connects to daemons on all hosts and transfers basic information to enable mpid to start submonitor Table 4 2 Startup phase 1 Figure 4 1 Startup phase 1 ScaMPI User s Guide Version 1 9 0 36 4 2 Startup 4 2 2 Phase 2 Description mpisubmon is started by mpid and connects to mpimon for a control socket information to enable mpisubmon to start userprograms proc esses are transferred mpisubmon creates shared memory areas Submonitors starts Table 4 3 Startup phase 2 mpi t S Hos Figure 4 2 Startup phase 2 ScaMPI User s Guide Version 1 9 0 37 4 2 3 Phase 3 4 2 Startup Step Description Processes start and enter MPI_Init mpisubmon starts all processes to be run on the host where it is running Processes synchronize When the processes have received all control information they will signal mpi mon that they are ready to run mpimon will then send a star
49. t package contains some MPI test programs If ScaM PI tst is installed on your system you have MPI test sources and binaries under the opt scali examples src and bin directories These programs are given a short presentation here together with simplerun instructions It is of course possible to use the included makefiles to compile the MPI test sources No details are mentioned sincethe compile link and run process are already described in the first example hello world program 2 2 1 A producer consumer MPI test program Producer is a simple producer consumer program Processes with rank 0 1 2 N 2 1 send data while process nii n 2 1 n 1 receive data Process 0 will send to process n 1 and process 1 will send to process n 2 and so on The producer program parameters are li i is the loop count ScaMPI User s Guide Version 1 9 0 13 2 2 MPI test programs nj ji sthenumber of bytes to transfer for each send operation As a first test run producer between any pair of two nodes nodeX and nodeY mpimon producer 1 1 n 1024 nodeX nodeY A single process is started on each node and a single message of size 1024 bytes are transferred from the process on nodeX to the process on nodeY The program should return TEST COMPLETE Repeat the test for all pairs of nodes N is the number of hosts must be a even number for this test The program should return TEST COMPLETE mpimon producer 1 1 n 1024 lt node
50. t run ning messageto all processes Processes return from MPI_Init and start run ning User program takes control Table 4 4 Startup phase 3 mpimon Figure 4 3 Startup phase 3 ScaMPI User s Guide Version 1 9 0 38 4 3 Stopping 4 3 Stopping Step Description Processes enter Processes signal to mpimon that they have entered finalize MPI_Finalize Processes synchronize Processes wait for an all stopped message from mpimon sent when all processes are in finalize Processes leave Processes terminate and submonitors release shared memory segments MPI_Finalize and exit mpimon terminates Table 4 5 Stopping steps ScaMPI User s Guide Version 1 9 0 39 4 4 Communication resources 4 4 Communication resources ScaMPI uses a on demand scheme for allocating resources This means that resources will be allocated when needed All resources reside in shared memory and are allocated by mpisubmon on demand from the sender process 4 4 1 Resources 4 4 1 1 Channel A channel is a unidirectional connection between a sender and a receiver There will be onechannel per sender receiver pair for each communicator Each ringbuffer entry is 64 byte and contains the message envelope Sender Receiver Ringbutfer Figure 4 4 Channel ScaMPI User s Guide Version 1 9 0 40 4 4 Communication resources 4 4 1 2 Eagerbuffer E agerbuffer is a buffered
51. tions Refer to the TotalView User s Guide documentation for TotalView specific explanations What is TotalView TotalView is the recommended parallel debugger on a Scali system running ScaM PI e The TotalView graphical parallel debugger has support for the most important parallel programming abstractions threads MPI PVM and HPF e TotalView has a fast and easy to learn GUI with the possibility of data visualisation a useful aid in debugging numerical programs ScaMPI User s Guide Version 1 9 0 25 3 5 Debugging ScaMPI applications e TotalView is designed for multiprocessing and offer the debugger features a programmer expects 3218 R PMB MPI1 6 threads Figure 3 1 TotalView main window ScaMPI User s Guide Version 1 9 0 26 3 5 Debugging ScaMPI applications PMB MPI1 S 154 Aa Function main argc 0x00000001 lt 1 argv 0x0003ac78 gt Oxeffffclo gt Local variables LC INFO Compound Object BList 0x0003f258 gt Compound Obj Bmark 0x0003f258 gt Compound Obj NFAIL 0x00000000 lt 0 NSUCCESS Ox00000000 lt 0 Pi 0x0003d880 gt PingPong WIDIIDIIIIIDIDDIIIIIIN Function gt gt Alltoall in Alltoall c MIRADO 10 t1 MPI_litimet gt s forti 0 i lt n_sampleti ierr MPI_Alltoallic_info gt s_buffer s_num c_info gt s_data_type c_info gt r_buffer r_num c_info gt r_data_type c_info gt communicator MPI_ERRHAND ierr gt CHK_DIFF lt Alltoal
52. uster of SCI interconnected nodes where each node is a multiprocessor workstation or PC running either Solaris Linux or Windows NT To get the full computational power of a Scali system it is necessary to use the message passing library ScaMPI ScaMPl utilises shared memory on intra node communication and the fast SCI interconnect on inter node communication Any parallel MPI program can be run with ScaMPI and benefit from the SCI performance This document describes in detail how to use ScaMPI 1 1 Purpose of the ScaMPI User s guide This document describes the Scali implementation of MPI ScaMPI version 1 1 Its purpose is twofold e tosupply the user with enough information to use ScaM PI e togive the interested reader an overview of the ScaMPI implementation 1 2 Scope of the ScaMPI User s guide This document has the following layout Chapter 2 explains how to get started running your first MPI program with ScaMPI Chapter 3 describes how to compile link run debug and profile ScaMPI programs Chapter 4 describes internal design and functionality of ScaM PI Chapter 5 explains what to do if you have problems and gives a list of common errors and solutions Chapter 6 explains what to do if you need assistance from Scali Chapter 7 gives a list of related documentation you may consult for additional information Appendix A explains how to install ScaMPI The normal user can safely ignore this appendix 1 3 Who should read th
53. v buf cnt dtype MPI_ANY_SOURCE MPI_ANY TAG comm sts doStuff This sequence works ok for implementations that have one receive queue for all senders ScaMPI have one receive queue per sender This gives messages from different senders the possibility to bypass each other The problem with the sequence ScaMPI User s Guide Version 1 9 0 47 5 2 Namespace pollution above is that a sent message could arrive in thetimegap between the probe finishes and the receives matches a message i e it is not certain that the message that probe matched is the same as the one that recv matches The correct sequence should be 1 while MPI_Probe MPI_ANY_SOURCE MPI_ANY_TAG comm sts if sts gt MPI_TAG SOME_VALUE MPI_Recv buf cnt dtype sts gt MPI_SOURCE sts gt MPI_TAG comm sts doStuff 5 1 2 Unsafe MPI programs Some programs may run with MPICH and not ScaMPI because of different buffering behaviour Unsafe MPI programs may require resources that are not always guaranteed by ScaMPI and dead lock If you want to know more about how to write portable MPI programs see e g 2 5 2 Namespace pollution TheScaM PI library is written in C and wehave prefixed all our classes with MPI_ Depending on the compiler used the user may run into problems if he she has C code with prefix MPI_ We have also a few global variables that could cause problems Name Description MPI_log
54. y using ScaMPI you can safely skip section A 1 about installation Section A 2 contains an overview of the file system layout of Scali software under opt scali A 3 give references to useful 3party parallel software A 1 Installing A 1 1 Requirements Before you can run an MPI program with ScaMPI the following requirements must be satisfied e Access toa cluster of SCI interconnected nodes with supported SCI drivers installed or access to a single node e The network file system must give a single file system image from all nodes This restriction regards only the path to programs and files used when executing an MPI program e The supported gcc compiler must be installed A license daemon scald is installed on preferably a front end of the cluster The proper license file is obtained from Scali at support scali com e After installing ScaMPI you get an mpi daemon mpid on each node of the cluster It is important to install the licenses properly before you install ScaM PI Otherwise you have to manually start mpid on each node of the cluster after installing the license software e Finally you can compile and link your program and run it with mpimon e Therequired software and instructions areavailable on a Scali cdrom or downloadable from http www scali com e Theinstall instructions given here are appropriate for installation on a single node at a time Refer to the ScaM PI Installation Guide for support of automatic instal

ScaMPI User's guide

Contents

Download Pdf Manuals

Related Search

Related Contents