Mellanox Technologies
Contents
1. cccsccccsssssscccsssssccccsscsccccesienccesebecssseceees 11 35 HOPS USED ix Aaahhh A nn 11 3 1 1 Default or Common Case Usage 0 0 t A A a 11 31 2 INon Default Usage stir a atli SA a a tees 11 O A A e AA 13 S221 sOSmshOpions O E a TA 13 3 2 1 1 Setting Some Options A Mr 14 3 2 2 Building a Basic SM Flow Hello World oooooccnocccnnccccnocccnnnnccnnnos 15 3 2 2 1 Binding osmsh to an IB port sm sstessberssesssrersrrressrrssrrrsrrrsrrrsrrrrsnrrr rr rn nr 15 3 2 2 2 Starting WED A its 15 3 2 2 3 Shutting down osmsh Y iia 15 3 2 2 4 A Complete Basic Flow ann 16 3 2 3 Interactive Flow coloidal 16 3 224 Data Modelos o ide e ts tes pde o ee e mete 17 3 2 4 1 Accessing the Data Model Using Identifiers ssosssosressressrrnnr 17 Did A De ACCESSINE NEI A ii 18 3 2 4 3 OEA Info ti 18 3 2 5 LID It Policy 0 naga 20 3 2 6 Routing Policy Integrating a User s Routing Engine 22 3 2 7 Configuring Quality Of Service ooooocnconncnnnnonononaconnnanononaconono nono nocnnonecnnn 23 3 2 8 Configuring Partitions VLANS cocoococcooccconocncnnnnnonnnnonnncnonnncnnnnncconnnccnnnos 23 Ded ti S ise eke O A TET 25 Mellanox Technologies Rev 0 3 OpenSM User s Manual About this Manual This manual describes the OpenSM package OpenSM is a Subnet Manager for the initialization of InfiniBand com pliant devices This manual is organ2. physp id phys port identifier block the VLArb table block The following is an example demonstrating how to use these objects get the current value of the VL Arbitration table object set vl osm_vl_arb_tbl_mad cget vl_entry replace some entries set vl lreplace vl 0 O 1 100 set the VL Arb object with your changes osm vl arb tbl mad configure vl_entry vl get a particular node set nodes lindex osm get nodes 0 get a phys port to work with we use port 1 set phys osm get physp node 1 now send the VL Arb to this physical port osm vl arb tbl mad send phys 1 verify the result by query of the phys port VL Arb table NOTE you need to wait for the SubnAdm GetResp VLArbTable to arrive wait time gt retry count subnet timeout processing time osm_physp_t vl arb get phys 3 2 8 Configuring Partitions VLANs InfiniBand provides extensive support for partitioning a large fabric into smaller sometimes overlapping partitions It does that by specification of hardware assisted packet filtering based on a packet s source partition at both input and output ports of every IB device A port is part of a partition if the PKey Partition Key is stored in its Partition Key Table Mellanox Technologies Rev 0 3 24 Using opensm osmsh and osmtest Thus osmsh provides the means to modify partition table contents and set the appropriate fields in the PortInfo and Nodelnfo th
3. Tcl objects As such they have two flavors for their usage identifiers and objects 3 2 4 1 Accessing the Data Model Using Identifiers As described above the identifiers can be obtained by using access functions Once an object identifier is stored in a variable each one of its attributes can be obtained by using a get method or modified using a set method The format of the get set methods is lt class gt _ lt attribute gt _ lt getlset gt For Example obtain all discovered nodes set nodes osm_get_nodes get the first node set node lindex nodes 0 get the node description osm_node_t_node_desc_get node Mellanox Technologies Rev 0 3 Using opensm osmsh and osmtest 18 3 2 4 2 Accessing Objects Any identifier can be converted into a Tcl Object As such like any other Tcl object it provides the standard cget and configure methods for inspecting and modifying its attributes In order to convert an identifier into an object use the following command lt class gt lt obj name gt this lt obj identifier gt Example using the previous identifier stored in node osm_node_t myNodeObj this node Once declared the lt obj_name gt can be used in conjunction with the standard configure and cget commands Example following the previous one myNodeObj cget node_desc gt MT47396 Infiniscale iii Mellanox Technologies 3 2 4 3 Objects Content Info The fol
4. Execute the following command tar xvfz osm XXX tgz 3 Execute the following command osm XXX install Mellanox Technologies Rev 0 3 OpenSM User s Manual 3 Using opensm osmsh and osmtest This section of the manual describes the provided executables and their usage 3 1 opensm opensm is a simple command line executable that serves as both a Subnet Manager and a Subnet Administrator It can be run with or without specifying any command line options 3 1 1 Default or Common Case Usage By entering opensm on the command line without any additional options the default settings will bechosen These defaults were designed to meet the common case usage on clusters with up to a few hundred nodes Thus in this default mode opensm will scan the IB fabric initialize it and sweep occasionally for changes OpenSM attaches to a specific IB port on the local machine and configures only the fabric connected to it If the local machine has other IB ports OpenSM will ignore the fabrics connected to those other ports In default operation mode OpenSM will present the available ports and prompt for a port number t attach to The run will be logged into two files var log messages and tmp osm log The first file will register only general major events whereas the sec ond will include details of reported errors All errors reported in this second file should be treated as indicators of IB fabric health issues Note that when a fa
5. Mellanox Technologies OpenSM User s Manual Rev 0 3 Mellanox Technologies 2 Copyright 2004 Mellanox Technologies Inc All Rights Reserved OpenSM User s Manual Document Number 2277UM Mellanox Technologies Inc 2900 Stender Way Santa Clara CA 95054 U S A www Mellanox com Tel 408 970 3400 Fax 408 970 3403 Mellanox Technologies Ltd PO Box 586 Hermon Building Yokneam 20692 Israel Tel 972 4 909 7200 Fax 972 4 959 3245 Mellanox Technologies Rev 0 3 Table of Contents Table OF Contents seecsvssdesscncsisseesesssescous scescdsasacaseegedosdascasascasescoucesassuscuostdesecsepsiece 3 Ab ut this Manual issi aka assa st se aa 5 O tatascaiagsentaseasnoseeanoss 7 Ld Key Concepts and Terms siria Ad de E ZE A 7 1 2 Contents of OpenSM Package tala 8 13 Dependencies yin OR ER ON STEN 8 2 Installation of OpenSM Package ccssccssssccssssccsssscsssccssscssssssssssssssessseseess 9 2 1 Installation Prerequisites issi svis ssnssiksdin dus s i Sks h ls d S RA ina SARA R coa ngdd 9 2 2 Installing the OpenSM Software lt A cccssgsccscseadevesassecesctuasaiciaaetuctaatedaceasanduatens 9 2 2 1 Installation on top of Standard Mellanox IB Stack Distributions 9 2 2 2 Non standard OpenIB Based Installation 00200e0000e efa 9 2 2 3 Non standard VAPI Based Installation seessesseesesssrsssrss osoderderssren 10 3 Using opensm osmsh and osmtest
6. OWN return lindex GuidLidStat 0 return Use the first available port active or init set port guid OSM_GetFirstAvailablePortGuid if port_guid puts E Fail to find any available port exit puts Attaching OpenSM to Port port_guid if losm bind port_guid puts E Fail to bind OSM to port_guid exit SWEEP IT osm_sweep Better to wait for it then miss something osm_wait_for_subnet_up Loop forever until shut down while osm exit flag after 5000 Shut down officially osm destroy exit 3 2 3 Interactive Flow Several commands are provided to enable running the sweep in a step by step manner It is recommended to turn off the sweep_on_trap option when running interactively Otherwise any trap received will trigger a new sweep thus complicating the task of tracking the program state Table 3 Manual and Interactive Sweep Synopsis Usage osm_start_sweep Initiates a new complete sweep immediately Note that requesting another sweep in the middle of a running one will be ignored silently osm_run discover Run the first single step Discover the fabric Rev 0 3 Mellanox Technologies OpenSM User s Manual Table 3 Manual and Interactive Sweep Synopsis Usage osm_run_lid_assign Run the second single step Must be called right after the discovery step osm run calc lfdb Run the third sin
7. Requires build option bmlx 2 Execute the following command tar xvfz osm XXX tgz 3 Use the distribution type in the command osm XXX install bhpc or osm XXX install b mlx 2 2 2 Non standard OpenIB Based Installation not part of Mellanox distribution At this stage the OpenIB stack installation process is not fully standardized Therefore it is not possible to provide a standard build option for it yet Instead you will need to find where the VAPI and OpenIB C header files vapi h and ts_ib_useraccess h are installed and set the two environment variables TS HOME and MTHOME as described in the following instructions 1 Define the environment variable TSHOME to point to the directory where the file ts_ib_useraccess h resides Define the environment variable MTHOME to point to the directory above the VAPI include subdirectory Make sure that you can see the file MTHOME include vapi h 3 Execute the following command tar xvfz osm XXX tgz 4 Execute the following command osm XXX install vendor ts Mellanox Technologies Rev 0 3 Installation of OpenSM Package 10 2 2 3 Non standard VAPI Based Installation not part of Mellanox distribution If you have installed VAPI in a non standard manner i e not under usr Mellanox perform the following steps 1 Define the environment variable MTHOME to point to the directory above the VAPI include subdirectory such that the file MTHOME include vapi h exists 2
8. age 7 e Installation of OpenSM Package page 9 e Using opensm osmsh and osmtest page 11 1 1 Key Concepts and Terms Throughout this manual there is frequent reference to various concepts and terms which are common to the general audience of System Administrators There is also reference to InfiniBand specific concepts and terms a part of which are briefly defined in the list to follow Other terms may be defined where relevant in this document however the IB specification remains the main reference for all missing and existing IB terms IB devices Integrated Circuits implementing InfiniBand compliant communication e IB Fabric Cluster Subnet A set of IB devices connected by IB cables e Subnet Manager SM One of several entities involved in the configuration and control of the subnet e Master Subnet Manager The subnet manager that is authoritative that has the reference configuration information for the subnet e Standby Subnet Manager A subnet manager that is currently quiescent and not in the role of a master SM by agency of the master SM e Subnet Administrator SA An application normally part of the Subnet Manager that implements the interface for querying and manipulating subnet management data Mellanox Technologies Rev 0 3 Overview 8 e LID An address assigned to a port data sink or source point by the Subnet Manager unique within the subnet used for directing packe
9. at control partition based packet filtering The mechanism provided is exactly the same as the one described in the previous section and includes support for the following MADs Table 6 Pre Allocated MAD Objects Required for Partitioning Object Send method Synopsis Send Parameters osm port info mad lt physp id gt physp id phys port identifier as obtained from osm get physp osm node info mad lt node id gt node id node identifier as obtained from osm get nodes osm pkey tbl mad lt physp id gt lt block gt physp id phys port identifier block the PKey table block Rev 0 3 Mellanox Technologies OpenSM User s Manual 3 3 OsmTest OsmTest provides a test suite for OpenSM and OsmSh Its executable is invoked by typing osmtest OsmTest has the following capabilities and testing flows e It creates an inventory file of all available Nodes Ports and PathRecords including all their fields e It verifies the existing inventory with all the object fields and matches it to a pre saved one e A Multicast Compliancy test An Event Forwarding test e A Service Record registration test An RMPP stress test A Small SA Queries stress test It is recommended that after installing OpenSM the user should run osmtest f c to generate the inventory file and immediately afterwards run osmtest f a to test OpenSM Another recommendation o
10. bject provides the standard cget and con figure methods known to the experienced TCL programmer To get the list of available osmsh options use the fol lowing command osm_opts configure gt m_key sm_key subnet prefix m key lease period sweep interval max_wire_smps transaction timeout sm priority Imc max op vis reassign lids reassign_lfts ignore other sm single thread no multicast option disable multicast force log flush subnet timeout packet life time head of queue lifetime local phy errors threshold overrun errors threshold polling_timeout polling retry number force heavy sweep sweep on trap max port profile port profile switch nodes Rev 0 3 Mellanox Technologies OpenSM User s Manual To retrieve the value of an option use osm_opts cget lt option gt gt lt value gt Example osm_opts cget sweep_interval gt 10 And finally setting an option osm_opts configure lt option gt lt value gt gt lt value gt Example osm_opts configure sweep_interval 20 gt 20 3 2 2 Building a Basic SM Flow Hello World Any osmsh flow should include the following stages binding osmsh to an IB port starting a sweep and shutting down osmsh 3 2 2 1 Binding osmsh to an IB port To bind osmsh to an IB port you will need to find a local IB port which is not in DOWN state The following com mand allows you to inspect the status of all IB ports os
11. ble value for a gt 1000nodes cluster is 1000ms Default Timeout value is 100ms p lt PRIORITY gt This option specifies the SM s PRIORITY Only if there is a need to explicitly control which SM priority lt PRIORITY gt This will affect the handover cases where master should be the master is chosen by priority and GUID v This option increases the log verbosity level The first v will print to the stdout a summary table of the verbose The v option may be specified multiple times discovered fabric to further increase the verbosity level Mellanox Technologies Rev 0 3 Using opensm osmsh and osmtest 12 Table 1 opensm Command Line Options Option Effect When to Use Option V This option sets the verbosity to its maximum level and forces log flushing Use this option to investigate an error or send a bug report f lt file_name gt log_file lt file_name gt This option defines the log file By default the log goes to tmp osm log To send it to standard output use f stdout For your convenience only Note that if you use V the log file might be too large for the tmp 0 once This option causes OpenSM to configure the subnet once then exit Ports remain in the ACTIVE state For testing purposes only r reassign_lids This option causes OpenSM to reassign LIDs to all endnodes Specifying r on a running subnet may dis rupt subn
12. ctively e accessing the data model e defining LID assignment policy e defining routing policy e defining Quality of Service policy e defining Partition policy 3 2 1 osmsh options osmsh provides full access to all opensm internal options which control various aspects of its operation The avail able options are listed in the following table Table 2 OpenSM OsmSh exposed options Option Default and Units Usage m_key 0 MKey used by the SM Set PortInfo sm_key 0 SMKey used by the SA to qualify a query as trusted subnet_prefix Oxf800000000000000 The subnet prefix to be used by SM SA m_key_lease_period 0 MKey lease period included in Set PortInfo sweep_interval 10 sec Interval between sweeps max_wire_smps 1 Number of simultaneous SMPs on the wire transaction_timeout 100 msec The time between a request and its expected response sm_priority 1 The priority of the SM with respect to other SMs Ime 0 2ime is the number of LIDs assigned to each port max_op_vls 1 The maximal number of operational VLs used Mellanox Technologies Rev 0 3 14 Using opensm osmsh and osmtest Table 2 OpenSM OsmSh exposed options Option Default and Units Usage reassign_lids FALSE If true new LIDs will be assigned reassign_Ifts TRUE If true existing LFT values are ignored on first sweep ignore_other_sm FALSE If true no
13. eavy_sweep sweep_on_trap max_port_profile port_profile_switch_nodes Mellanox Technologies Rev 0 3 OpenSM User s Manual 1 Overview The InifiniBand Architecture Specification Volume 1 Release 1 1 defines extensively and explicitly how InfiniBand IB compliant devices are to be managed There it is stated that routing and other management policies for each compliant device should be exported to a centralized entity called Subnet Manager SM This standardization of management significantly reduces the costs of hardware when compared to the costs of traditional distributed net work management policies OpenSM is an InfiniBand compliant Subnet Manager It is provided in two flavors a fixed flow executable called opensm and a fully configureable version called osmsh through TCL extension The two are accompanied by a testing application called osmtest The target audience of this User s Manual is System Administrators who have installed InfiniBand hardware and need to run a Subnet Manager in order to initialize it Since OpenSM implements an SM as defined in the IB specifi cation it will not be defined here again Instead to understand what OpenSM implements the uSer is kindly referred to the following chapters of that specification Management Model 13 Subnet Management 14 and Subnet Administration 15 This User s Manual describes the OpenSM package in the following sections e Overview p
14. ellanox Technologies Rev 0 3 Using opensm osmsh and osmtest 20 3 2 5 LID assignment Policy Real world fabrics require hot plug support i e it should be possible for two disjoint subnets to be unified by sim ply connecting two switches If the two disjoint subnets have overlapping LID assignments these LID collisions must be resolved Reassigning the LIDs will result in the loss of packets and major re configuration of the Unicast forwarding tables LFT will be needed To avoid such re configuration havoc osmsh supports a method by which one can pre assign LIDs to specific IB devices This pre assignment is referred to as LID assignment policy One way to perform this is to base the assign ment on a static list of GUIDs by this it is possible to statically pre assign each existing GUID to a LID Since osmsh and opensm assigns LIDs only after completing a full discovery of the subnet the pre assignment should be performed after the osmsh discovery stage To fulfill this requirement osmsh allows registering a user defined procedure or Tcl expression to be executed between the discovery stage and the assignment of LIDS This is accomplished by using the following command osm reg pre lid assign cmd lt tcl expression or procedure name gt Finally the command provided for associating a LID with a Port GUID is osm physp set lid lt portld gt lt lid gt where portld is a physical port identifier returned b
15. ess lt level gt This option runs the specified stress test instead of the normal test suite Stress test options are s 1 Single MAD response SA queries s 2 Multi MAD RMPP response SA queries Default stress testing is not performed t lt milliseconds gt timeout lt milliseconds gt This option specifies the time in milliseconds used for transaction timeouts request to response Default 100ms V verbose This option increases the log verbosity level The v option may be specified multiple times to further increase the verbosity level Mellanox Technologies Rev 0 3 Using opensm osmsh and osmtest 26 Table 7 OsmTest Command Line Options Option Effect V This option sets the verbosity level to the maximum and forces log flushing lt file name gt This option is used to specify the log file name log file lt file name gt By default the log goes to standard output d lt number gt This option specifies debug behavior The number following d selects the debug option to be enabled debug lt number gt can be specified multiple times d 0 Unused d 1 Do not scan and compare Path Record should be used on large clusters as Paths Nodes d 2 Force log flushing after each log message d 3 Unused Default no debug options are enabled Mellanox Technologies Rev 0 3
16. et traffic Default OpenSM attempts to preserve existing LID assignments resolving multiple use of same LID Should rarely be used Use only if re numbering of all the LIDs is required Note that when using multiple SMs for redundancy this option should NEVER be used lt LMC gt lmc lt LMC gt This option specifies the subnet LMC value The number of LIDs assigned to each port is 22LMC The LMC value must be in the range 0 7 LMC values gt 0 allow multiple paths between ports Default OpenSM defaults to LMC 0 which allows one path between any two ports Use when path migration option is required Note that in order to take advantage of path migration the Connection Manager or any other method of RC communication setup should take additional steps LMC values gt 0 should only be used if the subnet topology provides multiple paths between ports ie multiple interconnects exist between switches i lt eq ignore guids file gt ignore guids lt eq ignore guids file gt This option provides means to define a set of ports by guids that will be ignored by the link load equalization algorithm If there are some ports on the fabric that are rarely used like a dedicated OpenSM node it is possible to specify their guids This way their BW will be ignored by the routing algorithm d lt number gt debug lt number gt This option specifies debug behavior The number follo
17. f OsmTest usage is to create the inventory when the IB fabric is stable and occasionally run osmtest v to verify that nothing has changed The following table provides a full description of all osmtest options Table 7 OsmTest Command Line Options Option Effect g lt GUID in hex gt guid lt GUID in hex gt OsmTest will bind to the port with the provided GUID Default is to present to user the available GUIDs and to select one of them Use this option to avoid the interaction required in order to select the port lt is possible to enter echo 1 opensm to select the first GUID i lt filename gt inventory lt filename gt This option specifies the name of the inventory file Normally osmtest expects to find an inventory file with which it validates real time information received from the SA during testing Default osmtest uses the file osmtest dat f lt clalv slelflmla gt flow lt cla v s e f m q gt The actual test flow run by OsmTest C create an inventory file with all nodes ports and paths a run all validation tests expects an input inventory v only validate the given inventory file S run service registration un registration and lease e run event forwarding test f flood the SA with queries according to the stress mode m multicast flow q QoS info Dump VLArb and SLtoVL tables Default flow all the above but QoS s lt level gt str
18. gle step Calculate the min hop count tables osm run set lfdbs Assign routes by setting the LFTs of all the switches osm run set mfdbs Calculate and Set the Multicast Forwarding Tables osm run arm Bring all links to ARM state if are in INIT osm run activate Bring all links to ACTIVE state osm run continuous Resume continuous non single step sweep mode If you plan to use interactive mode it is highly recommended to install the Tcl extensionnamedstclreadline This will provide you superior interactive features in osmsh command completion editing history and log 3 2 4 Data Model osmsh provides access to some of the internal data structures of the SM in the form of object identifiers A set of access functions can be used to obtain these identifiers The access functions and the format of the identifiers are described in the following table Table 4 Exposed objects and their access functions Object Type Class Access Function Identifier Format Nodes osm_node_t osm_get_nodes node lt node guid gt Ports osm_port_t osm_get_ports port lt port guid gt Physical Ports osm physp t osm get physp lt node gt lt num gt physp lt node guid gt _ lt port_num gt Switches osm switch t Osm_get_switches switch lt node guid gt Example of Usage A Simple Report Eine puts OSMSH Discovered llength osm_get_nodes nodes llength osm_get_switches switches osmsh objects are standard Swig
19. handoff compliancy single_thread TRUE If true use a single thread for SMP processing no_multicast_option FALSE If true no multicast support by SA ClassPortInfo disable multicast FALSE If true no multicast GSI support force log flush TRUE If true force flush of the log file on every log subnet_timeout 18 dec time 4us 2SUbnet_timeout_ Used for Trap resend packet_life_time 20 dec time 4us 2Plttimeout Max life timesfor packet on the switch The default value turns off this mechanism head of queue lifetime 20 dec time 4us 2h04_timeout_ Max time for a packet at the head of the Tx queue The default value turns off this mechanism local_phy_errors_threshol 8 The number of consecutive PHY errors that will cause a Trap d overrun_errors_threshold 8 The number of buffer overrun errors that will cause a Trap polling_timeout 1000 msec Time between polls of the other Master SM polling_retry_number 4 Number of failing other Master SM polls that will cause re dis covery force_heavy_sweep FALSE If true makes every sweep scan through the entire subnet sweep_on_trap TRUE Start a heavy sweep when trap is received max_port_profile XX Deprecated do not allow link over subscription above this value port profile switch nodes FALSE If true will include switch nodes in the link subscription counting Otherwise ignore them 3 2 1 1 Setting Some Options The osmsh options are exposed to TCL as a TCL object As such this o
20. ized in the following manner e Chapter provides an overview of the OpenSM package page 7 e Chapter 2 provides instructions for the installation of the OpenSM package page 9 e Chapter 3 describes how to use the tools included this OpenSM package page 11 Intended Audience The target audience of this User s Manual is System Administrators who have installed InfiniBand hardware and need to run a Subnet Manager SM in order to initialize it Related Documentation For InfiniBand related issues please refer to the following specification e InifiniBand Architecture Specification Volume 1 Release 1 1 For implementing user level applications that interface with the InfiniBand Subnet Administrator send MADs through the SMI or GSI please refer to the following manual e OpenSM Vendor Layer API and Programmer s Manual Rey 0 1 Conventions Throughout this document TCL shell commands are typed in the following format command gt where the second line is the command return For Example osm_opts configure gt m_key sm_key subnet_prefix m_key_lease_period sweep_interval max_wire_smps transaction_timeout sm_priority Imc max_op_vls reassign_lids reassign_lfts ignore other sm single_thread no_multicast_option disable_multicast force_log_flush subnet_timeout packet_life_time head_of_queue_lifetime local_phy_errors_threshold overrun_errors_threshold polling_timeout polling_retry_number force_h
21. l 3 2 7 Configuring Quality of Service osmsh provides only rudimentary support for configuring Quality of Service As Quality of Service involves setting a device table of three different types SL2VL map VL Arbitration map and PortInfo osmsh provides the means to set these attributes and send them over to remote devices To achieve this osmsh pre allocates a single Tcl object containing each of these types The object provides the standard cget and configure methods by which the user can inspect and modify the content of an attribute Two additional methods are provided 1 send allows sending the attribute to a remote device 2 clear resets the attribute value to zero Note that the response for the send method which translates into a standard SubnAdm Set MAD is a standard Get Response MAD As such osmsh will accept this response and update the internal database with the Set result so the algorithm can verify the Set was successful The Objects listed in the following table are pre allocated Table 5 Pre Allocated MAD Objects Required for QoS Flows Object Send method Synopsis Send Parameters osm_port_info_mad lt physp id gt physp id phys port identifier as obtained from osm_get_physp osm slvl tbl mad lt physp id gt lt in port gt physp id phys port identifier in port input port number osm vl arb tbl mad lt physp id gt lt block gt
22. lowing code can be loaded into osmsh and used to dump out the content of every identifier or object proc objDump obj catch obj cget atts puts Object Dump a foreach attr lindex atts 0 set an string range attr 1 end puts an obj cget attr Puts 7 proc identifierDump class id if catch class obj this id el puts e else objDump _ obj rename obj set node lindex osm_get_nodes 0 gt node 0x0002c90120267d40 identifierDump osm node t node gt Object Dump this node 0x0002c90120267d40 node_info base_version 1 class_version 1 node_type 2 num_ports 8 sys_ guid 0x0000000000000000 node_guid 0x0002c90120267d40 port guid 0x0002c90120267d40 partition cap 32 device id 43132 revision 160 port num vendor id 50332361 node_desc MT43132 Mellanox Technologies discovery_count 1 physp tbl size 9 Mellanox Technologies Rev 0 3 OpenSM User s Manual osm node t myNode this node objDump myNode gt Object Dump this node 0x0002c90120267d40 node info base version 1 class version 1 node type 2 num_ports 8 sys guid 0x0000000000000000 node guid 0x0002c90120267d40 port guid 0x0002c90120267d40 partition cap 32 device id 43132 revision 160 port num vendor id 50332361 node desc MT43132 Mellanox Technologies discovery count 1 physp tbl size 9 19 M
23. m_get_local_ports_info gt guid lid1 link state1 guid2 lid2 link state2 Example osm_get_local_ports_info gt 0x0002c901093d91c1 0x0400 DOWN 0x0002c901093d91c2 0x0500 ACTIVE The first port reported HCA port 1 is down eithermo cable is connected to the port or the other side of it is con nected to a non active device power off or no driver The next step is to bind osmsh to a specific port using the following command osm_bind lt port guid gt gt 0 1 0 success 1 error Example continued osm_bind 0x0002c901093d91c2 gt 0 3 2 2 2 Starting a sweep To start a sweep simply invoke osm_sweep gt 0 Before progressing to inspecting the results wait for the subnet to be initialized osm_wait_for_subnet_up gt 0 3 2 2 3 Shutting down osmsh To shut down osmsh use the following command osm_destroy gt 0 Mellanox Technologies Rev 0 3 Using opensm osmsh and osmtest 16 3 2 2 4 A Complete Basic Flow To complete the basic flow a routine to find the first active port is provided The following is an example of a full flow that can be saved to a file and run as is The first part of this program is an example of the routine mentioned above osmsh Find the first available port that is not DOWN and return its GUID proc OSM_GetFirstAvailablePortGuid foreach GuidLidStat osm_get_local_ports_info set portState lindex GuidLidStat 2 if portState D
24. se might be invalid if catch set port osm get physp node OJH continue only first port if switch set port_guid osm_physp_t_port_guid_get port switch node only port 0 requires a lid if info exists GUID TO LID TBL port_guid set lid GUID_TO_LID_TBL port_guid osm physp set lid port lid puts OSM PrelLidAssign Seting Port port_guid Lid lid else puts OSM_PreLidAssign Ignoring undef sw port port guid port_guid finally register the function to be invoked during the sweeps osm_reg_pre_lid_assign_cmd OSM_PreLidAssign Mellanox Technologies Rev 0 3 Using opensm osmsh and osmtest 22 3 2 6 Routing Policy Integrating a User s Routing Engine The osmsh routing algorithm basically equalizes the link load between all the links with the minimal hop count to the target LID Being a good generic algorithm it might be adequate for many IB fabrics however it is usually not opti mal for configurations with non heterogeneous endnodes To provide the means to load or calculate a better Unicast routing scheme osmsh supports registering a user defined Tcl expression to be evaluated before the internal algo rithm for Unicast Linear Forwarding Table assignment is invoked A simple interface for assigning a single LFT entry is also provided Note that for complex dynamic cases where the topology frequently changes an online router might be needed Tcl is not the most adeq
25. tal and non recoverable error occurs opensm will abort Both log files should include the message SUBNET UP if opensm was able to setup the subnet correctly 3 1 2 Non Default Usage It is possible for the user to run opensm with settings other than the default ones Table 1 lists the opensm command line options in the first column the effect of each option in the second and tips on when to use each option in the last Table 1 opensm Command Line Options Option Effect When to Use Option g lt GUID in hex gt OpenSM will bind to the port with the provided GUID To avoid the interaction required in order to select the guid lt GUID in hex gt Default is to present to user the available GUIDs and to port It is possible to enter select one of them echo 1 opensm to select the first GUID s lt interval gt This option specifies the number of seconds between To minimize unneeded sweeps it is possible to set this sweep lt interval gt subnet sweeps Specifying s O disables sweeping value to 0 This will cause a single sweep and only traps Default OpenSM sweeps with intervals of 10 seconds will cause new sweeps On large clusters it is recom mended to set this to a value higher than 60 t lt milliseconds gt This option specifies the time in milliseconds This value should be changed only on large subnets A timeout lt milliseconds gt used for transaction timeouts request to response reasona
26. ther distributions Therefore it is very possible that OpenSM is already installed on the user s machine However for new separate releases of OpenSM stand alone installation instructions are provided below 2 1 Installation Prerequisites Before the installation of the OpenSM package it is necessary to guarantee the following requirements e Either a VAPI or an OpenIB driver is available on the system e The Tcl8 3 or Tcl8 4 package needs to be installed Standard Linux installations normally include this package However if you are unable to run the command tclsh8 3 or tclsh8 4 on your machine yowwill need to down load a Tcl package from one of the sources making it available on the WWW e The latest OpenSM release has been downloaded from the OpenIB site downloads page or from Mellanox s docs mellanox com 2 2 Installing the OpenSM Software As mentioned before OpenSM may be installed on top of the OpenIB stack or on top of VAPI Installation instruc tions for both options are provided below The following routines assume you were able to obtain the OpenSM tar gzip archive named osm XXXX tgz 2 2 1 Installation on top of Standard Mellanox IB Stack Distributions 1 Make sure you know what type of standard installation exists on your system e HPC standard distribution is installed under usr local ib hpc Requires build option b hpc e VAPI is installed at usr mellanox look for usr mellanox include vapi h
27. ts within the subnet Unicast Linear Forwarding Tables LFT A table that exists in every switch providing the port through which packets should be sent to each LID Multicast Forwarding Tables A table that exists in every switch providing the list of ports to forward received multicast packet The table is organized by MLID 1 2 Contents of OpenSM Package The OpenSM package contains the following executables and libraries opensm A Subnet Manager and Administrator It should be used for standard simple installations osmsh A Tel shell extended with the full set of opensm commands and its data model It should be used in case non stan dard policies for Quality of Service LID assignment routing etc are needed osmtest A simple application to test opensm and osmsh It is capable of exercising most of the SA queries and provide clear feedback on their success osmv_svc shared library Contains services which enable user level applications to interface with the SA send MADs through the SMI or GSI An extensive API definition is provided by the document OpenSM Vendor Layer API and Programmer s Manual Rev 0 1 1 3 Dependencies OpenSM in its current form is available on top of VAPI and OpenIB stacks osmsh is dependent upon TCL revision 8 3 and later Mellanox Technologies Rev 0 3 OpenSM User s Manual 2 Installation of OpenSM Package OpenSM is bundled into the releases of VAPI OpenIB and various o
28. uate language for implementing this algorithm rather it is better to code it in C C and dynami cally load a shared object into osmsh The Tcl should only be used to invoke the main routing task The following command assigns an LFT entry for a given LID on a given switch osm_switch_fdb_set lt switchld gt lt lid gt lt port_num gt switchld is a switch identifier as returned by osm get switches The following command registers the given Tcl command to be invoked before the Unicast LFT assignment osm_reg_ucast_fdb_assign_cmd lt tcl expression or procedure name gt The code below provides an example of how to use these interfaces for loading pre calculated routing This proc will use the preloaded FDB data and pre assign them It assumes a global variable holding a map of switch guid to list of LID PORT pairs for each switch proc OSM_SetSwitchesFDBs global OSM_SW_FDB track number of pre assignments set num 0 go over all switches and check if their we have pre assigned FDB foreach sw osm_get_switches get the sw guid set guid string range sw 7 end ok we got some FDB entries to set if info exists OSM SW FDB guid foreach lid port num OSM SW FDB guid osm switch fdb set sw lid port num incr num puts OSM SetSwitchesFDBs Pre Assigned num FDB entires osm_reg_ucast_fdb_assign_cmd OSM_SetSwitchesFDBs Mellanox Technologies Rev 0 3 OpenSM User s Manua
29. wing d selects the debug option can be specified multiple times d 0 Ignore other SM nodes d 1 Force single threaded dispatching d 2 Force log flushing after each log message d 3 Disable multicast support d 4 Put OpenSM in memory tracking mode d 10 Put OpenSM in testability mode Default No debug options are enabled These options are not normally needed Rev 0 3 Mellanox Technologies OpenSM User s Manual 13 3 2 osmsh osmsh short for OpenSM Shell is intended to provide an open customization solution for Subnet Management To facilitate full customization and configuration of subnet management it is required to define policies that control how the following tasks are performed e LID assignment e Unicast Routing e Partitioning of the subnet similar to the VLAN concept e Setting of Service Levels and their relative arbitration rules Rather than defining a specific syntax for the definition of each policy the user is allowed to write arbitrary algo rithms which directly interact with the lower level interface controlling the fabric Such algorithms can be imple mented in TCL or as shared C C libraries The algorithm may use an additional small set of osmshcommands as well as access the internal data model of the SM The API and data model are described in the following sections e osmsh extended control options e building a basic SM flow e running intera
30. y osmsh The code on the following page provides a complete example demonstr ting how this LID assignment policy fea ture can be used in practice Mellanox Technologies Rev 0 3 OpenSM User s Manual PROC assoc lt key gt lt key value list gt given a key and a list of ley value pairs get the pair proc assoc key key list foreach kv key list if lindex kv 0 key return lrange kv 1 endl return PROC OSM_PreLidAssign this routine will pre assign lids based on a global variable mapping of port guid to lid GUID_TO_LID_TBL guid gt lid proc OSM_PreLidAssign global GUID_TO_LID_TBL go over all nodes foreach node osm_get_nodes need to decide if it will have a single guid or multiple set nodelnfo osm_node_t_node_info_get node set nodeType assoc node_type nodelnfo set numPorts assoc num_ports nodelnfo if not a switch we can assign lids to each port if nodeType 2 go over all physical ports available and set lid for set pn 1 pn lt numPorts incr pn might be invalid phys port if catch set port osm_get_physp node pn continue set port_guid osm_physp_t_port_guid_get port if info exists GUID TO LID TBL port guid set lid GUID TO LID TBL port guid catch osm physp set lid port lid puts OSM PreLidAssign Seting Port port_guid Lid lid else puts OSM_PreLidAssign Ignoring undef port port guid port_guid el
Download Pdf Manuals
Related Search