- Oracle Documentation
Contents
1. on page 14 12 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 10 Determine your MAC addresses and whether they are to be global or locally administrated Note Each MAC address must be unique and not 00 00 00 00 00 00 Note Only even numbers are supported for the most significant byte of the MAC address unicast 11 Activate the VIOA by assigning a MAC address to it FabMan gateway_name gt createwnic connector guid guid mac mac_address pkey default where connector is the name of the connector OA ETH 1 to 0A ETH 4 and 1A ETH 1 to 1A ETH 4 m guid is the global unique identifier of the target port on the host associated with the VIOA m mac_address is the MAC address to be assigned to the VIOA Note You can use the information from Step 8 as a guide For example to assign the MAC address of a0 a5 91 95 30 9a to the VIOA for port GUID 21280001A0A591 through connector 0A ETH 2 FabMan gateway_name gt createvnic OA ETH 2 guid 0021280001A0A591 mac a0 a5 91 95 30 9a pkey default VNIC created FabMan gateway_name gt The VIOA is created and an Ethernet over InfiniBand data link is bound to the VIOA 12 Verify that the VIOA is active FabMan gateway_name gt showvnics ID STATE FLG IOA_GUID NODE IID MAC VLN PKEY GW 7 UP N 00 21 28 00 01 A0 A5 91 solaris01 0000 a0 a5 91 95 30 9a NO ffff OA ETH 22. Sun InfiniBand QDR Gateway Switch Product Notes 13 4 UP N 00 03 BA 00 01 00 E3 71 mnm34 60 0000 02 02 02 02 02 03 NO ffff OA ETH 1 5 UP N 00 03 BA 00 01 00 E3 71 mnm34 60 0002 02 02 02 02 02 04 NO ffff OA ETH 1 3 DISABLED N 00 03 BA 00 01 00 E3 71 mnm34 60 0000 02 02 02 02 02 02 NO ffff OA ETH 1 6 UP N 00 03 BA 00 01 00 E3 72 mnm34 60 0000 02 02 02 02 02 05 NO ffff OA ETH 1 FabMan gateway_name gt Note Until the IP interface is created on the data link using the ipadm command at the Oracle Solaris host the STATE of the VNIC is WAIT IOA 13 Configure an IP address on the data link See Configure IP Addresses on the Data Link Oracle Solaris on page 14 V Configure IP Addresses on the Data Link Oracle Solaris 1 Become superuser of the Oracle Solaris 11 host 2 Verify that the data link corresponding to the activated VIOA has been created For example dladm show phys LINK MEDIA STATE SPEED DUPLEX DEVICE vnet0 Ethernet up 0 unknown vneto 1bp0 Infiniband up 32000 unknown ibp0 ibp1 Infiniband down 2000 unknown ibp1 net5 Ethernet up 10000 full eoib0 net4 Ethernet up 10000 full eoibl net6 Ethernet unknown 10000 full eoib2 The dladm show phys command displays all data links corresponding to all hardware devices Similarly the dladm show link command displays the class MTU and state for all data links For example dladm show link LINK CLASS
3. gateway_name m CLI telnet gateway_name 1234 where gateway_name is the host name or IP address of the management controller Note The Oracle ILOM stack requires at least 2 minutes to become operational after a reboot The next time you log in to the gateway this message is displayed FW upgrade completed successfully on Mon Oct 17 18 36 14 IST 2012 Please run the fwverify CLI command to verify the new image This message will be cleared on next reboot 12 If the Subnet Manager was disabled in Step 2 log in to Oracle ILOM access the restricted Linux shell and enable the Subnet Manager ssh 1 ilom admin gateway_name ilom admin gateway_name s password password gt show SP Fabric_Mgmt NOTE show on Fabric_Mgmt will launch a restricted Linux shell User can execute switch diagnosis SM Configuration and IB monitoring commands in the shell To view the list of commands use help at rsh prompt Use exit command at rsh prompt to revert back to ILOM shell FabMan gateway_name gt enablesm Starting IB Subnet Manager OK Starting partitiond daemon OK FabMan gateway_name gt Sun InfiniBand QDR Gateway Switch Product Notes 25 13 Verify the firmware version FabMan gateway_name gt version SUN DCS gw version 2 0 8 1 Build time Feb 6 2013 09 47 52 FPGA version 0x33 SP board info Manufacturing Date 2009 06 24 Serial Number NCD3R0853 Hardware Revision 0x00
4. Use the ibnetdiscover command The usage output of the deletevnic command indicates that connectors 0A ETH and 1A ETH are supported They are not Stopping or starting the environment daemon on a gateway running as a fabric manager or monitored by a fabric manager changes the FabricMIB configuration due to the new status of that fabric element Should an snmpwalk of the FabricMIB be issued at the same time the FabricMIB library reloads the updated configuration internal file pointers might be mishandled initiating a segmentation fault There is no workaround at this time During a firmware upgrade or downgrade certain symbolic links might not be created or become lost Consequently the Fabric Monitor will fail to load when requested Workaround Reperform the upgrade or downgrade using the force option When two or more gateways within the InfiniBand fabric have adjacent gateway instance numbers for example 16 and 17 and the gateway with the lower instance number 16 has more than one LAG enabled there is a port ID collision between the two gateways Workaround Configure all gateways to have even instance numbers 2 4 6 with the setgwinstance command On rare occurrence when rebooting the gateway the gateway might experience a hard hang The last entry in the system log is kernel mlx_bx_core 0000 05 00 0 go bit not cleared Workaround Power cycle the gateway by removing both power cords from the power s
5. 11 MB available If not enough space is available you must delete files from the config directory 5 Verify that there is at least 1 MB available in the var 1og directory df h var log Filesystem Size Used Avail Use Mounted on dev hda3 16M 3 6M 11M 25 var log In this example there are 11 MB available If not enough space is available you must delete files from the var 1log directory 6 Verify that there is at least 150 MB free memory available Copyright aaa Version ILOM 3 0 r47111 c 2010 Oracle and or its affiliates All rights reserved free m total used free shared buffers cached Mem 498 104 393 0 12 47 buffers cache 45 453 Swap 0 0 0 In the buffers cache row of the free column there should be at least 150 MB free memory In this example there are 453 MB available If not enough memory is available you must exit nonessential applications that are running 7 Start the Oracle ILOM shell spsh Oracle R Integrated Lights Out Manager You are now in the Oracle ILOM shell You can use the exit command to return to the Linux shell 22 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 8 Begin the upgrade process gt load source URI pkgname where m URI is the uniform resource indicator for the host where the gateway firmware package is located The FTP and HTTP protocols are
6. Search tab The Patch Search window updates Click the Product or Family Advance link The Patch Search window updates In the Product Is drop down menu select Sun Network QDR Infiniband Gateway Switch In the Release Is drop down menu select Sun Network QDR Infiniband Gateway Switch x y z Where x y z is the version number of the firmware package to be acquired For example 2 0 8 Click outside of the drop down menu Click Search The Patch Search window expands with the search results In the Patch Name column click the respective patch number link For example 16295011 The Patch Search window reformats Click Read Me to display the README file Click Download The File Download window opens Click the filename zip link to initiate the download For example p16295011_208_Generic zip Indicate where the file should be saved The file is downloaded and saved In your receiving directory decompress the filename zip file The firmware is in the SUN_DCS_gw_x y z tar gz file The README file contains the latest information about the firmware release Unpack the gz file gtar zxvf SUN_DCS_gw_X y z tar gz 18 The extracted files are displayed Move the gateway firmware package filename pkg to a directory on a host that is accessible by Oracle ILOM 20 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 19 Upgrade
7. here Set Up VIOAs Oracle Solaris Note If the VIOA is to be assigned to a VLAN you cannot create the VIOA from the gateway Instead use the procedure in Gateway Administration creating Oracle Solaris VNICs 1 Become superuser of the Oracle Solaris 11 host For this procedure the Oracle Solaris 11 host name is solaris01 2 Display the HCA GUIDs and port GUIDs configured on the host dladm show ib LINK HCAGUID PORTGUID PORT STATE PKEYS ibp0 21280001A0A590 21280001A0A591 1 up FFFF ibp1 21280001A0A590 21280001A0A592 2 up FFFF 3 Record the host name HCA GUIDs and port GUIDs In this example the HCA GUID is 21280001A0A590 and the port GUIDs are 21280001A0A591 and 21280001A0A592 4 From the management controller of the gateway display the HCAs recognized by the gateway Ca Ca Ca Ca Ca FabMan gateway_name gt ibhosts 0x0021280001A0A590 ports 0x00212856cd22c040 ports 0x0002c903000891aa ports 0x00212800013ece9e ports 0x0003ba000100e370 ports FabMan gateway_name gt N NO ON N N solaris01 SUN IB QDR GW switch mnm34 97 Bridge 1 mnm34 54 HCA 1 mnm34 55 HCA 1 mnm34 60 HCA 1 Sun InfiniBand QDR Gateway Switch Product Notes 11 5 Compare the output of the ibhosts command with the recorded host name HCA GUIDs and por t GUIDs The port GUIDs are the HCA GUID 1 port 1 and HCA GUID 2 port 2 respectively 6 Verify that the gateway recogn
8. prefix correctly createvlan command does not support both hex and decimal P_Key value input clear option of setgwethport command does not restore default parameters checkguidfilesftree is displayed with the help all command but is not supported Powering on the gateway randomly displays these errors at BridgeX ports BX_0 and BX_1 e Peer RevErrors Peer VL15Dropped e XmtDiscards There is no workaround at this time Though the Oracle ILOM web interface supports Internet Explorer the Fabric Monitor does not Workaround Use Mozilla Firefox 3 0 or 3 6 Using the Mozilla Firefox 6 0 browser only navigational control is possible with the Oracle ILOM web interface No informational output is displayed Workaround Use Mozilla Firefox 3 0 or 3 6 If the Subnet Manager is disabled before starting a partition configuration session using the smpartition start command confusing messages are displayed upon issuing the command Workaround Ensure that the Subnet Manager is enabled before starting a partition configuration session GUIDs can be displayed as 0021280001a1623a 00 21 28 00 01 a1 62 3a or 0x0021280001a1623a The latter version is not recognized by the createvnic command Workaround Truncate the 0x prefix of the GUID for the createvnic command The construct of the P_Key and its membership bit requires the value to be in hexadecimal Providing and displaying the P_Key in decimal would complicate iden
9. produits ou services tiers ou leur utilisation eo n ga Adobe PostScript Contents Using This Documentation v Sun InfiniBand QDR Gateway Switch Product Notes 1 Known Problems 1 General Information and Issues 4 Reserved VLAN Identifiers 4 Jumbo Frames 4 Subnet Manager 4 Snapshot Dataset Information 4 Software Information and Issues 5 Main Board Management Controller and Chassis Serial Numbers High Availability in Partitions 5 Email Alert Rules 5 Time Zone Support 5 Firmware Update Considerations 6 Host Software and Firmware 7 Symbol Errors on Internal Ports 7 Commands Available to Restricted Linux Shells 7 Hardware Information and Issues 8 Ethernet Support 8 5 iv Number of LAGs 8 Unusable Ports 8 Documentation Information and Issues 8 smpartition Command Guidelines 8 VNIC Configuration on Linux Hosts in Gateway Manual Mode 10 Corrections to Creating Virtual I O Adapters Oracle Solaris 11 v Set Up VIOAs Oracle Solaris 11 v Configure IP Addresses on the Data Link Oracle Solaris 14 SNMP V3 Protocol Passwords 16 Features and Functionality Documented 16 v Acquire the BXOFED Software 16 v Acquire the ConnectX 2 Firmware 18 Upgrading the Gateway Firmware 19 v Acquire the Gateway Firmware Package CLI 19 v Upgrade the Gateway Firmware CLI 21 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 Using This Documentation This document provide las
10. supported m pkgname is the name of the firmware package in the transfer directory For example using the FTP protocol gt load source ftp root changeme 123 45 67 99 tmp sundcs_ gw repository _ 2 0 8 1 pkg Downloading firmware image This will take a few minutes Note If you are upgrading from firmware version 2 0 X you can use the force option to disable version number checking and force the upgrade The firmware is downloaded The upgrade begins A caution is displayed and you are asked to commit to the upgrade NOTE Firmware upgrade will upgrade firmware on SUN DCS gw Kontron module I4 and BridgeX Upgrade takes few minutes to complete ILOM will enter a special mode to load new firmware No other tasks should be performed in ILOM until the firmware upgrade is complete Are you sure you want to load the specified file y n 9 Answer y to the prompt to commit to the upgrade The upgrade begins Setting up environment for firmware upgrade This will take few minutes Starting SUN DCS gw FW update I4 fw upgrade from 7 3 0 INI 1 to 7 4 1010 INI 1 Upgrade started Upgrade completed INFO I4 fw upgrade from 7 3 0 INI 1 to 7 4 1010 INI 1 succeeded Sun InfiniBand QDR Gateway Switch Product Notes 23 Performing operation BX A BX fw upgrade from 8 3 3166 INI 4 to 8 4 2812 INI 6 Upgrade started Upgrade completed INFO BX fw up
11. 06 Firmware Revision 0x0103 BIOS version NOW1R113 BIOS date 07 20 2009 FabMan gateway_name gt In the first line of the output is SUN DCS gw version x y z w where x y z w is the version of the firmware upgraded or downgraded For example 2 0 8 1 14 Verify the firmware integrity FabMan gateway_name gt fwverify Checking all present packages LR NS BGR a ete ERE ELE ESR BEB CREAR REREAD LR AS AN Rue ae ee RG ARR RR OK Checking if any packages are missing DST VISA Ek de ere aS ee ER A es Baan twain ee cae LG SS Gog SN Eh teal ah oe te ar ee ee ne OK Verifying installed files chin ere ets Mudie de aba dome Al ae tone atlanta Ncw pue Saat ie Re gow alta ca Ren ES ge Oe a Se OS ne Serbes nee OK FabMan gateway_name gt 26 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013
12. MTU STATE OVER vnet0 phys 1500 up T 14 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 1bp0 phys 65520 up ibp1 phys 65520 down net5 phys 1500 up net4 phys 1500 up net6 phys 1500 unknown 3 Create an IP interface on the data link ipadm create ip lin where link is the name of the data link For example ipadm create ip net6 Note When you create the IP interface on the data link the respective VNIC STATE as seen with the showvnics command goes from WAIT IOA to UP 4 Verify that the interface was created ipadm show if IFNAME CLASS STATE ACTIVE OVER eoib2 ip down no Note You must create the IP interface before an IP address is assigned to it 5 Assign an IP address to the IP interface ipadm create addr T static a I P_address 24 IP_interface v4 where m IP_address is the IP address to be assigned to the IP interface m IP_interface is the interface corresponding to the data link and VIOA For example to assign the IP address of 192 168 17 24 to IP interface eoib2 ipadm create addr T static a 192 168 17 24 24 eoib2 v4 Sun InfiniBand QDR Gateway Switch Product Notes 15 6 Verify that the IP address was properly assigned ipadm show addr P_interface v4 where IP_interface is the interface corresponding to the data lin
13. Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 yY KG S u n Part No E26705 05 a February 2013 Revision A ORACLE Copyright 2012 2013 Oracle and or its affiliates All rights reserved This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws Except as expressly permitted in your license agreement or allowed by law you may not use copy reproduce translate broadcast modify license transmit distribute exhibit perform publish or display any part in any form or by any means Reverse engineering disassembly or decompilation of this software unless required by law for interoperability is prohibited The information contained herein is subject to change without notice and is not warranted to be error free If you find any errors please report them to us in writing If this is software or related software documentation that is delivered to the U S Government or anyone licensing it on behalf of the U S Government the following notice is applicable U S GOVERNMENT RIGHTS Programs software databases and related documentation and technical data delivered to U S Government customers are commercial computer software or commercial technical data pursuant to the applicable Federal Acquisition Regulation and agency specific ja ieee oleae regulations As such the use duplicati
14. an alternative time zone m America Argentina Salta m America Argentina San_Luis Sun InfiniBand QDR Gateway Switch Product Notes 5 America Bahia_Banderas a America Kralendijk m America Lower_Princes America Matamoros m America Metlakatla m America North_Dakota Beulah a America Ojinaga m America Santa_Isabel m America Santarem m America Sitka a Antarctica Macquarie m Asia Ho_Chi_Minh m Asia Kathmandu m Asia Kolkata a Asia Novokuznetsk m Pacific Chuuk m Pacific Pohnpei Firmware Update Considerations If you are going to downgrade the firmware to a version earlier than 2 0 you must remove user partitions and depopulate the Subnet Manager nodes list Refer to Gateway Administration removing partitions for firmware downgrade If you are going to downgrade from firmware 2 0 x to 1 3 4_1 or earlier you might see these type of messages in the var log message file after the downgrade lda Unknown config parameter ErrLogCount 100 lda Unknown config parameter ErrLogTimeInterval 100 The ErrLogCount and ErrLogTimeInterval configuration parameters introduced in firmware 2 0 x are unknown to firmware version 1 3 4 1 and earlier LDAs The LDA logs these messages and ignores them from there on The messages appear once per LDA startup and are harmless If you later upgrade from firmware 1 3 4 1 to 2 0 x and the ErrLogCount and ErrLogTimeInterval configuration parameters are absent the 2 0 x LDA uses the com
15. chip in the HCA must be updated to version 2 7 000 or higher 1 2 Open a web browser on the host that will receive the ConnectX 2 firmware Go to this URL http support oracle com Oracle s My Oracle Support page is displayed Sign in if you already have an account The dashboard page is displayed Note If you do not have an account you must register 4 10 11 12 13 Click the Patches amp Updates tab The Patches amp Updates page is displayed In the Patch Search window click the Product or Family Advanced The Patch Search window updates In the Product Is drop down menu select your HCA For example Sun Dual Port 4x QDR InfiniBand IB HCA PCle ExpressModule In the Release Is drop down menu select the latest firmware version For example FW25408 v2 7 8130 Click outside of the drop down menu In the Platform Is drop down menu select the Oracle Solaris appropriate for your host For example Oracle Solaris on x86 64 64 bit Click outside of the drop down menu Click Search The Patch Search window expands with the search results In the Patch Name column click the respective patch number link For example 12610332 The Patch Search window reformats Click Read Me to display the README file 18 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 14 15 16 17 18 Click Downloa
16. d The File Download window opens Click the filename zip link to initiate the download For example p12610332__Solaris86 64 zip Indicate where the file should be saved The file is downloaded and saved In your receiving directory decompress the filename zip file The ConnectX 2 firmware is in the fw ConnectX2 rel 2_7_version bin file For example fw ConnectX2 rel 2_7_8130 375 3697 01 bin Refer to your HCA documentation for instructions on how to upgrade the ConnectX 2 firmware Upgrading the Gateway Firmware In Gateway Remote Management firmware version numbers are provided as x y X y z and x y z w Currently these numbers are 2 0 2 0 8 and 2 0 8 1 respectively The following two procedures describe how to acquire and upgrade the firmware through the Oracle ILOM CLI V Acquire the Gateway Firmware Package CLI 1 Open a web browser on a host that is on the same Ethernet network as the management controller to receive the firmware update Go to this URL http support oracle com Oracle s My Oracle Support page is displayed Sign in if you already have an account The dashboard page is displayed Note If you do not have an account you must register 4 Click the Patches amp Updates tab The Patches and Updates page is displayed Sun InfiniBand QDR Gateway Switch Product Notes 19 10 11 12 13 14 15 16 17 In the Patch Search window click the
17. d erreurs et vous invite le cas ch ant a lui en faire part par crit Si ce logiciel ou la documentation qui l accompagne est conc d sous licence au Gouvernement des Etats Unis ou toute entit qui d livre la licence de ce logiciel ou l utilise pour le compte du Gouvernement des Etats Unis la notice suivante s applique U S GOVERNMENT RIGHTS Programs software databases and related documentation and technical data delivered to U S Government customers are commercial computer software or commercial technical data pursuant to the applicable Federal Acquisition Regulation and agency specific Pell os regulations As such the use duplication disclosure modification and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract and to the extent applicable by the terms of the Government contract the additional rights set forth in FAR 52 227 19 Commercial Computer Software License December 2007 Oracle America Inc 500 Oracle Parkway Redwood City CA 94065 Ce logiciel ou mat riel a t d velopp pour un usage g n ral dans le cadre d applications de gestion des informations Ce logiciel ou mat riel n est pas con u ni n est destin tre utilis dans des applications risque notamment dans des applications pouvant causer des dommages corporels Si vous utilisez ce logiciel ou mat riel dans le cadre d applications dangereuses il est de votre res
18. ert rule to ipmipet There is no impact to the InfiniBand fabric sometimes does not work Workaround After setting the alert rule to ipmipet set the level to disable and then set the level to the desired value 15693303 BridgeX reset maximizes symbol The setgwinstance setgwsl and setgwsystemname error counter commands reset the BridgeX chip causing symbol errors to maximize Workaround Follow the procedure described in Symbol Errors on Internal Ports on page 7 15696509 IP addresses within the Subnet By default the DHCP server is not assured to assign a Manager nodes file does not match consistent IP address to a Subnet Manager node upon reboot those assigned by DHCP Workaround Manually configure the DHCP server to assign a specific and unique IP address to each Subnet Manager node 15703751 No way to set Fabric Monitor The Fabric Monitor will timeout after 15 minutes of idle time console timeout Workaround You must login to the web interface and launch the Fabric Monitor again Bug Description Workaround 15723028 15747266 15747583 15747938 15747967 15750278 15750534 15755116 BridgeX ports indicate random errors in Fabric Monitor Fabric Monitor does not support Internet Explorer Oracle ILOM web interface not compatible with Mozilla Firefox 6 0 smpartition start command gives unclear message if the Subnet Manager is disabled createvnic command should handle GUIDs with 0x
19. gh X Open Company Ltd This software or hardware and documentation may provide access to or information on content products and services from third parties Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third party content products and services Oracle Corporation and its affiliates will not be responsible for any loss costs or damages incurred due to your access to or use of third party content products or services Copyright 2012 2013 Oracle et ou ses affili s Tous droits r serv s Ce logiciel et la documentation qui accompagne sont pee par les lois sur la propri t intellectuelle Ils sont conc d s sous licence et soumis des restrictions d utilisation et de divulgation Sauf disposition de votre contrat de licence ou de la loi vous ne pouvez pas copier reproduire traduire diffuser modifier breveter transmettre distribuer exposer ex cuter publier ou afficher le logiciel m me partiellement sous quelque forme et par uelque proc d que ce soit Par ailleurs il est interdit de proc der a toute ing nierie inverse du logiciel de le d sassembler ou de le d compiler except a es fins d interoperabilit avec des logiciels tiers ou tel que prescrit par la loi Les informations fournies dans ce document sont susceptibles de modification sans pr avis Par ailleurs Oracle Corporation ne garantit pas qu elles soient exemptes
20. grade from 8 3 3166 INI 4 to 8 4 2812 INI 6 succeeded BX fw upgrade from 8 3 3166 INI 4 to 8 4 2812 INI 6 Upgrade started Upgrade completed INFO BX fw upgrade from 8 3 3166 INI 4 to 8 4 2812 INI 6 succeeded I4 status FW UPDATE SUCCESS I4 update succeeded on A I4 already up to date on none I4 update failed on none BX status FW UPDATE SUCCESS BX update succeeded on A B BX already up to date on none BX update failed on none Performing operation SUN DCS gw firmware update SUN DCS gw Kontron module fw upgrade from 1 3 2 1 to 2 0 8 1 Please reboot the system to enable firmware update of Kontron module The download of the Kontron firmware image happens during reboot After system reboot Kontron FW update progress can be monitored in browser using URL http GWsystem OR at OS command line prompt by using command telnet GWsystem 1234 where GWsystem is the hostname or IP address of SUN DCS GW Firmware update is complete 10 Exit the Oracle ILOM CLI shell gt exit exit 24 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 11 Reboot the gateway to enable the new firmware Refer to Gateway Administration restarting the entire gateway Note The restart process takes between 4 to 5 minutes to complete You can monitor the update progress through m Web browser http
21. izes the Oracle Solaris 11 host correctly If not check the cabling between the gateway and the Oracle Solaris 11 host 7 Determine if a VIOA has already been assigned to the Oracle Solaris 11 host FabMan gateway_name gt showvnics E3 71 mnm34 60 0000 02 02 02 02 02 03 NO ffff E3 71 mnm34 60 0002 02 02 02 02 02 04 NO ffff 3 DISABLED N 00 03 BA 00 01 00 E3 71 mnm34 60 0000 02 02 02 02 02 02 NO ffff ID STATE FLG IOA_GUID 4 UP N 00 03 BA 00 01 00 1 OA ETH 1 5 UP N 00 03 BA 00 01 00 OA ETH 1 OA ETH 1 6 UP N 00 03 BA 00 01 00 OA ETH 1 FabMan gateway_name gt E3 72 mnm34 60 0000 02 02 02 02 02 05 NO ffff Note Until the IP interface is created on the data link using the ipadm command at the Oracle Solaris host the STATE of the VNIC is WAIT IOA 8 Examine the output by column m STATE The state of the VIOA a IOA_GUID The port GUID of the VIOA The GUID might be listed several times m NODE The host name of the node hosting the VIOA m MAC The MAC address assigned to the VIOA There might be several MAC addresses a PKEY The partition key assigned m GW The physical connector cabled to the node or host 9 Determine your next steps a If no port GUID of the Oracle Solaris 11 host is listed or there are no MAC addresses assigned to the port GUID go to Step 10 m Otherwise go to Configure IP Addresses on the Data Link Oracle Solaris
22. k and VIOA For example ipadm show addr eoib2 v4 ADDROBJ TYPE STATE ADDR eoib2 v4 static ok 192 168 17 24 24 SNMP V3 Protocol Passwords In sections of Gateway Remote Management authentication and privacy passwords are described for SNMP service user accounts and SNMP V3 protocol commands For the user accounts and commands the MD5 and SHA authentication passwords are 8 to 12 characters in length and the DES privacy password must be exactly 8 characters long Features and Functionality Documented The features and functionality described in the gateway documentation has been updated to reflect the firmware starting with version 2 0 Upgrading your gateway firmware to the most current version helps increase gateway functionality See Upgrading the Gateway Firmware on page 19 V Acquire the BXOFED Software 1 Open a web browser on a host that will receive the BXOFED software 2 Go to this URL http support oracle com Oracle s My Oracle Support page is displayed 3 Sign in if you already have an account The dashboard page is displayed 16 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 Note If you do not have an account you must register 4 10 11 12 13 14 15 16 17 Click the Patches amp Updates tab The Patches amp Updates page is displayed In the Patch Search window click the Prod
23. ly in the documentation For example the VNIC specific configuration file contains the DEVICE parameter which format is described as DEVICE name where name is suggested to be ethXX and XX is a user specified sequence number VNIC names of eth0 eth1 to eth7 have been problematic with Linux kernels A solution is to use this format for the VNIC name ethgateway_instance_VNIC_instance where m gateway_instance is the gateway instance number as returned by the showgwconfig command a VNIC_instance is a sequential VNIC ID number assigned to the VNIC You can view the VNIC ID numbers with the showvnics command An example VNIC name might be eth03_03 Using this new format the filename for the VNIC specific configuration file becomes ifcfg ethgateway_instance_VNIC_instance For the previous example the configuration filename is ifcfg eth03_03 For consistency the VNIC specific configuration file must contain these two parameters DEVICE name HWADDR mac where m name is ethgateway_instance_VNIC_instance as described previously m mac is the MAC assigned to the VNIC 10 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 Corrections to Creating Virtual I O Adapters Oracle Solaris In Gateway Administration there are several errors in the procedures for creating VIOAs in Oracle Solaris Those procedures are corrected
24. on disclosure modification and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract and to the extent applicable by the terms of the Government contract the additional rights set forth in FAR 52 227 19 Commercial Computer Software License December 2007 Oracle USA Inc 500 Oracle Parkway Redwood City CA 94065 This software or hardware is developed for general use in a variety of information management applications It is not developed or intended for use in any inherently dangerous applications including applications which may create a risk of personal injury If you use this software or hardware in dangerous applications then you shall be responsible to take all appropriate fail safe backup redundancy and other measures to ensure the safe use Oracle orporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications Oracle and Java are registered trademarks of Oracle and or its affiliates Other names may be trademarks of their respective owners AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc UNIX is a registered trademark licensed throu
25. ot 4 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 Software Information and Issues Main Board Management Controller and Chassis Serial Numbers The gateway documentation describes how to retrieve the chassis serial number using the showfruinfo command or the SYS MB Oracle ILOM target These methods actually display the serial number of the main board and the management controller respectively and not the gateway chassis The gateway chassis serial number is provided on the pull out tab on the left side front of the gateway chassis adjacent to power supply 0 High Availability in Partitions To allow communication fail over between HCAs belonging to the same operating system instance the HCA ports must be members of the same partition and have identical membership type full or limited Having both full and limited port memberships within a partition for the same operating system instance creates a configuration instability that might cause subtle communication problems Email Alert Rules You must specify the value of the email_custom_sender property of an email alert rule because the alert does not use the custom_sender property of the SP clients smtp target Time Zone Support The following time zones are only supported in firmware versions 1 3 4 2 0 7 2 and 2 0 8 1 If you upgrade or downgrade to a firmware version other than 1 3 4 2 0 7 2 or 2 0 8 1 you must set
26. piled default values Consequently no LDA messages regarding these configuration parameters are recorded 6 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 Host Software and Firmware To configure and use VNICs from an InfiniBand host the host must have the following software and firmware installed m BXOFED software See Acquire the BXOFED Software on page 16 m ConnectX 2 firmware version 2 7 000 or higher See Acquire the ConnectX 2 Firmware on page 18 Both of these software packages are available from Oracle Symbol Errors on Internal Ports When starting the gateway or when restarting the Bridge Manager bxm there might be 65535 symbol errors on the internal links between the 4 switch chip and the BridgeX chips These errors are identified in hardware commands such as getportcounters or in InfiniBand commands such as ibdiagnet Workaround Manually reset the error counters using the following commands getportcounters 1 R getportcounters 2 R getportcounters 3 R getportcounters 4 R Commands Available to Restricted Linux Shells Typing help all within either the SYS Switch_Diag SYS Gateway_Mgmt or SYS Fabric_Mgmt restricted Linux shells lists the commands available to those shells In the output the checkguidfilesftree command is listed however the command is not supported Conversely the dconfig command is not listed in the o
27. ponsabilit de prendre toutes les mesures de secours de sauvegarde de redondance et autres mesures n cessaires son utilisation dans des conditions optimales de s curit Oracle Corporation et ses affili s d clinent toute responsabilit quant aux dommages caus s par l utilisation de ce logiciel ou mat riel pour ce type d applications Oracle et Java sont des marques d pos es d Oracle Corporation et ou de ses affili s Tout autre nom mentionn peut correspondre des marques appartenant d autres propri taires qu Oracle AMD Opteron le logo AMD et le logo AMD Opteron sont des marques ou des marques co d Advanced Micro Devices Intel et Intel Xeon sont des marques ou des marques d pos es d Intel Corporation Toutes les marques SPARC sont utilis es sous licence et sont des marques ou des marques d pos es de SPARC International Inc UNIX est une marque d pos e conc d e sous licence par X Open Company Ltd Ce logiciel ou mat riel et la documentation qui l accompagne peuvent fournir des informations ou des liens donnant acc s des contenus des produits et des services manant de tiers Oracle Corporation et ses affili s d clinent toute responsabilit ou garantie expresse quant aux contenus produits ou services manant de tiers En aucun cas Oracle Corporation et ses affili s ne sauraient tre tenus pour responsables des pertes subies des co ts occasionn s ou des dommages caus s par l acc s des contenus
28. t minute information regarding the installation administration and service of the Sun Network QDR InfiniBand Gateway Switch from Oracle m Related Documentation on page v m Feedback on page v m Access to Oracle Support on page vi Related Documentation Documentation Links Sun Network QDR InfiniBand Gateway Switch http docs oracle com cd E26699_01 Firmware Version 2 0 Oracle Solaris 11 OS http www oracle com goto Solaris11 docs Oracle Integrated Lights Out Manager ILOM 3 0 http docs oracle com cd E19860 01 All Oracle products http docs oracle com Feedback Provide feedback about this documentation at http www oracle com goto docfeedback Access to Oracle Support Oracle customers have access to electronic support through My Oracle Support For information visit http www oracle com pls topic lookup ctx acc amp id info or http www oracle com pls topic lookup ctx acc amp id trs visit if you are hearing impaired vi Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 Sun InfiniBand QDR Gateway Switch Product Notes These product notes provide last minute late breaking information regarding the gateway These notes pertain to the release of the 2 0 8 1 firmware for the gateway Known Problems The following table describes known problems with the gateway Bug Description Workaround 15560533 Setting an al
29. the gateway firmware See Upgrade the Gateway Firmware CLI on page 21 V Upgrade the Gateway Firmware CLI Note If you are going to downgrade the firmware to a version earlier than 2 0 you must remove user partitions and depopulate the Subnet Manager nodes list Refer to Gateway Administration removing partitions for firmware downgrade 1 Open an SSH session as user root and connect to the management controller by specifying the controller s host name For example ssh 1 root gateway_name root gateway_name s password password where gateway_name is the host name of the management controller Initially the password is changeme 2 If the Subnet Manager is running on the management controller disable the Subnet Manager with the disablesm command disablesm Stopping partitiond daemon OK Stopping IB Subnet Manager OK 3 Verify that there is at least 150 MB available in the tmp directory df h tmp Filesystem Size Used Avail Use Mounted on tmpfs 250M 240K 249M 1 tmp In this example there are 249 MB available If not enough space is available you must delete files from the tmp directory Sun InfiniBand QDR Gateway Switch Product Notes 21 4 Verify that there is at least 1 MB available in the config directory df h config Filesystem Size Used Avail Use Mounted on dev hda2 16M 3 6M 11M 25 config In this example there are
30. tifying the state of the membership bit There is no workaround From the command usage the clear option of the setgwethport command should restore to default parameters This situation does not happen Workaround Use the clear option Using the help all command within the Oracle ILOM SYS Fabric_Mgmt target displays the checkguidfilesftree command This command has been deprecated and is not available 2 Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 Bug Description Workaround 15755133 15755727 15756465 15758329 15758406 15761792 15819653 fdconfig command is unavailable from the SYS Fabric_Mgmt target Need tool to display InfiniBand topology with physical InfiniBand entities Usage of deletevnic command is wrong Stopping or starting the envd daemon initiates a segmentation fault for the FabricMIB Fabric Monitor might fail to load BridgeX manager dies with a base port id collision detected error message Gateway hard hangs after reboot The dconfig command is required for configuring the FabricMIB and should be available from the Oracle ILOM SYS Fabric_Mgmt target The command is not available Workaround Log in as the root user where the fdconfig command is available At present no command provides the relationship of InfiniBand fabric GUIDs LIDs and IB ports to each other in a simplified manner Workaround
31. uct or Family Advanced The Patch Search window updates In the Product Is field type BridgeX Possible products are suggested Click the most appropriate link The Release Is field might autopropogate with the most current version In the Release Is drop down menu select the most current version of the BridgeX OFED software For example BridgeX OFED 1 5 1 Click outside of the drop down menu Click Search The Patch Search window expands with the search results In the Patch Name column click the patch number link respective to your platform For example 12621910 The Patch Search window reformats Click Read Me to display the README file Click Download The File Download window opens Click the filename zip link to initiate the download For example p1i2621910 151 Linux x86 64 zip Indicate where the file should be saved The file is downloaded and saved In your receiving directory decompress the filename zip file The BXOFED software is in the BXOFED 1 5 1 version_for Oracle tgz file There are also readme release notes installation guide and user manual files in the filename zip file Read the README release notes and installation guide files for information on how to install the BXOFED software Sun InfiniBand QDR Gateway Switch Product Notes 17 V Acquire the ConnectX 2 Firmware For your host to properly interface with the gateway the firmware of the ConnectX 2
32. upplies and then reattaching them Sun InfiniBand QDR Gateway Switch Product Notes 3 General Information and Issues Reserved VLAN Identifiers When creating VLANs use VLAN identifiers 2 through 4094 According to the IEEE 802 1Q specification VLAN ID 0 is used for the priority tag and VLAN ID 1 is usually reserved for a switch or bridge management VLAN Jumbo Frames Consider the following issues for Jumbo Frames m True Jumbo Frames are not supported a The maximum MTU is limited by the MTU of the InfiniBand fabric a The MTU of the HCA is 2048 by default but can be increased up to 4096 m The default MTU of the I4 switch chip and BridgeX chips is 4096 The gateway supports Jumbo Frames up to the InfiniBand limit of 4096 bytes You must configure the entire InfiniBand fabric to use a 4096 MTU This configuration includes configuring the Subnet Manager InfiniBand devices and the ConnectX HCAs You can configure a ConnectX HCA by activating the set_4k_mtu parameter of the HCA s mlx4_core module Subnet Manager Access to the VNICs through the gateway requires an active Subnet Manager for the InfiniBand fabric The management controller within the gateway is already configured with a Subnet Manager You can enable the Subnet Manager with the enablesm command Snapshot Dataset Information The normal fruid and full datasets of the snapshot utility are currently equivalent and contain the same data in the snapsh
33. utput but is supported by the SYS Fabric_Mgmt restricted Linux shell Sun InfiniBand QDR Gateway Switch Product Notes 7 Hardware Information and Issues Ethernet Support The gateway supports 10 Gb sec Ethernet networks The following table lists supported hardware to construct one splitter cable for 10 GbE networks Description Part Number Quantity Needed Optical splitter cable X2127A 1 10 GbE QSFP transceiver X2124A 1 10 GbE SFP transceiver X2129A 4 Number of LAGs The gateway supports a maximum of 16 LAGs Unusable Ports The ports covered by the Do Not Remove plastic tab are unusable at the time of this document Do not remove the tab Documentation Information and Issues smpartition Command Guidelines When partitioning the InfiniBand fabric with the smpartition command consider these guidelines during the configuration session Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2 0 February 2013 Valid P_Keys for partitioning are 0x1 through 0x7f f However both 0x1 and 0x7fff are predefined P_Keys and cannot be created or deleted with the smpartition command Default entries ports in the predefined partitions cannot be removed However when you add ports into the predefined partitions using the smpartition add command the added ports take precedence over the default ALL_SWITCHES and ALL_CAS port groupings The SELF and ALL port groupings are not supported b
34. y the smpartition command To specify all ports for a partition use the ALL_SWITCHES port grouping in conjuction with the ALL_CAS port grouping in the smpartition add command To remove the ipoib flag from a partition use the smpartition modify flag command without any additional arguments The command removes the ipoib flag successfully but erroneously issues this warning Missing port parameter Could not modify partition If you use EolB and VNIC functionality in a nondefault partition you must specify the appropriate BridgeX chips InfiniBand port GUIDs to the partition When you know the VNIC s Ethernet connector this table provides the respective BridgeX chip s portname Ethernet Connector BridgeX Portname OA ETH 1 Bridge 0 2 OA ETH 2 Bridge 0 2 OA ETH 3 Bridge 0 1 OA ETH 4 Bridge 0 1 1A ETH 1 Bridge 1 2 1A ETH 2 Bridge 1 2 1A ETH 3 Bridge 1 1 1A ETH 4 Bridge 1 1 When you know the Portname you can look in the output of the showgwports command for the respective PortGUID Using the table and the showgwports command together when you know the Ethernet connectors used for your VNICs you can find the respective BridgeX chip s InfiniBand port GUIDs to add to the nondefault partition Sun InfiniBand QDR Gateway Switch Product Notes 9 VNIC Configuration on Linux Hosts in Gateway Manual Mode If you are creating VNICs in Gateway Manual Mode you must follow these conditions VNIC names appear frequent
Download Pdf Manuals
Related Search