Home

Mellanox Firmware Tools (MFT) User Manual

1. Switch System Je i n A A gt d i 4 l o leJis ls ole bo I I I l 4 I J gt M M E E N R Note In Band capabilities are not supported by Mellanox Ethernet Adapters 1 1 Supported Operating Systems Please refer to the release notes of your version for supported platforms and kernels Unless explicitly specified the usage of the tools is identical for all operating systems Mellanox Technologies 11 Rev 2 4 Introduction 1 2 MFT Access to Hardware Devices Table 1 lists the Mellanox devices supported by MFT the supporting tools and the access meth ods to these devices Table 1 Supported Mellanox Devices HW Access Method Device Type Product Name PCI RC In Band HCA InfiniBand MT2760xA0 Connect IB V V V VPI Network Adapter MT27508 ConnectX 3 V V V MT27528 ConnectX 3 Pro V V V MT27708 ConnectXG 4 V V V Ethernet Adapter NIC MT25408 ConnectX EN V V MT25408 ConnectX 2 EN MT27508 ConnectX 3 EN Switch MT58100A0 SwitchX amp y V V MT48436 InfiniScale amp IV y2 V V Switch IBTM v2 V V Bridge MT64102 Bridgex v Y Y 1 For managed switch products only MFT tools access Mellanox devices via the PCI Express interface via a USB to I2C adapter Mellanox P N MTUSB 1 or via vendor specific MADs over the InfiniBand fabric In Band In Band device access requires
2. In order to print the firmware traces it is required to Debug firmware is burnt and loaded on the device e The driver is up meaning For adapters with on board memory The SYS ENABLE command has been executed For adapters without on board memory MemFree The RUN FW command has been executed The desired trace mask is set see the m flag below The mst driver must be started prior to running itrace tool To start itrace 1 Start the mst driver mst start or mst restart 2 Enter the following command itrace options IRISC NAME where IRISC NAME The iRISC for which traces are to be printed This can be specified once anywhere in the command line as a special option without the leading hyphen Run itrace h to get a list of iRISC names for each adapter device h help Displays help about itrace usage m mask TRACE MASK Sets the Trace Mask To enable generating trace messages for an iRISC the trace mask register must be set according to the specifications in the device s Programmer s Reference Manual Setting or clearing bits of the trace mask register enables or disables respectively the generation of specific types of trace messages The TRACE MASK parameter must be a hexadecimal or decimal number and its value will be written into the trace mask register Changing trace mask will not change or remove messages previously stored in the trace buffer so disabled types of messages can still be displ
3. ence between the running firmware and the firmware we are upgrading to de mixfwreset Synopsis mlxfwreset d device level lt 0 5 gt y query reset where d device device Device to work with 1 level lt 0 5 gt Run reset with the specified reset level y yes Answer yes on prompt v version Print tool version h help Show help message and exit query Query for reset level required to load new firmware reset Execute reset Level Reset Levels Reset levels depends on the extent of the changes introduced when updating the device s firm ware The tool will display the supported reset levels that will ensure the loading of the new firm ware Those reset levels are 1 In Service Firmware Update ISFU allows for a smooth firmware upgrade Mellanox Technologies 63 Rev 2 4 Firmware Generation Configuration and Update Tools 0 Full ISFU 1 Driver restart link management will remain up 2 Driver restart link management will be down 3 Driver restart and PCI reset 4 Warm reboot 5 Cold reboot performed by the user Examples of mlxfwreset Usage 1 To query device reset level after firmware update use the following command line Supported reset levels for loading firmware on device dev mst mt4113_pciconf0 Example 2 To reset device in order to load new firmware use the following command line 64 Mellanox Technologies
4. with_msix with_un known with_i2cdev with_Ipcdev Start mst server to allow incoming connection Default port is 23108 Stop mst server Establish connection with specified host on specified port default port is 23108 Add devices on remote peer to local devices list lt hostname gt may be host name as well as an IP address Remove all remote devices on specified hostname lt hostname gt port should be specified exactly as in the mst remote add command 16 Mellanox Technologies Rev 2 4 mst ib add OPTIONS Add devices found in the IB fabric for inband access local_hca_id local_h Requires OFED installation and an active IB link ca_port If local_hca_id and local_hca_port are given the IB subnet connected to the given port is scanned Otherwise the default subnet is scanned Options e discover tool lt discover tool gt The tool that is used to dis cover the fabric Supported tools ibnetdiscover ibdiagnet default ibdiagnet e add non mlnx Add non Mellanox nodes e topo file topology file A prepared topology file which describes the fabric For ibnetdiscover provide an output of the tool For ibdiagnet provide LST file that ibdiagnet generates e use ibdr Access by direct route MADs Available only when using ibnetdiscover tool for SwitchX and ConnectIB devices NOTE if a topology file is specified device are taken from it Otherwise a discover tool is run to disco
5. FPGA mst device interface v version Display version info h help Display help message r read lt addr gt Read debug register in address w write data lt addr gt Write data to debug register in address rr read reg name Read debug register with name wr write reg data name Write data to debug register with name b burn rbf file gt Burn image on flash read all Read all debug registers show all Show all debug registers clear semaphore Unlock flash controller semaphore reset Reset flash controller 68 Mellanox Technologies Rev 2 4 Examples of mlx_fpga Usage Adding FPGA mst Device Interface apps 13 mst start with fpga apps 13 mst status ST modules ST PCI module is not loaded ST PCI configuration module is not loaded ST devices No MST devices were found nor MST modules were loaded You may need to run mst start to load MST modules FPGA devices dev mst 0000 82 00 0 fpga Burning the FPGA s flash device using the mlnx fpga burning tool minx fpga tool burns a rbf file onto the FPGA flash device The rbf file must be generated according to the instructions listed in the relevant programmable adapter card User Manual Run the following command to burn the rbf file mlx fpga d device burn image filename rbf Debugging the Tool Reading One Debug Register mlx fpga d device read 0x0 Writing One Debug Register mlx fpga d device write 0x0 0x0
6. 10 10 3000 FW Release Date 29 4 2014 Description UID GuidsNumber Step Base GUID1 0002c903002e 500 8 1 Base GUID2 0002c903002ef508 8 1 Base MAC1 00000002c92ef500 8 1 Base MAC2 00000002c92ef508 8 1 Image VSD Device VSD VSD PSID MT 1240110019 flint i tmp connect ib bin uid 000123456abcd sg Updating GUID section OK Updating ITOC section OK Restoring signature OK flint i tmp connect ib bin q Image type FS3 FW Version 10 10 3000 FW Release Date 29 4 2014 Description UID GuidsNumber Step Base GUID1 000000123456abcd 8 1 Orig Base GUID1 0002c903002ef500 8 Base GUID2 000000123456abd5 8 Orig Base GUID2 0002c903002ef508 8 Base MAC1 000000000056abcd 8 Orig Base MAC1 00000002c92ef500 8 Base MAC2 000000000056abd5 8 Orig Base MAC2 00000002c92ef508 8 Image VSD Device VSD VSD PSID MT 1240110019 4 4 Change GUIDs and MACs for the ConnectX 4 device flint d dev mst mt4115 pciconf0 guid e41d2d0300570fc0 mac 0000e41d2d570fc0 ocr sg W Firmware flash cache access is enabled Running in this mode may cause the firmware to hang Updating GUID section OK Updating ITOC section OK Restoring signature OK flint d dev mst mt4115 pciconf0 q Image type FS3 FW Version 12 0100 5630 FW Release Date 23 3 2015 Description UID GuidsNumber Mellanox Technologies 49 Rev 2 4 Firmware Generation Configuration and Update Tools Base GUID e41d2d0300570
7. Connect IB Expansion ROM command limitation note Mellanox Technologies 7 Rev 2 4 Table 1 Revision History Table Sheet 3 of 3 Date Revision Description July 2013 1 80 Reorganized the Firmware Tools and Utilities section Added the following sections e Section 3 6 mlxfwreset Loading Firmware on 5th Generation Devices Tool on page 63 Section 4 1 fwtrace Utility on page 71 Section 4 11 mlxuptime Utility on page 83 Appendix F Accessing Remote InfiniBand Device by Direct Route MADs on page 99 Appendix F Accessing Remote InfiniBand Device by Direct Route MADS on page 99 Appendix G Update Package for Mellanox Firmware on page 101 April 2013 1 70 Added the following sections e Section Mellanox Connect IB Switch IBTM and ConnectX 4 Initial Burning Options on page 59 Section Connect IBG Examples on page 61 Section 4 2 itrace Utility on page 73 Section 4 8 mlxdump Utility on page 80 Section 4 7 mlxtrace Utility on page 78 e Section 4 9 mlxmcg Utility on page 80 e Section 4 10 pckt drop Utility on page 82 e Appendix E Remote Access to Device by Sockets Removed the following sections e Section 1 2 Software Prerequisites Section 2 1 4 Exit Return Values Updated the following sections Section mlxburn Synopsis on page 56 Section Examples of mlxburn Usage on p
8. Press C to skip I Done I Tracer was configured successfully Device frequency 276MHz Starting event streaming Reading new events 774774193803 I2 Mad received on port 1 QP 0 774774215444 I2 process set get pkey table on port 1 set get 0 block 1 774775079296 I2 Mad received on port 1 QP 0 774775120645 I2 port state changed from INIT to ARM 774775166315 I2 process set get port info on port 1 set get 1 status 0x0 774775335890 172 Mad received on port 1 QP 0 774775367205 I2 port state changed from ARM to ACTIVE 774775410880 I2 process set get port info on port 1 set get 1 status 0x0 774786733806 I3 process MAD IFC on port 1 774786744859 I3 process set get port info on port 1 set get 0 status 0x0 72 Mellanox Technologies Rev 2 4 4 2 itrace Utility The itrace utility extracts and prints trace messages generated by the firmware of a ConnectX family adapter devices These trace messages inform developers of software drivers about inter nal status events critical errors etc for each iRISC Trace messages generated by iRISCs are stored in the trace buffer The trace buffer is located in host memory for MemFree adapter cards i e without on board memory and in adapter memory for adapter cards with on board mem ory The utility is a command line application controlled by command line parameters It prints trace messages in text format to the console 4 2 1 itrace Usage
9. c0 4 Base MAC 0000e41d2d570 c0 4 Image VSD Device VSD PSID MT 2190110032 add note GUIDs and MACs can be changed separately on ConnectX4 3 4 5 3 Preparing a Binary Firmware Image for Pre assembly Burning In some cases OEMs may prefer to pre burn the flash before it is assembled on board To gener ate an image for pre burning for 4th generation devices ConnectX and newer use the mlx burn striped image flag The striped image file layout is identical to the image layout on the flash hence making it suitable for burning verbatim When pre burning the GUIDs MACS inside the image should be unique per device The follow ing are two methods to pre burn an image You can choose the best method suitable for your needs Method 1 Pre burn an Image with Blank GUIDs MACs In this method the image is generated with blank GUIDs and CRCs The GUIDs are set after the device is assembled using the flint sg command To set GUIDs take less than 1 second when running on an image with blank GUIDs through a PCI device A device that is burnt with blank GUIDs MACS will not boot as a functional network device as long as the GUIDs MACS are not set de To pre burn an image with blank GUIDs MACSs 1 Generate a striped image with blank GUIDs mlxburn fw fw ConnectX3 rel mlx MCX354A FCB A2 A5 ini wrimage fw ConnectX3 rel bin striped image blank guids I Generating image I Image generation completed su
10. lt image name gt rom Mellanox Technologies 43 Rev 2 4 Firmware Generation Configuration and Update Tools 3 4 5 3 4 5 1 Example flint d dev mst mt4099 pci cr0 rrom example mrom flint d dev mst mt4099 pci cr0 q Image type FW Version Rom Info Device ID Description GUIDs MACs VSD PSID FS2 Z5 FW Release Date 4 5 2014 type PXE version 3 4 225 devid 4099 proto VPI 4099 Node Portl Port2 Sys image 45214030001b8a0 f 45214030001b8al 45214030001b8a2 45214030001b8a3 4521401b8al 4521401b8a2 MT 1090120019 To remove the Expansion ROM use the following command flint d mst device drom Example flint d dev mst mt4099 pci cr0 drom Removing ROM image OK Restoring signature OK Setting GUIDs and MACs To set GUIDs MACS UIDs for the given device use the sg set guids command with the guid s uid s and or mac s flags 4th Generation Devices On 4th generation devices the sg command can operate on both the image file and the image on the flash When running the sg command on an image on the flash if the GUIDs MACs UIDs in the image are non blank the flint will re burn the current image using the given GUIDs MACS UIDs 1 Change the GUIDs MACS on a device flint d dev mst mt4099 pciconf0 q W Running quick query Skipping full image integrity checks Image type FW Version FW Release Date Device ID
11. port may cause the system to cease working To solve this issue 1 Disable SRIOV in bios 2 Reboot server 3 Change num of VFs 4 Enable SRIOV in bios Mellanox Technologies 87 Rev 2 4 Troubleshooting 5 2 5 3 88 Mellanox Technologies Installation Related Issues Issue Cause Solution Unable to install MFT pack Insufficient privileges 1 Copy the MFT package to tmp vmware and age on ESXi platform and continue with the installation If the issue the following message is printed on the screen Got no data from pro cess persists reboot the ESX server and try again 2 Use full file path of the MFT package Note an additional reboot will be required after completing the installation Firmware Burning Related Issues Issue Cause Solution The following message is printed on screen when try ing to query burn a Connect IB device E Cannot open Device dev mst mt4113 pciconfO B14 Operation not permit ted MFE CM DIF GO BIT BUSY Using an outdated firmware version with the Connect IB adapter 1 Unload MLNX_OFED driver etc init d openibd stop 2 Add ocr option to the flint command For example flint d dev mst mt4113 pciconf0 ocr q The following message is reported on screen when try ing to remove the expansion ROM using the drom option E Remove ROM failed The device FW contains common FW ROM Product Version The
12. 1 sfx extract dir mydir sfx extract only Extracting to mydir mlxfwmanager 20130717 1 Done Run the firmware update command mydir mlxfwmanager 20130717 1 mlxfwmanager u Mellanox Technologies 105 Rev 2 4 Troubleshooting Appendix H Secure Host Feature Secure host is the general term for the capability of a device to protect itself and the subnet from malicious software through mechanisms such as blocking access of untrusted entities to the device configuration registers directly through pci_cr or pci_conf and indirectly through MADs WARNING A Once a hardware access key is set the hardware can be accessed only after the correct key is provided e Ifakey is lost please refer to Appendix H 1 5 Recover Lost Key on page 107 The hardware access in this mode is allowed only if a correct 64 bits key is provided d The secure host feature requires a MLNX_OFED driver installed on the machine de H 1 Using the Secure Host H 1 1 Generating Burning a Firmware supporting Secure Host Feature 1 Make sure you have INI and mlx files suitable for the device Both files are available for download at http www mellanox com page custom firmware table a Addcr protection en true under HCA section in the INI file b Generate an image using mlxburn for example run mlxburn fw fw 4099 rel mlx conf secure host ini wrimage fw 4099 secure bin 2 Burn the image on the device using flint fl
13. 4 0460 Update required Found 1 device s requiring firmware update Device 1 Release notes Version 2 33 5000 contains the following features bug fixes 1 Virtual QoS support 2 RX buffer optimizations for DSCP mode 3 SMBUS ARP support 4 RDMA Retransmission optimization 5 NVCONFIG UAR BAR change support 6 Sideband connectivity improvements IPMI NCSI For full list of features and bug fixes please see full release notes at ConnectX3 http www mellanox com pdf firmware ConnectX3 FW 2 33 5000 release notes pdf ConnectX3Pro http www mellanox com pdf firmware ConnectX3Pro FW 2 33 5000 release notes pdf Please wait while downloading MFA s 100 Device 1 Updating FW Done Restart needed for updates to take effect 28 Mellanox Technologies Rev 2 4 3 2 3 4 Downloading Firmware Images and Firmware Update Packages To download firmware images firmware update packages use the following command line mlxfwmanager download DownloadDir download device DeviceType download os OS download type lt DownloadType gt Example a Downloading Firmware Images Firmware Update Packages mlxfwmanager download tmp DownloadDir download device ConnectX download os All download type self extractor Files 0 mlxfwmanager linux ConnectX3 X3Pro 20140811 1 mlxfwmanager windows ConnectX3 X3Pro 20140811 exe 2 mlxfwmanager linux6 4 ConnectX3 X3Pro 20140811 3 mlxf
14. Ax bin d dev mst SW MT52000 000011111101a24c lid 0x0006 mlx4 0 1 ConnectX 4 Examples e To generate a failsafe image file for ConnectX 4 device without burning enter mlxburn fw FW fw ConnectX 4 mlx conf FW MCX456A ECA Ax ini wrimage fw ConnectX 4 MCX 456A ECA Ax bin base guid 0x0002c903002ef500 e To update firmware on a ConnectX 4G device enter mlxburn i fw ConnectX 4 MCX456A ECA Ax bin d dev mst mt4113 pciconf0 Mellanox Technologies 61 Rev 2 4 Firmware Generation Configuration and Update Tools Exit Return Values The following exit values are returned e 0 successful completion gt 0 an error occurred 62 Mellanox Technologies Rev 2 4 3 6 mlxfwreset Loading Firmware on 5th Generation Devices Tool mlxfwreset Mellanox firmware reset tool enables the user to load updated firmware without having to reboot the machine mlxfwreset supports 5th Generation HCAs with ISFU supported Firmware It is supported only on Linux Tool Requirements e OFED driver to be installed and enabled e Access to device through configuration cycles pciconf Firmware supporting ISFU e Version 10 10 3000 or above for Connect IB e Version 12 0100 0000 or above for ConnectX 4 e Device s firmware updated with latest MFT burning tools Flint MIxburn Mlxfwman ager Supported devices Connect IB and ConnectX 4 Exact reset level needed to load new firmware may differ as it depends on the differ
15. Choose printed TS format hex dec possible values hex dec default dec print delta Enable printing delta between events in cycles default off f print file FilePath Print parsed event to the given file and not to stdout enable db check Enable events DB checks default off Examples 1 To generate a configuration file run the following command mlxtrace create cfg tmp 2 Choose the suitable cfg file depending on the device you are using and run the following command to generate a trc file mlxtrace d dev mst mt4099 pci cr0 c tmp mlxtrace_cx3 cfg m MEM o connectx3 trc 3 To generate a trc file with a maximal size of 100 MB run the following command mlxtrace d dev mst mt4099 pci cr0 c tmp mlxtrace cx3 cfg m MEM s 100 o connectx3 trc Mellanox Technologies 79 Rev 2 4 Debug Utilities 4 8 4 8 1 4 9 4 9 1 mlxdump Utility The mlxdump utility dumps device internal configuration data and other internal data such as counters state machines The data can be used by for hardware troubleshooting It can be applied to all Mellanox adapter devices BridgeX device and 4th generation switch devices The tool has 3 run modes fast normal full while the default is fast the full mode dumps all available data but might run slower than normal and fast modes mlxdump Usage The mst driver must be started prior to running mlxdump tool gt mlxdump d mst dev sn
16. Configures port 2 working only ConnectX 4 mode 1 For supported devices only listed in this section P The default value for the parameters listed in the table above is firmware dependent o Before setting the number of VFs in SR IOV please make sure your system can support that num ber of VFs If your hardware and software cannot support that number this may cause your sys P tem to cease working Therefore mlxconfig protects the user by making sure that when setting adl SR IOV parameters the value of NUM_OF_VFS PCI_BAR_SIZE must not exceed 512 Also NUM OF VFS must not exceed the limit defined by the firmware 127 VFs upper bound The same calculation applies to BAR size settings 32 Mellanox Technologies Rev 2 4 D PCI BAR SIZE refers to the PCI BAR size per function either physical or virtual Troubleshooting o Examples of mlxconfig Usage Querying Current Device Configuration To query the current device configuration use the following command line Example Mellanox Technologies 33 Rev 2 4 Firmware Generation Configuration and Update Tools Setting Device Configuration To set the device configuration use the following command line mlxconfig d device set Parameters Example mlxconfig d dev mst mt4099 pciconf0 set WOL MAGIC EN P2 1 NUM OF VFS 24 Device type ConnectX 3 PCI device dev mst mt4099 pcionf0 Configurations Current New
17. Description GUIDs ACs VSD PSID FS2 Bho Sah 5050 4 5 2014 4099 Node Portl Port2 Sys image 45214030001b8a0 45214030001b8al 45214030001b8a2 45214030001b8a3 4521401b8al 4521401b8a2 MT_1090120019 flint d dev mst mt4099 pciconf0 guid 0x452140300abadaba mac 0x300abadaba sg 44 Mellanox Technologies Rev 2 4 W GUIDs are already set re burining image with the new GUIDs You are about to change the Guids Macs Uids on the device New Values Current Values Node GUID 452140300abadaba 45214030001b8a0 Portl GUID 452140300abadabb 45214030001b8a1 Port2 GUID 452140300abadabc 45214030001b8a2 Sys Image GUID 452140300abadabd 45214030001b8a3 Portl MAC 00300abadaba 4521401b8al Port2 MAC 00300abadabb 4521401b8a2 Do you want to continue y n n y Burning FS2 FW image without signatures OK Restoring signature Qi flint d dev mst mt4099 pciconf0 q Image type FS2 FW Version 2 31 5050 FW Release Date 4 5 2014 Device ID 4099 Description Node Portl Port2 Sys image GUIDs 452140300abadaba 452140300abadabb 452140300abadabc 452140300abadabd MACs 00300abadaba 00300abadabb VSD PSID MT_1090120019 2 Change the GUIDs MACs on an image file flint i tmp image bin q Image type fs2 FW Version 2 31 5050 FW Release Date 4 5 2014 Device ID 4099 Description Node Portl Port2 Sys image GUIDs 45214030001b8a0 f45214030001b8al 45214030001b8a2 45214030001b8a3 M
18. MEM buffer size in MB dump Dump file name m mask Trace class mask use to enable multiple classes or use integer format e g m class1 class2 or OxffOOff00 1 level Trace level v version Print tool s version and exit Mellanox Technologies 71 Rev 2 4 Debug Utilities Device Specific Info e Connect IB and ConnectX 4 Irisc names 10 iron i2 13 14 15 16 17 all Trace classes DEBUG INIT INIT ICM ICM FREE LIST ICM BLOCK ALLOC CMD IF PHY IB PHY RX ADAP PHY EYE OPN PHY COMMON PHY MANAGER PHY PLL BLOCK ALLOC ICM ACCESS MAD RXT CHECKS I2C TRANSPORT FW LL RX ER RORS DEBUG TRACER PROFILING MANAGEMENT FLASH STEERING Example fwtrace d mlx5 0 i all s I Found FW string db cache file going to use it mlxtrace d mlx5 0 m MEM c tmp itrace 8153 cfg S I Tracer Configuration E Mode Collector I Activation Mode Memory Mode I Memory Access Method NA I Configuration File Path tmp itrace 8153 cfg I Output file Trace File Path mlxtrace tre I User Buffer Size NA MBytes Use Stream Mode YES I Configure Only NO I Only Snapshot Skip Configuration Stage NO T gt Continuous fill NO 1 gt Print timestamp in hh mm ss nsec format NO 1 gt Output file for streaming STDOUT I Delay between samples 0 usec I Device is cib I Configuring Tracer I Invalidating kernel buffer
19. MTUSB 1 USB to I2C Bridge Success gt To stop the service mst stop Stopping MST Mellanox Software Tools driver set Unloading MST PCI module Success gt To print the current status of Mellanox devices mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST devices dev mst mt4099 pciconf0 PCI configuration cycles access domain bus dev fn 0000 0b 00 0 addr reg 88 data reg 92 Chip revision is 01 dev mst mt4099 pci cr PCI direct access domain bus dev fn20000 0b 00 0 bar 0xd2600000 Size 0x100000 Chip revision is 01 dev mst mtusb 1 USB to I2C adapter as I2C master iSerial 0x1683 To show the devices status with detailed information mst status v PCI devices DEVICE TYPE MST PCI RDMA NET NUMA ConnectX4 rev 0 dev mst mt4115 pciconf0 08 00 0 mlx5 0 net ib2 1 08 00 1 mlx5 1 net ib3 zi ConnectIB rev 0 dev mst mt4113 pciconf0 0b 00 0 mlx5 2 net ib4 1 net ib5 ConnectX3 rev 1 dev mst mt4099 pciconf0 ConnectX3 rev 1 dev mst mt4099 pci cr0 0e 00 0 mlx4 0 net ib0 1 net ibl I2C devices MST Serial dev mst mtusb 1 Ox1B5C 18 Mellanox Technologies Rev 2 4 In case the device has Function Per Port FPP enabled on it the physical functions information will appear under the same mst device Example DEVICE TYPE MST PCI RDMA NET NUMA ConnectX4 rev 0 dev mst mt4115 pciconf0 08 00 0 mlx5 0 net ib2 1 08 00 1 mlx5 1 n
20. Rev 2 4 Example 3 Driver restart and PCI reset Continue with reset y N y I Stopping Driver Done I Sending Reset Command To Fw Done I Resetting PCI Done I Starting Driver Done I Restarting MST Done I FW was loaded successfully When running the reset command without specifying a reset level the minimal reset level will be performed 3 To reset a device with a specific reset level to load new firmware use the following command line 4 mlxfwreset d dev mst mt4113 pciconfO l 4 reset Example Requested reset level for device dev mst mt4113 pciconf0 4 Warm Reboot Continue with reset y N y I Sending reboot command to machine The system is going down for reboot NOW 3 6 1 mlxfwreset Limitations The following are the limitations of mlxfwreset e Executing a reset level that is lower than the minimal level as shown in query com mand will yield an error and a reboot will be required in order to load the new firm ware Attempting to run the tool again will return an error When updating the firmware of a device using flint mlxburn mlxfwmanager it is required for the burning tool to update the new firmware in an ISFU manner in order to be able to run this tool The user should make sure that the message I To load new FW run mlxfwreset or reboot machine is present in the burning tool s output e After a successful reset execution attempting to query or reset again will yield a
21. SRIOV EN ji 1 NUM OF VFS 16 24 WOL MAGIC EN P1 0 0 WOL MAGIC EN P2 0 dL Apply new Configuration y n n y Applying Done I Please reboot the system to load new configurations Resetting Device Configuration to Default To reset the device configuration to default use the following command line mlxconfig d lt device gt reset Example mlxconfig d dev mst mt4099 pciconf0 reset Reset configuration for device dev mst mt4099 pciconf0 y n n y Applying Done I Please power cycle device to load new configurations mlxconfig d dev mst mt4099 pciconf0 query Device 1 Device type ConnectX 3 PCI Device dev mst mt4099 pciconf0 Configurations Current SRIOV EN 1 NUM OF VFS 8 WOL MAGIC EN P1 0 WOL MAGIC EN P2 0 34 Mellanox Technologies Rev 2 4 3 3 1 Using mlxconifg with PCI Device in Bus Device Function BDF Format In order to access device in BDF format via configuration cycles use pciconf as prefix to the device Example mlxconfig d pciconf 000 03 00 0 q Device 1 Device type ConnectX 3 PCI Device pciconf 000 03 00 0 Configurations Current SRIOV EN il NUM_OF VFS 16 WOL MAGIC EN P1 0 WOL MAGIC EN P2 0 3 3 2 Using mlxconfig to Set VPI Parameters In order to set VPI parameters through mlxconfig use the following command line mlxconfig d device set LINK TYPE Pl link type LINK TYPE P2 link type gt Example Configuring both ports as InfiniB
22. Showing all debug registers apps 13 4 mlx fpga d dev mst 0000 82 00 0 fpga show all thernet ethernet hca ethernet port ips mac rx mac config rx transfer control thernet ethernet hca ethernet port ips mac rx mac config rx transfer status thernet port ips mac rx mac config rx padcrc control thernet ethernet hca ethernet port ips mac rx mac confi S thernet ethernet hca e rx crccheck control S e e e thernet ethern e t count msb t hca ethernet port ips mac rx mac config rx overflow truncated pack thernet ethernet hca ethernet port ips mac rx mac config rx overflow truncated pack t count lsb thernet ethernet hca ethernet port ips mac rx mac config rx overflow dropped pack t count msb thernet ethernet hca ethernet port ips mac rx mac config rx overflow dropped pack t count lsb thernet ethernet hca oO thernet port ips mac rx mac config rx custom preamble forward e e G e G e G E S e S e thernet ethernet hca od 10g oO thernet port ips mac rx mac config rx timestamp registers rx peri Mellanox Technologies 69 Rev 2 4 Firmware Generation Configuration and Update Tools thernet ethernet hca ethernet port ips mac rx mac config rx timestamp registers rx fns ad ustment 10g thernet ethernet hca ethernet port ips mac rx mac config rx timestamp registers rx ns adjust ent 10g thernet ethernet hca ethernet port ips mac rx mac config rx t
23. dev mst mt4099 pci cr0 sxl itrace read memory 174712 bytes each dot denotes 2048 bytes IRISC Trace Viewer Mellanox ConnectX mft 4 0 0 20 built on Mar 17 2015 16 18 04 Git SHA Hash 161be23 FW Version 2 33 9910 11 03 2015 16 44 34 00000003 00000004 00000005 00000006 00000007 00000008 00000009 0000000a 0000000b 0000000c 0000000d 00000029 0000002b 0000002c 0000002d 0000002e 0000002f c1b59e4e dda895e4 dda89760 dda89868 dda97cc dda97e47 dda97 4 f dda9a8f6 dda9aa6e dda9ab79 ddaaadcl ddaee521 ddaee8ce ddaee9f2 ddaef0d5 ddaef2d9 ddaef6aa SCHD exegpc valid2freed 0x0 vec busy valid 0x00000010 SCHD SQP 0x000400 exes super scheduler busy done SCHD writing QpState SQPSTATE GOOD IDLE SCHD exegpc valid2freed 0x0 vec busy valid 0x00000010 SCHD SQP 0x000400 exes super scheduler busy SCHD writing QpState SQPSTATE GOOD IDLE SCHD exegpc valid2freed 0x0 vec busy valid 0x00000010 SCHD SQP 0x000400 exes super scheduler busy SCHD writing QpState SQPSTATE GOOD IDLE SCHD exegpc valid2freed 0x0 vec busy valid 0x00000010 SCHD SQP 0x000400 exes super scheduler busy INFO IPCdata 00 0x01abcd0a 0000002a ddaee60c INFO IPCdata 01 0x00000014 MAD exes mad QPN 0x000000 nda nds 0x7c58d014 SCHD SQP 0x000000 sqpc access db algorithm INC SCHD exes scheduler try to insert SCHD SQP 0x000000 exes scheduler chosen SCHD EXES GO 0x0 74 Mellano
24. device in phy addr dev addr format a addr data Access a single MDIO reg If data is specified the reg is written Otherwise it is read Addr and data should be in hex format g mdio gw Select which mdio gw 0 or 1 to use Default is O h Show usage v Show tool version Methods for sending MDIO Transactions mlxmdio will attempt to send the MDIO transaction through a firmware interface if supported on supported devices only if the firmware interface is not supported by the device the tool by default will attempt to send the transaction through the mdio gateway default is 0 If g lt 011 gt flag is specified the tool will attempt to send the MDIO transaction through the specified gate way Sending MDIO transactions via FW requires specification of the PCI device de Example 1 To read mlxmdio register run the following command mlxmdio d dev mst mt4099 pciconf0 m 0x2 0x1 a 0x0 4081 2 To write mlxmdio register run the following command mlxmdio d dev mst mt4099 pciconf0 m 0x2 0xle a 0x103 0x1 3 To read mlxmdio register through gateway run the following command mlxmdio d dev mst mt4099 pciconf0 m 0x0 0x1 a 0x3 g 0 688b 86 Mellanox Technologies Rev 2 4 5 Troubleshooting You may be able to easily resolve the issues described in this section If a problem persists and you are unable to resolve it yourself please contact your Mellanox representative or Mellanox
25. eee cee cece eee 55 Figure 32 MTUSB 1 De VIG une eis Fie e heres peer beaded He EN FEES 95 Figure 4 UPMF Package Generation Flow seeeeeeeeee eee eee 102 Mellanox Technologies 5 Rev 2 4 Document Revision History Table 1 Revision History Table Sheet 1 of 3 Date Revision Description June 2015 2 4 Added the following sections Section 3 8 mlx_fpga Burning and Debugging Tool for Mellanox Devices with FPGA on page 68 Section L Burning a New ConnectX 4 Device on page 114 Updated the following sections Section 2 MFT Installation on page 14 Section 3 1 Mellanox Software Tools mst Service on page 16 Section 3 5 mlxburn Firmware Image Generator and Burner on page 55 Section 3 6 mlxfwreset Loading Firmware on 5th Generation Devices Tool on page 63 Section 3 6 mlxfwreset Loading Firmware on 5th Generation Devices Tool on page 63 Section 4 2 itrace Utility on page 73 Section 4 7 mlxtrace Utility on page 78 Section 4 12 wqdump Utility on page 83 Section 5 Troubleshooting on page 87 e Section J 2 PCI Vital Product Data VPD on page 110 January 2015 2 3 Added the following sections Section 3 7 mlxphyburn Burning Tool for Externally Managed PHY on page 66 Section K Burning a New Switch IB Device on page 112 Updated the following sections e Supported Configurations and their
26. fw dir conf fw conf file nofs nofs img striped image format BINARY IMAGE dev type device type exp rom exp rom file exp rom dir exp rom dir force conf dir conf dir fwver vpd vpd rw vpd prog rw rw keywords file vpd set keyword lt keyword assignment gt set pxe en lt port1 port2 enable disable gt prof file profiles file gt query lie condam S ES damit CHA ooon amm 1 The fwver flag is not supported in Connect IB Switch IBTM and ConnectX 4 devices where dev type mellanox device mlxburn must know the device type in order to work properly number gt Use this flag if device type auto detection fails Example dev_type 4099 Supported Mellanox device types HCAs NICs 25408 25418 26418 26428 25448 26448 26468 26478 25458 26458 26438 26488 4097 4098 4099 4100 4101 4102 4103 4104 4105 4106 4107 4108 4109 4110 4111 4112 e Switches 43132 48436 48437 48438 51000 Bridges 64102 64122 1016 fw mellanox fw file Specify Mellanox FW released Firmware File to use file exten sion is mlx for HCA and BIN for an MT47396 switch conf lt parameter set file gt FW configuration file ini Needed when generating image not using dev flag or if configuration auto detection fails conf dir dir When specified the auto detected configuration files will be looked for in the given directory instead of in the firmware file dire
27. get more information Verify entire flash SW reset the target un managed switch device This command is supported only in the In Band access method Burn the specified ROM file on the flash Remove the ROM section from the flash Read the ROM section from the flash Query ROM in given image Burn Block Burns the given image as is No checks are done Set GUIDs Set read only VPD For FS3 image only Set manufacture GUIDs For FS3 image only Set the VSD Read the fw image on the flash Dump Configuration print fw configura tion file for the given image Dump Hash print hash file for the given image Set Update the HW access key which is used to enable disable access to HW The key can be provided in the command line or interactively typed after the com mand is given NOTE The new key is activated only after the device is reset Mellanox Technologies 39 Rev 2 4 Firmware Generation Configuration and Update Tools hw access enable disable key Enable disable the access to the HW The key can be provided in the command line or interactively typed after the com mand is given Low Level Flash Operations hw query Query HW info and flash attributes e rase addr Erase sector rw lt addr gt Read one dword from flash ww addr data Write one dword to flash wwne lt addr gt Write one dword to flash without sector erase wb data file addr Write a data block to flas
28. gt may be host name as well as an IP address mst remote del lt host Removes all remote devices on a specified hostname lt host name port name gt port should be specified exactly as in the mst remote add command Example The example below shows how to query the firmware of a device in the server side machine mft from the client side machine mft1 Run mst status in the server side rootemft mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST devices dev mst mt26428 pciconf0 PCI configuration cycles access domain bus dev fn 0000 0b 00 0 addr reg 88 data reg 92 Chip revision is B0 dev mst mt26428 pci cro PCI direct access domain bus dev fn 0000 0b 00 0 bar 0xd2600000 size 0x100000 Chip revision is B0 dev mst mtusb 1 USB to I2C adapter as I2C master Mellanox Technologies 97 Rev 2 4 Troubleshooting 2 Start the mst server in the server side rootGdev l vrt 059 005 mst server start 3 Add mst remote device in the client side rootGdev l vrt 059 006 mst remote add dev 1 vrt 059 005 4 Show the mst device in the client side which contains remote devices for the server side machine rootemftl mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST devices dev mst mt4099 pciconf0 PCI configuration cycles access domain bus dev fn 0000 0b 00 0 addr reg 88 data reg 92 Chip re
29. guid 0xe41d2d0300570fc0 mac 0x0000e41d2d570fc0 sg 6 Set Write protection on the last sector using flint flint d dev mst mt521 pciconf0 ocr hw set Flash0 WriteProtected Top 8 SubSectors 7 Enable flash quad SPI IO operations flint d dev mst mt521 pciconf0 ocr hw set QuadEn 1 gt To view flash settings flint d dev mst mt521 pciconf0 ocr hw query To view assigned GUIDs flint d dev mst mt521 pciconf0 ocr q gt To change a GUID after the initial burn flint d dev mst m4115 pciconf0 ocr guid 0xe41d2d0300570fc0 sg To change a MAC after the initial burn flint d dev mst m4115 pciconf0 ocr mac 0x0000e41d2d570fc0 sg gt To change a GUID and derive MAC from it after the initial burn run flint d dev mst m4115 pciconf0 ocr uid 0xe41d2d0300570fc0 sg IL MACS will be derived from the GUID according to guid2mac guid formula described in section J 1 Mellanox Technologies 115
30. in VMware Updated the following sections e Section Example VIBs installation on page 14 e Section on page 15 Section 4 12 wqdump Utility on page 83 December 2013 1 90 Removed the qq flag from the document Removed sub section On Pre ConnectX Devices Added the following sections Section 4 12 wqdump Utility on page 83 Updated the following sections Section 2 MFT Installation on page 14 e Section 3 1 Mellanox Software Tools mst Service on page 16 e Section Example VIBs installation on page 14 Section mlxburn Synopsis on page 56 Section SwitchX Switch Examples on page 60 Section InfiniScale IV Switch Examples on page 60 Section Command Parameters on page 39 Section 3 4 5 5 Disabling Enabling Access to the Hardware on page 52 Section 3 4 6 Flint mlxburn Limitations on page 54 Section 3 6 mlxfwreset Loading Firmware on 5th Generation Devices Tool on page 63 Section 3 2 1 Querying the Device on page 25 Section 3 2 2 Updating the Device on page 27 Section 4 4 mlxi2c Utility on page 75 Section 4 5 1 i2c Usage on page 76 Section 4 7 mlxtrace Utility on page 78 Section 4 12 wqdump Utility on page 83 Section J 3 Burning a New Connect IB Device on page 110 October 2013 1 80 Updated section Section Command Parameters on page 39 added a
31. macs as appears in the given image Commands affected burn Burn vsd as appears in the given image do not keep existing VSD on flash Commands affected burn Do not save the ROM which exists in the device Commands affected burn Do not attempt to take device data sections from device sections will be taken from the image FS3 Only Commands affected burn Do not attempt to take device data sections from device sections will be taken from the image FS3 Only Commands affected burn Version info Write this string of up to 208 characters to VSD when burn Non interactive mode assume answer yes to all questions Commands affected all Access to flash using FW ConnectX3 ConnectX3Pro Device Only Commands affected all 38 Mellanox Technologies Rev 2 4 1 When accessing SwitchX via I2C or PCI the override_cache_replacement flag must be set Command Parameters The flint utility commands are Common FW Update and Query b urn q uery full v erify swreset Expansion ROM Update brom lt ROM file gt drom rrom lt out file gt qrom Initial Burn Production bb sg guids_num lt num gt step_size lt size gt nocrc set vpd vpd file smg guids num lt num gt step size lt size gt Sv Misc FW Image operations ri out file dc out file dth out file HW Access Key set key key Burn flash Query misc flash firmware characteristics use full to
32. mlxburn Limitations 0 0 0 eee eee eee 54 3 5 mlxburn Firmware Image Generator and Burner 4 55 3 5 1 Generating and Burning Firmware 00 0 0 cee eee ee eee 55 3 5 2 Customizing Firmware as ees ne esen eea eao a e i a eee eee 55 3 6 mlxfwreset Loading Firmware on 5th Generation Devices Tool 63 3 6 mlxfwreset Limitations lees Ie 65 3 7 mlxphyburn Burning Tool for Externally Managed PHY 66 3 8 mlx fpga Burning and Debugging Tool for Mellanox Devices with FPGA 68 Chapter 4 Debug Utilities ice e re e VT E EI Oe See EE T ee eee ee 7L 4T fwtrace Utility ois esI ete ae gebe e te ede oe 71 41 1 fwtraceUsage em Rem Hea RRESHSTER RETE RENDER 71 4 2 trace U lity 2 her pie ee ta oh eed HOR Gi ARES RS 73 42 4 atrace UsaBe sesso iret be T AAE eta ED PED P TU TS eA 73 4 3 3mstdump Utility i sutacu oin ERAS ES I SIRE NADA RR EA 75 4 3 1 mstduinp Usage oos eked ot ebur eb isubqu bu Pub es 75 Mellanox Technologies 1 Rev 2 4 4 4 malxi2 Utilityz i sei ek oe at ee aS Ea Oe ee eae ee 75 44 T mlxu2c U SABe sos beber OEEREEEDOERESHRIUOPPLPES EPIS E 75 AS ZC U lity s cse ER eoa Xt Ree EN Ee SAIS PE EPI dH vede 76 4 5 1 26 Usage xbesrRERRE ERE EE REIR Godhead dpe EUMD Gehan EA 76 4 5 2 ExitR turn Valles oec ta pea Rp ERR REN REOR TI 4 6 mget temp Utility oc oo e cemere reb e Was TI 4 6 1 mget temp Usage s sos tue EE Cee Vot
33. mlxfwmanager deflated 57 adding mlxphyburn deflated 60 sfx auto run command mlxfwmanager u log on update ssl certificate ca bundle crt current dir argv mlxphyburn auto run command mlxphyburn device i phy mfa Firmware 1 37 10 N32722 cld b Log name tmp mlxfwmanager 20141126 2 10g 104 Mellanox Technologies Rev 2 4 G3 Updating Firmware Using an UPMF Updating the firmware is done by simply executing the UPMF Most of the command line options of the mlxfwmanager tool apply also for the UPMF For further detail please refer to Section 3 6 mlxfwreset Loading Firmware on 5th Generation Devices Tool on page 63 Some of the most commonly used command line options are force Force firmware update even if the firmware in the UPMF is not newer than the one on the device yes Non interactive mode assume yes to all questions list List the supported part numbers for which a firmware is available in the package In addition to the mlxfwmanager tool command line options the UPMF has 2 additional options Additional UPMF self extractor options sfx extract dir dir Use lt dir gt for temporary files during execution sfx extract only Do not execute only extract files to a location specified through the sfx extract dir option Sfx no pause Do not wait for user keypress after completion Note This flag is used in Windows OSs Extraction Example mlxfwmanager 20130717
34. mlxfwmanager update tool into a single standalone self extracting executable The UPMF generation is supported on Linux and Windows Being an executable file the UPMF should be prepared for Linux and Windows separately Usage mlx fwsfx gen source dir FW images directory out dir output directory sfx name lt sfx file name gt phy support phy img lt phy img gt Mellanox Technologies 101 Rev 2 4 Troubleshooting where source dir Directory containing Mellanox firmware images to be included in the package This option may be used more than once to specify more than one source directory out dir Specifies the output directory certificate SSL certificate phy support Generate extractor with mlxphyburn support phy img PHY firmware image sfx name The self extracting executable filename The default name is mlxfwmanager YYYYMMDD lt build number where build number is the previous maximum build number existing in the output directory incremented by one Figure 4 UPMF Package Generation Flow Fw FW FW FW exp exp exp exp rom rom rom rom ee p AV a Mixfwmanager tool t UT From MFT package gt mlx fwsfx gen tool Released to end users mixfwmanager mixfwmanager version version update package update package for Linux for Windows 102 Mellanox Technologies Rev 2 4 G2 2 UPMF Generation Example The below ex
35. the local IB port to be in the ACTIVE state and connected to an IB fabric B All MFT tools address the target hardware device using an mst device name This name is assigned by running the command mst start in Windows it is not required to run the mst start command for PCI and I2C access In Band devices can be assigned by running the mst ib add command To list the available mst device names on the local machine run mst status Local PCI devices may also be accessed using device aliases Supported aliases are e PCI device bus dev fn E G 03 00 0 OFED RDMA device E G mlx4 0 12 Mellanox Technologies Rev 2 4 Network interface with net prefix E G net eth2 Run mst status v to list the devices and their available aliases The format of an mst device name is as follows Via PCI mt4099 pci crX where X is the index of the adapter on the machine _crX devices access the adapter directly recommended if possible confX devices use configuration cycles to access the adapter For example mt25418 pci cr0 e Via USB to I2C adapter For example mtusb 1 Via Remote device dev mst mft 23108 devemst mt4103 pci cr0 e Via ibdr device For example dev mst CA MTA113 serverl HCA 3 ibdr 0 mlx 5 0 1 or ibdr 0 mlx5 0 1 e Via In Band string lid lid number For example dev mst CA MT4099 mft HCA 1 lid 0x0002 or simply lid 2 The mst ib add command adds devic
36. 1 and 2 GUIDs are ignored and should be set to 0 NOTE A GUID is a 16 digit hexadecimal number If less than 16 digits are pro vided leading zeros will be inserted MAC s Optional for Ethernet and VPI adapters and switches e If 2 MACs are provided macs flag they will be assigned to Port 1 and Port 2 respec tively If only one MAC is provided mac flag it will be assigned to Port 1 MAC 1 will be assigned to Port 2 e f no mac macs flag is provided the current LIDs will be preserved on the device NOTE A MAC is a 12 digit hexadecimal number If less than 12 digits are pro vided leading zeros will be inserted Mellanox Technologies 41 Rev 2 4 Firmware Generation Configuration and Update Tools gt To burn a firmware image 3 4 2 1 Update the firmware on the device keeping the current GUIDs and VSD Note This is the common way to use flint flint d dev mst mt4099 pci cr0 i fw 4099 2 31 5050 MCX354A FCB A2 bin burn Update the firmware on the device specifying the GUIDs to burn flint d dev mst mt4099 pci cr0 i fw 4099 2 31 5050 MCX354A FCB A2 bin guid 1234567dead beef burn Update the firmware on the device specifying the MACS to burn flint d dev mst mt4099 pci cr0 i fw 4099 2 31 5050 MCX354A FCB A2 bin mac 1234567dead beef burn Burn the image on a blank Flash device This means that no GUIDs are currently burnt on the device therefore they must be supplied with gui
37. 3 serverl HCA 3 ibdr 0 mlx5 0 1 dev mst SW MT51000 switchl ibdr 0 2 mlx5 0 1 4 Run any tool against the devices above flint d dev mst CA MT4113 serverl HCA 3 ibdr 0 mlx5 0 2 v FS3 failsafe image 0x00000038 0x00000f4f 0x000f18 BOOT2 OK 0x00201000 0x0020101f 0x000020 ITOC Header OK 0x00203000 0x0020323 0x000240 FW MAIN CFG OK 0x00204000 0x0020437 0x000380 FW BOOT CFG OK 0x00205000 0x002057 f 0x000800 0x00206000 0x002060f 0x000100 0x00207000 0x002195e3 0x0125e4 0x0021a000 0x0021e3a7 0x0043a8 HW MAIN CFG OK HW BOOT CFG OK PCI CODE OK IRON PREP CODE OK mta SS ors sacco Eee nee Mellanox Technologies 99 Rev 2 4 Troubleshooting 0x0021 000 0x00226bab 0x007bac PCIE LINK CODE OK 0x00227000 0x002a888f 0x081890 MAIN CODE OK 0x002a9000 0x002a95bf 0x0005c0 POST IRON BOOT CODE OK 0x002aa000 0x002aa3ff 0x000400 IMAGE INFO OK 0x002aa400 0x002b3e7b 0x009a7c FW ADB OK 0x002b3e7c 0x002b4277 0x0003fc DBG LOG MAP OK 0x002b4278 0x002b427 0x000008 DBG FW PARAMS OK 0x003 a000 0x003fbfff 0x002000 NV DATA OK 0x003 d000 0x003fd1ff 0x000200 DEV INFO OK 0x003 000 0x003 f13f 0x000140 MFG INFO OK 0x003 140 0x003ff13f 0x000000 VPD RO OK FW image verification succeeded Image is bootable 100 Mellanox Technologies Rev 2 4 Appendix G Update Pac
38. 3f 4e3f 0 5000 L2 1 cA totos m zo so oo 25 ee ee e GA ee 535 SSS A S000 Jig gt 1 ee Oi eOOsSe200s 0e i co co ce so es ae se ec SO doe 66 c 66fc 0 5000 L2 A IER RIDE PIE IRIE RIEIE gt gt gt ee ee HOO Opus 6a3e 6a3e 0 5000 L2 1 ec Bersm OE M E MEO 734b 734b 0 5000 L2 il 3j o3 St z008 1L co co cse se es we we ce dtu Duplicated MCGS Count 1000 1000 0 5000 L2 A ee reU se se se ee ee 9 e ee S95 AQ 2048 4002 4002 0 5000 L2 X ee Qs scole co ec e ae e oh sa ec GNU duae 2046 16 Unique rules 4108 Total fee0 40054 40055 40056 40057 40058 40059 4005a 4005b 4005c 4005d 4005e 4005f 40060 40061 40062 40063 40064 40065 40066 40067 40068 40069 4006a 4006b 4006c 4006d 4006e 4006f 40070 40071 40072 40073 40074 40075 40076 40077 40078 40079 4007a fef8 40054 40055 40056 40057 40058 40059 4005a 4005b 4005c 4005d 4005e 4005f 40060 40061 40062 40063 40064 40065 40066 40067 40068 40069 4006a 4006b 4006c 4006d 4006e 4006f 40070 40071 40072 40073 40074 40075 40076 40077 40078 40079 4007a 4 10 pckt drop Utility The pckt drop utility corrupts the next transmitted packet from a ConnectX family adapter port 4 10 1 pckt drop Usage The mst driver must be started prior to running the pckt_drop utility To start the pckt drop util ity 1 Start the mst driver mst start or mst restart 82 Mellanox Technologies Rev 2 4 2 Run the pckt_drop with the following command line syntax d device Spe
39. 4099 pciconf0 I Burning Process might take a few minutes I Device burned and verified 66 Mellanox Technologies Rev 2 4 Query Example mlxphyburn d dev mst mt4099 pciconf0 q I attempting to burn PHY Fw on device dev mst mt4099 pciconf0 Flash Type Atmel AT25DF041A FW version 1 37 Image ID 1 37 10 InterfaceMasters N32722 Apr 14 2014 12 21 00 Image ROM ID 0 Mellanox Technologies 67 Rev 2 4 Firmware Generation Configuration and Update Tools 3 8 mix fpga Burning and Debugging Tool for Mellanox Devices with FPGA mlx fpga tool allows the user to burn private code on Mellanox programmable adapter cards with an on board FPGA This tool burns a rbf file onto the FPGA flash device The rbf file must be generated according to the instructions listed in the relevant programmable adapter card User Manual des The tool also enables the user to read write individual registers in the QDR memory or read them all at once Tool Requirements e A ConnectX 9 3 Pro adapter card with an FPGA device Starting the mst service with the fpga lookup flag mst start with_fpga mix fpga Synopsis mlx_fpga d device read lt addr gt write addr value b image path gt mlx fpga d device lt read reg name write reg name value show all read all gt mlx_fpga d device clear semaphore reset gt where d device device
40. ACs 00300abadaba 00300abadabb VSD PSID MT_ 1090120019 flint i tmp image bin guid 0002c9000abcdef0 mac 02c90abcdef0 sg You are about to change the Guids Macs Uids on the device New Values Current Values Node GUID 0002c9000abcdef0 45214030001b8a0 Port1 GUID 0002c9000abcdef1 45214030001b8a1 Port2 GUID 0002c9000abcdef2 45214030001b8a2 Sys Image GUID 0002c9000abcdef3 45214030001b8a3 Portl MAC 02c90abcdef0 00300abadaba Port2 MAC 02c90abcdef1 00300abadabb Do you want to continue y n n y Restoring signature OK Mellanox Technologies 45 Rev 2 4 Firmware Generation Configuration and Update Tools flint i tmp image bin q Image type FS2 FW Version Ao Sil S050 FW Release Date 4 5 2014 Device ID 4099 Description Node Porti Port2 Sys image GUIDs 0002c9000abcdef0 0002c9000abcdef1 0002c9000abcdef2 0002c9000abcdef3 MACs 02c90abcdef0 02c90abcdef1 VSD PSID MT 1090120019 46 Mellanox Technologies Rev 2 4 3 4 5 2 5th Generation Devices On 5th Generation devices the sg command can operate on both the image file and the image on the flash When running the sg command on an image on the flash uid flag must be speci fied For ConnectX 4 guid mac flags can be specified By default 8 GUIDs will be assigned for each port starting from base base 1 up until base 7 for port 1 and base 8 up until base 15 for port 2 To change the step size and the number of GUIDs per p
41. E Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Hakidma 26 Sunnyvale CA 94085 Ofer Industrial Park U S A Yokneam 2069200 www mellanox com Israel Tel 408 970 3400 www mellanox com Fax 408 970 3403 Tel 4972 0 74 723 7200 Fax 4972 0 4 959 3245 Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CoolBox CORE Direct GPUDirect InfiniBridge InfiniHost InfiniScale Kotura Kotura logo MetroX MLNX OS PhyX ScalableHPC SwitchX TestX UFM Virtual Protocol Interconnect Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies Ltd CyPU ExtendX FabricIT FPGADirect HPC X Mellanox Care Mellanox CloudX Mellanox Open Ethernet Mellanox PeerDirect Mellanox Virtual Modular Switch MetroDX NVMeDirect StPU Switch IBTM Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 2329 Rev 2 4 Table of Content Fable of Contents osecias ids ee ia heise ed RR ANNEXE x Vu E PA a xa ERA Xa d Laistof Tables aves e RR IR Ex xx cin AE AN REIR RUN RUN ER RR o ow A List OF BIGUECS 2 54 5 35 ERR RCEESCOV DECORE DU ECL UL EM RUR RE AE OE RUE ED Chapter 1 Introduction 14 v ARE RERUCRS e
42. JM Mellanox TECHNOLOGIES Connect Accelerate Outperform Mellanox Firmware Tools MFT User Manual Rev 2 4 www mellanox com Rev 2 4 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAG
43. Mellanox Technologies Rev 2 4 3 2 1 Querying the Device To query a specific device use the following command line mlxfwmanager d device query e To query all the devices on the machine use the following command line mlxfwmanager query Examples a Query the device mlxfwmanager d 09 00 0 query Querying Mellanox devices firmware Device 1 Device Type ConnectX3 Part Number MCX354A FCA A2 A4 Description ConnectX 3 VPI adapter card dual port QSFP FDR IB 56Gb s and 40GigE PCIe3 0 x8 8GT s RoHS R6 PSID MT 1020120019 PCI Device Name 0000 09 00 0 Portl GUID 0002c9000100d051 Port2 MAC 0002c9000002 Versions Current Available FW 21584 550 50 25975000 Status Update required Found 1 device s requiring firmware update Please use u flag to perform the update b Query all the devices Querying Mellanox devices firmware Device 1 Device Type Connect IB Part Number MCB192A FCA Al Description Connect IB Host Channel Adapter single port QSFP FDR 56Gb s PCIe2 0 x16 RoHS R6 PSID MT 1220110030 PCI Device Name dev mst mt4113 pciconf0 Portl GUID 0002c903002e 500 Port2 GUID 0002c903002e 501 Versions Current Available FW 2 a Akal ALAS 10 10 4000 Status Update required Device 2 Mellanox Technologies 25 Rev 2 4 Firmware Generation Configuration and Update Tools Device Type ConnectX3 Part Number MCX354A FCA A2 A4 Description ConnectX 3 VPI adap
44. Parameters on page 32 Examples of mlxconfig Usage on page 33 Section 4 1 fwtrace Utility on page 71 e Section 4 2 1 itrace Usage on page 73 Section G2 2 UPMF Generation Example on page 103 August 2014 2 2 Added the following sections Section 3 6 mlxfwreset Loading Firmware on 5th Generation Devices Tool on page 63 Section 3 1 Mellanox Software Tools mst Service on page 16 Section I Booting HCA Device in Livefish Mode on page 108 Updated the following sections Section 3 6 mlxfwreset Loading Firmware on 5th Generation Devices Tool on page 63 Section G Update Package for Mellanox Firmware on page 101 Section flint Synopsis on page 36 6 Mellanox Technologies Rev 2 4 Table 1 Revision History Table Sheet 2 of 3 Date Revision Description May 2014 2 1 Added the following sections Section 3 2 mlxfwmanager Firmware Update and Query Tool on page 23 Section H Secure Host Feature on page 106 Updated the following sections Section 3 6 mlxfwreset Loading Firmware on 5th Generation Devices Tool on page 63 Section Supported Configurations and their Parameters on page 32 e Section Burn Example on page 66 e Section Example VIBs installation on page 14 February 2014 2 0 Removed the sub section How to Run flint in VMware Removed the sub section How to run mstdump
45. Program the VPD W tag the writable section file gt of the VPD with the data given in the rw keywords file File lines format KEYWORD VALUE In order to set binary data to a keyword add BIN to the key word name in this case the data is a hexadecimal string of even length Example file V1 MY ASCI KEYWORD V2 BIN 1234abcd White spaces before and after VALUE are trimmed vpd_rw on Linux only Display also the read write section of the PCI VPD of the given device vpd_set_keyword lt keyword Add or change a keyword value in the VPD W tag the writable assignment section of the VPD with the data given in the keyword assign ment string The string format is identical to a line in the rw key words file described above Other keywords in the VPD W tag are not affected by this operation wrimage lt fw image file gt Write the image to the given file 1 The VPD query may not be enabled on certain board types Also VPD operations are available only for devices with a PCI interface 2 Running multiple VPD access commands in parallel on the same device by mlxburn or any other VPD access tool may cause the commands to fail VPD access commands should be run one at a time 58 Mellanox Technologies Rev 2 4 Mellanox Connect IBG Switch IB and ConnectX 40 Initial Burning Options These options are applicable for Mellanox Connect IB HCA only de The following options are relevant when g
46. ROM cannot be removed separately B9 Updating only the EXP ROM FlexBoot for recent firmware images which requires adding the allow rom change option Allow allow rom change option to the flint command For example flint d mst device allow rom change drom Generating a firmware image file on Windows fails and the following message is printed on screen E File C Users Administrator ps ini Line 1 Invalid syn tax E Image generation failed child pro cess exited abnor mally Using a firmware configura tion file ini which was generated by PowerShell text redirection flint d lt mst_device gt dc gt fw conf file gt ini Generate the firmware configuration file ini using CMD edit and continue with generating the firmware image file Rev 2 4 Issue Cause Solution Burning command fails and the following message is printed on screen E Can not open 06 00 0 Can not obtain Flash sema phore 63 You can run flint clear semaphore d device to force semaphore unlock See help for details Semaphore can be locked due to e Another process is burning the firmware at the same time Failure in the firmware boot Burning process was force fully killed If no other process is taking place at the same time run the following command flint d device clear sema phore OR Reboot the machine Burning t
47. Support at support mellanox com 5 1 General Related Issues Issue Cause Solution Adapter is no longer identi fied by the operating system after firmware upgrade Happens due to burning the wrong firmware on the adapter firmware corruption or adapter s hardware fail ure Power cycle the server If the issue persists extract the adapter and contact Mellanox Support Server is booting in loop not completing boot after performing adapter firm ware upgrade Happens due to burning the wrong firmware on the adapter firmware corruption or adapter s hardware fail ure Extract the adapter and contact Mellanox Support Some of the 5th generation devices are represented with only one mst device dev mst mt4113_pciconfx in the output of mst status For 5th generation devices there is only one method available for accessing the hardware For example Connect IB device is repre sented by dev mst mt4113_pciconfx mst device When querying a 5th generation device use the conf mst device for example dev mst mt4113_pciconfx Enabling hardware access after configuring new secure host key fails The new configuration of the secure host key was not loaded by the driver Restart the driver before enabling the hard ware access again Server not booting after enabling SRIOV with high number of VFs Setting number of VFs larger than what the Hard ware and Software can sup
48. X eas Gea eee re siete DE 1 4 Supported Operating Systems lees 11 12 MET Access to Hardware Devices llle 12 Chapter 2 MFT Installation Roe RR EE EE inve 14 Chapter 3 Firmware Generation Configuration and Update Tools 16 3 Mellanox Software Tools mst Service 0 0 0 cece eee 16 3 11 EMU scooter me DI VER PUER HEROES Sabb Rhee EE DAE Et 16 341 2 Wid OWS ii s o E e Sigs te wed a OP REX RICE NS 19 3 L3 FreeBSD i oun OIN E A eR bee bo x eon ER Resina 21 BLA BSI 4 ttr t as WIRE ROR AAA sia RR Rc BUDE ade secet eC ERR nd 22 3 2 mlxfwmanager Firmware Update and Query Tool 0 23 3 2 Querymg the Device cscs bensr nrden naea EE AE PRESS Soto p 25 3 2 2 Updating the Device sd op be be bete e pe e E ERES 27 3 2 33 Updating the Device Online 0 ee ee eee 27 3 3 mlxconfig Changing Device Configuration Tool 0 31 3 3 1 Using mlxconifg with PCI Device in Bus Device Function BDF Format 35 3 3 2 Using mlxconfig to Set VPI Parameters eese 35 3 4 flint Firmware Burning Tool eleeeeeeee eee 36 3 4 1 Burning a Firmware Image 0 0 00 40 3 4 Querying the Firmware Image 0 0 0 eee ee eee 42 3 4 3 Verifying the Firmware Image 0 0 eee eee ees 43 3 4 4 Managing an Expansion ROM Image 0 0 00 00 e eee eee 43 3 4 5 Setting GUIDs and MACs 00 ee ccc eee eee 44 3 4 6 Flint
49. achine use the following command line mlxfwmanager u d device i lt FileName gt Example mlxfwmanager u d 0000 09 00 0 i fw ConnectX3 rel 2 32 5000 mfa Querying Mellanox devices firmware Device 1 Device Type ConnectX3 Part Number MCX354A FCA A2 A4 Description ConnectX 3 VPI adapter card dual port QSFP FDR IB 56Gb s and 40GigE PCIe3 0 x8 8GT s RoHS R6 PSID MT 1020120019 PCI Device Name 0000 09 00 0 Port1 GUID 0002c9000100d051 Port2 MAC 0002c9000002 Versions Current Available FW Ao ddl SOS 2 32 5000 Status Update required Found 1 device s requiring firmware update 3 2 3 Updating the Device Online To update the device online on the machine from Mellanox site use the following command line mlxfwmanager online u d lt device gt 1 If only PXE rom needs update please add f to the command line Mellanox Technologies 27 Rev 2 4 Firmware Generation Configuration and Update Tools Example mlxfwmanager online u d 0000 09 00 0 y Querying Mellanox devices firmware Device 1 Device Type Part Number Description ConnectX3Pro MCX354A FCC Ax ConnectX 3 Pro VPI adapter card dual port QSFP FDR IB 56Gb S and 40GigE PCIe3 0 x8 8GT s RoHS R6 PSID PCI Device Name 0000 09 00 0 Portl GUID Port2 GUID Versions FW PXE Status MT 1090111019 0002c903006e955e1 0002c903006e955e2 Current Available DID ONS Ae A SOOO 3 4 0460 3
50. age 59 Section flint Synopsis on page 36 8 Mellanox Technologies Rev 2 4 About this Manual The document describes MFT Rev 2 4 features tools content and configuration Intended Audience This manual is intended for system administrators responsible for managing and debugging firm ware for Mellanox devices Common Abbreviations and Acronyms Term Description MFT Mellanox Firmware tools MST Mellanox Software tools and it s the name of the script that starts stops the driver used by MFT tools mlx Extension of the text firmware file which contains all the firm ware content ini Extension of the firmware configuration file which is in INI format and contains card specific configurations bin Extension of the binary firmware file which is a combination of INI and mlx file MFA Extension of the a firmware file that contains several binary files of firmware for different cards boards 4th Generation Family Contains the following devices ConnectX 3 Pro Infniscale IV SwitchX SwitchX amp 2 and BridgeX ConnectX 3 5th Generation Family Contains the following device Connect IBG Switch IBTM ConnectX 4 Reference Documents and Downloads e To download firmware images and their release notes see http www mellanox com page software overview ib e Mellanox OFED for Linux is a software stack that can be downloaded from http www mellan
51. ample packs 3 firmware binaries named fw ConnectX 3 1 bin fw ConnectX 3 2 bin fw ConnectX 3 3 bin located in the directory tmp fw ConnectX 3 dir into a Linux UPMF package named tmp mlxfwmanager 20141126 1 mlx fwsfx gen source dir tmp fw ConnectX 3 dir out dir tmp Package name tmp mlxfwmanager 20141126 1 Contents Source dirs tmp fw ConnectX 3 dir Adding file etc mft ca bundle crt Sfx stub file usr bin mlx sfx stub Creating intermediate MFA archive from binary files fw ConnectX 3 1 bin fw ConnectX 3 2 bin fw ConnectX 3 3 bin mfa tool usr bin mlx mfa gen mfa cmd usr bin mlx mfa gen p tmp Pcgs82KTxr srcs mfa s tmp fw ConnectX 3 dir Adding bins from tmp fw ConnectX 3 dir Files copied 3 Querying images Files queried 3 Compressing this may take a minute Archive tmp Pcgs82KTxr srcs mfa Total time 0m2s Adding file tmp Pcgs82KTxr srcs mfa Adding file usr bin mlxfwmanager pci Creating zip tmp Pcgs82KTxr zippackage zip adding srcs mfa deflated 0 adding mlxfwmanager pci deflated 57 adding ca bundle crt deflated 45 sfx auto run command mlxfwmanager pci u log on update ssl certificate ca bundle crt current dir argv Log name tmp mlxfwmanager 20141126 1 10g Mellanox Technologies 103 Rev 2 4 Troubleshooting G 2 3 UPMF Generation with PHY Binary Example The below example packs 3 firmware binaries named fw ConnectX 3 1 bin fw ConnectX 3 2 bi
52. and mlxconfig d dev mst mt4103 pci cr0 set LINK TYPE Pl 1 LINK TYPE P2 1 Device 1 Device type ConnectX3Pro PCI device dev mst mt4103 pci cr0 Configurations Current New SRIOV EN il It NUM_OF VFS 8 LINK TYPE Pl N A LINK TYPE P2 N A LOG BAR SIZE 3 t es e ee Apply new Configuration y n n y Applying Done I Please reboot machine to load new configurations Mellanox Technologies 35 Rev 2 4 Firmware Generation Configuration and Update Tools 3 4 flint Firmware Burning Tool The flint Flash interface utility performs the following functions Burns a binary firmware image to the Flash device attached to an adapter bridge or switch device Burns an Expansion ROM image to the Flash device attached to a ConnectX family adapter devices Queries for firmware attributes version GUIDs UIDs MACS PSID etc e Enables executing various operations on the Flash memory from the command line for debug production e Disables enables the access to the device s hardware registers and changes the key used for enabling This feature is functional only if the burnt firmware supports it flint Synopsis flint switches command parameters Switches Options allow psid change banks banks blank guids clear semaphore d evice device dual image flash params type log2 Size num of flashes Allow burning a FW image with a different PSID Parame
53. apshot options Where o file dump file name m mode run mode fast normal full h help Show help message Examples To generate mlxdump udmp while running in fast mode mlxdump d dev mst mt4099 pci cr0 snapshot To generate mlxdump udmp while running in full mode mlxdump d dev mst mt4099 pci cr0 snapshot m full To generate mlIxdump 13 1 2013 udmp while running in normal mode mlxdump d dev mst mt4099 pci cr0 snapshot m normal o mlxdump 13 1 2013 udmp mlixmcg Utility The mlxmcg tool displays the current multicast groups and flow steering rules configured in the device Target users Developers of Flow Steering aware applications This tool dumps the internal steering table which is used by the device to steer Ethernet packets and Multicast IB packets to the correct destination QPs Each line in the table shows a single filter and a list of destination QPs Packets that match the fil ter are steered to the list of destination QPs mlxmcg is not supported against In band device o mixmcg Usage The mst driver must be started prior to running mlxmcg tool To start mlxmcg 80 Mellanox Technologies Rev 2 4 1 Optional for Windows OSs Start the mst driver mst start or mst restart 2 Enter an mlxmcg command that complies with the following command syntax mlxmcg OPTIONS where h d DEV f FILE p PARAMS q help dev DEV file FILE
54. ation Configuration and Update Tools override_cache_re placement s ilent striped_image uid lt UID gt uids lt UIDs gt use image guids use image ps use image rom ignore dev data ignore dev data v vsd lt string gt y es use_fw Allow accessing the flash even if the cache replacement mode is enabled NOTE This flag is often referred to as ocr NOTE This flag is intended for advanced users only Running in this mode may cause the firmware to hang Do not print burn progress flyer Commands affected burn Use this flag to indicate that the given image file is in a striped image format Commands affected query verify BridgeX 5th Generation devices only Derive and set the device UIDs GUIDs MACs WWNs UIDs are derived from the given base UID according to Mellanox Methodologies Commands affected burn sg BridgeX only 29 space separated UIDs must be specified here The specified UIDs are assigned to the following fields respectively GO MAC PIO G0O MAC PII G0 MAC PI2 GO MAC PEO G0 MAC PE1 G0 MAC PE2 GO MAC PE3 GO FC WWPN PO0 GO FC WWPN P1 G0 FC WWPN P2 GO0 FC WWPN P3 G0 IB NODE GUID G0 IB PORT GUID GO FC WWNN GI MAC PI0 GI MAC PII GI MAC P2 GI MAC PE0 GI MAC PEI GI MAC PE2 GI MAC PE3 G1 FC WWPN PO GI FC WWPN P1 GI FC WWPN P2 GI FC WWPN P3 GI IB NODE GUID GI IB PORT GUID GI FC WWNN IB SYS TEM GUID Commands affected burn sg Burn guids uids
55. ayed by itrace if they were previously generated w wait Runs itrace in wait mode itrace will exit only if you press lt Ctrl C gt This is not the default behavior of itrace Without the w option itrace will exit if there have been no new traces in the last 0 5 seconds 1 This step in not required in Windows Mellanox Technologies 73 Rev 2 4 Debug Utilities d d DEVICE nomap no propel v version c color D dump Example Specifies the name of the mst device driver for accessing the cr space The default value is dev mst mt4099 pci cr0 To run itrace via the I2C interface use this option to specify the following d device where the device is an I2C device such as mtusb 1 Sets itrace not to directly access memory via memory mapping for reading the trace buffer but to use the adapter memory access Gateway instead By default itrace accesses the memory directly If the cr space device specified by the d parameter is one of the I2C devices nomap is switched on Sets itrace not to animate the propeller in wait mode w option By default ani mation is enabled Prints the MFT version and exits Enables color in trace output Dumps the trace buffer and exits This option is useful for debugging itrace it dumps the contents of the trace buffer in row format For Linux device names should be listed with the dev mst prefix For Windows no prefix is required itrace d
56. ba 00300abadabb VSD MELLANOX PSID MT 1090120019 3 4 5 5 Disabling Enabling Access to the Hardware The secure host feature enables ConnectX 9 family devices to block access to its internal hard ware registers The hardware access in this mode is allowed only if a correct 64 bits key is pro vided The secure host feature requires a MLNX_OFED driver installed on the machine dies To disable enable access to the hardware 1 Set the key flint d dev mst mt4099 pci cr0 set key 22062011 Setting the HW Key OK Restoring signature OK A driver restart is required to activate the new key dec 2 Access the HW while HW access is disabled fint d dev mst mt4099 pci cr0 q E Cannot open dev mst mt4099 pci cr0 HW access is disabled on the device E Run flint d dev mst mt4099 pci cr0 hw access enable in order to enable HW access 52 Mellanox Technologies Rev 2 4 3 Enable HW access flint d dev mst mt4099 pci cr0 hw access enable Enter Key xeeeck 4 Disable HW access flint d dev mst mt4099 pci cr0 hw access disable WARNING 1 Once a hardware access key is set the hardware can be accessed only after the correct key is provided 4 2 If a key is lost there is no way to recover it using the tool The only way to recover from a lost key is to Connect the flash not present jumper on the card e Boot in flash recovery mode e Re burn FW e Re set the HW access key For
57. ccessfully Burn the image to a flash using an external burner Optional After assembly query the image on flash to verify there are no GUIDs on the device flint d dev mst mt26428 pci cr0 q Image type FS2 FW Version 2 31 5050 FW Release Date 4 5 2014 Device ID 4099 Description Node Portl Port2 Sys image 50 Mellanox Technologies Rev 2 4 GUIDs ego ie eee ei fee as ease et ea esas mee ee Tape eye uses eee estate MACs IEEE IRIE IE TEIE EE TERCIETESETEOETE TEE TETEIE YE TEST VSD n a PSID MT 1090120019 W GUIDs MACs values and their CRC are not set 4 Set the correct GUIDs Since the image is with blank GUIDs this operation takes less than 1 second flint d dev mst mt4099 pci cr0 guid 0x0002c9030abcdef0 mac 0x0002c9bcdef1 sg 5 Query the image on flash to verify that the GUIDs are set correctly flint d dev mst mt4099 pci cr0 q Image type FS2 FW Version Desi OO FW Release Date 4 5 2014 Device ID 4099 Description Node Porti Port2 Sys image GUIDs 0002c9030abcdef0 0002c9030abcdef1 0002c9030abcdef2 0002c9030abcdef3 MACS 0002c9bcdef1 0002c9bcdef2 VSD n a PSID MT 1090120019 Method 2 Pre burn an Image with Specific GUIDs MACs for Each Device In this method a base image is generated with arbitrary default GUIDs and then updated with the correct GUIDs for each device gt To pre burn an image with Specific GUIDs MACs for Each Device 1 Generate the base image with arbi
58. cece cece eee ee eee 108 I 1 Booting Card in Livefish Mode 1 0 op E ccc cece nent ences 108 L2 Booting Card in Normal Mode 0 cece m 108 L3 Common Locations of Flash Present Pins 108 Appendix J Burning a New Connect IB Device cece cee cece ee ee cece esc e S 110 KL GUIDsandMAGCS te cate oe epu waa oe a ea Ai ae dre eee ew ied DUE cay wae Seaman 110 J 2 PCI Vital Product Data VPD 0 0 0 0 ces 110 J 3 Burning a New Connect IB Device lseeeseeeeeeeee Re 110 Appendix K Burning a New Switch IB Device cece cece ccc ceccccecescece 112 K 1 Burning the Switch IB Device 2 0 0 00 ec I 112 Appendix L Burning a New ConnectX 4 Device cece cece cece cere eene eee L14 L 1 Burning the ConnectX 4 Device 0 0 cece teen nee nes 114 Mellanox Technologies 3 Rev 2 4 List of Tables Table 1 Revision History Table sis 2 02 eee II ae 6 Table 2 Supported Mellanox Devices 0 cece eh 12 Table 3 mst start Supported OPCODES 0 0 eee eens 17 Table 4 Supported Configurations and their Parameters 0 0 0 0 cece eee eee 32 Ilable 5 PSID format i1 x tora y dae beset es we IER ed 90 Table 6 MTUSB 1 Package Contents 4 Mellanox Technologies Rev 2 4 List of Figures Figure 1 Mellanox Firmware Tools A Scheme of Operation 0 00 0 e eee eee 11 Figure 2 FW Generation and Burning 2 0
59. chnologies 95 Rev 2 4 Troubleshooting D 1 2 System Requirements The MTUSB 1 is a USB device which may be connected to any Personal Computer with a USB Host Adapter USB Standard 1 1 or later and having at least one USB connection port D 1 3 Supported Platforms MTUSB 1 supports the same platforms that are supported by the MFT tools package D 2 Hardware Installation To install the MTUSB 1 hardware please execute the following steps in the exact order 1 Connect one end of the USB cable to the MTUSB 1 and the other end to the PC 2 Connect one end of the I2C cable to the MTUSB 1 and the other end to the system board you wish to control via the I2C interface If the system board uses a 3 pin connector instead of a 9 pin connector connect the appropriate converter cable as an extension to the I2C cable on the 9 pin end then connect its 3 pin end to the system board D 3 Software Installation The MTUSB 1 device requires that the Mellanox Firmware Tools MFT package be installed on the machine to which MTUSB 1 is connected see Section 2 MFT Installation on page 14 of this manual for installation instructions For a Windows machine it is also required to install the MTUSB 1 driver visit http www dio lan com to download this driver This driver is required for the first use of the MTUSB 1 device Once you have the requirements installed you may verify that your MTUSB 1 device is detected by MFT software as desc
60. cify the mst device to configure Required h help Print this help screen and exit m mode Specify operating mode Supported modes are EDP Inserts error on next transmitted data packet Default EDP p port Select which port to configure Use 1 2 for port1 port2 respectively or b for both Default b v version Print the application version and exit Example pckt drop d dev mst mt4099 pci cr0 p 1 The example above shows how to use the pckt drop to corrupt a packet from port 1 4 11 mlxuptime Utility The mlxuptime is a Mellanox firmware which prints Mellanox devices up time and measured configured frequency 4 11 1 mlxuptime Usage mlxuptime d s h v where d Mst device name s Sampling interval for measuring frequency default 1 sec h Print help and exit v Print tool version and exit Example Print all info mlxuptime d dev mst mt4099 pci cro easured core frequency 427 099818 MHz Device up time 10 01 20 456344 h m s usec Print the uptime and configured frequency only mlxuptime d dev mst mt4099 pci cr0 m Configured core frequency 427 083333 MHz Device up time 53 31 00 162464 h m s usec 4 12 wqdump Utility The wqdump utility dumps device internal work queues A work queue is an object containing a Queue Pair Context QPC which contains control information required by the device to execute I O operations on that QP and a work
61. ction 3 5 2 Customizing Firmware on page 55 3 5 1 Generating and Burning Firmware The mlxburn firmware update flow is composed of two separate stages image generation and image burning In the image generation stage a given Mellanox firmware release in mlx format for adapters bridges and 4th generation switches ConnectX and newer and in BIN file for mat for InfiniScale III switches is processed together with a board specific configuration ini file to generate a burnable firmware image This image is burnt to the Flash EERPROM attached to a Mellanox device in the second stage The burning process retains device specific data such as GUIDs UIDs MACs VSD and BSN Also the burn process is failsafe by default Figure 2 FW Generation and Burning N d Exp ROM mlxburn runs both stages by default but it may perform only one by means of command options If the wrimage is specified see Section mlxburn Synopsis only image generation is performed Specifying the image option skips the image generation stage and loads the pro vided image generated in a previous run of mlxburn using the wrimage option 3 5 2 Customizing Firmware A Mellanox firmware image can be customized usually to fit a specific board type The custom ization is done by using a FW parameter set file in the image generation stage This file has a ini Mellanox Technologies 55 Rev 2 4 Firmware Generat
62. ctory Applicable for burn operation dev lt mst dev gt Burn the image using the given mst device 56 Mellanox Technologies Rev 2 4 exp rom lt exp rom file gt Integrate the given expansion rom file to the FW image The given file may be in img or bin rom raw binary format e If the exp rom file is set to AUTO expansion rom file is auto detected from the files rom in the exp rom dir see below NOTE Exp rom auto detection is done for devices that are already burned with an exp rom image e f exp rom AUTO is specified for a device with no exp rom it would be burnt with no exp rom To add exp rom to a device manually supply the exp rom file to use exp rom dir exp rom dir The directory in which to look for expansion rom file when exp rom AUTO is specified By default exp rom files are searched in fw file directory exp rom force None interactive mode Assume yes for all user questions format BINARY IMAGE gt Specify which image format to use Can be specified only with the wrimage flag Default is BINARY fw dir dir When specified the auto detected fw files will be looked for in the given directory Applicable for burn operation conf dir list When specified the auto detected configuration files will be dirl dir2 dirn looked for in the given directories instead of in the firmware file directory Applicable for burn operation fwver e When a device is given Displa
63. d guids by the burning command More over the burn process cannot be failsafe when burning a blank Flash therefore the nofs flag must be specified flint d dev mst mt4099 pci cr0 i fw 4099 2 31 5050 MCX354A FCB A2 bin nofs guid 12345678 burn Read FW from the device and save it as an image file flint d dev mst mt4099 pci cr0 ri Flash Image Copy bin MT48436 InfiniScale IV switch Burn the image on a blank Flash device This means that no GUIDs are currently burnt on the device therefore they must be supplied with guid guids by the burning command Moreover the burn pro cess cannot be failsafe when burning a blank Flash therefore the nofs flag must be specified flint d dev mst mtusb 1 i tmp fw is4 bin nofs guids 0002c9000100d060 0 0 0002c9000100d060 b MT48436 InfiniScale IV switch inband firmware update flint d lid 0x18 i tmp fw is4 bin b MT58100 SwitchX switch Burn the image on a blank Flash device Meaning no GUIDs MACs are currently burnt on the device therefore they must be supplied with guid guids and mac macs by the burning command More over the burn process cannot be failsafe when burning a blank Flash therefore the nofs flag must be specified flint d dev mst mtusb 1 i tmp fw sx bin nofs guids 000002c900002100 0 0 000002c900002100 macs 0002c9002100 0002c9002101 b MT58100 SwitchX switch inband firmware update flint d lid 0x18 i tmp fw sx bin b Querying the Firm
64. e set to 0 mlxburn d dev mst mtusb 1 fw fw sx mlx conf MSX6025F Al ini guids 000002c900002100 0 0 000002c900002100 macs 0002c9002100 0002c9002101 nofs InfiniScale IV Switch Examples e Burn an MTS3600 switch system via I2C a Set the I2C network to access the InfiniScale IV switch mlxi2c d dev mst mtusb 1 p IS4 PRIM b Burn mlxburn dev dev mst mtusb 1 fw fw 1S4 mlx e Burn an MTS3600 switch system using the In Band access method mlxburn dev dev mst SW MT48438 lid 0x0003 fw fw IS4 mlx e Generate and burn a new MTS3600 via I2C a Setthe I2C network to access the InfiniScale IV switch mlxi2c d dev mst mtusb 1 p IS4 PRIM b Burn the new image the flash is still blank specifying the Node and System GUIDs Note that 4 GUIDs in quotes should be specified as an argument to the guids flag The 2 middle GUIDs are ignored by the InfiniScale IV and should be set to 0 mlxburn dev dev mst mtusb 1 fw fw IS4 mlx conf MTS3600Q 1UNC_Al ini guids 0002c9000100d060 0 0 0002c9000100d060 nofs Generate and Burn a new MT3600 switch system via I2C in 2 steps a Generate the image mlxburn fw fw IS4 mlx conf MTS36000 1UNC Al ini wrimage fw is4 bin 60 Mellanox Technologies Rev 2 4 b Burn using flint tool flint d dev mst mtusb 1 i tmp fw is4 bin nofs guids 0002c9000100d060 0 0 0002c9000100d060 b To generate an MT48436 image and perform an In Band update of th
65. e device with LID 0x18 enter mlxburn fw fw IS4 mlx dev lid 0x18 BridgeX Gateway Examples To update firmware on BridgeX device with the configuration file ini auto detected enter mlxburn d dev mst mt64102 pci crl fw fw BridgeX rel mlx To generate a failsafe image file for the same BridgeX above without burning enter mlxburn fw fw BridgeX rel mlx conf MTB4020 PCO Al ini wrimage fw BridgeX bin Toupdate firmware on the same BridgeX above with the configuration file ini explic itly specified enter mlxburn fw fw BridgeX rel mlx dev dev mst mt64102 pci cri conf MTB4020 PCO0 Al ini To burn a firmware binary file for a BridgeX device enter mlxburn image fw BridgeX bin dev dev mst mt64102 pci crl Connect IB Examples e To generate a failsafe image file for Connect IB device without burning enter mlxburn fw FW fw ConnectIB mlx c FW MCB194A FCA Al ini wrimage fw ConnectIB MCB194A FCA Al bin base guid 0x0002c903002ef500 To update firmware on a Connect IB device enter mlxburn i fw ConnectIB MCB194A FCA Al bin d dev mst mt4113 pciconf0 Switch IB Examples To generate a failsafe image file for a Switch IB device without burning enter mlxburn fw FW fw SwitchIB mlx c FW MSB7700 E Ax ini wrimage fw SwitchIB MSB7700 E Ax bin base guid 0x0002c903002ef500 To update firmware on a Switch IBTM device enter mlxburn i fw SwitchIB MSB7700 E
66. e sek we B ERES TI 47 3nlxtrace Utility lt p etsy aet ao bebe ger teet SEU got tp es 78 4ST A amlxtrace Usage cocer heey Sea ER Soha ew Ghee 78 48 mlxdump Utility i oeil ben eR Id E RARE eR RET EY 80 4 8 mlxdump Usage cc ese ree eet este eret es TUE 80 49 mlxmcg U lity ese epe I eR RUE Sen E WEM REA Ra OS 80 4 9 T mmeg Usage ooo Tbe eb t eue Iusbuustebt ebbe genet 80 4 10 pekt drop Utility esesta ssi remise 9 uo mei Kee ceeded aes 82 4 10 1 pckt drop Usage ssec eR RD RR lob OE elk CER HERES 82 4 ET mlxupume Utility e tbt Be das ee ae ie Ags et 83 4 11 11 mixuptime Usage s soecerrAps ERR e Aopen Rd E ed EB phase 83 4 12 wqdump Utility cr e eene eto phd ep RR 83 4 12 1 wqdump Usage een LERMRERSRRERN ERES RES MR eater SEE 84 4 13 mlxmdio Utility lllelleeeeeeeee I 86 4 13 1 mlxmdio Usage 5E te cea a esee Dd UE ee papier 86 Chapter 5 Troubleshooting celere nn nn nn nnn 87 5 1 General Related Issues l l 87 5 2 Installation Related Issues 2 0 0 0 0 0 eee nee 88 5 3 Firmware Burning Related Issues 0 cece eee 88 Appendix A PSID Assignment ccc cece cece cece cece ehh hh hrs 90 A T PSD Field Structure vos ode cranes Pes ede eters caste taste acne Ee Mte Ales e ga m obedire ele cts 90 A 2 PSID Assignment and Integration Flow 0 0 0 0 ccc eee tenn eee 90 Appendix B Flow Examples mlxburn 0 cece cece ccc cece cece e
67. ectX adapter device with the configuration file ini auto detected enter mlxburn fw fw ConnectX3 rel mlx dev dev mst mt4099 pci cr0 To generate a failsafe image file for the same adapter above without burning enter mlxburn fw fw ConnectX3 rel mlx conf MCX354A FCB A2 A5 ini wrimage fw 4099 bin 1 The arguments of the guids and macs flags must be provided within quotation marks for example mlxburn macs 0002c900001 0002c900002 Mellanox Technologies 59 Rev 2 4 Firmware Generation Configuration and Update Tools To update firmware on the same adapter above with the configuration file ini explic itly specified enter mlxburn fw fw ConnectX3 rel mlx dev dev mst mt4099 pci cr0 conf CX354A FCB A2 A5 ini SwitchX amp Switch Examples Burn an MSX6025 switch system using the In Band access method mlxburn dev dev mst SW MT51000 000002c900002100 lid 0x000E fw fw sx mlx Generate an MT48436 image and perform an In Band update of the device with LID 0x18 mlxburn dev lid 0x000E fw fw sx mlx e Generate and burn a new MSX6025 via I2C e Set the I2C network to access the InfiniScale IV switch mlxi2c d dev mst mtusb 1 p SX Burn the new image the flash is still blank specifying the Node GUID system GUID base MAC and Switch MAC Note that 4 guids in quotes should be specified as an argument to the guids flag The 2 middle GUIDs are ignored by SwitchX and should b
68. ee e hh hn 91 Appendix C In Band Access to Multiple IB Subnets 0c c cece c cece cece cscs cees 93 Appendix D MTUSB 1 USB to I2C Adapter cece cece cece reece eee rh 95 D OVERVIEWS Sees tees hot ed ore va e SEAN See ues Bie ue Aes DEO wee RN SH aan Obes Faw Mewes 95 0 0 1 Package Contents 95 0 0 2 System Requirements 96 0 0 3 Supported Platforms 96 D 2 HardwareInstallation llleeeeeeeeeeee I nee Ea 96 D 3 Software Installation 0 0 unuunu urnen nerne ereere erreneren reren 96 Appendix E Remote Access to Device by Sockets eeeeeeeeeee enhn 97 E OSerVIe Weed es chin Peds tees Nig WG area UP RID MEM A E ARE RE EIER 97 Appendix F Accessing Remote InfiniBand Device by Direct Route MADs 99 Appendix G Update Package for Mellanox Firmware eee 101 Gl OVERVIEW 5 eure de dete ed ee Sd a Se pene De oR metae red e MNT CES 101 0 0 4 Update Package for Mellanox Firmware Features 101 G 2 Update Package for Mellanox Firmware Generation Flow 0 0 00 cee cence eae 101 0 0 5 mlx_fwsfx_gen Usage 101 2 Mellanox Technologies Rev 2 4 0 0 6 UPMF Generation Example 103 G 3 Updating Firmware Using an UPMF 0 0 cnet ete 105 Appendix H Secure Host Feature 0 ccc cece cece cece cece cee ce sce cs n nn n nn nns 106 H 1 Using the Secure Host 0 cee eee ee nee 106 Appendix I Booting HCA Device in Livefish Mode cc
69. enerating an image for initial burning The image con tains the VPD and the GUIDs that are in a read only area on flash vpd r file vpd r file gt base guid GUID where vpd r file vpd r Embed the given VPD Read Only section in the generated image The file vpd r file should contain the vpd read only section and the first dword of the vpd write able section The file is in binary format and its size must be a multiple of 4 bytes Please refer to PCI base spec for VPD structure info base guid lt GUID gt Set the given GUID as the image base GUID The base GUID is used to derive GUIDs and MACs for the HCA ports It is assumed that 16 GUIDs base guid to base guid 15 are reserved for the card Note On ConnectX 4 only GUIDs will be derived according to the HCA configuration base mac MAC Set the given MAC as the image base MAC The base MAC is used to derive MACS for the HCA ports according to the device configuration ConnectX 4 only Additional mIxburn Options The following is a list of additional options Please see Chapter 3 2 mlxfwmanager Firmware Update and Query Tool for the HCA options use image ps skip is mac s guid s sysguid vsd uid s log blank guids flash params allow psid change no flash verify use image rom ocr use image guids bank Examples of mlxburn Usage Host Channel Adapter Examples To update firmware on an MT25408 Conn
70. es in the format for adapters and bridges CA device id ib node description lid lid number e for switches SW device id lid lid number See Step 3 in Appendix B Flow Examples mlxburn on page 91 for instructions on how to obtain the device LID e Via PCI user level lt bus dev fn gt For example if you run spci d 15b3 Mellanox devices and PCI Device IDs will be displayed sbin lspci d 15b3 02 00 0 Ethernet controller Mellanox Technologies Unknown device 6368 rev a0 Mellanox Technologies 13 Rev 2 4 MFT Installation 2 MFT Installation OS Linux 1 3 4 5 Install Download the Linux MFT package from the Mellanox Manage ment Tools webpage http www mellanox com products management tools php Untar the downloaded package Allow packaging of bins to self executing file Run install sh For OEM only install sh oem Start the mst driver by running mst start NOTE It is possible to customize some installation parame ters such as the target installation path Run install sh help for details Windows The installation is EXE based 1 FreeBSD 1 VMware 1 ad Download the Windows MFT package from the Mellanox Man agement Tools webpage http www mellanox com products management_tools php Double click the EXE file and follow the instructions presented by the installation wizard Download the FreeBSD MFT package from the Mellanox Man agemen
71. et ib3 zi 3 1 2 Windows mst Synopsis mst status v help server start stop ib add del version remote add del hostname Commands and Switches Description There are no mst start or stop operations in Windows de mst server start port Start mst server to allow incoming connection Default port is 23108 mst server stop Stop mst server mst remote add host Establish connection with specified host on specified port name port default port is 23108 Add devices on remote peer to local devices list hostname may be host name as well as an IP address mst remote del host Remove all remote devices on specified hostname lt hostname gt port name port should be specified exactly as in the mst remote add command mst ib del Remove all inband devices mst status Print current status of Mellanox devices mst version Print the version info Mellanox Technologies 19 Rev 2 4 Firmware Generation Configuration and Update Tools mst ib add OPTIONS local_hca_id local_h ca port lst file mst help Examples of mst Usage Add devices found in the IB fabric for inband access Requires MLNX_WinOF installation and an active IB link If local hca id and local hca port are given the IB subnet connected to the given port is scanned Otherwise the default subnet is scanned If an Ist file is specified devices are taken from this file Otherwise ibnetdiscover tool is run to discover
72. explains how to assign a new PSID for a user customized FW and how to indicate to the burning tools that a new PSID exists Please change FW parameters with caution A faulty setting of FW parameters may result in undefined behavior of the burnt device v PSID Field Structure The PSID field is a 16 ascii byte character string If the assigned PSID length is less then 16 characters the remaining characters are filled with binary Os by the burning tool Table 4 provides the format of a PSID Table 4 PSID format Vendor symbol Board Type Symbol Board Version Symbol Parameter Set Number Reserved 3 characters 3 characters 3 characters 4 characters 3 characters filled with 0 Example A PSID for Mellanox s MHXL CF128 T HCA board is MT 0030000001 where MT Mellanox vendor symbol 003 MHXL CF128 T board symbol 000 Board version symbol 0001 Parameter Set Number PSID Assignment and Integration Flow To assign and integrate the new PSID to produce the new FW 1 Write the new FW configuration file in INI format 2 Assign it with a PSID in the format described above Use your own vendor symbol to assure PSID uniqueness If you do not know your vendor symbol please contact your local Mellanox FAE 3 Set the PSID parameter in the new FW configuration file 90 Mellanox Technologies Rev 2 4 Appendix B Flow Examples mlxburn To update an MT48436 InfiniSca
73. fault dev mst mtusb 1 Affected commands all h Print this help information v Print version and exit Commands p lt i2c_component gt Route the i2c path to the indicated i2c component scan Scan the i2c slave addresses Example e Point to a SwitchX device Jl a lxi2c d dev mst mtusb 1 p SX Pointto an InfiniScale IV device to enable accessing it directly by firmware utilities a lxi2c d dev mst mtusb 1 p IS4 PRIM Display the addresses of all I2C accessible devices a lxi2c d dev mst mtusb 1 scan 45 i2c Utility The i2c utility provides low level access to the I2C bus on any Mellanox switch platform enabling the user to read or write data 4 5 1 i2c Usage The mst driver must be started prior to running i2c tool To start i2c 1 Start the MST driver mst start or mst restart 2 Run i2c with the following command line syntax i2c OPTIONS device cmd i2c addr addr lt data gt where h Prints this message a addr width Sets address width in bytes to the specified value May be 0 1 2 or 4 Default 1 d data width Sets data width in bytes to the specified value May be 1 2 or 4s Default is 1 x data len Presents each byte of data as two hexadecimal digits such as 013C20343B Note that this option is mutually exclusive with the d option The remaining parameters are device Valid mst device 1 This ste
74. flint d device e addr where device The device the sector is erased from addr The address of a word in the sector that you want to erase Example flint d dev mst mtusb 1 e 0x1000 3 4 06 Flint mlIxburn Limitations k When running flint mlxburn via an MTUSB 1 device a burn query command may take up to 45 minutes to complete k To accelerate the burn process add the flag no_flash_ verify to the command line which skips the flash verification step This flag however does not verify if the image is burnt correctly a Burning an image to a ConnectX 3 adapter in Flash recovery mode may fail on some server types that use PCIe spread spectrum The tool may not be able to recognize the device s PCI CONFO or the image burn may not complete successfully S To burn the device use the MTUSB 1 connection 54 Mellanox Technologies Rev 2 4 3 5 mlxburn Firmware Image Generator and Burner mlxburn is a tool for firmware FW image generation and or for burning a firmware image to the Flash EEPROM attached to a Mellanox device Both functions or a single function of mlxburn can be activated by means of command line options see Section mlxburn Synopsis It can also query for firmware attributes e g firmware version GUIDs etc and VPD info of adapter cards and switch systems mlxburn allows for customization of standard Mellanox firmware for OEM specific needs e g a specific adapter board type See Se
75. further details please refer to Appendix H Secure Host Feature on page 106 3 4 5 6 Reading a Word from Flash To read one dword from Flash memory use the following command line where flint d device rw addr device The device the dword is read from addr The address of the word to read Example flint d dev mst mt4099 pci cr0 rw 0x20 3 4 5 7 Writing a dword to Flash To write one dword to Flash memory use the following command line flint d device ww addr data where device The device the dword is written to addr The address of the word to write data The value of the word Example flint d dev mst mt4099 pci conf01 ww 0x10008 0x5a445a44 3 4 5 8 Writing a dword to Flash Without Sector Erase To write one dword to Flash memory without sector erase use the following command line flint d device wwne addr data where device The device the dword is written to Mellanox Technologies 53 Rev 2 4 Firmware Generation Configuration and Update Tools addr The address of the word to write data The value of the word Example flint d dev mst mt4099 pci cr0 wwne 0x10008 0x5a445a44 Note that the result may be dependent on the Flash type Usually bitwise and between the speci fied word and the previous Flash contents will be written to the specified address 3 4 5 9 Erasing a Sector To erase a sector that contains a specified address use the following command line
76. h wbne addr size data Write a data block to flash without sector erase rb addr size out file Read a data block from flash The following commands are non failsafe when performed on a 5th generation device sg smg sv and set vpd des Manufacture GUIDs are similar to GUIDs However they are located in the protected area of the flash and set during production By default firmware will use GUIDs unless specified otherwise po during production 3 4 1 Burning a Firmware Image The FLINT utility enables you to burn the Flash from a binary image To burn the entire Flash from a raw binary image use the following command line flint d device i fw file guid GUID guids 4 GUIDS gt mac MAC macs 2 MACs gt burn where device Device on which the flash is burned fw file Binary firmware file 40 Mellanox Technologies Rev 2 4 GUID s Optional for InfiniBand adapters and 4th generation switches One or four GUIDs f 4 GUIDS are provided guids flag they will be assigned as node Port 1 Port 2 and system image GUIDs respectively e If only one GUID is provided guid flag it will be assigned as node GUID Its values 1 2 and 3 will be assigned as Port 1 Port 2 and system image GUID respectively e f no guid guids flag is provided the current GUIDs will be preserved on the device NOTE For 4th generation four GUIDs must be specified but Ports
77. h Present pins Existence and location of Flash Present pins depends on the board manufacture de 1 Supported boards only 108 Mellanox Technologies ail i ar E d Br E ic aii TER NE LE ij th a j i ib Mellanox Technologies 109 Rev 2 4 Troubleshooting Appendix J Burning a New Connect IB Device When burning a flash for the first time the initial image should contain the correct GUIDs and VPD for the device Subsequent firmware updates will not change these initial setting flint for OEM is required for burning Connect IB for the first time au J 1 GUIDs and MACs The Connect IB image contains the Node Port and System GUIDs and Port MACs to be used by the card To simplify GUIDs assignment the mlxburn tool can derive the MACs and GUIDs from a single base GUID according to Mellanox methodology Description UID Number Step Port1 GUID base 8 il Port2 GUID base 8 8 1 Port1 MAC guid2mac base 8 il Port2 MAC guid2mac base 8 8 il 1 guid2mac guid is guid gt gt 16 amp Oxffffff000000 guid amp Oxffffff Meaning remove the 2 middle bytes of an 8 bytes GUID to generate a 6 bytes MAC J 2 PCI Vital Product Data VPD The VPD information is returned by the firmware upon VPD access from the PCI configuration header e The vp ro file last 3 bytes are the vp rw tag id and length e The size of the vp x file including the above 3 bytes should be a mul
78. iguration and Update Tools 3 3 1 3 1 1 Firmware Generation Configuration and Update Tools Mellanox Software Tools mst Service Linux This script is used to start mst service and to stop it It is also used in other operations with Mel mst Synopsis mst lt command gt switches lanox devices such as in resetting or enabling remote access Commands and Switches Description mst start with msix with unknown with i2cdev with lpc dev mst stop mst restart with m six with unknown with i2cdev with lpc dev mst server start port mst server stop mst remote add host name gt port mst remote del lt host name gt port Create special files that represent Mellanox devices in directory dev mst Load appropriate kernel modules and saves PCI configuration headers in directory var mst_pci After successfully completion of this command the mst driver is ready to work and you can invoke other Mellanox tools like Infiniburn or tdevmon You can configure the start command by edit the configuration file etc mft mst conf for example you can rename you devices Options e with_msix Create the msix device e with unknown Do not check if the device ID is supported e with_i2cdev Create Embedded I2C master Stop Mellanox mst driver service remove all special files directories and unload kernel modules Just like mst stop followed by mst start
79. imestamp registers rx period ult speed B 0 3B Oo uu o thernet ethernet hca ethernet port ips mac rx mac config rx timestamp registers rx fns ad ustment mult speed a o 70 Mellanox Technologies Rev 2 4 4 Debug Utilities 4 1 fwtrace Utility The fwtrace utility extracts and prints trace messages generated by the firmware running on 5th generation devices Connect IB and above iRISC These trace messages inform developers of software drivers about internal status events critical errors etc Trace messages generated by iRISCs are stored in the trace buffer The trace buffer is located in host memory The tool also supports mem free mode with which uses a device internal small buffer By default the firmware does not print trace messages Please contact your FAE for more details on how to enable firmware tracing Memory mode on Connect IB device is supported only by PCI mst devices des 4 1 1 fwtrace Usage 1 Start the mst driver mst start or mst restart 2 Enter the following command fwtrace options where h help Print this help message and exit d device mst device name fw strings Fw strings db file containing the FW strings tracer mode Tracer mode FIFO MEM real ts Print real timestamps in hh mm ss nsec format i irisc Irisc name See below for full list of irisc names 8 stream Run in streaming mode c cfg HW tracer events cfg file S buf size HW tracer
80. int d dev mst mt4099 pci cr0 i fw 4099 secure bin b 3 For changes to take effect reboot is required H 1 2 Setting the Secure Host Key To set the key run flint d dev mst mt4099 pci cr0 set key 22062011 Setting the HW Key OK 106 Mellanox Technologies Rev 2 4 Restoring signature OK A driver restart is required to activate the new key o H 1 3 Disabling Enabling Access to the Hardware 1 Access the hardware while hardware access is disabled flint d dev mst mt4099 pci cr0 q E Cannot open dev mst mt4099 pci cr0 HW access is disabled on the device E Run flint d dev mst mt4099 pci cr0 hw access enable in order to enable HW access 2 Enable hardware access flint d dev mst mt4099 pci cr0 hw access enable Enter Key 3 Disable hardware access flint d dev mst mt4099 pci cr0 hw access disable H 1 4 Removing secure host feature gt To remove the secure host feature Step 1 Make sure you have INI and MLX file suitable for the device a Remove cr protection en true from the INI if present b Generate the image using mlxburn for example run mlxburn fw fw 4099 rel mlx conf unsecure host ini wrimage fw 4099 unsecure bin Step2 Burn the firmware on the device make sure hardware access is enabled prior to burning flint d dev mst mt4099 pci cr0 i fw 4099 unsecure bin b Step 3 Execute a driver restart in order to load the unsecure firmware
81. ion Burn the InfiniScale IV switch 92 Mellanox Technologies J Rev 2 4 Appendix C In Band Access to Multiple IB Subnets In most cases an adapter is connected to a single InfiniBand subnet The LIDs InfiniBand Local IDs on this subnet are unique In this state the device access MADs are sent to the target LID from the first active port on the first adapter on the machine In case that the different IB ports are connected to different IB subnets source IB port on the local host should be specified explicitly The device name would be in the format lt any string gt lid lt lid number gt source adapter name source IB port number For example On Linux lid 3 mlx4 0 1 On Windows lid 3 0 1 Say we have the following setup Host H1 HCA mlx4_0 HCAmlx4 1 port 1 port 2 ee Switch1 Switch2 Host H2 Host H3 H1 host has 2 adapters Port 1 of the first adapter is connected to Switch 1 and port 2 of the sec ond adapter is connected to Switch 2 Since the 2 adapters on the H1 are not connected to the each other there are 2 separate IB subnets in this setup Subnet1 nodes H1 Switch 1 and H2 Subnet2 nodes H1 Switch 2 and H3 Mellanox Technologies 93 Rev 2 4 Troubleshooting Running ibv_devinfo command on H1 would list the 2 adapter names For ConnectX adapt ers the names would be mlx4_0 and mlx4 1 Running mst ib add would add ib devices from the default port first active po
82. ion Configuration and Update Tools format Each parameter set file has a unique parameter set ID PSID which is kept in the device Flash EEPROM and allows retaining device configuration during future FW updates During a device FW update mlxburn reads the PSID from the device and uses the correspond ing ini file when generating the FW image mlxburn searches for the files in the same directory of the FW release When mlxburn is used to generate an image file or when no corresponding parameter set file is found the user should explicitly specify which parameter set file to use To produce an image file the user needs to provide the option wrimage lt target file gt To actu ally burn the image to the Flash EEPROM attached to a Mellanox adapter or switch device the user needs to specify the option dev mst device gt see the synopsis section below If run in burning mode mlxburn auto detects the firmware parameter set with which the device was previously burnt It locates and uses this parameter set file to generate the appropriate image for the device by merging the FW release with the specific parameter set required To inhibit image generation the image lt pre generated image file gt should be used It instructs mlxburn to use the given file for burning the device mlxburn Synopsis mlxburn h v dev mst device wrimage fw image fw mellanox fw file image fw image img dir img direcory fw dir
83. isable the Write protection flint d dev mst mt521 pciconf0 ocr hw set Flash0 WriteProtected Disabled 3 Burn the entire flash using the flint tool flint d dev mst mt521 pciconf0 i fw ConnectX4 MCX454 Ax bin ocr ignore dev data allow psid change nofs yes burn 4 Set Write protection flint d dev mst mt521 pciconf0 ocr hw set Flash0 WriteProtected Top 8 SubSectors 5 Enable flash quad SPI IO operations flint d dev mst mt521 pciconf0 ocr hw set QuadEn 1 114 Mellanox Technologies J Rev 2 4 Method 2 Generating Firmware Image with Blank GUIDs Burning and Setting GUIDs on the Device In order to burn a new Switch IB device follow the steps below 1 Generate the initial image VPD for the specific device using the mlxburn tool The generated image occupies full flash size mlxburn fw FW fw ConnextX4 mlx c FW MCX454 Ax ini wrimage fw ConnectX4 MCX454 Ax bin vpd r file vpd r data bin 2 Disable the Write protection flint d dev mst mt521 pciconf0 ocr hw set Flash0 WriteProtected Disabled 3 Bur the entire flash using the flint tool flint d dev mst mt521 pciconf0 i fw ConnectX4 MCX454 Ax bin ocr ignore dev data allow psid change nofs yes burn 4 Set device manufacture GUIDs and MACS run flint d dev mst mt521 pciconf0 ocr guid 0xe41d2d0300570fc0 mac 0x0000e41d2d570fc0 smg 5 Set device GUIDs and MACs run flint d dev mst mt521 pciconf0 ocr
84. kage for Mellanox Firmware G 1 Overview Update Package for Mellanox Firmware UPMF is a new method used to distribute firmware to end users Instead of providing multiple binary files one for each board type and burning them using the flint tool the UPMF method requires only a single standalone file The UPMF is a self extracting executable that contains a set of Mellanox firmware binary images and the mlxfwmanager firmware update tool When executed the UPMF e Extracts its content into a temporary location e Scans the locally installed Mellanox devices firmware versions Performs firmware updates if needed e Cleans up temporary files P The UPMF method and its related tools are released as Beta in MFT 3 1 0 The current firmware update method updating firmware using flint and bin file is still supported G 1 1 Update Package for Mellanox Firmware Features e Single file per firmware release Simple one click firmware update Compact size achieved by efficient compression of the firmware images Noinstallation required G2 Update Package for Mellanox Firmware Generation Flow The mix fwsfx gen tool is used for OEMs that wish to create their own UPMFs that contain their own customized firmware images To install the m1x fwsfx gen tool the installation script should be run with the oem com mand line option G 2 1 mlx fwsfx gen Usage This tool packs the firmware images provided in the input directory and the
85. le IV switch device having a specific GUID for example 0002c90200415190 or LID the following are the recommended steps to update the device firm ware For Linux device names should be listed with the dev mst prefix For Windows no prefix is required ade 1 Make sure all subnet ports are in the active state One way to check this is to run opensm the Subnet Manager rootemymach gt etc init d opensmd start opensm start OK 2 Make sure the local ports are active by running ibv_devinfo Obtain the device LID There are two ways to obtain it a Using the mst ib add command The mst ib add runs the ibdiagnet ibnetdiscover tool to discover the InfiniBand fabric and then lists the discovered IB nodes as an mst device These devices can be used for access by other MFT tools root mymach gt mst ib add I Discovering the fabric Running opt bin ibdiagnet skip all I Added 3 in band devices To list the discovered mst inband devices run mst status root mymach gt mst status MST modules MST PCI module loaded MST PCI configuration module loaded Inband devices dev mst CA MT25418 sw005 HCA 1 lid 0x0001 dev mst SW MT47396 lid 0x0011 dev mst SW MT48438 lid 0x0003 rootemymach Mellanox Technologies 91 Rev 2 4 Troubleshooting b Using the ibnetdiscover tool run 4 Run mlxburn with the LID retrieved in step 3 above to perform the In Band burning opera t
86. loading file s to tmp DownloadDir is done successfully 30 Mellanox Technologies Rev 2 4 3 3 mlxconfig Changing Device Configuration Tool The mlxconfig tool allows the user to change some of the device configurations without burning the firmware once more The configuration is also kept after reset Tool Requirements e OFED WinOF driver to be installed and enabled e Access to the device through the PCI interface pciconf pci cr Firmware supporting changing device configurations feature e Version 2 31 5000 or above for ConnectX 3 ConnectX 3 Pro e Version 10 10 6000 or above for Connect IB e Supported devices ConnectX 3 ConnectX 3 Pro Connect IBG and ConnectX 4 Changing device configurations enabled For changes after a successful configuration to take effect reboot the system de mlxconfig Synopsis mlxconfig d device yl set parameters query reset gt where d dev device Performs operation for a specified mst device y yes Answers yes in prompt v version Displays version info h help Displays help message q ueryl Queries current supported configurations force Enables the user to skip some advanced checks performed by the tool s et Sets configurations to a specific device r eset Resets configurations to their default value 1 Query command will query a single device if a device is specified Otherwise it will query all devices on
87. n fw ConnectX 3 3 bin located in the directory tmp fw ConnectX 3 dir and a PHY image tmp Firmware 1 37 10 N32722 cld into a Linux UPMF package named tmp mlxfw manager 20141126 2 mlx fwsfx gen source dir tmp fw ConnectX 3 dir out dir tmp phy support phy img tmp Firmware 1 37 10 N32722 cld Creating tmp C04TldeQHr phy mfa direcotry Package name tmp mlxfwmanager 20141126 2 Contents Source dirs tmp fw ConnectX 3 dir Adding file etc mft ca bundle crt sfx stub file usr bin mlx sfx stub Creating intermediate MFA archive from binary files fw ConnectX 3 1 fw ConnectX 3 2 fw ConnectX 3 3 Files copied 3 Files queried Compressing bin bin bin Querying images 3 mfa tool usr bin mlx mfa gen mfa cmd usr bin mlx mfa gen p tmp YaH5BAoQ8g srcs mfa s tmp fw ConnectX 3 dir Adding bins from tmp fw ConnectX 3 dir this may take a minute Archive tmp YaH5BAoQ8q srcs mfa Total time 0m1 Adding file t Adding file u Copying tmp Fi Adding file t Adding file u Creating zip t adding srcs adding ca bu S sr bin rmware_ sr bin mfa de ndle cr mp YaH5BAoQ8q srcs mfa mlxfwmanager 1 37 10 N32722 cld to tmp C04TldeQHr phy mfa mp Firmware 1 37 10 N32722 cld mlxphyburn mp YaH5BAoQ8q zippackage zip flated 0 t deflated 45 adding phy mfa stored 0 adding phy mfa Firmware 1 37 10 N32722 cld deflated 44 adding
88. n error as the load new firmware command was already sent to the firmware Mellanox Technologies 65 Rev 2 4 Firmware Generation Configuration and Update Tools 3 7 mlxphyburn Burning Tool for Externally Managed PHY Mlxphyburn Mellanox PHY burn tool allows the user to burn firmware of an externally man aged PHY The tool burns and verifies a pre compiled binary PHY firmware image on the PHY s flash It is supported only on Linux Tool Requirements e ConnectX 3 ConnectX 3 Pro with an externally managed PHY A device that has access to the PHY flash module e MLNX OFED driver if installed should be down Access to the device through the PCI interface pciconf pci cr Firmware version that supports access to an externally managed PHY e Version 2 33 5000 and above mlxphyburn Synopsis mlxphyburn d lt device gt i phy fw image b urn q uery where d dev device Device which has access to the PHY i img PHY fw image PHY firmware image v version Display version info ssh help Display help message b urn Burn given PHY image on the device s PHY q uery Query PHY FW on device If no device is specified mlxphyburn will attempt to burn the PHY firmware image on all mst devices on the machine c Examples of mlxphyburn Usage Burn Example mlxphyburn d dev mst mt4099 pciconf0 i Firmware 1 37 10 N32722 cld burn I attempting to burn PHY Fw on device dev mst mt
89. new Switch IB device in order to set these initial settings Subsequent firmware updates will not change these settings flint for OEM is required for burning Switch IB for the first time des For information regarding GUIDs MACs and VPD please refer to Appendix F Accessing Remote InfiniBand Device by Direct Route MADs on page 99 K 1 Burning the Switch IB Device The examples below are for managed switches For unmanaged switches connect an MTUSB adapter Appendix D MTUSB 1 USB to I2C Adapter on page 95 to the device and use the appropriate mst device dev mst mtusb Method 1 Generating Firmware with Specific GUIDs and Burning it on the Flash In order to burn a new Switch IB device follow the steps below 1 Generate the initial image with the correct GUIDs and VPD for the specific device using the mlxburn tool The generated image occupies full flash size mlxburn fw FW fw SwitchlB mlx c FW MSB7700 E Ax ini wrimage fw SwitchlB MSB7700 E Ax bin base guid 0x0002c903002ef500 vpd r file vpd r data bin 2 Disable the Write protection flint d dev mst mt583 pciconf0 ocr hw set Flash0 WriteProtected Disabled flint d dev mst mt583 pciconf0 ocr hw set Flashl WriteProtected Disabled 3 Burn the entire flash using the flint tool flint d dev mst mt583 pciconf0 i fw SwitchIB MSB7700 E Ax bin ocr ignore dev data allow psid change nofs yes burn 4 Set Write protecti
90. nox devices mst help Print this help information mst version Print mst version information mst server start port Start mst server to allow incoming connection Default port is 23108 mst server stop Stop mst server Examples of mst Usage To print the current status of Mellanox devices Mellanox Technologies 21 Rev 2 4 Firmware Generation Configuration and Update Tools 3 1 4 ESXi mst Synopsis mst command switches Commands and Switches Description mst start Create special files that represent Mellanox devices in direc tory dev Load appropriate modules After successfully complet ing this command the mst driver will be ready to work st stop Stop Mellanox mst driver service and unload the kernel modules st restart mst stop followed by mst start st server start p port Start mst server to allow incoming connection port Default port is 23108 st server stop Stop the mst server st status Print current status of Mellanox devices Options v run with a high verbosity level print more info on each device mst version Print the version info Examples of mst Usage To print the current status of Mellanox devices Native opt mellanox bin mst status MST devices mt4099 pciconf0 mt4099 pci cr0 VMK Linux opt mellanox bin mst status MST devices dev mt4099 pciconf0 dev mt4099 pci cr0 To show the devices status with detailed information opt mellan
91. nter changeable with the mlxfwmanager d dev mst device format mlxfwmanager Synopsis d dev DeviceName h help v version query query format Format u update i image file FileName D image dir DirectoryName f force y yes no l list content archive names clear semaphore exe rel path log L log file LogFileName no progress o outfile OutputFileName nofs ssl certificate Certifi cate online online query PSIDs key key download DirectoryName download default download device Device download os OS download type Type where dl dev DeviceName Perform operation for specified mst device s Run mst status com mand to list the available devices Multiple devices can be specified delimited by semicolons A device list containing semicolons must be quoted hl help Show this message and exit vl version Show the executable version and exit query Query device s info query format Format Query Online query output format XML Text default Text ul update Update firmware image s on the device s il image file FileName Specified image file to use DI image dir DirectoryName Specified directory instead of default to locate image files fl force Force image update yl yes Answer is yes in prompts no Answer is no in prompts Mellanox Technologies 23 Rev 2 4 Firmware Generation Configuration and Update T
92. o parser mode default off Mode CollectMode d device MstDev Mst device m mode Mode Activation mode FIFO HW BUFFER MEM KER NELB UFFER possible values FIFO MEM a mem_access MemMethod Memory access method OB_GW MEM DMEM FMEM VMEM possible values OB_GW MEM DMEM FMEM VMEM c cfg CfgFile Mixtrace configuration file 78 Mellanox Technologies Rev 2 4 o tre_file TrcPath Output TRC file path default mlxtrace trc C config only Configure tracer and exit default off n snapshot Take events snapshot this assumes previous run with con fig only default off s buf size BufSize User buffer size MB default 1 S stream Don t save events to file parse it immediately default off ignore old events Ignore collecting old events default off g continuous fill Do not stop recording stopping only with C keep filling user s buffer cyclicly default off sample delay Delay Delay between samples when polling new events in usec default 0 keep running Keep the HW tracer unit running after exit default off Mode ParseMode i input TrcFile Input file default mlxtrace trc csv mode Enable csv output format default off print ts Print timestamp events default off r real ts Print real timestamps in hh mm ss nsec format default off print raw Print event bytes in each line header default off ts format format
93. on flint d dev mst mt583 pciconf0 ocr hw set Flash0 WriteProtected Top 2 SubSectors 5 Enable flash quad SPI IO operations flint d dev mst mt583 pciconf0 ocr hw set QuadEn 1 112 Mellanox Technologies J Rev 2 4 Method 2 Generating Firmware Image with Blank GUIDs Burning and Setting GUIDs on the Flash In order to burn a new Switch IB device follow the steps below 1 Generate the initial image VPD for the specific device using the mlxburn tool The generated image occupies full flash size mlxburn fw FW fw SwitchlB mlx c FW MSB7700 E Ax ini wrimage fw SwitchlB MSB7700 E Ax bin vpd r file vpd r data bin 2 Disable the Write protection flint d dev mst mt583 pciconf0 ocr hw set Flash0 WritePro tected Disabled flint d dev mst mt583 pciconf0 ocr hw set Flashl WritePro tected Disabled 3 Burn the entire flash using the flint tool flint d dev mst mt583 pciconf0 i fw SwitchIB MSB7700 E Ax bin ocr ignore dev data allow psid change nofs yes burn 4 Set device manufacture GUIDs flint d dev mst mt583 pciconf0 ocr uid 0x0002c903002ef500 smg 5 Set device GUIDs flint d dev mst mt583 pciconf0 ocr uid 0x0002c903002ef500 sg 6 Set Write protection lint d dev mst mt583 pciconf0 ocr hw set Flash0 WriteProtected Top 2 SubSectors flint d dev mst mt583 pciconf0 ocr hw set Flashl WriteProtected Top 1 SubSectors 7 Enable flash quad SPI IO operations flint d dev ms
94. ool fails with the following message E Unsupported binary version 2 0 please update to lat est MFT package The binary version is incom patible with the burning tool Update MFT to the latest package mlxburn tool fails to gener ate a firmware image and displays the following mes sage E Unsupported MLX file version 2 0 please update to lat est MFT package The MLX file version is incompatible with the image generation tool mlxburn Update MFT to the latest package mlxburn tool fails to gener ate a firmware image and displays the following mes sage E Perl Error Image generation tool uses tool version 1 5 0 that is not sup ported for creating a bin file for this FW version FW requires mic version 2 0 0 or above Please update MFT package mic The MLX file version is incompatible with the image generation tool mlxburn Update MFT to the latest package Mellanox Technologies 89 Rev 2 4 Troubleshooting j Appendix A PSID Assignment A 1 A 2 In some cases OEMs or board manufacturers may wish to use a specific FW configuration not supplied by Mellanox After setting the new FW parameters in an INI file the user should assign a unique PSID Parameter Set ID to this new configuration The PSID is kept as part of the FW image on the device NVMEM The firmware burning tools use this field to retain FW settings while updating FW versions This appendix
95. ools ll list content archive names clear semaphore exe rel path log Ll log file LogFileName no progress ol outfile OutputFileName nofs online online query psid PSIDs key key download DirectoryName download default download device Device download os OS download type Type ssl certificate Certificate List file Directory content used with image dir and image file flags Display archive names in listing Force clear of the flash semaphore on the device No command is allowed when this flag is used NOTE May result in system instability or flash corruption if the device or another application is currently using the flash Exercise caution Use paths relative to the location of the executable Create a log file Use a specified log file Do not show progress Write to a specified output file Burn image in a non failsafe manner Fetch required FW images online from Mellanox server Query FW info PSID s are comma separated Key for custom download update Download files from server to a specified directory Use Default values for download Options are ConnectX Connect IB ConnectX 4 e default All Options are only for sfx e Linux x64 e Windows e Windows x64 Linux PPC64 Linux PPC64le e FBSDIO 64 e FBSDIO 32 e FBSDIO 1 64 e FBSDII 64 e default All MFA self extractor default All SSL certificate For secure connection 24
96. ort specify guids_num lt num gt step size lt size gt to the sg command 1 Change GUIDs for device flint d dev mst mt4113 pciconf0 q Image type FS3 FW Version 10 10 3000 FW Release Date 29 4 2014 Description UID GuidsNumber Step Base GUID1 0002c903002e 500 8 1 Base GUID2 0002c903002ef508 8 iL Base MAC1 00000002c92ef500 8 1 Base MAC2 00000002c92ef508 8 1 Image VSD Device VSD VSD PSID MT 1240110019 flint d dev mst mt4113 pciconf0 uid 0002c123456abcd ocr sg W Firmware flash cache access is enabled Running in this mode may cause the firmware to hang Updating GUID section OK Updating ITOC section OK Restoring signature OK flint d dev mst mt4113 pciconf0 q Image type FS3 FW Version 10 10 3000 FW Release Date 29 4 2014 Description UID GuidsNumber Step Base GUID1 00002c123456abcd 8 1 Orig Base GUID1 0002c903002ef500 8 Base GUID2 00002c123456abd5 8 Orig Base GUID2 0002c903002ef508 8 Base MAC1 000000002c56abcd 8 Orig Base MAC1 00000002c92ef500 8 Base MAC2 000000002c56abd5 8 Orig Base MAC2 00000002c92ef508 8 Image VSD Device VSD VSD PSID MT 1240110019 Mellanox Technologies 47 Rev 2 4 Firmware Generation Configuration and Update Tools 2 Change GUIDS for device specifying guids_num and step_size 48 Mellanox Technologies Rev 2 4 3 Change GUIDs for image flint i tmp connect ib bin q Image type FS3 FW Version
97. ox bin mst status v PCI devices DEVICE TYPE MST BEN RDMA NET NUMA ConnectX3 rev 1 mt4099 pciconf0 Connect X3 rev 1 mt4099 pci cro 0b 00 0 For further information on In Band and Remote Access please refer to Appendix C In Band Access to Multiple IB Subnets on page 93 Appendix F Accessing Remote InfiniBand Device by Direct Route MADs on page 99 and Appendix E Remote Access to Device by Sockets on page 97 22 Mellanox Technologies Rev 2 4 3 2 mlxfwmanager Firmware Update and Query Tool The mlxfwmanager is a Mellanox firmware update and query utility which scans the system for available Mellanox devices only mst PCI devices and performs the necessary firmware updates mlxfwmanager utility has the following flavors e mlxfwmanager operates on mst devices and device aliases and requires MFT installa tion and running mst start mlxfwmanager pci operates on local Mellanox PCI devices and does not require MFT installation mlxfwmanager_pci gets a PCI device in format bus dev fn as an input to the device option Other than the input device format the mlxfwmanager and mlxfwmanager_pci tools share the same command line and functionality For further information on firmware update please refer to Appendix G Update Package for Mellanox Firmware on page 101 The examples throughout the document use pci bus dev fn format However all the exam ples are i
98. ox com gt Products gt Adapter IB VPI Drivers gt Linux SW Driver e Mellanox WinOF for Windows is a software stack that can be downloaded from http www mellanox com gt Products gt Adapter IB VPI Drivers gt Windows SW Driver e ibdiag tools run man ibdiagnet for details on a machine with OFED installed e Mellanox OFED for ESXi is a software stack that can be downloaded from http www mellanox com gt Products gt Adapter IB VPI Drivers gt VMware Driver e Mellanox OFED for FreeBSD is a software stack that can be downloaded from http www mellanox com gt Products gt Adapter IB VPI Drivers gt VMware Driver Mellanox Technologies 9 Rev 2 4 MFT Web Page Please visit http www mellanox com gt Products gt InfiniBand VPI Drivers gt Firmware Tools for downloads 10 Mellanox Technologies Rev 2 4 1 Introduction The Mellanox Firmware Tools MFT package is a set of firmware management and debug tools for Mellanox devices MFT can be used for Generating a standard or customized Mellanox firmware image Querying for firmware information Burning a firmware image to a single Mellanox device The list of the available tools in the package can be found in the Release Notes document Figure 1 Mellanox Firmware Tools A Scheme of Operation Mellanox USB TO C Adapter Mellanox InfiniScale IV Mellanox SwitchX Based Based Switch System pp
99. p in not required in Windows 76 Mellanox Technologies J Rev 2 4 cmd Command May be r ead or w rite i2c addr I2C slave address lt addr gt Address of length addr_width inside I2C target device to read write operation Note that the lt addr gt value is ignored if lt addr_witdh gt 0 lt data gt Data bytes of length data_width to write to target device All parameters are interpreted as hexadecimal values A Examples 1 Read two bytes from address 0 of target I2C device at address 0x56 i2c a 2 d 2 dev mst mtusb 1 r 0x56 0x00 0000 2 Write two bytes to the address above then read them i2c a 2 d 2 dev mst mtusb 1 w 0x56 0x00 0x1234 i2c a 2 d 2 dev mst mtusb 1 r 0x56 0x00 3412 3 Read as separate 16 bytes in hexadecimal format starting from address 0 of the target device above i2c a 2 x 16 dev mst mtusb 1 r 0x56 0x00 12340000000000000000000000000000 4 5 2 Exit Return Values The following exit values are returned e 0 successful completion gt 0 an error occurred 4 6 mget_temp Utility The mget_temp utility reads the hardware temperature from Mellanox Technologies devices with temperature sensors all Mellanox devices and prints the reading in Celsius degrees 4 6 1 mget_temp Usage The mst driver must be started prior to running mget_temp To run mget_temp 1 Start the mst driver mst start or mst restart 2 Run mget_temp with the follo
100. params PARAMS quiet version hopcount advanced Show this help message and exit mst device to use required MCG dump file to use for debug Mcg params MCG ENTRY SIZE HASH TABLE SIZE MCG TABLE SIZE default is 64 32768 65536 Do not print progress messages to stderr Print tool version Add hopCount column Show all rules This will display all the current multicast groups and flow steering rules configured in the device Mellanox Technologies 81 Rev 2 4 Debug Utilities Example Command mlxmcg d dev mst mt4099 pci cr0 MCG table size 64 K entries Hash size 32 K entries Entry size 64 B Progress HHHHHLLLL Bucket Index ID Prio Proto DQP Port VLAN MAC SIP DIP I MAC I VLAN VNI L4 SPort DPort Next QPs 00 00 all ec DEL BS Sb oe Ga ce ee a ee AOE aA af9 fee0 0 5000 IPv6 ff0e 0000 0000 0000 0000 0000 6000 0001 8012 SB e3f e3f 05000 L2 2 ee 1 580 5c72500 800 8 8 cc me seo ee as es So BIA OQ WISE ALS 0 5549008 p cc 2 is obs 0o Sac 06 1l oo ec a ee ee se ee BA Aboot 26fc 26fc 0 5000 L2 2 se ff ff ff ff ff ff 8018 40048 FAIS ABNS 0 OO ly a 2 e 23933 809 5005 0 8 Hl o 2 sc sa eo 99 e em Bu MODUS 4000 4000 0 0 alll 2 os ce mG GS Go uo s e a MOOSE iL 45d7 45d7 0 5000 L2 1 ee Oils Os SecO0sOUgitl e ee so gt QUO opus 4af9 fef8 0 5000 IPv6 0e 0000 0000 0000 0000 0000 e000 0001 8002 SB 4e
101. queue buffer which is a virtually contiguous memory buf fer allocated when creating the QP Mellanox Technologies 83 Rev 2 4 Debug Utilities The dumped data can be used for hardware troubleshooting It can be applied on ConnectX amp and Connect IB Mellanox adapter devices wqdump on all ConnectX devices except for ConnectX 4 devices is not supported against in band devices Tom 4 12 1 wqdump Usage The mst driver must be started prior to running the wqdump utility To start the wqdump utility 1 Start the mst driver mst start or mst restart 2 Run wqdump WQDump WODump d device DeviceName gt lt source ContextType gvmi Gvmi qp ContextNumber dump DumpType fi StartIndex num NumberOfItems format Format address Address size Size v version h help clear_semaphore gw_access where d device DeviceName Device name source ContextType Type of context to dump Options are Snd Rev Cmp Srq Ege con nect X MCGconnect IB MKC SXDC FullQp gvmi Gvmi Guest VM ID connect IB only qp ContextNumber Context number to dump dump DumpType Dump Type Options are WQ QP WQ_QP ALL_QPC ALL_VALID_QPNS ICM fi StartIndex Index of first element to dump Default 0 num NumberOfItems Number of elements to dump from buffer Default keep reading format Format Output format options are text raw dw Default text address Address Memor
102. ribed below 1 Start the mst driver Enter mst start or mst restart if mst start was run earlier 2 To obtain the list of mst devices enter mst status If MTUSB 1 has been correctly installed mst status should include the following device in the device list it generates On Linux dev mst mtusb 1 On Windows mtusb 1 1 This step in not required in Windows 96 Mellanox Technologies Rev 2 4 Appendix E Remote Access to Device by Sockets E 1 Overview The mst device on a machine can be accessed server side remotely for debugging purposes using the minimum set of tools from another machine client side which may have more tools or faster machine To do so The mst server should run on the server side machine Run mst server start The client side should add the mst server side Run mst remote add server side machine IP After remote devices are added to the mst list device in the client side you can run any tool that accesses the mst devices of the server side as seen in the example below Usage of relevant command Command Description mst server start port Starts mst server to allow incoming connection Default port is 23108 mst server stop Stops mst server mst remote add host Establishes connection with a specified host on a specified port default name gt port port is 23108 Adds devices on remote peer to local the devices list e lt hostname
103. rn fw FW fw ConnectIB mlx c FW MCB194A FCA Al ini wrimage fw ConnectIB MCB194A FCA Al bin vpd r file vpd r data bin 2 Disable the Write protection flint d dev mst mt511 pciconf0 ocr hw set Flash0 WriteProtected Disabled 3 Burn the entire flash using the flint tool flint d dev mst mt5111 pciconf0 i fw ConnectIB MCB194A FCA Al bin ocr ignore dev data allow psid change nofs yes burn 4 Set device manufacture GUIDs run flint d dev mst mt511 pciconf0 ocr uid 0x0002c903002ef500 smg 5 Set device GUIDs run flint d dev mst mt511 pciconf0 ocr uid 0x0002c903002ef500 sg 6 Set Write Protection on the last sector using the flint tool For devices using Winbond flash flint d dev mst mt511 pciconf0 ocr hw set Flash0 WriteProtected Top 1 SubSectors 7 Enable flash quad SPI IO operations flint d dev mst mt511 pciconf0 ocr hw set QuadEn 1 gt To view flash settings run flint d dev mst mt511 pciconf0 ocr hw query gt To view assigned GUIDs run flint d dev mst mt511 pciconf0 ocr q To change a GUID after the initial burn run flint d dev mst mt4113 pciconf0 ocr uid 0x0002c903002ef500 sg Mellanox Technologies 111 Rev 2 4 Troubleshooting Appendix K Burning a New Switch IB Device Upon first time flash burning the GUIDs and VPD of the device are required to be set on the flash The sections below demonstrate two methods of burning a
104. rt on the first adapter only Subnet1 nodes would be listed To add the nodes of the second subnet the source adapter and port should be specified to the mst ib add command in the following format mst ib add hca name hca port Example 1 Add nodes of both subnets Run mst ib add mlx4_0 1 mst ib add mlx4 1 2 2 List the devices mst status dev mst CA MT25418 H1 HCA 1 lid 0x0001 mlx4 0 1 dev mst CA MT25418 H2 HCA 1 lid 0x0005 mlx4 0 1 dev mst SW MT51000 Switchl lid 0x0003 mlx4 0 1 dev mst CA MT25418 H1 HCA 1 lid 0x0010 mlx4 1 2 dev mst CA MT25418 H3 HCA 1 lid 0x0012 mlx4 1 2 dev mst SW MT51000 Switch2 lid 0x0005 mlx4 1 2 You can use the above device names with the MFT tools Fo 94 Mellanox Technologies Rev 2 4 Appendix D MTUSB 1 USB to I2C Adapter D 1 Overview The MTUSB 1 is a USB to IC bus adapter This chapter provides the user with hardware and software installation instructions on machines running Linux or Windows operating systems Figure 3 MTUSB 1 Device D 1 1 Package Contents Please make sure that your package contains the items listed in Table 5 and that they are in good condition Table 5 MTUSB 1 Package Contents Item Quantity Description MTUSB 1 device 1 USB to C bus adapter USB cable 1 USB_A to USB_B 1 8m I2C cable 1 9 pin male to male cable 1 5m Converter cable 2 9 pin female to 3 pin small large 0 3m Mellanox Te
105. service openibd restart H 1 5 Recover Lost Key If a key is lost there is no way to recover it using the tool The only way to recover it is to Step 1 Connect the flash not present jumper on the card Step 2 Reboot the machine Step3 Re burn firmware Step 4 Remove the flash not present jumper Step 5 Reboot the machine Step 6 Re set the hardware access key Mellanox Technologies 107 Rev 2 4 Troubleshooting Appendix I Booting HCA Device in Livefish Mode In case a MLNX HCA fails to boot properly and is not being identified by the system due to a corrupt firmware the user is able to boot the card in livefish mode in order to re burn the card To do so a direct access to the card is needed By connecting the two flash present pins using a jumper while the machine is powered off the card will boot in Flash not present mode the firm ware will not be loaded from the flash i e livefish l1 Booting Card in Livefish Mode To boot the card in livefish mode Step 1 Power off the machine Step 2 Locate the Flash preset pins on the HCA Step3 Close the two pins using a jumper Step 4 Power on the machine 1 2 Booting Card in Normal Mode gt To boot the card in normal mode Step 1 Power off the machine Step 2 Take off the jumper connected to the Flash Present pins on the HCA Step 3 Power on the machine 1 3 Common Locations of Flash Present Pins The following photos show common locations of the Flas
106. t HCA The HCA ignores this value Commands affected burn sg guids lt GUIDs gt 4 GUIDs must be specified here The specified GUIDs are assigned to the following fields repectively node port1 port2 and system image GUID NOTE port2 guid must be specified even for a single port HCA The HCA ignores this value It can be set to 0x0 Commands affected burn sg h elp Prints this message and exits hh Prints extended command help i mage image Binary image file Commands affected burn verify log lt log_file gt Prints the burning status to the specified log file mac lt MAC gt 1 MAC address base value 2 MACs are automatically assigned to the following values mac gt portl mac 1 gt port2 Commands affected burn sg macs lt MACs gt 2 MACs must be specified here The specified MACs are assigned to port1 port2 repectively Commands affected burn sg NOTE mac macs flags are applicable only for Mellanox Technolo gies ethernet products no Non interactive mode assume answer no to all questions Commands affected all no flash verify Do not verify each write on the flash nofs Burns image in a non failsafe manner allow rom change Allows burning removing a ROM to from Firmware image when product version is present 1 The mac and macs options are applicable only to Mellanox Technologies Ethernet adapter and switch devices Mellanox Technologies 37 Rev 2 4 Firmware Gener
107. t Tools webpage http www mellanox com products management_tools php Untar the downloaded package Run install sh Download the MFT for VMware vib package from http www mellanox com products management tools php Install the package Run 4 esxcli software vib install v MST Vib gt 4 esxcli software vib install v MFT Vib gt NOTE For VIBs installation examples please see below Reboot system Start the mst driver Run opt mellanox bin mst start Example VIBs installation e VMK esxcli esxcli Native esxcli esxcli Uninstall Stop the mst driver by running mft uninstall sh 1 Go to Add or remove programs 2 Remove WinMFT64 depending on the platform type Uninstall MFT on FreeBSD run the following command mft uninstall sh 1 Uninstall the package Run esxcli software vib remove n mft 2 Uninstall the mst e VMKlinux esxcli software vib remove n net mft Native esxcli software vib remove n nmst 3 Reboot system Software vib install v tmp net mst 4 0 0 22 10EM 600 0 0 2295424 vib software vib install v tmp mft 4 0 0 22 10EM 600 0 0 2295424 x86 64 vib software vib install v tmp nmst 4 0 0 22 10EM 600 0 0 2295424 x86 64 vib software vib install v tmp mft 4 0 0 22 10EM 600 0 0 2295424 x86 64 vib 14 Mellanox Technologies 1 The path is temporary and will hold only until reboot Mellanox Technologies 15 Rev 2 4 Firmware Generation Conf
108. t mt583 pciconf0 ocr hw set QuadEn 1 gt To view flash settings flint d dev mst mt583 pciconf0 ocr hw query To view assigned GUIDs flint d dev mst mt583 pciconf0 ocr q gt To change a GUID after the initial burn flint d dev mst m52000 pciconf0 ocr uid 0x0002c903002ef500 sg Mellanox Technologies 113 Rev 2 4 Troubleshooting Appendix L Burning a New ConnectX 4 Device Upon first time device burning the GUIDs MACs and VPD of the device are required to be set on the flash The sections below demonstrate two methods of burning a new ConnectX 4 device in order to set these initial settings Subsequent firmware updates will not change these settings flint for OEM is required for burning ConnectX 4 for the first time P For information regarding GUIDs MACs and VPD please refer to Appendix F Accessing Remote InfiniBand Device by Direct Route MADs on page 99 L 1 Burning the ConnectX 4 Device Method 1 Generating Firmware with Specific GUIDs and Burning it on the Device In order to burn a new ConnectX 4 device follow the steps below 1 Generate the initial image with the correct GUIDs and VPD for the specific device using the mlxburn tool The generated image occupies full flash size mlxburn fw FW fw ConnextX4 mlx c FW MCX454 Ax ini wrimage fw ConnectX4 MCX454 Ax bin base guid 0xe41d2d0300570fc0 base mac 0x0000e41d2d570fc0 vpd r file vpd r data bin 2 D
109. ter Set ID than the one currently on flash Note that changing a PSID may cause the device to malfunction Use only if you know what you are doing Set the number of attached flash devices banks Burn the image with blank GUIDs and MACS where applicable These values can be set later using the sg command see details below Commands affected burn Force clear the flash semaphore on the device No command is allowed when this flag is used NOTE May result in system instability or flash corruption if the device or another application is currently using the flash Exercise caution Device flash is connected to Commands affected all Make the burn process burn two images on flash previously default algorithm Current default failsafe burn process burns a single image in alternating locations Commands affected burn Use the given parameters to access the flash instead of reading them from the flash Supported parameters e Type The type of the flash such as M25PXxx M25Pxx SST25VFxx W25QxxBV W25Xxx AT25DFxxx S25FLXXXP log2size The log2 of the flash size num of flashes the number of the flashes connected to the device 36 Mellanox Technologies Rev 2 4 guid lt GUID gt GUID base value 4 GUIDs are automatically assigned to the follow ing values guid gt node GUID guid 1 gt portl guid 2 gt port2 guid 3 gt system image GUID NOTE port2 guid will be assigned even for a single por
110. ter card dual port QSFP FDR IB 56Gb s and 40GigE PCIe3 0 x8 8GT s RoHS R6 PSID MT 1020120019 PCI Device Name dev mst mt4099 pci cr0 Port1 GUID 0002c9000100d051 Port2 MAC 0002c9000002 Versions Current Available FW Ay dik SOS A 32 5501000 Status Update required Found 2 device s requiring firmware update Please use u flag to perform the update c Query XML mlxfwmanager query query format XML lt Devices gt Device pciName dev mst mt4099 pci cr0 type ConnectX3 psid MT_1200111023 part Number MCX354A FCA A2 A4 gt lt Versions gt lt FW current 2 1 0065 available 2 32 5000 gt lt Versions gt lt MACs portl 02c90abcdef0 port2 02c90abcdef1 gt lt Status gt update required lt Status gt lt Description gt ConnectX 3 VPI adapter card dual port QSFP FDR IB 56Gb s and 40GigE PCIe3 0 x8 8GT s RoHS R6 lt Description gt lt Device gt Device pciName dev mst mt4113 pciconf0 type ConnectIB psid MT 1220110030 part Number MCB192A FCA A1 Versions FW current 2 11 1258 available 10 10 4000 gt lt Versions gt GUIDs port1 0002c903002ef500 gt MACs port1 0002c903002ef501 gt Status update required Status Description Connect IB Host Channel Adapter single port QSFP FDR 56Gb s PCIe2 0 X16 RoHS R6 Description Device Devices 26 Mellanox Technologies Rev 2 4 3 2 2 Updating the Device To update a device on the m
111. the machine 2 The use of the force flag is not recommended and is intended for advanced users only 1 For ConnectX 4 support is at beta level and it only supports VPI settings Mellanox Technologies 31 Rev 2 4 Firmware Generation Configuration and Update Tools Supported Configurations and their Parameters Table 3 Supported Configurations and their Parameters Feature Description Parameter Values SR IOV Sets the device virtu SRIOV EN 0 disable alization parameters Enables or disables virtualiza 1 enable tion NUM OF VFS NUM 1 to 127 maximal number of Sets the number of virtual virtual function may be functions to allocate smaller as it depends on the PCI BAR size and available system resources Wake on LAN Enables disables WOL MAGIC EN P1 0 disable Wake on LAN fea Enables or disables Wake on enable ture magic packet for port 1 WOL MAGIC EN P2 0 disable Enables or disables Wake on enable magic packet for port 2 BAR size Sets the BAR size LOG BAR SIZE 0 to 9 maximal size may be that the system allo Log base 2 of the number of smaller as it depends on SR cates per physical megabytes to be allocated per IOV settings and virtual function physical and virtual function VPI settings Configures the port s VPI SETTINGS PORT1 1 InfiniBand working mode Configures port 1 working 2 Ethernet mode 3 VPI auto sense VPI SETTINGS PORT2 ConnectX 3 ConnectX 3 Pro
112. the fabric Options e discover tool discover tool The tool used to discover the fabric Supported tools ibnetdiscover ibdiagnet default ibnetdiscover NOTE The discover tool argument is intended only for parsing pur poses thus you need to specify an Ist file with it e add non mlnx Add non Mellanox nodes e use ibdr Access by direct route MADs Available only when using ibnetdiscover tool for SwitchX and ConnectIB devices e no format check Do not check the format of the given local hca id The expected format of the local hca id is ibv_device 0 9 e topo file lt topology files gt A prepared topology file which describes the fabric For ibnetdiscover provide an output of the tool For ibdiagnet provide and Ist file that ibdiagnet generates NOTE If a topology file is specified the devices are taken from it Otherwise a discover tool is run to discover the fabric Print this help information To print the current status of Mellanox devices mst status MST devices mt4099 pci cro mt4099 pciconf0 mtusb 1 mtusb 2 To show the devices status with detailed information mst status v MST devices mt4099 pci cr0 bus dev fn 13 00 0 mt4099 pciconf0 bus dev fn 13 00 0 mtusb 1 iSerial 0xlccc mtusb 2 iSerial 0x1cd5 20 Mellanox Technologies Rev 2 4 3 1 3 FreeBSD mst Synopsis Commands and Switches Description mst status Print current status of Mella
113. tiple of 4 J 3 Burning a New Connect IBe Device The VPD and GUIDs are stored in the last sector on flash that can be set as Write protected after the initial firmware burn J 3 1 To burn the Connect IBe device Method 1 Generating Firmware With Specific GUIDs and Burning on the Flash 1 Generate the initial image with the correct GUIDs and VPD for the specific device using the mlxburn tool The generated image occupies full flash size mlxburn fw FW fw ConnectIB mlx c FW MCB194A FCA Al ini wrimage fw ConnectIB MCB194A FCA Al bin base guid 0x0002c903002ef500 vpd r file vpd r data bin 2 Disable the Write protection flint d dev mst mt511 pciconf0 ocr hw set Flash0 WriteProtected Disabled 110 Mellanox Technologies Rev 2 4 3 Burn the entire flash using the flint tool flint d dev mst mt511 pciconf0 i fw ConnectIB MCB194A FCA Al bin ocr ignore dev data allow psid change nofs yes burn 4 Set Write protection on the last sector using the flint For devices using Winbond flash flint d dev mst mt511 pciconf0 ocr hw set Flash0 WriteProtected Top 1 SubSectors 5 Enable flash quad SPI IO operations flint d dev mst mt511 pciconf0 ocr hw set QuadEn 1 Method 2 Generating Firmware Image With Blank GUIDs Burning and Setting GUIDs on the Flash 1 Generate the initial image with VPD for the specific device using the mlxburn tool The gen erated image occupies full flash size mlxbu
114. trary default GUIDs mlxburn fw fw ConnectX3 rel mlx c MCX354A FCB A2 A5 ini wrimage fw ConnectX3 rel bin striped image 2 Per device set the device specific GUIDs in the image flint i fw ConnectX3 rel bin guid 0x0002c9030abcdef0 mac 0x0002c9bcdef1 striped image sg 3 Optional Query the image to verify the GUIDs are set The striped_image flag must be specified when querying a striped image flint i fw ConnectX3 rel bin striped image q Image type FS2 FW Version 2 31 5050 FW Release Date 4 5 2014 Device ID 4099 Description Node Porti Port2 Sys image GUIDs 0002c9030abcdef0 0002c9030abcdef1 0002c9030abcdef2 0002c9030abcdef3 ACs 0002c9bcdef1 0002c9bcdef2 VSD n a PSID MT 1090120019 Mellanox Technologies 51 Rev 2 4 Firmware Generation Configuration and Update Tools Now the fw ConnectX3 rel bin image can be pre burned to the flash After the assembly the device would be fully functional 3 4 5 4 Setting the VSD To set the vsd for the given image device 4th generation use the sv command with vsa flag Example flint d dev mst mt4099 pci cr0 vsd MELLANOX sv Setting the VSD OK Restoring signature OK flint d dev mst mt4099 pci cr0 q Image type FS2 FW Version A Si S050 FW Release Date 4 5 2014 Device ID 4099 Description Node Portl Port2 Sys image GUIDs 45214030001b8a0 f45214030001b8al 45214030001b8a2 45214030001b8a3 ACs 00300abada
115. ver the fabric mst ib del Remove all inband devices mst status Print current status of Mellanox devices Options e y run with high verbosity level print more info on each device mst save Save PCI configuration headers in directory var mst pci mst load Load PCI configuration headers from directory var mst pci mst version Print the version info Using mst conf File in Linux Edit the etc mft mst conf configuration file to configure the start operation in Linux only The configuration file consists of lines of rules Every line will be a rule for mst start It must be valid and the rules should be unique There should also be no duplication of new names and or serials The rule general format is the following SOPCODE PARAMS Table 2 mst start Supported OPCODES OPCODES Definition Description RENAME renames mst devices Rule format RENAME TYPE NEW NAME ID Supported types MTUSB where ID is the iSerial e Example RENAME USB my Ox2A4C Effect MTUSB with serial 0x2A4C will be renamed to dev mst mymtusb 1 always mtusb 1 will be concatenated to the new name Mellanox Technologies 17 Rev 2 4 Firmware Generation Configuration and Update Tools Examples of mst Usage gt To start Mellanox mst driver service mst start Starting MST Mellanox Software Tools driver set Loading MST PCI module Success Loading MST PCI configuration module Success Create devices
116. vision is 01 dev mst mt4099 pci cr0 PCI direct access domain bus dev fn 0000 0b 00 0 bar 0xd2600000 size 0x100000 Chip revision is 01 Remote MST devices dev mst mft 23108 devamst mt26428 pciconf0 Chip revision is B0 dev mst mft 23108 devemst mt26428 pci cro Chip revision is B0 dev mst mft 23108 Gdevamstemtusb 1 5 Access a remote mst device from the client side rootemftl flint d dev mst mft 23108 0devamstGmt4099 pci cr0 q Image type FS2 FW Version 232 O92 FW Release Date 17 8 2014 Rom Info type PXE version 3 4 300 devid 4099 proto VPI Device ID 4099 Description Node Portl Port2 Sys image GUIDs 0002c90300e6e4e0 0002c90300e6e4e1 0002c9030066e4e2 0002c90300e6e4e3 ACS 0002c9e6e4e1 0002c9e6e4e2 VSD n a PSID MT 1090120019 98 Mellanox Technologies Rev 2 4 Appendix F Accessing Remote InfiniBand Device by Direct Route MADs gt To access a SwitchX switch or Connect IB device remotely by direct route MADs 1 Make sure the local ports are connected to a node or more ibstat or ibv_devinfo 2 Obtain the device direct route path mst ib add use ibdr discover tool ibnetdiscover mlx5 0 1 I Discovering the fabric Running ibnetdiscover s C mlx5 0 P 1 I Added 2 in band devices 3 List the discovered direct route device mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST devices dev mst CA MT411
117. ware Image To query the FW image on a device use the following command line flint d device q To query the FW image in a file use the following command line flint i image file q 42 Mellanox Technologies Rev 2 4 where device Device on which the query is run image file Image file on which the query is run Examples a Query the FW on the device flint d dev mst mt4099 pciconf0 query b Query the FW image file flint i 25408 2 30 5000 MCX354A FCB A2 bin query 3 4 3 Verifying the Firmware Image To verify the FW image on the Flash use the following command line He lie Cl glewnuse v e To verify the FW image in a file use the following command line flint i image file v where device Flash device to verify image file Image file to verify Examples flint d dev mst mt4099 pci cr0 v flint i image file bin v 3 4 4 Managing an Expansion ROM Image To burn an Expansion ROM image use the following command flint d mst device brom image name mrom The brom command installs the ROM image on the Flash device or replaces an already existing one Example flint d dev mst mt4099 pci cr0 brom example mrom Current ROM info on flash N A New ROM info type PXE version 3 4 255 devid 4099 proto VPI Burning ROM image OK Restoring signature OK e To read an Expansion ROM image to a file use the following command flint d lt mst device gt rrom
118. wing command line syntax mget_temp OPTIONS 1 This step in not required in Windows Mellanox Technologies 77 Rev 2 4 Debug Utilities where h Print this message d dev mst device name version Display version info Example on how to read a device temperature mget temp d dev mst SW MT51000 0002c903007e76a0 lid 0x0002 4 7 mlxtrace Utility The mlxtrace utility is used to configure and extract HW events generated by different units in Mellanox devices The utility generates a dump trc file which contains HW events that assist us with debug troubleshooting and performance analysis Events can be stored in host memory if driver is up or in a small on chip buffer always available depending on the utility running mode In order to run the utility it s required to have a configuration file first this file should be provided by Mellanox representative A dump file mlxtrace trc will be generated by end of run file name can be controlled by o flag this file should be sent to Mellanox representative for further diagnostics troubleshooting Memory mode on Connect IB device is supported only by PCI mst devices de 4 7 4 mlxtrace Usage The mst driver must be started prior to running the mlxtrace tool For MEM buffer mode driver must be up also Enter the following command e mlxtrace options h help Print help and exit v version Print version default off p parse Move t
119. wmanager windows64 ConnectX3 X3Pro 20140811 exe Release Note Fixed Minor Issues Perform Download y N y Please wait while downloading Files to tmp DownloadDir 0 mixfwmanager linux ConnectX3 X3Pro 20140811 Done 1 mlxfwmanager windows ConnectX3_ X3Pro 20140811 exe Done 2 mlxfwmanager linux64 ConnectX3 X3Pro 20140811 Done 3 mlxfwmanager windows64 ConnectX3 X3Pro 20140811 exe Done Downloading file s to tmp DownloadDir is done successfully Mellanox Technologies 29 Rev 2 4 Firmware Generation Configuration and Update Tools b Downloading firmware images firmware update packages using custom key mlxfwmanager download tmp DownloadDir download device ConnectX download os All download type All key last_release lt Files gt 0 fw ConnectX3 X3Pro 20140610 mfa 1 mixfwmanager linux ConnectX3 X3Pro 20140610 2 mlxfwmanager windows ConnectX3 X3Pro 20140610 exe 3 mlxfwmanager linux64 ConnectX3 X3Pro 20140610 4 mixfwmanager windows64 ConnectX3 X3Pro 20140610 exe Release Note Fixed Minor Issues Perform Download y N y Please wait while downloading Files to tmp DownloadDir 0 fw ConnectX3 X3Pro 20140610 mfa Done mlxfwmanager linux ConnectX3 X3Pro 20140610 Done mlxfwmanager windows ConnectX3_ X3Pro 20140610 exe Done mlixfwmanager linux64 ConnectX3 X3Pro 20140610 Done mlxfwmanager windows64 ConnectX3 X3Pro 20140610 exe Done 4H 00 PNO oH Down
120. x Technologies Rev 2 4 4 3 mstdump Utility The mstdump utility dumps device internal configuration registers The dump file is used by Mellanox Support for hardware troubleshooting purposes It can be applied on all Mellanox adapter devices BridgeX device and 4th generation switch devices 4 3 1 mstdump Usage The mst driver must be started prior to running mstdump tool To start mstdump 1 Start the mst driver mst start or mst restart 2 Enter an mstdump command that complies with the following command syntax mstdump full mst device gt dump file gt where the u11 flag dumps all internal registers On BridgeX devices using the u11 flag may have undesired side effects and requires resetting the device Example root mymach mstdump dev mst mt4099 pci cr0 gt mt4099 dmp This dumps the internal configuration data of the device into the file mt25408 dmp 4 4 mlxi2c Utility The mlxi2c utility provides a way to route the I2c bus to Mellanox 4th generation switches 4 4 1 mlxi2c Usage The mst driver must be started prior to running mlxi2c To start mlxi2c 1 Start the mst driver mst start or mst restart 2 Run mlxi2c with the following command line syntax mlxi2c switches command parameters 1 This step in not required in Windows Mellanox Technologies 75 Rev 2 4 Debug Utilities Switches Options d lt device gt mst i2c device name de
121. y Address size Size Memory size in bytes v version Show tool version and exit h help Show usage clear semaphore Force clear semaphore gw access Force get QPC by GW access Connect X Family Examples Print all valid qpns The example below will dump all valid qpns of type mcg context wgdump d dev mst mt4099 pci cr0 source mcg dump ALL VALID QPNS e Dump mcg qp 84 Mellanox Technologies Rev 2 4 The example below will dump mcg context number 0x10 wqdump d dev mst mt4099 pci cr0 source mcg dump QP qp 0x10 Dump other qpns The example below will dump snd context number 0x10 in a raw format wqdump d dev mst mt4099 pci cr0 source snd dump QP qp 0x10 format raw Dump wq The example below will dump send work queue buffer number 0x42 wgdump d dev mst mt4099 pci cr0 source snd dump wq qp 0x42 Dump mcg qp by GW access The example below will dump mcg context number 0x10 by GW access wqdump d dev mst mt4103 pci cr0 source mcg dump QP qp 0x10 gw access Mellanox Technologies 85 Rev 2 4 Debug Utilities 4 13 4 13 1 mlxmdio Utility The mlxmdio tool is used to read write MDIO registers Clause 45 on Boards with externally managed PHY mlxmdio Usage To run mlxmdio use the following line mlxmdio lt d mst dev lt m phy addr dev addr a addr data g mdio gw where d device mst device m mdio id The mdio id of the target
122. y current loaded firmware version e When a FW file is given fw flag Display the file FW version Note The wver flag is not supported in Connect IB devices h Display a short help text image fw image file Do not generate image Use the given fw image instead img dir image directory Do not generate image Select the image to burn from the bin in the given directory nofs When specified burn process will not be failsafe nofs img When specified generated image will not be failsafe and burn process will not be failsafe Mellanox Technologies 57 Rev 2 4 Firmware Generation Configuration and Update Tools query Query the HCA Switch device for firmware details e g Firm ware Version GUIDs etc In addition to the above flags Mlxburn can also accept the fol lowing flags options which are passed to the underlying burning tool banks use_image_ps skip_is mac s guid s sysguid vsd ndesc bsn pe i2c se i2c is3 i2c no uid s log blank guids flash_params allow psid change no flash verify use image rom override cache replacement use image guids See the flint tool documentation for HCA 4th gen switches Bridge burning options v Print version info and exit V INFORM WARNING DEBUG gt Set verbosity level Default is WARNING vpd Display the read only section of the PCI VPD Vital Product Data of the given device Vpd prog rw rw keywords on Linux only

Mellanox Firmware Tools (MFT) User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents