Mellanox OFED for Linux Release Notes
Contents
1. Table 7 Known Issues Continued Index Issue Description Workaround 73 74 75 Resources Limitation The device capabilities reported may not be reached as it depends on the system on which the device is installed and whether the resource is allocated in the kernel or the userspace Occasionally a user process might experience some memory shortage and not function properly due to Linux kernel occupation of the system s free memory for its internal cache To free memory to allow it to be allocated in a user pro cess run the drop_caches procedure below Performing the following steps will cause the kernel to flush and free pages den tries and inodes caches from memory causing that mem ory to become free Note As this is a non destructive operation and dirty objects are not free able run sync first To free the pagecache echo 1 proc sys vm drop caches To free dentries and inodes echo 2 proc sys vm drop caches To free pagecache dentries and inodes echo 3 proc sys vm drop caches Setting more IP addresses than the available GID entries in the table results in failure and the update gid table error message is displayed GID table of port 1 is full Can t add address message 20 Mellanox Technologies Rev 2 1 1 0 0 Table 7 Known Issues Continued Index Issue Description Workaround 76 Ethernet P2. 3 Pro Rev 2 30 8000 and above e ConnectX 3 Rev 2 30 8000 and above e ConnectX 2 Rev 2 9 1000 and above 1 ConnectX 2 does not support all the new functionality of MLNX OFED 2 0 3 XXX For the complete list of the supported features per HCA please refer to the MLNX OFED User Manual 8 Mellanox Technologies Rev 2 1 1 0 0 For official firmware versions please see http www mellanox com content pages php pg firmware download 7 Compatibility MLNX OFED Rev 2 1 1 0 0 is compatible with the following Table 4 MLNX OFED Rev 2 1 1 0 0 Compatibility Matrix Mellanox Product Description Version SwitchX amp InfiniBand MSX6036 MSX6035 MSX6536 w w MLNX OS version 3 3 3000 Ethernet MSX1036 MSX1016 MSX1024 w w MLNX OS version 3 3 3000 FabricIT EFM Tested IPoIB Verbs and OpenSM priority handover e SLES 11 x64 w w ConnectX VPI PCIe 2 0 5GT s IB QSFP QDR 10GigE ConnectX VPI 10GigE IB QDR IS5030 w w FabricIT EFM version 1 1 2700 FabricIT BXM MBX5020 w w FabricIT BXM version 2 1 2000 Unified Fabric Manager UFM v4 6 MXM v2 1 ScalableUPC v2 2 ScalableSHMEM v2 2 FCA v2 5 OMPI v1 6 4 MVAPICH v1 9a CD v1 0 Mellanox Technologies 9 J Rev 2 1 1 0 0 Change Log History 8 Change Log History 8 1 Changes in Rev 2 1 1 0 0 From Rev 2 0 3 0 0 Table 5 New Features Changes and Fixes in v2 1 1 0 0 Cat
3. Continued Index Issue Description Workaround 10 The physical port MTU indicates the port capa In order to change the IPoIB bility default value was changed to 4k whereas MTU to 4k edit the the IPoIB port MTU logical MTU default OpenSM partition file in the value is 2k as it is set by the OpenSM section of IPoIB setting as follow Default Oxffff ipoib mtu 5 ALL full Where mtu 5 indicates that all IPoIB ports in the fabric are using 4k MTU mtu 4 indicates 2k MTU 11 Occasionally when using IPoIB in Connected Use the Datagram mode mode the connection might get closed and recov ered only after several minutes 12 Fallback to the primary slave of an IPoIB bond does not work with ARP monitoring https bugs openfabrics org show_bug cgi id 1990 13 Whenever the IOMMU parameter is enabled in To avoid such issue the kernel it can decrease the number of child Decrease the amount of the interfaces on the device according to resource RX receive buffers module limitation parameter the default is The driver will stuck after unknown amount of 512 Decrease the number of RX rings sys fs or ethtool in new kernels Avoid using IOMMU if not required child interfaces creation 14 System might crash in skb_checksum_help Use UD mode in ipoib while performing TCP retransmit involving pack ets with 64k packet size A similar out to the below will be printed k
4. IBV M WOE CAP RO ENABLE From IBV M WQE CQE WAIT CAP to IBV M WOE CAP COE WAIT e From IBV M WQE CALC CAP to IBV M WQE CAP CALC SEND The enumerations IBV M WOE CAP CALC RDMA WRITE WITH IMM was added Mellanox Technologies 25 J Rev 2 1 1 0 0 Bug Fixes History 11 Bug Fixes History Table 12 lists the bugs fixed in this release Table 12 Fixed Bugs List I Descrintion Discovered Fixed in RRS SH in Release Release l mlx4_core Restored port types as they were when recovering 2 0 2 0 5 2 1 1 0 0 from an internal error 2 Added an N A port type to support port type array 2 0 2 0 5 2 1 1 0 0 module param in an HCA with a single port 3 SR IOV Fixed memory leak in SR IOV flow 2 0 2 0 5 2 0 3 0 0 4 Fixed communication channel being stuck 2 0 2 0 5 2 0 3 0 0 5 mlx4 en Fixed ALB bonding mode failure when enslaving 2 0 3 0 0 2 1 1 0 0 Mellanox interfaces 6 Fixed leak of mapped memory 2 0 3 0 0 2 1 1 0 0 T Fixed TX timeout in Ethernet driver 2 0 2 0 5 2 0 3 0 0 8 Fixed ethtool stats report for Virtual Functions 2 0 2 0 5 2 0 3 0 0 9 Fixedan issue of VLAN traffic over Virtual Machine 2 0 2 0 5 2 0 3 0 0 in paravirtualized mode 10 Fixed ethtool operation crash while interface down 2 0 2 0 5 2 0 3 0 0 11 IPoIB Fixed memory leak in Connected mode 2 0 2 0 5 2 0 3 0 0 12 Fixed an issue causing IPoIB to avoid pke
5. and Software Requirements essosssessoseseosessose 8 Chapter 6 Supported HCAS 6 065555 6c ok SAWS OGRE oa ee BAERS EW RS 8 Chapter 7 Compatibility ee ca 36a wos tases oho So Ws Apo mg tw Coder RC Wee 9 Chapter 8 Change Log History cc cc cece cece cece rnnt 10 8 1 Changes in Rev 2 1 1 0 0 From Rev 2 0 3 0 0 0 0 0 cee ee eee ee eee 10 8 2 Changes in Rev 2 0 3 0 0 From Rev 2 0 2 0 5 0 0 0 cece eee 10 8 3 New Features in Rev 2 0 2 0 5 0 0 0 ccc ccc eee nee 11 Chapter 9 Known ISSUes ove bere eR E a RE OREN MR RE RU REG OM ORE ORE 12 Chapter 10 API Changes 5 ze e si sch ca eR OAGE ona wool NA Rho e CRACK RC RUR RERC oie wicca ac 24 10 1 API Changes in MLNX OFED Rev 2 1 1 0 0 n annaua aaaea 24 10 1 1 Verbs Extension and Verbs Experimental APIs 0 0 0 eee 24 10 2 API Changes in MLNX OFED Rev 2 0 3 0 0 0 0 0 cece cece 25 10 3 API Changes in MLNX OFED Rev 2 0 2 0 5 2 0 0 0 cece eee eee 25 Chapter 11 Bug Fixes History 0 ccc ec ccc ccc cece cece cece cere cence 27 Mellanox Technologies 3 J Rev 2 1 1 0 0 List Of Tables Table 1 Mellanox OFED for Linux Software Components 000 c cece eee eee 6 Table 2 Supported Platforms and Operating Systems 0 0 0 c cece 7 Table 3 Additional Software Packages 0 0 ccc cent enna 8 Table 4 MLNX OFED Rev 2 1 1 0 0 Compatibility Matrix cece eee 9 Table 5 New Features Changes and Fixes in v2 1
6. be achieved while connected to Mellanox SX10XX switch series Mellanox Technologies 5 J Rev 2 1 1 0 0 Content of Mellanox OFED for Linux 3 Content of Mellanox OFED for Linux Mellanox OFED for Linux software contains the following components Table 1 Mellanox OFED for Linux Software Components Components Description OpenFabrics core and ULPs IB HCA drivers mlx4 mlx5 core Upper Layer Protocols IPoIB SRP and ISER Initiator OpenFabrics utilities OpenSM IB Subnet Manager with Mellanox proprietary Adaptive Routing Diagnostic tools e Performance tests MPI e OSU MPI mvapich2 1 9 1 stack supporting the InfiniBand interface e Open MPI stack 1 6 5 and later supporting the InfiniBand interface MPI benchmark tests OSU benchmarks Intel MPI benchmarks Presta PGAS ScalableSHMEM v2 2 supporting InfiniBand MXM and FCA ScalableUPC v2 2 supporting InfiniBand MXM and FCA HPC Acceleration packages Mellanox MXM v2 1 p2p transport library acceleration over Infiniband e Mellanox FCA v2 5 MPI PGAS collective operations acceleration library over InfiniBand KNEM Linux kernel module enabling high performance intra node MPI PGAS communication for large messages Extra packages e ibutils2 ibdump MFT Sources of all software modules under conditions mentioned in the modules LICENSE files except for MFT OpenSM plugins ibutils2 and ibdump Documentation 6
7. cq e ibv create qp ex e ibv modify qp 40 Using libnll 1 3 26 or earlier requires Ibv create ah protection by a lock for multi threaded applications 41 Driver Start When reloading the driver using the etc init d openibd restart command on XenServer6 1 loading of mlx4 en driver might fail with Unresolved Symbols errors This message can safely be ignored 42 Out of memory issues may rise during drivers load depending on the values ofthe driver module parameters set e g log num cq 16 Mellanox Technologies Rev 2 1 1 0 0 Table 7 Known Issues Continued Index Issue Description Workaround 43 When reloading starting the driver using the Remove the third party etc init d openibd the following messages RPM non MLNX OFED are displayed if there is a third party RPM or drivers directory run driver installed depmod and then rerun Module mlx4 core does not belong to etc init d openibd MLNX OFED restart or Module mlx4 core belong to rpm name gt which is not a part of MLNX OFED 44 Occasionally when trying to repetitively reload the nes hardware driver on SLES11 SP2 a soft lockups occurs that required reboot 45 In ConnectX 2 if the driver load succeeds the informative message below is presented convey ing the below limitations If port type is IB the number of maximum sup ported VLs is 4 If port type is ETH then the maximum pri
8. module The dev assign str module parameter is not backward compatible In the current version this parameter is using dec imal number to describe the InfiniBand device and not hexadecimal number as it was in previous versions in order to uniform the mapping of device function numbers to InfiniBand device numbers as defined for other module parameters e g num v s and probe vf Mellanox Technologies 15 J Rev 2 1 1 0 0 Known Issues Table 7 Known Issues Continued Index Issue Description Workaround 37 ABI Compatibility MLNX OFED Rev 2 1 1 0 0 is not ABI compati Recompile the application ble with previous MLNX OFED OFED versions over the new MLNX OFED version 38 System Time Loading the driver using the openibd script when no InfiniBand vendor module is selected for example mlx4 ib may cause the execution of the sbin start udev script In RedHat 6 x and OEL6 x this may change the local system time 39 Verbs Verbs for the following features are subject to change e Core Direct Shared memory region Contiguous pages Flow steering Verbs subject to changes are iby post task e ibv query values ex e iby query device ex e iby poll cq ex e ibv reg shared mr ex iby reg shared mr e ibv modify cq ibv create cq ex e ibv modify qp ex ibv reg mr ibv post send e jibv dealloc mw e ibv alloc mw ibv bind mw iby query device e ibv poll
9. over IPoIB When working with ISCSI over IPoIB LRO must be disabled even if IPoIB is set to connected mode due to a a bug in older kernels which causes a kernel panic 68 MLNX OFED sources MLNX OFED includes the OFED source RPM packages used as a build platform for kernel code but does not include the sources of Mellanox pro prietary packages 69 InfiniBand Utilities When running the ibdiagnet check nodes info on the fabric a warning specifying that the card does not support general info capabilities for all the HCAs in the fabric will be displayed Run ibdiagnet skip nodes info 70 mlx5 Driver Atomic Operations over Connect IB M are not supported 71 General On ConnectX 2 ConnectX 3 Ethernet adapter cards there is a mismatch between the GUID value returned by firmware management tools and that returned by fabric driver utilities that read the GUID via device firmware e g using Ibstat MIxburn flint return Oxffff as GUID while the utilities return a value derived from the MAC address For all driver firmware software pur poses the latter value should be used N A Please use the GUID value returned by the fabric driver utilities not Oxfffff 72 Uplinks On rare occasions ConnectX 3 Pro adapter card may fail to link up when performing parallel detect to 40GbE Restart the driver Mellanox Technologies 19 J Rev 2 1 1 0 0 Known Issues
10. user increases receive send a buffer it might consume all the memory when few child s interfaces are created 2 The hardware address suffix of IPoIB interfaces in MLNX OFED v2 0 3 0 0 is a instead of 8 to indicate the TSS support 3 The size of send queue in Connect IBTM cards cannot exceed 1K 4 In 32 bit devices the maximum number of child interfaces that can be created is 16 Creating more that might cause out of memory issues 5 The default IPoIB operating mode in ConnectX family adapter cards is UD and CM in Connect IBTM 6 Changing the IPoIB mode CM vs UD requires the interface to be in down state 7 IPoIB interface does not function properly if a third party application changes the PKey table We recommend modifying PKey tables via OpenSM 8 When creating a new child interface in an over loaded kernel a dmesg print is displayed advising the user to try again in a few seconds 9 Out of memory issue might occur due to overload To calculate the allowed of interfaces created memory per each IPoIB interface check the follow ing e Num rings min num cores on that device 16 Ring size 512 by default it is module parameter UD memory 2 num rings ring size 8K e CM memory ring size 64k Total memory UD mem CM mem 12 Mellanox Technologies Rev 2 1 1 0 0 Table 7 Known Issues
11. 1 0 0 0 0 00 cece cece eee eee 10 Table 6 New Features Changes and Fixes in v2 0 3 0 0 0 00 cc cece cece eee eens 10 Table 7 7 Knowinlssues odes fauna Pad ERI S DE Ed edo E ene phe refe ded sce aaa 12 Table 8 API Changes in MLNX OFED Rev 2 0 3 0 0 sssssseesseee eh 24 Table 9 Verbs Extension and Verbs Experimental APIS 0 0 0 c cece cece eee 24 Table 10 API Changes in MLNX OFED Rev 2 0 3 0 0 l 0 20 0 c cece eee eee 25 Table 11 API Changes in MLNX_OFED Rev 2 0 2 0 5 2 00 00 eee eee 25 Table 2 Fixed Bugs List l i oru wi RAS be ae oe ele ee ue ee Saale 27 4 Mellanox Technologies J Rev 2 1 1 0 0 1 Overview These are the release notes of Mellanox OFED for Linux Driver Rev 2 1 1 0 0 Mellanox OFED is a single Virtual Protocol Interconnect VPI software stack and operates across all Mellanox network adapter solutions supporting the following uplinks to servers 10 20 40 and 56 Gb s InfiniBand IB 10 40 and 56 Gb s Ethernet 2 5 or 5 0 GT s PCI Express 2 0 e 8 GT s PCI Express 3 0 2 Main Features in This Release MLNX_OFED Rev 2 1 1 0 0 provides the following new features Signature Verbs T10 PI at beta level e RoCE Time Stamping e PeerDirect nline Receive Ethernet Performance Counters Memory Window VMA bundled with MLNX OFED DCT support at beta level eIPoIB multicast support 1 56 GbE is a Mellanox propriety link speed and can
12. DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 4972 0 4 959 3245 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB amp CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDXTM Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Rev 2 1 1 0 0 Table of Contents Table of Contents 2 0534 she uuo A seks exec seeks eins ou cies oe eb eeduses ee wes 3 Last Of Tables 52e qd HR ehm CK Kx one t Se aire KL dine ea X Rod eR Ra onim OS 4 Chapter 1 Overview ocu ose Oe PNE Re oa eu ia ox PU acu d prat rd oes qoia aos 5 Chapter 2 Main Features in This Release 0 cece cece eee cece eee eees 5 Chapter 3 Content of Mellanox OFED for Linux eee 6 Chapter 4 Supported Platforms and Operating Systems 0 eceeeees 7 Chapter 5 Hardware
13. Mellanox TECHNOLOGIES Mellanox OFED for Linux Release Notes Rev 2 1 1 0 0 Last Modified 18 February 2014 www mellanox com Rev 2 1 1 0 0 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
14. Mellanox Technologies J Rev 2 1 1 0 0 4 Supported Platforms and Operating Systems The following are the supported OSs in MLNX_OFED Rev 2 1 1 0 0 Table 2 Supported Platforms and Operating Systems Operating System Platform RHEL CentOS 6 3 x86_64 PPC64 RHEL CentOS 6 4 x86_64 PPC64 RHEL CentOS 6 5 x86 64 SLES11 SPI x86 64 SLES11 SP2 x86 64 PPC64 SLES11 SP3 x86 64 PPC64 OEL 6 1 x86 64 OEL 6 2 x86 64 OEL 6 3 x86 64 OEL 6 4 x86 64 Citrix XenServer Host 6 x 1686 Fedora 18 x86 64 Fedora 19 x86 64 Ubuntu 12 04 x86 64 Ubuntu 13 04 x86 64 Ubuntu 13 10 x86 64 Debian 6 0 7 x86 64 Debian 6 0 8 x86 64 Debian 7 1 x86 64 Debian 7 2 x86 64 kernel 3 10 kernel 3 11 kernel 3 12 If you wish to install OFED on a different kernel you need to create a new ISO image using mlnx add kernel support sh script ae See the MLNX OFED User Guide for instructions 7 Upgrading MLNX_OFED on your cluster requires upgrading all of its nodes to the newest version as well Mellanox Technologies 7 J Rev 2 1 1 0 0 Hardware and Software Requirements 5 Hardware and Software Requirements The following are the hardware and software requirements of MLNX OFED Rev 2 1 1 0 0 Linux operating system Administrator privileges on your machine s Disk Space IGB For the OFED Distribution to compile on your machine some software packages of your operat i
15. Rev 2 0 3 0 0 Table 10 API Changes in MLNX OFED Rev 2 0 3 0 0 Name Description XRC The following verbs have become deprecated struct ibv xrc domain ibv open xrc domain Struct ibv srq ibv create xrc srq int ibv close xrc domain int ibv create xrc rcv qp int ibv modify xrc rcv qp int ibv query xrc rcv qp int ibv reg xrc rcv gp int ibv unreg xrc rcv qp 10 3 API Changes in MLNX OFED Rev 2 0 2 0 5 The following are the API changes in MLNX OFED v2 0 2 0 5 Table 11 API Changes in MLNX OFED Rev 2 0 2 0 5 Name Description Libibverbs Extended speeds Missingtheext active speed attribute from the struct ibv port attr e Removed function ibv ext rate to int e Added functions ibv rate to mbpsandmbps to ibv rate Raw QPs QP types IBV_QPT RAW PACKET and IBV QPT RAW ETH are not sup ported Contiguous pages Added Contiguous pages support e Addedfunctionibv reg shared mr Libmverbs 24 Mellanox Technologies Rev 2 1 1 0 0 Table 11 API Changes in MLNX_OFED Rev 2 0 2 0 5 Name Description The enumeration IBV M WR CALC was renamed to IBV M WR CALC SEND The enumeration IBV M WR WRITE WITH IMM was added Inthe structure ibv m send wr the union wr send was renamed to wr calc sendand wr rdma was added The following enumerations were renamed From IBV M WQE SQ ENABLE CAP to IBV M WOE CAP SQ ENABLE From IBV M WQE RQ ENABLE CAP to
16. dct e int ibv exp query dct struct ibv dct dct struct ibv exp dct attr attr 10 1 1 Verbs Extension and Verbs Experimental APIs e Verbs Extension API Verbs extension API defines OFA APIs extension scheme to detect ABI compatibility and enable backward and forward compatibility support Verbs Experimental API Verbs experimental API defines MLNX OFED APIs extension scheme which is similar to the Verbs extension APT This extension provides a way to introduce new features before they are integrated into the formal OFA API and to the upstream kernel and libs The following are the Verbs Extension and Verbs Experimental APIs in MLNX_OFED Rev 2 1 1 0 0 Table 9 Verbs Extension and Verbs Experimental APIs API Type APIs Verbs Extension API ibv_post_task ibv_query_values_ex ibv_query_device_ex ibv_create_flow ibv_destroy_flow ibv poll cq ex ibv reg shared mr ex ibv open xrcd ibv close xrcd ibv modify cq ibv create srq ex ibv get srg num ibv create qp ex ibv create cq ex ibv open qp ibv modify qp ex Mellanox Technologies 23 J Rev 2 1 1 0 0 API Changes Table 9 Verbs Extension and Verbs Experimental APIs API Type APIs Verbs Experimental API e ibv_exp_create_qp e jbv exp query device ibv exp create dct e ibv exp destroy dct e ibv exp query dct 10 2 API Changes in MLNX OFED Rev 2 0 3 0 0 The following are the API changes in MLNX OFED
17. efault vmalloc setting is 128M It is recommended to add each time 64M of memory until desired ring size can be allocated Please note that in case vmalloc size is too big the OS will fail to boot so please use caution when adding additional memory For more info refer to http www mythtv org wiki Common Problem vmalloc too small On OEL6 1 with uek1 2 6 32 x x x el6uek ker nel when the number of RX ring is smaller than TX rings kernel issue the following call trace will be shown in the kernel log WARNING at net core dev c 2077 get rps cpu 0x70 0x2b9 Kernel panic might occur during traffic over IPv6 on kernels between 3 12 rc7 and 3 13 rc1 kernel issue Kernel panic might occur during fio splice in ker nels before 2 6 34 rc4 Use kernel v2 6 34 rc4 which provides the follow ing solution baff42a net Fix oops from tcp collapse when using splice On Debian 6 0 7 kernel panic may occur when changing the number of TX channels above the default value 8 On kernels that do not support multiqueues the number of TX channels represents the number of TX rings The maximal number of TX channels is 16 14 Mellanox Technologies Rev 2 1 1 0 0 Table 7 Known Issues Continued Index Issue Description Workaround 24 Transmit timeout might occur on RH6 3 as a result of lost interrupt OS issue In this case the following message will be shown in dm
18. egory Description EoIB EoIB is supported only in SLESIISP2 and RHEL6 4 Connect IBTM Added the ability to resize CQs IPoIB Reusing DMA mapped SKB buffers Performance improvements when IOMMU is enabled minx en Added reporting autonegotiation support Added Transmit Packet Steering XPS support Added reporting 56Gbit s link speed support Added Receive Flow Steering RFS support in UDP Added Low Latency Socket LLS support Added check for dma mapping errors eIPoIB Added non virtual environment support Hypervisor support KVM and XenServer 8 2 Changes in Rev 2 0 3 0 0 From Rev 2 0 2 0 5 Table 6 New Features Changes and Fixes in v2 0 3 0 0 Sheet 1 of 2 Category Description Operating Systems Additional OS support e SLES11SP3 Fedoral6 Fedoral7 Drivers Added Connect IB support Installation Added ability to install MLNX OFED with SR IOV support Added Yum installation support EoIB EoIB at beta level is supported only in SLESIISP2 and RHEL6 4 mlx4 core Modified module parameters to associate configuration values with specific PCI devices identified by their bus device function value format mlx4 en Reusing DMA mapped buffers major performance improvements when IOMMU is enabled Added Port level QoS support 10 Mellanox Technologies Rev 2 1 1 0 0 Table 6 New Features Changes and Fixes in v2 0 3 0 0 Sheet 2 of 2 Categ
19. erfor In a system with more than 61 VFs the 62nd VF mance Counters and onwards is assigned with the SINKQP coun ter and as a result will have no statistics and loopback prevention functionality for SINK coun ter 77 Since each VF tries to allocate 2 more QP counter for its RoCE traffic statistics in a system with less than 61 VFs if there is free resources it receives new counter otherwise receives the default counter which is shared with Ethernet In this case RoCE statistics is not available 78 In ConnectX 3 when we enable function based loopback prevention for Ethernet port by default i e based on the QP counter index the dropped self loopback packets increase the IfRxError Frames Octets counters 79 RoCE Not configuring the Ethernet devices or indepen Restart the driver dent VMs with a unique IP address in the physical port may result in RoCE GID table corruption 80 If RDMA CM is not used for connection man agement then the source and destination GIDs used to modify a QP or create AH should be of the same type IPv4 or IPv6 81 Since the number of GIDs per port is limited to 128 there cannot be more than the allowed IP addresses configured to Ethernet devices that are associated with the port Allowed number is e 127 for a single function machine e 15 for a hypervisor in a multifunction machine n for a guest in a multifunction machine where n is
20. ernel BUG at net core dev c 1707 invalid opcode 0000 1 SMP RIP 0010 ffffffff81448988 5 skb_checksum_help 0x148 0x160 Call Trace lt IRQ gt ffffff814488383 5 dev hard start xmit4 0x3e3 0x530 ffffff8144Cc8055 dev queue xmit 0x205 0x550 ffffff8145247d neigh connected_output 0xbd 0x1 15 Changing the GUID of a specific SR IOV guest after the driver has been started causes the ping to fail Hence no traffic can go over that Infini Band interface Mellanox Technologies 13 J Rev 2 1 1 0 0 Known Issues Table 7 Known Issues Continued Index Issue Description Workaround 16 send queue size over Connect IB adapter cards cannot be larger than 1024 17 18 19 20 21 22 23 Ethernet Ethernet PV VLAN Guest transparent Tagging VGT is only supported in openvswitch and not in standard Linux vBridges and libvirt For more information please see http lib virt org formatnetwork html Setting VLAN tag section Changing the ring size on 32 bit system may result in failure due to lack of memory Therefore mlx4 en will not be able to vmap enough mem ory and the below message will be printed in dmesg vmap allocation for size 528384 failed use vmalloc size to increase size In this case user can enlarge the vmalloc memory by adding vmal loc lt size gt to grub conf D
21. esg do_IRQ 0 203 No irq handler for vec tor irg 1 25 26 27 28 29 30 eIPoIB On rare occasions upon driver restart the follow ing message is shown in the dmesg cannot create duplicate filename class net eth ipoib interfaces No indication is received when eIPoIB is non functional Run ps ef grep ipoibd to verify its func tionality eIPoIB requires libvirtd python eIPoIB supports only active backup mode for bonding eIPoIB supports only VLAN Switch Tagging VST mode on guests IPv6 is currently not supported in eIPoIB 3 32 33 34 35 XRC Legacy API is deprecated thus when recompiling applications over MLNX OFED v2 0 3 x x warnings such as the below are displayed rdma c 1699 warning ibv open xrc domain is deprecated declared at usr include infiniband ofa verbs h 72 rdma c 1706 warning ibv create xrc srq is deprecated declared at usr include infiniband ofa verbs h 89 These warnings can be safely ignored XRC is not functional in heterogeneous clusters containing non Mellanox HCAs XRC options do not work when using qperf tool Use perftest instead XRC over ROCE in SR IOV mode is not func tional Out of memory issue might occur due to overload of XRC receive QP with non zero receive queue size created XRC QPs do not have receive queues 36 mlx4 ib
22. ior to changing the ment port protocol from InfiniBand to Ethernet 53 After changing port type using Use udev rules for persis connectx port config interface ports names tent naming configuration can be changed For example ib1 gt ibO if port For further information changed to be Ethernet port and port2 left IB please refer to the User Manual 54 Flow Steering Flow Steering is disabled by default To enable it set the parame ter below as follow log num mgm entry siz e should set to 1 55 IPv4 rule with source IP cannot be created in SLES 11 56 RFS is not supported in SLES11 57 Quality of Service QoS is not supported in XenServer Debian 6 0 and in OEL6 1 and 6 2 with uek kernel 58 Driver Uninstall A Kernel panic occurs if you uninstall the driver Remove the midule after without deleting the SR IOV module params uninstalling and prior to mlx4 core s num vfs inthe file etc mod restarting the driver probe d mlx4 core conf On the next boot you will get the panic and machine will boot up 59 Installation When upgrading from an earlier Mellanox OFED Stop the old OFED stack version the installation script does not stop the etc init d openibd earlier version prior to uninstalling it stop before upgrading to this new version 60 Upgrading from the previous OFED installation Reboot after installing the to this release does not unload the kernel module driver Ipoib helper 61 Ins
23. ng system OS distribution are required To install the additional packages run the following commands per OS Table 3 Additional Software Packages Operating System Required Packages Installation Command RHEL OEL Fedora yum install pciutils python gcc gfortran libxml2 python tesh libnl 1686 libnl libnl devel expat glib2 tcl libstdc bc tk XenServer yum install pciutils python libxml2 python libnl expat glib2 tcl bc libstdc tk OpenSUSE zypper install glib2 tools pciutils python libxml2 python tesh libnl 1 1 32bit libstdc 46 expat libnl 1 1 devellibnl 1 1 tcl be tk SLES 11 SP1 zypper install pciutils python libxml2 python tesh libnl libstde 43 libnl devel expat glib2 tcl bc libnl 1586 tk SLES 11 SP2 zypper install pciutils python libnl 32bit libxml2 python tesh libnl libnl devel libstdc 46 expat glib2 tcl bc tk SLES 11 SP3 zypper install pciutils python libnl 32bit libxml2 python tesh libstde 43 libn libnl devel expat glib2 tcl bc tk Ubuntu Debian apt get install dpkg autotools dev autoconf libtool automake1 10 automake m4 dkms debhelper tcl tcl8 4 chrpath swig graphviz tcl dev tcl8 4 dev tk dev tk8 4 dev bison flex dpatch zlib1 g dev curl libcurl4 gnutls dev python libxml2 libvirt bin libvirtO libnl dev libglib2 0 dev libgfortran3 6 Supported HCAs MLNX OFED Rev 2 1 1 0 0 supports the following Mellanox network adapter cards e Connect IB Rev 10 10 2000 and above ConnectX
24. onnec tion establishment 92 Ungraceful power cycle of an initiator connected with Targets DDN Nimbus NetApp may result in temporary stale connection messages when initiator reconnects 93 ISER On SLES11 the ib iser module does not load Add a dummy interface using iscsiadm iscsiadm m iface I ib iser o new iscsiadm m iface I ib iser o update n iface transport na me v ib iser In SLES10 SP3 and Ubuntu12 04 need to update user space open iscsi package to version 2 0 873 Trying to disconnect a session while the session is undergoing a reconnect flow may result in discon nection hang Restart iscsid Note Please be aware that doing so might cause rmmod process to hang as the ib_iser module will not be unloaded Unloading ib iser during session disconnect event may result in kernel panic ISER interop Ora cle Sun ZFS storage 7420 Connection establishment occurs twice which may cause ISER to log a stack trace 22 Mellanox Technologies Rev 2 1 1 0 0 10 API Changes 10 1 API Changes in MLNX OFED Rev 2 1 1 0 0 The following are the API changes in MLNX OFED Rev 2 1 1 0 0 Table 8 API Changes in MLNX OFED Rev 2 0 3 0 0 Name Description Dynamically Connected DC The following verbs were added struct ibv dct ibv exp create dct struct ibv context context struct ibv exp dct init attr attr e int ibv exp destroy dct struct ibv dct
25. ority for VLAN tagged is 3 mlx4 core 0000 0d 00 0 command SET PORT 0xc failed in param 0x120064000 in mod 0x2 op mod 0x0 fw status 0x40 46 Operating Systems RHEL 5 X and SLES 10 SPX are currently not supported 47 SR IOV When using legacy VMs with OFED 2 0 2 0 5 hypervisor the enable 64b cqe eqe parameter must be set to zero on the hypervisor It should be set in the same way that other module parameters are set for mlx4 core at module load time For example add options mlx4 core enable 64b cqe eqe 0 asa line in the file etc modprobe d mlx4 core conf 48 Enabling SR IOV requires appending the intel iommu on option to the relevant OS in file boot grub grub conf Without that SR IOV cannot be loaded 49 rdma cm does not support UD QPs 50 SR IOV can be enabled only when using the firmware version embedded in the MLNX OFED v2 0 3 0 0 driver 51 When SR IOV is disabled in the system BIOS a Enable Sr IOV in the BIOS PCI issue is noticed in Ubuntu v12 04 3 with Linux kernel v3 8 which affects NICs of several manufacturers including Mellanox s preventing them from operating Mellanox Technologies 17 J Rev 2 1 1 0 0 Known Issues Table 7 Known Issues Continued Index Issue Description Workaround 52 Port Type Manage OpenSM must be stopped pr
26. ory Description IPoIB Reduced memory consumption Limited the number TX and RX queues to 16 Default IPoIB mode is set to work in Datagram except for Connect IB adapter card which uses IPoIB with Connected mode as default Storage iSER at GA level 8 3 New Features in Rev 2 0 2 0 5 SR IOV for both Ethernet and InfiniBand at Beta level e RoCE over SR IOV at Beta level e eIPoIB to enable IPoIB in a Para Virtualized environment at Alpha level Contiguous pages Internal memory allocation improvements Register shared memory Control objects QPs CQs Ethernet Performance Enhancements NUMA related and others for 10G and 40G OFED VMA integration to a single branch Ethernet Time Stamping at Beta level Flow Steering for Ethernet and InfiniBand at Beta level Raw Eth QPs e Checksum TX RX Flow Steering Errata Kernel upgrade support e YUM update support e Storage iSER at Beta level and SRP e 64bit wide counters port xmit recv data packets unicast mcast VERSION query API library and headers 1 SR IOV Ethernet Time Stamping and Flow Steering are ConnectX 3 HCA capability Mellanox Technologies 11 J Rev 2 1 1 0 0 Known Issues 9 Known Issues The following is a list of general limitations and known issues of the various components of this Mellanox OFED for Linux release Table 7 Known Issues Index Issue Description Workaround 1 IPoIB When
27. tallation using Yum does not update HCA See Updating Firmware firmware After Installation in OFED User Manual 62 On SLES11 1 the package libnl 1586 is required Perform one of the follow to install MLNX_OFED ing Install the libnl 1586 RPM from the SLES11 1 32bit installation disk Install MLNX OFED with the following flag with out 32bit 63 When using bonding on Ubuntu OS the ifenslave package must be installed 18 Mellanox Technologies Rev 2 1 1 0 0 Table 7 Known Issues Continued Index Issue Description Workaround 64 total vfs lt 0 63 gt installation parameter is no longer supported Use enable sriov installation parameter to burn firmware with SR IOV support The number of vir tual functions VFs will be set to 16 For further information please refer to the User Manual 65 Driver Unload Openibd stop can sometime fail with the error Unloading ib cm FAILED ERROR Module ib cm is in use by ib ipoib Re run openibd stop 66 Fork Support Fork support from kernel 2 6 12 and above is available provided that applications do not use threads fork is supported as long as the parent process does not run before the child exits or calls exec The former can be achieved by calling wait childpid and the latter can be achieved by application specific means The Posix system call is supported 67 ISCSI
28. the number of virtual functions 82 A working IP connectivity between the RoCE devices is required when creating an address han dle or modifying a QP with an address vector 83 MLNX OFED v2 1 1 0 0 is not interoperable with older versions of MLNX OFED 84 Unloading mlx4 en while a rdma cm session is established can cause a kernel panic 85 Storage SLES11 SP1 When running multipath rescan while new devices are added to mpath tables multipath may not find all the device mappers 86 Older versions of rescan_scsi_bus sh may not rec If encountering such issues ognize some newly created LUNs it is recommended to use the c flag 87 SRP Reconnecting to a target during host reset stage Run rescan scsi may result in devices going Offline bus sh r 88 SRP Interop The driver is tested with Storage target vendors recommendations for multipath conf extensions ZFS DDN TMS Nimbus NetApp Mellanox Technologies 21 J Rev 2 1 1 0 0 Known Issues Table 7 Known Issues Continued on boot Index Issue Description Workaround 89 DDN Storage Fusion DDN does not accept non default P Key connec 10000 target tion establishment 90 Oracle Sun ZFS stor Occasionally the first command to a LUN may age 7420 not be serviced aborted and cause a successful re connection to the target 91 ZFS does not accept non default P Key c
29. y value O 2 0 2 0 5 2 0 3 0 0 for child interfaces 26 Mellanox Technologies
Download Pdf Manuals
Related Search