Home
Technical Preview Guide
Contents
1. RoCE Log Message Description Type lt pci bus info gt lt hca_name gt Prints the information about the RoCE PCI Information lt speed gt lt model_number gt port function For example lt port_num gt 0000 04 00 1 Emulex OneConnect RoCE HCA 10Gbps OneConnect OCe14000 port 1 lt pci bus info gt ocrdma lt D gt driver Driver loaded successfully on the device Information loaded successfully ocrdma_add leaving ret lt D gt Adding device failed with Error D Error ocrdma_ dispatch ibevent Received unknown event from the hardware Error unknown type 0x lt D gt ocrdma_dispatch ibevent Fatal Device reported a fatal event Error event received ocrdma process acqe lt D gt Invalid event code lt DD gt reported on the device Error invalid evt code 0x lt DD gt lt D gt ocrdma process mcqe cqe for Invalid completion tag reported Error invalid tag0x lt D gt expected 0x lt DD gt ocrdma_ wait _mge cmpl lt D gt Mailbox failed because of timeout Error mailbox timeout fw not responding crdma_mbx_cmd Completion and Extended status in case of Error cqe_ status 0x lt D gt mailbox errors ext_status 0x lt DD gt opcode 0x lt D gt subsystem 0x lt DD gt Opcode and subsystem ids of the failed mailbox Error commands ocrdma_irq handler Fatal EQ Full detected Error Error EQ full eq id Ox lt D gt eqe 0x lt DD gt o
2. cccccecccccccccccccccccccsssssccessccccceseceesecscccccssssesees 18 QOS Behavior viesedesscscaia gies cette wSsSedkawane aleckasdeekaecute sleet dodtedacsaeciaeeeedawediwe 18 0Ce14000 Series Defaults ccc ecce esse ce eec eee seeeeeeeeeeseeeeeseeeeseeseeeeseeeeseeeees 18 QoS Configuration Guidelines sesssesscsseesosssscesesscsssesosesecccsssesosesseese 19 Priority GOUPS sscciceacsceescicicesweles testitakse EARE MANENE ERNEA AREEN EAA dies ead dieleauietew eves 19 WAIE CONUION ETE IE E MENE EE E P E E AE E O E E 19 DCBX Enabled SwitChiesscssssccrsreresis stres eves cv EEn P rA E EREE EREEREER AKERE A E 19 DCBX Disabled Switch Connection generic pause mode seeesssessseseesesssessesssss 21 Updating the Adapter Firmware for ROCE ssssssscsssesceosececoesescossececosececoeesee 21 Determine Firmware Version sssssssesssssesssesssossososeoseosessesessesessessesses 21 Updating the Firmware Manually sessssssssssssesesesssessesseesssesesssseessssesseseee 22 Updating the Firmware using OneCommand Manger ssssssessesesesssseosssseseeseeo 22 Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A 3 D6 EMULEX Table of Contents 4 4 Troubleshooting osrccsecsccccirwesossceonsiscsorsas enaren i ae iee a iii Log Messages cursiisrriiisireirsitiscrte nsise nes sst t ETE EEVEE N ETENEE NEER EEEE a RoCE Error Log Messages and their Descriptions ssssssssssssssesssessesssssssssesse
3. VLANs other than 0 and 1 are used MTU Configuration A maximum MTU of 4200 is allowed for RoCE therefore the corresponding Ethernet interface for a given RoCE interface should be configured for a MTU of greater than 4200 See Interface Configuration on page 11 to find the Ethernet interface for a given RoCE interface Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A is 3 RoCE Configuration for the OCe14000 Series Adapters 14 DG EMULEX Advanced Applications For example ifconfig eth4 mtu 4200 The following procedure must be performed when changing the MTU on a network interface or a switch port 1 Unmount all the existing NFS RDMA mounts 2 Change MTU on host interface and the switch 3 Remount the NFS RDMA mounts Advanced Applications NFS over RDMA Notes e When enabling NFS over RDMA on RHEL 6 4 systems refer to the note on page 8 in the Installing OFED section e On NFS servers where fsid needs to be specified while exporting file systems ensure that they are unique for each exported file system e The usage of memory regions MRs by the NFS over RDMA service is high by default 32 It has been found that there is no performance loss by reducing the MR usage to a lower level This gives flexibility to the number of mounts that can be allowed over a single port For example to lower the default MR usage to 16 add the following line to the etc rc local file sysctl w sun
4. connection o When the OCe14000 series adapter is connected to a DCBX disabled switch it will be in generic pause mode Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A 3 RoCE Configuration for the OCe14000 Series Adapters 19 Advanced Configuration gt G EMULEX QoS Configuration Guidelines Priority Groups It is advisable to split traffic into two or more priority groups e one priority group for RoCE e other groups for non RoCE traffic Many RoCE applications use TCP IP for out of band connection establishment Therefore it is advisable to allocate sufficient bandwidth to non RoCE priority groups L2 Flow Control When a port is running in generic pause mode RoCE latencies can be adversely affected In this situation it is advisable to configure RoCE to use PFC for better results For switches and adapters that do not support PFC RoCE can continue to operate in generic pause mode Bandwidth allocation can be still done for RoCE versus NIC traffic However this allocation cannot be guaranteed since all of the outgoing traffic can be paused in case of congestion DCBX Enabled Switch Switch Configuration At this time none of the known switch vendors for example Arista Brocade Cisco and Juniper allow configuring priority for RoCE specific traffic Priority 5 must be manually enabled on the switch under a priority group other than the FCoE iSCSI NIC priority group Note In the absence of priority
5. gt lt EMULEX RoCE Driver Version 10 2 for Linux Technical Preview Guide P010083 01A Rev A Emulex Connects Servers Storage and People gt G EMULEX Copyright 2014 Emulex All rights reserved worldwide No part of this document may be reproduced by any means or translated to any electronic medium without the prior written consent of Emulex Information furnished by Emulex is believed to be accurate and reliable However no responsibility is assumed by Emulex for its use or for any infringements of patents or other rights of third parties which may result from its use No license is granted by implication or otherwise under any patent copyright or related rights of Emulex Emulex the Emulex logo AutoPilot Installer AutoPilot Manager BlockGuard Connectivity Continuum Convergenomics Emulex Connect Emulex Secure EZPilot FibreSpy HBAnyware InSpeed LightPulse MultiPulse OneCommand OneConnect One Network One Company SBOD SLI and VEngine are trademarks of Emulex All other brand or product names referenced herein are trademarks or registered trademarks of their respective companies or organizations Emulex provides this manual as is without any warranty of any kind either expressed or implied including but not limited to the implied warranties of merchantability or fitness for a particular purpose Emulex may make improvements and changes to the product described in this manual at any time and without any notice
6. www openfabrics org downloads OFED Note When installing OFED 3 5 1 packages on the RHEL 6 4 operating system perform the following additional step to enable NFS over RDMA Change the following lines in the install pl file from NFSRDMA if Skernel m 3 5 or SDISTRO SLES11 2 RHEL6 23 to NFSRDMA if kernel m 3 5 or SDISTRO SLES11 2 RHEL6 234 Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A 7 2 Installing and Uninstalling Q gt lt EMULEX Installing the RoCE Driver for the OCe14000 Series Adapters If OFED has already been installed it must be uninstalled modified and reinstalled 1 Opena web browser and navigate to www openfabrics org downloads OFED 2 Download the appropriate tgz tarball file 3 Extract the downloaded OFED x tgz tarball to the tmp directory tar xvzf tmp OFED 3 5 tgz 4 Install OFED cd tmp OFED 3 5 install pl all Note Package dependencies must be resolved for a successful installation 5 When installation is complete do one of the following e reboot the system e restart OFED by typing service openibd restart RoCE Driver Installation To install the RoCE driver 1 Download the following package from the Emulex website elx ocrdma dd lt release gt lt version gt tar gz 2 Copy the package to tmp and run tar xvzf elx ocrdma dd lt release gt lt version gt tar gz 3 Change directory to the RoCE packa
7. 5 at the switch side the OCe14000 series adapter will continue to be configured for PFC mode for priority 5 This can result in packet losses unrecoverable errors or infinite retries for RoCE traffic Perform the following steps to configure the switch Create a priority group 1 PG 1 for RoCE traffic Assign priority 5 to PG 1 Assign the appropriate bandwidth for example 90 to PG 1 Create PG 2 or something different from PG 1 Assign NIC traffic to PG 2 Assign the remaining bandwidth to PG 2 for example 10 Enable PFC on the switch ports Set both switch ports to pass all VLAN traffic OO ONO ae ON Note Some switches have jumbo fame size support disabled by default on the port and or global level Enable jumbo frame support or set MTU to at least 4200 Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A is 3 RoCE Configuration for the OCe14000 Series Adapters 20 DG EMULEX Advanced Configuration Host Configuration 1 Enable PFC using the OneCommand Manager see the OneCommand Manager Application User Manual 2 Create a VLAN 3 Assign an appropriate IP address to the VLAN interface Example Switch PFC Configuration Note This example is for a Cisco switch that is connected to the OCe14000 series adapter By default Cisco is configured with two priority groups enabled which are fixed and cannot be deleted e Default group name default group e FCoE group name fcoe group Perform
8. Emulex assumes no responsibility for its use nor for any infringements of patents or other rights of third parties that may result Periodic changes are made to information contained herein although these changes will be incorporated into new editions of this manual Emulex disclaims any undertaking to give notice of such changes Emulex 3333 Susan Street Costa Mesa CA 92626 Note References to OCe11100 series products also apply to OCe11100R series products Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A Table of Contents gt G EMULEX Table of Contents Listof Tables citacswatenntwidaantwrdunausvawesiaserdansshacansacannarkwdusinkstaasvebianmadaanees 5 1e INCFORPU CLIO a cca wccessuandaceeuviatauase vesiav waastacensasaiantouaieaviayshadaneddenaaidaened 6 OVEIVICW wi tccnnccannncs ssneinnhsaknennis diannnbdnnsarnanospeeiesneerbisdheewieansanemeaeeieerisaemenesed 6 S pported Adapters ccswcsvicsceutcacay sandra pias ssaerosussodeceasucsaseerehideetassebinsinastenss 6 OFED OVEIrVICW sis kirin rE EEEE EEEN EEEE EENE EE EATER SONEA 6 DDE VIALS 20 cc ncsasienesnesenneneipenesinmmecsaneeendanrnbadsavactaasenenunaseastacebenscesseeneness 6 2 Installing and Uninstalling vscsviwscascavs siwencasncsnwsicinssasadausedessnabesnecransonewanced 8 Installing the RoCE Driver for the OCe14000 Series Adapters sssesssssessssssesseo 8 Installing OPED sisscxssisneiasuustasneiwcssscunsdaxccweusSsavasdandydavernwadeusbvasaenevatee
9. RDMA over converged Ethernet receive mode storage area network Small Computer System Interface Sockets Direct Protocol SUSE Linux Enterprise Server service level interface single root I O virtualization SCSI RDMA Protocol Transmission Control Protocol transmit mode virtual local area network P010083 01A Rev A 7 7 2 Installing and Uninstalling amp DG EMULEX Installing the RoCE Driver for the OCe14000 Series Adapters 2 Installing and Uninstalling Installing the RoCE Driver for the OCe14000 Series Adapters RoCE is a network protocol that allows remote direct memory access over an Ethernet network RoCE is a link layer protocol which allows communication between any two hosts in the same Ethernet broadcast domain Network intensive applications like networked storage or cluster computing require a network infrastructure with a high bandwidth and low latency The advantages of RDMA over other network application programming interfaces are lower latency lower CPU load and higher bandwidth The following operating systems are supported for RoCE e SLES 11 SP2 e RHEL 6 4 Installing OFED The following table lists the available OFED versions that are compatible with the supported operating systems Table 2 1 Compatible OFED Versions OFED Version Operating System SLES 11 SP2 RHEL 6 4 OFED 3 5 YES NO OFED 3 5 1 NO YES Supported OFED packages can be downloaded from the following website https
10. c pause mode 1 Host Configuration On the host and peer systems ensure that Tx and Rx pause flow control is enabled using the operating system standard tools on all of the ports interfaces which are RoCE enabled a To verify status ethtool a ethx b To configure ethtool A ethX autoneg on loff rx on off tx on off 2 Switch configuration a Enable Tx and Rx generic pause flow control on each port b Some switches have jumbo frame size support disabled by default on the port and or global level Enable jumbo frame support or set MTU to at least 4200 Updating the Adapter Firmware for RoCE Note This section can be skipped if the adapter already has the required firmware version and RoCE profile Determine Firmware Version The adapter firmware should be upgraded to version 10 2 x x The upgrade can be done manually or by using the OneCommand Manager application 1 Determine if the firmware needs to be updated List the ocrdma interfaces by typing Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A 3 RoCE Configuration for the OCe14000 Series Adapters 22 gt G EMULEX Updating the Adapter Firmware for ROCE ibdev2netdev 2 Each ocrdma interface maps to an Ethernet interface ethX Use ethtool i ethX to determine the firmware version on the adapter If the reported firmware version does not match the version listed above then the firmware for that adapter needs to be updated The lat
11. crdma_mbx create cq lt D gt Requesting more CQ entries than what the device Error max cqe 0x lt DD gt supports requester cqe 0x lt DDD gt ocrdma set create gp sq cmd The total number of SQ entries requested is more Error req max send _wr 0x lt D gt than what the device supports ocrdma set create qp rq cmd The total number of RQ entries requested is Error req max _recv_wr 0x lt D gt more than what the device supports Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A gt G EMULEX 4 Troubleshooting 24 Table 4 1 RoCE error log messages and their descriptions Continued Log Messages RoCE Log Message Description Type ocrdma mbx create srq req The total number of SRQ entries requested is Error max wr 0x lt D gt more than what the device supports ocrdma_mbx create_qp lt D gt Failed to create qp Error rg err ocrdma_mbx create_qp lt D gt Failed to create qp Error sq_err ocrdma parse dcbxcfg_rsp DCBX state is disabled in the adapter Information DCBX state is disabled ocrdma parse dcbxcfg_rsp PFC is disabled in the adapter Information pfc is disabled ocrdma init service level Failed to get the DCBX configuration from the Error status lt D gt adapter ocrdma is using default service Using the default service level Information level ocrdma_ init hw status lt D gt Initialization of the hardware faile
12. d Error ocrdma_alloc_ resources lt D gt Failed to allocate driver resources Error error Unable to allocate ib device ib_alloc_device failed Error ocrdma_add stat No space in Response data for debugfs request is more than Error stats buff the stats buff size ocrdma_ alloc stats mem stats _ Failed to allocate memory for statistics Error mbox allocation failed command ocrdma_ alloc stats mem stats _ Failed to allocate memory for debugfs Error debugfs mem allocation failed ocrdma_ update stats stats The Statistics command failed from the Error mbox failed with status lt D gt hardware ocrdma_ query port lt D gt Invalid port specified in Query port Error invalid _port 0x lt DD gt ocrdma_ modify port D Invalid port specified in Modify port Error invalid _port 0x lt DD gt ocrdma_ dealloc_ ucontext pd D Dealloc ucontext requested on a PD which is in Information Freeing in use pdid 0x lt DD gt use ocrdma_ get _ dma _ mr err invalid Invalid access rights while allocating lkey Error access rights ocrdma_dereg mr lt D gt fw not De registration of MR failed because firmware is Information responding not responding ocrdma_ copy cq uresp lt D gt copy Failed to copy cq create response Error error cqid 0x lt DD gt Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A 4 Troubleshooting 25 Log Messages gt G EMULEX Table 4 1 RoCE error log messages and their descr
13. deeneus 8 ROCE Driver INstallations oce ccsscsssvensasveeceavece ceeds oveasawe te cedade cess ovaarwete taeadeees 9 Uninstalling the ROCE Driver sssessesssesssesossssesessecsssssosseseseeseeeossesesesseee 10 3 RoCE Configuration for the OCe14000 Series Adapters scccscssssscccees 11 Basic Confid ration a 5s55cc ccnsscresuinsicneagnatawasmauduvesksmecseausseressesessmeasweesswecdeweda 11 Interface CONfiQuratiOn cc cece cece eee ceceeeeeessensseeeeeeeeeeeeeeeessssaaaeeeeees 11 Sample Applications cc cccccce cece cece eee neres DEC cece eeeeeeeeseeeeeseeeeeeeeeeeeeaags 12 Setting the Profile ciccscsvecasevedeesdeesvsgeee oteevedeesseiewsas evens teetedec die eeeevasestee te 12 Gontiguring VLAN Scsi cc cevennticcdeecestecccactapiincdsecects ve decas tencdeteecti vecweeueses cdasews 13 MTU Configuration cccccccccccccccccccccccccccccccccsesesccccesccccssssssessccecesseseeees 13 Advanced Applications isis icctcdcccvccdsascdiieceneicceineeincioiesenedcherecdassceincene ems 14 NES over RDMA iced cesses te ewieecvecedeene vee weweeesviewesevewsue ater aeeeedeevvewswe races ened 14 Server CONPISUPALION oa aie ccessens tne riiseni p Epona n e dee ecteiene das eenwe RE Ei 14 Client CONfISUrATION cccvecscewcscevaweeveesecwoddeenenecvaneedvesadevedes tome deceedebbeuveete twee 15 Enabling RDMA on all Four Ports on an OCe14000 Series Adapter eeeeeeeeeeee 16 Advanced Configuration
14. dev Follow the standard procedure to assign a valid IP address to the desired Ethernet interface that corresponds to the RoCE port You can assign an IP address to eth4 to use ocrdma0 for RoCE Example output ibv_devinfo l 2 HBAs found Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A is 3 RoCE Configuration for the OCe14000 Series Adapters 12 DG EMULEX Basic Configuration ocrdmal ocrdma2 ibdev2netdev ocrdma0 port 1 gt ethO Up ocrdmal port 1 gt ethl Up ocrdma2 port 1 gt eth2 Up ocrdma3 port 1 gt eth3 Up Sample Applications The RoCE connectivity can be tested using the following OFED i built tools applications ibv rc pingpong ib _ send bw ib read oo ib write_bw Note The use of these commands assume that the client ocrdma0 and server ocrdma0 interfaces are configured with IPs 11 192 168 x and 11 192 168 x respectively Examples ibv rc pingpong Server ibv_rc_pingpong g 0 d ocrdma0 Client ibv_rc_pingpong g 0 d ocrdma0 11 192 168 x ib _ send bw Server ib_send_bw d ocrdma0 Client ib_send_bw d ocrdma0 11 192 168 x ib read bw Server ib_read_bw d ocrdma0 Client ib_read_bw d ocrdma0 11 192 168 x ib write bw Server ib_write_bw d ocrdma0 Client ib_write_bw d ocrdma0 11 192 168 x Setting the Profile The RoCE profile can be enabled by using PXESelect BIOS or the OneCommand Manager application e To configure the adapter using PXESelect BIOS refer to the Boo
15. est firmware can be downloaded from the Emulex website The name of the firmware file will have a format of ocl4 x x x x ufi oc14 in the filename refers to the OCe14000 series NIC adapter and x x x x refers to the version Updating the Firmware Manually Note These steps need to be done only once for each adapter Perform the following steps to update the firmware 1 The firmware download command needs to be invoked once for each adapter by specifying any Ethernet interface ethX configured on the adapter 2 List the ocrdma interfaces ibdev2netdev 3 Update the firmware for ethx copy ocl4 x x x x ufi to lib firmware ethtool f ethX ocl4 ufi Note Multiple ocrdma interfaces may be present on a adapter so by using the mapping of ocrdma interfaces to Ethernet interfaces through ibdev2netdev and ethtool i ethx only one Ethernet interface per adapter may be selected Updating the Firmware using OneCommand Manger If the OneCommand Manager application is used to the update the firmware refer to the appropriate section of the OneCommand Manager Application User Manual for the procedure Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A 4 Troubleshooting 23 Log Messages gt G EMULEX 4 Troubleshooting Log Messages RoCE Error Log Messages and their Descriptions Table 4 1 lists ROCE error log messages and their descriptions Table 4 1 RoCE error log messages and their descriptions
16. ges directory cd tmp X X X X Linux RoCE 4 Select the required package based on the host on which the driver is being installed 5 The current Linux distribution on the host may be determined by typing lsb release i 6 Extract the selected package using tar 7 Change directory to the resulting directory from the tar extraction command to find the installer script elx_roce_install sh 8 Run the installer script to install the required RoCE driver and user library RPMs Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A ys is 2 Installing and Uninstalling 10 gt lt EMULEX Installing the RoCE Driver for the OCe14000 Series Adapters Uninstalling the RoCE Driver To uninstall the RoCE driver type elx_roce_install sh uninstall Note SLES 11 platforms may prevent loading of unsupported modules by default In such cases the installer may fail to load the modules even though the RPMs are installed If this occurs try manually loading the ocrdma module after doing one of the following e set allow_unsupported_modules to 1 in etc modprobe d unsupported modules e specify allow unsupported on the command line Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A is 3 RoCE Configuration for the OCe14000 Series Adapters 1 DG EMULEX Basic Configuration 3 RoCE Configuration for the OCe14000 Series Adapters Notes e RoCE multichannel is not supported e SR IOV is n
17. iptions Continued RoCE Log Message Description Type ocrdma check _ qp params lt D gt Validation checks during QP create Error unsupported qp type 0x lt DD gt requested ocrdma check qp params lt D gt Validation checks during QP create Error unsupported send wr 0x lt DD gt requested ocrdma check qp params lt D gt supported send wr 0x lt DD gt ocrdma check _ qp params lt D gt Validation checks during QP create Error unsupported recv_wr 0x lt DD gt requested ocrdma_check _ qp params lt D gt supported recv_wr 0x lt DD gt ocrdma_check_qp params lt D gt Validation checks during QP create Error unsupported inline data size 0x lt DD gt requested ocrdma_ check _ qp params lt D gt supported inline data size 0x lt DD gt ocrdma check qp params lt D gt Validation checks during QP create Error unsupported send sge 0x lt DD gt requested ocrdma check qp params lt D gt supported send sge 0x lt DD gt ocrdma check _ qp params lt D gt Validation checks during QP create Error unsupported recv_sge 0x lt DD gt requested ocrdma_check _ qp params lt D gt supported recv_sge 0x lt DD gt ocrdma_check qp params lt D gt Validation checks during QP create Error Userspace can t create special QPs of type 0x lt DD gt ocrdma_check _qp params lt D gt Validation checks during QP create Error GSI special QPs already created ocrdma check
18. ot supported when RoCE is enabled e Itis highly recommended that PFC is enabled as the default mode when possible while using RoCE See Advanced Configuration on page 18 to enable QoS for RoCE In addition VLAN interfaces must be configured and used for RoCE traffic in order for PFC to work correctly see Configuring VLANs on page 13 VLANs greater than 1 should be used for best interoperability RoCE profiles can be specified in the OneCommand Manager GUI application the OneCommand Manager CLI application and the PXE Boot utility Regardless of the utility you use follow these guidelines to select the appropriate RoCE profile e Choose the RoCE 2 profile for NFS on Linux Note Check the Implementer s Lab on the Emulex website for any updated information on additional use cases for the RoCE 2 profile e For the RoCE 1 profile check the Implementer s Lab on the Emulex website for any updated information on use cases for the RoCE 1 profile For specific information on selecting RoCE profiles see the applicable manual e OneCommand Manager Application User Manual e OneCommand Manager Command Line Interface User Manual e Boot for NIC iSCSI FCoE and RoCE Protocols User Manual Basic Configuration Interface Configuration Identify the RoCE interfaces and their corresponding NIC interfaces by using the following commands To list the ocrdma interfaces ibv_devinfo l To list the corresponding NIC interfaces ibdev2net
19. qp params lt D gt Consumer QPs should not use the CQ of GSI QP Error Consumer QP cannot use GSI CQs ocrdma_ copy qp uresp lt D gt user Failed to copy the QP create response back to Error copy error the user ocrdma_ create qp lt D gt Failed QP create command Error error lt DD gt ocrdma_ modify qp lt D gt invalid Parameter error while trying to modify the QP Error attribute mask 0x lt M gt specified for qpn 0x lt QP gt of type 0x lt T gt old_qps 0x lt OS gt new _qps 0x lt NS gt Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A gt G EMULEX Table 4 1 RoCE error log messages and their descriptions Continued 4 Troubleshooting 26 Log Messages RoCE Log Message Description Type ocrdma build inline sges Failed to build inline SGEs Error supported len 0x lt L gt unspported len req 0x lt LL gt ocrdma_ update wc invalid Invalid opcode received from the hardware Information opcode received 0x x completion ocrdma_reg mr status lt D gt Failed to register MR Error ocrdma_resolve dmac fail to Failed to resolve MAC address Error resolve mac_addr Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A
20. r showmount lt server roce ip gt Note lt server_roce_ip gt is the NIC IP address of the corresponding RoCE interface on the NFS Server For example showmount e 11 192 168 1 Export list for 11 192 168 1 export everyone 4 Mount the file system mount t nfs4 lt server roce ip gt lt path gt o rdma port 20050 lt mount point gt For example mount t nfs4 11 192 168 1 o rdma port 20050 mnt 5 Verify the NFS mount using RDMA cat proc mounts grep lt mount point gt Note The Proto field should be rdma Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A gt G EMULEX 3 RoCE Configuration for the OCe14000 Series Adapters 16 Advanced Applications Enabling RDMA on all Four Ports on an OCe14000 Series Adapter RDMA cannot run on all four ports on the OCe14000 series 4 port adapter due to insufficient MRs To enable RDMA on all four ports decrease the sunrpc rdma_slot_table_entries parameter from the default value of 32 by performing the following steps 1 Determine the total number of client server connections desired per adapter A connection is defined as a link between a single client and a server and is not considered an NSF mount There can be any number of mounts on a single connection 2 Using Table 3 1 look up the appropriate value for sunrpc rdma_slot_table_entries Table 3 1 Setting sunrpc rdma_slot_table_entries Max Connections Per Adapter sunrpc
21. rdma_slot_table_entries 32 4 31 5 24 6 20 7 17 8 15 9 13 10 12 11 11 12 10 13 9 15 8 17 7 20 6 24 5 31 4 41 3 62 2 Note Setting sunrpc rdma_slot_table_entries to a value less than 8 is not recommended because a significant performance drop will occur 3 To change the value of sunrpc rdma_slot_table_entries add the following line to the etc rc local file sysctl w sunrpc rdma_slot_table entries N where N is the desired value Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A is 3 RoCE Configuration for the OCe14000 Series Adapters 17 DG EMULEX Advanced Applications 4 Reboot the system for the new settings to take effect Note The total number of connections is dependent on the number of MRs used per connection and the number of MRs used per connection is dependent on the value of sunrpc rdma_slot_table_entries within the Linux kernel The parameter sunrpc rdma_slot_table_entries can have a value between 2 and 32 with the default being 32 If the default of 32 is used the total number of client to server connections is only three for the entire adapter which is why all four ports of a 4 port adapter cannot be connected by default Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A 3 RoCE Configuration for the OCe14000 Series Adapters 18 Advanced Configuration gt EMULEX Advanced Configuration This section desc
22. ribes the configuration and behavior aspects of RoCE QoS on the OCe14000 series adapters QoS Behavior e Supported o Limited QOS configuration via the OneCommand Manager o A single traffic class group for RoCE per port o A single RoCE priority in PFC mode o Bandwidth allocation for priority groups e Not supported o RoCE Multichannel is not supported OCe14000 Series Defaults Note If generic pause is used ensure that switches have the proper support for this feature Use PFC with priority 5 if the switch does not support generic pause e Adapter boot time o PFC is disabled on all the ports at adapter boot time in the NIC RoCE profile o Generic pause is enabled on all the ports at adapter boot time in the NIC RoCE profile e Back to back connection OCe14000 to OCe14000 o PFC is disabled by default o Generic pause is enabled on that port e DCBX enabled switch connection o When the OCe14000 series adapter is connected to a DCBX enabled switch it will shift the mode from generic pause to PFC mode o The OCe14000 series adapter configures RoCE traffic for priority 5 o Manually enables priority 5 on a switch under a priority group other than a FCoE ISCSI NIC priority group o Inthe absence of priority 5 at the switch side the OCe14000 series adapter continues to be configured for PFC mode for priority 5 This can result in packet losses unrecoverable errors or infinite retries for RoCE traffic e DCBX disabled switch
23. rpc rdma slot table entries 16 See Table 3 1 on page 16 for additional values that can be used e The NFS over RDMA service available with OFED 3 5 1 and OFED 3 5 has been known to fail in certain error conditions It is recommended to use Emulex packaged NFS RDMA modules until future OFED releases address the issue Server Configuration 1 Load and configure the ocrdma driver library 2 Load the RDMA transport module by typing modprobe svcrdma 3 Start the NFS server by typing etc init d nfs start for RHEL or service nfsserver start for SLES Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A is 3 RoCE Configuration for the OCe14000 Series Adapters 15 DG EMULEX Advanced Applications 4 Configure the server Listen port number for RDMA transport echo rdma 20050 gt proc fs nfsd portlist 5 Configure the etc exports file by adding the required entries echo lt path gt rw fsid 0 insecure no subtree check async no root squash gt gt etc exports For example echo export rw insecure no_ subtree check async no root squash gt gt etc exports cat etc exports export rw insecure no_ subtree check async no_ root squash 6 Export the file system configured in etc exports fexportfs a Client Configuration 1 Load and configure the ocrdma driver library 2 Load the RDMA client module modprobe xprtrdma 3 List the file system exported by the NFS serve
24. ssess 23 Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A gt G EMULEX List of Tables 5 List of Tables Table 2 1 Compatible OFED Versions itsscsesavvewcndevan vv svadedadveescaediecetweavsdacdaneecivecess 8 Table 3 1 Setting sunrpc rdma_slot_table_Entrie s ccccc cee eeeeesceeeeeeeeeeeeeeeeees 16 Table 4 1 RoCE error log messages and their GeSCriptionS cece eeeee ee eee eee eeeees 23 Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A 1 Introduction 6 DG EMULEX Overview 1 Introduction Overview This technical preview guide provides requirements and steps to install and configure the Emulex RDMA over converged Ethernet RoCE device driver in the Linux environment This guide is applicable for the following operating systems e Red Hat Enterprise Linux RHEL version 6 4 e SUSE Linux Enterprise Server SLES version 11 SP2 Supported Adapters The OCe14000 family of adapters is supported OFED Overview The OpenFabrics Enterprise Distribution OFED is open source software for Remote Direct Memory Access RDMA and kernel bypass applications OFED is used in business research and scientific environments that require highly efficient networks storage connectivity and parallel computing The software provides high performance computing sites and enterprise data centers with flexibility and investment protection as computing evolves towards applications that require e
25. t for NIC iSCSI FCoE and RoCE Protocols User Manual for more information on the PXESelect BIOS utility Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A is 3 RoCE Configuration for the OCe14000 Series Adapters 13 DG EMULEX Configuring VLANs e To configure the adapter using the OneCommand Manager application refer to the OneCommand Manager Application User Manual or the OneCommand Manager Command Line Interface User Manual The profile can be confirmed from the Ethernet driver load messages in var log messages For example Active profile ID ROCE 2 Note If the correct profile is not reported update the profile ID using the OneCommand Manager or PXESelect BIOS Configuring VLANs To configure a VLAN interface perform the following steps 1 Load the 8021q module if necessary by typing modprobe 8021q 2 Create a VLAN interface by typing vconfig add eth lt x gt lt vlan id gt For example vconfig add eth4 100 3 Configure an IP address on the VLAN interface by typing ifconfig eth lt x gt lt vlan id gt x x x x up For example ifconfig eth4 100 11 192 168 2 netmask 255 255 255 0 up 4 Verify the configuration From the server type ibv rc _ pingpong g 1 d ocrdma0 Note g 1 corresponds to the GID index for the first VLAN From the client type ibv rc pingpong g 1 d ocrdma0 11 192 168 1 Notes e If QoS is set to use PFC the interfaces must be configured with VLANs e RoCE PFC works well if
26. the following steps 1 Use default group as the non RoCE priority group see step 4 on page 19 2 Create another group for priority 5 such as PG 5 see step 1 on page 19 3 Set90 bandwidth to the PG 5 group and 10 to the default group No other changes are required to the default group or fcoe group The following are example switch PFC configurations Cisco Global QoS Configuration Global QoS configuration on the Cisco Switch class map type qos roce match qos 5 class map type queuing roce match qos group 5 class map type network gos roce match gos group 5 policy map type qos roce class roce set gos group 5 class class fcoe set gos group 1 class class default policy map type queuing roce class type queuing roce bandwidth percent 90 class type queuing class fcoe bandwidth percent 0 class type queuing class default bandwidth percent 10 Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A is 3 RoCE Configuration for the OCe14000 Series Adapters 21 DG EMULEX Updating the Adapter Firmware for RoCE policy map type network gos roce class type network qos roce pause no drop mtu 4200 class type network qos class default mtu 9216 class type network qos class fcoe pause no drop mtu 2158 system qos service policy type qos input roce service policy type queuing input roce service policy type queuing output roce service policy type network gos roce DCBX Disabled Switch Connection generi
27. xtreme speeds massive scalability and utility class reliability Some operating system distributions ship OFED in box It is recommended to only install the OFED version for the supported operating system Refer to Table 2 1 on page 8 for a list of compatible OFED versions Note The entire set of OpenFabrics Software from which modules and patches are selected to form OFED releases resides on the OpenFabrics servers and is available for download Abbreviations API application programming interface BIOS basic input output system CPU central processing unit DCBX data center bridging exchange FCoE Fibre Channel over Ethernet HCA host channel adapter IOV I O virtualization IP Internet Protocol IPL initial program load Emulex RoCE User Guide Version 10 2 for Linux P010083 01A Rev A gt G EMULEX iSCSI MR MPI MTU NAS NFS RDMA NIC OFED PCI PCIe QoS RDS RDMA RHEL RoCE Rx SAN SCSI SDP SLES SLI SR IOV SRP TCP Tx VLAN Emulex RoCE User Guide Version 10 2 for Linux 1 Introduction Abbreviations internet Small Computer System Interface memory region parallel message passing maximum transmission unit network attached storage network file system over ROMA network interface card or controller OpenFabrics Enterprise Distribution Peripheral Component Interconnect Peripheral Component Interconnect Express quality of service remote desktop service remote direct memory access Red Hat Enterprise Linux
Download Pdf Manuals
Related Search
Related Contents
DUALHIGH SPEED RS232 ISOLATION TRANSCEIVER Jura Capresso C9 Coffeemaker User Manual Catálogo - Hanna Instruments Portugal Now - Boyles Fitness Equipment Samsung ML-3561N manual do usuário Braun BP3510 blood pressure unit Norsk, svenska and english user manual Manual del usuario Introducción Siemens Gigaset 4010 User's Manual Copyright © All rights reserved.
Failed to retrieve file