Home

Mellanox Grid Director 4700

1. 3 11 Maximum Heat Dissipation Table 12 Maximum Heat Dissipation Maximum heat dissipation max for full configuration 22 206 BTU Hour 3 12 Airflow Table 13 4700 Airflow Rated volumetric air flow for a fully configured chassis 1300 CFM Cubic Feet per Minute 3 13 Physical Characteristics Table 14 4700 Physical Characteristics 4700 Features Characteristics 19 rack mountable chassis 19U height Dimensions Width 19 440 mm Depth 22 75 578 mm Height 25 6 650 mm Weight 128 to 163 lbs 58 to 74 kg depending on configuration Minimum rack depth required for the rail kit N A Maximum rack depth required for the rail kit N A Table 15 HyperScale Physical Characteristics HyperScale Features Characteristics HyperScale rack mountable chassis 20U height 3 14 Weights and Dimensions Table 16 Weights and Dimensions Metric 2 English 1 Unpacked Modules D cm W cm H cm Kg D in W in H in Lbs Packed Modules D W H Kg sMB CM 33 22 4 5 1 1 13 8 7 1 77 2 42 40 28 7 5 1 4 15 7 11 3 3 30 T Mellanox Grid Director 4700 Installation Manual INE e Oo oe a n oa wma a a am a a e ann a asa as a o A A EEE mp E ea A ow on oa w s ons no na an os mm an GE EECHER O A or os w s os a IC eo os oos aas sa lia os en ooa aas aa lia os M ar IC eo os oos aas oa dia oe um a s as o nn os gt o o oe u os oe ss ne C kEKKISkisis Rack Mount Kit for
2. Linux shell prompt The character stands for the Linux shell prompt Mellanox CLI Guest Mode Switch gt Mellanox CLI Guest Mode Mellanox CLI admin mode Switch Mellanox CLI admin mode String lt gt or Strings in lt gt or are descriptions of what will actually be shown on the screen for example the contents of lt your ip gt could be 192 168 1 1 Management GUI label item New Network Management GUI labels and item name New Environment names appear in bold whether or not the name is explicitly displayed for example buttons and icons User text entered into Manager Env1 Network1 Note the quotes The text entered e g to assign as the name of a does not include the quotes logical object Introduction 1 1 Introduction This chapter provides a description of the Mellanox Grid Director 4700 chassis and its system boards 1 1 Mellanox Grid Director 4700 Overview Mellanox s fourth generation Grid Director 4000 series of smart switches addresses the growing size and complexity of clusters by providing high interconnect bandwidth advanced carrier class management and a unique HyperScale stackable architecture The Grid Director 4700 is a high performance ultra low latency and fully non blocking InfiniBand switch for high performance clusters With configurations of up to 324 ports or double sided 648 ports of 40 Gb s per port InfiniBand connectivity
3. 4 3 Packing the Grid Director 4700 Switch If you need to move the Grid Director 4700 switch to a different physical location pack the equipment according to the following procedure gt To pack the Grid Director 4700 chassis 1 Insert the foam grip on the pallet 2 Position the chassis on the foam Verify that the two sFU 40H horizontal fan unit handles at front of chassis point to the two niches in the bottom foam 3 Place the foam plate then the top foam on top of the chassis The foam protection padding fits snugly over the chassis top 4 Insert the accessories in the top foam dedicated locations 5 Slide the cardboard box over the chassis and foam with the flaps facing up Make sure it sits well on the pallet indent 6 Fasten the box and secure it with the pallet using packing straps gt The packing list and part numbers bar codes are glued to the box Safety labels are printed on the box as Packing and Unpacking 1 Figure 13 4700 Switch Packed 4 J T Mellanox Grid Director 4700 Installation Manual 5 Preparing for Installation This chapter describes how to prepare your site gt Read this chapter before you install the switch A 5 1 Tools Required for Installation The following tools and materials are required for mounting the Grid Director 4700 chassis Table 31 Required Tools and Materials uanti escription Quantity Descripti 1 Phi
4. 4700 with HyperScale a Boards aus 75 125 122 1634 295 492 1 lin 2 54cm 2 1 kg 2 205 lbs 3 15 LED Indicators This section specifies the LED indicators of the different Grid Director 4700 modules 31 sLB 4018 Line Board LEDs Table 17 sLB 4018 Line Board LEDs 4700 Specifications Indication LED Description Remarks Link State The two LEDs in each pair are as follows Indicates the state of the 18 Amber LEDs e Green ON indicates physical link up QSFP connectivity 18 Green LEDs e Amber ON indicates logical link up HW only The LED states are as follows e OFF no link detected e ON the port has a link e Blinking Green indicates errors Amber indicates data activity Power Green ON indicates that all board voltage levels are HW only operating normally OFF indicates a problem in power supply to the board Info Blue General purpose LED for management use User only OFF No attention required Blinking Chassis attention required Card Ready Green OFF Card not Ready ON Card ready Blinking Slowly Card loading Blinking Fast Card problem Fabric Board sFB 4700 LEDs Table 18 Fabric Board sFB 4700 LEDs Indication LED Description Remarks Link Each pair of LEDs indicates the state of ports of each 18 pairs of internal link 18 Amber LEDs Line Board indication LEDs Each 18 Green LEDs The two LEDs in each pair are as follows pair of LE
5. Installing the 19 In Long Bracket 63 Figure 26 Using Ejectors to Install a Line Board 66 Figure 27 Using Ejectors to remove a Line Board 66 Figure 28 Using Ejectors to Install a Fabric Board 66 Figure 29 Using Ejectors to remove a Fabric Board 67 Figure 30 Front Panel of the sLB 4018 Line Board oocccnnnnccccnnnociconnnoccccnnnoncccnn noc cnn nn 69 Figure 31 Front Panel of the Management Board 70 Figure 32 Front Panel of the Power Supply Module ooococcccnnccconocnoncccconccononanonononanoccnnnc cnn nc nnnn nn narcos 71 Figure 33 sFU 40V Vertical Fan Unit Front Panel 73 Figure 34 4700 Cable Dressing 80 Figure 35 HyperScale Cabling Guide Bracket for Switches with HyperScale Fabric Boards SEBA e bs e Es 80 CM T Contents Figure 36 Cable Management for Switches with HyperScale Fabric Boards s s s 81 Figure 37 Cabling Dressing between Switches Installed Back 0o Back 83 Figure 38 QSFP CGonpnechor Dimenslons cece eeeeaeeeeaeeeeeeeeeeaeeesaaeseeeeeseeeesaeeesaeeeeneeees 86 Figure 39 Straight through Cables aie uean arna aara da anta 88 Figure 40 Cross connect Cables esessesssessseeestsnettttttttttnttnutn uttu tttnnsttnnntnstustnnnnnsnnsanna nnns nn nanna 88 List of Tables Table 1 Module Configuration per Chassis oooccccnnncinncinnncccnonccnnonanonannnano cnn nn naar nana 18 Table 2 Feature Summary Table AA 19 Table 3 4700 Basic Config ratioN issnin iinit nn 19 Table 4 Spare Parts and Acc
6. It is recommended to start the insertion manually to ensure that the insertion is correct When using an electric screwdriver to fasten the board screws in place do not use more A than Torque 3 force as this could damage the screws 7 3 Installing Fabric Boards Up to nine fabric boards can be installed at the front of the 4700 chassis to achieve a fully non blocking system 67 J Installing System Boards Refer to LED Indicators on page 31 for a comprehensive list of LEDs LED description and functionality gt To insert a Fabric Board into the chassis 1 Verify that the ejectors are fully opened 2 Carefully seat the board into the side guide rails Slowly slide the board into the chassis until the module guide pins engage the chassis The ejectors begin to engage the chassis and start moving When sliding in the board push from the center of the board front panel and not by pushing the ejectors Make sure that the front panel of the Fabric board is completely vertical is pushed all the way in and is leveled with the front panel of the chassis 68 T Mellanox Grid Director 4700 Installation Manual p Gently press the ejectors inward Tighten the two security screws Verify the following LED conditions Power LED ON Info LED OFF e Rdv LED Starts to blink slowly gt To extract a Fabric Board from the chassis 1 Release the security screws 2 Pull the ejec
7. and line boards have ejectors that lock the board in place and serve as levers for seating or extracting e The ejectors snap inward to lock the board in place e Two screws on each board complete the seating and lock of the board in place The following figures illustrate the ejectors action the photos below are of the previous design the new design has black plastic covers 65 j Installing System Boards Line Board Press the ejector inward on both ends of the Line Board Figure 26 Using Ejectors to Install a Line Board Pull the ejector outward on both ends of the Line Board Figure 27 Using Ejectors to remove a Line Board It is recommended to insert boards from top to bottom starting from slot 1 down to 18 Fabric Board Press the ejector inward at the top and at the bottom of the Fabric Board Figure 28 Using Ejectors to Install a Fabric Board 66 Mellanox Grid Director 4700 Installation Manual Pull the ejector outward on both ends of the Line Board Figure 29 Using Ejectors to remove a Fabric Board The ejector mechanism on the Management Board sMB CM is the identical to the one used on the Line and Fabric boards e Itis recommended to tighten the two board screws before starting to operate the system Refer to Torque Settings for torque specifications e When inserting the screws make sure that they are inserted directly into their groove and that they slide in easily
8. the Grid Director 4700 delivers an impressive 51 8 Tb s of non blocking bandwidth with between 100 and 300 nanoseconds of port to port latency Additionally the switch s 648 ports can be divided between two racks for weight distribution and greater ease of cabling The switch s HyperScale architecture provides a unique inter switch link capability for stacking multiples of 324 ports to form highly scalable cost effective and low latency fabrics As a result I O bottlenecks are removed allowing applications to operate at maximum efficiency As the industry s largest QDR switching solution the efficient Grid Director 4700 s smart design provides unprecedented levels of performance and makes it easy to build clusters that can scale out to thousands of nodes T Mellanox Grid Director 4700 Installation Manual Figure 1 Grid Director 4700 with Standard Fabric Boards Front Figure 2 Grid Director 4700 with HyperScale Fabric Boards Front Introduction 1 Figure 3 Grid Director 4700 Rear T Mellanox Grid Director 4700 Installation Manual 1 2 Main Features Ultra low latency between 100 and 300 nanoseconds port to port 324 QDR 40Gb s auto negotiating QSFP InfiniBand ports in a 19U switch 648 port option can be divided between two racks for weight distribution and greater ease of cabling Unique HyperScale architecture allows scaling to thousands of nodes with a single tier of switches under 4
9. Grid Director 4700 switch It is assumed that the readers are familiar with the InfiniBand technology and terminology Related Documentation For additional information refer to the following documents e Grid Director Family Getting Started Guide LIT 00037 e Grid Director Family User Manual DOC 00785 e Grid Director Family Release Notes DOC 00962 e Regulatory and Compliance Reference Guide DOC 00859 Document Conventions The following lists conventions used in this document gt NOTE Identifies important information that contains helpful suggestions gt T Contents CAUTION Alerts you to the risk of personal injury system damage or loss of data a personal injury or a malfunction of the hardware or software Be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents before you work on any equipment A WARNING Warns you that failure to take or avoid a specific action might result in Typography The following table describes typographical conventions in Mellanox documentation All terms refer to isolated terms within body text or regular table text unless otherwise mentioned in the Notes column Term Construct Example Notes Text Block File name pathname opt ufm conj gv cfg Console session code gt flashClear lt CR gt Complete sample line or block Comprises both input and output The code can also be shaded
10. Inside p Right Inside Ledge Kate Outer opening Inner opening Front of Rack e Use all 8 screws Note the order of the nuts and washers e Select the inner openings or outer openings as shown depending on the rack vertical rails locations Inside ledges are used for sliding the chassis along the rail Procedure Summary Perform each step of the following procedure for both rails 1 Use 8 32 5 8 screws and flat washers and spring provided with the kit to assemble the rail positioning the insert in front of the bracket 2 For each of left and right rails take a bracket and attach an insert the two inserts are identical At this stage position the two parts to fit the distance between the rack rails of each side Make sure that you are using the right bracket for the right rail right side when facing the front of the rack and the left bracket for the left rail left side when facing the front of the rack The insert is identical for both rails The rail supports six or eight 8 32 5 8 screws Eight is recommended 3 Select the counter sunk holes in the bracket 4 Assemble the rail with flat washers spring washers and nuts IMPORTANT Do not tighten the screws at this time 5 Align the rail mounting holes with the mounting holes in the equipment rack 6 Install the rail kit by positioning the rail brackets in alignment with the adequate mark on the rack 56 T Mellanox Grid Director 4700 Installat
11. TT eersnee senrago ATA gt TAL d AIS ORIO Covo 0000 200000 00008 h 06090010 gt To insert a power supply unit into the chassis Carefully seat the unit into the designated slot Slowly slide the unit into the chassis until the unit is completely engaged on the chassis edge N Tighten the screws e Verify that you have the correct number of power supplies installed when powering up the system Using fewer than the required power supplies for your specific configuration can cause malfunction e All power supplies must be powered simultaneously If not powered simultaneously the system may not start properly gt To extract a power supply from the chassis 1 Disconnect power cable 2 Loosen the screws LA Extract the power supply by pulling the handles This product may contain sealed lead acid nickel cadmium nickel metal hydride lithium or lithium ion battery The battery must be recycled or disposed of properly Recycling facilities may not be available in your area Batteries or packaging for batteries are labeled in accordance with European Directive 2006 66 EC concerning batteries and accumulators and waste batteries and accumulators The Directive determines the framework for the return and recycling of used batteries and accumulators as applicable throughout the European Union For Taiwan Gi Please recycle batteries HE E AY Installing System Boards This mark
12. applies only to countries within the European Union EU This label is applied to various batteries to indicate that the battery is not to be thrown away but rather reclaimed upon end of life per this Directive gt In accordance with the European Directive 2006 66 EC batteries and accumulators are labeled to indicate that they are to be collected separately and recycled at end of life The label on the battery may also include a chemical symbol for the metal concerned in the battery Pb for lead Hg for mercury and Cd for cadmium Users of batteries and accumulators must not dispose of batteries and accumulators as unsorted municipal waste but use the collection framework available to customers for the return recycling and treatment of batteries and accumulators Customer participation is important to minimize any potential effects of batteries and accumulators on the environment and human health due to the potential presence of hazardous substances 7 7 Fan Units sFUs The 4700 chassis is pre assembled at the factory with the fan units installed at the front of the chassis The fan units are hot swappable modules At the system startup the fan speed is set to turbo mode until adequate temperature is reached The fan speed is then set to normal mode The 4700 hosts a vertical fan unit SFU 40V and a horizontal fan unit sFU 40H When replacing one of the 4700 fan units the other takes over and changes to turbo mode to ensure pro
13. from the Mellanox 4700 chassis power supply Cable wiring 3x14AWG Power Requirements Table 7 4700 Power Requirements Power supply modules sPSU S with N 1 or N N Up to 6 power supplies redundancy Note Each power supply has its own power cord Electrical rating 100 120 200 240V 50 60Hz 60A 12 8A per power feed MAX Typical power consumption QDR full configuration 4000 Watt Leakage current 2254 V mA gt 6mA RMS Note that leakage exceeds 3 5 mA when more than one power supply is being used 27 j 4700 Specifications Indicators Operating voltage Power factor Temperature rating Operational AC and DC Status LEDs 100 240 VAC 50 60 Hz auto sensing 0 98 typical at 230Vac full load 0 98 typical at 115Vac full load 0 45 degrees Centigrade 3 9 3 Power Characteristics Table 8 Power Characteristics Attribute Specification Comments Power Entry 54 90 Volts 47 63 Hz auto sensing Power Rating 1400 Watts 220V per module 1200 Watts 110V per module Nameplate Power Rating VA Input 100V 240Vac 16A Output 48Vdc 29A Nameplate Power Rating at 100V 12A Nameplate Power Rating at 200V 8A Power Factor W V A 0 98 typical at 230 Vac full load 0 98 Watts Volts x Amps x Power typical at 115 Vac full load Factor Power supply efficiency 89 typical at 230Vac full load rated A measure of the ability of the power 8
14. hardware is properly installed it is ready for optional system configuration To configure the system software refer to Mellanox Grid Director 4000 Family User Manual Troubleshooting Solving Operation Problems When a startup problem is encountered it is usually attributed to a single component which is more difficult to isolate than a problem to a subsystem When troubleshooting we suggest that you start by troubleshooting each separate subsystem in the 4700 since there are fewer subsystems than components The 4700 chassis consists of the following subsystems e Power supplies that operate whenever system power is connected 75 j Verifying Installation 1 e Chassis fan units which operate when the system power is connected The fan units will not continue to operate when power is disconnected e Management Board CLI Eth e Line Board sLB 4018 e Fabric Boards sFB 4700 sFB 4700x2 The following are simple checks you can make to determine if there is a fan problem e Listen to the fan units to determine they are operating e Check for any obstructions restricting airflow through the 4700 If you determine that the fan is not operating contact a customer service representative 8 4 2 Identifying Startup Problems gt Perform the following steps when you connect the power cord to the 4700 1 Listen to the fans operation If they do not operate the fans may need to be replaced Continue to Step 2 to determi
15. harmful interference la IN Aala aA 2 This device must accept any interference received gt s Vogt wei including interference that may cause undesired eet ae CSA CIL No 60950 1 Canada ICESNMB 003 Class Classe A N420 Ce m ng COMM TIAARMERMHRECT ORMENRNERA CUA RUBBER RIZADO ET TOBARRA A 5 LS EK EK LENA VCCI A 3 22 Gost R Certification Mellanox products comply with the Gost R Russian safety regulations 3 23 KCC Korea Certification Mellanox products comply with the Korean KCC regulations 3 24 Oscillator Speeds Oscillator speeds are shown in the following table Table 26 Oscillator Speeds Mellanox PN Description Frequency Card 141C00005 CRYSTAL PECL156 25MHZ 156 25 MHz sFB 4700 sFB 4700 2X JOE75 SMD H 1 4MM sLB 4018 OSC 00005 OSCILLATOR PECL 100MHZ 7X5 100MHz sFB 4700 sFB 4700 2X X 1 9MM 50PPM sLB 4018 sMB CM 141C71200 OSCILLATOR 125MHZ 5 0 X 7 0 X 125MHz sFB 4700 sFB 4700 2X 2 0 MM sLB 4018 sMB CM 38 T Mellanox Grid Director 4700 Installation Manual Mellanox PN Description Frequency Card OSC 00009 OSCILLATOR 2 5V HCMOS TTL 66 66MHz sFB 4700 sFB 4700 2X 66 66MHZ 50 PPM sLB 4018 sMB CM 141C65000 OSCILLATOR 50MHZ 7 5 X 5 0 X SOMHzZ sFB 4700 sFB 4700 2X 1 4 MM 3 3V 50PPM sLB 4018 OSC 00008 OSCILLATOR 3 3V CMOS 48MHZ 48MHz sMB CM 50 PPM OSC 00007 OSCILLATOR HCMOS TTL 25MHz sMB CM 25MHZ OSC 00
16. normal operation by anticipating and correcting environmental conditions before they exceed the maximum operating range Refer to Environmental Specifications on page 35 for operating and non operating environmental site requirements 5 6 Chassis Grounding Grounding connection is performed at the customer s site Connect the chassis to a reliable earth ground Ensure that the host is connected to earth ground during normal use e The ground wire should always be hooked up first and disconnected last e Never defeat the ground conductor or operate the equipment in the absence of a suitably installed ground conductor Contact the appropriate electrical inspection authority or an electrician if you are uncertain that suitable grounding is available 48 T Mellanox Grid Director 4700 Installation Manual gt The Grid Director 4700 is delivered with a grounding stud kit The system must be permanently grounded using the grounding stud be The grounding kit includes the following items e A 10 32 5 8 screw e 3 spring washers e 2 nuts 10 e Ground label e Safety warning label This grounding procedure requires a 6 AWG Grounding Cable 4 Figure 14 4700 Grounding Connection Figure 15 Ground Warning Label HIGH TOUCH CURRENT EARTH CONNECTION ESSENTIAL BEFORE CONNECTING SUPPLY Preparing for Installation The Grid Director 4700 provides two grounding options as shown For convenience ground
17. ports Up to 18 Fabric Board sFB 4700 Includes 36 internal ports Up to 9 HyperScale Fabric sFB 4700 2X Includes 36 internal and 12 external CXP Up to 9 Board 12X ports Power Supply sPSU S AC DC Power supply Up to 6 Vertical Fan Unit sFU 40V Vertical Fan Unit 1 Horizontal Fan Unit sFU 40H Horizontal Fan Unit 1 Management Board sMB CM Chassis Management Board Up to 2 Cable Management N A Right cabling guide brackets for cable 1 right Right management Cable Management N A Left cabling guide brackets for cable 1 left Left management Optional cabling guide N A Optional kit for 648 ports must be 1 bracket kit for ordered separately Includes HyperScale HyperScale Fabric cabling guide bracket and long 19 Board brackets 2 1 1 1 Summary of Cabling Management and Rail Kit e Standard chassis configuration 324 ports Cable management standard Rail kit a 19 bracket attached to the chassis e HyperScale Fabric Board chassis configuration 648 ports Ra OO T Mellanox Grid Director 4700 Installation Manual Cable management standard Rail kit 19 bracket attached to the chassis KIT 00022 includes 3 parts must be purchased separately for the 648 port configuration 1 HyperScale cabling guide bracket 2x bracket long 8 screws 2 1 2 Specifications Summary The following table summarizes the Grid Director 4700 specifications Table 2 Feature Summary Table Featu
18. seeen asero ii clas 77 9 1 Cabling Guide Brackets Installation on Either Sides of the Chassis Line Boards Cables 77 9 1 1 Installing Cabling Guide Drackets 78 9 1 2 InfiniBand Cabling on Either Side of the Chassis for Line Cables 79 9 2 Cabling Guide Brackets Installation under the Chassis Fabric Board Cables 80 9 2 1 Installing Cabling Guide Brackets for Switches with HyperScale Fabric Boards 81 9 2 2 Connecting the InfiniBand Cable between Fabric Boarde ssssnnsssnnneseernnnenean 81 9 3 InfiniBand Cabling Guidelines ooooccncccnnninn nnnnnncccnncccnnornnnncnnnn nc cnn nano rca cnn rca rara 83 9 3 1 InfiniBand Cabling Do s and Don ts ccccceeeeeeeeeeeeeeeeeeeeeeceeeeeseaeeesaeeeeeeseneees 84 9 4 Replacing Line Boards EE 84 9 5 Connecting to the Management Porte 84 5 j Contents 1 Appendix A Cabling Information and Specifications ccccccescesseeeseeeeeeeeesseeseseeeeneeeeeees 86 AAs ee e EE 86 A 2 IB Port Cable Specification ooooocncicinnninnnnnnnnccnnncnnnrnnccnn nc 87 A3 1GbE Ports Management 88 List of Figures Figure 1 Grid Director 4700 with Standard Fabric Boards Front 11 Figure 2 Grid Director 4700 with HyperScale Fabric Boards Front 11 Figure 3 Grid Director 4700 Heart 12 E Is A A O O 15 Figure 5 Grid Director 4700 Switching Topologw nano nn nano cnn nn anar nnnann cnn 21 Figure 6 Grid Director 4
19. this document are registered trademarks or trademarks of Mellanox Voltaire is a registered trademark of Mellanox Technologies Ltd All other trademarks mentioned in this document are the property of their respective owners Copyright 2011 Mellanox Technologies Inc All rights reserved Mellanox TECHNOLOGIES Mellanox Technologies Inc Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 972 4 959 3245 Fax 408 970 3403 Copyright 2011 Mellanox Technologies All rights reserved Mellanox BridgeX ConnectX CORE Direct InfiniBlast InfiniBridge InfiniHost InfiniRISC InfiniScale InfiniPCI PhyX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd FabricIT and SwitchX are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners Document Number DOC 00726 A06 T Contents Contents A a A T 8 UW lte Te d EE 10 1 1 Mellanox Grid Director 4700 OvervieW oooocnccccnnnoccccnnnococcnnnnncnnnn nono cnn nano cnn rnnnn nn rra 10 1 2 Main FOatures o AS 13 1 3 Product Architect Guoman id 14 1 3 1 Main Buldim Ee 14 1 32 Reset Buttons ii A A 16 E E ue Din EE 17 2 System Configurations sc Neue ed EE geg 18 21 Configuration SUMMA diia 18 2 1 1 Modu
20. two parts of the rail in Step 1 IMPORTANT If you do not tighten the screws the chassis cannot be mounted safely in the rack 4 Repeat steps 1 to 3 to assemble the other rail and mount it on the other side of the chassis the rails are now ready for chassis installation IMPORTANT Ensure box to box vertical alignment in all four connections See that the bottom side of the rail is aligned with the 1U marking on the rack rail 6 3 Mounting the 4700 in a Rack Standard Installation This section provides instructions on how to mount the 4700 in a rack Screws and clip nuts to connect the chassis to the rack and rail kit to rack are not supplied by Mellanox Make sure that the grounding stud is installed before performing the steps below see A Chassis Grounding on page 48 Prerequisites e Rail kit 19 bracket attached to the chassis To install the 4700 chassis Wi 1 Lift the chassis using a hydraulic chassis lift or another adequate tool and place it on the rails with front of the chassis towards you as shown in the figure below 2 Slide the chassis into place Note At least two people are required to perform this operation 3 Secure the chassis onto the rack by aligning the holes in the front of the chassis with the mounting holes in the equipment rack s vertical front rail 60 T Mellanox Grid Director 4700 Installation Manual 4 Secure the chassis through the holes in the Front Rack rail us
21. 00 nanoseconds of latency Available bandwidth up to 51 8 Tbps of non blocking bandwidth Simple and fast device management Fully managed by Mellanox Unified Fabric Manager UFM 13 Introduction 1 e Support for longer and more varied cable options e Zero down time with no single point of failure and real time fault notifications e QSFP ports supporting either copper or optical cables e Redundant hot swappable Management Boards power supplies and fans meet stringent availability requirements e InfiniBand specification 1 2 compliant 1 3 Product Architecture The Mellanox Grid Director 4700 is a modular rack mountable chassis for a wide range of applications Highlights of the Grid Director 4700 architecture include e Upto eighteen 18 line boards with 18 QSFP InfiniBand QSFP external interfaces and 18 QSFP InfiniBand links for connectivity between the Line cards and the internal Switch fabric e Supports up to nine vertical fabric boards e Supports one or two redundant hot swappable Management Boards sMB for fabric management and chassis management when two Management boards are installed they support failover capabilities in the event of failure of one of the boards e Up to 6 power supply units sPSU S with N 1 N N redundancy e Front to rear air cooling system consisting of a horizontal fan unit sFU 40H and a vertical Fan Unit sFU 40V for the Grid Director 4700 e Cabling for InfiniBand management
22. 00 port on the switch is equipped with transmit and receive circuits that can adjust to fit different types of link characteristics and either optical or copper cables The QSFP cables transmit and receive up to 10 Gbps per channel total up to 40 Gbps per port over a single active passive optical or copper cable Passive copper cables do not consume any additional power while active copper and optic cables consume up to 2 5 Watt You can use the QSFP module with Copper or Optical cables You can install any combination of the cables in any of the QSFP cages A 2 IB Port Cable Specifications The following table details the cabling specifications for the 4036E 4X InfiniBand ports Table 33 4X InfiniBand Port Cabling Specifications Type Maximum Cable Length 4XQDR AWG Length ft Length m Mellanox P N 26 22 96 7 CBL 00158 26 16 4 5 CBL 00086 30 9 84 3 CBL 00185 30 6 57 2 CBL 00084 30 3 28 1 CBL 00083 30 1 64 0 5 CBL 00113 4XQDR 26 26 25 8 CBL 00157 4036E HCA 26 22 96 7 CBL 00158 26 16 4 5 CBL 00086 30 9 84 3 CBL 00185 30 6 57 2 CBL 00084 30 3 28 1 CBL 00083 30 1 64 0 5 CBL 00113 Type Maximum Cable Length 4XQDR Length ft Length m Mellanox P N 9 84 3 New PN Finisar 16 4 5 New PN Finisar 32 8 10 CBL 00102 49 2 15 CBL 00103 65 6 20 CBL 00104 87 j System Cabling Type Max
23. 011 OSCILLATOR HCMOS TTL 25MHz sFB 4700 sFB 4700 2X 25MHZ 2 5V sLB 4018 OSC 00010 OSC HCMOS TTL 25MHZ 1 8V 25MHz sMB CM SOPPM TR 4NS MAX 7X5MM OSC 00006 OSCILLATOR SE 1 8V 19 2MHZ 19 2MHz sMB CM 50PPM 3 25 Shock and Vibration Test Reports Prepare the Grid Director 4700 to ship in a rack by carefully following the rail kit installation instructions as detailed in Assembling and Mounting the Rails The Grid Director 4700 is susceptible to shock and vibration if the rail kit installation instructions are not followed precisely The following tables detail the shock and vibration test reports compliant with shipping in a rack 3 25 1 4700 Test for Transportation without a Rack Table 27 Vibration Test Description Vibration Axes 3 axes X Y Z Non Operating Random Frequency range 5 300Hz Vibration Vibration level Z axis 1 03g RMS X Y axis 0 6988 RMS Vibration test for each axis 60 minutes Operating Sine Frequency range 5 500Hz Vibration Vibration level 0 1G peak Vibration test for each axis 40 minutes Operating Shock Vibration Level Z axis 5g peak half sine Shock test for Z axis 3 shocks 10ms duration each shock 39 j 4700 Specifications Table 28 Free Fall Test Description Drop Height 0 1 meter Number of drops of test One drop performed on the bottom face 2 edges and 2 corners of the test item item Total number of drops 5 d
24. 2 2001 EN 61000 4 5 1995 Al 2001 E E E N 61000 4 6 1996 Al 2001 N 61000 4 8 1993 A1 2000 N 61000 4 11 1994 A1 2001 36 T Mellanox Grid Director 4700 Installation Manual 3 20 3 20 1 3 20 2 EN 61000 3 3 95 A1 01 Voltage fluctuations and flicker e Japan VCCI Technical Requirements V 3 2001 04 CISPR 22 1997 A1 2000 A2 2002 Class A e Australian New Zealand C Tick AS NZS CISPR 22 04 e KCC Korea Restricted Hazardous Substances Certification RoHS 6 InfiniBand Certification IBTA 1 2 Declarations Declaration of Conformity Mellanox products comply with the Radio amp Telecommunications Terminal Equipment Directive 99 5 EEC the EMC Directive 2004 108 EC and of the Low Voltage Directive 2006 95 EC Hazardous Substances RoHS 6 Compliance Declaration Mellanox products comply with the RoHS directive 4700 Specifications 3 21 Label s The Grid Director 4700 label example below shows safety and EMC text and logos Figure 11 4700 Label example See Installation Instructions 99 4700 before connecting to the Power Supply Voir la notice d installation Volraire avant de reccorder au r eseau PPPPPPPP RRR Vordem ansch liessen ans A NON E get MMWWY Yxxxxxx 100 120 200 240V 50 60Hz 60A S N MIN eme 8 a y Gr This device complies with part 15 of FCC rules Operation is subject to the following two conditions e 1 This device may not cause
25. 5 typical at 115Vac full power supply to convert source load rated power AC power into usable DC power 3 9 4 Power Data per Module Table 9 Power Data by Module Module Name Module Watts Module Configured Total Watts Qty Fan Tray sFU 40H 380 1 380 Fan Tray sFU 40V 250 1 250 Management Board sMB CM 45 2 90 18 port Line Board sLB 4018 110 18 1980 Fabric Board sFB 4700 110 9 990 HyperScale Fabric Board sFB 4700 X2 210 9 1890 3 9 4 1 Calculating the Required Number of sPSU S Power Supply Modules To find the minimum number of power supply modules required for a given switch configuration 1 Add up all DC power consumption numbers according to the built configuration 2 Increase the number of sPSU S modules according to redundancy requirements A single sPSU S unit provides up 1400 Watts 28 T Mellanox Grid Director 4700 Installation Manual This formula provides margin for momentary power peaks and enhances reliability by preventing the power subsystem from continuously operating at maximum power 3 For redundancy N 1 or N N configure additional power supplies Table 10 Module Power Data Mellanox Grid Director 4700 Description Module Name DC Power Consumption Watt Basic chassis configuration 4700 675 0 Single management board sMB Horizontal fan units sFU 40H Vertical fan unit sFU 40V Two power supply units 2 x SPSU S sFB 9 QDR connections fabric board sFB 990 0 sMB sMB C
26. 700 HyperScale Spine Switching Topology 22 Figure 7 HyperScale Mesh Configuration oooooconnncinnccnnncccnononnnonnnnnoccnnncn ron n nano rana nn nana rana rana 23 Figure 8 HyperScale Edge Configuration ran nn nano nn c nn aran naar 23 Figure 9 SFU 40H Horizontal Fan Um 25 Figure 10 4700 Line Board Slot Numbering cents eaaeeesaaeseeeeeseeeesaeeesaeeeeneeees 26 Figure 11 4700 Label example ooonoccccnnnicccononoconnnonccnnnnnnccn rn 38 Figure 12 Center of Gravity Data ooooonnncccincccnonccnnocnnonccnnoncncnnnn nono nn nana rra r arrancan rana n nana 41 Figure 13 4700 Switch Packed sis cic tiie dd cis ean it ibid 46 Figure 14 4700 Grounding Connection 49 Figure 15 Ground Warning Label 49 Figure 16 Grounding Holes for A 200 50 Figure 17 4700 Fully Configured N 1 Redundancy c ooocccccccccoccnononanonaccnoncnonona nano nnnnnn cnn nn cnn nc nnnn nn nnncccns 53 Figure 18 Overview of Assembled Rail kt 56 Figure 19 Positioning the Insert in Front of the Right Bracket 58 Figure 20 Assembling the Rail Bracket AAA 58 Figure 21 Right Rail Assembly top and Left Rail Assembly bottom ooonnccccnnnncicinonaccconnnaccccnnnacccnnn 58 Eigure 22 ANNM er aisea Re e ea ET anA SRI RARAY iia 59 Figure 23 Securing the 4700 Chassis for Transport Regular ConfiguratiON oooncccinncinnnnninnnncncccccn 62 Figure 24 Securing the 4700 Chassis for Transport HyperScale Configuration cooocccinnnnnnnnncncccccns 62 Figure 25
27. 700 in a Rack Standard Installation ccccccceceeeeeeeeeeeeeeeeeeeeeeeaeeseeeeees 60 6 4 Mounting the HyperScale Chassis in a Rack ssssssssssresinesrnssrnssrrssrnssrnssrnssrnssrnssrnssrnssrnn 62 6 5 4700 Power Up civil hee ie ei ie oie ene i ee eects 64 7 Installing System Boards ccseccsceceeeseeeeseeseneeeeneeeeeeaeeesaeseseeeeeeeeeseaesasaaesgaeeeeeeeeeseeseseaeeneeeeeees 65 7 1 Chassis Board Contouratton snene 65 TA d RENE 65 7 2 Inserting Extracting Boards AA 65 EC SEHR eet ee cui ei aci 67 7 4 Installing the Line Board telBA018 nana ncccnnnns 69 7 5 Installing the Management Board SMB CM occccnccccnocinononcconcccnoncnononcnnno cc nan na ran nnnnnn ranma 70 7 6 Installing the Power Supply Unit ebGUl Ei 70 dl Ran Uns is FU Sa ia 72 7731 Vertical Fan Unit SFU 40V ccoo a aaa 73 7 1 2 Horizontal Fan Umt SFU 40H ccoo ic aia ae 74 8 Verifying Installation cccccceceeeceeeeeee ee eeenee seen nnee seen nnee eee snee rre 75 BI Verifying Start Up Operations ssnstinin airinn aaaea airia aa aa aas aaaea aait 75 8 2 Verifying Network Connections oooccccccccnnnnconocnnnnccccnnnnnnnnnnn non nnnnc cra nn nn ranma nnnr ranas 75 33 Where to Go Ni ed oe do 75 8 4 Troubleshooting soene i iA Ed le ee 75 8 4 1 Solving Operation Problems esseesseesseeeseeesnnesn nesr nestneernstnnnnnstnntnsstnnsrnssrnnsnnne 75 8 4 2 Identifying Startup Problems non nnnn cnn rca 76 9 System Cabling
28. Ds indicates e G ON indicates physical link the state of two ports of reen indicates physical link u HW only pay p each Line Board e Amber ON indicates logical link up The LED states are as follows e OFF no link detected e ON link detected e Blinking Green indicates errors Amber indicates data Power Green HW only ON indicates that all board voltage levels are operating normally OFF indicates a problem in power supply to the board Info Blue General purpose LED for management use 32 T Mellanox Grid Director 4700 Installation Manual Indication LED Description Remarks User only OFF No attention required Blinking Chassis attention required Card Ready Green OFF Card not Ready ON Card ready Blinking Slowly Card loading Blinking Fast Card problem HyperScale Fabric Board sFB 4700 2X LEDs Table 19 HyperScale Fabric Board sFB 4700 2X LEDs Indication LED Description Remarks Link Each LED indicates the state of ports of each Line 18 pairs of internal link 18 Green LEDs Board indication LEDs Each LED HW only e ON indicates physical link up indicates the state of two ports f each Line Board e OFF no link detected EE e Blinking indicates errors CXP Link The two LEDs in each pair are as follows Indicates the state of the 36 36 Amber LEDs e Green ON indicates physical l
29. M 45 0 Modules for the Mellanox Grid Director 4700 Description Module Name DC Power Consumption Watt sLB 4018 18 QSFP ports per sLB 4018 sLB 4018 110 0 Add 1 5 Watts for each connected QSFP optical module Spare Parts for the Mellanox Grid Director 4700 Description Module Name DC Power Consumption Watt sFU 40H Horizontal Fan Unit sFU 40H 380 0 sFU 40V Vertical Fan Unit sFU 40V 250 0 3 10 Calculating Heat Output in BTUs To compute the approximate BTUs for a given configuration find the power consumption in Watts e g from Power Data Per Module see and multiply by 3 408 Example calculations for specific configurations are shown in the following table Table 11 BTUs for QDR Sample Configurations Configuration Modules Included Power Watts Heat Output BTUs 18 ports per sLB 4018 9 sFB 4700 1 sLB 4018 1 sFU 40H 1 2 041 W 6 965 sFU 40V 1 sMB 2 sPSU S 108 ports 6 sLBs x 18 9 sFB 4700 6 sLB 4018 1 sFU 40H 1 2 674W 9 123 port per sLB sFU 40V 1 sMB 2 sPSU S 29 j 4700 Specifications Configuration Modules Included Power Watts Heat Output BTUs 324 ports QSFP copper 9 sFB 4700 18 sLB 4018 1 sFU 40H 1 4 244W 14 479 sFU 40V 2 sMB 3 sPSU S 324 ports QSFP copper 9 sFB 4700x2 18 sLB 4018 1 sFU 40H 1 5 279W 18 010 HyperScale Fabric Boards sFU 40V 2 sMB 4 sPSU S
30. Mellanox TECHNOLOGIES Mellanox Grid Director 4700 Installation Manual www mellanox com NOTE MELLANOX TECHNOLOGIES INC AND ITS AFFILIATES MELLANOX FURNISH THIS DOCUMENT AS IS WITHOUT WARRANTY OF ANY KIND MELLANOX DISCLAIMS ALL WARRANTIES EXPRESS OR IMPLIED INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE NON INFRINGEMENT AND THOSE ARISING FROM A COURSE OF PERFORMANCE A COURSE OF DEALING OR TRADE USAGE MELLANOX SHALL NOT BE LIABLE FOR ANY ERROR OMISSION DEFECT DEFICIENCY OR NONCONFORMITY IN THIS DOCUMENT AND DISCLAIMS ALL LIABILITY INCLUDING LIABILITY FOR INFRINGEMENT OF ANY INTELLECTUAL PROPERTY RIGHTS RELATED TO THE INFORMATION CONTAINED IN THIS DOCUMENT No license expressed or implied to any intellectual property rights is granted under this document This document as well as the software described in it are furnished under a separate license and shall only be used or copied in accordance with the terms of the applicable license The information in this document is furnished for informational use only is subject to change without notice and should not be construed as any commitment by Mellanox Except as permitted by the applicable license no part of this document may be reproduced stored in a retrieval system or transmitted in any form or by any means without the express written consent of Mellanox Names and logos identifying products of Mellanox in
31. am Plate Carton Plate placed at the top of the rear of the chassis to protect the opening ZA Ia j AJo Tn lr Bottom Foam aa Packing and Unpacking Part Number Description 8 Cardboard Box 9 Pallet 4 2 Unpacking the Grid Director 4700 gt To unpack the Grid Director 4700 chassis 1 Carefully cut the packing straps and open the carton box 2 Remove the box from around the chassis by sliding it upwards as shown The Grid Director 4700 chassis in its partially unpacked state is shown as follows D The upper packing foam covering the Grid Director 4700 contains the following components Rail kit Cabling guide brackets cable management 44 T Mellanox Grid Director 4700 Installation Manual Screw kit Grounding kit Console cable Power cables Getting Started Short Guide Mellanox Solution CD with documentation and software utilities fastened on top of the Grid Director 4700 chassis wrapped in an antistatic bag 3 Remove all of the accessories listed above and as well as the packing foam on top of the chassis 4 Lift the Grid Director 4700 from the pallet and place it on a flat surface Due to the weight of the Grid Director 4700 chassis two people are required to perform this Use the four handles at top of chassis to carry or move the unit A Use safe lifting procedures when moving the unit to avoid personal injury
32. amage to the cables Tie wraps and zip ties can damage the cables Velcro tie is the recommended option e Do not bend the InfiniBand cables too sharply The recommended minimum bend radius is 4 10 cm 79 j System Cabling The following figure shows the 4700 cable dressing Figure 34 4700 Cable Dressing 9 2 Cabling Guide Brackets Installation under the Chassis Fabric Board Cables HyperScale cabling guide bracket are installed at the front of the rack between two preinstalled switches The bracket is designed to accommodate adequate cable bend radius The cabling guide bracket is part of a kit for setting up a 648 port configuration Selecting the optimal InfiniBand cable length depends on the overall cluster configuration and requires proper planning to produce orderly and maintainable cabling e The 4700 space in the Rack should be free from obstructions such as power strips LZ e Itis recommended that the rack rails that overlap with the 19U of the 4700 chassis are free of PDU and power inlets Figure 35 HyperScale Cabling Guide Bracket for Switches with HyperScale Fabric Boards sFB 4700X2 80 T Mellanox Grid Director 4700 Installation Manual 9 2 1 9 2 2 Installing Cabling Guide Brackets for Switches with HyperScale Fabric Boards i The HyperScale chassis must be pre installed in the rack with the 2 x 19 long bracket The cabling guide bracket for HyperScale Fabric Boards is instal
33. an damage the cables Velcro tie is the recommended option e Do not bend the InfiniBand cables too sharply The recommended minimum bend radius is 4 10 cm 82 T Mellanox Grid Director 4700 Installation Manual The following figure shows cabling between two 4700 installed back to back for a 648 port configuration Figure 37 Cabling Dressing between Switches Installed Back to Back 9 3 InfiniBand Cabling Guidelines It is recommended to dress the cables connected to the InfiniBand connectors of each sLB into two groups left and right and fasten them down gently after connectivity has been verified The cables from the same half of the Line Boards are bundled on the corresponding hangers and will be routed down along the Rack side wall Cables should be arranged so that each Line board s cables don t interfere with the neighbor Line Board insertion or removal System Cabling 9 3 1 InfiniBand Cabling Do s and Don ts Do not kink the cable Do not over bend the cable behind the connector Do not twist the connector If it is necessary to re orient the connector start the adjustment further down the cable at least 75 cm away from the connector so the twist occurs gradually along the cable instead of abruptly near the connector 9 4 Replacing Line Boards If the cabling is done according to the guidelines described in the InfiniBand Cabling guidelines above sLB 4018 Line Boards can be rep
34. and power located at rear of chassis e Local and remote management and provisioning using CLI Command Line Interface 1 3 1 Main Building Blocks The Grid Director 4700 rear panel has 18 slots in which you can install the Line Boards sLB 4018 T Mellanox Grid Director 4700 Installation Manual The following figure is a simplified functional block diagram of the Grid Director 4700 components Figure 4 4700 Block Diagram Up to 18 Midplane Backplane sFU 40H sFU 40V The main building blocks of the Grid Director 4700 chassis are listed and described in the following sections Grid Director Midplane The Grid Director includes a passive Midplane that connects the InfiniBand differential signals control signals and power between the different boards in the chassis Fabric Board signals are routed from the front of the chassis to the Midplane Line Boards are routed from the rear of the chassis to the Midplane The Midplane is not field serviceable Line Boards The rear side of the Grid Director 4700 can host up to up to eighteen 18 Port QDR 40Gbps Line Boards sLB 4018 Each line board has 18 external QSFP interfaces and 18 internal 4X InfiniBand interfaces for connectivity between the Line cards and the Switch fabric Line boards are installed at the rear of the chassis in a horizontal position Fabric Boards The Grid Director 4700 supports up to nine Fabric Boards sFB that are installed a
35. bly horizontal fan unit a second one is on the sMB CM Management Board front panel Management Reset To reset the Grid Director 4700 push any of the reset buttons using a thick wire or tip of a pen Pressing the Reset button for one second resets the sMB CM Management Board only pressing the Reset button for at least six seconds resets the entire chassis E T Mellanox Grid Director 4700 Installation Manual 1 4 Management The Grid Director 4700 includes smart device management that provides a simple interface for deploying troubleshooting maintaining and upgrading the switch With a simple to use CLI interface routine tasks such as monitoring the switch operation or upgrading software and firmware are made simple l System Configurations 1 2 System Configurations 2 1 Configuration Summary This section summarizes the configuration for the Grid Director 4700 modules 2 1 1 Module Configuration per Chassis The following table provides a comprehensive list of modules per chassis as well as the maximum number of modules that can be installed in a 4700 system Some of the components are provided with the basic configuration chassis The cabling guide bracket kit KIT 00022 for HyperScale Fabric Board chassis with 648 ports is optional Table 1 Module Configuration per Chassis Module Name Description Number of Modules per chassis Line Board sLB 4018 Includes 18 QSFP
36. board which allow the IB differential signals control signals and power connectivity between the different boards in the chassis e Cables for InfiniBand management and power all located at the rear e Up to 324 InfiniBand QSFP 40 Gbps ports e Bisectional switch bandwidth QDR 51 8 Tbps 3 2 4700 Fabric Board sFB 4700 Specifications e Up to 9 hot swappable Fabric Boards running at Quad Data Rate QDR which support the switching Clos topology e Hot swappable operation with no traffic disruptions e On board high performance low power CPU for low level chassis management purposes e EEPROMS to store the switch ASIC ERP Embedded RISC processor code e Indicators Physical connectivity and logical connectivity LEDs per Line Board link port power Rdy and Info LED e On board temperature monitoring reported to Management system e Thermal shutdown for over temperature sensing 3 3 4700 Line Board sLB 4018 Specifications e Up to 18 sLB 4018 hot swappable Line Boards sLB running at 40 Gbps Quad Data Rate QDR e 18 auto sensing QSFP InfiniBand ports 40 Gbps e Hot swappable operation with no traffic disruptions e All ports support copper or optical cables e Indicators Physical connectivity and logical connectivity LEDs per port power Rdy and info LEDs e On board high performance low power CPU for low level chassis management purposes e On board temperature monitoring reported to Management system e Thermal shutdown for
37. ck and the left bracket for the left rail left side when facing the front of the rack The insert is identical for both rails 57 Rack Mounting the 4700 Chassis The rail supports six or eight 8 32 5 8 screws Eight is recommended Figure 19 Positioning the Insert in Front of the Right Bracket Inner Opening Outer Opening Figure 20 Assembling the Rail Bracket 2 Select the round holes in the bracket 3 Assemble the rail with flat washers spring washers and nuts 4 Do not tighten the screws at this time Figure 21 Right Rail Assembly top and Left Rail Assembly bottom T Mellanox Grid Director 4700 Installation Manual 2 Align the rail mounting holes with the mounting holes in the equipment rack as displayed previously 1 Position the rail brackets in alignment with the 1U mark on the Rack Note you can install any equipment adjacent to the rail kit as long as you align the rail kit with the U mark Figure 22 U Alignment U Alignment For M6 Screws For M5 Screws 1 U Mark 2 Attach the front side of the rail to the rack s front vertical side rail with 2 screws and the rear side of the rail to the rack s rear vertical side rail with 2 screws The following figure shows the left rail completely assembled and what it looks like when it is attached to the rack Rack Mounting the 4700 Chassis 1 3 Go back and tighten the screws that were used to connect the
38. d Director 4700 with HyperScale Fabric Board spine InfiniBand internal switching topology Grid Director 4700 HyperScale Spine Switching Topology Figure 6 va The Grid Director 4700 topology feature eighteen symmetrical Line Boards sLB and nine interconnect Fabric Boards sFB 4018 includes one 36 Each Fabric Board sFB 4700 includes one 36 Each Line Board sLB Each HyperScale Fabric Board sFB 4700 switch chips The LEDs on the Fabric Board modules indicate proper connectivity and traffic flow between each line board and the respective fabric board 22 Mellanox Grid Director 4700 Installation Manual 2 3 3 HyperScale Configurations HyperScale Mesh Configuration Figure 7 HyperScale Mesh Configuration HE i i ii d E e Multiple 4700 chassis can be connected in a mesh e This configuration uses 4700 stacking fabric boards and 4700 switches e Ideal for over 648 node counts e CXP CXP cables are used between switches HyperScale Edge Configuration Figure 8 HyperScale Edge Configuration min SEE eed Em mun 3 ZC mm um up E mm on mu mg Zu mum mp e Combines 4700 and 4036 switches e CXP 3QSFP cables are used between switches e Ideal for 324 648 node counts 23 j 4700 Specifications 3 4700 Specifications 3 1 4700 Architecture e 9 slots of Fabric boards in front of the chassis e 18 slots for Line Boards e Based on passive Midplane
39. e rack in line with the Line Boards and are designed to accommodate adequate cable bend radius The cabling guide brackets are part of the basic chassis configuration b Selecting the optimal InfiniBand cable length depends on the overall cluster configuration and requires proper planning to produce orderly and maintainable cabling HyperScale cable guide requires an additional 1U of rack space making the total height 20U instead of 19U The 4700 space in the Rack should be free from obstructions such as power strips The rack mounted cabling guide bracket should be installed with its bend facing out otherwise it will prevent Line Boards from being inserted or removed It is recommended that the rack rails that overlap with the 19U of the 4700 chassis are free of PDU and power inlets The 4700 cabling guide brackets are illustrated in the following section 77 j System Cabling 9 1 1 Installing Cabling Guide Brackets el en on en on on aa UR en en en en en es kg sg LS gt ee en 2 s WW WW ev kd O CW e Left and Right brackets are marked L for left and R for right When installed make sure that the grips face out The rack gripping brackets shown are installed on the 4700 rack s rear rail gt To install the rack gripping cabling guide brackets 1 Install the cabling guide brackets at the rear of the rack itself on the rail and not on the 4700 chassis It is not important from t
40. ed the 19 in bracket and screws from the kit see Figure 6 10 Tf the chassis is installed disassemble the original 19 in rack mount brackets from the chassis and assemble the 19 in long bracket Note Count five slots to the left of the HyperScale cabling guide bracket and then install it In the front of the rack Install the HyperScale cabling guide bracket under the rail kit as decribed in Chapter 9 on page 77 Mount the chassis in the rack as described in Assembling and Mounting the Rails Note it is important to leave enough space in the rack for the second 4700 X2 chassis 5 In the back of the rack install the regular cable management as described in Chapter 9 Secure the chassis onto the rack 1 Align the holes in the front of the chassis with the mounting holes in the equipment rack s vertical front rail 2 Secure the chassis through the holes in the Front Rack rail using 16 screws 8 for each side and adequate clip nuts To stabilize the chassis for transport secure the 4700 chassis onto the rail by using three 8 32 screws 3 8 tooth washers and flat washers insert them into the oval holes and the threaded holes in the chassis side walls see Figure 6 10 Note Once the rack is installed on the premises these screws should be taken away to allow chassis removal in case the sides of the rack are not accessible Figure 25 Installing the 19 In Long Bracket Rack Mounting the 4700 Cha
41. eps accompanied by figures showing the components as they appear aligned and then assembled one by one Loss J T Mellanox Grid Director 4700 Installation Manual 6 2 1 6 2 2 Rail Kit and Mounting Components The 4700 package includes e Rail kit e Mounting ears or 19 brackets that are factory installed at the front of the chassis The supplied rail kit is adjustable to allow for mounting the chassis in racks of varying depths The rail bracket kit includes the following items e Left bracket left rail e Right bracket right rail e 2 inserts with elongated holes same for both left and right rails e 8 32 screws 5 8 qty 2 x 8 e 8 32 screws 3 8 qty 2 x 3 to be used in case of chassis transportation in a rack e 8 32 nuts qty 2 x 11 e M4 flat washers qty 2 x 11 e Spring washers qty 2 x 8 e Tooth washers qty 2 x 3 gt Clip nuts are not provided b Rail Kit Assembly Overview This section shows a diagram of an assembled rail kit and provides the procedures for assembling the rail kit Rack Mounting the 4700 Chassis 1 The following image shows the two rails completely assembled and displays the left rack assembled to the rack and how the left rail looks when it is attached to the rack Figure 18 Overview of Assembled Rail Kit Rear of Rack screw Left Bracket Z Right Bracket Flat Washers Rail Mounting Holes Spring Washers Q Nut Nylac D Left
42. essories ccconoccccconoccccnononnnccnnnnncccnnonnncnnnnnnncnnnnnnnnnnnn rn rr naar rr nr naar rn rnnnnrrnnnnnes 20 Table 5 Torque Settings for Module Assembly ssssseesseesseeseessnrssrnstrnestnsrnssrnsstnssrnnsrnssrnssrnnnrnnnnnnt 26 Table 6 Clearance Heouirements A 27 Table 7 4700 Power Requirement A 27 Table 8 Power Characteristics Ata 28 Table 9 Power Data Dy Modula tsi coctel aa 28 Table 10 Module Power Data ANEREN 29 Table 11 BTUs for QDR Sample Configurations nn nn canon narrar rn 29 Table 12 Maximum Heat Disspaton cnn cnnnn ainan cnc nano kaaa EE nc ran rancia 30 Tape TS ATO Eine 30 Table 14 4700 Physical Characteristics oooonoocccnonocccccononnncnanonnnccononnncnnno nn nc nano nn nc canon nn rnnnr nn nrcnnnnnncnnns 30 Table 15 HyperScale Physical Charactertstce non nnnnn nn nncnnnnnnnnnns 30 Table 16 Weights and Dmensions nn cnn nn ttn Ennn nn nn cnn cnn nnnna nen 30 Table 1 7sLB 4018 Line Board LEDS ii a 32 Table 18 Fabric Board sFB 4700 LEDS sirai anaana aa e EA A AAAA AE 32 Table 19 HyperScale Fabric Board SFB 4700 2X LEDS ccccsceeeceeeeeeeeeeeeeeeeeeeeeeseaeeesaeeeeeeeeaes 33 Table 20 Management Board eMp CMiLEIe 33 Table 21 sFU 40H Horizontal Fan Unit LEDS 000 00 eee ee enne ee ee eee ee eee ae ee ee tans ee canon cnn nano n cnc naar nccnnnnns 34 Table 22 sPSU S Power Supply LED 35 Table 23 4700 Environmental Specifications 0 cccccecsceceeceeceeeeeeeeeeeeeeeceeeeeseae
43. g for Installation 1 Refer to Power Data and Calculating Heat Output in BTUs for 4700 power and heat ratings To prevent a loss of input power verify that the total maximum load on the circuit A supplying power to the power supply is within the current ratings of the wiring and breakers 5 3 Rack Requirements The 19U high 4700 chassis is mounted in a standard 19 rack with the fabric boards facing the front of rack and the Line Board InfiniBand ports and sMB CM facing the rear of the rack The rack should support the mechanical and environmental characteristics of a fully populated 4700 chassis as listed in Environmental Specifications The rack mounting holes conform to the IEA 310 standard for 19 inch racks Precautions should also be taken to guarantee proper ventilation and ambient temperature is maintained for air intake at the front of the chassis and exhaust in the rack The HyperScale cable guide requires an additional 1U of rack space making the total height 20U instead of 19U b 5 4 Clearance Requirements You should be certain to provide sufficient clearance around the racks to install and maintain the Grid Director 4700 Clearance specifications are detailed in Chapter 3 on page 27 5 5 Site Environment Specifications The system can continue to operate within specified environmental ranges however a measurement that approaches the minimum or maximum of a range indicates a potential problem You can maintain
44. he Management Board Power Green OFF indicates a problem in power supply to the board HW Only ON indicates that all board voltage levels are operating normally Info Blue General purpose LED for management use User only OFF No attention required Blinking Chassis attention required PSU OK Green OFF N A AC DC power supply fault ON System Power OK indication Indicates that at least one of the main AC DC Blinking System P bl i E power supplies has an AC or DC fault Eth 1 and 2 Two LEDs Link Green and Activity Orange Eth 1 and 2 3 15 5 sFU 40H Horizontal Fan Unit LEDs Table 21 sFU 40H Horizontal Fan Unit LEDs Indication LED Description Remarks Temp Orange ON Indicates an over temperature fault on the chassis OFF Indicates normal temperature levels sFU 40V Orange ON Vertical fan unit fault indication Indicates that at least one of the five fans in the vertical fan unit has failed OFF All fans in the vertical fan unit are OK sFU 40H Orange ON Horizontal fan unit fault indication Indicates that at least one of the eight fans in the horizontal fan unit has failed OFF All fans in the horizontal fan unit are OK PS Orange AC DC power supply fault indication ON Indicates that at least one of the main AC DC power supplies has an AC or DC fault OFF All PSUs are OK La J T Mella
45. he cable management perspective where on the rack the 4700 is installed 2 Install the cabling guide brackets in parallel with the 4700 chassis so that each Line Board lines up with its corresponding hanger on the bracket 78 T Mellanox Grid Director 4700 Installation Manual 9 1 2 InfiniBand Cabling on Either Side of the Chassis for Line Cables It is recommended to dress the cables connected to the InfiniBand connectors of each line board into two groups left and right and fasten them down gently after connectivity has been verified The cables from the same half of the line boards are bundled on the corresponding hangers and will be routed down along the rack side wall Cables should be arranged so that each line board s cables do not interfere with the neighbor line board insertion or removal gt To connect an InfiniBand cable to an sLB 4018 Line Board 1 Route the cable to the nearest cabling guide bracket hanger position 2 Position the InfiniBand cable connector with the pulling tab at top and push the connector in place 3 Verify that the cable is inserted properly into the connector by gently tugging the cable verify that the Link State green colored LED lights and does not flash indicating a good physical connection 4 Gently fasten in place with Velcro tie or tie wraps e Avoid using tie wrap guns or similar tools do not fasten the InfiniBand cable too tightly as this may cause irreversible d
46. imum Cable Length 98 4 30 CBL 00105 164 50 CBL 00106 328 1 100 CBL 00115 525 150 CBL 00212 984 2 300 CBL 00200 A 3 1GbE Ports Management Use modular RJ 45 straight through UTP cables to connect the 10 100 1000 Fast Ethernet ports to external switches routers that relay management traffic Use modular RJ 45 cross connect cables to connect to end systems The following figures show straight through cables and cross connect cables Figure 39 Straight through Cables Pin number Pin number 1 z 1 3 q 3 ds 4 74 7 8 Fe e 8 Figure 40 Cross connect Cables Pin number Pin number 1 lt 3 A OOS SSS SSS The 10 100 1000 Fast Ethernet port supports a RJ 45 connector 88
47. ing 16 screws 8 for each side and adequate clip nuts as shown 5 To stabilize the chassis for transport secure the 4700 chassis onto the rail by using three 8 32 screws 3 8 tooth washers and flat washers insert them into the oval holes and the threaded holes in the chassis side walls see Figure 6 10 Rack Mounting the 4700 Chassis 1 Once the rack is installed on the premises these screws should be taken away so that you can remove the chassis in case the sides of the rack are not accessible Figure 23 Securing the 4700 Chassis for Transport Regular Configuration For transportation use A for regular configuration Figure 24 Securing the 4700 Chassis for Transport HyperScale Configuration For transportation use B for HyperScale configuration 6 4 Mounting the HyperScale Chassis in a Rack The installation kit includes e Rack mount bracket for recessed installation supporting cabling from both sides e HyperScale cabling guide bracket for CXP cables e Screws you will need to save the screws from the original brackets use screws from the kit 62 T Mellanox Grid Director 4700 Installation Manual gt To mount the HyperScale 4700 chassis into the rack 1 2 Install the rail kit as described in Assembling and Mounting the Rails on page 54 Swap the standard 19 in rack mount bracket with the 19 in long bracket provided in the kit using the screws from the unmount
48. ing can be performed from any of these locations Figure 16 Grounding Holes for 4700 Grounding hole gt 4700 Front Grounding hole Install the grounding stud before you mount the Grid Director 4700 chassis onto the rack IX 50 Mellanox Grid Director 4700 Installation Manual gt To install the Grounding Stud 1 Insert a 10 32 5 8 screw through the Grounding hole rear to front as shown REAR 1 10 32 5 8 Screw Grounding Hole EN Safety Warning Label 8 Ground Label to Ground E H e Spring Washer Ei 6 Hex Nut 10 D i Spring Washer 5 Grounding Cable L Spring Washer 4 CG Ground Hex Nut 10 Add a spring washer onto the 10 32 5 8 thread this time front to rear Add a nut 10 Add another spring washer R A WwW N Insert the 6 AWG grounding cable Connect the other end of the grounding cable to a grounding point at your site 6 Add another spring washer Make sure that the grounding cable is tightly secured between the two spring washers 7 Secure everything with a grounding nut 10 8 Apply the Ground label under the grounding nut 9 Apply the Safety Warning label on the side of the chassis near the grounding stud 51 J Preparing for Installation 1 5 7 Power Requirements Power requirements are useful for planning the power distribution system needed to support the Grid Director 4700 Heat dissipation is an importan
49. ing the Grid Director 4700 chassis 4 1 Package Contents The Grid Director 4700 is supplied in a heavy duty tri wall cardboard box secured with packing straps For information about the various 4700 configurations and items shipped with the chassis refer to Chapter 1 on page 18 The basic 4700 shipping contents are as follows e Grid Director 4700 e Horizontal fan unit sFU 40H e Vertical fan unit sFU 40V e Management board SsMB CM e Two power supply units sPSU S The Grid Director 4700 chassis is shipped with a horizontal fan unit sFU 40H and a vertical fan unit sFU 40V installed at the front of the chassis and two power supplies at the rear of the chassis Additional components include e Rail kit for rack installation e Cabling guide brackets e One power cord per power supply e One console cable e A grounding kit e Mellanox Getting Started Short Guide e Two product CDs one containing the software one containing the documentation Verify that the package contains the items detailed in the packing list before you begin to install the Grid Director 4700 Check for loose parts or any visible damage to the contents If any damaged or missing parts are detected notify your local Mellanox representative 42 Mellanox Grid Director 4700 Installation Manual Part Number Description Grid Director 4700 Top Foam Cable Management Mounting Rails Left and Right Fo
50. ink up external QSFP port tivity 36 Green LEDs e Amber ON indicates logical link up STE HW only The LED states are as follows e OFF no link detected e ON the port has a link e Blinking Green indicates errors Amber indicates data activity Power Green ON indicates that all board voltage levels are HW only operating normally OFF indicates a problem in power supply to the board Info Blue General purpose LED for management use User only OFF No attention required Blinking Chassis attention required Card Ready OFF Card not Ready Green ON Card ready Blinking Slowly Card loading Blinking Fast Card problem Management Board SMB CM LEDs Table 20 Management Board S MB CM LEDs Indication LED Description Remarks Fans Green OFF All fan drawers are OK Two fan drawers 33 j 4700 Specifications Indication LED Description Remarks ON Vertical and Horizontal fan unit fault indication Indicates that at least one of the 13 fans has failed Temp Orange OFF Indicates normal temperature levels Temp Orange ON Indicates an over temperature fault on the chassis FM Active Green OFF Fabric Manager is not active ON Fabric Manager running on the Management Board Blink Fabric Manager is standby on the Management Board DM Active Green OFF Device Manager is not active ON Device Manager running on t
51. ion Manual 7 Attach the front side of the rail to the rack s front vertical side rail with 2 screws for each side and the rear side of the rail to the rack s rear vertical side rail with 2 screws for each side IMPORTANT If you do not tighten the screws the chassis cannot be mounted safely in the rack 8 Repeat steps to 3 to assemble the other rail and mount it on the other side of the chassis the rails are now ready for chassis installation IMPORTANT When mounting the rail ensure box to box vertical alignment on all four rack rails See that the bottom side of the rail is aligned with the 1U marking on the rack rail 6 2 3 Rail Kit Assembly Procedure This section presents the procedure for assembling and mounting the rail kit Each step is accompanied by a figure showing the components as they appear aligned and then assembled one by one In addition eight screw sets are included comprising one screw one nut and two washers gt To assemble the rail 1 Use 8 32 5 8 screws and flat washers and spring provided with the kit to assemble the rail positioning the insert in front of the bracket as shown below 1 For each of left and right take a bracket and attach an insert the two inserts are identical At this stage position the two parts to fit the distance between the rack rails of each side Make sure that you are using the right bracket for the right rail right side when facing the front of the ra
52. laced without interfering with traffic of operation of adjacent Line Boards even when the 4700 is fully populated with Line Boards It is permissible to connect and disconnect InfiniBand cables while the 4700 is powered on gt To replace a Line Board 1 Carefully remove the InfiniBand cable connections from the Line Board connectors located at the rear side of the chassis It is recommended to gently secure the removed cables using a Velcro tie neatly placing the cables on the outside of cable guide bracket rails 2 Remove a Line Board as described in Inserting Extracting Boards 3 Insert a new Line Board as described in Inserting Extracting Boards 4 Replace the cable connections 9 5 Connecting to the Management Ports The Mellanox 4700 has management ports located at top back of the 4700 This includes two interfaces a serial interface for local console access and a 10 100 1G Ethernet interface for network access The serial port can be used to connect to a PC running terminal emulation software for 4700 management tasks The Ethernet port is connected to an IP network and is used to run a CLI via Telnet It also includes a USB type A connection One USB 2 0 Host interface allows you to increase the CPU flash memory by a connection to a standard Disk On Key device Through the management network you can manage the 4700 using a Command Line Interface CLI Refer to the relevant User Manual for more information If the c
53. le Configuration per Chassis ocooccconnncnnncnnnncccconccnnnnnnnnannnnncc cnn cnn nn nn nana nnnnccnns 18 2 1 2 Specifications SUMMALY cooooocnnnccnccononccnnonnnononnnano cc nn cnn nn nn nn nana carr annn rana nan 19 2 2 4700 Configuration and Part Numbers ssssessseesseessesssessrsenesertenntetnttnttnnntnnnnnnsnnnnnnnnnnnnnt 19 23 S Witch Internal Topology earnen aea a ia a aa aaa Aaa aa a ae aera a Eaa ia 20 2 3 1 HyperScale Fabric BOardS ooocccciccconococonccononccononanonannnncc narran ran cc 20 2 3 2 Switching Topologies A 21 23 3 HyperScale M Configurations oooonnicccnnnncccnnnnocccnnnnenccnnnnarncnn nn cnn rc 23 3 4700 Specifications iii E 24 KN elen e 24 3 2 4700 Fabric Board sFB 4700 Gpechflcatons rra 24 3 3 4700 Line Board SLB 4018 Specifications oonoccinnininnnnninnccnncccnnncnnccnnccrnnrnncrnnnnrcnnn 24 3 4 4700 Management Board sMB CM Specifications ooooocncccininnnnnnnnnncnaccccnncnnncrnnnnccnncn 25 3 5 Chassis Slot NuMberinQ ooooinncnnnninnnnnnnccnnncccnrcnnnnrn nana cc 25 A elt UE ne EE 26 3 7 Module Configuration Rules oooocncccnnininnnnnnncccnocccnnornn narran cnc 27 3 8 Clearance REQUIFeMeNtS umi E 27 3 9 Power Dala iii ll ted ds 27 3 9 1 Type of Power Cord on the Power Supply oooocinocccnncccconccononanonacccancnnnnncnano na nana cnn 27 3 9 2 Power Requirement S aeriene aiei nc 27 3 9 3 Power Characteristics a 28 3 9 4 Power Data per Module oooomocccnonnncccinnncccconocccn
54. le Management Right Delivered with the KIT 00023 switch Cable Management Left KIT 00024 Cabling guide bracket kit for HyperScale Fabric Board It Set must be ordered per KIT 00022 includes the following parts chassis e 2x 19 Long bracket MEC 00576 e Cable management for Spine in 648 configuration MEC 00593 e Screws for assembly 380J30011 2 3 Switch Internal Topology High performance computing clusters typically utilize Clos networks sometimes referred to as Fat Tree or Constant Bisectional Bandwidth CBB networks to construct large node count non blocking configurations A Clos network is a switch topology in which integrated non blocking switch chips crossbars with a relatively low number of ports are used to build a non blocking switch topology supporting a much larger number of endpoints The Grid Director 4700 features an internal Clos network with Line Boards as the edge leaf switches and Fabric Boards as the core spine switches Each Line Board switch chip is connected to each switch chip on the Fabric Boards through one or more InfiniBand connections The path from one external port to another node may require three switch hops e Hop at a Line Board e Hop 2 at a Fabric Board e Hop 3 at another Line Board 2 3 1 HyperScale Fabric Boards HyperScale is an interconnect technology enabling large scale fabrics at reduced latency cost and complexity Mellanox HyperScale fabric cards ha
55. led on the 4700 rack s front rail in a horizontal position in parallel and between two HyperScale switches gt To install a cabling guide bracket for HyperScale fabric boards Install the cabling guide bracket at the front of the rack under the rail kit on the rack itself between the two 4700 using four screws to secure the bracket in place Figure 36 Cable Management for Switches with HyperScale Fabric Boards ta a e Connecting the InfiniBand Cable between Fabric Boards gt To connect a cable between Fabric Boards 1 Ina Fabric Board installed at the top chassis position the InfiniBand cable connector with the pulling tab at top and push the connector in place 2 Route the cable straight down to the nearest cabling guide bracket hanger position Y e y JA A y j R SE I System Cabling 3 Select a Fabric board in the bottom chassis and position the InfiniBand cable connector with the pulling tab at the right of the cable and push the connector in place 4 Verify that the cable is inserted properly into the connector by gently tugging the cable verify that the Link State green colored LED lights and does not flash indicating a good physical connection 5 Gently fasten all cables in place with Velcro tie or tie wraps e Avoid using tie wrap guns or similar tools do not fasten the InfiniBand cable too tightly as this A may cause irreversible damage to the cables Tie wraps and zip ties c
56. llips screwdriver 1 Utility knife 1 Measuring tape 1 Crimping tool 1 Wire stripping tool 1 Digital Voltmeter DVM Floating nuts Phillips screws 5 2 Site Planning It is essential that you plan the proper location and layout of your equipment rack Equipment placed too close together or in an inadequately ventilated area can cause an increase in temperature and inadequate cooling conditions In addition be sure to place the system panels so that they are accessible and easy to reach system panels inaccessible and difficult to maintain Plan your site configuration and prepare your site before installation to ensure normal operation and avoid unnecessary maintenance Refer to Environmental Specifications for the operational and non operational environmental site requirements for the Grid Director 4700 The system can continue to operate within specified environmental ranges however a measurement that approaches the minimum or maximum of a range indicates a potential problem You can maintain normal operation by anticipating and correcting environmental conditions before they exceed the maximum operating range Verify the site power for the type of device you are installing Power requirements are useful for planning the power distribution system needed to support the Grid Director 4700 Heat dissipation is an important consideration for sizing the air conditioning requirements for an installation 47 j Preparin
57. m modules and provide an alarm indicator in the event of a high temperature condition InfiniBand Switch Silicon Firmware Each Line Board and Fabric Board incorporates InfiniBand 36 port switch ASIC devices with locally attached memory containing firmware that controls basic functionality of the devices The firmware controls low level link and physical layers functionality and may be upgraded via Ethernet or RS 232 on the Management Board or in band via any of the InfiniBand Line Board ports InfiniBand products are RoHS 6 compliant Management The Grid Director 4700 provides a command line interface CLI interface for flexible access the switch comprehensive management capabilities Device Management DM e Control a device e g a switch or a device cluster e Remote management upgrades logs alarms Management Board The Grid Director supports either one or two hot swappable Management Boards Power Supplies sPSU S Mellanox Grid Director switches support multiple power supply modules sPSU S configured in redundant hot swap current sharing mode for N 1 or N N redundancy depending on the configuration and suitable for 100 240 VAC mains voltage auto sense The Grid Director supports up to six power supply modules 1 3 2 Reset Buttons In the rare event of a 4700 system lock up the Reset button can be used to reset the system The system has two Reset buttons one is located on the front panel of the sFU 40H fan assem
58. mmodate racks of differing depth The 4700 rail kit is adjustable to accommodate a distance of depths of 24 60 cm to 40 100 cm between the rack s EIA rails In addition the chassis is delivered with cable management cabling guide brackets The HyperScale cable guide requires an additional 1U of rack space making the total height 20U instead of 19U The mounting process consists of the following stages 1 Standard chassis installation up to 324 port configuration Assembling and mounting the rail Mounting the chassis on the rail in a rack Installing the cable management 2 HyperScale chassis installation up to 648 port configuration Assembling and mounting the rail Remove the standard 19 brackets keep the screws for installation of the 19 brackets Install the 19 long brackets part of P N KIT 000222 Slide and mount the chassis into the rack Installing the cable management Install the HyperScale cabling guide bracket Note that you should count 5 slots to the left of the bracket prior to installing it Refer to Chapter 9 on page 77 for more information Because of the 4700 chassis weight at least two people are required to mount this unit in A a rack 6 2 Assembling and Mounting the Rails This section describes how to assemble and mount the rails e Rail kit and mounting components e Overview of the assembling and mounting procedure e Assembling and mounting procedure in detail the st
59. ne if the power supplies are operational If you determine that the power supplies are functioning normally and that the fans are faulty contact a customer service representative If the 4700 fan does not function properly at initial startup there are no installation adjustments that you can make contact a customer service representative 2 Check the power supply LEDs on the rear panel The power LEDs illuminate immediately upon the connection of power to the 4700 If the LEDs are not on the power supplies may need to be replaced 3 Check the Link LEDs on the Line Boards front panel for the various ports In the event of a system lock up in the Mellanox 4700 the reset button can be used to reset and reboot the system The system has two Reset buttons one is located on the front panel of the sFU 40H horizontal fan unit and one on the sMB CM management board panel To reset the switch push the button using a thick wire or tip of a pen until the system reboots Remove the wire or pen immediately afterwards 4 Verify that the PC terminal emulation program is set correctly and that the PC is connected properly to the Console management port if the boot information and CLI prompt are not displayed 76 T Mellanox Grid Director 4700 Installation Manual H System Cabling 9 1 Cabling Guide Brackets Installation on Either Sides of the Chassis Line Boards Cables The 4700 cabling guide brackets are installed at the rear of th
60. nnnnnccncnnnn cnn nn nr 28 3 10 Calculating Heat Output in BTUe nn 29 3 11 Maximum Heat Ditsempaton narnia a 30 312 e Wisteria stateless E 30 3 13 Physical Characteristics oooonnnniicinnininnnnnnnnnnnnncncrccnnnrn nr 30 3 j Contents 1 3 14 Weights and RR tu Ee 20 315 LED MALNES ee Eed ege iaa 31 3 15 11 sLB 4018 Line Board LEDS cerea eE A EAA 32 3 15 2 Fabric Board SFB 4700 LEDS A 32 3 15 3 HyperScale Fabric Board sFB 4700 2X LEDS 0ooconccccnoccconoccconcccconncnnoncnnnannnanccnnns 33 3 15 4 Management Board SMB CM LEDS 0oooncccinnccnnncccconcccnonnnnnoncnanoccnnnn cnn nn nana cnnancccns 33 3 15 5 sFU 40H Horizontal Fan Unit LEDS A 34 3 15 6 sPSU S Power Supply LEDS oooncccnnccnnoccconocnnonoccconccnnnnnnn acc nac c cnn cnn nn nc anna cnn nnccns 35 3 16 Environmental Specifications ooomcccinnidinnnnnnnnccnnnccnnrannccnnncccrnrnc nn 35 3 17 ACOUSTIC Data E 35 3 18 Reliability MTBE aio rad 35 3 19 GertificationS iii tllav initia cies Meee laden ies 36 3 20 Declara A A tege deelen oh ET T eebe diene EE Eed 37 3 20 1 Declaration of Conformity o ooocnnnnninnccnncccncocnnnnonnnnnoccnnncn nan nn nano rana nn cnn annr rana ranma 37 3 20 2 Hazardous Substances RoHS 6 Compliance Declaration oooooconncnnnnnnnnnnnnnnccc 37 8221 A aa en ee Le T E E 38 Le e Et ee EE TEE 38 3 23 KCC Korea Certification ooononmicniididnn nnonnnnnnnnccnoncncrnccnnrc rr 38 Ee ele EE 38 3 25 Shock and Vibration Test Hepor
61. nox Grid Director 4700 Installation Manual 3 15 6 sPSU S Power Supply LEDs Table 22 sPSU S Power Supply LEDs Indication Description DC ON Green ON Indicates that the DC power source is present AC ON Green ON Indicates that the AC power source is present 3 16 Environmental Specifications Table 23 4700 Environmental Specifications Ambient Operating Temperature 32 to 113 F 0 to 45 C Operating Humidity 15 to 80 non condensing Operating Altitude 0 to 9843 ft 3000m Storage Temperature 13 to 158 F 25 to 70 C Storage Humidity 5 to 90 non condensing Storage Altitude 0 to 15 000 ft 4570m 3 17 Acoustic Data Acoustic noise is reported in accordance with ISO 7779 ECMA 74 standard 3 18 Reliability MTBF The Mean Time between Failures MTBF for individual switch components is shown in the following tables These figures indicate the mean elapsed time until a single component fails in the module whether or not that failure impacts the operation of the switch Note that in many cases redundant modules can be configured which dramatically increases the overall system Mean Time between Critical Failures MTBCF MTBF per 4700 model types e Model A The system configuration is a fully populated chassis with 18 line cards 9 fabric cards 3 power supplies 1 Management Board and 2 fan modules Horizontal and Vertical drawers e Model B The system configuration is a fully p
62. oards 7 1 Chassis Board Configuration Boards must be installed in their designated slot to enable their full functionality All boards are hot swappable and therefore can be inserted and extracted without turning off electrical power on the 4700 chassis The 4700 chassis is factory pre assembled with fan units Additional boards and modules are shipped according to the system configuration and are added during installation Refer to System Configurations on page 18 for a list of the possible chassis configurations 7 Every slot that is not populated with a module must be covered with a blank panel for proper cooling gt 7 1 1 4700 Chassis The front panel of a Grid Director 4700 includes e Nine slots for Fabric Boards sFB e sFU 40H horizontal fan unit e sFU 40V vertical fan unit The rear panel of a Grid Director 4700 includes e 18 slots which can be filled with line boards 324 ports e Two slots for management boards SMB e Six bays for power supply modules the power supplies can be installed in any of the bays without restriction See Mellanox Grid Director 4700 Overview on page 10 for illustrations of front and rear panels of a fully populated 4700 chassis 7 2 Inserting Extracting Boards When handling electronic boards wear grounding wrist straps to avoid ESD damage to A the card Do not touch the Midplane with your hand or any metal tool or you could shock yourself Fabric boards management boards
63. onfiguration calls for out of band management of the 4700 either using a serial connection or over Ethernet the management network must be connected to the corresponding Loss J T Mellanox Grid Director 4700 Installation Manual management ports on the rear of the Chassis Both types of management network can be connected simultaneously gt To connect to the Management port Ethernet Interface 1 Connect the end of the management cable with the single RJ 45 connector to the Management port on the 4700 2 Connect the end of the cable with Ethernet connector to the appropriate devices network gt To connect to the Management port Serial Interface 1 Connect the end of the management cable with the single RJ 45 connector to the Management port on the 4700 2 Connect the end of the cable with the DB 9 serial connector to a PC or a console The PC console must support VT100 terminal emulation and should be equipped with terminal emulation software such as HyperTerminal or ProComm Plus 3 Make sure that your PC console terminal emulation software is configured to communicate with the 4700 via hardware flow control Configure the baud rate and character format of the PC terminal emulation program to match these RS 232 management port default characteristics 38 400 baud Eight data bits One stop bit No parity Flow Control None System Cabling Appendix A Cabling Information and Specifications Thi
64. opulated chassis with 18 line cards 9 double fabric cards 4 power supplies 1 Management Board and 2 fan modules Horizontal and Vertical drawers Table 24 Reliability MTBF Specifications 4700 QDR Director FPMH MTBF Hours Model A 66 308 15 081 Model B 75 431 13 257 35 j 4700 Specifications Table 25 MTBF per Module Data Chassis Module Description MTBF Hours sFB 4700 9 QDR port connections fabric board 519 426 sFB 4700 2X 9 QDR port connections x2 fabric board 345 941 sLB 4018 sLB 4018 18 QSFP ports per sLB 4018 419 866 sMB CM Management Management board 441 273 Board sFU 40H Horizontal Fan Horizontal fan unit 708 754 Unit FRU sFU 40V Vertical Fan Unit Vertical fan unit 913 145 FRU sPSU S Power supply 2 303 086 3 19 Certifications e cMETus UL 60950 1 07 Second Ed e CSA C22 2 No 60950 1 07 Second Ed e cTUVus UL60950 1 07 Second Ed e S Mark Argentina e CB Certificate and report according to IEC 60950 1 Second Ed e KCC Korea e Gost R e CE according to EN 60950 1 06 Second Ed e S Mark EMC Certifications e FCC Part 15 Subpart B Class A and testing to ANSI 63 4 e Industry Canada ICES 003 CAN CSA CEI IEC CISPR 22 02 e CE EN55022 1998 A1 2000 A2 2003 EN55024 1998 A1 2001 A2 2003 EN 61000 3 2 00 A2 05 Harmonic current emissions EN 61000 4 2 1995 Al 98 A2 2001 EN 61000 4 3 02 A1 2002 EN 61000 4 4 1995 Al 01 A
65. over temperature sensing e Data throughput QDR 2 88 Tbps 24 T Mellanox Grid Director 4700 Installation Manual 3 4 3 5 e Port to port Latency Under 100 nanoseconds e Data Virtual Lanes 8 e Management Virtual Lanes 1 e MTU 4096 Bytes max e Maximum hops 1 hop between ports on the same board 3 hops between different Line Boards in the same chassis 4700 Management Board sMB CM Specifications Two hot swappable Management Boards each including on board CPU LEDs Refer to LED Indicators on page 31 Connectors 10 100 1000 Ethernet RJ 45 EIA TIA 232 Console RJ 45 Reset button Resets the Management Board or the entire chassis Chassis Slot Numbering Slot numbering at the front of the chassis for Management Board and Fabric Board is from left to right indication on the horizontal fan unit sFU 40H Figure 9 sFU 40H Horizontal Fan Unit sFB1 sFB2 sFB3 sFB4 sFB5 sFU 40V sFB6 sFB7 sFB8 sFB9 Slot numbering at the rear of the chassis for Line Boards is from top to bottom indication on the cable guide brackets 25 4700 Specifications Figure 10 4700 Line Board Slot Numbering UR US en en en en e 10 11 12 13 14 15 16 17 18 Left Cabling Right Cabling Guide Guide For specific instructions refer to System Cabling on page 77 3 6 Torque Settings Table 5 Torque Settings for Module Assembly Assembly Torq
66. per system ventilation The 4700 fan units can be replaced within 5 minutes without interruption to the operation of the chassis and without disconnecting it from power 72 T Mellanox Grid Director 4700 Installation Manual 7 7 1 Vertical Fan Unit sFU 40V The sFU 40V vertical fan unit shown in the following figure is part of the 4700 air cooling system it provides cooling of the Line Boards and the management boards It includes 5 fans with fan speed control fan speed is dynamically controlled as a function of temperature Figure 33 sFU 40V Vertical Fan Unit Front Panel A Never remove the sFU 40V and sFU 40H at the same time at least one fan unit must be installed at all times gt To replace the sFU 40V 1 Loosen the four screws that secure the sFU 40V to the chassis 2 Remove the sFU 40V by sliding it out of the chassis 73 j Installing System Boards 3 Within five minutes replace with a new sFU 40V and place and tighten the four screws 4 Listen for proper sFU 40V fan operation 7 7 2 Horizontal Fan Unit sFU 40H The sFU 40H horizontal fan unit shown in the following figure is part of the 4700 air cooling system it provides cooling of the Fabric Boards It includes eight fans with fan speed control fan speed is dynamically controlled as a function of temperature LED indications provide an onsite report regarding the operating conditions of the air cooling system modules and
67. provide an alarm indicator in the event of a high temperature condition sue sFU 40H A 0 0 or 980980 a gt To replace the sFU 40H horizontal fan unit 1 Loosen the four screws that secure the sFU 40H horizontal fan unit to the chassis 2 Remove the sFU 40H horizontal fan unit by sliding it out of the chassis 3 Replace with a new sFU 40H horizontal fan unit and tighten the four screws within five minutes 4 Listen for proper fan operation and check the LEDs 74 T Mellanox Grid Director 4700 Installation Manual 8 1 8 2 8 3 8 4 8 4 1 Verifying Installation Verifying installation of the 4700 consists of making sure that it starts up properly and that the network connections are operational Verifying Start up Operations gt To verify that the 4700 starts up properly 1 Observe the power supply LEDs on the fans module front panel to make sure the power supplies are operable 2 Listen and check for airflow to make sure the fan assembly is operating 3 Observe system LEDs to verify that the system is running and operational 4 If any of these conditions are not met refer to Troubleshooting to isolate and if possible resolve the problem Verifying Network Connections Verifying the network connections consists of making sure that the following network ports are operational e InfiniBand e 10 100 1G Ethernet management Where to Go Next Once you have verified that the 4700
68. re 4700 Rack Height 19U HyperScale Rack Height 20U Rack Width 19 Switch Type Director Backplane Bandwidth 51 8 Tbps Line Board Slots 18 Fabric Board Slots 9 Max of InfiniBand Ports 324 40 Gbps InfiniBand Ports Optional 648 40 Gbps InfiniBand Ports Interface Type QSFP 40 Gbps QDR QSFP ports with support for optical adapters and cables Redundant Components Power Supplies sPSU S Management Boards sMB CM Embedded Chassis Management CLI Yes 2 2 4700 Configuration and Part Numbers The following table details the basic configuration of the 4700 chassis Table 3 4700 Basic Configuration Product Description Product Module P N Name Grid Director 4700 18 Slot QDR Basic Configuration Grid Director 4700 VLT 30040 sFB 4700 QDR Fabric Board sFB 4700 VLT 30041 sFB 4700 2X QDR Fabric Board sFB 4700 2X VLT 30042 19 j l System Configurations Product Description Product Module P N Name sLB 4018 18 Port QDR Line Board sLB 4018 VLT 30043 sMB CM Chassis Management Board sMB CM VLT 30044 sPSU S 1 4KW AC Power Supply Module sPSU S VLT 30046 sFU 40H Horizontal Fan Unit sFU 40H VLT 30047 F sFU 40V Vertical Fan Unit sFU 40V VLT 30048 F The above parts can be ordered separately The following table provides the list of additional spare parts and accessories Table 4 Spare Parts and Accessories Description Comment P N Cab
69. redundancies The Grid Director 4700 supports up to six load sharing 1400W power supplies with separate AC power connections Each power supply must be powered from separate sockets with circuit breaker 16A for Europe or 20A for North America The amount of required power supplies per system is set according to system configuration When the Grid Director 4700 is fully configured it supports N 1 redundancy as shown in the example below 52 Mellanox Grid Director 4700 Installation Manual Figure 17 4700 Fully Configured N 1 Redundancy AC Power Source 110V 220V Internal Full Configuration The following table describes the Grid Director 4700 N 1 N N and N M redundancies Table 32 4700 QDR N 1 N N Redundancy Configuration Modules Included Non redundant N 1 N N or N M power supplies 18 ports QSFP 9 sFB 4700 2 3 2 1 4 2 2 RES 1 sLB 4018 1 sFU 40H 1 sFU 40V 1 sMB CM 2 sPSU S 162 ports QSFP 9 sFB 4700 Z 3 2 1 4 2 2 copper 9 sLB 4018 1 sFU 40H 1 sFU 40V 1 sMB CM 2 sPSU S 324 ports QSFP 9 sFB 4700 3 4 3 1 6 3 3 copper 18 sLB 4018 1 sFU 40H 1 sFU 40V 2 sMB CM 3 sPSU S 53 j Rack Mounting the 4700 Chassis 1 6 Rack Mounting the 4700 Chassis 6 1 Mounting Process Summary The 4700 chassis is mounted in a standard 19 rack with the Fabric Boards facing the front of the rack The 4700 is provided with an adjustable rail kit to acco
70. rops 3 26 Export Information Composite Theoretical Performance CTP is a calculation measured in MTOPS millions of theoretical operations per second The 460EX 460GT contain a 440H6 core not a 460 core it is called 460 to distinguish it as a 90 nm core The CTP in MTOPS for the PPC440 core 17 18 F where F is the CPU clock rate in MHz Refer to the following table accordingly Table 29 Export Information CTP value for the PPC440 core 460EX 460GT 567 MTOPS 600MHz CTP value for the PPC440 core 460EX 460GT 944 MTOPS 1000MHz ECCN 5A991C 3 27 Shipping Restrictions None 0 J Mellanox Grid Director 4700 Installation Manual 3 28 3 29 41 j Center of Gravity Data This section provides the Grid Director 4700 center of gravity specifications The measurement origin starts at the front lower left corner of the chassis Figure 12 Center of Gravity Data Center of gravity for minimum typical and maximum configurations are shown in the following table Table 30 Center of Gravity Data 4700 Width X mm Depth Y mm Height Z mm Empty Chassis 220 250 345 Basic Configuration 220 270 330 Full Configuration 220 300 320 Statement of Volatility Refer to the Grid Director 4700 Statement Of Volatility document Packing and Unpacking 4 Packing and Unpacking This chapter provides step by step instructions for packing and unpack
71. s appendix provides supplemental information and specifications A 1 QSFP Cable QSFP Quad Small Form factor Pluggable Key Features e Small size high density hot pluggable connector with four transmit and four receive channels e Supports both types of cables Copper active or passive cable Copper passive cable equipped with two simple passive connectors and a standard copper cable Copper active cable equipped with an active device to optimize signal quality Fibre optic cable delivered as either Fibre cable with two separate modules Active optic cable where cable and optic adaptors are delivered as a single unit permanently attached cable e Data rate 2 5 Gbps SDR 5 Gbps DDR and up to 10 Gbps QDR per channel e QSFP optic modules use standard MPO parallel 12 fibre jacketed ribbon optical connector e Provide manageability discovery and identification capability such as vendor identification module part number serial number cable type using low speed two wire LC interface e Active cables Integrates digital diagnostic capability to monitor link performance e Latching mechanism e Physical dimensions Figure 38 QSFP Connector Dimensions 72mm max A 18 35mm 8 5mm QSFP to CX4 cables are available to connect 4700 equipped with QSFP with legacy DDR SDR systems equipped with CX4 connectors 86 T Mellanox Grid Director 4700 Installation Manual QSFP Cable Overview Every 47
72. seeaaeseeeeeseaeeeseaeeseeeenaes 35 Table 24 Reliability MTBF Specifications cc cccecceesceceeeeeceeeeeeeaeeeeeeeceeeeesaaeeeeaaeseeeeeseaeeesuaeeseeeeeaes 35 Table 25 MTBF per Module Dataa trceeccsiasees sa tidad 36 Table 26 Oscillator SpeedS AA 38 Table 27 Vibrations EE 39 Fable 28 Free Falls ai A Me ee ie ee et 40 IES 40 Table 30 Center of Gravity Data ricimosiro tratada taras 41 Table 31 Required Tools and Materials 0 ccccccccccceecceceeeeeeeeeceaeeeeeeeseeceaaeaeeeeeeeseeenaeeeeeeeeeseesenaeess 47 Table 32 4700 QDR N 1 N N Redundancy ecccocccccoccconoconononononccnnnncno nn nn nan n cnn nn nan nn 53 Table 33 4X InfiniBand Port Cabling Specifications narran cnn 87 7 j Contents 1 Preface About this Manual This manual provides installation instructions for the Mellanox Grid Director 4700 40Gb s InfiniBand Switch the product specifications unpacking and installation information unit power up and initiation and troubleshooting procedures gt Refer to the official and latest product release notes for last minute updates al Technical support may be obtained directly from e Your regional distributor from whom this product was ordered e Your OEM customer representative For further information and assistance go to http www mellanox com content pages php pg support_index Audience The manual is intended primarily for system administrators who are authorized to install a
73. ssis 1 6 5 4700 Power Up Once the 4700 chassis is secure within the rack electrical cables from the power distribution source can be fitted in preparation for connection Power cords are supplied by Mellanox according to the system configuration one per power supply unit 3 Verify that you have the correct number of power supply modules installed gt 1 Ensure that all site power and grounding requirements have been met before connecting the chassis to a power source 2 Before turning on the power verify that all boards power supplies and fan tray modules are properly inserted 3 Connect the power supply cords to the AC connectors located at the rear of the chassis All LED indicators on the Fabric Board and Line Board briefly light up and then turn off just after power is turned on This is a normal part of the power on initialization sequence After initialization is completed the link LED indicators on the Fabric Board and Line Board are in a state representing the link status of their corresponding ports If there are not enough sPSUs installed the system won t power up or in some cases only part of the system will power up and other parts will remain unpowered Determine the minimum number of PSU required and to add at least 1 PSU as redundant N 1 or full redundant N N using the method described in Power Data per Module Lo J T Mellanox Grid Director 4700 Installation Manual 7 Installing System B
74. t Board provides one RS 232 port and one Ethernet port for management capabilities At a given time only one sMB CM Management Board can operate as a master active while the other board is in standby mode passive Refer to LED Indicators on page 31 for a comprehensive list of LEDs LED description and functionality gt To insert a Management Board into the chassis 1 Pull the ejector to a fully open position 2 Carefully seat the board into the side guide rails Slowly slide the board into the chassis until the module guide engage the chassis then the ejectors begin to engage on the chassis edge 3 Push the ejector inwards 4 Tighten the two security screws 5 Verify the following LED conditions Power LED ON Info LED OFF To extract a Management Board from the chassis Disconnect all cables from the specific board Release the security screws wn rv Pull the ejector outwards and slowly slide the board out 7 6 Installing the Power Supply Unit sPSU S Power supply modules are installed in the rear of the 4700 chassis Ideally each power supply module line cord should be connected to a different circuit for maximum fault tolerance in the event of power problems 70 T Mellanox Grid Director 4700 Installation Manual Refer to the Regulatory Compliance and Safety Reference Manual before connecting the power supplies to mains Figure 32 Front Panel of the Power Supply Module ESTI
75. t consideration for sizing the air conditioning requirements for an installation Verify the available power source at the site for the type of device you are installing Refer to Power Data on page 27 for power specifications and heat ratings To prevent a loss of input power verify that the total maximum load on the circuit A supplying power to the power supply is within the current ratings of the wiring and breakers The Grid Director 4700 requires multiple power connections using standard 3 wire AC power cords including a safety ground The input voltage is auto adjusting for 100 240 VAC 50Hz or 60Hz power connection If you do not use the power cords supplied by Mellanox the power cords should be rated for 15A or higher The Grid Director 4700 includes hot swappable power supply units sPSU S with N 1 or N N redundancy A non redundant power configuration includes the minimum number of power supplies required by the system for operation N N redundancy implies that each non redundant power supply is fully backed up by a redundant power supply which is routed to a different power source This provides for full power backup in case the first power source is disconnected N 1 implies that all the system non redundant power supplies are connected to a single power source and are backed up by one spare power supply N M is for spare power supplies N stands for required units while M stands for spares This covers 2 1 as well as 3 2
76. t the front of the chassis 324 port chassis The sFB 4700 QDR Fabric Board is installed in a standard 324 port chassis in a vertical position Each sFB 4700 has 36 internal QSFP InfiniBand ports 648 port chassis The HyperScale Fabric Board is installed in a 648 port chassis in a vertical position Each HyperScale Fabric Board has 36 internal QSFP InfiniBand ports and 12 external 12x InfiniBand ports with CXP connectors The Fabric Boards determine the total bandwidth of the switching Clos topology within the chassis Grid Director Power Backplane The power supply unit sPSU S and horizontal fan unit sFU 40H modules are connected to the system through the backplane which provides the mechanical control and electrical interface to the controllable AC DC 48V power supplies It allows electric current flow ts Introduction 1 supplies power to the Midplane via a special power harness and provides connectivity to the Midplane and the fan units sFU 40H Air Flow Cooling System The Grid Director 4700 uses front to back air flow cooling and provides full internal redundancy monitored fans Fan speed is dynamically controlled as a function of temperature The air cooling system includes a horizontal fan unit sFU 40H containing eight fans and A vertical fan unit sFU 40V containing five fans LED indicators on the horizontal fan unit sFU 40H provide an on site report on the operating conditions of the air cooling syste
77. tors outwards and slowly slide out the board 7 4 Installing the Line Board sLB 4018 Up to 18 Line Boards sLB 4018 can be installed at the rear side of the 4700 chassis There are 18 external QSFP ports all in one row and 18 internal ports on each sLB 4018 There are no upper or lower rows The InfiniBand ports are numbered from 1 18 from left to right Figure 30 Front Panel of the sLB 4018 Line Board For more information see LED Indicators on page 31 and System Cabling on page 77 When installing the line boards start at the bottom of the chassis and work your way up gt To insert a Line Board into the chassis 1 Carefully seat the board into the side guide rails Slowly slide the board into the chassis until the ejectors begin to engage on the chassis edge Do not exert excess force on the connectors when inserting the Line Board 2 Push the ejectors inwards 3 Tighten the two security screws 4 Verify the following LED conditions Power LED ON Info LED OFF Rdv LED Start to blink slowly To extract a Line Board from the chassis Disconnect all cables Release the security screws DON y Pull the ejectors outwards and slowly slide the board out 69 j Installing System Boards 7 5 Installing the Management Board SMB CM You can either install one or two sMB CMss in the back of the 4700 chassis Figure 31 Front Panel of the Management Board The Managemen
78. ts cnc 39 3 25 1 4700 Test for Transportation without a Hack 39 3 26 Export Information vit cara 40 ET leien et a a et cs he aE cbs 40 3 28 Genter of Gravity Data i208 caine een dd eed 41 3 29 Statement of Volete Ae td det 41 4 Pa kng and Unpacking ii ea edie ae ee ees 42 SN Geen 42 4 2 Unpacking the Grid Director A 700 44 4 3 Packing the Grid Director 4700 Gwitch coran rnncc cnn rn 45 59 Preparing tor InstallatiOn 00000 odos 47 5 1 Tools Required for Installation A 47 5 2 Site Planning A A va ete i la ee ae 47 539 Rack Requite ment 48 5 4 Clearance Heouirements esn tesn nesr nesr netr netrnstrnsennstnnnstnnstnnensstnnsnnnsrnnsnnnnn nne 48 5 5 Gite Environment Specifications oooncccnnnininncnnnncccnoncnncnrnna nan nnnnn cc nr nano rana rana 48 5 6 Ghassis Grounding BET 48 5 7 Power Requirements conc ci re cn ov eect 52 6 Rack Mounting the 4700 Chassis sccssscssseeesseeeeeseeeseeseseeeeeeneeseeseseaesasneeeneeeeseaeseseaeeneeeeeees 54 a re T Contents 6 1 Mounting Process SUMMaly tac dll 54 6 2 Assembling and Mounting the Rails ooooncocnnnnnccnnnnnccccnnnoccccnnnoncccnanancccnnnanccnnnnn cnc cnnnnnccnnnnna 54 6 2 1 Rail Kit and Mounting Components sssesseesseeseeseeeneesnesenrtennretnnetnnstnnntnnsnnssrnsnnt 55 6 2 2 Rail Kit Assembly COvervlew 55 6 2 3 Rail Kit Assembly Procedure ceccccceceecccceeeeeceeeeeeeaeeeeeeceeeeneeaeeeseeeaeeeeneeaeenenses 57 6 3 Mounting the 4
79. ue in Ib sLB 4018 Line Board to chassis 2 or use manual screwdriver sMB Management Board to chassis 2 or use manual screwdriver Rail kit part A to part B 10 Rail to chassis 10 26 T Mellanox Grid Director 4700 Installation Manual 3 7 3 8 3 9 3 9 1 3 9 2 Assembly Torque in Ib Fabric Board to chassis 2 or use manual screwdriver Module Configuration Rules The Grid Director 4700 supports up to two Management Boards Two Management Boards are used for redundancy failover implementation Use slot one left when only one Management Board is installed For more information refer to Installing the Management Board SMB CM The Grid Director 4700 supports any Fabric Board configuration between 1 and 9 When installing the line boards start at the bottom of the chassis For more information refer to Installing the Line Board sLB 4018 Clearance Requirements Table 6 Clearance Requirements Area Recommended Minimum Clearance Front of the rack 36 90 cm for technician access and maintenance Rear side of the rack 24 60 cm for human accessibility Sides of the rack No limits Space below the chassis No limits Space on top of the chassis No limits Power Data Type of Power Cord on the Power Supply Mellanox provides a 15 Amp cord 2 meter jumper with a universal plug for a power distribution unit PDU for each power supply The cord is detachable
80. ve double the capacity of standard fabric cards as well as external 12x ports This combination enables new modular configurations that allow scaling to large fabric sizes with the minimum amount of ports and 20 blocking islands and minimize application les Topologi tching Lowest latency of 400 nanoseconds vs 500 750 nanoseconds Ultimate application performance large non specific tuning layout cables wasted for inter switch connectivity HyperScale combines the best of Hypercube There are fewer cables to wire price simplicity cabling and Fat tree performance The following figure illustrates the Grid Director 4700 InfiniBand internal switching e Each 324 node scalable unit is fully independent failures upgrades topology Figure 5 Grid Director 4700 Switching Topology HyperScale features Sw Mellanox Grid Director 4700 Installation Manual o 2 3 2 V SH Vy v Q opami adi rie SINS K gt d Y CR VC e 4 SE s 2 CH L S VN Y N Y 21 System Configurations 4 KA AN Y SINS SN D RA N SS i SS RT ill l ai KKAAAAA KAMAAAAA a if D U AY KRAAN KAMAAAAA NX DE IE KKAAAAAA WT W NAM W ODO sc GH V J port 40Gb s InfiniBand port 40Gb s InfiniBand switch chip port 40Gb s InfiniBand switch chip 2X includes two 36 The following figure illustrates the Gri

Mellanox Grid Director 4700

Contents

Download Pdf Manuals

Related Search

Related Contents