Cisco UCS® Deployment Manual - Support
Contents
1. continued Metric Description P2_TEMP_SENS Temperature of power supply 2 CIMC XML API Metrics The metric collection script that is used to collect the preceding metrics is located on the head node at cm local apps cluster tools ext ucs ucsmetrics Plots from the data values obtained from these sensors can then be viewed in the monitoring inteface of cmgui Bright Cluster Manager File Monitoring View Bookmari Bright Cluster Manager Monitoring Ele Grid Groups LA DDP3 P1 A2 TEMP C le PVIT P2 V IA VICPBIE 0 TMP1 C rau IOUT A globalnet PA VICP81E_0_TMP2 C i es SE 3 internalnet le DDR3_P1_B2_TEMP C ipminet le DDR3_P1_D1_TEMP C SEES Power Distribution Uni E P1V1_AUX V 0 Software Images HA PVDDQ EF V defaultimage l VICP81E_0_TMP3 C 0 Node Categories E PVTT_DDR_AB V E default LA boardconsumedPower N A 0 Head Nodes E boardinputvoltage V 6GB outof 31535 GiB Ooutof 1 25 0ct 2013 16 10 00 25 Oct 2013 17 05 00 s 001 9993 j igi ucs head BA hoardinputCurrent A 25 0ct 2013 16 06 17 N A N A NIA N A Racks BA boardtrontTemp C 1 ke boardioh2Temp C A Chassis BA boardioh1Temp C node001 DIMM_A1 node001 DIMM_B1 node001 DIMM_D1 node001 DIMM_C1 Virtual SMP Nodes E boardambientTemp C 0 Nodes E boardrearTemp C E nedeo LoM i Est detay ES node001 DIMM_A1 C E node001 DIMM D1 E Cloud Nodes2. Authorization Bo Authentication PA RISER2_OUTLETTMP C PA PISEP1 OUTLETTMP C be PVDDQ AB V le DDP3 P1 D2 TEMP C be PSU2_VOUT V Err Fin IA P1_TEMP_SENS C 2 FP_TEMP_SENSOR C PA DDP3 P1 AL TEMP C PA PISERI_INLET_TMP C E PVDDQ_CD V PA PSU2 TEMP C Ep SSB V b DDP3 P1 C1 TEMP C E MLOM TEMP SENS C PA PVPLL_P1 V PA PVSA_P2 V 2 DDP3_P2_E1_TEMP C lg DDR3_P1_B1_TEMP C E P3V3_V_MOIN V l PVDDQ GH V IA PVTT DDR EF V be POV75 AUX V port DDP GH V lg DDP3 P2 G1 TEMP C LA PVCCP F n rau TEMP C LA PVSA P1 IA PSU1_VOUT V pu Spa b P3V3 BAT V MOIN V rive aa E DDP3 P2 H2 TEMP C E DDR3 P1 A2 TEMP C pm F n b VICP81E_0_TMP1 C IA PSU2_IOUT A PA VICP81E 0 TMP2 C LA DDP3 P1 B2 TEMP C E DDP3 P1 D1 TEMP C PA PIVI_AUXV Bright Computing Inc node001 RISER2_INLET_TMP node001 DDR3_P2_H1_TEMP node001 P2_TEMP_SENS 4oc 35c 24 0ct 2013 18 10 00 24 0ct 2013 18 15 00 node001 PVCCP P1 node001 PVTT P1 node001 PVTT DDR GH node001 PVTT DDR AB Liv w 09V osv 07V 24 Oct 2013 17 20 00 Figure 6 2 UCS metrics monitoring 24 Oct 2013 18 15 00 GB oo 315 35 GB s 0 99936 Est delay UCS Profiles In this appendix XML schemas are listed for the following Cisco UCS profiles HPC Appendix A 1 Custom Appendix A 2 Default Appendix A 3 A 1 UCS Profile HPC lt UCSProfile gt
3. Bright Cluster Manager 7 1 Cisco UCS Deployment Manual Revision 6754 Date Fri 20 Nov 2015 2 5 Bright Computing 2015 Bright Computing Inc All Rights Reserved This manual or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Bright Computing Inc Trademarks Linux is a registered trademark of Linus Torvalds PathScale is a registered trademark of Cray Inc Red Hat and all Red Hat based trademarks are trademarks or registered trademarks of Red Hat Inc SUSE is a registered trademark of Novell Inc PGI is a registered trademark of NVIDIA Corporation FLEXIm is a registered trademark of Flexera Software Inc ScaleMP is a registered trademark of ScaleMP Inc All other trademarks are the property of their respective owners Rights and Restrictions All statements specifications recommendations and technical information contained herein are current or planned as of the date of publication of this document They are reliable as of the time of this writing and are presented without warranty of any kind expressed or implied Bright Computing Inc shall not be liable for technical or editorial errors or omissions which may occur in this document Bright Computing Inc shall not be liable for any damages resulting from the use of this document Limitation of Liability and Damages Pertaining to Bright Computing Inc The Bright Cluster Manager product principally consists o
4. lt topSystem gt lt biosUnit gt lt biosSettings gt elas CRU gt lt biosVfIntelHyperThreadingTech vpIntelHyperThreadingTech disabled gt biosVfCoreMultiProcessing vpCoreMultiProcessing all gt lt biosVfIntelVirtualizationTechnology vpIntelVirtualizationTechnology enabled gt biosVfIntelVTForDirectedIO vpIntelVTDCoherencySupport enabled N vpIntelVTForDirectedIO enabled vpIntelVTDATSSupport enabled biosVfCPUPerformance vpCPUPerformance hpc biosVfHardwarePrefetch vpHardwarePrefetch enabled lt biosVfAdjacentCacheLinePrefetch vpAdjacentCacheLinePrefetch enabled gt biosVfDCUPrefetch vpStreamerPrefetch enabled vpIPPrefetch enabled gt lt biosVfDirectCacheAccess vpDirectCacheAccess enabled gt biosVfCPUPowerManagement vpCPUPowerManagement energy efficient gt biosVfEnhancedIntelSpeedStepTech vpEnhancedIntelSpeedStepTech enabled gt lt biosVfIntelTurboBoostTech vpIntelTurboBoostTech enabled gt lt biosVfProcessorC6Report vpProcessorC6Report enabled gt lt biosVfProcessorClE vpProcessorClE enabled lt biosVfPStateCoordType vpPStateCoordType HW ALL gt lt biosVfCPUFrequencyFloor vpCPUFrequencyFloor disabled gt lt biosVfCPUEnergyPerformance vpCPUEnergyPerformance energy efficient MEMORY gt biosVfSelectMemoryRASConfiguration vpSelectMemoryRASConfiguration maximum performanc
5. FRB 2 Timer LOM port state OS Boot Watchdog Timer Lom ports al state Os Boot Watchdog timer Poncy p d E Chassis E Vitual SMP Nodes Putty key pad Cesar Nodes Redirection after POST Always Enable reegt E node002 Terminal ype Gumo E Cloud Nodes Set link speed MIC Nodes PCI option ROMs 0S Boot Watchdog Timer Timeout Slot link speed UCSM Boot Order Rule Sitt state Slo state L5 Slotmezz state Niele Groups 0 Hadoop HDFS D OpenStack AL Users Groups Processor Conriguration Memory coniguration USE conniguration 2 Workload Management IE Monitoring Configuration Intel R Hyper Threading Technology dsened Authorization B Authentication Select memory RAS configuration Legacy USB support enabled 5 Number of Enabled Cores G DRAM clock throtting USB Emul 6064 Cenablea Enhanced Intel R Speedstep Technology DRAM refresh rate Allusb devices Cenatiea Intel R Turbo Boost Technology Low Voltage DDR mode USB port front Cananea Intel R VT NUMA optimized Cenabled USB portinternal enabled Intel R VTd Altitude oom USB port KVM enabled Intel R VTd ATS Support Demand scrub WS USB port rear abled Imol R VT d Coherency Support Patrol scrub USB port SD card CPU energy performance Channel inter leave Gata USB portY media enabled CPU frequency floor Cdisat
6. 30 10 00 BC 16 65 30 20 00 BC 16 65 30 10 00 BC 16 65 30 Load UCS XML from file Show Live Import from Category Figure 5 4 Overview of vNICs and vHBAs in cmgui O Bright Computing Inc 5 2 VIC Adapters 23 Double clicking on a vNIC interface row opens an edit dialog figure 5 5 Property classes that can be modified for a vNIC interface are e vNIC Settings Receive Side Scaling usNIC Profile Each of these property classes has several device properties that can be changed The Save button of the dialog saves the changes via CIMC directly Enable TCP TX Offload Checksum Generation vy Enable Large Receive v Receive Side Scaling Enable TCP Receive Side Scaling Enable IPv4 RSS Enable TCP IPv4 RSS Enable IPv6 RSS Enable TCP IPv6 RSS usNIC Profile usNIC Profile Class Of Service Cancel Figure 5 5 Edit vNIC properties using cmgui Double clicking on a vHBA interface will open an edit dialog figure 5 6 Property classes that can be modified for a vHPB interface are General Settings Error Recovery Fiber Channel Properties Queue Profiles Boot Table Persistent Bindings Device properties in these classes can be changed The Save button of the dialog saves the changes via CIMC directly O Bright Computing Inc 24 UCS Node And Category Configuration FC SAN Boot Enable Persistent LUN Binding Error Recovery Enable FCP Error Recov
7. E DIMM_B1 C zs Emm E 0 MIC Nodes E DIMM_C1 C 0 GPU Units E DIMM_D1 C e 0 Other Devices E DIMM EL C E Node Groups E DIMM F1 C 260 dh Users amp Groups A DIMM_G1 C 2 Workload Manageme i PA Monitoring Configuratii E ipmihealth E Authorization 25 0c0 2013 16 05 54 N A N A N A N A ul nale B Authentication 25 0cu2013 16 10 00 25 Oct 2013 17 05 00 node001 DIMM HI node001 DIMM F1 node001 DIMM EL node001 DIMM H1 D node00 1 DMM Di 2 M ir nodes B IEN RSEN node00DIMMCH js default AIl Ethernet switches DAI IB switches BAI Myrinet switches QAI Power Distribution Units AI rack sensors AI Chassis 1 All GPU Units 2 All generic devices 25 0ct 2013 16 10 00 25 0ct 2013 17 05 00 25 Oct 2013 16 01 37 28 C 30 C 29 C 28 C Figure 6 1 Plot of CIMC metrics obtained via XML API O Bright Computing Inc 31 Ele Monitoring View Bookmarks Ele Grid Groups Bright Cluster Manager Bright Cluster Manager Monitoring E Switches ES switcho1 LJ Networks externalnet globalnet 2 internalnet 2 ipminet ij Power Distribution Units El Software Images s defaultimage 0 Node Categories f default 0 Head Nodes E ucs head Racks ESj Chassis E Virtual SMP Nodes 0 Nodes E node001 Gi Cloud Nodes MIC Nodes E GPU Units Gi Other Devices Node Groups di Users amp Groups Workload Management BA Monitoring Configuration
8. all the nodes in the category is hardly likely to be useful Instead clicking on the button in category mode simply prompts the user to select a specific node and launches a console for the selected node Bright Computing Inc Monitoring Metrics that can be measured and monitored through CIMC on a C series rack server are covered in this chapter The majority of the metrics are reported via IPMI while some metrics are measured through the CIMC XML API Chapter 9 of the Administrator Manual has more information about metric collection and metric collection scripts The following table shows a list of metrics that are monitored from the CIMC using the XML API Metric Description boardconsumedPower Power consumed by board boardinputVoltage Input voltage across board boardinputCurrent Input current into board boardfrontTemp Front temperature of board boardiohlTemp Temperature of IOH1 on board boardioh2Temp Temperature of IOH2 on board boardambientTemp Ambient temperature of board boardrearTemp Temperature of rear of board DIMM A1 Temperature of DIMM A1 DIMM B1 Temperature of DIMM B1 DIMM C1 Temperature of DIMM C1 DIMM D1 Temperature of DIMM D1 DIMM E Temperature of DIMM E1 DIMM F1 Temperature of DIMM F1 DIMM G1 Temperature of DIMM G1 DIMM H1 Temperature of DIMM H1 Pl TEMP SENS Temperature of power supply 1 continues O Bright Computing Inc 30 Monitoring
9. bright71 export CIMC_FIRMWARE cm shared apps hwvendor ciscou cs current firmware c220 cimc upd pkg c220 m3 cimc full 1 5 4 bin root bright71 service ipmi start root bright71 cm shared apps hwvendor ciscoucs current firmware l wup cimc update CIMC_FIRMWARE The update takes several minutes The administrator must wait for it to complete It is completed when a message appears prompting the administrator to run CIMC firmware activation 4 1 2 CIMC Firmware Activation The head node CIMC firmware can be activated as follows Example root bright71 cm shared apps hwvendor ciscoucs current firmware f wup cimc activate 4 1 3 CIMC Firmware Version View The head node CIMC firmware can be viewed as follows Example Bright Computing Inc 16 UCS Firmware root bright71 cm shared apps hwvendor ciscoucs current firmware f wup cimc show version Firmware Update Utility v1 1 Build 9 Copyright c 2011 Cisco Systems Inc All rights reserved Platform detected C240M3 Series 4 1 4 BIOS Firmware Update The head node BIOS firmware can be updated as follows Example root bright71 export BIOS FIRMWARE cm shared apps hwvendor ciscouN cs current firmware c220 bios UCS C220 BIOS 1 5 4 0 CAP root bright71 cm shared apps hwvendor ciscoucs current firmware b iosup n p u SBIOS_FIRMWARE For the head node BIOS firmware update The update takes several minutes The hosts a
10. ext cm ibv pingpong test To verify correctness of MPI traffic the cm usnic ring test program can be run The following example shows how to run it as the cmsupport user su cmsupport module load openmpi gcc cluster tools ext cm usnic ring test O Bright Computing Inc Nexus Switch Configuration This chapter recommends configuration settings for a Cisco Nexus Switch that is part of a Cisco UCS C Series rack mount cluster The following instructions are based on the Cisco Nexus Operating System NX OS version 6 0 running on a Cisco Nexus 3548 switch and assume command line access to the switch The administrator should consult the appropriate switch and NX OS documentation for possible changes before proceeding with the configuration The administrator executes the following steps The switch is logged into details on logging in are to be found in the switch documentation The configuration mode is entered configure terminal The interfaces are selected and flow control is set interfac thernet 1 1 48 flowcontrol receive on flowcontrol send on The network qos policies are set class map type network qos classl policy map type network qos my network policy class type network qos classl pause no drop system qos Service policy type network qos my network policy policy map type network qos jumbo class type network qos class default mtu 9216 system qos Service policy type network qos jumb
11. regular GE network is not availabl the 10GE network and only the corresponding network interf s needt ssigned on internalnet on the Network Interfaces screen However if itis still required to define a separate IOGE network alongside the regular GE ne rk and if the LOGE network should be used as the default management network then please remember to selectthe 10GE network as the Management Network in the next screen BMC Network Configuration Co0c0c0c0c0 cecececec0c0 c0000000 Will nodes have IPMI ILO CIMC compatible BMCs Yes No Please select a BMC network type CIMC X Use dedicated BMC port or share BMC with first Ethernet Interface Dedicated X To which Ethernet segment are the BMCs connected External Network Use DHCP to obtain BMC IP addresses Yes No Automatically configure BMC when node boots Yes No Cancel Go Back Continue Figure 2 3 Select Ethernet segment for BMC e f the BMC is to use DHCP to obtain BMC IP addresses it should be set to do so in the same section of figure 2 3 e Ifa separate layer 3 network was chosen and is to be used for node booting then 10genet should be selected as the management network figure 2 4 O Bright Computing Inc UCS Cluster Deployment Procedure Networks Q Networks English US Ges Management network 10genet internalnet externalnet Network parameters externalnet Name Base address Netmask Domain name Default gateway N
12. settings of the node category the change request is immediately sent to the CIMC of the node This is done by CMDeamon on the head node as already explained in the other sections of this chapter 5 9 UCS Category Settings In Bright Cluster Manager defining a configuration at category level means that compute nodes in that category inherit the specified configuration The default category that is defined in a standard vanilla installation of Bright Cluster Manager is preconfigured with the UCS profile section 5 7 The UCS profile can be selected during head node installation but by default it takes the default value Appendix A 3 In this case all nodes by default therefore simply inherit the default UCS profile The list of nodes to which the UCS profile is applied to is conveniently listed at the top of the UCS tab of the specific category figure 5 11 Selecting an individual node from there with a double click then opens up the node item and allows configuration adjustments per node as described in the preceding sections of this chapter X Bright Cluster Manager ooo Eile Monitoring View Bookmarks Help H default E Bright z 0 UCS Cluster RESOURCES My Clusters lt ces Disk Setup RAID Setup Static Routes FS Mounts FSExports Roles Nodes GPU Settings ucs v Ej Bright 7 0 UCS Cl Gj Switches Eg switchO1 gt y Networks P y Power Distribu gt Software Image
13. vpCPUPowerManagement energy efficient gt biosVfEnhancedIntelSpeedStepTech vpEnhancedIntelSpeedStepTechN enabled lt biosVfIntelTurboBoostTech vpIntelTurboBoostTech enabled biosVfProcessorC6Report vpProcessorC6Report enabled gt biosVfProcessorC1E vpProcessorClE enabled lt biosVfPStateCoordType vpPStateCoordType HW ALL lt biosVfCPUFrequencyFloor vpCPUFrequencyFloor disabled biosVfCPUEnergyPerformance vpCPUEnergyPerformance energy efficient 1 MEMORY biosVfSelectMemoryRASConfiguration vpSelectMemoryRASConfiguration maximum performance lt biosVfDRAMClockThrottling vpDRAMClockThrottling performance gt lt biosVfNUMAOptimized vpNUMAOptimized enabled lt biosVfLvDIMMSupport vpLvDDRMode performance mode lt biosVfDramRefreshRate vpDramRefreshRate auto gt biosVfMemoryInterleave vpChannelInterLeave auto vpRankInterLeave auto lt biosVfPatrolScrub vpPatrolScrub enabled lt biosVfDemandScrub vpDemandScrub enabled biosVfAltitude vpAltitude 300 m biosSettings lt biosUnit gt topSystem UCSProfile O Bright Computing Inc A 3 UCS Profile Default 35 A 3 UCS Profile Default lt UCSProfile gt lt topSystem gt lt biosUnit gt lt biosSettings gt lt CPU gt biosVfIntelHyperThreadingTech vpIntelHyperThreadingTech disabled gt biosVfCoreMultiProces
14. 064 Enhanced Intel R Speedstep Technology enabled J TT DRAM Tetresn rate m 3 Ailusb devices Intel R Turbo Boost Technology enabled KR Low Voltage DDR mode power saving mode v USB port front Intel R VT enabled gt NUMA optimized enabled USB port internal Intel R VT d enabled H Altitude 20 m X USB port KVM Intell VI d ATS Support anabiad D Demand scrub anablac USR nort rear o NEEDS Figure 5 10 KVM Console Launched From KVM Button An error is reported by cmgui if CIMC does not provide a valid JNLP file or if javwas is not found 5 7 Importing UCS Settings From An XML File Configuration and settings for different components in CIMC for a particular node can be imported from an XML file The list of specifications in the XML file is called the UCS profile The Load UCS XML from file button launches a file browser to help import of the XML file After import the contents populate all the UCS subtabs Clicking the Save button stores the imported changes in the CMDaemon database and CIMC is notified of the requested changes Appendix A has the XML schema details for the UCS profile 5 8 Applying UCS Settings Changes UCS settings for a node can be applied while the node is up O Bright Computing Inc 28 UCS Node And Category Configuration If a node has been provisioned at least once before and is known to Bright Cluster Manager then when changes are made to the UCS settings of the node or to the UCS
15. 5 3 explains NTP related settings that can be configured for the CIMC Section 5 4 describes syslog parameters and other settings that can be configured for the CIMC Section 5 5 describes fault logging Section 5 6 explains how to launch a KVM session to the node console Section 5 7 describes how to import CIMC UCS related settings from an XML file into the UCS configuration settings of the node Section 5 9 describes how node categories are used in managing UCS configuration 5 1 BIOS Settings Figure 5 1 shows a view of the BIOS subtab properties that can configured O Bright Computing Inc 20 UCS Node And Category Configuration Bright Cluster Manager lexus ES Brigt ruk Custer My Clusters overview Tasks Settings System Intormaton Services Process Management fH Brighttrunk Cluster Model UCSC C240 M35 Firmwares Locator LED Vendor Cisco Systems Inc cmc Lem Storage Status Serial FCHLG17VOUL BIOS C240M3 1 5 41 0 Build Dato 11 13 2013 Power Supplio UUID 54F92905 9245 40CC 878B AC483AB455A4 Lom 2 1284 Temperature stengt wire 2 Storage 2 1202344471 Ee e Luis VHBAs 2 Ft Server Status e S intemanet Ga Power Distribution Units s Bios C Sotware images m defaultimage ES Console redirection LOM and PCIe Slots Configuration Server Management default Ej Head Nodes Redirection sablea mmm SENS Baud rate Cem Ei How contol Crone LOM port state
16. CS C220 BN IOS 1 5 4 0 CAP cm shared apps hwvendor ciscoucs current firmware bio sup n p u BIOS FIRMWARE For the compute node BIOS firmware updates The BIOS firmware update takes several minutes The hosts automatically reboot after the biosup command has completed n some cases the hosts do not power on automatically The hosts must then be powered on manually for example via cmgui O Bright Computing Inc 18 UCS Firmware 4 2 5 BIOS Firmware Version View The compute node BIOS firmware can be viewed as follows Example root bright71 cmsh bright71 device bright71 gt device pexec b n node001 node004 cm shared apps hwven dor ciscoucs current firmware biosup i Nodes down node001 bright71 device node002 BIOS Update Utility v2 Build 8 Copyright c 2013 Cisco Systems Inc All rights reserved Vendor Cisco Systems Inc Version C220M3 1 5 4 0 111320130449 StartAddr 61440 0x f000 BIOSReleaseDate 11 13 2013 BIOS RomSize 127 0x7 f BIOS BIOSCharacteristics 1066113152 0x3f809880 O Bright Computing Inc UCS Node And Category Configuration This chapter describes the management of UCS configuration for individual head and compute nodes using cmgui Section 5 1 explains the BIOS Settings tab which allows the BIOS settings of the node to be modified directly Section 5 2 explains how to modify properties of VIC adapter interfaces Section
17. RE Iq i 0d bout ThisManual i EE A e E E e EIE VS Sg re iii 0 2 About The Manuals In General iii 0 3 Getting Administrator Level Support iii Introduction 1 11 Cisco Unified Computing System 1 12 UCS Integration in Bright Cluster Manager 1 UE WE e ia do tee eee a car ay adeb ev iba e 2 UCS Cluster Deployment Procedure 3 ZT Pre install Checklist Ae A der A d ale GELS XudskXsdleg xA 3 2 28 Switch Configuration o a eae A ria e 4 23 Head Node Installation ee 4 24 Compute Nodes Provisioning ooo 9 25 Firmware Upgrade On Servers o 9 2 6 Post Installation Configuration aooaa 9 e RENE 10 2 7 1 Verifying The usNIC Installation o ooo e 10 2 7 2 Running MPI Applications eA 11 Nexus Switch Configuration 13 UCS Firmware 15 4 1 Head Node Firmware Upgrade 15 411 ClIMCFirmmwareUpdate oo 15 41 2 EIMC Firmware Activation 2 2 weg a ERR 15 4 1 3 CIMC Firmware Version View 15 414 BIOS Firmware Update eee 16 4 1 5 BIOS Firmware Version View aoaaa aaa 16 42 Compute Node Firmware Upgrade oaaao aa 16 421 CIMC Firmware Update 16 4 2 2 CIMC Firmware Activation 17 4 2 3 CIMC Firmware Version View 2 ee 17 424 BIOS Firmware Update inie nor de uec edem e ee wR ed aS 17 4 2 5 BIOS Firmware Version View aoaaa aaa 18 UCS Node And Category Configuration 19 JA BIOS SENES usce Ret s Sum ebd tud ane AE seres uud as be re
18. Rate vpDramRefreshRate auto lt biosVfMemoryInterleave vpChannelInterLeave auto vpRankInterLeave auto gt lt biosVfPatrolScrub vpPatrolScrub disabled gt lt biosVfDemandScrub vpDemandScrub enabled gt lt biosVfAltitude vpAltitude 300 m gt lt biosSettings gt lt biosUnit gt lt topSystem gt lt UCSProfile gt Bright Computing Inc
19. The KVM button launches a request to CIMC to obtain the JNLP file containing the information to make connection to the console If the CIMC replies with a valid JNLP file then cmgui checks if Java Web Start typically javaws exists in the default path If javaws is found then cmgui uses a down loaded JNLP file to automatically launch the KVM console figure 5 10 ucs head Overview Tasks Settings System Information Services Process Management Network Setup Static Routes FS Mounts FS Exports Roles Hadoop Ceph OpenStack GPUSettings ucs Nol Model UCSC C240 M3S Firmwares Vendor Cisco Systems Inc CIMC 1 5 4 Serial FCHI617V0UL BIOS C240M3 1 5 41 0 Build Date 11 13 2013 UUID 54F92905 9245 40C C 878B AC483AB455A4 Lom 2 1 2 84 vNICs 2 Storage 2 120 234 1471 vHBAs 2 PCI Bios VIC Adapters NTP Serings Syslog Senings Fail _ File View Macros Tools Help Ef kvm gj virtual Media Console redirection Server Management CentOS release 5 Final Redirection lisabled M Kernel 2 6 32 431 3 1 e16 x86 64 on an x86 64 FRB 2 Timer m Baud rate 115200 ucs head login Watchdog Timer Flow control none ichdog Timer P Putty key pad N ichdog Timer T Redirection after POST Always Enable gt UCSM Boot Order Rule Terminal type un M Processor Configuration USB Configuration Intel R Hyper Threading Technology disabled Legacy USB support Number of Enabled Cores all USB Emul 6
20. abled nor rmm Import from Category Figure 5 2 Smaller Window View Of The BIOS Subtab In cmgui In the BIOS subtab the different BIOS properties that can be configured are grouped into boxes based on their functionality These subtabs are Console redirection LOM and PCIe Slots Configuration LOM Lan On Motherboard e Server Management Processor Configuration Memory Configuration USB Configuration Onboard Storage e OPI Configuration QPI Quick Path Interconnect Other Settings PCI Configuration BIOS Boot Order If a BIOS property is modified then the Save button can be clicked to update the settings via CIMC For example in figure 5 3 the property Intel R Hyper Threading Technology has been modified to enabled Save has become clickable and clicking it requests CIMC to update the property O Bright Computing Inc 22 UCS Node And Category Configuration ger Buunmamns now bright70 ucs Ej Bright7 0UCS Cluster Model UCSC C240 M3S Firmwares Locator LED o Vendor Cisco Systems Inc CIMC 1 5 4 Storage Status Serial FCH1617VOUL BIOS C240M3 1 5 4f 0 Build Date 11 13 2013 Power Supplies UUID 54F92905 9245 40CC 878B AC483AB455A4 Lom 2 1 2 84 Temperature vNICs 2 Storage 2 120 234 1471 Fans e vHBAs 2 PCI Server Status Processor Configuration Memory Configuration Intel R Hyper Threading Technology enabled Select memory RAS configuration maximum perf
21. cation using the installed example programs 2 7 1 Verifying The usNIC Installation The following helps verify that usNIC has been installed correctly and is working e Listing the usnic verbs kernel module and related dependencies lsmod grep usnic verbs usnic verbs 78776 0 ib core 73747 13N ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad iw_cxgb4 iw_cxgb3 usnic_verbs enic 67741 1 usnic_verbs The preceding output may vary depending on the other kernel modules installed Viewing The Configuration Of usNIC enabled Interfaces ibv devinfo hca id usnic 1 State PORT ACTIVE 2 hcaid usnic 0 state PORT ACTIVE 2 Verifying that rdma service is running and is using the usNIC related modules Bright Cluster Manager enables the service by default and is also configured to start on boot as chkconfig output should show chkconfig list rdma rdma O off l off 2 0n 3 on 4 on See 6 off service rdma status Low level hardware support loaded none found Upper layer protocol modules ib ipoib User space access modules rdma ucm ib ucm ib uverbs ib umad usnic verbs Connection management modules rdma cm ib cm iw cm O Bright Computing Inc 2 7 Using usNIC 11 2 7 2 Running MPI Applications Tranmission of the usNIC network packages between the hosts can be verified by running the following program su omsupport module load cluster tools
22. du 19 5 2 VIC Adapters 5 cus A Ade E C eee dere see eer darts 22 5 9 NTE EE e ere BER e tuts dee etd e te e de S tos ee trit 24 5 4 Syslog SettifigS ome cR Cb Rubus Rx e gne ADM VIS qu ie E eee E 25 Table of Contents B D AA dos aoc ea eae pon oh dut evt to d Ht Mos bol Mur dog 26 56 The REVM BULDODL ou a EB IPEA d ux ure d eds 26 5 6 1 KVM Console Requirements eA 27 57 Importing UCS Settings From An XML File o ooo o 27 5 8 Applying UCS Settings Changes o eA 27 5 95 UCS Cat gory Settings amis a hue E auae muet ie EE a ene 28 Monitoring 29 UCS Profiles 33 Act UCES Profile HBO A A eme Sak ele te De us 33 A2 UCS Profile Custonr embara a a ra 34 A3 UCS Profiles Default ss ua abe st AA a AA 35 Preface Welcome to the UCS Deployment Manual for Bright Cluster Manager 7 1 0 1 About This Manual This manual is aimed at helping cluster administrators understand and manage Cisco UCS server capa bilities that are integrated into Bright Cluster Manager The administrator is expected to be reasonably familiar with the Bright Cluster Manager Administrator Manual 0 2 About The Manuals In General Regularly updated versions of the Bright Cluster Manager 7 1 manuals are available on updated clus ters by default at cm shared docs cm The latest updates are always online at http support brightcomputing com manuals The Installation Manual describes installation procedures for a bas
23. e lt biosVfDRAMClockThrottling vpDRAMClockThrottling performance gt biosVfNUMAOptimized vpNUMAOptimized enabled lt biosVfLvDIMMSupport vpLvDDRMode performance mode biosVfDramRefreshRate vpDramRefreshRate auto biosVfMemoryInterleave vpChannelInterLeave auto vpRankInterLeave auto gt O Bright Computing Inc 34 UCS Profiles lt biosVfPatrolScrub vpPatrolScrub enabled gt lt biosVfDemandScrub vpDemandScrub enabled gt lt biosVfAltitude vpAltitude 300 m lt biosSettings gt lt biosUnit gt topSystem UCSProfile A 2 UCS Profile Custom lt UCSProfile gt lt topSystem gt lt biosUnit gt lt biosSettings gt Stes CPU gt lt biosVfIntelHyperThreadingTech vpIntelHyperThreadingTech disabled gt lt biosVfCoreMultiProcessing vpCoreMultiProcessing all gt lt biosVfIntelVirtualizationTechnology vpIntelVirtualizationTechnology enabled gt biosVfIntelVTForDirectedIO vpIntelVTDCoherencySupport enabled N vpIntelVTForDirectedIO enabled vpIntelVTDATSSupport enabled biosVfCPUPerformance vpCPUPerformance hpc gt biosVfHardwarePrefetch vpHardwarePrefetch enabled gt biosVfAdjacentCacheLinePrefetch vpAdjacentCacheLinePrefetch enabled biosVfDCUPrefetch vpStreamerPrefetch enabled vpIPPrefetch enabled gt lt biosVfDirectCacheAccess vpDirectCacheAccess enabled gt lt biosVfCPUPowerManagement
24. edure is recommended During installation onto Cisco UCS the following choices must be made Hardware vendor Cisco UCS must be chosen as the hardware vendor on the node information screen if it was not chosen during download of the Bright Cluster Manager ISO UCS Profile figure 2 1 This profile determines the default UCS configuration that is applied for the head node and the default category of compute nodes Appendix A has more on the default profile types O Bright Computing Inc 2 3 Head Node Installation Bright Cluster Manager Installer Nodes English US o The numer of nodes node start number node b name and node digits are relevant only for the v compute no nd do not include the head no Q Please enter node information k O kd Nodes Number of racks v Number of nodes 1 Q E Node start number 1 Qo Node base name node Node digits o Q Node Hardware Manufacturer Cisco UCS M o Cisco UCS Profile hpc x o e o e Q i Cancel Go Back Continue Figure 2 1 Selecting The UCS Profile The following items a separate layer 3 network for 10 Gig E node booting can besetinthe Additional High Speed Networks section of figure 2 2 O Bright Computing Inc 6 UCS Cluster Deployment Procedure Additional Network Configuration English US e k o Additional High Speed Networks x Additional high speed interconnect 10 Gig E M Create
25. el and interact directly with the Cisco UCS VIC 1225 adapter when sending and receiving network packets A Bright Cisco UCS cluster has all the required software pre installed and configured for Cisco usNIC functionality to work out of the box The following are installed and configured by default The Linux kernel verbs driver for usNIC kmod usnic verbs The Linux kernel driver for enic kmod enic The user space library libibverbs plugin for usNIC 1ib usnic verbs The latest Open MPI software that provides full support for usNIC The services that required are enabled to start on boot rdma for usnic verbs The recommended BIOS and CIMC usNIC settings are pre configured Section 2 7 covers the following Cisco usNIC topics in more detail e Verifying that usNIC is working and available in user space Running MPI applications using usNIC O Bright Computing Inc UCS Cluster Deployment Procedure This chapter describes a step by step procedure to deploy Bright Cluster Manager on Cisco UCS hard ware Withit an experienced cluster administrator should quickly be able to get Bright Cluster Manager running on a Cisco UCS cluster A Cisco UCS cluster is also called a domain in Cisco terminology and in Bright Cluster Manager 7 1 implies a combination of Cisco UCS C Series Rack Servers and supported fabric interconnects or fabric extenders Full support for Cisco UCS B Series Blade Servers is planned for later Following the d
26. eployment procedure does not require extensive knowledge of Bright Cluster Man ager or Cisco UCS A brief explanation of the task is given at each step and references to other docu mentation are given if necessary However only references to the Bright Cluster Manager manuals are provided Further Cisco UCS documentation can be obtained from Cisco or the appropriate hardware vendors The procedure follows 2 1 Pre install Checklist Before starting deployment the following should be checked or considered This prevents delays or having to restart at a later stage of the deployment Choosing the node booting and provisioning network In a Cisco UCS domain nodes can boot from the regular 1GE network interface or from the Cisco VIC 10GE interface On most UCS C Series rack servers the 1GE interface is the first device in the network boot order If the interface used to boot and provision compute nodes is the Cisco VIC 10GE interface then Cisco VIC ports must be cabled to the 10GE switch or fabric extender that is connected to the switch The first boot device in the network boot order must be set to the Cisco VIC that the node should boot from the 1GE interface then The 1GE ports must be cabled to a 1GE switch or to the 10GE switch through a GBIC converter The first boot device in the network boot order must be set to the 1GE interface that the node should boot from Choosing the CIMC BMC network It is b
27. ery Link Down Timeout Port Down UO Retries Port Down Timeout Error Detect Timeout Resource Allocation Timeout Fiber Channel Properties Interrupt Mode Figure 5 6 Edit vHBA properties using cmgui 5 3 NTP Settings Enabling and configuring the NTP clients via CIMC can be done using the NTP Settings subtab figure 5 7 O Bright Computing Inc 5 4 Syslog Settings 25 X Bright Cluster Manager 6086 Eile Monitoring View Bookmarks Help 4 bright70 ucs Ej Bright7 0UCS Cluster ves My Clusters E v Bright 7 0 U v Lj Switches Model UCSC C240 M3S Firmwares Eg switc Vendor Cisco Systems Inc CIMC 1 5 4 P gj Networks Serial FCH1617V0UL BIOS C240M3 gt 3 Power Di UUID 54F92905 9245 40CC 878B AC483AB455A4 Lom 2 1 2 84 b Gi Software vNICs 2 Storage 2 120 23 gt O Node Ca VHBAs 2 PCI v jjHead No es bright gt E Chassis gt Virtual S v 3j Nodes amp g node amp node P E Cloud N gt MIC Nodes b C3 GPU Units gt Other De gt Node Gr gt E Hadoop d Users amp 25 Workloa y Enable NTP Server 1 Server 2 Server 3 Server 4 KVM Load UCS XML from file Show Live Import from C Figure 5 7 Configuring NTP Settings Using cmgui 5 4 Syslog Settings Logging controls for the CIMC are configured from the Syslog Setti
28. est to connect the dedicated BMC ports to an external network which is accessible from outside the cluster This makes reaching the remote console of the O Bright Computing Inc 4 UCS Cluster Deployment Procedure node easier if there is a communication failure between the head and compute nodes However the BMC network can also be configured to be internal to the cluster RAID configuration It is best to consult with the hardware vendor for known issues with a spe cific RAID configuration for the RAID controller on the servers RAID volumes can be configured by choosing Ctrl H WebBios during boot 2 2 Switch Configuration The servers in a Cisco UCS cluster are typically connected via a 10GE Cisco Nexus Switch running the NX OS operating system The switch should be configured according to the recommended procedures for proper functioning of the cluster and for extracting maximum network performance Chapter 3 Nexus Switch Configuration has details on the configuration options and recommended settings The switch must be configured properly before the compute nodes can be provisioned or provi sioning is likely to fail Other issues may also occur due to bad configuration even if provisioning is succesful 2 3 Head Node Installation The head node now has to have Bright Cluster Manager installed onto it Chapter 1 of the Bright Cluster Manager Installation Manual has a quick start guide for this A bare metal installation proc
29. etwork type externalnet DHCP DHCP DHCP Ethernet 10genet Figure 2 4 Select 10 Gig E Network As Management Network Otherwise the default internalnet can be used Inthe Network Interfaces screen figure 2 5 the correct interface that will be used for management and external networks on the head node must be chosen By default eth0 is assigned an IP address on the internal network ethl is assigned an IP address on the external network If the Cisco VIC interface is to be used for management and node booting then the network of the corresponding interface the one identified by the enic driver should be set to 10genet if the separate 10genet was chosen setto internalnet otherwise O Bright Computing Inc 2 4 Compute Nodes Provisioning 9 Bright Cluster Manager Installer Network Interfaces English US 2 l Head node Network Interface e followir d ied r k rk interfaces will tur 1 tl ead nod o Q nterface Network IP Address x tho 1ed X 00 externalnet DHCP Y 1 iic 1 1 X a eth3 enic nternalne 10 141 255 254 Q Network Interfaces u m DHCP ju wl ite rf k por e trol rigl I Je Network Interf llowir i ied network interfaces v t uj tu mati a 1 N f i OOTIF nternalne X v 9 9 O x e Figure 2 5 Set network of Cisco VIC interface 2 4 Compute Nodes Provisioning After the head node installation is done the compute nodes must be installed The follo
30. f free software that is licensed by the Linux authors free of charge Bright Computing Inc shall have no liability nor will Bright Computing Inc provide any warranty for the Bright Cluster Manager to the extent that is permitted by law Unless confirmed in writing the Linux authors and or third parties provide the program as is without any warranty either expressed or implied including but not limited to marketability or suitability for a specific purpose The user of the Bright Cluster Manager product shall accept the full risk for the qual ity or performance of the product Should the product malfunction the costs for repair service or correction will be borne by the user of the Bright Cluster Manager product No copyright owner or third party who has modified or distributed the program as permitted in this license shall be held liable for damages including general or specific damages damages caused by side effects or consequential damages resulting from the use of the program or the un usability of the program including but not limited to loss of data incorrect processing of data losses that must be borne by you or others or the inability of the program to work together with any other program even if a copyright owner or third party had been advised about the possibility of such damages unless such copyright owner or third party has signed a writing to the contrary Table of Contents Table of Contents is A ea ee Ee I Re en eR RA
31. ftware and typically managed by a cluster manager such a Bright Cluster Manager In such a setup they are normally connected with a Cisco Nexus switch The C Series rack servers are managed and monitored through the Cisco Integrated Management Controller CIMC The CIMC can be accessed through a Web based GUI a CLI and IPMI like any other baseboard manage ment controller The CIMC also exposes an XML API which is a programmatic interface to the CIMC and which accepts XML documents through HTTP HTTPS requests The CIMC XML API implements a subset of the methods available in the Cisco UCS Manager XML API The scope of the CIMC XML API is limited to a single C Series server For extensive and detailed documentation on Cisco UCS the hardware vendor or the Cisco website should be consulted The next section explains how Bright Cluster Manager 7 1 integrates into the Cisco UCS domain and makes use of the CIMC XML API to configure monitor and manage the rack servers 1 2 UCS Integration in Bright Cluster Manager Bright Cluster Manager 7 1 can configure monitor and manage C Series rack servers directly that is in the standalone mode or via UCS Manager Deployment and management of UCS C Series servers from Bright Cluster Manager is designed to be easy in this version Complete support for configuring and managing B Series blade chassis and servers are intended for future versions of Bright Cluster Manager O Bright Computing Inc 2 Introduc
32. ic cluster The Administrator Manual describes the general management of the cluster The User Manual describes the user environment and how to submit jobs for the end user The Cloudbursting Manual describes how to deploy the cloud capabilities of the cluster The Developer Manual has useful information for developers who would like to program with Bright Cluster Manager The OpenStack Deployment Manual describes how to deploy OpenStack with Bright Cluster Man ager The Hadoop Deployment Manual describes how to deploy Hadoop with Bright Cluster Manager The UCS Deployment Manual describes how to deploy the Cisco UCS server with Bright Cluster Manager If the manuals are downloaded and kept in one local directory then in most pdf viewers clicking on a cross reference in one manual that refers to a section in another manual opens and displays that section in the second manual Navigating back and forth between documents is usually possible with keystrokes or mouse clicks For example lt A1t gt lt Backarrow gt in Acrobat Reader or clicking on the bottom leftmost naviga tion button of xpdf both navigate back to the previous document The manuals constantly evolve to keep up with the development of the Bright Cluster Manager envi ronment and the addition of new hardware and or applications The manuals also regularly incorporate customer feedback Administrator and user input is greatly valued at Bright Computing So any co
33. ied Rank inter leave CPU performance Cenrenrise CPU power management enery elficen Execute disable prefetch Adjacent Cache Line Pretetcher Direct Cache Access Support Figure 5 1 Full width View Of The BIOS Subtab In cmgui If this manual has been printed on A4 paper then figure 5 1 has an unreadable text size The text in the figure can however be read with the Bright released PDF file version of this manual if it is on a display and if zooming in on the figure is allowed If running cmgui itself to display the UCS tabs screen with its subtabs then the full width view is quite readable on a relatively modest 21 monitor with a resolution of 1280x1024 or better A zoomed in view that shows some details and is thus helpful for A4 paper readers is shown in figure 5 2 Bright Computing Inc 5 1 BIOS Settings 21 View Bookmarks Help a bright70 ucs Bright 7 0 UCS Cluster Model UCSC C240 M3S Firmwares Locator LED e Vendor Cisco Systems Inc CIMC 1 5 4 Storage Status Serial FCH1617VOUL BIOS C240M3 1 5 4f 0 Build Date 11 13 2013 Power Supplies UUID 54F92905 9245 40CC 878B AC483AB455A4 Lom 2 1 2 84 Temperature vNICs 2 Storage 2 120 234 1471 Fans e vHBAs 2 PCI Server Status Console redirection LOM and PCIe Slots Configuration Redirection disabled ER LOM port state Enabled ER Baud rate 115200 x LOM port state Enabled ul Flow control C gt LOM ports all state En
34. iscoucs current firmware fwup cimc update CIMC FIRMWARE The text between the quote marks is one long line using backslashes to indicate continuation on the printed page as is customary in the Bright Cluster Manager manuals The CIMC firmware update takes several minutes The administrator must wait for it to complete It is completed when a message appears prompting the administrator to run CIMC firmware activation 4 2 2 CIMC Firmware Activation The compute node CIMC firmware can be activated as follows Example root bright71 cmsh bright71 device bright71 gt device pexec b n node001 node004 service ipmi start X cm shared apps hwvendor ciscoucs current firmware fwup cimc activate The CIMC firmware activation will take several minutes The administrator must wait for it to complete It is completed when the CIMC completion message is seen 4 2 30 CIMC Firmware Version View The compute node CIMC firmware can be viewed as follows Example root bright71 cmsh bright71 device bright71 device pexec n node001 node004 service ipmi start cm N shared apps hwvendor ciscoucs current firmware fwup cimc show version 4 2 4 BIOS Firmware Update The compute node BIOS firmware can be updated as follows Example root bright71 cmsh bright71 device bright71 gt device pexec b n node001 node004 export BIOS FIRMWARE V cm shared apps hwvendor ciscoucs current firmware c220 bios U
35. layer 3 10 Gig E network Yes 7 No ka Allow booting over 10 Gig E Yes No Note Creation of a s Additional Networks o SC ate 10GE network alon Q the regular GE network and if the 10GE network should be u staut management networ A then please remember to selectthe 10GE network as the Management Network in the next screen o o BMC Network Configuration o Will nodes have IPMIALO CIMC compatible BMCs e Yes No Q Please select a BMC network type CIMC X o Use dedicated BMC port or share BMC with first Ethernet Interface Dedicated X o To which Ethernet segment are the BMCs connected internal Network M Q Q Create new layer 3 network i e IP subnet for BMC interfaces e Yes No Q Use DHCP to obtain BMC IP addresses Yes No Q Automatically configure BMC when node boots Yes No Q I Cancel Go Back Continue Figure 2 2 Allowing Separate Layer 3 Networking And Node Booting For 10 Gig E An External Network should be selected for the BMC Ethernet segment figure 2 3 O Bright Computing Inc 2 3 Head Node Installation 7 Bright Cluster Manager Installer Additional Network Configuration English US UA AA Additional High Speed Networks Additional high speed interconnect 10 Gig E v Create layer 3 10 Gig E network Yes No Allow booting over 10 Gig E Yes No Note Creation of a separate 10GE layer 3 network is not strictly ne Additional Networks management network and or if a
36. m ments suggestions or corrections will be very gratefully accepted at manuals brightcomputing com 0 3 Getting Administrator Level Support Unless the Bright Cluster Manager reseller offers support support is provided by Bright Computing over e mail via support brightcomputing com Section 10 2 of the Administrator Manual has more details on working with support Introduction This chapter introduces how Cisco Unified Computing System Cisco LICS is integrated with Bright Cluster Manager It introduces the following topics Section 1 1 gives a brief overview of Cisco UCS Section 1 2 discusses how servers in Cisco UCS are managed and integrated in Bright Cluster Manager 1 4 Cisco Unified Computing System Cisco UCS is a data center server platform that comprises compute servers switching fabric and man agement software The physical chassis used in the platform is available in two models the B Series blade chassis with blade servers the C Series rack mount servers The hardware in both models can be managed by the UCS Manager software which typically runs on the fabric interconnect s When the C Series servers are used as part of UCS they perform the same functions as the B Series blade servers A cluster instance that is managed by UCS Manager is called a Cisco UCS domain in Cisco terminology The C Series rack servers can also operate in a standalone environment Here standalone means without the UCS Manager so
37. ngs subtab figure 5 8 The properties that can be configured are e the Remote Syslog Server Primary e the Remote Syslog Server Secondary e the Local Logging Bright Computing Inc 26 UCS Node And Category Configuration es bright70 ucs E Bright7 0UCS Cluster Model UCSC C240 M3S Firmwares Locator LED Qo Vendor Cisco Systems Inc CIMC 1 5 4 Storage Status Seria FCH1617V0UL BIOS C240M3 1 5 4f 0 Build Date 11 13 2013 Power Supplies UUID 54F92905 9245 40CC 878B AC483AB455A4 Lom 2 1 2 84 Temperature Q vNICs 2 Storage 2 120 234 1471 Fans e vHBAs 2 PCI Server Status Syslog Settings Fault Remote Syslog Server Primary Remote Syslog Server Secondary Local Logging Enable T Enable Logging Severity HostnamellP HostnamellP Load UCS XML from file Show Live Import from Category Figure 5 8 Configuring Syslog Settings Using cmgui 5 5 Fault Logs The Fault Log subtab figure 5 9 shows a summary of faults instances reported by the CIMC System event logs and CIMC authentication and audit logs are not captured at the moment The fault summary is initially empty at the start of a cmgui session Clicking on the Refresh button retrieves and displays the list of faults The values displayed persist in the subtab until a click of the Re resh button retrieves the latest fault logs The values are not refreshed automatically nitoring View Bookmarks Help a3 node002 Zlusters irigh
38. o The qos is set O Bright Computing Inc 14 Nexus Switch Configuration configure terminal class map type qos classl match cos 2 policy map type gos my_qos_policy class type qos class1 set qos group 1 The running qos can be displayed show running config ipqos O Bright Computing Inc UCS Firmware This chapter explains how to upgrade the CIMC and BIOS firmware of multiple C Series servers after a Bright Cluster has been installed that is after the head node and compute nodes are in the status UP and are running Bright Cluster Manager installed on UCS comes pre installed with the latest Cisco UCS firmware packages that were available at the time of installation The firmware files are placed in a shared location accessible to all the nodes cm shared apps hwvendor ciscoucs If different firmware files are to be used then placing them in that location is recommended 4 14 Head Node Firmware Upgrade The example that follows uses the head node C Series server model C220 The variables CIMC FIRMWARE and BIOS FIRMWARE are set in this example to point to firmware files for Cisco UCS version 1 5 4 The path and names of the files must be changed if the firmware files are in a different location The examples are based on where the files are located on a default Bright Cluster Manager installation 4 1 1 CIMC Firmware Update The head node CIMC firmware can be upgraded as follows Example root
39. ormance latform default Number of Enabled Cores DRAM clock throttling Balanced X Enhanced Intel R Speedstep Technology EI DRAM refresh rate 2x Intel R Turbo Boost Technology enabled Low Voltage DDR mode power saving mode v Load UCS XML from file Show Live Import from Category Figure 5 3 Edit BIOS tokens using cmgui 5 2 VIC Adapters The VIC Adapters subtab as seen in figure 5 4 shows the list of physical vNIC and vHBA interfaces that have identified themselves to the system along with a summary of their settings Properties of a physical interface cannot be modified but vNIC and vHBA interface properties can be modified w Bookmarks Help Bright 7 0 UCS Cluster Model UCSC C240 M3S Firmwares Locator LED Q Vendor Cisco Systems Inc CIMC 1 5 4 Storage Status Serial FCH1617VOUL BIOS C240M3 1 5 4f 0 Build Date 11 13 2013 Power Supplies UUID 54F92905 9245 40CC 878B AC483AB455A4 Lom 2 1 2 84 Temperature vNICs 2 Storage 2 120 234 1471 Fans e vHBAs 2 Server Status VIC Adapters Physical Interfaces Port ID MAC Address Link State Transport Admin Speed Operating Spee 0 BC 16 65 30 AF BE up CE 10Gbps 10Gbps d BC 16 65 30 AF BF down CE 10Gbps Virtual Ethernet Interfaces vNICs _ Namew MAC Adde YM wi mes UplimkP v Gs VL vi VLAN Mo vi iSCSIB Chen Y _ Port Prof Wi Uplink Failov vii _WWPN MIO WANN ov UplinkPot vw 20 00 BC 16 65
40. s ha Node Categories Bios Profile VNIC Profile vHBA Profile NTP Settings Syslog Settings a v Head Nodes Hostname A BIOS Fir wi CIMC Fir w Storage Fir w vNICsw vHBAsw Dot si Ww each node002 node001 E bright70 ucs Console redirection LOM and PCle Slots Configuration b Racks b E Chassis Redirection EN LOM port0 state gt Cj Virtual SMP N Baud rate v LOM port1 state v Nodes d s node001 Flow control LOM ports all state node002 P C3 Cloud Nodes Putty key pad y PCI option ROMs e ER gt 3 MIC Nodes 1 AS P 4 GPU Units KVM Load UCS XML from file gt Other Devices Figure 5 11 The Category UCS Tab While in the UCS tab of the category subtabs allow UCS configuration just as for individual nodes So BIOS settings section 5 1 VIC adaptors section 5 2 NTP settings section 5 3 syslog settings section 5 4 fault logging section 5 5 and importing a UCS profile section 5 7 can all be configured at the just like was done for at the node level The difference being that the configuration profile thus created is applied at the category level and can be overridden at node level Launching a KVM console section 5 6 is not included in the preceding list of category configuration settings This is because having the KVM button work directly at the category level to open up many consoles for
41. sing vpCoreMultiProcessing all lt biosVfIntelVirtualizationTechnology vpIntelVirtualizationTechnology enabled lt biosVfIntelVTForDirectedIO vpIntelVTDCoherencySupport enabled vpIntelVTForDirectedIO enabled vpIntelVTDATSSupport enabled biosVfCPUPerformance vpCPUPerformance enterprise biosVfHardwarePrefetch vpHardwarePrefetch enabled gt biosVfAdjacentCacheLinePrefetch vpAdjacentCacheLinePrefetch enabled gt biosVfDCUPrefetch vpStreamerPrefetch enabled vpIPPrefetch enabled gt lt biosVfDirectCacheAccess vpDirectCacheAccess enabled lt biosVfCPUPowerManagement vpCPUPowerManagement custom biosVfEnhancedIntelSpeedStepTech vpEnhancedIntelSpeedStepTech enabled gt lt biosVfIntelTurboBoostTech vpIntelTurboBoostTech enabled lt biosVfProcessorC6Report vpProcessorC6Report disabled gt lt biosVfProcessorC1E vpProcessorClE disabled lt biosVfPStateCoordType vpPStateCoordType HW ALL gt biosVfCPUFrequencyFloor vpCPUFrequencyFloor enabled gt biosVfCPUEnergyPerformance vpCPUEnergyPerformance custom MEMORY biosVfSelectMemoryRASConfiguration vpSelectMemoryRASConfiguration maximum performance lt biosVfDRAMClockThrottling vpDRAMClockThrottling performance gt biosVfNUMAOptimized vpNUMAOptimized enabled biosVfLvDIMMSupport vpLvDDRMode performance mode biosVfDramRefresh
42. t 7 0 U ES Bright 7 0 UCS Cluster Model Firmwares LocatorLED Vendor CIMC Storage Status Serial BIOS Power Supplies Q UUID Lom Temperature vNICs Storage Fans Q vHBAs PCI Server Status S Nr slog Settings Fauit Log UU Time Severity Code DN Probable cause Description E Vinual S Fri Jan 24 16 26 21 warning F0996 sys rack unit 1 equipment degraded Storage Local disk 2 is degraded please H Nodes board storage SAS check if rebuild or copyback of drive is required node SLOT MEZZ pd 2 ij Cloud N MIC Nodes lj GPU Units H Other De Node Gr E le A 2 Figure 5 9 CIMC Fault Logs In cmgui 5 6 The KVM Button The CIMC provides a KVM console The KVM console is an interface that emulates direct keyboard video and mouse connections to the server It also allows mapping physical device locations to virtual drives that are accessible during the KVM session A KVM console is launched within cmgui by clicking the KVM button The button is found on the status bar of the UCS configuration settings subtabs of each node O Bright Computing Inc 5 7 Importing UCS Settings From An XML File 27 5 6 1 KVM Console Requirements The following must be installed on the host that is running cmgui in order for cmgui to launch the KVM console JRE Java Runtime Environment version 1 5 0 or greater Java Web Start application IcedTea WebStart on Linux
43. tion Chapter 2 describes a simplified deployment of a Bright UCS cluster that should serve the most common needs After deployment interaction with the rack servers is performed through the CIMC XML API The CIMC XML API presents all information from a rack server as a hierarchical tree comprising several managed objects and classes This information is stored as properties of a node or a category of nodes in Bright Cluster Manager These properties can then be modified monitored and managed using standard Bright Cluster Manager tools such as the cluster management GUI cmgui section 2 4 of the Administrator Manual or the cluster management shell cmsh section 2 5 of the Administrator Manual Bright Cluster Manager allows the administrator to modify BIOS tokens and BIOS boot order modify Cisco VIC Adapter and interface properties assign and modify usNIC profiles for Cisco VIC interfaces modify Fibre Channel Adapter and Interface properties launch remote KVM console to nodes modify network related settings of a CIMC such as NTP and syslog monitor metrics related to power supply fans temperature voltage and storage automatically configure CIMC user and network Chapter 5 explains how to accomplish the preceding tasks in detail 1 3 Cisco usNIC The Cisco user space NIC usNIC is a vital feature that improves the performance of MPI applications running in an HPC environment The MPI applications bypass the kern
44. utomatically reboot after biosup has completed n some cases the hosts do not power on automatically The hosts must then be powered on manually for example via cmgui 4 1 5 BIOS Firmware Version View The head node BIOS firmware can be viewed as follows Example root bright71 cm shared apps hwvendor ciscoucs current firmware b iosup i BIOS Update Utility v2 Build 8 Copyright c 2013 Cisco Systems Inc All rights reserved Vendor Cisco Systems Inc Version C240M3 1 5 4 0 111320130505 StartAddr 61440 0x f000 BIOSReleaseDate 11 13 2013 BIOS RomSize 127 0x7 BIOS BIOSCharacteristics 1066113152 0x3f809880 4 2 Compute Node Firmware Upgrade The following example assumes that the compute nodes use the C Series server model C220 and that the variables CIMC FIRMWARE and BIOS FIRMWARE point to the respective firmware files The paths and names of the files used in the example are defaults and may need adjustment if the firmware locations are elsewhere 4 2 1 CIMC Firmware Update The compute node CIMC firmware can be updated as follows Example O Bright Computing Inc 4 2 Compute Node Firmware Upgrade 17 root bright71 cmsh bright71 device bright71 device pexec b n node001 node002 export CIMC FIRMWAREN cm shared apps hwvendor ciscoucs current firmware c220 cimc upd pkg cV 220 m3 cimc full 1 5 4 bin service ipmi start cm shared apps hwvendo r c
45. wing procedure installs the compute nodes Ensuring the switch has been configured correctly section 2 2 Ensuring that nodes are configured to boot from the network in their BIOS settings Booting the compute nodes servers Carrying out the compute node installation The Bright Cluster Manager quick start installation guide section 1 3 of the Installation Manual has a simplified guide of this process for the adminis trator 2 5 Firmware Upgrade On Servers If the firmware for different components in Cisco UCS servers needs to be upgraded then Chapter 4 UCS Firmware has instructions on downloading and upgrading the firmware for all servers in the cluster 2 6 Post Installation Configuration Bright Cluster Manager can configure and manage all important components of a Cisco UCS server from the UCS tab in cmgui or from the ucs submode in cmsh This is discussed in the following sections O Bright Computing Inc 10 UCS Cluster Deployment Procedure Chapter 5 possible configuration options for the head node and individual compute nodes are described as well as for node categories Appendix 6 an overview of all metrics that are monitored from the CIMC of a server is provided 2 7 Using usNIC After a Bright UCS cluster has been deployed Cisco usNIC is configured and ready to use out of the box Section 2 7 1 explains how to verify the usNIC installation Section 2 7 2 describes how to verify MPI communi
Download Pdf Manuals
Related Search