Home

Sun Netra CP3060 Blade Server Userճ Guide

1. MPC boots 2 3 4 5 IPMC enables all voltage rails and proceeds if DC OK 6 7 MPC becomes operational 8 IPMC asserts SYSTEM POK ENABLE in FPGA CSRs Chapter 5 Hardware and Functional Descriptions 5 25 9 SYSTEM POK to JBus to PCI E bridge F PWR GOOD is released causes CPU hard reset and vBSC commences with host reset sequence 0 9 5 3 1 5 3 2 5 26 Power The following list provides an overview of the Sun Netra CP3060 blade server power rails m Sun Netra CP3060 blade server derives dual 48V power from the ATCA backplane m Sun Netra CP3060 blade server has on board or power converters for the following a 48V to 12V 3 3V standby 2 5V standby 1 2V standby m 12V to SV 3 3V 2 5V 1 8V 1 5V 1 2V 1 1V 1 0V 0 9V Power Input The Sun Netra CP3060 blade server is powered from dual redundant 48 V power supply inputs from the backplane power connector It takes redundant 48V as input and derives the other necessary power using DC DC converters The design complies to the ATCA power distribution specification such as m Inrush current limiting hot plug capability m Input under voltage over voltage and transient protection m EMI filtering for conducted emissions m Isolated Management Power m External holdup energy storage capacitors to meet OV transient for 5msec m A B Feed loss alarm Fuses As required by the PICMG 3 0 R1 0 specification the Sun Netra CP3060 blade server
2. TABLE 5 1 IPMB L Addresses Device IPML L Address AMC GA 2 0 AMC slot 0x72 Gnd Gnd NC Reset MMC IPMC also provides individual reset ENABLE pin on the AMC connector to the AMC slot by logically AND ing the Reset MMC with the AMC PS1 input Reset MMC is driven by the H8S GPIO pins AMC Module Presence The PS1 signal on the AMC connector provides presence indication This signal is wired to an H8 GPIO pin AMC Hot Plug Module Insertion PCI E When an AMC module is installed the IPMC is signaled and will initially enable management power 3 3V to the module The IMPC will then have the ability to communicate with the module through the IPMI interface IPMB_L If the shelf manager controller ShMC determines that the module is supported it signals the IPMC to enable payload power 12V to the module The IPMC controller will also assert HP_PRSNT and HP_BUTTON ATTN inputs to the PCI E switch hot plug controller This assertion will cause the switch to generate a hot plug interrupt or Presence detect change to the processor to indicate that the blade server is hot inserted This triggers a system event in the Solaris OS which causes an automatic connect configure operation When this operation completes the processor asserts the HW_PWR_LED output pin of the PCI E switch hot plug controller This output is wired to the H8 GPIO pins AMC Hot Plug Module Removal PCI E An operator can initiate module rem
3. The Zone 2 connector provides the following signals m Two 1000BASE T Ethernet Base Fabric channels m Two 1000 SERDES ports on the Extended Fabric interface m Synchronization Clock interface supporting CLK1A B and CLK2A 2B FIGURE A 5 shows the Zone 2 connectors Appendix A Physical Characteristics A 9 FIGURE A 5 Zone 2 Connectors 410101 J10001 TLTR Z 1_ T TR HG Hf GEE EE HG H m m Jj Feld d Fd de G m lim m in lom mm mm m Gils m m wll mim mim m FG F f 7 7 7 FG rmm 7 77 EIE Ellm mm wll me Lf Ejim mim wll u il c asi PEB DG E a a emo RER Pe CFEFEFEEEEEL AAAH C m m m x mo oe me ont C i m m m wt lw l m ain oie BG glee falar oc f is BG gr fm fm fm fm m fm fm fa Fm Allm w x x oie oe oe BG A m lal a Uae LY JJ 0 uL ELM BEA Z 09876543 2 1 10 9 8 7 6 5 4 3 2 1 TABLE A 5 gives the Zone 2 J10101 connector pin assignments TABLE A 5 Zone 2 J10101 Connector Pin Assignments Row A B c D E F G H 1 n c n c P23 F2 P23 F2 n c c P23 F2 P2 F2 TERM RX2 TERM RX2 TERM RX3 TERM RX31 2 GBE EXTB T GBE EXTB T GBE EXTBR GBE EXTBR n c n c P23 F2 P23 F2 XO P X0 N XO P X0 N TERM RX1 TERM_RX1 3 n c n c P23F1 P23 F1_ n c n c P23 F1 P23 F1 TERM RX2 TERM RX2
4. J10202 J10201 J10301 IG FI SS E RIE G Diy a C G B m m nmm om fax fa a fu fac A o mm m mmy mMM N 109 8 7 6 5 4 3 2 1 10 9 8 7 6 5 43 2 1 109 8 7 6 5 4 3 2 1 TABLE A 7 gives the Zone 3 J10202 connector pin assignments TABLE A 7 Zone 3 J10202 Connector Pin Assignments Row A B BG C D DG E F FG 1 PCIE Tx7 p PCIE Tx7 n GND PCIE Rx7 p PCIE Rx7 n GND n c n c GND 2 PCIE Tx6 p PCIE Tx6 n GND PCIE Rx6 p PCIE Rx6 n GND n c n c GND 3 PCIE Tx5 p PCIE Tx5 n GND PCIE Rx5 p PCIE Rx5 n GND n c n c GND 4 PCIE Tx4 p PCIE Tx4 n GND PCIE Rx4 p PCIE Rx4 n GND n c n c GND 5 PCIE Tx3 p PCIE Tx3 n GND PCIE Rx3 p PCIE Rx3 n GND n c n c GND 6 PCIE Tx2 p PCIE Tx2 n GND PCIE Rx2 p PCIE Rx2 n GND n c n c GND 7 PCIE_Tx1_p PCIE_Tx1_n GND PCIE_Rx1_p PCIE Rx1 n GND n c n c GND 8 PCIE Tx0 p PCIE Tx0 n GND PCIE Rx0 p PCIE Rx0 n GND n c n c GND 9 n c n c GND n c n c GND n c n c GND 10 n c n c GND n c n c GND n c n c GND A 12 Netra CP3060 Board User s Guide April 2009 TABLE A 8 gives the Zone 3 J10201 connector pin assignments TABLEA 8 Zone 3 10201 Connector Pin Assignments Row A B BG Cc D DG E F FG 1 AMCO IO1 AMCO IO2 AMCO IO3 AMCO IO4 AMCO IO5 AMCO IO6 GND 2 AMCO IO7 AMCO IO8 AMCO IO9 AMCO IO10 AMCO IO11 AMCO IO12 3 AMCO IO24 AMCO IO23 AMCO IO22 AMCO IO21 AMCO IO20 AMCO IO19 4 AMCO IO18 AMCO IO17 AMCO IO16 AMCO IO15 GND AMCO IO14 AMCO IO13 5 AMCO IO36 AMCO IO35 GND AMCO IO34 AMCO IO33 A
5. Manual system reset Setting the OBP auto boot variable to false causes the system to stop at the ok prompt the next time the blade server is reset Netra CP3060 Board User s Guide April 2009 4 3 2 4 3 3 Caution Obtaining the ok prompt suspends all application and operating system software After you issue firmware commands and run firmware based tests from the ok prompt the system might not be able to resume where it left off If possible back up system data before starting accessing the ok prompt Also exit or stop all applications and warn users of the impending loss of service For information about the appropriate backup and shutdown procedures see Solaris system administration documentation Auto Boot Options The system firmware stores a configuration variable called auto boot which controls whether the firmware will automatically boot the operating system after each reset The default setting for Sun platforms is true Normally if a system fails power on diagnostics auto boot is ignored and the system does not boot unless an operator boots the system manually An automatic boot is generally not acceptable for booting a system in a degraded state Therefore the Sun Netra CP3060 server OpenBoot firmware provides a second setting auto boot on error This setting controls whether the system will attempt a degraded boot when a subsystem failure is detected Both the auto boot and auto boot on error switches
6. The H8S provides a local IPMI interface wired to the AMC slot IPMB L The AMC IPMI interface is isolated through LTC4300A and controlled by H8 GPIO pins ATCA Hot Swap Latch Hot swap is supported by monitoring of the hot swap handle switch The handle switch goes directly to one of the GPIO pins on the H8S Payload Shutdown in Response to Hot Swap The hardware supports both non graceful shutdown and graceful shutdown of payload in response to a hot swap event In case of non graceful shutdown firmware on the IPMC will disable the voltage rails before turning the blue LED on In case of graceful shutdown IPMC writes to the SHUTDOWN_REQUEST bit in the FPGA which will cause an interrupt to be sent to the JBus to PCI E bridge and the MPC When shutdown is complete MPC signals this to IPMC through a status bit in the FPGA LEDs The Sun Netra CP3060 blade server supports three LEDs compliant with ATCA specification m Green LED Healthy status m Amber LED Fault condition Chapter 5 Hardware and Functional Descriptions 5 19 5 1 6 6 5 1 6 7 5 1 6 8 5 20 m Blue LED Indicates that blade server is safe for removal hot swap activity when on and hot swap in progress when blinking The LEDs are controlled by H8 Power Control The BMR HSS is able to control enable disable power rails to the payload Extended Interface E keying Control The BMR H8S provides E keying for the Extended interfaces to the midplane
7. Thus the addresses listed here all represent the read form of the given device s address that is Addr lt 0 gt 0 These devices are solely accessed by the H8 on the motherboard MB if not otherwise noted in TABLE 5 3 Note that a number of addresses are the same This is possible because they are on different PC segments controlled by the DC MUX The Location column indicates the FRU on which the physical device resides The Standby column indicates whether this device is readable when the system is in standby mode that is powered off but receiving its main power source In other words a YES in this column indicates that the H8 which is powered from the systems standby power rail can access this device at all times whether the system is powered on or off 5 1 6 10 System Monitor and Thresholds The system monitor ADM1026 is a highly integrated and highly sophisticated multi function system monitor including Analog to Digital Conversion ADC Digital to Analog Conversion DAC Temperature Sensing and GPIO pins The Analog Devices ADM1026 is used for system monitoring functions The ADM1026 is interfaced with the IPMC and the IPMC firmware is responsible for monitoring these sensors Voltage Monitoring The ADM1026 measures most of the board voltages The ADM1026 has one temperature sensor embedded in the device itself and supports two remote sensing channels requiring external diodes for temperature sensing These remote sensors
8. load base auto boot network boot argument S boot command boot file boot device true false 0 15 true false true false true false 0 n 0 n 9600 8 n 1 virtual conso le screen virtual conso le keyboard true false 0 n true false protocol key value variable name variable name variable name false false false true 80 34 9600 8 n 1 virtual conso le virtual conso le false 16384 true none boot none disk net If true include name fields for plug in device FCodes SCSI ID of the Serial Attached SCSI controller If true use custom OEM logo otherwise use Sun logo If true use custom OEM banner If true enable ANSI terminal emulation Sets number of columns on screen Sets number of rows on screen Serial management port baud rate bits parity stop handshake The serial management port only works at the default values Power on output device Power on input device If true boot automatically after system error Address If true boot automatically after power on or reset Arguments to be used by the PROM for network booting Defaults to an empty string network boot arguments can be used to specify the boot protocol RARP DHCP to be used and a range of system knowledge to be used in the process For further information see the eeprom 1M man page or your Solaris Reference Manual Action following a boo
9. PICMG mandates that the software hooks for E keying be present but it does not mandate that anything needs to be done in hardware in response to this There is no hardware support for E keying in the base interfaces E keying for the Extended interfaces is accomplished by setting the SIG DET lines active PC Architecture FIGURE 5 6 is a block diagram of the I2C architecture The subsequent subsections contain a description of the I2C Devices and Address map The MPC has default access to the MPC RUN DC bus Netra CP3060 Board User s Guide April 2009 a 2 c E m e o N O Es o o gt g aan o ag H t 8 DIMMO 0 xA0 DIMMO 1 xA2 DIMM1 O xA4 DIMM1_1 I2C_DDRL xA6 DIMM2 O xA8 DIMM2 1 xAA DIMM3 O xAC DIMM3 1 12C DDRL xAE PPC RUN 12C SMM150 CPU_VCore x50 CY28401 xDC DS1338 xDO OBP HV xA4 SP NVRAM xA2 SUN FRU ID xA8 FRU PC SCC MAC xA6 RTM PC RTM FRU ID xAA PPC_I2C ADM1032 12C Selector PPC855 x4C ADM1026 x5C IPM1 FRU xAC SEL EEPROM ki DC LM80 x5E DC FRU xAE SPI BCM5388 AMC Slot PMI_L x72 FPGA DC Enable IPMI A IPMI B LEDs IMPC_I2C SIPL Chapter 5 Hardware and Functional Descriptions Handle switch 5 21 5 1 6 9 PC Device Address Map TABLE 5 3 shows the complete I C address map for the PC bus TABLE 5 3 I2C Address Map Standby Address Location Po
10. The following command is a single long entry Do not press the Return key until you have typed the entire text string shown in the following command TABLE 2 smosservice add u root p root password x mediapath image directory x platform sparc sun4v Solaris_n x cluster SUNWCXall x locale locale Where m root password is the root password for the installation server m image directory is the path to the directory where the Solaris install image is stored m Nis the Solaris OS version you are using m locale is the locale that you want to use Refer to the smosservice 1M man page for more information and options For example TABLE 3 smosservice add u root p root password x mediapath export install x platform sparc sun4v Solaris 10 x cluster SUNWCXall x locale en US Where m root password root password m image directory export install m locale en_US m n 10 for Solaris 10 3 4 Netra CP3060 Board User s Guide April 2009 4 Download and install additional patches Refer to the Sun Netra CP3060 Blade Server Product Notes 819 4966 for the latest information on the patches available for the Sun Netra CP3060 blade server The document can be downloaded from the following web site http www sun com documentation Follow the instructions in the Sun Netra CP3060 Blade Server Product Notes 819 4966 for downloading and applying patches to a diskless clients boot serve
11. When the OS is at run level 0 the ok prompt appears This prompt indicates that the OpenBoot firmware is in control of the system There are a number of scenarios under which OpenBoot firmware control can occur By default before the operating system is installed the system comes up under OpenBoot firmware control m When the auto boot OpenBoot configuration variable is set to false the system boots to the ok prompt m When the operating system is halted the system transitions to run level 0 in an orderly way m When the operating system crashes the system reverts to OpenBoot firmware control m During the boot process when there is a serious hardware problem that prevents the operating system from running the system reverts to OpenBoot firmware control m When a serious hardware problem develops while the system is running the operating system transitions smoothly to run level 0 m When the OS is deliberately placed under the OpenBoot firmware control in order to execute firmware based commands Getting to the ok Prompt There are different ways of reaching the ok prompt The methods are not equally desirable See TABLE 4 2 for details TABLE4 2 Ways of Accessing the ok Prompt Access Method What to Do Graceful shutdown of From a shell or command tool window issue an appropriate the Solaris OS command for example the shutdown or init command as described in Solaris system administration documentation
12. ident system 1 18 05 06 27 SMI SVRA 1 5 SYSTEM SPECIFICATION FILE set pcie pcie aer ce mask 0x1 3 If the entry is not there add it Use an editor to edit the etc system file and add the entry 4 Reboot the server 3 4 3 4 1 Installing Diskless Clients The following procedures describe how to create a boot server for diskless clients and how to add new diskless clients to the patched boot server For additional instructions on installing diskless clients refer to the appropriate Solaris Documentation Collection at the Sun Documentation web site at http www sun com documentation You must have a superuser password on your diskless server to perform the following tasks Creating a Boot Server for Diskless Clients Note This procedure sets up a boot server by starting the operating environment services required for diskless clients Once you have set up the boot server see Section 3 4 2 Adding a Diskless Client on page 3 5 for instructions on adding diskless clients to the boot server 1 Verify that the IP addresses for all other network interfaces on the boot server have corresponding hostnames in the hosts database Chapter 3 Software Installation 3 3 2 Log in to the network server as superuser and change to the usr sadm bin directory TABLE 1 cd usr sadm bin 3 Use the smosservice command to add boot services to the installation server Note
13. 2 4 1 1 Installing a DDR 2 DIMM Memory Module 2 7 2 4 1 2 Removing a DDR 2 DIMM Memory Module 2 8 2 4 2 Installation of Optional Compact Flash Card 2 9 2 4 3 Configuring Rear Transition Module Hardware 2 11 25 Installing the Sun Netra CP3060 BladeServer 2 11 2 5 1 Installing the Sun Netra CP3060 Blade Server With a Rear Transition Module 2 11 2 5 1 1 Installing a Rear Transition Module 2 12 2 5 2 Installing the Sun Netra CP3060 Blade Server 2 15 2 6 Connecting External I O Cables 2 17 27 Installation of Optional AMC Modules 2 18 2 7 0 1 Installing an Optional AMC Module 2 19 3 Software Installation 3 1 3 1 Operating Systems and Patches 3 1 3 2 Firmware Updates 3 2 33 Mandatory etc systemFile Entry 3 2 v To Check and Create the Mandatory etc systemFile Entry 2 3 4 Installing Diskless Clients 3 3 3 4 1 Creating a Boot Server for Diskless Clients 3 3 3 4 2 Adding a Diskless Client 3 5 35 Downloading and Installing SunVTS Software 3 7 4 Firmware and Blade Server Management 4 1 Netra CP3060 Board User s Guide April 2009 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 System Firmware 4 1 Power On Self Test Diagnostics 4 2 4 21 POST Test Coverage 4 2 4 2 2 POST Diagnostic and Error Message Format 4 3 OpenBoot Firmware 4 3 43 1 Getting to the ok Prompt 4 4 43 2 Auto Boot Options 4 5 4 3 3 OpenBoot Commands 4 5 4 3 3 1 probe ide Command 4 6 4 3 3 2 show devs Command 4 6 4 3 8 3 Checking Network Using watch net and watch
14. FIGURE A 4 shows the pin assignments Appendix A Physical Characteristics A 7 FIGURE A 4 Power Distribution Connector Zone 1 P10 TABLE A 4 lists the power connector pin assignments TABLE A 4 Power Distribution Connector Pin Assignments Pin Number Name Description 1 Reserved Reserved 2 Reserved Reserved 3 Reserved Reserved 4 Reserved Reserved 5 HAO HAO Hardware Address Bit 0 6 HA1 HA1 Hardware Address Bit 1 7 HA2 HA2 Hardware Address Bit 2 8 HA3 HA3 Hardware Address Bit 3 9 HA4 HA4 Hardware Address Bit 4 10 HA5 HA5 Hardware Address Bit 5 11 HA6 HA6 Hardware Address Bit 6 12 HA7 P HA7 P Hardware Address Bit 7 Odd Parity Bit 13 SCL_A IPMB Clock Port A 14 SDA_A IPMB Data Port A 15 SCL_B IPMB Clock Port B 16 SDA_B IPMB Data Port B 17 Unused 18 Unused A 8 Netra CP3060 Board User s Guide April 2009 A 4 5 TABLEA 4 Power Distribution Connector Pin Assignments Continued Pin Number Name Description 19 Unused 20 Unused 21 Unused 22 Unused 23 Unused 24 Unused 25 SHELF GND Shelf Ground 26 LOGIC GND Logic Ground 27 ENABLE B Enable B 28 VRIN A Voltage Return A 29 VRIN B Voltage Return B 30 EARLY A 48V Early A 31 EARLY B 48V Early B 32 ENABLE A Enable A 33 48V A 48V A 34 48V B 48V B Data Transport Connector Zone 2 The data transport connector consists of one 120 pin HM Zd connector labeled P23 with 40 differential pairs This is called the Zone 2 connector
15. Remote server Netra CP3060 RTM Serial connection onneECUo Terminal console Ethernet Ng connection NS li ATCA shelf Netra CP3060 blade server Diskless client that boots through network from a remote server 1 3 1 AMC Modules The Sun Netra CP3060 blade server has one AMC slot to provide additional I O to the front panel The AMC slot supports AMC compatible SATA disk drives and PCI E cards See Section A 4 2 AMC Connector on page A 5 for more information 1 3 2 Rear Transition Module The optional Sun Netra CP3060 RTM installs into the rear of the ATCA enclosure opposite the Sun Netra CP3060 blade server see FIGURE 1 5 The RTM connects with the host node board s Zone 3 rear I O connectors and includes a serial port and 10 100BASE T Ethernet port both with RJ 45 connectors see FIGURE 1 4 FIGURE 1 4 Sun Netra CP3060 Rear Transition Module 1 8 Netra CP3060 Board User s Guide April 2009 Zone 3 connectors Ethernet port Serial port FIGURE 1 5 shows the physical relationship between the blade server the rear transition module and the midplane in a typical ATCA system Chapter 1 Introduction to the Sun Netra CP3060 Blade Server 1 9 FIGURE 1 5 Relationship of the Sun Netra CP3060 Blade Server Midplane and RTM Zone 3 connectors Netra CP3060 RTM Netra CP3060 blade server installed from rear installed from front 7 GR PIG AG EG c2 ja mam b
16. TERM RX3 TERM_RX3 4 GBE EXTA T GBE EXTA T GBE EXTAR GBE EXTA R n c n c P23 F1 P23 F1 X0 P X0 N X0 P X0 N TERM RX1 TERM_RX1 5 GBE BASE GBE BASE GBE BASE GBE BASE GBE BASE GBE BASE GBE BASE GBE BASE Z2A TRO P Z2A TRO N Z2A TR1 P Z2A TRI N Z2A TR2 P Z2A TR2 N Z2A TR3 P Z2A TR3 N 6 GBE BASE GBE BASE GBE BASE GBE BASE GBE BASE GBE BASE GBE BASE GBE BASE Z2B TRO P Z2B TRO N Z2B TR1 P Z2B TRI N Z2B TR2 P Z2B TR2 N Z2B TR3 P Z2B TR3 N 7 n c n c n c n c n c n c n c n c 8 n c n c n c n c n c n c n c n c 9 n c n c n c n c n c n c n c n c 10 n c n c n c n c n c n c n c n c A 10 Netra CP3060 Board User s Guide April 2009 TABLE A 6 gives the Zone 2 J10001 connector pin assignments TABLE A 6 Zone 2 10001 Connector Pin Assignments Row A B Cc D E F G H 1 CIK1A P CIK1A N Clk1B_P Clk1B_N CIk2A_P Clk2A_N Clk2B_P Clk2B_n 2 n c n c n c n c n c n c n c n c 3 n c n c n c n c n c n c n c n c 4 n c n c n c n c n c n c n c n c 5 n c n c n c n c n c n c n c n c 6 n c n c n c n c n c n c n c n c 7 n c n c n c n c n c n c n c n c 8 n c n c n c n c n c n c n c n c 9 n c n c n c n c n c n c n c n c 10 n c n c n c n c n c n c n c n c A 4 6 RTM Connector Zone 3 The Sun Netra CP3060 blade server provides all the I O connections for rear access through the Zone 3 RTM connector The connector view and the pinouts for the Zone 3 connectors are shown in FIGURE A 6 Appendix A Physical Characteristics A 11 FIGURE A 6 Zone 3 Connectors
17. The organization of the memory subsystem with regard to the connections between the UltraSPARC T1 processor and the memory DIMMs is shown in FIGURE 5 3 Because the UItraSPARC T1 processor accesses memory with a 128 bit data path plus ECC DIMMs must be installed in pairs The DIMMs are standard DDR 2 Registered VLP DIMMs Specific vendors DIMMs are qualified by Sun and supplied as orderable options for system configuration Chapter 5 Hardware and Functional Descriptions 5 7 5 1 2 2 5 8 FIGURE 5 3 DDR Memory Diagram Addresses lt 13 0 gt BA lt 1 0 gt RAS_L CAS_L WE L DIMM Pair 0 PG e DDR DIMM L I gt DDR DIMM f UltraSPARC T1 128 Data 16 ECC Memory Subsystem RAS Features The ECC scheme employed by the UltraSPARC T1 memory controller provides single bit correct double bit detect ECC protection across the 128 bits of data in each bank of memory Also each DIMM provides an industry standard 256 byte Serial Presence Detect SPD PROM of which 128 bytes are available to the system for dynamic FRU data Plans are being made to use this 128 bytes for dynamic FRU data such as soft error rate information The Sun Netra CP3060 blade server also supports the Chip kill detect ECC scheme allowing the detection of up to 4 bits in error as long as they are not in the same DRAM This is made possible by limiting the type of DDR 2 memor
18. are used for measuring CPU die temperature Chapter 5 Hardware and Functional Descriptions 5 23 The ADM1026 measures the voltages listed in TABLE 5 4 The ADM1206 and H8 firmware monitors these voltages When the voltages are within regulations the blade server functions with no warnings When any voltage goes out of regulation beyond approximately 7percent a Critical Warning is generated and presented to the shelf manager When any voltage goes out of regulation beyond approximately 10 the H8 initiates a blade server shutdown TABLE5 4 Voltage Sensor Thresholds Non Critical Thresholds Critical Thresholds Non Recoverable Thresholds Sensor Lower Upper Lower Upper Lower Upper 12V 11 77V 12 47V 11 47V 12 73V 11 28V 12 92V 5V VCC 4 86V 5 15V 4 76V 5 25V 4 05V 5 36V 3V Main 3 24V 345V 3 17V 3 50V 3 10V 3 57V 3V Standby 3 24V 345V 317V 3 50V 3 10V 3 57V 2 5VBAT 2 38V 252V 2 31V 257V 2 27V 2 61V 1 0V 1 15V 1 10V 1 17V 1 07V 1 19V 1 05V 1 21V 1 1V 1 05V 1 12V 1 03V 1 14V 1 00V 1 16V 1 2V 1 15V 1 22V 1 12V 1 24V 1 10V 1 26V 1 5V 1 43V 1 52V 1 40V 1 55V 1 37V 1 58V 0 9V VTTL 0 86V 0 91V 0 84V 0 93V 0 83V 0 94V 0 9V VTTR 0 86V 0 91V 0 84V 0 93V 0 83V 0 94V 1 8V DDR2L 1 72V 1 83V 1 69V 1 86V 1 65V 1 90V 1 8V DDR2R 1 72V 1 83V 1 69V 1 86V 1 65V 1 90V VCCP 2 5V 2 38V 254V 2 34V 2 58V 2 29V 2 63V 1 2V Standby 0 96V 1 46V 0 85V 1 50V 0 81V 1 56V AMCSite1 12V 12 609V 12 84V 13 00V AMCSite1 3 3V 3 11V 3 35V 3 01V 3 39V 2 9
19. 1 1 and FIGURE 1 2 show the Sun Netra CP3060 blade server Chapter 1 Introduction to the Sun Netra CP3060 Blade Server 1 3 FIGURE 1 1 Sun Netra CP3060 Blade Server Front Panel 1 Top latch 6 Out of Service LED yellow 2 AMC Hard Disk Drive optional 7 OK LED green 3 Ethernet port A RJ 45 8 Hot Swap LED blue 4 Ethernet port B RJ 45 9 Hot Swap switch and bottom latch 5 Serial port RJ 45 ttya 1 4 Netra CP3060 Board User s Guide April 2009 FIGURE 1 2 Sun Netra CP3060 Blade Server Top View AMC slot Compact enclosure Flash connector AMC HDD optional posocaco eecoceccscocecos inmrmmmmmomUETIS posocoecooscoscooscoscooscosq q posccecoceccecocecescoceccs poscoecoceccecoceccecocoscosG poseceecocscoscocecescoceces dD posooescooscoocooscoscoc0cOoSQ posocscoonscoscooscosco0scosd baaaosooo ecoscoc scoscocscosQpoo LI poscascogceccecocebtscoce Processor under heat sinks OE Zone 2 connectors A J Zone 1 power connector Chapter 1 Introduction to the Sun Netra CP3060 Blade Server 1 5 1 6 TABLE 1 1 lists features of the blade server TABLE 1 1 Feature Summary for the Sun Netra CP3060 Blade Server Feature CPU Memory Power requirement PICMG compliance Node board support Operating system Internal I O connections to ATCA midplane External I O AMC I O IPMI system management Hot swap support Description One Ultr
20. 2 20 AMC device 2 18 collecting network information 2 4 DIMM modules 2 5 preparation 2 3 summary 2 4 tools needed 2 2 hot swap information 4 15 LED indicator 4 16 support 1 11 l I O connections front panel 1 7 IDE bus 4 6 init Solaris command 4 4 installation AMC device 2 18 DIMM modules 2 7 overall procedure 2 4 preparation 2 3 tools needed 2 2 installing rear transition modules 2 12 integrated drive electronics IDE Intelligent Platform Management controller IPMC 4 1 Internet Assigned Numbers Authority IANA B 1 IPMC Intelligent Platform Management controller 4 1 IPMI system management 1 6 L LED hot swap 4 16 LEDs Active A 3 Fault A 3 front panel A 3 hot swap A 3 Index 2 Netra CP3060 Board User s Guide April 2009 M MAC address 2 4 label 1 13 manual system reset 4 4 memory 1 6 DIMM module 2 5 message POST error 4 2 4 3 N netfunction NetFn B 1 O ok prompt ways to access 4 4 OpenBoot commands probe ide 4 6 show devs 4 6 OpenBoot configuration variables auto boot 4 5 P part number locating 1 13 PICL 4 17 pinouts serial port A 4 platform information and control library PICL 4 17 POST error messages 4 2 4 3 POST see also power on self test POST 4 2 power requirements 1 6 power module 5 28 power on self test POST see POST power on sequence 5 26 preparation installation 2 3 probe ide command OpenBoot 4 6 PR
21. ATCA chassis midplane Note When the RTM is used with the Sun Netra CP3060 blade server shielded cables are required for serial I O ports Unshielded cables can be used on Ethernet ports to satisfy EMI compliance standards The shields for all shielded cables must be terminated on both ends The customer can order the Sun Netra CP3060 RTM build a custom card or buy from an IHV A minimal set of I O must provide a boot path for the host board and a path for console I O to deliver commands and to read board and system status 1 10 Netra CP3060 Board User s Guide April 2009 Possible boot and console configurations are described in TABLE 1 2 Sun Microsystems provides the Sun Netra CP3060 blade server and a compatible Sun Netra CP3060 RTM The other configurations require IHV hardware TABLE 1 2 I O Configurations 1 0 Hardware Required Description Ethernet Sun Netra CP3060 blade server Default boot path uses the Ethernet port when the blade server Sun Netra CP3060 RTM runs in diskless client configuration RTM supplied as an option for rear access SATA Through the AMC connector SATA devices can be used for local booting Serial data Compact Flash Sun Netra CP3060 blade server Sun Netra CP3060 RTM Sun Compact Flash card Serial port on front panel provides a path for a console I O The serial port can be used on either the Sun Netra CP3060 blade server or Sun Netra CP3060 RTM but both ports cannot be u
22. Fabric I O Region m The signals denoted SATA are wired to two of the SATA controller ports These are port 2 and port 3 in the AMC specification part of the Common Options Region Appendix A Physical Characteristics A 5 TABLE A 3 AMC Connector Pin Assignments Basic Side Extended Side Basic Side Extended Side Pin Signal Pin Signal Pin Signal Pin Signal 85 Gnd 86 Gnd 42 12V 129 RTM TX3 N 84 12V 87 PCIE R4 N 41 AMC CONN ENABLE L 130 RTM TX3 P 83 Gnd 88 PCIE RX4 P 40 Gnd 131 Signal 82 Gnd 89 Gnd 39 SATA1 AMC TX N 132 RTM RX3 N 81 AMCCON REFCLK N 90 PCIE TXAN 38 SATA1 AMC TX P 133 RTM RX3 P 80 AMCCON REFCLK P 91 PCIE TX4 P 37 Gnd 134 Gnd 79 Gnd 92 Gnd 36 SATA1 AMC RXX N 135 RTM TX4 N 78 CLK2 N 93 PCIE RX5N 35 SATA1 AMC RXX P 136 RTM TX4 P 77 CLK2 P 94 PCIE RX5 P 34 Gnd 137 Gnd 76 Gnd 95 Gnd 33 SATAO AMC TX N 138 RTM RX4 N 75 CLK1 N 96 PCIE TX5 N 32 SATA0 AMC TX P 139 RTM RX4 P 74 CLK1 P 97 PCIE TX5 P 31 Gnd 140 Gnd 73 Gnd 98 Gnd 30 SATAO AMC_RXXN 141 RTM TX5 N 72 12V 99 PCIE RX6 N 29 SATA0 AMC RXX P 142 RTM TXx5 P 7 DC AMCCONN SDA 100 PCIE RX6 P 28 Gnd 143 Gnd 70 Gnd 101 Gnd 27 12V 144 RTM RX5 N 69 PCIE RX3 N 102 PCIE TX6 N 26 Gnd 145 RTM RX5 P 68 PCIE RX3 P 103 PCIE TX6 P 25 Gnd 146 Gnd 67 Gnd 104 Gnd 24 n c 147 RTM TX6 N 66 PCIE TX3 N 105 PCIE RX7N 23 n c 148 RTM TX6 P 65 PCIE TX3 P 106 PCIE RX7 P 22 Gnd 149 Gnd 64 Gnd 107 Gnd 21 n c 150 RTM RX6 N 63 PCIE RX2 N 108 PCIE TX7 N 20 n c 151 RTM RX6 P 62 PCIE RX2 P 109 PCIE T
23. IMPLIED WARRANTY OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE OR NON INFRINGEMENT ARE DISCLAIMED EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID Copyright O 2009 Sun Microsystems Inc 4150 Network Circle Santa Clara California 95054 Etats Unis Tous droits r serv s Cette distribution peut comprendre des composants d velopp s par des tierces parties Des parties de ce produit pourront tre d riv es des syst mes Berkeley BSD licenci s par l Universit de Californie UNIX est une marque d pos e aux Etats Unis et dans d autres pays et licenci e exclusivement par X Open Company Ltd Sun Sun Microsystems le logo Sun Netra Sun Ray le logo Netra et le logo Solaris sont des marques de fabrique ou des marques d pos es de Sun Microsystems Inc ou ses filiales aux Etats Unis et dans d autres pays Toutes les marques SPARC sont utilis es sous licence et sont des marques de fabrique ou des marques d pos es de SPARC International Inc aux Etats Unis et dans d autres pays Les produits portant les marques SPARC sont bas s sur une architecture d velopp e par Sun Microsystems Inc L utilisation de pieces detachees ou d unites centrales de remplacement est limitee aux reparations ou a l echange standard d unites centrales pur les produits exportes conformement a la legislation americaine en matiere d exportation Sauf autorisation par les autorites des Etats nis l utilisation d unites centrales
24. S lala ZW Zone 1 Zone 2 Zone 3 RTM Chapter 5 Hardware and Functional Descriptions 5 3 5 1 1 5 1 1 1 5 4 UltraSPARC T1 Processor This section provides a brief description of the salient features of the UltraSPARC T1 processor its package and its socket Overview The UltraSPARC T1 multicore processor is the basis of the Sun Netra CP3060 blade server The UltraSPARC T1 processor is based on chip multithreading CMT technology that is optimized for highly threaded transactional processing The UltraSPARC T1 processor improves throughput while using less power and dissipating less heat than conventional processor designs Depending on the model purchased the processor has four six or eight UltraSPARC cores Each core equates to a 64 bit execution pipeline capable of running four threads The result is that the 8 core processor handles up to 32 active threads concurrently Additional processor components such as L1 cache L2 cache memory access crossbar DDR2 memory controllers and a JBus I O interface have been carefully tuned for optimal performance See FIGURE 5 2 The CPU is a high performance highly integrated superscaler UltraSPARC T1 processor implementing the 64 bit SPARC V9 RISC architecture It contains a 128 Kbyte instruction cache and 64 Kbyte data cache The chip has an inbuilt 3 Mbyte L2 cache that uses a 32 byte cache line Netra CP3060 Board User s Guide April 2009 5 1 1 2 7 Multit
25. are RJ 45 connectors The controller autonegotiates to either 10BASE T 100BASE T or 1000BASE T The Ethernet connector pin numbering is shown in FIGURE A 2 FIGURE A 2 Ethernet RJ 45 Connector 12345678 TABLE A 1 shows the Ethernet connector pin assignments The 1000BASE T signal names are in parentheses TABLE A 1 Ethernet Port Connector Pin Assignments Pin Signal Name Pin Signal Name 1 TXD BI_DA 5 not used BI DC 2 TXD BI DA 6 RXD BI DB 3 RXD BI_DB 7 notused BI DD 4 not used BI DC4 8 not used BI DD Serial Port FIGURE A 3 contains the connector pin assignments for the front panel serial port A 4 Netra CP3060 Board User s Guide April 2009 A 4 2 FIGURE A 3 Front Panel Serial Port Diagram 87654321 TABLE A 2 shows the serial port connector pin assignments TABLE A 2 Serial Port RJ 45 Connector Pinouts Pin Signal Name Pin Signal Name 1 RTS 5 DCD 2 DTR 6 RXD 3 TXD 7 DSR 4 GND 8 CTS AMC Connector The Sun Netra CP3060 blade server provides a B style 170 pin connector The pinout of the AMC slot is provided in TABLE A 3 m The signals denoted RTM are wired to the Zone 3 RTM connector These are port 12 through port 20 in the AMC specification part of the Extended Options Region m The signals denoted PCI E are connected with an x8 port on the PCI E switch These are port 4 through port 11 in the AMC specification part of the
26. critical functions of the blade server and responds to commands from the shelf manager controller ShMC and reports events BMR H8S uses a serial connection to communicate with the host CPU through the service processor MPC Field Programmable Gate Array FPGA The ATCA backplane provides redundant 48V power connection and the Sun Netra CP3060 blade server derives the necessary power by using DC DC converters Standby power is generated separately from 48V and provided for hardware management circuitry The BMR H8S IPMC subsystem and the FPGA are powered from standby power FIGURE 5 1 is a block diagram of the Sun Netra CP3060 blade server Netra CP3060 Board User s Guide April 2009 FIGURE 5 1 Block Diagram D GNG DDR2 DDR2 DDR2 DDR DDR2 DDR2 DDR2 DDR2 16MB OBP Flash 4MB SDRAM 64MB XBus 2 ER MPC885 MPOBUS FPGA sel UltraSPARC T1 r ke D JBus to PCI E bridge Serial g c Port RJ 45 F o x4 x8 a Leaf A Leaf B g o 0 o PCI E switch x4 x4 x8 x4 x8 GBE NIC GBE NIC Southbridge Env 8 port switch Compact Ic H8 Flash IPMC AMC 12V slot Quad PHY DC DC e 3 al 8 g wy Ww a 83 af al m F 0 o ai m S wj ui ol o Ethernet Ethernet Q Q 2 a 8 S Port Port wil wl wy poe 2 RJ 45 HRI 45 o o 7 H FH gl FENCE Jo xa 3 i B3 a la amp 2 5 S 3 s o glo Slo 3 a a S 2
27. must be set to true to enable an automatic degraded boot To set the switches type ok setenv auto boot true ok setenv auto boot on error true Note The default setting for auto boot on error is false The system will not attempt a degraded boot unless you change this setting to true In addition the system will not attempt a degraded boot in response to any fatal nonrecoverable error even if degraded booting is enabled For examples of fatal nonrecoverable errors see OpenBoot Configuration Variables on page 9 OpenBoot Commands You type the OpenBoot commands at the ok prompt Two of the OpenBoot commands that can provide useful diagnostic information include m probe ide Chapter 4 Firmware and Blade Server Management 4 5 4 3 3 1 4 3 3 2 4 6 m show devs For a complete list of OpenBoot commands and more information about the OpenBoot firmware refer to the OpenBoot 4 x Command Reference Manual An online version of the manual is included with the OpenBoot Collection AnswerBook that ships with Solaris software probe ide Command The probe ide command communicates with all Integrated Drive Electronics IDE devices connected to the IDE bus This is the internal system bus for media devices such as the DVD drive Caution If you used the halt command or the Stop A key sequence to reach the ok prompt issuing the probe ide command can hang the system CODE EXAMPLE 4 2 shows sample output from the prob
28. pour proceder a des mises a jour de produits est rigoureusement interdite LA DOCUMENTATION EST FOURNIE EN L ETAT ET TOUTES AUTRES CONDITIONS DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES DANS LA MESURE AUTORISEE PARLA LOI APPLICABLE Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE A L APTITUDE A UNE UTILISATION PARTICULIERE OU A L ABSENCE DE CONTREFACON s x DI cle cA TA Adobe PostScript Contents Preface xi Introduction to the Sun Netra CP3060 Blade Server 1 1 1 1 1 2 1 3 1 4 1 5 1 6 Overview of the Sun Netra CP3060 BladeServer 1 1 Features of the Sun Netra CP3060 Blade Server 1 2 Sun Netra CP3060 Blade Server System Configurations 1 7 1 3 1 AMC Modules 1 8 13 2 Rear Transition Module 1 8 Hot Swap Support 1 11 System Requirements 1 11 15 1 Hardware Requirements 1 12 1 5 2 Software Requirements 1 12 Technical Support and Warranty 1 13 1 6 1 Blade Server Part Number Serial Number and Revision Number Identification 1 13 Hardware Installation 2 1 2 1 2 2 2 3 Equipment and Operator Safety 2 1 Materials and Tools Required 2 2 Preparing for the Installation 2 3 iv 2 3 1 Checking Power Thermal Environmental and Space Requirements 2 3 2 3 2 Determining Local Network IP Addresses and Host Names 2 4 2 3 3 Installation Procedure Summary 2 4 24 Configuring the Blade Server Hardware 2 5 2 4 1 Installation of DDR 2 DIMM Memory Modules 2 5
29. provides parity protection over the multiplexed address data bus and another provides parity over the control signals Address and control parity errors cause a fatal reset while data parity errors are treated on a per instance basis depending on the producer and consumer of the data I O Subsystem The I O subsystem implements the JBus and PCI E fabrics including the JBus to PCI E bridge one PCI E switch one AMC slot two Intel PCI E dual gigabit Ethernet chips a PC E UliM1575 Southbridge chip and miscellaneous logic Chapter 5 Hardware and Functional Descriptions 5 9 5 1 4 1 5 10 JBus to PCT E Bridge The JBus to PCI E bridge is a low cost high performance JBus to dual PCI E host bridge similar to the PC based Northbridge On the host bus side the JBus to PCI E bridge supports a coherent split transaction 128 bit JBus interface On the I O side the JBus to PCT E bridge supports two 8x lane industry standard PCT E interfaces In addition the JBus to PCI E bridge supports the following interfaces functionality to facilitate a richer system feature set JBus Interface The JBus unit detects which transactions present on JBus are targeted for the JBus to PCI E bridge accepts and queues those transactions and coordinates with the appropriate destination unit within the JBus to PCI E bridge to which the address and data are sent When the UltraSPARC T1 processor is the target the converse is true in this case the JBus uni
30. shipped with the memory module Caution Do not remove the DDR 2 DIMM from its antistatic container until you are ready to install it on the card Handle the module only by its edges Do not touch module components or metal parts Always wear a grounded antistatic wrist strap when handling modules 1 Locate the DDR 2 DIMM connectors on the Sun Netra CP3060 blade server Select the connectors where you will install the memory module see FIGURE 2 1 If you need to replace an existing memory module with a new module see Section 2 4 1 2 Removing a DDR 2 DIMM Memory Module on page 2 8 for instructions on removing the DDR 2 DIMM module 2 Remove the DDR 2 DIMM from its protective packaging holding the module only by the edges 3 Insert the bottom edge of the DDR 2 DIMM into the bottom of the slot s hinge style connector see FIGURE 2 2 The socket and module are both keyed which means the module can be installed one way only With even pressure push simultaneously on both upper corners of the DDR 2 DIMM until its bottom edge the edge with the gold fingers is firmly seated in the connector Chapter 2 Hardware Installation 2 7 FIGURE 2 2 Installing a DDR 2 DIMM Memory Module el e N Caution Do not rock the DDR 2 DIMM into place Ensure that all contacts engage at the same time You will feel or hear a click when the DDR 2 DIMM properly seats in the connector 4 Press the top edge of the DDR 2 DIMM
31. support and options m Elimination of PCI connectivity between the blade servers in the system and reallocation of connectivity to serial interconnects eliminating single points of failure m Mandatory use of Intelligent Platform Management Interface IPMI management interfaces m Flexible user I O m Power and thermal management guidelines enforced by the management infrastructure m Separation of control and data traffic by supporting the Base PICMG 3 0 and Extended PICMG 3 1 interfaces The ATCA standard consists of the PICMG 3 0 PICMG 3 1 PICMG 3 2 and PICMG 3 3 specifications The Sun Netra CP3060 blade server complies with m PICMG 3 0 the base specification that defines the mechanical power distribution system management data transport and regulatory guidelines m PICMG 3 1 which builds upon the PICMG 3 0 base specification and the IEEE 802 3 2003 standard 1 2 1 2 Features of the Sun Netra CP3060 Blade Server The Sun Netra CP3060 blade server provides two 1000BASE T Ethernet interfaces for the Base interface a requirement of PICMG 3 0 and two serializer deserializer SERDES gigabit Ethernet interfaces PICMG 3 1 for the Extended interface The Base interface is used as the control interface and the Extended interface can be used for data traffic Both Base and Extended interfaces are configured as Dual Star topologies Sun Netra CP3060 blade server features include m Single wide ATCA module Compli
32. 0 Zone 3 Connectors A 12 x Netra CP3060 Board User s Guide April 2009 Preface The Sun Netra CP3060 Blade Server User s Guide provides information about features installation configuration functional hardware components and physical properties of this blade server The Sun Netra CP3060 Blade Server User s Guide is written for system integration engineers field applications and service engineers and others involved in the integration of these blade servers into systems How This Document Is Organized Chapter 1 provides an overview of the Sun Netra CP3060 blade server Chapter 2 provides instructions on hardware installation Chapter 3 provides instructions on the software configuration Chapter 4 provides information about the Sun Netra CP3060 firmware Chapter 5 provides hardware and functional descriptions of the Sun Netra CP3060 blade server Appendix A provides information about the physical characteristics of the Sun Netra CP3060 blade server Appendix B describes the Sun specific OEM defined Intelligent Platform Management Interface IPMI commands xi Using UNIX Commands This document might not contain information on basic UNIX commands and procedures such as shutting down the system booting the system and configuring devices Refer to the following for this information m Software documentation that you received with your system m Solaris Operating System documentation which is at httpe d
33. 1V 3 39V Temperature Monitor The ADM1026 also monitors the CPU die or junction temperatures of the CPU on the Sun Netra CP3060 blade server The ADM1206 and H8 firmware reports a minor major or critical alarm when the temperature of either CPU goes beyond the 5 24 Netra CP3060 Board User s Guide April 2009 thresholds listed in TABLE 5 5 Also if the temperature of either CPU goes beyond the Emergency H8 Shutdown value listed in TABLE 5 5 the H8 initiates a blade server shutdown independent of the shelf manager TABLE 5 5 CPU Temperature Alarms Minor Alarm Major Alarm Critical Alarm Emergency H8 Sensor UNC UC UNR Shutdown CPU Temp 1 gt 80 C 290 C 102 C gt 105 C CPU Temp 2 gt 80 C gt 90 C gt 102 C gt 105 C Blade Server gt 60 C gt 70 C gt 88 C gt 90 C Temp The H8 temperature alarm equivalents are Minor Alarm Upper Non Critical UNC Major Alarm Upper Critical UC Critical Alarm Upper Non Recoverable UNR Emergency H8 Shutdown Emergency H8 Shutdown EMR Caution These voltage and temperature thresholds should not be changed under normal operating conditions 2 2 Power on Sequence The following list describes the power on sequence 1 Sun Netra CP3060 blade server installed into a midplane slot in ATCA shelf Aux power applied to IPMC and the FPGA IPMC enables front end DC DC 48V 12V converter IPMC checks power ok from front end DC DC
34. 2 Features of the Sun Netra CP3060 Blade Server on page 1 2 m Section 1 3 Sun Netra CP3060 Blade Server System Configurations on page 1 7 m Section 1 4 Hot Swap Support on page 1 11 m Section 1 5 System Requirements on page 1 11 m Section 1 6 Technical Support and Warranty on page 1 13 1 1 Overview of the Sun Netra CP3060 Blade Server The Sun Netra CP3060 blade server is a high performance single blade server computer based on one UltraSPARC T1 multicore processor and designed for high availability in a switched network computing environment This blade server is compliant with Advanced Telecom Computing Architecture ATCA specifications PICMG 3 0 and PICMG 3 1 and can achieve greater performance levels than compactPCI cPCI standards based products targeted for telco markets The PICMG PCI Industrial Computer Manufacturers Group standards committee has developed the new ATCA or PICMG 3 x standard to address the issues posed by previous standards based on cPCI and cPSB PICMG 2 x The PICMG 3 x specification brought the following changes to the existing PICMG 2 x family of products m Larger board space 8U high compared to 6U for cPCI which allows more features and processing power m On board power supplies deriving local power from redundant 48V power from the midplane rather than separate power supplies m 6 HP slot width allowing greater component height m Advanced Mezzanine card AMC
35. 780 pci 0 pci 1 pci 0 ide 1f 1 disk pci 780 pci 0 pci 1 pci 0 ide 1f cdrom pci 780 pci 0 pci 1 pci 0 ide 1f disk virtual devices 100 ipmi f virtual devices 100 flashupdate e virtual devices 100 led ed virtual devices 100 explorer c virtual devices 100 sunmc b virtual devices 100 sunvts a virtual devices 100 fma 9 virtual devices 100 echo 8 virtual devices 100 loop 6 virtual devices 100 loop 7 virtual devices 100 rtc 5 virtual devices 100 ncp 4 virtual devices 100 console 1 virtual devices 100 flashprom 0 virtual devices 100 nvram 3 openprom client services packages SUNW asr packages obp tftp packages dropins packages terminal emulator packages disk label packages deblocker packages SUNW builtin drivers 0 ok Checking Network Using watch net and watch net all Commands The watch net diagnostics test monitors Ethernet packets on the primary network interface The watch net all diagnostics test monitors Ethernet packets on the primary network interface and on any additional network interfaces connected to the system blade server Good packets received by the system are indicated by a period Errors such as the framing error and the cyclic redundancy check CRC error are indicated with an X and an associated error description Netra CP3060 Board User s Guide April 2009 4 3 4 To start the watch net diagnostic test type the watch net command at the ok prompt 0 ok watch net In
36. I DMA tests JBus to PCI E loop back test 7 PCIE Tests Verify PCI E Bus configuration Verify VID DIC registers for all onboard PCI device Verify link status of all onboard PCI E channel POST Diagnostic and Error Message Format POST diagnostic and error messages are displayed on a console The format of the these messages is the following Core ID Strand ID ERROR TEST fest name Core ID Strand ID H W under test description Core ID Strand ID Repair Instruction Core ID Strand ID MSG error message body Core ID Strand ID END ERROR The following is an example of a POST error message TABLE 4 1 3 2 gt ERROR TEST L2 Cache Functional 3 2 gt H W under test Core 12 Cache 3 2 gt Repair Instructions Replace items in order listed by H W under test above 3 2 gt MSG No way found to match tag address 00000000 00600000 state 3 3 2 gt END_ ERROR 4 3 OpenBoot Firmware The Solaris OS installed operates at different run levels For a full description of run levels refer to the Solaris system administration documentation Chapter 4 Firmware and Blade Server Management 4 3 4 3 1 4 4 Most of the time the OS operates at run level 2 or run level 3 which are multiuser states with access to full system and network resources Occasionally you might operate the system at run level 1 which is a single user administrative state However the lowest operational state is run level 0
37. If only nonfatal errors are detected by POST or OpenBoot firmware the system attempts to boot if auto boot is true and auto boot on error is true Nonfatal errors include the following 4 12 Netra CP3060 Board User s Guide April 2009 m SATA subsystem failure In this case a working alternate path to the boot disk is required m Ethernet interface failure m Serial interface failure m Memory failure Given a failed DIMM the firmware will unconfigure the entire logical bank associated with the failed module Another nonfailing logical bank must be present in the system for the system to attempt a degraded boot Note If POST or OpenBoot firmware detects a nonfatal error associated with the normal boot device the OpenBoot firmware automatically unconfigures the failed device and tries the next in line boot device as specified by the boot device configuration variable m Ifa fatal error is detected by POST or OpenBoot firmware the system does not boot regardless of the settings of auto boot or auto boot on error Fatal nonrecoverable errors include the following Any CPU failed m All logical memory banks failed a Flash RAM cyclical redundancy check CRC failure m Critical field replaceable unit FRU PROM configuration data failure m Critical application specific integrated circuit ASIC failure 4 5 Automatic System Recovery Automatic system recovery ASR consists of self test features and an autoco
38. MCO IO32 AMCO IO31 6 AMCO IO30 AMCO IO29 AMCO IO28 AMCO IO27 AMCO IO26 AMCO IO25 GND 7 n c n c n c n c n c n c 8 n c n c n c n c n c n c 9 n c n c n c n c n c n c 10 n c n c n c n c n c n c TABLE A 9 gives the Zone 3 J10301 connector pin assignments TABLEA 9 Zone 3 10301 Connector Pin Assignments Row A B BG Cc D DG E F FG 1 n c n c n c n c n c 12V 2 n c n c 5V n c n c 5V 12V 12V 5V 3 RIM SERI_ RTM_SERI_ 3 3V RTM_SERI_ RTM_SERI_ 3 3V RTM_SER1_ RTM_SER1_ 3 3V CTS DTR DCD DSR RXD TXD 4 RTM SERI n c 3 3V n c n c 3 3V n c n c 3 3V RTS 5 n c n c 5V n c n c 5V n c n c 5V 6 RTM_SATA1_ RTM_SATA1_ GND RTM_ETH_ RTM_ETH_ GND RTM SATA1 RTM SATA1 GND RXP RXN LED 1 LED 2 TX N TX P 7 RIM PRSNT 3V STBY GND RTM DC RTM DC GND 3V STBY 3V STBY GND N SDA SCL 8 RIM SATA2 RTM_SATA2_ GND RTM_ETH_ RIM ETH_ GND RTM SATA2_ RTM_SATA2_ GND RXP RXN LED3 LED4 TX N TX P 9 RIM ETH RIM ETH_ GND 2 5V 2 5V GND RIM ETH_ RTM_ETH_ GND TRD_3P TRD 3N TRD 2P TRD 2N 10 RTM ETH RIM ETH_ GND 2 5V 2 5V GND RIM ETH_ RTM_ETH_ GND TRD_1P TRD 1N TRD 0P TRD ON Appendix A Physical Characteristics A 13 A 4 7 TOD Clock Battery Holder The TOD clock battery holder is located under the AMC enclosure see FIGURE A 1 The TOD battery must be type CR 1225 with a minimum of 4ma abnormal charging current rating for example a Panasonic CR 1225 Dispose of batteries properly in accordance with manufacturer s instructions and local regulations Caution
39. OM contents 4 1 R rear transition module features 1 8 removing DDR 2 DIMM module 2 8 requirements hardware 1 12 safety 2 1 software 1 6 1 12 reset manual system 4 4 scenarios 4 9 RJ 45 Ethernet A 4 RTM rear transition module 1 8 S serial number locating 1 13 port A 4 show devs command OpenBoot 4 6 shutdown Solaris command 4 4 software hot swap information 4 15 PICL 4 17 requirements 1 6 Solaris commands init 4 4 shutdown 4 4 Sun OEM IPMI commands B 1 Get RTM Status B 3 Get Version B 1 Sun Services 1 13 SunVTS software 3 7 support contacting 1 13 system reset scenarios 4 9 T TOD clock battery 5 28 A 14 battery holder A 14 tools needed 2 2 Ww warranty 1 13 watch net 4 8 Index 3 Index 4 Netra CP3060 Board User s Guide April 2009
40. RE 2 2 FIGURE 2 3 FIGURE 2 4 FIGURE 2 5 FIGURE 2 6 FIGURE 2 7 FIGURE 2 8 FIGURE 2 9 FIGURE 2 10 FIGURE 4 1 FIGURE 5 1 FIGURE 5 2 FIGURE 5 3 Figures Sun Netra CP3060 Blade Server Front Panel 1 4 Sun Netra CP3060 Blade Server Top View 1 4 Sun Netra CP3060 Blade Server in an ATCA Shelf Enclosure 1 8 Sun Netra CP3060 Rear Transition Module 1 9 Relationship of the Sun Netra CP3060 Blade Server Midplane and RTM 1 10 Sun Netra CP3060 Blade Server Barcode Labeling 1 14 DDR 2 DIMM Memory Locations 2 6 Installing a DDR 2 DIMM Memory Module 2 8 Removing a DDR 2 DIMM Memory Module 2 9 Compact Flash Card Connector 2 10 Installing the Sun Netra CP3060 Rear Transition Module 2 12 RTM Injector Ejector Latch and Locking Screw 2 14 Installing Node Blade Server Into Chassis Slot 2 16 Sun Netra CP3060 Blade Server Latches and Locking Screws 2 17 Removing AMC Slot Filler Panel 2 19 Inserting AMC Module Into AMC Connector 2 20 Hot Swap Latch and Hot Swap LED 4 16 Block Diagram 5 3 UltraSPARC T1 Multicore Processor Block Diagram 5 5 DDR Memory Diagram 5 8 FIGURE 5 4 FIGURE 5 5 FIGURE 5 6 FIGURE A 1 FIGURE A 2 FIGURE A 3 FIGURE A 4 FIGURE A 5 FIGURE A 6 AMC Power and Management Infrastructure 5 14 Clock Synchronization 5 16 I C Block Diagram 5 22 Sun Netra CP3060 Blade Server Layout A 2 Ethernet RJ 45 Connector A 4 Front Panel Serial Port Diagram A 5 Power Distribution Connector Zone 1 P10 A 8 Zone 2 Connectors A 1
41. Risk of explosion if battery is replaced by an incorrect type To install the battery slide the battery into the holder with the side labeled facing up A 14 Netra CP3060 Board User s Guide April 2009 APPENDIX B Sun OEM IPMI Commands This appendix contains the following sections m Section B 1 Get Version Command on page B 2 m Section B 2 Get RTM Status Command on page B 3 The commands described in this appendix are specific to Sun Netra CP3060 blade server designed by Sun Microsystems The Internet Assigned Numbers Authority IANA number assigned to Sun Microsystems is 42 Refer to http www lana org assignments enterprise numbers for more information about IANA number assignments The netfunction NetFn used for these commands is 0x2E which is the OEM netfunction as defined in the IPMI specification For this netfunction the first three data bytes in the request packet must be this IANA number and the first three bytes in the response packet following the completion code are the IANA number For Sun ATCA node blade servers these three bytes are 00 00 2A The Sun OEM IPMI commands are listed in TABLE B 1 and described in the following sections TABLE B 1 Sun OEM IPMI Commands Command Opcode Syntax Get Version 0x80 GET VERSION Get RTM status 0x88 GET RTM STATUS B 1 B 1 Get Version Command Get Version returns the IPM controller IPMC firmware version and Standby CPLD version Byte
42. SS 4v SUN microsystems Sun Netra CP3060 Blade Server User s Guide Sun Microsystems Inc www sun com Part No 819 4967 11 April 2009 Revision A Submit comments about this document at http www sun com hwdocs feedback Copyright 2009 Sun Microsystems Inc 4150 Network Circle Santa Clara California 95054 U S A All rights reserved This distribution may include materials developed by third parties Parts of the o may be derived from Berkeley BSD systems licensed from the University of California UNIX is a registered trademark in the U S and in other countries exclusively licensed through X Open Company Ltd Sun Sun Microsystems the Sun logo Netra Sun Ray the Netra logo and the Solaris logo are trademarks or registered trademarks of Sun Microsystems Inc or its subsidiaries in the U S and other countries All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc in the U S and other countries Products bearing SPARC trademarks are based upon architecture developed by Sun Microsystems Inc Use of any spare or replacement CPUs is limited to repair or one for one D procement of CPUs in products exported in compliance with U S export laws Use of CPUs as product upgrades unless authorized by the U S Government is strictly prohibited DOCUMENTATION IS PROVIDED AS IS AND ALL EXPRESS OR IMPLIED CONDITIONS REPRESENTATIONS AND WARRANTIES INCLUDING ANY
43. UNW asr 5a0000006b ufs file system ufs file system 5a00000072 chosen chosen 5200000079 openprom openprom 5a00000086 client services client services 5a00000090 options options 5a00000096 aliases aliases 5a000000be virtual memory virtual memory 5a000000 7 iscsi iscsi 5a000004e1 For more information on the PICL framework refer to the picld 1M man page 4 9 4 20 Mandatory etc system File Entry A mandatory entry must be listed in the etc system file to ensure the optimal functionality of the server The following entry must be in the etc system file set pcie pcie aer ce mask 0x1 Check that the entry is present before deploying the server To Check and Create the Mandatory etc system File Entry 1 Log in as superuser 2 Check the etc system file to see if the mandatory line is present TABLE 4 10 more etc system ident system 1 18 05 06 27 SMI SVRA 1 5 SYSTEM SPECIFICATION FILE set pcie pcie aer ce mask 0x1 Netra CP3060 Board User s Guide April 2009 3 If the entry is not there add it Use an editor to edit the etc system file and add the entry 4 Reboot the server Chapter 4 Firmware and Blade Server Management 4 21 4 22 Netra CP3060 Board User s Guide April 2009 cHAPTER B Hardware and Functional Descriptions This chapter contains the following sections m Section 5 1 Hardware Architecture on pag
44. X7 P 19 Gnd 152 Gnd 61 Gnd 110 Gnd 18 12V 153 RTM TX7 N 60 PCIE DO N 111 RIM TX0 N 17 Gnd 154 RTM TX7 P 59 PCIE TX2 P 112 RTM TXO P 16 Gnd 155 Gnd 58 Gnd 113 Gnd 15 n c 156 RTM RX7 N 57 12V 114 RIM RXON 14 n c 157 RTM RX7 P 56 IC AMCCONN SCL 115 RTM RXO0P 13 Gnd 158 Gnd A 6 Netra CP3060 Board User s Guide April 2009 TABLE A 3 AMC Connector Pin Assignments Continued Basic Side Extended Side Basic Side Extended Side Pin Signal Pin Signal Pin Signal Pin Signal 55 Gnd 116 Gnd 12 n c 159 RIM TX8 N 54 PCIE RX1 N 17 RIMTXIN 1 n c 160 RIM TX8 P 53 PCIE RX1 P 118 RIM TX1 P 10 Gnd 161 Gnd 52 Gnd 19 Gnd 9 12V 162 RTM RX8 N 51 PCIE TX1 N 120 RIM RXLN 8 nGnd 163 RTM RX8 P 50 PCIE TX1 N 121 RTM RXIP 7 Gnd 164 Gnd 49 Gnd 122 Gnd 6 Gnd 165 TCLK 48 PCIE RX0 N 122 RIM TX2N 5 n c 166 TMS 47 PCIE RX0 P 124 RIM DO P i 3 3v AMC 167 TRST 46 Gnd 125 Gnd 3 AMC_PS1_L 168 TDO 45 PCIE TX0 N 126 RIM RX2N 2 12V 169 TDI 44 PCIE TX0 P 127 RTM RX2 P 1 Gnd 170 Gnd 43 Gnd 128 Gnd A 4 3 A 4 4 Compact Flash Connector The Compact Flash connector is a type I II connector Midplane Power Connector Zone 1 The Sun Netra CP3060 blade server uses a Zone 1 power connector It provides support for the following signals m Two 48 volt DC power feeds four signals each eight signals total m Two IPMB ports two signals each four signals total m Geographic address eight signals The analog test and ring voltage pins are left unconnected
45. aSPARC T1 processor 4 6 or 8 core CPU core speed 1 0 GHz Eight DDR 2 240 pin Very Low Profile VLP DIMMs buffered registered 1 Gbyte or 2 Gbyte DIMMs for a total memory of 16 Gbytes ECC supported ATCA 3 0 standard of 200 watts maximum PICMG 3 0 R1 0 PICMG 3 1 R1 0 Functions as a CPU node board with the Solaris software package Solaris 10 6 06 OS and subsequent compatible versions with supported Netra patches Dual gigabit Ethernet for Base interface Dual SERDES interface as Extended interface Dual IPMI channel connects to the midplane for communicating with the Shelf Management card Two 10 100 1000 BASE T Ethernet ports on front panel One asynchronous serial port for maintenance on front panel One AMC slot with front panel access for AMC I O Rear access support using Sun Netra CP3060 RTM One 10 100 Mbps Ethernet port One asynchronous serial port Access is available in front and rear when RTM is present Note Either serial port can be used but only one of the serial ports can be used at a time Provision for adding an independent hardware vendor IHV supplied AMC disk or PCI E card on front panel Uses IPMI communications with baseboard management controller BMC performs advanced system monitoring ASM on local board interface for example temperature sense FRU ID and control Basic full and high availability HA hot swap support for node board AMC disk card hot swap su
46. and some on board I O devices The software then generates messages that can be useful in determining the nature of a hardware failure POST can run even if the system is unable to boot If POST detects a faulty component it is disabled automatically preventing faulty hardware from potentially harming any software If the system is capable of running without the disabled component the system boots when POST is complete For example if one of the processor cores is deemed faulty by POST the core is disabled and the system boots and runs using the remaining cores POST diagnostic and error message reports are displayed on a console 4 2 1 POST Test Coverage The POST diagnostics include the following tests 1 UltraSPARC T1 Processor Tests MMU Memory Management Unit all cores DMMU TLBs tags data RAM tests IMMU TLBs tags data RAM tests Caches all cores 2 Cache L1 Icache L1 Dcache FPU Floating Point Unit Functional Register Interrupts 2 Memory Tests up to 2 Gbyte DIMM SDRAM data line test SDRAM address line test SDARM cell integrity Test MOVing inversions memory test 3 POST Image Tests e POST PROM checksum test POST memory checksum test 4 ECC Error Test 4 2 Netra CP3060 Board User s Guide April 2009 4 2 2 5 XBUS SRAM Test 6 JBus to PCIE Bridge Tests nternal registers test JBus interrupts JBus interrupts PCI E MSI Interrupts test PLX Interconnect test PC
47. ant with PCIMG3 0 3 1 specification m UltraSPARC T1 4 core 6 core or 8 core 1 0 GHz processor m Service processor MPC885 for CPU reset boot partition and fault management architecture FMA m Memory subsystem m Fight standard DDR 2 Very Low Profile VLP DIMM sockets Netra CP3060 Board User s Guide April 2009 m 200 MHz DIMMs buffered and registered with error correcting code ECC m Up to 16 Gbytes of DIMM memory with 2 Gbyte DIMMs m JBus to PCI E application specific integrated circuits ASICs to bridge processors and the PCI E I O subsystem m Two on board 1000BASE T Ethernet interfaces used as ATCA Base Interface m Two on board 1000 SERDES Ethernet interfaces used as ATCA Extended Interface m One AMC slot for x8 PCI E I O expansion or compatible disk drive Compliant with PCIMG AMC 0 1 3 specifications m Two 3 Gbps Serial Advanced Technology Attachment SATA ports routed to AMC connector m Two 10 100 1000 BASE T Ethernet RJ 45 ports on front panel m One asynchronous serial port on front panel m Compact Flash socket to support an 8 Gbyte user flash type I II memory card m System management support using Pigeon Point Systems Intelligent Platform Management IPM controller providing a redundant IPMI channel to communicate with the ATCA shelf manager m Rear I O access using a compatible Sun Netra CP3060 rear transition module RTM m Maximum power consumption 200 watts including Sun Netra CP3060 RTM FIGURE
48. ardware requirements Netra CP3060 Board User s Guide April 2009 2 3 2 94 Preparing for the Installation Read the following subsections before starting to install these blade servers In addition do the following 1 2 Become familiar with the contents of the referenced documentation Verify that all listed hardware and software is available see Section 1 5 System Requirements on page 1 11 Check power thermal environmental and space requirements see Section 2 3 1 Checking Power Thermal Environmental and Space Requirements on page 2 3 Verify that local area network LAN preparations are completed see Section 2 3 2 Determining Local Network IP Addresses and Host Names on page 2 4 Ensure that the host names and their network IP addresses are allocated and registered at the site Checking Power Thermal Environmental and Space Requirements Verify that you meet the following requirements Your enclosure specifications support the sum of the specified maximum blade server power loads See Section 5 3 Power on page 5 27 for blade server power specifications Facility power loading specifications can support the rack or enclosure requirements Your enclosure specifications support the cooling airflow requirements The Sun Netra CP3060 blade server fits a standard ATCA shelf or chassis If your installation requirements are different contact your field application engine
49. ate Array 5 18 5 1 5 3 XBus External devices 5 19 5 1 6 Intelligent Platform Management Controller 5 19 5 1 6 1 Intelligent Platform Management Bus 5 19 5 1 6 2 Interface to the MPC 5 19 vi Netra CP3060 Board User s Guide April 2009 5 1 6 3 IPMB L Interface 5 20 5 1 6 4 ATCA Hot Swap Latch 5 20 5 1 6 5 LEDs 5 20 5 1 6 6 Power Control 5 20 5 1 6 7 Extended Interface E keying Control 5 21 5 1 6 8 D2C Architecture 5 21 5 1 6 9 PC Device Address Map 5 22 5 1 6 10 System Monitor and Thresholds 5 24 5 2 Power on Sequence 5 26 53 Power 5 27 5 3 1 Power Input 5 27 5 3 2 Fuses 5 27 5 3 8 ATCA Power Module 48V to 12V 5 28 5 3 4 On Board DC DC Regulators 5 28 5 3 5 TOD Clock Battery 5 28 A Physical Characteristics A 1 A 1 Form Factor A 1 A 2 Layout A 1 A 3 Front Panel A 2 A 3 1 Visual Indicators A 3 A 3 2 Ports A 3 A 3 3 AMC Slot A 3 A 4 Connectors and Pinout A 3 A 4 1 Front Panel Connectors A 3 A411 Ethernet Ports A 4 A412 Serial Port A 4 A 4 2 AMC Connector A 5 A 4 3 Compact Flash Connector A 7 Contents vii viii A 4 4 A 4 5 A 4 6 A 4 7 Midplane Power Connector Zone 1 A 7 Data Transport Connector Zone 2 A 9 RTM Connector Zone 3 A 11 TOD Clock Battery Holder A 14 B Sun OEM IPMI Commands B 1 B 1 Get Version Command B 2 B 2 Get RTM Status Command B 3 Index Index 1 Netra CP3060 Board User s Guide April 2009 FIGURE 1 1 FIGURE 1 2 FIGURE 1 3 FIGURE 1 4 FIGURE 1 5 FIGURE 1 6 FIGURE 2 1 FIGU
50. ce path The Solaris OS provides some predefined device aliases for the network devices so that you do not need to type the full device path name TABLE 4 8 lists the network device aliases the default Solaris OS device names and associated ports for the Sun Netra CP3060 blade server The devalias command can be used to display the device aliases TABLE 4 8 Network Device Aliases Default Solaris 10 OS Device Alias Device Name net net0 e1000g0 net1 e1000g1 net2 e1000g2 net3 e1000g3 Port Base Interface Ethernet A Management Ethernet A Ethernet port A on front panel RTM Ethernet port Base Interface Ethernet B Management Ethernet A Ethernet port A on front panel Extended Interface Ethernet A PICMG 3 1 Extended Interface Ethernet B PICMG 3 1 4 8 Retrieving Device Information You use the Solaris platform information and control library PICL framework for obtaining the state and condition of the Sun Netra CP3060 blade server The PICL framework provides information about the system configuration that it maintains in the PICL tree Within this PICL tree is a subtree named frutree which represents the hierarchy of system field replaceable units FRUs with respect to a root node in the tree called chassis The frutree represents physical resources of the system The PICL tree is updated whenever a change occurs in a device s status Chapter 4 Firmware and Blade Server Management 4 17 4 18 TABLE 4 9 show
51. ctions built into the Southbridge are not used functional blocks of the device Chapter 5 Hardware and Functional Descriptions 5 15 5 16 are essential to the Sun Netra CP3060 blade server s system architecture Relative to the PCI E interconnect the Sun Netra CP3060 blade server uses a 4x link to access the M1575 The major functional blocks used are listed below and described in the paragraphs which follow m UltraDMA IDE controller The primary port is used to provide the connection to the Compact Flash and it supports speeds of 66 100 133 m SATA controller with four 3 Gbps SATA ports Configuring the internal devices of the Southbridge is very simple There is a central switching routing block inside the Southbridge chip that maps the PCI E incoming data to the intended downstream device Thus the Southbridge can be viewed as a collection of PCI devices within a single package The functional units within Southbridge which appear to firmware as unique PCI devices are listed in TABLE 5 2 The Southbridge s standard PCI device configuration is used to control the device behavior on the PCI bus and to allocate PCI Memory and I O space address ranges needed by the device TABLE5 2 Southbridge PCI Devices Bus Device Function IDSEL Function Name Vendor ID Device ID Bus 0 Device 25 Function0 NA PCI Bridge 10B9h 5249h PCI IDE Controller The IDE controller is resident inside the Southbridge This is a dual controller bu
52. d of time Prepare the blade server by opening the injector ejector latches FIGURE 2 6 Carefully align the edges of the blade server with the card guides in the appropriate slot FIGURE 2 7 It might be helpful to look into the enclosure to verify correct alignment of the rails in the guides Taking care to keep the blade server aligned in the guides slide the blade server in until the injector ejector latches engage the card cage Netra CP3060 Board User s Guide April 2009 FIGURE 2 7 Installing Node Blade Server Into Chassis Slot 6 Taking care to keep the blade server aligned in the guides slide the blade server in until the injector ejector latches engage the card cage 7 Push the blade server slightly into the midplane connectors and then close the latches to seat the blade server in the connectors FIGURE 2 8 When the lower latch is closed the blue Hot Swap LED blinks while the blade server is initializing The blue LED turns off and the green OK LED lights when the blade server is ready Chapter2 Hardware Installation 2 15 FIGURE 2 8 Sun Netra CP3060 Blade Server Latches and Locking Screws Top latch Locking screw Botton hot swap latch Locking screw 8 Tighten the locking screws to ensure the blade server is secured into the shelf see FIGURE 2 6 2 6 Connecting External I O Cables External I O cables are connected to the Sun Netra CP3060 blade server or to the Sun N
53. e cpu cpu cpu cpu 5a000000 c 5a00000168 5a00000188 5a000000e1 Netra CP3060 Board User s Guide April 2009 CODE EXAMPLE 4 3 obp prtpicl Command Output Continued cpu cpu 5a0000020c cpu cpu 5a0000021a cpu cpu 5a00000228 cpu cpu 5a00000236 cpu cpu 5a00000244 cpu cpu 5a00000252 cpu cpu 5a00000260 cpu cpu 5a0000026e cpu cpu 5a0000027c cpu cpu 5a0000028a cpu cpu 5a00000298 cpu cpu 5a000002a6 cpu cpu 5a000002b4 cpu cpu 5a000002c2 cpu cpu 5a000002d0 cpu cpu 5a000002de cpu cpu 5a000002ec cpu cpu 5a000002fa pci pciex 5a00000308 pci pciex 5a0000032a pci pciex 5a00000347 pci pciex 5a00000363 ide ide 5a00000384 ide ide 5a000003a8 dad block 5a000003d3 pci pciex 5a000003ea network network 5a00000407 network network 5a00000438 pci pciex 5a00000455 pci pciex 5a0000046f pci pciex 5a00000487 network network 5a000004a7 network network 5a000004c4 pseudo devctl 5a000004f6 picl 5a0000001e ib ib 5a00000032 packages packages 5a0000003e SUNW builtin drivers SUNW builtin drivers 5a00000044 deblocker deblocker 5a0000004a disk label disk label 5a00000051 terminal emulator terminal emulator 5a00000057 dropins dropins 5a0000005e obp tftp obp tftp 5a00000065 Chapter 4 Firmware and Blade Server Management 4 19 CODE EXAMPLE 4 3 prtpicl Command Output Continued SUNW asr S
54. e 5 1 m Section 52 Power on Sequence on page 5 26 m Section 53 Power on page 5 27 5 1 Hardware Architecture The Sun Netra CP3060 blade server is an ATCA node blade server based on the UltraSPARC T1 processor It is hot swappable to an ATCA midplane and supports two 1000BASE T Ethernet interfaces as Base interface and two 1000 SERDES Ethernet interfaces as Extended interface to support redundant Dual Star topology The Sun Netra CP3060 blade server supports eight standard DDR 2 Very low profile VLP DIMMs which can support the maximum memory of 16 Gbytes The I O subsystem is designed around the JBus to PCI E ASIC which is the bridge between the processor bus JBus and the PCI E links The JBus to PCI E bridge has one PCI E x4 link Leaf A and one PCI E x8 link Leaf B Leaf A is wired to a gigabit Ethernet controller which provides the SERDES Extended interface Leaf B is wired to the PCI E switch ASIC PLX8532 and provides the Base interface The PCI E switch ASIC on Leaf B connects to the Southbridge the AMC slot the Zone 3 RTM connector and a gigabit Ethernet controller for the Base interface and maintenance ports The Southbridge has a built in IDE controller that provides support for Compact Flash and a SATA controller with four 3 Gbps SATA ports of which two are connected to the AMC connector and the two are connected to the Zone 3 RTM connector 5 1 5 2 The BMR H8S IPM Controller IPMC monitors all the
55. e RTM in that particular slot Installing the Sun Netra CP3060 Blade Server With a Rear Transition Module A compatible RTM must be used with the Sun Netra CP3060 blade server for rear I O access The RTM enables access to the network to a boot device and to a console terminal You can use the Sun Netra CP3060 RTM or you might design your own rear transition module Chapter2 Hardware Installation 2 11 FIGURE 25 Installing the Sun Netra CP3060 Rear Transition Module Zone 3 connectors Netra CP3060 RTM Netra CP3060 blade server installed from rear installed from front w m jm Nad 9 9a N ATCA chassis midplane 2 5 1 1 Installing a Rear Transition Module 1 Verify that you have taken the necessary antistatic precautions 2 Go to the rear of the system and choose an appropriate slot for the RTM RTMs must be installed inline behind the accompanying node blade server For example if the accompanying node blade server is installed in slot 3 its RTM must be installed at the back of the system in slot 3 See FIGURE 2 5 2 12 Netra CP3060 Board User s Guide April 2009 3 Remove the slot filler panel from the selected node blade server slot if necessary 4 Get the RTM from the shipping kit 5 Perform any card specific hardware procedures if necessary Refer to the documentation that you received with the card for more information 6 Prepare the card by opening the injector ejector latche
56. e diskless client 9 0 Downloading and Installing SunVTS Software SunVTS software is a comprehensive suite that tests and validates the Sun Netra CP3060 blade server by verifying the configuration and function of most hardware controllers and devices on the blade server SunVTS software is used to validate a system during development production inspection troubleshooting periodic maintenance and system or subsystem stressing SunVTS software can be tailored to run on machines ranging from desktops to servers with modifiable test instances and processor affinity features You can perform high level system testing by using the appropriate version of SunVTS software For detailed information on SunVTS support and downloads refer to the following web site http www sun com oem products vts Ensure that the SunVTS software version is compatible with the Solaris OS version being used Information on the version of the SunVTS software installed can be found in the file opt SUNWvts bin version For the latest version of SunVTS document go to http www sun com documentation Chapter3 Software Installation 3 7 Note For security reasons only a superuser is permitted to run SunVTS software Installation and starting instructions are included with the software when it is downloaded 3 8 Netra CP3060 Board User s Guide April 2009 CHAPTER 4 Firmware and Blade Server Management This chapter contains the
57. e from its shipping kit and place it on an antistatic surface 5 Insert the AMC module through the cutout and into the AMC slot FIGURE 2 10 2 18 Netra CP3060 Board User s Guide April 2009 FIGURE 2 10 Inserting AMC Module Into AMC Connector AMC module AMC disk drive shown 6 Carefully plug the AMC module into the AMC connector FIGURE 2 10 Ensure that the AMC module is seated correctly in the connector Caution Do not use excessive force when installing the AMC module into the slot You might damage the AMC connector on the Sun Netra CP3060 blade server causing permanent damage to the AMC module or the blade server If the AMC module does not seat properly when you apply even pressure remove the AMC module and carefully reinstall it 7 Refer to the AMC module documentation for software and cabling installation instructions Chapter 2 Hardware Installation 2 19 2 20 Netra CP3060 Board User s Guide April 2009 Chapter 2 Hardware Installation 2 21 CHAPTER 3 Software Installation This chapter contains the following sections m Section 3 1 Operating Systems and Patches on page 3 1 m Section 3 2 Firmware Updates on page 3 2 m Section 3 3 Mandatory etc system File Entry on page 3 2 m Section 34 Installing Diskless Clients on page 3 3 m Section 3 5 Downloading and Installing SunVTS Software on page 3 7 od Operating Systems and Patches The Sun Netra CP3060 blade se
58. e ide command CODE EXAMPLE 4 1 probe ide Command Output 0 ok probe ide Device 0 Primary Master ATA Model FUJITSU MHV2040BH Device 1 Primary Slave ATA Model Device 2 Secondary Master Not Present Device 3 Secondary Slave Not Present show devs Command The show devs command lists the hardware device paths for each device in the firmware device tree CODE EXAMPLE 4 2 shows some sample output CODE EXAMPLE 4 2 show devs Command Output 0 ok show devs pci 7c0 pci 780 cpu 17 Netra CP3060 Board User s Guide April 2009 CODE EXAMPLE 4 2 show devs Command Output Continued cpu 1 cpu 1 cpu 1 cpu 1 cpu 1 cpu 1 cpu 1 cpu f cpuGe cpu d cpu c cpu b cpu a cpu 9 cpu 8 cpu 7 cpu 6 cpu 5 cpu 4 cpu 3 cpu 2 cpu 1 cpu 0 virtual devices 100 OD H N W bb UY o virtual memory memory m0 800000 aliases options openprom chosen packages pci 7c0 network 0 1 pci 7c0 network 0 pci 780 pci 0 pci 780 pci 0 pci 9 pci 780 pci 0 pci 8 pci 780 pci 0 pci 2 pci 780 pci 0 pci 1 pci 780 pci 0 pci 2 network 0 1 pci 780 pci 0 pci 2 network 0 pci 780 pci 0 pci 1 pci 0 pci 780 pci 0 pci 1 pci 0 ide 1f 1 pci 780 pci 0 pci 1 pci 0 ide 1f Chapter4 Firmware and Blade Server Management 4 7 4 3 3 3 4 8 CODE EXAMPLE 4 2 show devs Command Output Continued pci 780 pci 0 pci 1 pci 0 ide 1f 1 cdrom pci
59. e power from the equipment chassis when servicing a system or installing a hardware upgrade If the chassis cannot be placed upon a grounded antistatic mat connect a grounding strap between the facility electrical input ground usually connected to the equipment chassis and facility electrical service ground m Use an antistatic wrist strap when Removing a blade server from its antistatic bag Connecting or disconnecting blade servers or peripherals The other end of the strap lead should be connected to one of the following m A ground mat m Grounded chassis metalwork m A facility electrical service ground m Keep blade servers in the antistatic bags until they are needed m Place circuit blade servers that are out of their antistatic bags on an antistatic mat if one is available The mat must be grounded to a facility electrical service ground Do not place blade servers on top of an antistatic bag unless the outside of the bag also has antistatic protective properties m Remove a blade server from its antistatic bag only when wearing a properly connected ground strap 22 2 2 Materials and Tools Required This section provides information on the materials and tools required to perform installation The minimum tools required to perform installation are m Phillips screwdrivers No 1 No 2 optional m Antistatic wrist strap m Terminal console See Section 1 5 1 Hardware Requirements on page 1 12 for information on h
60. e zone locale is the client s locale name service is the client S nameservice name seroer is the nameserver s hostname Refer to the smdiskless 1M man page for more information and options For example TABLE 6 smdiskless add i 129 144 214 999 e 8 0 20 22 b3 aa n client host x os sparc sun4v Solaris 10 x root export root client host x swap export swap client host x swapsize 999 x tz US Pacific x locale en US x ns NIS x nameserver nameserver host Where ip address 129 144 214 999 ethernet address 8 0 20 22 b3 aa host name client host n 10 for Solaris 10 swap size 128 time zone US Pacific locale en_US name_service NIS name_server nameserver_host 3 6 Netra CP3060 Board User s Guide April 2009 You must type your superuser password again after typing this command The installation process should take roughly 5 minutes per client and about 15 30 minutes for the operating environment service to install however no progress is displayed on screen while the process is running Do not cancel or kill the process until the process has successfully completed You should see messages similar to the following after a few moments confirming that the command went through successfully the second time TABLE 7 Login to client host as user root was successful Download of com sun admin osservermgr cli OsServerMgrCli from client host was successful 6 Boot th
61. ee Section 4 5 1 1 To Enable Automatic System Recovery on page 4 14 Enabling and Disabling Automatic System Recovery The automatic system recovery ASR feature is not activated until you enable it Enabling ASR requires changing configuration variables in ALOM as well as OpenBoot To Enable Automatic System Recovery 1 At the ok prompt type ok setenv auto boot true Ok setenv auto boot on error true 2 To cause the parameter changes to take effect type TABLE 4 6 ok reset all The system permanently stores the parameter changes and boots automatically when the OpenBoot configuration variable auto boot is set to true its default value Netra CP3060 Board User s Guide April 2009 45 12 Note To store parameter changes you can also power cycle the system using the front panel Power button To Disable Automatic System Recovery 1 At the ok prompt type Ok setenv auto boot on error false 2 To cause the parameter changes to take effect type TABLE 4 7 ok reset all The system permanently stores the parameter change Note To store parameter changes you can also power cycle the system using the front panel Power button After you disable the automatic system recovery ASR feature it is not activated again until you re enable it 4 6 4 6 1 Hot Swap Information The Sun Netra CP3060 blade server supports hot swapping and incl
62. egistered m 1 Gbyte and 2 Gbyte DDR 2 modules supported m DIMMs are installed as shown in FIGURE 2 1 Note You cannot mix 1 Gbyte and 2 Gbyte DIMMs The Sun Netra CP3060 blade server supports DDR 2 DIMM memory modules that have the following characteristics m Each DIMM has a 72 bit wide data bus 64 8 ECC and up to 14 address bits m Maximum height of the DIMM module is 0 72 inches Very Low Profile VLP m Supports single bank or dual bank SDRAM DIMMs m Memory controller supports 128 bit data plus 9 bit error correcting code ECC m Maximum of 16 Gbytes Chapter 2 Hardware Installation 2 5 For additional information see Section 5 1 2 DDR 2 Memory Subsystem on page 5 7 FIGURE 2 1 shows the location of the DIMMs FIGURE 2 1 DDR 2 DIMM Memory Locations 0080008003000000320 090020000002dp 60000008000200580030 000000000000 ba saasoggs 9s86 9 b 960 DIMM1 DIMMO DIMM5 DIMM4 DIMM12 DIMM13 DIMM8 DIMM9 Channel 0 Channel 1 Channel 3 Channel 2 Note Channels 1 and 2 are used in four DIMM configurations Channels O and 3 are only used in eight DIMM configurations 2 6 Netra CP3060 Board User s Guide April 2009 2 4 1 1 Installing a DDR 2 DIMM Memory Module The following procedure provides a general guide for installing additional memory However for directions on the installation process of the memory DDR 2 DIMMs on the Sun Netra CP3060 blade server refer to the documentation that
63. er Chapter2 Hardware Installation 2 3 2 3 2 2 0 9 2 4 Determining Local Network IP Addresses and Host Names Collect the following information to connect hosts to the local area network LAN Ask your network administrator for help if necessary You can use TABLE 2 1 to record this information This information is not needed for a standalone installation TABLE2 1 Local Network Information Information Needed Your Information IP address and host name for each Sun Netra CP3060 client Domain name Type of name service and corresponding name server names and IP addresses for example DNS and NIS or NIS Subnet mask Gateway router IP address NES server names and IP addresses Web server URL Local IP addresses are not needed if they are assigned by a network DHCP server You might need the MAC Ethernet addresses of the local hosts to make nameserver database entries The MAC address can be seen in the console output while booting to the ok prompt It can also be derived from the host ID seen on the barcode label see Section 1 6 1 Blade Server Part Number Serial Number and Revision Number Identification on page 1 13 Installation Procedure Summary The steps in this section summarize the Sun Netra CP3060 blade server installation at a high level Be sure to read the details in Section 2 4 Configuring the Blade Server Hardware on page 2 5 before installing the blade server T
64. er serial number Blade Server part number CPU configuration Memory configuration TREE Illli EET EAE IIlllllll llllll Illllllll IIlllllllllllll UB I lllllll Illlll IllllllllI IIlllll llI SunSN 1005LCB 0628Wxsx SunSn 1005LCB 0626Wxxxx SunPN 501 7313 07 Rev 06 IIIIIlllllll ERU E ELA SunPN 501 7508 01 Rev 03 2900000 pococscoscooocos pasocaco 08200000320000030p0 po80080008008000800300080030p200000820025000 bo00280000002000060200000080p000000000000000 posoonco00CO00000000200000020p200002220020800 joeooscoc 000000000200080080p000000000000000 p00028000000200000020008002080000088000000000 ho8008000600800000020008002050000090000000000 haa2295000008000000250080030p000000020000000 posaaspoaopodoooot dtoge a a PE III IIIIIIiILi 1IIII ANA ARA TABA SunSN 1005LCB 0626Wxxx TELE AWA VUELTA SunPN 501 7654 01 Rev 03 Note You might find the labels shown in FIGURE 1 6 on other locations on your blade server Your particular blade server configuration might also appear different from the illustration 1 14 Netra CP3060 Board User s Guide April 2009 CHAPTER 2 Hardware Installation This chapter describes the hardware installation procedures for the Sun Netra CP3060 blade server and contains the following sections m Section 2 1 Equipment and Operator Safety on page 2 1 m Section 2 2 Materials and Tools Required on page 2 2 m Section 2 3 Preparing for the I
65. etra CP3060 RTM when a rear transition module is used Information on connecting each of these cables follows m For Ethernet connections category 5e or better network cable is required One end of the Ethernet cable is connected to a suitable 10 100 1000BASE T switch and the other end to one of the Ethernet ports on the Sun Netra CP3060 blade server Both Ethernet ports A and B are available on the Sun Netra CP3060 blade server front panel 2 16 Netra CP3060 Board User s Guide April 2009 Note A single 10 100BASE T Ethernet port is also available on the Sun Netra CP3060 RTM when installed Refer to the Sun Netra CP3060 Rear Transition Module User s Guide 819 6689 for more information Use the bge device names shown in TABLE 2 2 when configuring the Ethernet ports TABLE2 2 bge Device Names Ethernet Ports Solaris 10 OS Ethernet Port A on front panel and e1000g0 Ethernet port on the RTM Ethernet Port B on front panel e1000g1 m An asynchronous serial I O cable can be attached from serial communication devices to the RJ 45 serial port on the Sun Netra CP3060 blade server s front panel Note A serial port is also provided on the Sun Netra CP3060 RTM Either the serial port on the Sun Netra CP3060 blade server or the serial port on Sun Netra CP3060 RTM can be used but only one of the ports should be used at one time Once a serial cable is connected use the tip utility on the host to establish a full duplex term
66. following sections m Section 4 1 System Firmware on page 4 1 m Section 4 2 Power On Self Test Diagnostics on page 4 2 m Section 4 3 OpenBoot Firmware on page 4 3 m Section 4 4 Error Handling Summary on page 4 12 m Section 4 5 Automatic System Recovery on page 4 13 m Section 4 6 Hot Swap Information on page 4 15 m Section 47 Network Device Aliases on page 4 17 m Section 4 8 Retrieving Device Information on page 4 17 4 1 System Firmware The Sun Netra CP3060 blade server contains a modular firmware architecture that gives you latitude in controlling boot initialization You can customize the initialization test the firmware and even enable the installation of a custom operating system This platform also employs the Intelligent Plattorm Management controller IPMC described in Section 5 1 6 Intelligent Platform Management Controller on page 5 19 which controls the system management hot swap control and some blade server hardware The IPMC configuration is controlled by separate firmware The Sun Netra CP3060 blade server boots from the 4 Mbyte system flash PROM device that includes the power on self test POST and OpenBoot firmware 4 1 4 2 Power On Self Test Diagnostics Power on self test POST is a firmware program that helps determine whether a portion of the system has failed POST verifies the core functionality of the system including the CPU modules motherboard memory
67. he procedure to set up and configure a Sun Netra CP3060 blade server in a system includes the following steps 1 Configure the blade server s physical hardware For example install memory or Compact Flash as necessary 2 Configure the rear transition module RTM as necessary Netra CP3060 Board User s Guide April 2009 3 Optional Physically install the Sun Netra CP3060 RTM into the chassis 4 Physically install the Sun Netra CP3060 blade server into the chassis 5 Connect the nodes to a local network Alternatively the blade server can be run as a standalone system without a network connection 6 Optional Install an AMC module on the Sun Netra CP3060 blade server 7 Install the operating system and patches as necessary See Section 3 1 Operating Systems and Patches on page 3 1 2 4 2 4 1 Configuring the Blade Server Hardware This section lists hardware installation and settings that might apply to your blade server configuration Read and perform the procedures as necessary before installing the Sun Netra CP3060 blade server into the chassis Installation of DDR 2 DIMM Memory Modules The Sun Netra CP3060 blade server supports a total of 8 DIMMs and a maximum memory capacity of 16 Gbytes using eight 2 Gbyte DIMMs FIGURE 2 1 shows the location of the DIMMs In addition to the on board memory the Sun Netra CP3060 blade server accommodates the following m Fight standard DDR 2 DIMMs buffered and r
68. hreaded pipe 7 Instruction i cache 4 threads SPARC SPARC SPARC SPARC SPARC SPARC SPARC SPARC COre core core core core core core core t t tt tt 3 nan Crossbar pipeline l Ps Data s cache Shared L2 cache bo lt maha ll LAM ll PRAM lij PEAN lij FIGURE 5 2 UltraSPARC T1 Multicore Processor Block Diagram Cores and Cache Each of the 8 SPARC cores has support for 4 threads for a total of 32 threads This support consists of a full register file per thread with most ASI ASR and privileged registers replicated per thread The 4 threads share the instruction cache data cache and TLBs Each TLB is 64 entry Each core then has 16 Kbytes of primary Instruction Cache I cache and 8 Kbytes of primary Data Cache D cache each of which is parity protected with redundant rows and columns for repair Lastly there is 3 Mbyte unified L2 cache which is 12 way associative ECC protected along with redundant rows and columns This cache is shared among the 8 internal cores and is connected through an internal crossbar The UltraSPARC T1 processor provides no external cache interface Chapter 5 Hardware and Functional Descriptions 5 5 5 1 1 3 5 1 1 4 5 1 1 5 5 1 1 6 5 1 1 7 5 6 Memory Controller The UltraSPARC T1 processor contains four i
69. ice is a PCI E based gigabit Ethernet controller with two fully independent Media Access Control MAC and physical layer PHY ports The PCI E interface is fully compliant to the PCI Express Specification Revision 1 0a The Sun Netra CP3060 blade server utilizes two controllers one for the Base Fabric and one for the Extended Fabric Both Ethernet interfaces are compliant to the IEEE 802 3 standard for Chapter 5 Hardware and Functional Descriptions 5 11 5 1 4 4 5 1 4 5 5 12 m 1000BASE T m 100BASE TX m 10BASE T m 1000BASE SX L SERDES optical fiber A serial SPI 16 KByte EEPROM is attached off each device to provide it with configuration data PHY configuration data MAC address etc 10 100 1000BASE T Ethernet Base Interface The Sun Netra CP3060 blade server provides two gigabit Ethernet 10 100 1000BASE T interfaces to meet the Base Interface requirements of ATCA The Ethernet controller is used to provide dual Base Interfaces and is an integrated MAC and GMII These controllers are connected to a x4 PCT E link The controllers are coupled to the ATCA backplane for TPE network connection Each interface supports 10BASE T 100BASE TX and 1000BASE T operation conforming to the IEEE802 3 specification reference 11 The controllers individually negotiate transfer speeds when their network links are established 1000 SERDES Ethernet for Extended Interface The Sun Netra CP3060 blade server provides two gigabit Ether
70. inal connection with the Sun Netra CP3060 blade server At the UNIX prompt in a command tool or shell tool type TABLE 2 3 tip 9600 dev ttya for serial port Zu Installation of Optional AMC Modules An Advanced Mezzanine card AMC is a card or module that provides additional functionality to the Sun Netra CP3060 blade server The blade server contains one AMC slot in which you can install an optional AMC device see FIGURE 2 1 for the location of these slots The AMC device can be installed and removed via a cutout in the front panel while the Sun Netra CP3060 blade server is installed in the chassis Chapter 2 Hardware Installation 2 17 2 7 0 1 Installing an Optional AMC Module Note The following procedure provides a general set of instructions for installing AMC modules on the Sun Netra CP3060 blade server Refer to the AMC module manufacturer s documentation for specific instructions on installing these devices 1 Retrieve the wrist strap from the adapter s shipping kit 2 Attach the adhesive copper strip of the antistatic wrist strap to the metal chassis Wrap the other end twice around your wrist with the adhesive side against your skin 3 Remove the AMC slot filler panel from the blade server s front panel FIGURE 2 9 FIGURE 2 9 Removing AMC Slot Filler Panel NA K IRE 8 EB S In 3 lt del ma PF ii aNg we B LI 7 fiu AMC slot filler panel NG 4 Retrieve the AMC modul
71. ion on the data The memory interface provides a standard Single bit correct Double bit Detect ECC protection across the 128 bits of data for a total memory width of 144 bits In addition the JBus interconnect is parity protected UltraSPARC T1 Processor Speed The UltraSPARC T1 processor supports a core frequency of 1 GHz The maximum supported JBus speed is 200 MHz and maximum supported DDR 2 speed is 200 MHz Netra CP3060 Board User s Guide April 2009 bali 5 1 2 1 The processor s core speed JBus and DDR 2 speeds in the Sun Netra CP3060 blade server depend upon total blade server power budget which shall not exceed 200W as per the PICMG 3 0 R1 0 specification DDR 2 Memory Subsystem The Sun Netra CP3060 blade server uses industry standard DDR 2 VLP 0 72 inch DIMMs Each of the four memory controller units MCUs inside the UltraSPARC T1 processor can control up to four DIMMs The Sun Netra CP3060 blade server only supports two DIMMs per MCU The Sun Netra CP3060 blade server supports up to 2 Gbyte VLP DIMMs for a total system memory capacity of 16 Gbytes Some of the key features of the memory subsystem are m Eight 240 pin JEDEC Standard DDR 2 Registered DIMM slots two DIMM slots per MCU m Minimum clock frequency of 200 MHz 400MT sec m Memory controller supports 144 bits 128 bits data 16 bits ECC m 16 Gbytes max 4 Gbytes 8 Gbytes 16 Gbytes m Peak memory bandwidth of 25 6 Gbytes sec 200MHz DIMMs
72. k On Chip 64 MB Disk on Chip DoC is used to store FMA logs and LDOM configuration Field Programmable Gate Array The SSI interface from the UltraSPARC T1 processor connects to a Field Programmable Gate Array FPGA that provides an internal 32 Kbyte SRAM access to external OpenBoot flash PROM through an XBus and access to the IPMC and the MPC The FPGA serves as a gateway between the UltraSPARC T1 and the MPC subsystem and provides support functionality for the IPMC It provides the following functionality m 32 Kbyte SRAM integrated used as Mailbox Data Channel and scratch pad for POST The SRAM is accessible both from the UltraSPARC T1 processor through the SSI interface and from H8 through the H8 bus m H8 interface providing IPMC support m MPC interface m SSI Interface UltraSPARC T1 m BUS Interface and Arbiter Round Robin Chapter 5 Hardware and Functional Descriptions 5 17 5 1 5 5 5 1 6 5 18 m Bus Clock Control m Power sequence control of DC DCs on board m Interrupts m Data Channel Fast Mailbox Control Initialization The FPGA configuration is performed after an FPGA reset when the configuration is downloaded from the PROM The PROM can be updated using a cable and the Xilinx programming header JTAG or from the MPC XBus External devices Flash PROM for OpenBoot The 4 Mbyte flash PROM is used for the OpenBoot and POST firmware Intelligent Platform Management Controller The Rene
73. napposcoacooscoacooscoadhoooco oo d n god posanecocecceccosttscoce Serial port connector Zone 2 connectors SA Huni Zone 1 connector A 3 Front Panel The single slot wide 8U front panel was designed to meet PICMG 3 0 R 1 0 and other specifications A 2 Netra CP3060 Board User s Guide April 2009 A 3 1 A 3 2 A 3 3 Visual Indicators The Sun Netra CP3060 blade server has the following indicators on the front panel m Green LED Blade Server s healthy status or user programmable ACTIVE m Amber LED Blade Server s fault condition FAULT m Blue LED Indicates safe removal hot swap activity The front panel s Ethernet ports do not have LED indicators Ports The Sun Netra CP3060 blade server has the following ports on the front panel m One serial port RJ 45 m Two 10 100 1000BASE T Ethernet ports RJ 45 AMC Slot The Netra CP3060 blade server has an AMC I O full height single width slot that is accessable through the AMC cutout on the front panel A 4 A 4 1 Connectors and Pinout FIGURE A 1 shows all the basic I O connectors to the front and the rear of the Sun Netra CP3060 blade server Front Panel Connectors The front panel has the following connectors m Two 10 100 1000BASE T Ethernet ports RJ 45 m One serial port RJ 45 m AMC connector via AMC slot Appendix A Physical Characteristics A 3 A 4 1 1 A 4 1 2 Ethernet Ports The Ethernet connectors
74. nce Bits 7 to1 0 Bits 0 RTM presence 0 RTM not detected 1 RTM detected Appendix B Sun OEM IPMI Commands B 3 Example Ierminal Mode B8 00 88 00 00 2A Request BC 00 88 00 00 00 2A 01 Response B 4 Netra CP3060 Board User s Guide April 2009 Index A AMC devices 1 8 installing 2 18 AMC I O 1 6 auto boot OpenBoot configuration variable 4 5 automatic system recovery ASR disabling 4 15 enabling 4 14 barcode labels locating 1 13 board layout A 1 C cabling 2 17 collecting network information 2 4 compliance NEBS 1 7 PICMG specifications 1 6 components 1 12 configurations I O configurations 1 11 D descriptions lt endrange gt 5 28 descriptions lt startrange gt 5 1 device information retrieving 4 17 device paths hardware 4 6 diagnostics POST 4 2 SunVTS software 3 7 DIMM memory module described 2 5 installing 2 7 removing 2 8 requirements 2 5 E electric shock caution 2 1 error handling summary 4 9 Ethernet RJ 45 A 4 F features 1 6 CPU 1 6 hot swap 1 6 memory 1 6 power requirements 1 6 Flash update 1 7 form factor A 1 front panel A 2 A 4 connections 1 7 frutree entries and properties 4 18 hierarchy 4 17 fuses 5 27 G graceful system halt 4 4 Index 1 H halt gracefully advantages of 4 4 hardware 5 1 5 28 hardware architecture 5 1 hardware device paths 4 6 hardware installation 2 1 to
75. ndependent DDR 2 memory controllers and data interfaces Unlike standard DDR 2 memory interfaces in the PC industry the UltraSPARC T1 processor uses a 144 bit datapath to main memory Thus the DIMMs in a UltraSPARC T1 processor memory subsystem are always accessed two at a time Because each DIMM provides 64 bits of data plus ECC the resulting data width is 128 bits plus ECC The clock speed of the memory subsystem is nominally 200 MHz which yields a data bit rate of 400 Mbps using the dual data rate signaling inherent to DDR 2 SDRAM technology Instruction Set The UltraSPARC T1 processor implements the standard SPARC V9 instruction set along with the standard UltraSPARC III extensions including the VIS instruction set Interval Arithmetic support and a special Prefetch Enhancement Interrupts The UItraSPARC T1 processor follows the interrupt dispatch mechanisms laid down in the SunV architecture In this model interrupts are pushed into the CPU as Mondo Vectors which in UltraSPARC T1 systems take the form of Interrupt packets sent over the JBus interconnect from the JBus to PCT E bridge into UltraSPARC T1 processor UItraSPARC T1 RAS Features For reliability availability and serviceability RAS the UltraSPARC T1 processor provides parity protection on its internal cache memories including tag parity and data parity on the D cache and I cache The internal 3 Mbyte L2 cache has parity protection on the tags and ECC protect
76. net allCommands 4 8 4 3 4 OpenBoot Configuration Variables 4 9 4 3 4 1 Viewing and Setting OpenBoot Configuration Variables 4 12 Error Handling Summary 4 12 Automatic System Recovery 4 13 4 5 1 Enabling and Disabling Automatic System Recovery 4 14 4 5 1 1 To Enable Automatic System Recovery 4 14 4 5 1 2 To Disable Automatic System Recovery 4 15 Hot Swap Information 4 15 4 61 Hot Swapping the Sun Netra CP3060 Blade Server 4 15 4 6 1 1 Hot Swap LED 4 16 Network Device Aliases 4 17 Retrieving Device Information 4 17 Mandatory etc systemFile Entry 4 20 v To Check and Create the Mandatory etc system File Entry 20 Hardware and Functional Descriptions 5 1 5 1 Hardware Architecture 5 1 Contents v 5 1 1 UltraSPARC T1 Processor 5 4 5 1 1 1 Overview 5 4 5 1 1 2 Cores and Cache 5 5 5 1 1 3 Memory Controller 5 6 5 1 1 4 Instruction Set 5 6 5 1 1 5 Interrupts 5 6 5 1 1 6 UltraSPARC T1 RAS Features 5 6 5 1 1 7 UltraSPARC T1 Processor Speed 5 6 5 1 2 DDR 2 Memory Subsystem 5 7 5 1 2 1 DIMMs 5 7 5 1 2 2 Memory Subsystem RAS Features 5 8 5 1 2 3 Serial Presence Detect 5 9 5 13 System JBus 5 9 5 1 3 1 JBus RAS Features 5 9 5 1 4 I O Subsystem 5 9 5 1 4 1 JBus to PCI E Bridge 5 10 5 1 4 2 PCI E Switch 5 11 5 1 4 3 PCI E to Dual GbE Controller 5 11 5 1 4 4 Ethernet Switch 5 12 5 1 4 5 AMC Slot 5 12 5 1 4 6 Southbridge 5 16 5 1 5 Service Processor MPC885 5 17 5 1 5 1 MPC Bus External Devices 5 18 5 1 5 2 Field Programmable G
77. net SERDES interfaces to meet the Extended Interface requirements of ATCA This Ethernet controller is used to provide dual Extended Interfaces and includes a MAC and GMII These controllers are connected to a x4 PCI E link Ethernet Switch The Ethernet switch provides Base Interface connectivity to the MPC885 AMC Slot The Sun Netra CP3060 blade server provides one AMC slot Extended Full Height Single Width Card used for PCI E x8 expansion In addition a dual SATA channel from the Southbridge are wired to the AMC slot The SATA channels are wired to port 2 and port 3 on the AMC connector common options region an AMC port is equal to a PCI E lane that is two differential pairs Netra CP3060 Board User s Guide April 2009 AMC Connectivity to RTM Port 12 20 8 differential signal pairs total of 32 signals on each AMC slot connector is wired to the Zone 3 RTM connector for AMC generated I O AMC Power and Management AMC require two power rails m 3 3V management power m 12V payload power maximum 21W Power management is handled by the ATCA ShMC An overview of the power and management signals is shown in FIGURE 5 4 FIGURE 5 4 AMC Power and Management Infrastructure 12V AMC AMC CP3060 AMC_3 3V_EN_L 3 3_AMC Module Management Power Control IPMB L Enable GA 2 0 IPMB L ENABLE PS1 Chapter 5 Hardware and Functional Descriptions 5 13 5 14 IPMB L Address See TABLE 5 1 for the IPMB L addresses
78. nfiguration capability to detect failed hardware components and unconfigure them By enabling ASR the server is able to resume operating after certain nonfatal hardware faults or failures have occurred If a component is monitored by ASR and the server is capable of operating without it the server automatically reboots if that component develops a fault or fails This capability prevents a faulty hardware component from stopping operation of the entire system or causing the system to fail repeatedly If a fault is detected during the power on sequence the faulty component is disabled If the system remains capable of functioning the boot sequence continues Chapter 4 Firmware and Blade Server Management 4 13 4 5 1 4 5 1 1 4 14 To support this degraded boot capability the OpenBoot firmware uses the 1275 client interface by means of the device tree to mark a device as either failed or disabled creating an appropriate status property in the device tree node The Solaris OS does not activate a driver for any subsystem marked in this way As long as a failed component is electrically dormant not causing random bus errors or signal noise for example the system reboots automatically and resumes operation while a service call is made Once a failed or disabled device is replaced with a new one the OpenBoot firmware automatically modifies the status of the device upon reboot Note ASR is not enabled until you activate it s
79. nnects up to downsteam devices via 1x 2x 4x or 8x links See PCI E Switch on page 11 for a better synopsis of the device The interface meets the PCI E 1 0a specification and is completely autonomous from the first PCI E interface on the JBus to PCI E bridge that is they are not electrically connected Interrupts The JBus to PCI E bridge employs a newly developed interrupt event queue that was necessitated by PCI Express being able to generate many more interrupts 2M than regular PCI which was limited to four The JBus to PCI E chip supports 1 K coalesced interrupts PCI E Switch The PCI E switch supports full non transparent crossbar bridging functionality that allows the system to isolate the I O memory domains by presenting the processor subsystem as a endpoint rather than another memory system The eight ports on the PCI E switch are highly configurable allowing for lane widths from x1 up to x16 The Sun Netra CP3060 blade server only uses lane widths of x4 and x8 which are hard coded using strapping resistors on the port_config pins The signal strengths of SERDES outputs are controlled by software to better optimize power and signal levels The four levels are off low typical and high The Sun Netra CP3060 blade server uses the typical setting The PCI E switch provides a PC Compliant Hot Plug Controller per port This is used at the port connected to the AMC modules PCI E to Dual GbE Controller This dev
80. nstallation on page 2 3 m Section 24 Configuring the Blade Server Hardware on page 2 5 m Section 2 5 Installing the Sun Netra CP3060 Blade Server on page 2 11 m Section 2 6 Connecting External I O Cables on page 2 17 2 1 Equipment and Operator Safety Refer to Important Safety Information for Sun Hardware Systems 816 7190 for general safety information Read the safety statements specific to the Sun Netra CP3060 blade server carefully before you install or remove any part of the system Caution Depending on the particular chassis design operations with open equipment enclosures can expose the installer to hazardous voltages with a consequent danger of electric shock Ensure that line power to the equipment is disconnected during operations that make high voltage conductors accessible The installer must be familiar with commonly accepted procedures for integrating electronic systems and with the general practice of Sun systems integration and administration Although parts of these systems are designed for hot swap 2 1 operation other components must not be subjected to such stresses Work with power connected to a chassis only when necessary and follow these installation procedures to avoid equipment damage This equipment is sensitive to damage from electrostatic discharge ESD from clothing and other materials Use the following antistatic measures during an installation m If possible disconnect lin
81. oes sun com xii Shell Prompts Shell Prompt C shell C shell superuser Bourne shell and Korn shell Bourne shell and Korn shell superuser machine name machine name Netra CP3060 Board User s Guide April 2009 Typographic Conventions Typeface Meaning Examples AaBbCc123 The names of commands files Edit your login file and directories on screen Use 1s a to list all files computer output You have mail AaBbCc123 What you type when contrasted 3 su with on screen computer output password AaBbCc123 Book titles new words or terms Read Chapter 6 in the User s Guide words to be emphasized These are called class options Replace command line variables You must be superuser to do this with real names or values To delete a file type rm filename The settings on your browser might differ from these settings Related Documentation For additional information about the Sun Netra CP3060 blade server and the Sun Netra CP3060 rear transition module RTM refer to the following documents Title Part Number Sun Netra CP3060 Blade Server Product Notes 819 4966 Sun Netra CP3060 Blade Server Getting Started Guide 819 4971 Sun Netra CP3060 Blade Server Programming Guide 819 4969 Sun Netra CP3060 Rear Transition Module Getting Started Guide 819 6692 Sun Netra CP3060 Rear Transition Module User s Guide 819 6689 Important Safety Information for Sun Hardware Systems p
82. ot request boot diag file from diag device If false After a boot request boot boot file from boot device Specifies recovery action after an error reset Default is boot none No recovery action boot System attempts to boot sync Firmware attempts to execute a Solaris sync callback routine Chapter 4 Firmware and Blade Server Management 4 11 4 3 4 1 Viewing and Setting OpenBoot Configuration Variables Halt the server to display the ok prompt m To display the current values of all OpenBoot configuration variables use the printenv command The following example shows a short excerpt of this command s output TABLE 4 4 ok printenv Variable Name Value Default Value local mac address true true fcode debug false false scsi initiator id 7 7 oem logo false false boot command boot boot boot file boot device disk net disk net use nvramrc false false nvramrc m Tosetor change the value of an OpenBoot configuration variable use the setenv command TABLE 4 5 Ok setenv error reset recovery none error reset recovery none m To set OpenBoot configuration variables that accept multiple keywords separate keywords with a space 4 4 Error Handling Summary Error handling during the power on sequence falls into one of the following three cases m If no errors are detected by POST or OpenBoot firmware the system attempts to boot if auto boot is true m
83. oval by opening the module handle which deactivates the hot swap switch When the hot swap switch on the module is deactivated the MMC in the module sends a module hot swap Module Handle Opened event to the IPMC If the ShMC permits blade server removal the IPMC asserts the HP_BUTTON input pin to the PCI E switch hot plug controller which Netra CP3060 Board User s Guide April 2009 will generate a hot plug event to the OS When the OS has quiesced the driver operating the AMC module the driver responds by writing to a CSR in the PCI E switch that causes de assertion of the HP PWR LED output pin This signal is monitored by IPMC When de asserted the IPMC informs the module to light the blue LED then turns off payload power and finally turns off management power Clock Synchronization The Sun Netra CP3060 blade server routes the clock synchronization signals from the midplane CLK1A 1B CLK2A 2B to the AMC slot connector CLK1 and CLK2 pins The control of which clock signal CLK1A or CLK1B and CLK2A or CLK2B is routed to the corresponding CLK1 and CLK2 pins is handled by IPMC FIGURE 5 5 Clock Synchronization CLK1A CLK1B Z2 CLK2A AMC connector Trans Mux MVLDS CLK2B CLK2B CLK2_SEL to IPMC 5 1 4 6 Southbridge The PCT E based Southbridge when used in a PC system implements virtually a complete desktop I O subsystem including Ethernet Although many of the Super I O fun
84. pport Netra CP3060 Board User s Guide April 2009 TABLE 1 1 Feature Summary for the Sun Netra CP3060 Blade Server Continued Feature Description Front panel One serial port RJ 45 access Two 10 100 1000BASE T Ethernet ports RJ 45 Cutout for installing AMC without removing blade server from system Building Network Equipment Building Systems NEBS Level 3 compliance Flash update Supported from downloaded file Note For EMI compliance of front access ports use shielded cables on all I O ports The shields for all shielded cables must be terminated on both ends 1 3 Sun Netra CP3060 Blade Server System Configurations Sun Netra CP3060 blade servers can be installed in an ATCA shelf or chassis shown in FIGURE 1 3 The blade servers can be deployed in various electrical configurations to suit each end user requirement For example the blade server can be configured to boot from a network as a diskless client with either a front panel or RTM network connection or from an optional Compact Flash card Alternatively industry standard Advanced Mezzanine Card AMC hardware from IHVs can be installed to provide local disk I O which can be used optionally as a boot path The Sun Netra CP3060 blade server has fixed on blade server memory and connectors for additional memory Chapter 1 Introduction to the Sun Netra CP3060 Blade Server 1 7 FIGURE 1 3 Sun Netra CP3060 Blade Server in an ATCA Shelf Enclosure
85. provides fuses on each of the 48V power feeds and on the return RTN connections The fuses on the return feeds are critical to prevent overcurrent situations when an O ring diode in the return path fails There are eight fuses on the blade server connected to 48V A 48V B RET A RET B EARLY A EARLY B ENABLE A and ENABLE B inputs The fuses are one time blow type fuse and need to be replaced when blown Netra CP3060 Board User s Guide April 2009 9 0 9 5 3 4 5 3 ATCA Power Module 48V to 12V The Sun Netra CP3060 blade server uses the ATCA power module solution The power module provides integrated ATCA power solution which supports PCIMG3 0 requirements including dual bus input DC isolation hold up hot plug and management power 3 3V standby It provides 12V intermediate bus as backend power Some of the salient features of the module are m 210W output power m Input to Output Isolation m 12V with current rating of 17 5A m 3 3V standby at 1 82A System Management Controller draws power from IPMI Power 3 3V STBY so that it can be functional even if back end logic is powered down m Isolated Remote ON OFF m Isolated A and B Bus detect signals m Operating Input Voltage 36V to 72V On Board DC DC Regulators The Sun Netra CP3060 blade server uses its own designed DC DC SIP modules for generating other onboard power All the voltages are generated from 12V input power TOD Clock Batte
86. ptional and must be ordered separately Acquire the following components if needed m Serial terminal or terminal emulation for console output m Cables for terminal and network connections m The Sun Netra CP3060 RTM and the AMC disks and PCI E cards are optional TABLE 1 3 lists ATCA and other minimum requirements met by the Sun Netra CP3060 blade server TABLE 1 3 ATCA and Other Minimum Requirements Met By the Sun Netra CP3060 Blade Server Sun Netra CP3060 Node Requirements Blade Server ATCA system enclosure for 8U boards includes chassis Yes midplane power supply Console output device or serial terminal Yes Boot device such as hard drive network or Compact Flash card Yes Peripheral device for network access Yes Intelligent Platform Management Controller IPMC Yes Software Requirements The Sun Netra CP3060 blade server supports the following versions of the Solaris OS m Solaris 10 OS and subsequent compatible versions Netra CP3060 Board User s Guide April 2009 Refer to the Sun Netra CP3060 Blade Server Product Notes 819 4966 for more Solaris OS information including a list of the required Netra software patches You can view and download the latest version of this manual at the following web site http www sun com documentation 1 6 1 6 1 Technical Support and Warranty Should you have any technical questions or support issues that are not addressed in the Sun Netra CP3060 blade
87. r 5 After the patches are installed follow the procedure in Section 3 4 2 Adding a Diskless Client on page 3 5 3 4 2 Adding a Diskless Client 1 Prepare a patched boot server for the diskless clients Follow the steps in Section 3 4 1 Creating a Boot Server for Diskless Clients on page 3 3 to create a boot server for the diskless clients 2 Log in to the patched boot server as superuser 3 Collect the following information for the diskless client you are adding m Client s IP address m Client s Ethernet address m Client s host name 4 Change directories to the usr sadm bin directory TABLE 4 cd usr sadm bin 5 Set up the diskless clients For each diskless client type the following command as superuser TABLE 5 smdiskless add i 1p address e ethernet address n host name x os sparc sun4v Solaris 5 x root export root host_name x swap export swap lost name x swapsize swap size x tz time zone x locale locale x ns name service x nameserver name_server Where m ip address is the client s IP address m ethernet address is the client s Ethernet address Chapter 3 Software Installation 3 5 host name is the client s host name n is the Solaris OS version you are using either 10 or 9 swap size is the size of the swap space that you will be using The default is 24 however your swap space should be the same amount as your memory time zone is the client s tim
88. re replacing the module you removed with a new DDR 2 DIMM install it as described in Section 2 4 1 1 Installing a DDR 2 DIMM Memory Module on page 2 7 Installation of Optional Compact Flash Card An optional IDE Compact Flash card can be installed on the Sun Netra CP3060 blade server The Compact Flash card is not hot swappable and there is no access to the card once the blade server is installed in an ATCA chassis Chapter 2 Hardware Installation 2 9 To install the Compact Flash card use the arrow on the card s label as a guide and insert the card into the Compact Flash connector see FIGURE 2 4 FIGURE 2 4 Compact Flash Card Connector Compact Flash connector 2 4 3 Configuring Rear Transition Module Hardware If you are using the Sun Netra CP3060 RTM refer to the Sun Netra CP3060 Rear Transition Module User s Guide 819 6689 You can also refer to the Sun Netra CP3060 RTM manual for detailed connector pin assignments 2 10 Netra CP3060 Board User s Guide April 2009 2 5 2 5 1 Installing the Sun Netra CP3060 Blade Server If you are installing the Sun Netra CP3060 blade server with the RTM first install the RTM at the rear of the server Then install the node card in the front of the server Even though you will be installing the RTM first look at the front of the server and locate the slot number where you will be installing the Sun Netra CP3060 blade server Then go to the back of the server and install th
89. rinted version only 816 7190 10 Except for Important Safety Information for Sun Hardware Systems all the documents listed are available online at http www sun com documentation Preface xiii Documentation Support and Training Sun Function URL Documentation http www sun com documentation Support http www sun com support Training http www sun com training Third Party Web Sites Sun is not responsible for the availability of third party web sites mentioned in this document Sun does not endorse and is not responsible or liable for any content advertising products or other materials that are available on or through such sites or resources Sun will not be responsible or liable for any actual or alleged damage or loss caused by or in connection with the use of or reliance on any such content goods or services that are available on or through such sites or resources xiv Sun Welcomes Your Comments Sun is interested in improving its documentation and welcomes your comments and suggestions You can submit your comments by going to http www sun com hwdocs feedback Please include the title and part number of your document with your feedback Sun Netra CP3060 Blade Server User s Guide part number 819 4967 11 Netra CP3060 Board User s Guide April 2009 CHAPTER 1 Introduction to the Sun Netra CP3060 Blade Server This chapter contains the following sections m Section 1
90. rver supports the Solaris 10 6 06 OS and subsequent compatible versions with supported Netra patches The Solaris OS software can be downloaded from the Sun download center at http www sun com download Note Refer to the Sun Netra CP3060 Blade Server Product Notes 819 4966 for information on the Netra patches at http www sun com documentation For information on these versions of the Solaris OS including installation see the appropriate Solaris Documentation Collection at the Sun Documentation web site at http www sun com documentation 3 1 3 2 Firmware Updates The Sun Netra CP3060 firmware updates can be downloaded from the Sun download center at http www sun com download Note For information on firmware updates refer to the Sun Netra CP3060 Blade Server Product Notes 819 4966 which are available online at http www sun com documentation 3 3 Mandatory etc system File Entry A mandatory entry must be listed in the etc system file to ensure the optimal functionality of the blade server The following entry must be in the etc system file set pcie pcie aer ce mask 0x1 Check that the entry is present before deploying the blade server V To Check and Create the Mandatory etc system File Entry 1 Log in as superuser 3 2 Netra CP3060 Board User s Guide April 2009 2 Check the etc system file to see if the mandatory line is present TABLE 3 1 more etc system
91. ry The TOD clock battery must be type CR 1225 with a minimum of 4ma abnormal charging current rating for example a Panasonic CR 1225 Caution Risk of explosion if battery is replaced by an incorrect type Dispose of batteries properly in accordance with manufacturer s instructions and local regulations Chapter 5 Hardware and Functional Descriptions 5 27 5 28 Netra CP3060 Board User s Guide April 2009 APPENDIX A Physical Characteristics Specifications for the Sun Netra CP3060 blade server are provided in the following sections m Section A 1 Form Factor on page A 1 m Section A 2 Layout on page A 1 m Section A 3 Front Panel on page A 2 m Section A4 Connectors and Pinout on page A 3 A 1 Form Factor The Sun Netra CP3060 blade server is a standard 8U form factor a single slot wide It complies with the board mechanical dimensions required by the PICMG 3 0 R1 0 Specification m 322 25 mm x 280 mm length x width m 1 2 inch wide front panel A 2 Layout The Sun Netra CP3060 blade server layout is shown in FIGURE A 1 FIGURE A 1 Sun Netra CP3060 Blade Server Layout AMC connector Compact Flash under enclosure connector AMC slot HDD installec Zone 3 connectors a ar Let eren F 1 j Ethernet port bosccecccsccssccsccecocesced 1 TOD clock EI poe 09000000000200090002qP q battery connectors P dpososs d o po00000020000000000200000056P d hidden aaa
92. s 8 9 and A are reserved for future use Command NetFn Opcode Reference Get Version Ox2E OEM 0x80 Data Bytes Type Byte Data Field Request data Bytel 00 Byte2 00 Byte3 2A Response data Bytel Completion code 00 OK C1 Command not supported CC Invalid data in request Refer to IPMI specification for more completion codes Byte2 00 Byte3 00 Byte4 2A Byte5 CPLD version Byte6 REV1 byte of IPMC firmware Byte7 REV2 byte of IPMC firmware Byte8 Reserved for future use ignore Byte9 Reserved for future use ignore ByteA Reserved for future use ignore Example Terminal Mode B8 00 80 00 00 2A BC 00 80 00 00 00 2A 02 02 00 00 00 00 Response m The IPMC version is read as B 2 Netra CP3060 Board User s Guide April 2009 lower nibble of REV1 high nibble of REV2 low nibble of REV2 In the preceding example the IPMC version is 2 0 0 m The CPLD version is read as lower nibble of CPLD version byte In the example the CPLD version is 2 B 2 Get RTM Status Command Get RTM Status can be used to detect the presence of a rear transition module RTM in the system Command NetFn Opcode Reference Get RTM Status Ox2E OEM 0x88 CPLD Specification Data Bytes Type Byte Data Field Request data Bytel 00 Byte2 00 Byte3 2A Response data Bytel Completion code 00 OK C1 Command not supported CC Invalid data in request Byte2 00 Byte3 00 Byte4 2A Byte5 RTM prese
93. s at the top and bottom of the card FIGURE 2 6 FIGURE 2 6 RTM Injector Ejector Latch and Locking Screw Locking screw Injector ejector latch Locking screw Injector ejector latch Chapter 2 Hardware Installation 2 13 2 042 2 14 10 11 Carefully align the edges of the card with the card guides in the appropriate slot It might be helpful to look into the enclosure to verify correct alignment of the rails in the guides Taking care to keep the blade server aligned in the guides slide the card in until the injector ejector latches engage the card cage Push the blade server into the midplane connectors and close the latches to seat the blade server in the connectors Tighten the locking screws to ensure that the blade server is secured into the shelf Install the Sun Netra CP3060 blade server into the shelf Go to Section 2 5 2 Installing the Sun Netra CP3060 Blade Server on page 2 15 for those instructions Installing the Sun Netra CP3060 Blade Server 1 If you have installed a Sun Netra CP3060 RTM go to the front of the system and locate the card slot where you installed the RTM at the rear of the system Remove the filler panel if necessary The filler panel is secured to the card cage using two screws one at the top of the filler panel the other at the bottom Store the filler panel in a safe place you might need to use it again if you have to remove a card for an extended perio
94. s the frutree entries and properties that describe the condition of the Sun Netra CP3060 blade server TABLE 4 9 PICL Frutree Entries and Description for the Sun Netra CP3060 Blade Server Frutree Entry Property Entry Description Example of Condition CPU location State State of the receptacle or slot Condition of the blade server or occupant CPU fru Condition CPU fru State CPU fru FRUType FRU type State of the blade server or occupant connected ok configured bridge fhs The prtpicl v command shows the condition of all devices in the PICL tree Sample output from the prtpicl command on the Sun Netra CP3060 blade server is shown in CODE EXAMPLE 4 3 CODE EXAMPLE 4 3 prtpicl Command Output prtpicl picl 5a00000001 platform sun4v 5a00000005 devctl 5a00000021 obp device 5a000000cf virtual devices 5a000000 4 obp device 5a00000103 5a00000113 5a00000120 5a00000128 5a00000138 5a00000148 5a00000158 scsi vhci memory virtual devices nvram nvram flashprom console ncp rcc serial obp device obp device loop obp device loop obp device echo obp device fma obp device sunvts obp device sunmc obp device 5a00000178 explorer obp device led obp device 5a00000198 ipmi obp device 5a000001a8 5a000001b8 5a000001c6 5a000001d4 cpu 5a000001e2 cpu cpu cpu cpu cpu cpu cpu 5a000001 0 5a000001f
95. sas H8S 2166 provides the IPM controller IPMC function on the Sun Netra CP3060 blade server The IPMC provides PICMG 3 0 board management functionality and it interfaces to the host CPU through a serial interface The IPMC subsystem is powered from the standby power The IPMC is responsible for the following m Dual buffered IPMB interfaces to connect to IPMB 0 m Serial payload interface to the host through MPC m IPMI L interface to the AMC boards m Power control of the entire Sun Netra CP3060 blade server m Power and reset control of the AMC board m Hot swap latch input and LED control m Power control m E Keying control m Environmental monitoring m Access to all environmental PC devices m Access to all I2C devices when MPC is reset Netra CP3060 Board User s Guide April 2009 5 1 6 1 5 1 6 2 5 1 6 3 5 1 6 4 5 1 6 5 Intelligent Platform Management Bus The BMR HS8S provides dual buffered Intelligent Platform Management Bus IPMB interfaces to the IPMB 0 bus on the PICMG 3 0 midplane The I2C channels on the H8S are connected the IPMB A and IPMB B through the LTC4300A IC buffers The PC buffers allow the board I C to be isolated from the midplane until the board is fully seated and the IZC bus on the midplane is idle Interface to the MPC The BMR HS8S provides one serial payload interface to the MPC This interface supports hardware flow control RTS Request To Send and CTS Clear To Send IPMB L Interface
96. sed at the same time If the serial port on Sun Netra CP3060 blade server is used the serial port on the Sun Netra CP3060 RTM should not be used and vise versa The Compact Flash connector can be used to add an optional 8 Gbyte user flash type I II memory card 1 4 Hot 5wap Support This section briefly discusses the hot swap support on the Sun Netra CP3060 blade Server There are three hot swap models described in the PICMG ATCA specification basic hot swap full hot swap and high availability HA hot swap Refer to the PICMG ATCA Specification which provides a detailed description of this subject This process uses hardware connection control to connect the hardware in an orderly sequence 1 5 System Requirements This section contains the system level hardware and software requirements for the Sun Netra CP3060 blade server Chapter 1 Introduction to the Sun Netra CP3060 Blade Server 1 11 1 5 1 1 5 2 1 12 Hardware Requirements Sun provides the following items for customer order m Sun Netra CP3060 node blade server m Sun Netra CP3060 RTM optional The RTM enables rear system I O access to the network to a boot device and to a console terminal shown in FIGURE 1 4 Refer to the Sun Netra CP3060 Rear Transition Module User s Guide 819 6689 for more information The Sun Netra CP3060 RTM is optional and must be ordered separately m Compact Flash card optional An IDE Compact Flash card is o
97. server documentation set or on the web site contact your local Sun Services representative This hardware carries a one year return to depot warranty For customers in the US or Canada call 1 800 USA 4SUN 1 800 872 4786 For customers in the rest of the world find the World Wide Solution Center nearest you by visiting our web site http www sun com service contacting solution html When you call Sun Services be sure to indicate if the Sun Netra CP3060 blade server was purchased separately and is not associated with a system Have the proper blade server identification information ready Be prepared to give the representative the blade server part number serial number and date code see FIGURE 1 6 Blade Server Part Number Serial Number and Revision Number Identification The Sun Netra CP3060 blade server part number serial number and revision can be found on labels located on the card see FIGURE 1 6 The Sun barcode labels provide the following information m SunsN Sun serial number for example 1005LCB 0626WM001M m SunPN Sun part number and dash number for example 501 7658 01 01 is the dash number m Rev Revision number of the part for example Rev 06 The Media Access Control MAC address label contains the MAC address for the blade server in printed and barcode form Chapter 1 Introduction to the Sun Netra CP3060 Blade Server 1 13 FIGURE 1 6 Sun Netra CP3060 Blade Server Barcode Labeling Blade Serv
98. t only the primary bus is utilized to connect the Compact Flash drive on the Sun Netra CP3060 blade server The bus can operate at DMA mode 4 speed Compact Flash The Sun Netra CP3060 blade server provides the option of supporting an IDE Compact Flash card In order to support higher memory capacity a Type II CF socket is provided The Compact Flash card is not hot swappable and there is no access to the CF socket once the blade server is installed in an ATCA chassis SATA controller A dual 3 Gbps SATA port is connected to the AMC connector Netra CP3060 Board User s Guide April 2009 5 1 5 5 1 5 1 5 1 5 2 Service Processor MPC885 The Sun Netra CP3060 blade server includes a MPC885 service processor subsystem used for LDOM configuration and Hypervisor interaction host system reset and boot support The Sun Netra CP3060 blade server uses the MPC885 to run the vBSC firmware on VxWorks A block diagram of the subsystem is shown in FIGURE 5 1 The following I O interfaces provided by MPC885 are used by the Sun Netra CP3060 blade server m 100T Connects to the Base Fabric switch m I2C Provides interface to I2C devices required by vBSC DIMM SPDs TOD NVRAM m UARTI Provides serial interface to IPMC Payload m UART 2 Provides console interface MPC Bus External Devices There are three MPC external devices m SDRAM 64MB SDRAM memory for the MPC with ECC m Flash 16 Mbyte flash for the MPC code m Dis
99. t command File from which to boot if diag switch is false Device s from which to boot if diag switch is false 4 10 Netra CP3060 Board User s Guide April 2009 TABLE 4 3 OpenBoot Configuration Variables Continued Variable Possible Values Default Value Description use nvramrc nvramrc security mode security password security badlogins verbosity diag switch error reset recovery boot true false variable name none full command variable name variable name max min none normal true false none sync false none none none none min false boot If true execute commands in NVRAMRC during server startup Command script to execute if use nvramrc is true Firmware security level Firmware security password if Security mode is not none never displayed Do not set this directly Number of incorrect security password attempts Controls the amount and detail of OpenBoot output Default is min none Only error and fatal messages are displayed on the system console min Notice error warning and fatal messages are displayed on the system console normal Summary progress and operational messages are displayed on the system console in addition to the messages displayed by the min setting max Detailed progress and operational messages are displayed on the system console If true After a bo
100. t sends out onto the JBus the transactions initiated from within the JBus to PCI E bridge such as from one of the PCI units I O caches and so on and presents them to the UltraSPARC T1 processor PCI Express PCI Express PCI E is a high speed point to point dual simplex chip interconnect It is designed as the next generation system bus interconnect replacing the aging PCI bus PCI E operates at 2 5 GHz and supports lane widths of x1 x2 x4 x8 x16 and x32 PCI E signaling is accomplished through the use of differential pairs per lane of the interface For a 1x link four individual lines are needed TX TX RX and RX Electrically the signaling levels are 1 2 volts PCI E transfers data using packets Each packet has a header which includes information about the packet and data payload The data payloads vary in size from 64 bytes to 4 Kbytes The maximum packet size supported for the JBus to PCI E bridge is 512 bytes Leaf A PCI E Interface The Leaf A 4x PCI E interface on the JBus to PCI E bridge connects to the Extended Fabric NIC The interface meets the PCI E 1 0a specification and is completely autonomous from the second PCI E interface on the JBus to PCI E bridge That is they are not electrically connected Netra CP3060 Board User s Guide April 2009 5 1 4 2 5 1 4 3 Leaf B PCI E Interface The Leaf B 8x PCI E interface on the JBus to PCI E bridge connects to the PCI E Switch chip The PLX chip then co
101. ternal loopback test succeeded Link is up Looking for Ethernet Packets is a Good Packet X is a Bad Packet Type any key to StOPs vic cae eid oe Sees Ka ee cn To start the watch net all diagnostic test type watch net al1 at the ok prompt 0 ok watch net all pci 1f 0 pci 1 1 network c 1 Internal loopback test succeeded Link is Up Looking for Ethernet Packets is a Good Packet X is a Bad Packet Type any key to stop OpenBoot Configuration Variables The OpenBoot configuration variables are stored in the OBP flash PROM and determine how and when OpenBoot tests are performed This section explains how to access and modify OpenBoot configuration variables For a list of important OpenBoot configuration variables see TABLE 4 3 Changes to OpenBoot configuration variables take effect at the next reboot TABLE 4 3 OpenBoot Configuration Variables Variable Possible Values Default Value Description local mac address true false true If true network drivers use their own MAC address not the server MAC address Chapter 4 Firmware and Blade Server Management 4 9 TABLE 4 3 Variable Possible Values OpenBoot Configuration Variables Continued Default Value Description fcode debug scsi initiator id oem logo oem banner ansi terminal screen columns screen rows ttya mode output device input device auto boot on error
102. toward the blade server until the retainer clips click into place see FIGURE 2 2 The small retainer clips on each side of the DDR 2 DIMM slot click into place in the notches on the sides of the DDR 2 DIMM 241 2 Removing a DDR 2 DIMM Memory Module You might need to remove a DDR 2 DIMM module from the Sun Netra CP3060 blade server if you are returning the DDR 2 DIMM module or the blade server for service or if you are replacing a module with another DDR 2 DIMM module Note Safely store the original factory shipped DDR 2 DIMM and related DDR 2 DIMM packaging You might wish to store any removed DDR 2 DIMM in the new DDR 2 DIMM packaging or use the packaging for service To remove a DDR 2 DIMM from the Sun Netra CP3060 blade server perform the following steps 1 Take antistatic precautions attach and electrically ground the wrist strap 2 8 Netra CP3060 Board User s Guide April 2009 2 4 2 Caution Always wear a grounded antistatic wrist strap when handling modules 2 Place the Sun Netra CP3060 blade server on an antistatic mat or on the blade server s antistatic bag if you do not have a mat available 3 For the DDR 2 DIMM you wish to remove simultaneously pull both DDR 2 DIMM retainer clips outward from the slot 4 Grasp the DDR 2 DIMM by the edges and carefully pull it out of its connector Place it in an antistatic bag FIGURE2 3 Removing a DDR 2 DIMM Memory Module L 1 5 If you a
103. udes a blue Hot Swap LED Hot Swapping the Sun Netra CP3060 Blade Server If the Solaris OS is running on a Sun Netra CP3060 blade server and you open the blade server s latches you see a message that the operating system will shut down in one minute When the blue LED on the blade server lights steadily you can safely remove the blade server Chapter 4 Firmware and Blade Server Management 4 15 4 6 1 1 Hot Swap LED The blue Hot Swap LED located on the front panel of the Sun Netra CP3060 blade server FIGURE 1 1 blinks when a hot swap is initiated and lights steadily when the blade server is ready to be removed from the system Unlatching the bottom latch on the Sun Netra CP3060 blade server initiates the hot swap sequence The LED lights steadily when the blade server can be safely removed from the system The reverse is true when a Sun Netra CP3060 blade server is installed into the system Once the Sun Netra CP3060 blade server is installed into the system and the bottom latch is latched the blue Hot Swap LED blinks until the blade server is ready and then turns off The green LED lights steadily when the blade server is ready FIGURE 4 1 shows the hot swap latch and Hot Swap LED FIGURE 4 1 Hot Swap Latch and Hot Swap LED Hot Swap LED blue Hot swap latch bottom latch 4 16 Netra CP3060 Board User s Guide April 2009 4 7 Network Device Aliases A device alias is a shorthand representation of a devi
104. wered I C Segment Accessed by Function Ox5C motherb Yes IPMC_I2C H8 Motherboard H W oard Monitor MB 0x98 MB Yes IPMC_I2C H8 JBus to PCI E bridge junction temperature monitor OxAC MB Yes IPMC DC H8 IPMI FRU OxAE DC DC Yes IPMC DC H8 DC DC FRU ID 0xA6 MB Yes IPMC_I2C H8 SEL EEPROM Ox5E DC DC Yes IPMC DC H8 DC DC temp sensor OxA8 MB Yes MCPFID DC MPC when operational Sun FRU ID or else H8 OxAE RIM Yes RTM DC RTM FRU ID 0xA4 MB Yes MCPFID I2C OBP HV 0xA6 MB Yes MCPEID_I2C SCC MAC 0x50 MB No MPCRUN DC MPC Voltage marginer for UItraSPARC T1 Core power 0xD0 MB No MCPFID I2C MPC RTC 0xDC MB No MPCRUN DC MPC Differential Buffer for PCI E 0xA0 MB No DDRL I2C MPC when operational Channel 0 DIMMO or else H8 0xA4 MB No DDRL DC Channel 0 DIMM1 0xA8 MB No DDRL DC Channel 1 DIMMO 0xAC MB No DDRL DC Channel 1 DIMM1 0xA0 MB No DDRR_I2C Channel 2 DIMMO 0xA4 MB No DDRR_I2C Channel 2 DIMM1 5 22 Netra CP3060 Board User s Guide April 2009 TABLE 5 3 I2C Address Map Continued Standby Address Location Powered I C Segment Accessed by Function 0xA8 MB No DDRR_I2C Channel 3 DIMMO 0xAC MB No DDRR DC Channel 3 DIMM1 0x72 AMC No Yes IMPI L H8 IPMI bus to AMC module AMC management power A few items are worth noting in the C address map shown above The Address column indicates the I2C address for the associated device Although the DC address space is only 7 bits the read write bit is appended as bit 0
105. y DIMMs to only include x4 organization Netra CP3060 Board User s Guide April 2009 5 12 53 5 1 3 5 1 3 1 5 1 4 Serial Presence Detect The Sun Netra CP3060 blade server supports autoconfiguration using serial presence detect SPD The DIMMs have small EEPROM devices that store their configuration data according to the SPD format MPC vBSC reads this data via the PC management bus and writes the data into the OpenBoot PROM A 128 byte region of the SPD EEPROM is available for user data storage allowing a limited form of FRU information to be implemented for the DIMMs System JBus The JBus is the main interconnect for the core of every UltraSPARC T1 based system The JBus is a memory coherent interconnect joining the UltraSPARC T1 processor to the JBus to PCT E bridge JBus provides a memory coherent 128 bit shared address data path between each of the JBus resident devices The JBus uses the Dynamic Termination Logic DTL signaling environment created initially for the UltraSPARC III and is a globally clocked bus designed to run with a maximum clock frequency of 200 MHz Memory coherence is maintained through the use of a snooping Modified Owned Exclusive Invalid MOESI cache protocol wherein all JBus agents must snoop their own cache hierarchies on every transaction that goes across the JBus interconnect JBus RAS Features JBus provides parity protection on all address and data transfers One set of parity signals

Sun Netra CP3060 Blade Server Userճ Guide

Contents

Download Pdf Manuals

Related Search

Related Contents