Home

NEC Express5800/320Fc Administration Manual

1. md20 active raidl sdd1 1 sdb1 0 143371968 blocks 2 2 UU md1l active raidl sdc2 1 sda2 0 2096384 blocks 2 2 UU md2 active raidl sdc3 2 F sda3 0 31647936 blocks 2 1 U md0 active raidl sdc1 1 sdal 0 2096384 blocks 2 2 UU unused devices lt none gt In this example md30 is a RAID 0 array made up of md20 and md21 The remaining stanzas are for RAID 1 arrays Note that md21 is operating in degraded mode It is missing a disk 2 1 _U Removing and Replacing Disks For disk fault tolerance disk mirrors must be maintained when disks or CPU I O enclosures are removed and replaced For information about the recommended partnering confirmation for internal disks see Configuring Internal Disks on page 5 3 This section discusses the following topics e Disk Insertion e Administering RAID Arrays 5 14 Express5800 ftServer System Administrator s Guide for the Linux Operating System Administering RAID Arrays Disk Insertion When you reinsert a pulled disk the OSM storage plugin attempts to match it with an existing disk If it finds a match it hot adds the mirror partitions on the inserted disk back into the existing RAID arrays and resynchronizes them see Resynchronization on page 5 18 Similarly if you replace a failed disk the OSM plugin automatically adds the replacement disk to a running RAID array Administering RAID Arrays This secti
2. Before performing an upgrade to a system in use perform a complete file system backup Stop ftISNMP services before upgrading the Express Builder If you upgrade the firmware to support the new Express Builder version backup the BIOS and BMC firmware files on your system If the upgrade fails or if you chose to return to the previous Express Builder version after the upgrade you will need these files to return the firmware to its earlier versions If you must restore an installation whose system files are corrupt this upgrade may fail To ensure that Express Builder is performing correctly you should either perform a complete Express Builder reinstallation or restore the system from backups Carefully read the current Release Notes Express5800 ftServer for the Linux Operating System document on http support necam com Distribution Release Notes may have been updated with information about restrictions or problems and workarounds software updates and document corrections that Release Notes from earlier distribution CDs do not have To update or reinstall Express5800 ftServer system software for the Linux operating system 1 Make sure that you have a backup system disk in case the installation fails or you want to return to the previous version of Express Builder See Creating a Backup System Disk on page 4 7 for details Ensure that your system is prepared for the upgrade by checking the items in Pre Installation Ch
3. Voltage sensors 1 150 1 152 Top I O element Top I O element 10 Mass storage controller 10 0 EIDE controller 05 00 0T SAS SATA controller 10 1 SAS SATA controller 05 01 0 USB controllers 10 2 USB host controller 05 02 0 05 02 2 VGA controller 10 3 Graphics controller 05 03 0 Ethernet controller 10 5 Ethernet card 04 02 0 04 02 1 PCI device 10 8 Core logic 03 01 0 PCI slots 9 10 11 10 9 10 11 Storage enclosure in top I O element 10 40 Internal disk slot 1 maps to sda 10 40 1 Internal disk slot 2 sdb 10 40 2 Internal disk slot 3 sdc 10 40 3 BMC 10 120 7 6 Express5800 ftServer System Administrator s Guide for the Linux Operating System The ftsmaint Command Table 7 1 Device Paths of ftServer Devices Page 3 of 3 Location Bottom I O element Device Path Fan speed sensor 10 140 Voltage sensors 10 150 10 162 Bottom I O enclosure 11 Mass storage controller 11 0 EIDE controller 7c 00 0 SAS SATA controller 11 1 SAS SATA controller 7c 01 0 USB controllers 11 2 USB host controller 7c 02 0 7c 02 2 VGA controller 11 3 Graphics controller 7c 03 0 Ethernet controller 11 5 Ethernet card 7b 02 0 7b 02 1 PCI device 11 8 Core logic 7a 01 0 PCI Slots 9 10 11 11 9 11 11 Storage e
4. v3view v3view 8 6 Express5800 ftServer System Administrator s Guide for the Linux Operating System Installing and Configuring fISNMP Run the following commands each time SNMP is restarted or write a script to manage this task snmpusm v3 u admin n 1 authNoPriv a MD5 A your_passwd localhost create v3user admin snmpusm v3 u v3user n l authNoPriv a MD5 A your_passwd localhost passwd old_passwd new_passwd These commands clone an initial template SNMPv3 user admin as v3user and then change the password of v3user The string old_passwd is the password previously assigned to the user admin Another way to add SNMPv3 users is to run the following three commands stop snmp net snmp config create snmpv3 user start _snmp This series of commands adds lines in the proper configuration files to add a user to the system Be sure to run stop_ snmp before running the net snmp config command which prompts you for a user name and password The command adds acreate user snmpv3 user MD5 password DES line entry into etc opt ft snmp snmpd conf file This line is automatically replaced with a key on SNMP restart because it contains the correct password The net snmp config command also puts an rwuser entry into the etc opt ft snmp snmpd conf file You will be prompted for a user name and password when you enter the net snmp config command You will need to configure snmpd conf with VACM entries for a
5. O operating system upgrade 4 2 operational states managing 8 25 state definitions 8 30 OpState reason definitions 8 31 state definitions 8 30 optional software components 2 6 P partition tables creating 5 6 displaying 5 4 partitions adding 5 6 managing 5 4 password default root 2 2 valid 2 2 path 7 3 PCI enumeration enabling 2 13 pedestal system orientation 2 4 R rack system orientation 2 4 RAID 5 9 arrays 5 2 5 9 checking the current state 5 14 creating and mounting a file system 5 14 problems 9 5 RAID 0 5 9 RAID 1 5 9 configured drives during installation 2 8 fault tolerance 1 1 system pair 2 17 two CPU I O enclosures 5 2 resynchronization 5 18 RAID arrays 5 9 README file for SNMP 8 3 reason definitions 8 31 recovering from a disaster 5 26 from an upgrade 4 8 Red Hat Linux CD ROMs 2 2 documentation 1 4 upgrading 4 4 reinstalling Linux operating system 4 5 reinstalling Express Builder 2 14 remote network management services 8 16 removing ftISNMP 8 29 rescue mode for Linux 2 15 restoring the Express Builder 4 6 restoring the Linux operating system preparation checklist 4 3 recovery from 4 8 resynchronization of RAID mirrors 5 18 root password default 2 2 RPM files 8 3 runlevel controlled process configuration 2 6 S safe bring down 8 28 SAS SATA drives checking 5 8 default configuration 2 5 5 8 firmware 2 5 RAID 2 5 SCSI drive
6. The following command creates an ext 3 journaled file system in the RAID 0 array created above mkfs ext3 dev md30 You can mount the file system on a convenient mount point big_ data as follows mkdir big data mount dev md30 big data To verify the work use the command af to show the size of the file system df big data Use the command 1s to show the lost found directory in the file system 1s 1 big data At this point be sure to add the mount to the etc fstab file so the file system is mounted during boot Reboot the system to make sure it works NOTE A single disk that is not part of a RAID array must have a file system mounted on it to ensure that its operation state is reported correctly Checking the Current State of RAID The mdstat file displays the current state of RAID It shows all running RAID arrays and their current status including which mirrors are present whether they are synchronized and more See Example 5 2 Setting Up the ftServer System 5 13 Removing and Replacing Disks NOTE The device names displayed in proc mdstat are the kernel names for each device These are different from the user device names displayed by the mdadm command Example 5 2 Checking the Current State of RAID cat proc mdstat Personalities raid0 raid1 md30 active raidO md21 1 md20 0 286743808 blocks 64k chunks md21 active raidl sdf1 1 sde1 0 143371968 blocks 2 1 _U
7. Firmware Rev ECO Level Min Partner ECO Level Serial MTBF Fault Count MTBF Last Timestamp MTBF Threshold MTBF Value MTBF Type Logic Revision Then run the following command opt ft bin ftsmaint ls 1 0 Combined CPU IO ONLINE DUPLEX SECONDARY ON OFF ON AA G94340 1390 37 0 401318 0 None 1200 0 useThreshold 18023 NOTE To determine what model your system is type the ftsmaint 1sSystem command The model is identified in the Description field Obtain the latest BIOS image for the Express5800 ftServer and copy it to the ftServer tmp folder Updating ftServer System Firmware 3 3 Updating the System BIOS 6 Perform the BIOS burn by issuing the following commands to one of the CPU I O enclosures opt ft bin ftsmaint bringDown 0 Completed bringDown on the device at path 0 opt ft bin ftsmaint burnProm Path and filename for the BIOS Updated firmware on the device at path 0 opt ft bin ftsmaint jumpSwitch 0 Transferred processing to the device at path 0 opt ft bin ftsmaint bringUp 1 Completed bringUp on the device at path 1 The preceding commands burn the BIOS to both CPUs as follows e bringDown Takes the CPU 0 element in the upper enclosure out of service e burnProm Burns the new BIOS to CPU 0 e jumpSwitch Brings CPU 0 up runs diagnostics on it synchronizes it with CPU 1 in lower enclosure and then takes CPU 1 down e bringUp Bring
8. If necessary upgrade the firmware see Chapter 3 Optionally upgrade the Linux operating system as described in Upgrading or Restoring the Linux Operating System on page 4 3 Before upgrading the operating system software check with the NEC Corporation of America Customer Support Center or your authorized NEC service representative to ensure that your ftServer system supports the new version Upgrade the Express Builder as described in Upgrading or Restoring Express Builder on page 4 6 Related Considerations The upgrade and reinstallation processes do not overwrite the following files if you have modified them e etc modprobe d ft network conf e etc sysconfig network scripts ifcfg bondo e etc sysconfig network scripts ifcfg bond1 e etc sysconfig network scripts ifcfg eth000010 e etc sysconfig network scripts ifcfg eth000011 e etc sysconfig network scripts ifcfg eth080010 e etc sysconfig network scripts ifcfg eth080011 If the RPM file on an upgrade CD contains updated versions of these files and if you have modified the original file the upgrade process saves the updated file to the etc OPT ft network scripts ARCHIVE directory giving each file a rpmnew extension To complete an upgrade to your system do one of the following e f you do not want to preserve your changes copy the rpmnew file to the appropriate directory but remove the rpmnew extension 4 2 Express5800 ftServer System Ad
9. NMS However you can use a commercial or open source NMS to manage the Net SNMP and ftISNMP packages remotely you can also manage these packages directly from a remote system using Net SNMP The SRA ftLinux MIB file must be provided on the managing host s and must support the Express Builder release installed on the managed system s If different systems use different Linux operating system releases the SRA ftLinux MIB file must reconcile differences or your SNMP NMS will not be able to manage mismatched object IDs The SRA ftLinux MIB file for a later Linux operating system release will likely work with the earlier Linux operating system releases but may require some adjustment for different defined objects in the proc file system The MIBs in ASN 1 encoded text form are located in opt ft mibs usr share snmp mibs and subordinate directories by default Note that the SRA ftLinux MIB file is present in the opt ft mibs directory and is named SRA ftLinux MIB txt Load all of the MIB files you require into the SNMP NMS certainly SRA ftLinux MIB will be necessary to manage ftServer objects Configure the SNMP NMS to avoid verbose OID object ID printouts that may clutter the display The minimum part of the OID needs to be displayed to provide the object s unique name 8 26 Express5800 ftServer System Administrator s Guide for the Linux Operating System Initial SNMP Testing NOTES 1 Net SNMP and ftISNMP do not require the SNMP
10. NMS and the package does not provide one Choice installation and configuration of the SNMP NMS is your responsibility 2 The SRA ftLinux MIB file is only useful for managing ftServer Linux based systems Initial SNMP Testing On a system with an SNMP aware NMS you start the NMS before starting SNMP servers Start the master agent and the ftISNMP subagents by typing start_snmp After this command completes master agent and subagent processes with the names snmpd ft lsubagent and ft ltrapsubagent should be running Verify this ps aux grep snmnp ps aux grep ftl In this command process is one of the process names listed above Any errors or warnings generated during the startup script s execution are posted to syslog and stderr See Deploying SNMP Agents and Subagents on page 8 145 for the default destinations of messages logged by the master agent and subagent processes and The ft lsubagent conf and ftltrapsubagent conf Files on page 8 133 for information on how to change the location of the subagent logs Note that there are many other commands available for managing Net SNMP and ftISNMP See ftISNMP Management Commands on page 8 137 and ftisnmp_scripts 8 for descriptions of other commands and read comments in the script files To terminate these processes run the stop_ snmp command Initial Testing of ftltrapsubagent The ftltrapsubagent conf file allows you to control trap filter
11. SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206134940 622184 300 RFC1213 MIB sysUpTime 0 Timeticks 12130 0 02 01 30 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 40 1 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING ONLINE SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206134941 625363 300 RFC1213 MIB sysUpTime 0 Timeticks 12231 0 02 02 31 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 5 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206134942 629238 300 RFC1213 MIB sysUpTime 0 Timeticks 12431 0 02 04 31 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrap
12. SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206135102 760559 300 RFC1213 MIB sysUpTime 0 Timeticks 21744 0 03 37 44 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 40 1 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING REMOVED SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206135117 760098 300 RFC1213 MIB sysUpTime 0 Timeticks 23244 0 03 52 44 SNMPv2 MIB snmpTrapOID 0O OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 120 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PRIMARY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206135132 761581 300 Simple Network Management Using Net SNMP and ftISNMP 8 37 Trap Filtering RFC1213 MIB sysUpTime 0 SNMPv2 MIB snmpTrapOID 0 Timeticks OID 26444 0 04 24 44 SRA ftLinux MIB ftcTrapGenericInformationTrap 0
13. SRA ftLinux MIB OID Values and Properties The SRA ftLinux MIB file contains all ftISNMP defined object identifiers and associated operation classes and properties with some notes inserted to assist you You can grep through the SRA ftLinux MIB txt file for information about OID values and properties or open it with an editor or browser to view and search for information The default installation places SRA ftLinux MIB at opt ft mibs SRA ftLinux MIB txt 8 32 Express5800 ftServer System Administrator s Guide for the Linux Operating System Trap Filtering Trap Filtering This section discusses the following topics e Trap Filtering Capability e Activating and Deactivating Trap Filtering e Trap Filtering Examples Trap Filtering Capability ftISNMP provides the ability to filter out transitional traps Traps are messages that inform you about network events Hardware components that go in and out of service trigger a number of traps that are seen at the management client Some of these traps are actually transitional state information for devices For example when you bring up a CPU element the CPU board s state changes from DIAGNOSTICS to INITIALIZING ONLINE and then DUPLEX However if you are interested in only the end state for example ONLINE and DUPLEX the trap filtering capability is useful Another reason to use the trap filtering capability is that some SNMP traps are triggered by obvious reason
14. Simple Network Management Using Net SNMP and fISNMP 8 11 SNMP Foundations and Concepts NOTE The Express Builder installation automatically creates the SRA ftLinux MIB file in the opt ft mibs directory while Net SNMP creates its MIBs in the usr share snmp mibs directory MIBs can be stored in a variety of locations but running SNMP agents must still be directed to the location of a MIB the first time it is to be used if the MIB is added after the agents have already started If the MIB was not in the path when the SNMP services were started and SRA ftLinux MIB exported the following example shows how to identify the SRA ftLinux MIB file to the SNMP tools snmpget m SRA ftLinux MIB v 2c c public myhost com ftcBdState 1 SRA ftLinux MIB ftcBdState 1 INTEGER duplex 21 As you begin to develop your own MIB or MIBs for your management requirements you can save a lot of work by adopting defined variables from other MIBs A large number of MIBs are defined by the IETF and are available as plain text files You should use standardized MIBs where they define objects to avoid non standard implementation of networked objects Do not alter standard MIBs If you need additional object definitions you can add another MIB or create your own The MIB defined objects can be queried and data recovered that provides a basis for SNMP agent operations You can create scripts that the SNMP agent or a subagent executes according to M
15. 01 58 29 SNMPv2 MIB snmpTrapOID 0O OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 0 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206134938 609247 300 RFC1213 MIB sysUpTime 0 Timeticks 11929 0 01 59 29 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PRIMARY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206134939 616628 300 8 34 Express5800 ftServer System Administrator s Guide for the Linux Operating System Trap Filtering RFC1213 MIB sysUpTime 0 Timeticks 12030 0 02 00 30 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 40 1 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING
16. 20 self checking 1 1 supported 2 7 host name configuring 5 32 hyperthreading disabling 5 32 installing ftISNMP 8 1 8 2 Express Builder after a failed attempt 2 14 overview 2 2 restoring an installation 4 6 Linux operating system 1 2 2 1 4 1 installer interfaces 2 7 overview 2 2 pre installation checklist 2 7 remote network management services 8 16 J jumpswitch 3 4 K kernel memory dump file management 7 13 L LEDs 2 6 9 1 simplex duplex indicator 8 28 legacy devices 5 1 USB storage device 6 3 Linux operating system 1 1 2 7 configuring 1 3 default setup 2 3 distribution 2 2 installation 1 2 2 1 network administration 1 2 pre installation checklist 2 7 recovering from an upgrade 4 8 restoring 4 1 system administration 1 2 upgrading 1 2 version information 2 4 logrotate 8 6 logs console 5 2 message 9 6 Ispci command and fault tolerance 5 30 lspci command 5 30 lsusb command 6 1 MAC addresses 5 31 management data storage devices 1 3 6 1 partitions 5 4 SNMP 8 1 memory dumps 7 13 MIB management information base 8 11 8 24 objects defined by 8 12 SRA ftLinux MIB 8 12 missing drivers 9 3 mounting a file system 5 14 Index 3 Index N naming devices 6 1 Net SNMP 8 1 8 26 basic commands 8 10 description 8 2 network administration 1 2 network management stations NMS 8 26 Network Time Protocol NTP NTP See Network Time Protocol
17. 37 27 gt opt ft bin ftsmaint ls Modelx H W Path Description State OPState FRev Fct AA G90730 0 Combined CPU IO ONLINE DUPLEX 0 AA M23100 0 0 1GB DDR 2 DIMM ONLINE ONLINE AA M23100 0 1 1GB DDR 2 DIMM ONLINE ONLINE 0 2 MISSING EMPTY 0 3 MISSING EMPTY 0 4 MISSING EMPTY 0 5 MISSING EMPTY 0 6 MISSING EMPTY 0 7 MISSING EMPTY 0 20 15 4 1 ONLINE ONLINE 0 20 130 Internal Temp NORMAL 0 20 150 12V NORMAL 0 23 000 ONLINE ONLINE 0 23 130 Internal Temp NORMAL 0 23 150 12V NORMAL 0 130 Ambient Air Temp NORMAL 0 140 Fanl NORMAL 0 141 Fan2 NORMAL 0 150 1 2V VTT NORMAL 0 151 1 8V VDD NORMAL 0 152 12V NORMAL AA G90730 1 Combined CPU IO ONLINE DUPLEX 0 AA M23100 1 0 1GB DDR 2 DIMM ONLINE ONLINE AA M23100 1 1 1GB DDR 2 DIMM ONLINE ONLINE 1 2 MISSING EMPTY 1 3 MISSING EMPTY 1 4 MISSING EMPTY 1 5 MISSING EMPTY 1 6 MISSING EMPTY 1 7 MISSING EMPTY 1 20 15 4 1 ONLINE ONLINE 1 20 130 Internal Temp NORMAL 1 20 150 12V NORMAL 7 10 Express5800 ftServer System Administrator s Guide for the Linux Operating System The ftsmaint Command AA G90730 AA U57500 AA U57500 AA D64200 AA D64300 AA G90730 1 23 1 23 130 1 23 150 1 130 1 140 1 141 1 150 1 151 1 152 LO L0
18. BIOS on the booting enclosure scans the list of bootable devices as configured in the BIOS looking for a device to boot When the search finds the disks they are analyzed from bottom to top in the boot enclosure When a disk with a boot partition is found it is booted The boot sequence is a multi stage process e The GRUB boot loader is loaded and started e The GRUB boot loader loads the second stage GRUB which selects the Linux kernel to boot loads and starts it The boot process uses the raw disk RAID is not involved NOTE For the boot sequence to work a bootable medium must be found in the active CPU I O enclosure For example if you try to boot the top CPU I O enclosure the boot sequence fails if sda is missing and there is no other bootable disk If a boot partition is found in sdb or sde it is booted If the boot fails the system switches the active CPU I O enclosure and tries again where it will find and boot sdd if present e When the kernel starts it loads the SCSI driver which in turn loads the SAS SATA driver e The SAS SATA driver spins up all the disks it discovers It also collects a list of all of the type Oxfd Linux RAID autodetect partitions e When the RAID 1 module is loaded it processes this list of mirrors and starts RAID arrays It processes mirrors from disks in the following order sda sdd sdb sde sde and sdf If a RAID array was not cleanly shut down a resync is started NO
19. Care must be taken with recycled disks The partition table and RAID superblocks that exist on the disk can confuse the system To remove a defective disk and insert a spare disk 1 2 3 Remove the defective disk from any RAID arrays that it belongs to using the instructions presented in Removing a Faulty Mirror on page 5 17 Physically remove the disk Insert a spare disk in the slot previously occupied by the defective disk The drive spins up automatically 5 20 Express5800 ftServer System Administrator s Guide for the Linux Operating System Administering RAID Arrays To verify that the spare disk is not in use Type the following commands and check the resulting output mdadm detail scan swapon s cat etc mtab To zero the spare disk Perform one of the following procedures e Zero the spare disk s RAID superblocks by typing a command such as the following for each partition on the spare disk substitute the device node of the partition you wish to zero for sdb1 in this example mdadm zero superblock dev sdb1 NOTE Zeroing the disk s RAID superblocks takes very little time but may not remove everything from the disk If you are concerned about this zero the entire disk as described in the following step e Zero the entire spare disk by typing a command such as the following substitute the device node of the disk you wish to zero for sdf in this example dd if dev zero
20. Firmware a character file rather than a binary file You can detect such corruption by computing a checksum with the md5sum command before and after copying A repeated BIOS burn failure is likely to be caused by a command syntax error or by using a damaged or inappropriate BIOS image file 10 If it is necessary to update the BMC firmware follow the procedure described in Updating BMC Firmware on page 3 5 11 If you now need to update the Linux operating system and Express Builder place the Express Builder CD in the upper CD ROM drive and reboot See the Release Notes Express5800 ftServer for the Linux Operating System and Chapter 2 for operating system update procedures Updating BMC Firmware Each I O element contains a socketed Baseboard Management Controller BMC chip Firmware updates are provided on a bootable CD ROM To check the BMC firmware version type the following command lopt t bin ftsmaint ls 10 120 Firmware Rev 7 0 0 The preceding example displays a BMC firmware version number of 7 0 0 for CPU element 0 J CAUTION Do not interrupt a BMC firmware update with a burn in progress Interrupting power can result in EEPROM damage or corrupted BMC firmware that requires field service or replacement I O elements to recover To update the BMC firmware from a CD 1 Insert the ExpressBuilder for Linux CD in the CD ROM drive in the CPU I O enclosure whose power button is lit 2 Log on to the s
21. MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING INITIALIZING SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051207143724 381849 300 RFC1213 MIB sysUpTime 0 Timeticks 467013 1 17 50 13 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 0 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING DUPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PRIMARY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051207143732 824273 300 RFC1213 MIB sysUpTime 0 Timeticks 467114 1 17 51 14 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 2 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING DUPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING SECONDARY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051207143733 828058 300 Example 8 4
22. SCSI revision 05 Host scsi4 Channel 00 Id 00 Lun 00 Vendor ATA Model ST380013AS Rev 3 00 Type Direct Access ANSI SCSI revision 05 Storage Device Definition The Linux operating system automatically creates device nodes for all devices in a system Miscellaneous SCSI devices such as scanners are generally mapped as dev sg devices Note that the Linux operating system also allows some non SCSl devices to be addressable as SCSI pseudo devices This can be useful to allow certain SCSI software packages to work with non SCSI devices Setting Up RAID Arrays This section discusses the following topics e RAID Array Overview e Creating a RAID 1 Array e Creating a RAID 0 Array e Creating and Mounting a File System e Checking the Current State of RAID RAID Array Overview RAID is the basis for fault tolerant file system availability As disks come in and go out of service the only way to keep the file system available is to mirror it on multiple disks with a disk in each CPU I O enclosure All of the file systems are created on RAID devices The system supports RAID 1 mirrored and RAID O striped on RAID 1 RAID 0 is configured using RAID 1 devices since the underlying devices must be fault tolerant Each RAID array has a number for example dev md23 that must be unique among the running RAID arrays The RAID array numbers are in the range 0 through 128 5 8 Express5800 ftServer System
23. STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206134945 675906 300 RFC1213 MIB sysUpTime 0 Timeticks 14244 0 02 22 44 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 3 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING OFFLINE SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PARENT EMPTY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206134945 675906 300 RFC1213 MIB sysUpTime 0 Timeticks 14444 0 02 24 44 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 4 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING OFFLINE SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PARENT EMPTY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206135004 762343 300 RFC1213 MIB sysUpTime 0 Timeticks 17244 0 02 52 44 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 5 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapG
24. The system log provides details about the device including its size scsi5 SCSI emulation for USB Mass Storage devices Vendor LEXAR Model JUMPDRIVE SECURE Rev 3000 Type Direct Access ANSI SCSI revision 02 SCSI device sdaz 506880 512 byte hdwr sectors 260 MB Additional Resources Linux Allocated Devices LANANA http www lanana org docs device list devices txt Managing Data Storage Devices 6 5 Additional Resources 6 6 Express5800 ftServer System Administrator s Guide for the Linux Operating System Chapter 7 Using ftServer Fault Tolerant Utilities and Software This chapter discusses the following topics e The tsmaint Command e Kernel Memory Dump File Management e Kernel Memory Dump File Management The Express5800 ftServer System Software for the Linux Operating System Express Builder provides a special command interface ftsmaint for managing the fault tolerant components of your ftServer system It also includes a monitoring and diagnostic package the Express Service Network that enables your ftServer system to interact with the Express Service Network When you configure the Express Service Network the NEC Corporation of America Customer Support Center or your authorized NEC service representative can receive alarm notifications when faults or other significant events occur on your system and can remotely diagnose problems The following sections explain how to use the ftsmaint
25. Without a Kickstart File 2 18 Additional Documentation and Resources 2 19 3 Updating ftServer System Firmware 3 Updating the System BIOS 3 1 Updating BMC Firmware 3 5 4 Updating the Operating System and Express5800 ftServer System Software 4 1 General Upgrade Considerations 4 2 Upgrade Requirements 4 2 Related Considerations 4 2 Upgrading or Restoring the Linux Operating System 4 3 Express5800 ftServer Kernel Modules 4 4 Upgrading the Linux Operating System 4 4 Restoring the Linux Operating System 4 5 Upgrading or Restoring Express Builder 4 6 Creating a Backup System Disk 4 7 Recovering from a Failed Software Upgrade 4 8 Related Information and Resources 4 8 5 Setting Up the ftServer System 5 1 Setting Up Internal Disk Storage 5 2 Internal Disk Storage Overview 5 2 The Console Log and the var log messages File 5 2 Configuring Internal Disks 5 3 Managing Partitions 5 4 iv Express5800 ftServer System Administrator s Guide for the Linux Operating System Contents Default Internal Disk Configuration for a Newly Installed System 5 8 Checking the Current State of the Internal Disk Subsystem 5 8 Storage Device Definition 5 9 Setting Up RAID Arrays 5 9 RAID Array Overview 5 9 Creating a RAID 1 Array 5 10 Creating a RAID 0 Array 5 12 Creating and Mounting a File System 5 14 Checking the Current State of RAID 5 14 Removing and Replacing Disks 5 15 Disk Insertion 5 16 Administering RAID Arrays 5 16 To Stop a RAID Array and Mo
26. a RAID 0 array from the RAID 1 arrays If you want to start the RAID O arrays automatically add entries for them to etc mdadm conf Otherwise the RAID 0 arrays are not started Automatic Reboot After Boot Monitoring Timeout When the system is booted into certain modes such as RAID repair mode the system heartbeat is not enabled After a defined period the default is 10 minutes the system is automatically rebooted if a heartbeat has not been received If your troubleshooting and repair requires more than the defined period you must disable boot monitoring in the BIOS during the boot sequence To disable boot monitoring 1 When the system is booting and the progress bar has started to fill press F2 to enter the BIOS Setup program An Entering Setup message appears but it may take several minutes for the BIOS Setup program to run 2 On the menu use the DOWN ARROW key to select Monitoring Configuration Press ENTER 3 Onthe Monitoring Configuration menu use the DOWN ARROW key to select Boot Monitoring 4 Select Boot Monitoring and use the PLUS SIGN key to change the value to Disabled 5 Press ESC to exit from the submenu 6 Inthe Setup Confirmation dialog box select Yes and press ENTER to save the new settings and exit from the BIOS Setup program After resolving the problem reenable boot monitoring during the next boot by following the same procedure but in step 4 change the value to Enabled Troubleshooti
27. a name for example sde This is the device s internal name that is displayed by commands such as proc mdstat and in the system log When mounting and unmounting a USB device do not use this name Instead use the name assigned by the udev command This name has the format sd usb For example sd1usb is the name of the device attached to port 1 of the root USB hub Managing Data Storage Devices 6 1 USB Storage Devices If a device is plugged into a USB hub the name has two numbers For example sd1 3usb is the name of the device attached to port 3 of a hub connected to port 1 of the root USB hub If you add another hub to the chain the device name would contain a third number The udevinfo command translates the internal name into the name assigned by the udev command For example the following command and output show that the udev command has assigned the name sd1usb to the device with the internal name sde udevinfo q name p sys block sde sdlusb The 1s command displays the device node for this device ls 1 dev sdlusb brw rw 1 root disk 8 64 Oct 24 16 18 dev sdlusb You can also translate the name assigned by the udev command into the internal name For example 1ls 1 dev sd usb brw rw 1 root disk 8 64 Oct 24 16 18 dev sdlusb 6 2 Express5800 ftServer System Administrator s Guide for the Linux Operating System USB Storage Devices Once you have the name assigned by the udev command in th
28. and ftISNMP 8 1 Installing and Configuring ftISNMP 8 1 ftISNMP Inventory 8 2 Manually Installing and Upgrading the ftISNMP RPM 8 3 ftISNMP Prerequisites 8 4 SNMP Configuration File Updates 8 4 The snmpd conf File 8 5 The ftlsubagent conf and ftltrapsubagent conf Files 8 5 Configuring SNMP to Start at System Initialization 8 6 Configuring SNMP for Service Management 8 6 SNMP Foundations and Concepts 8 8 ftISNMP Management Commands 8 9 The Basic Net SNMP Commands 8 10 MIBs 8 11 Some Objects Defined by Standard MIBs 8 12 SNMPv3 Support 8 14 SNMP s View of a Network 8 14 Extensions and Fault Tolerant SNMP Operation 8 15 Installing Remote Network Management Services 8 16 Configuring SNMP for Remote Service Management 8 17 Deploying SNMP Agents and Subagents 8 17 Verifying Traps 8 18 Managing SNMP 8 18 Testing Your SNMP Configuration 8 19 Managing ftServer Hardware Components 8 20 Example Managing Hardware 8 21 Testing Ethernet Ports 8 23 Example Testing Ethernet Ports 8 23 SNMP and MIBS 8 24 Device Enumeration 8 25 Express5800 ftServer System Administrator s Guide for the Linux Operating System Contents ftServer System Operation State Management 8 25 SNMP Network Management Station Considerations 8 26 Initial SNMP Testing 8 27 Initial Testing of ft ltrapsubagent 8 27 Initial Testing of ft lsubagent 8 29 Removing ftISNMP 8 29 OpState State Definitions 8 30 OpState Reason Definitions 8 31 GET and SET Operations for ft
29. changed message 5 33 creating a file system 5 14 D data rate of serial ports 2 6 default configuration notes 2 17 Ethernet configuration 2 5 Linux operating system 2 3 SAS SATA drive configuration 5 8 settings 2 5 system initialization 2 6 default password 2 2 device enumeration 7 5 8 25 device names 6 1 disaster recovery 5 26 disk drives inserting 5 16 removing 5 15 replacing 5 19 distribution Linux operating system 2 2 CD ROMs 1 2 2 2 separately released and optional Index 1 Index components 2 6 documentation 1 3 4 8 5 34 ftServer systems 1 4 Linux operating system 1 4 UNIX 1 4 drivers missing 9 3 dumps 7 13 system 7 13 duplex 8 31 LED indicator 8 28 E error log messages keyboard 9 6 mouse 9 6 Ethernet configuration 2 5 channel bonding 5 28 MAC addresses 5 31 naming Ethernet devices 5 26 PCI adapters adding 5 30 testing Ethernet ports 8 23 Express Service Network 7 1 architecture 1 1 optional software 1 3 Express Builder operational states 8 25 recovering from an upgrade 4 8 restoring 4 6 upgrading 1 2 using 1 3 F failed disk replacing 5 19 fault tolerance 7 1 Express Builder software 1 3 hardened drivers 5 1 hardware 1 1 RAID disk arrays 1 1 5 9 file systems availability 5 9 RAID 5 9 files and upgrade procedure 4 2 firmware 2 7 BIOS 2 7 BMC 2 7 SAS SATA drives 2 5 updating 2 2 3 1 3 5 floppy drives 6 4 fl
30. command Command m for help o Building a new DOS disklabel Changes will remain in memory only until you decide to write them After that of course the previous content won t be recoverable The number of cylinders for this disk is set to 17849 There is nothing wrong with that but this is larger than 1024 and could in certain setups cause problems with 1 software that runs at boot time e g old versions of LILO 2 booting and partitioning software from other OSs e g DOS FDISK OS 2 FDISK Warning invalid flag 0x0000 of partition table 4 will be corrected by w rite Command m for help Enter the n command to add a new partition Command m for help n Command action e extended p primary partition 1 4 Enter e or p to specify the desired type P Partition number 1 4 Setting Up the ftServer System 5 5 Setting Up Internal Disk Storage 5 Enter the partition number you wish to assign the choices depend on the type specified Partition number 1 4 1 First cylinder 1 8924 default 1 6 Enter the desired starting cylinder number for the partition or press ENTER to accept the default this example accepts the default Using default value 1 Last cylinder or size or sizeM or sizeK 1 8924 default 8924 7 Enter the desired last cylinder number for the partition the size in megabytes or kilobytes or press ENTER to accept the default this example accepts the default Using de
31. command and how to configure the Express Service Network and also information about managing dump files and about system load The ftsmaint Command The tsmaint command provides a control interface for managing your ftServer system s fault tolerant functions To see basic command options type the ftsmaint command The tsmaint command arguments support both device query and management tasks Some of the command arguments only apply to certain devices or systems which you must specify following the command argument Most of the ftsmaint command task arguments require an enumerated hardware specification argument following them indicated by path in the command descriptions that follow Any hardware path value shown in ftsmaint 1s output can be used as a path argument Using ftServer Fault Tolerant Utilities and Software 7 1 The ftsmaint Command The task arguments are as follows See also ft smaint 8 ftsmaint ls path This command displays the status of the hardware specified by the enumerated path Specifying a path displays a detailed status of the hardware at that path Omitting the path argument displays a less detailed table of all fault tolerant devices on the system See Device Path Enumeration on page 7 5 for more information Output from ftsmaint 1s path reflects what the OSM reports about the state of a given component Because of system latency this may not reflect the immediate state of the device How
32. command will not permit you to bring down a simplex device because this would disable the system ftsmaint bringUp path This command brings into service the CPU element I O element or CPU I O enclosure slot specified by path No other devices are supported ftsmaint burnProm fw_file path This command updates the firmware contained in the file fw_path into the EPROM devices on the ftServer device specified by path This command can only be used to update BMC and BIOS firmware ftsmaint clearMtbf path This command clears the MTBF value of the CPU I O enclosure CPU I O enclosure or CPU I O enclosure slot specified by path ftsmaint identify start stop path This command starts or stops the LEDs on the device specified by path The device can be a CPU board an I O board or an I O slot ftsmaint dump path This command generates a dump of the BMC or CPU element specified by path BMC dumps are stored in var crash date bmcx dmp where x is 10 or 11 CPU dumps are stored in var crash YYYY MM DD hh mm lvmcore See Table 7 1 for path values for BMC and CPU ftsmaint powerOn modem This command supplies electrical power to the modem ftsmaint powerOff modem This command removes electrical power from the modem ftsmaint reset modem This command restores modem settings to their factory defaults ftsmaint resetMtbf path This command resets the MTBF value of the CPU element I O element or CPU I O enclosure slot specifie
33. element 0 and a final octet 2 identifies CPU element 1 snmpset v 3 t 40 1 authNoPriv u v3user A new_passwd localhost 1 3 6 1 4 1 458 107 1 2 1 2 3 1 13 1s test To bring up a CPU element snmpset v 3 t 40 1 authNoPriv u v3user A new_passwd localhost 1 3 6 1 4 1 458 107 1 2 1 2 3 1 11 1 s test To bring down an I O element Note that a final octet 1 identifies I O element 10 and a final octet 2 identifies I O element 11 snmpset v 3 t 40 1 authNoPriv u v3user A new_passwd localhost 1 3 6 1 24 458 1 07 142 23 331 14 1 Test To bring up an I O element snmpset v 3 t 40 l authNoPriv u v3user A new_passwd localhost W326 cd 4 el Ab 8071 2 aS aS3e elo Cest Express5800 ftServer System Administrator s Guide for the Linux Operating System Managing SNMP To control firmware burn FWBURN snmpset v 3 t 40 1 authNoPriv u v3user A new_passwd localhost L 366 1 4 1 458 107 4 2 1 2 3 1 15 1L S FWBURN Example Managing Hardware In this example only relevant portions of the ft smaint command output are shown The following example illustrates bringing a CPU I O enclosure down and then back up Issue the following command to check the status of CPU element 0 I O element 10 opt ft bin ftsmaint ls 0 H W Path 0 Description Combined CPU IO State ONLINE Op State DUPLEX Reason SECONDARY LED Green ON LED Yellow OFF LED White ON The command output shows t
34. hd0 Checking if boot grub stagel exists no Checking if grub stagel exists yes Checking if grub stage2 exists yes Checking if grub e2fs stagel 5 exists yes Running embed grub e2fs stagel 5 hd0 16 sectors are embedded succeeded Running install grub stagel hd0 hd0 1 16 p hd0 0 grub stage2 grub grub conf succeeded Done grub gt quit On your own system replace the dev sdd shown in Example 5 4 with the device node for your spare disk In the root hd0 0 command the second zero is thenumber of the partition to GRUB GRUB partitions are zero based rather than one based so these commands actually indicate that partition 1 on dev sdd has been GRUBbed Setting Up the ftServer System 5 23 Administering RAID Arrays The duplex blank disk Command The duplex blank disk command prompts you for all of the information required to pair a spare disk with a running disk You can run it by typing opt ft bin duplex blank disk In Example 5 5 the command prompts you for information that is needed to pair a spare internal disk with the running system disk Example 5 5 Pairing a Spare Internal Disk with the Running System Disk opt ft bin duplex blank disk Device Path ID of blank disk e g 10 40 1 or 70 1 11 40 1 Device node s for 11 40 1 dev sdd Is this the correct blank disk device yes no y Device Path ID of source disk e g 10 40 1 or 70 1 10 40 1 Device node s for
35. managed object on a network is uniquely identifiable SNMP uses ISO Abstract Syntax Notation Standard 1 ASN 1 to place every SNMP object within the internet hierarchy of managed objects All these unique managed objects can be managed by their defined characteristics in the MIB While in the theoretical schema there is only one MIB it is usual to refer to any file that provides SNMP MIB definitions as a MIB MIBs can be formally registered and entered into defined namespace or used locally as experimental MIBs An SNMP server only knows of objects for which it has definitions This allows distributed SNMP services to co exist on networks without interfering with each other SNMP agents can however interact SNMP agents can act as subagents of a master agent A managed object can be a host computer or subsystem an arbitrary interactive device or a software application including an operating system basically anything whose interactivity over the network can be defined in a MIB so that it can be interfaced via SNMP Extensions and Fault Tolerant SNMP Operation While Net SNMP supports the security features of SNMPv3 it can also interact compatibly with distributed SNMP services that use SNMPv1 and SNMPv2 Net SNMP is the most widely adopted open source SNMP utility package This facilitates interfacing the Net SNMP and ftISNMP implementations with other servers deploying Net SNMP based distributed SNMP services and service management utiliti
36. may be defined will not obsolete existing SNMP servers and their managed objects SNMP s View of a Network SNMPv1 defines a simple and robust internet protocol based communications method for tracking the status of and managing almost any network interactive item that is sufficiently defined as an object in a MIB SNMP normally uses UDP protocol implemented on socket based IP communications but may also be implemented using TCP IP and another IP based protocol and also on non IP protocols such as RS 232 serial communications by spoofing an IP based communication or by piggybacking it on another transmission or transfer protocol SNMP can also take advantage of common security enhancements implemented over IP such as the Secure Socket Layer and other encryption authentication and remote access technologies provided by for example ssh the Openssh package SNMPv2 expands management capabilities of SNMPv1 by providing a mechanism for more easily defining the managed objects that SNMP communicates with SNMP refines SNMPv2 definitions and adds important security features Net SNMP supports SNMPv1 SNMPv2 and SNMPv3 protocols Because of the simple basic structure of 8 14 Express5800 ftServer System Administrator s Guide for the Linux Operating System SNMP Foundations and Concepts SNMP applications developed for any SNMP implementation tend to be easily adaptable and useful with other SNMP implementations Conceptually every
37. of dev sdf bs 1024k NOTE Zeroing the entire disk takes quite a long time but removes everything from the disk thus eliminating many problems To partition the spare disk to match the running disk 1 Save the partition table of the running disk to a file with a command like the following sfdisk d dev sda gt sda_partition_table 2 Write the saved partition table to the spare disk with a command like the following sfdisk dev sdd lt sda_partition_table Setting Up the ftServer System 5 21 Administering RAID Arrays Occasionally sfdisk returns the following error while writing the saved partition table to the spare disk Checking that no one is using this disk right now BLKRRPART Input output error This error indicates that the disk is currently in use so you should not repartition it Perform these steps to correct this error Unmount all file systems Swap off all swap partitions on this disk Use the no rereaa flag to suppress this check ao 9 Use the force flag to overrule all checks NOTE The preceding error does not occur if the spare disk already contained a valid partition table If you are sure that the spare disk is not in use force sfdisk to write the partition table by using the no reread flag To add partitions on the spare disk to RAID 1 arrays 1 Type the following command to determine which RAID 1 arrays the running disk belongs to mdadm detail s
38. operating system from NECAM the Linux operating system and Express Builder are preinstalled You do not need to reinstall or upgrade this software You should familiarize yourself with this chapter then log in To log in to the system 1 Login as root 2 Enter the root password A password must have at least six characters After you read and accept the necessary end user license agreements continue to Chapter 5 to begin configuring the ftServer system If you have a current installation that requires only minor upgrading review your Release Notes Express5800 ftServer for the Linux Operating System then see Chapter4 and if necessary Chapter 3 If some time has elapsed since your Express5800 ftServer system was delivered there may be updated documentation and software that may be useful to you NOTE Firmware updates may require Express Builder updates Express Builder updates may require firmware updates When updating 2 2 Express5800 ftServer System Administrator s Guide for the Linux Operating System Installation Overview Express Builder use the versions of firmware and software that are supplied on the ExpressBuilder for Linux 1 of 2 CD From time to time NECAM may issue an update to Express Builder See Chapter 4 for information about updating from an Express Builder update disk AN CAUTION The procedure described in this chapter is for a full installation or reinstallation of a supported Linux ope
39. physically remove the device Another state is Unknown The system will try to bring up any device and resolve it to a known state initializing it and performing diagnostics and if diagnostics are passed Simple Network Management Using Net SNMP and ftISNMP 8 25 SNMP Network Management Station Considerations bringing the device into an Online state for fault tolerant operations A partnered device on an ftServer system typically reaches a Simplex state if its partner is missing or not functioning or a Duplex state The interpretation of Duplex depends on the individual device type as shown in Table 8 1 Table 8 1 Meaning of Duplex for ftServer System Components CPU element A partner CPU element is present and online and the two partners are synchronized and running in lockstep I O element A partner I O element is present online and able to become primary to assume active compatibility 1 0 Device A partner I O device for example an Ethernet adapter is present online and available for failover Disk Drive A partner disk drive is online and the partitions of the two partners are mirrored and synchronized You can use ftISNMP to track and log these states and to control some operations See Table 8 2 for a complete list of operational states SNMP Network Management Station Considerations The ftISNMP package provides SNMP subagents Net SNMP and ftISNMP do not provide an SNMP capable network management station
40. reported on a mirror the mirror is marked faulty and it is no longer used The last active mirror is never marked faulty even if errors are reported against it The SCSI subsystem which comprises the SCSI midlayer and the low level drivers returns errors to RAID software when its error management code determines that the I O request cannot succeed When this happens RAID software marks the mirror faulty and retries reads on another mirror but otherwise ignores write errors Every time an active disk is pulled all outstanding I O is returned as errors When a disk is pulled all RAID members or mirrors that have active I O on the missing disk are marked faulty This may not be all RAID arrays that use the disk When a mirror becomes faulty the disk and CPU I O enclosure of the remaining active mirror becomes simplex and that enclosure is no longer safe to pull If you want to remove a good disk from a RAID array you must mark it faulty You can use the mdadm command to simulate an error and mark the mirror faulty mdadm dev md20 f dev sdb1 The proc mdstat file shows an F after a faulty mirror in the display md20 active raid F sdc1 1 sdf1 0 F 4095872 blocks 0 active chunks 2 1 _U Removing a Faulty Mirror Before removing a mirror check the proc mdstat file to make sure that it is marked faulty You can use the mdadm command to remove a faulty mirror from a RAID array as shown in the following example mdadm de
41. source O Begin the installation process with both CPU I O enclosures inserted and with the ftServer system halted A red LED is lit at the left side of each CPU I O enclosure Ensure that the green power button at the right side of the upper CPU I O enclosure is lit the lit LED indicates that it is the active primary enclosure If the upper CPU I O enclosure is not the active enclosure you must make it the active enclosure by performing the actions listed in step 2 of the installation procedure in Initial Linux Operating System and Express Builder Installation or Default Reinstallation on page 2 10 QO For security during installation and initial configuration isolate the ftServer system from networks and other communicating hosts O Be aware that a system with a newly installed operating system and Express Builder is not secure until it has been configured In addition you may want to change the default network settings Initial Linux Operating System and Express Builder Installation or Default Reinstallation The User Guide Setup contains detail procedures to install the Linux operationg system and the ftControl Software Installing the Operating System and Express5800 ftServer System Software 2 9 Initial Linux Operating System and Express Builder Installation or Default Reinstallation Avoiding CD ROM Mount Command Failure After Disabling SELinux During the installation of Express Builder you choose whether to ena
42. sraTraceLevel set to brief data flows to and from ftISNMP external items are traced With sraTraceLevel set to verbose internal items are also traced You can also change the location of the log file Agent and subagent startup and shutdown events are separately logged in syslog With trace levels other than off logs may grow rapidly depending on the number of managed objects and their activity In order to limit the size of the logs you can use logrotate to manage log size and archiving See ogrotate 8 Configuring SNMP to Start at System Initialization After installation of the Isb ft snmp package the t snmp initialization script is installed in the etc init d directory and the ft snmp service is added During system initialization the ft snmp service is automatically started at run level 3 4 or 5 This initialization script provides start stop restart and status functionality Configuring SNMP for Service Management There are several ways of adding SNMPv3 user accounts which allow local and remote access to SNMP services SNMP provides an etc opt ft snmp snmpd conf file that contains instructions and sample entries Uncomment the right lines then run the commands Uncomment these lines in etc opt ft snmp snmpd conf createUser admin MD5 your passwd DES rwuser v3user group v3usergroup usm admin group v3usergroup usm v3user view v3view included 136 21 access v3usergroup usm authNoPriv exact v3view
43. the device and the two disks In this example the device is changed to md30 and the two devices take the RAID 1 array names The result should look like the following ARRAY dev md30 level 0 DEVICES dev md20 dev md21 c Save the file and exit the editor 3 Make the RAID array with the following command mdadm create dev md30 level 0 raid devices 2 dev md20 dev md21 This command creates the RAID array and starts it If you type the mdadm command again it shows the active RAID array mdadm Q detail dev md30 dev md30 Version 00 90 01 Creation Time Wed Sep 28 15 20 08 2005 Raid Level raido Array Size 286743808 273 46 GiB 293 63 GB Raid Devices 2 Total Devices 2 Preferred Minor 30 Persistence Superblock is persistent Update Time Wed Sep 28 15 20 58 2005 State clean Active Devices 2 Working Devices 2 Failed Devices 0 Spare Devices 0 Chunk Size 64K Number Major Minor RaidDevice State o 9 20 0 active sync dev md20 1 9 21 1 active sync dev md21 UUID 5762850 7dd0c84 5c720b6b b328ca20 Events 0 1 5 12 Express5800 ftServer System Administrator s Guide for the Linux Operating System Setting Up RAID Arrays To stop the device use the mdadm command with the S argument as follows mdadm S dev md30 Creating and Mounting a File System The RAID arrays created in the preceding examples are raw disk block devices You can mount a file system on the RAID array
44. which physical device is available first based on which CPU I O enclosure is used during the boot Other System Configuration Information 5 30 In addition to setting up storage and network devices you may want to perform the following tasks to set up your system e Configure ftISNMP to allow remote management of your ftServer system See Chapter 8 for instructions e Disable hyperthreading Information about video display settings is also included Express5800 ftServer System Administrator s Guide for the Linux Operating System Other System Configuration Information You also need to perform the following configuration tasks using standard Linux procedures Configuring the IP address for the bondO and bond1 interfaces static or DHCP and gateway in etc sysconfig network scripts ifcfg bondo and etc sysconfig network scripts ifcfg bond1 Configuring DNS resolution for the system etc nsswitch conf and etc resolv conf Configuring static routes for the system etc sysconfig static routes Configuring the system hostname etc hosts and etc sysconfig network Configuring the system time zone etc sysconfig clock Disabling Hyperthreading Some system installations may disable hyperthreading to facilitate application execution A CAUTION Disable hyperthreading only if your system satisfies the minimum BIOS version requirements Contact the NEC Corporation of America Customer Support Center or your
45. 0 05 00 0 LO 1 05 01 0 10 2 05 02 0 05 02 1 05 02 2 10 3 05 03 0 10 4 10 5 04 02 0 eth000010 04 02 1 eth000011 10 6 10 7 10 8 03 01 0 10 9 04 01 0 eth000008 04 01 1 eth000009 10 10 10 11 10 40 10 40 1 10 40 2 LO LO LO LO LO LO LO LO LO LO LO LO LO LO 10 120 140 150 151 152 153 154 155 156 157 158 159 160 161 162 000 Internal Temp 12V Ambient Air Temp Fanl Fan2 1 2V VIT 1 8V VDD 12V Combined CPU IO Mass Storage Ctlr Fast Track TX EIDE Ctlr Mass Storage Ctlr PCI PCI X SATA Ctlr Serial Bus Ctlrs USB 1 0 Host Ctlr USB 1 0 Host Ctlr USB 2 0 Host Ctlr Display Ctlr ATI Rage Mobility Network Ctlr 2 port 1GB Enet NIC Network Interface 2 port 1GB Enet NIC Network Interface Bridge Bridge Misc ftSwitch Core Logic Network Ctlr 2 port 1GB Copper Enet NIC Network Interface 2 port 1GB Copper Enet NIC Network Interface SATA Enclosure 160GB SATA Disk Drive 74GB SATA Disk Drive Baseboard Management Ctlr Fan 12V ONLINE NORMAL NORMAL NORMAL NORMAL NORMAL NORMAL NORMAL NORMAL DUPLEX ONLINE ONLINE DUPLEX DUPLEX ONL ONL ONL ONL NE NE NE NE DUPLEX DUPLEX EMPTY DUPLEX DUPLEX DUPLEX DUPLEX DUPLEX 1 1 2 2 2 3V 5V 5V 5V 5V GB GB SATA VGA L1 L1 0 7c 00 0 3V CLK 3 3V 3 3Vs 3 3V GBE 5V 5Vs 12V Combined CPU IO Mass Storage Ctlr Fast Track TX EI
46. 1 Run an snmpwalk on ftcEtherInstanceName OID This gives you a list of EtherInstance names mapped to Ethernet device names snmpwalk v 1 c private t 40 localhost ftcEtherInstanceName STRING lo STRING sito STRING eth080010 STRING eth080011 STRING eth000010 STRING eth000011 SRA ftLinux MIB ftcEtherInstanceName SRA ftLinux MIB ftcEtherInstanceName SRA ftLinux MIB ftcEtherInstanceName SRA ftLinux MIB ftcEtherInstanceName SRA ftLinux MIB ftcEtherInstanceName SRA ftLinux MIB ftcEtherInstanceName Nu amp WN H I 2 Run an snmpwalk on ftcEtherDevPathID OID This gives you a list of EtherDevPath names mapped to EtherInstance names snmpwalk v 1 c private t 40 localhost ftcEtherDevPathID SRA ftLinux MIB ftcEtherDevPathID 1 STRING unknown SRA ftLinux MIB ftcEtherDevPathID 2 STRING unknown SRA ftLinux MIB ftcEtherDevPathID 3 STRING 10 5 SRA ftLinux MIB ftcEtherDevPathID 4 STRING 10 5 SRA ftLinux MIB ftcEtherDevPathID 5 STRING 11 5 SRA ftLinux MIB ftcEtherDevPathID 6 STRING 11 5 Simple Network Management Using Net SNMP and ftISNMP 8 23 SNMP and MIBS The instances of interest are the following Instance Name Instance 1 0 element Slot Device ftcEtherInstanceName 3 10 5 eth080010 ftcEtherInstanceName 4 10 5 eth080011 ftcEtherInstanceName 5 11 5 eth000010 ftcEtherInstanceName 6 11 5 eth000011 These are the
47. 1 0 ftSwitch Core Logic ONLINE ONLINE 11 9 Network Ctlr ONLINE ONLINE 0 AA U57500 7b 01 0 2 port 1GB Copper Enet NIC ONLINE ONLINE eth080008 Network Interface BROKEN BROKEN AA U57500 7b 01 1 2 port 1GB Copper Enet NIC ONLINE ONLINE eth080009 Network Interface BROKEN BROKEN 11 10 MISSING EMPTY 11 11 MISSING EMPTY 11 40 SATA Enclosure AA D64200 11 40 1 160GB SATA Disk Drive ONLINE DUPLEX 3 00 AA D64300 11 40 2 74GB SATA Disk Drive ONLINE ONLINE 33 0 11 120 Baseboard Management Ctlr ONLINE DUPLEX 4 0 0 11 140 Fan NORMAL 11 150 12V NORMAL 11 151 1 3V NORMAL 11 152 1 5V GB NORMAL 11 153 2 5V GB NORMAL 11 154 2 5V SATA NORMAL 11 155 2 5V VGA NORMAL 11 156 3V CLK NORMAL 11 157 3 3V NORMAL 11 158 3 3Vs NORMAL 11 159 3 3V GBE NORMAL 11 160 5V NORMAL 11 161 5Vs NORMAL 11 162 12V NORMAL IO Enclosure 11 is the Active Compatibility Node This is an ftServer 1 way DMR 3 2 GHz system P Package P3403R 1D Use lsLong to see this value Bringing System Components Down and Up You can use the ftsmaint command to bring down and restart a fault tolerant subsystem After bringing up a system it attempts to synchronize and duplex the corresponding components automatically 7 12 Express5800 ftServer System Administrator s Guide for the Linux Operat
48. 10 40 1 dev sda Is this the correct source disk device yes no y Source disk is partitioned partitioning blank disk to match Source disk partition 1 belongs to RAID 1 dev mdo Adding blank disk partition 1 to RAID 1 dev mdo mdadm hot added dev sdd1 Source disk partition 2 belongs to RAID 1 dev md2 Adding blank disk partition 2 to RAID 1 dev md2 mdadm hot added dev sdd2 Source disk partition 3 belongs to RAID 1 dev mdl Adding blank disk partition 3 to RAID 1 dev md1 mdadm hot added dev sdd3 Waiting for resync to complete before grubbing dev sdd1 Grubbing dev sdd1 5 24 Express5800 ftServer System Administrator s Guide for the Linux Operating System System Backup and Disaster Recovery System Backup and Disaster Recovery Your ftServer system provides many safeguards against losing data due to hardware failures However it cannot cover all contingencies so it is still important to perform regular backups and enact a good disaster recovery program Ethernet Devices Network interface naming on ftServer systems running a supported Linux distribution together with Express Builder is different from that on other Linux systems On ftServer systems physical devices are given names corresponding to their hardware location After installing Express Builder the interfaces associated with the Ethernet adapters are operational Multiple interfaces can be bonded together in a channel bonding interface Th
49. 3 Reason Codes Names and Definitions 8 31 Table 8 4 Set Operations Currently Implemented in ftISNMP 8 32 Tables ix Examples Example 5 1 Checking the Current State of the Internal Storage Subsystem 5 9 Example 5 2 Checking the Current State of RAID 5 15 Example 5 3 Resynchronization 5 18 Example 5 4 Running GRUB 5 24 Example 5 5 Pairing a Spare Internal Disk with the Running System Disk 5 25 Example 5 6 Default Configuration of Embedded Ethernet Devices 5 28 Example 7 1 Displaying System Status with the ftsmaint Command 7 10 Example 8 1 Traps that Can Occur for I O Element 11 When Trap Filtering Is Off 8 34 Example 8 2 Traps That Can Occur for I O Element 11 when Trap Filtering Is On 8 38 Example 8 3 Traps That Can Occur for When Trap Filtering Is Off 8 40 Example 8 4 Traps That Can Occur When Trap Filtering Is On 8 41 Example 9 1 Possible Keyboard and Mouse Error Messages at Boot Time 9 6 x Express5800 ftServer System Administrator s Guide for the Linux Operating System Preface The Express5800 ftServer System Administrator s Guide for the Linux Operating System documents tasks and information for system administrators of NECAM systems running a supported Linux distribution and ftControl system software sfor the Linux Operating System Express Builder This document is intended for system and network administrators using or migrating to Express5800 ftServer systems running a supported Linux distribut
50. 7 13 systems firmware 2 2 2 7 T testing Ethernet ports 8 23 SNMP configuration 8 19 ticks clock losing message 5 33 time zone configuring 5 32 traps filtering activating 8 33 deactivating 8 33 description 8 33 examples 8 34 verifying 8 18 troubleshooting 1 3 ftServer systems 1 3 9 1 U udev command 6 1 udevinfo command 6 3 umount command 6 3 uname command 2 4 up2date agent 4 4 updating BMC firmware 3 5 system BIOS 3 1 updating the Linux operating system 4 1 upgrading Express Builder 1 2 4 6 procedure 4 6 Linux operating system 1 2 4 1 pre upgrade checklist 4 3 recovery from an upgrade 4 8 upgrading the operating system 4 2 USB devices during system installation 2 5 Index 5 Index floppy drives 6 4 restoring after enclosure failure 6 3 solid state 6 4 storage 6 1 UTC See Coordinated Universal Time V version operating system identification 2 4 visual diagnostics 9 1 VND Virtual Network Devices 5 28 group configuration 5 28 default 5 28 Index 6 Express5800 ftServer System Administrator s Guide for the Linux Operating System
51. Administrator s Guide for the Linux Operating System Setting Up RAID Arrays Device files are created for the first 128 RAID arrays Use the mknod command see mknod 1 to create additional device files as needed The number is the minor device number and it is also used in the name The smaller numbers are used by the installer so it is convenient to add new RAID arrays above 10 When RAID arrays are intended to be moved between systems try to pick numbers that are unique among all of the systems The etc mdadm conf file which describes all of the RAID arrays for the system is created during installation This file contains one line for each RAID array When creating a new RAID array it is convenient to copy an existing line and modify it to suit your needs Here is a typical line ARRAY dev md2 super minor 2 You can create start and stop the RAID arrays You can manage RAID 1 arrays by adding and removing mirrors The following sections discuss working with RAID arrays This includes configuring RAID arrays administration and tools NOTE To ensure fault tolerant operation of your system only use the mdadm command to create RAID arrays Creating a RAID 1 Array The example in this discussion uses the disk pair sdb and sde The RAID array is called dev md20 and consists of sdb1 and sde1 To create a RAID 1 array 1 Select a pair of same sized disks and insert them into two corresponding slots in different CPU
52. Builder It discusses the following topics e General Upgrade Considerations e Upgrading or Restoring the Linux Operating System e Upgrading or Restoring Express Builder e Creating a Backup System Disk e Recovering from a Failed Software Upgrade e Related Information and Resources NOTES 1 If you want to update or reinstall an individual software package that is provided in a Red Hat Package Manager RPM file use the rpm command see rpm 8 2 The Linux operating system upgrade script has been tested only with firmware hardware and devices meeting design specifications of the Express5800 ftServer system and its system options See the Release Notes Express5800 ftServer for the Linux Operating System for issues that may relate to the upgrade process 3 If you need to perform a complete reinstallation rather than a release level upgrade or restoration see Chapter 2 for preparation and for the procedure Updating the Operating System and Express5800 ftServer System Software 4 1 General Upgrade Considerations General Upgrade Considerations When upgrading the Linux operating system or the Express Builder be aware of the following requirements and related considerations Upgrade Requirements First ensure that the system s BIOS and BMC firmware levels support the new Express Builder version You can obtain required versions of firmware from the ExpressBuilder for Linux CD
53. DE Ctlr Using ftServer Fault Tolerant Utilities and Software ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE MISSING ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE BROKEN ONLINE BROKEN MISSING MISSING ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONL ONL ONL ONL ONL ONL B O BROKEN E E NE NE NE NE NE NE ROKEN NE ry FY DUPLEX ONL NE DUPLEX NORMA NORMA NORMA NORMA NORMA NORMAI NORMAI NORMA NORMA NORMA NORMA NORMA NORMA Li Li Li Li NORMA DUPLEX ONLINE ONLINE Li 3 00 33 0 4 0 0 oo 1 1 oo f The ftsmaint Command 11 1 Mass Storage Ctlr ONLINE DUPLEX 0 7CO3 0L 0 PCI PCI X SATA Ctlr ONLINE DUPLEX 11 2 Serial Bus Ctlrs ONLINE ONLINE 0 7c 02 0 USB 1 0 Host Ctlr ONLINE ONLINE 7c 02 1 USB 1 0 Host Ctlr ONLINE ONLINE 7c 02 2 USB 2 0 Host Ctlr ONLINE ONLINE 11 3 Display Ctlr ONLINE DUPLEX 0 7c 03 0 ATI Rage Mobility ONLINE DUPLEX 11 4 MISSING EMPTY 11 5 Network Ctlr ONLINE DUPLEX 0 7b 02 0 2 port 1GB Enet NIC ONLINE DUPLEX eth080010 Network Interface ONLINE DUPLEX 7b 02 1 2 port 1GB Enet NIC ONLINE DUPLEX eth080011 Network Interface ONLINE DUPLEX 11 6 Bridge ONLINE ONLINE 0 11 7 Bridge ONLINE ONLINE 0 11 8 Misc ONLINE ONLINE 0 7a 0
54. Express Builder does not support them as fault tolerant devices To be supported as a fault tolerant device a device must have a special hardened driver that supports surprise removal and fault management by Express Builder Some system components such as data storage may require additional configuration The following sections discuss them as well as some of the automated features of the ftServer system that support fault tolerant operation and ease system administration Setting Up the ftServer System 5 1 Setting Up Internal Disk Storage Setting Up Internal Disk Storage This section discusses the following topics e Internal Disk Storage Overview e The Console Log and the var log messages File e Configuring Internal Disks e Managing Partitions e Default Internal Disk Configuration for a Newly Installed System e Checking the Current State of the Internal Disk Subsystem e Storage Device Definition Internal Disk Storage Overview ftServer systems support up to six internal Serial Advanced Technology Attachment SAS SATA disks three in each CPU I O enclosure You can use RAID 1 to mirror the disks in one enclosure with the corresponding disks in the other enclosure for fault tolerance You should not mirror disks in the same internal storage enclosure RAID 1 directs I O flow to the appropriate disks in the two CPU I O enclosures When a CPU I O enclosure or a disk in it is
55. Express5800 ftServer System Administrator s Guide for the Linux Operating System NEC Corporation of America 456 01 730 000 Notice The information contained in this document is subject to change without notice UNLESS EXPRESSLY SET FORTH IN A WRITTEN AGREEMENT SIGNED BY AN AUTHORIZED REPRESENTATIVE OF NEC NEC MAKES NO WARRANTY OR REPRESENTATION OF ANY KIND WITH RESPECT TO THE INFORMATION CONTAINED HEREIN INCLUDING WARRANTY OF MERCHANTABILITY AND FITNESS FOR A PURPOSE NEC assumes no responsibility or obligation of any kind for any errors contained herein or in connection with the furnishing performance or use of this document Software described in NEC a is the property of NEC and or its licensees b is furnished only under license and c may be copied or used only as expressly permitted under the terms of the license NEC documentation describes all supported features of the user interfaces and the application programming interfaces API developed by NEC and or its licensees Any undocumented features of these interfaces are intended solely for use by NEC personnel and are subject to change without warning This document is protected by copyright All rights are reserved No part of this document may be copied reproduced or translated either mechanically or electronically without the prior written consent of NEC Corporation of America Portions of this document are copyrights of Stratus Technologies and are reprinted her
56. GenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING OFFLINE SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206134944 633611 300 RFC1213 MIB sysUpTime 0 Timeticks 12435 0 02 04 35 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 0 Simple Network Management Using Net SNMP and ftISNMP 8 35 Trap Filtering SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING OFFLINE SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PARENT EMPTY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING RFC1213 MIB sysUpTime 0 Timeticks 12535 0 02 05 35 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 2 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING OFFLINE SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PARENT EMPTY SRA ftLinux MIB ftcTrapGenAction 0
57. I O enclosures This example uses the middle slot sdb of the upper CPU I O enclosure and the middle slot sde of the lower enclosure 2 When the disks have spun up partition them for the desired RAID array see Managing Partitions on page 5 4 You can mark the partitions with code 83 Setting Up the ftServer System 5 9 Setting Up RAID Arrays 3 Editthe etc mdadm conf file so that the new RAID array will start each time the system boots a Use an existing ARRAY line as a model Copy it to the bottom of the file b Edit the device and the two disks In this example the device is changed to md20 and the two disks are changed to dev sdb1 and dev sde1 The result should look like the following ARRAY dev md20 level 1 num devices 2 devices dev sdb1 dev sdel Always define RAID 1 arrays on disks in the embedded slots to have two disks and no spares c Save the file and exit the editor 4 Make the RAID array with the following command mdadm create dev md20 level 1 raid devices 2 dev sdb1 dev sdel This command creates the RAID array and starts it NOTE When an array is created proc mdstat shows that disks are in the process of resynchronizing but the LEDs on those disks do not light Although proc mdstat reports a resynchronization in progress none is occurring and no disk I O is involved In this case ignore the resynchronization information in proc mdstat 5 10 Express5800 ftServe
58. IB definitions Some Objects Defined by Standard MIBs For a practical implementation of SNMP a number of objects simply must be defined Some of these are introduced here 8 12 Express5800 ftServer System Administrator s Guide for the Linux Operating System SNMP Foundations and Concepts For UDP or TCP IP communications and collection of statistical data about communications and communications channels MIB II defines some necessary objects MIB II defines these objects for querying system interfaces at ip icmp tcp udp snmp The Net SNMP implementation requires basic support of the Host Resources MIB The objects defined in RFC1514 Host Resources MIB include hrSystem hrStorage hrDevice hrSWRun hrSWRunPerf hrSWinstalled hrSWRunID NOTE A Host Resources MIB should support these objects which are defined in RFC1514 A Newer Host Resources MIB may comply with RFC2790 which extends and replaces RFC1514 The Net SNMP Host Resources MIB implementation has been tested for the RFC1514 features The Net SNMP package also implements the Net SNMP version of the UCD Extensions MIB which defines these objects prTable memory laTable systemStats fileTable Simple Network Management Using Net SNMP and fISNMP 8 13 SNMP Foundations and Concepts SNMPv3 Support SNMPv3 support includes implementation of IETF RFCs 3410 through 3418 The third version of the Simple Network Management Protocol presented b
59. ID and the CPU I O enclosure are simplex for the active mirror Replacing a Failed Disk When you need to replace a failed disk the OSM plugin can automatically add the replacement disk to a running RAID array provided the following conditions exist e The replacement disk must be blank as defined by the current safe mode setting If safe mode is active zero the disk s partition table and RAID superblocks Then remove and reinsert the disk to start the automatic disk replacement For more information about safe mode see Configuring Safe Mode Do not reboot the system or stop and restart OSM after you remove the failed disk until you have inserted the replacement disk and it has synchronized with its partner The information necessary to perform automatic disk replacement is not persistent so if OSM is restarted the replacement disk must be paired using a different method e The failed disk must have been paired with one and only one partner disk For example if dev md4 consisted of partitions sda1 and sdb1 and dev md5 consisted of sdb2 and sdc2 automatic disk replacement would not work for disk sdb In addition partition numbers on the failed disk and its partner for any partitions belonging to RAID1 arrays must be the same e The failed disk must belong to a RAID 1 on top of a disk partition or multipath If the failed disk belongs to a RAIDO even if that RAIDO is part of a RAID1 the blank disk will not be added t
60. ISNMP MIB Objects 8 32 SRA ftLinux MIB OID Values and Properties 8 32 Trap Filtering 8 33 Trap Filtering Capability 8 33 Activating and Deactivating Trap Filtering 8 33 Trap Filtering Examples 8 34 9 Troubleshooting ftServer Systems 9 1 LED and Visual Diagnostics 9 1 System Boot Problems 9 1 Normal Boot Sequence 9 2 Possible Boot Problems 9 3 Missing Drivers Prevent Booting 9 3 GRUB Problem 9 4 RAID Problem 9 5 Automatic Reboot After Boot Monitoring Timeout 9 5 System Log Messages 9 6 Error and Log Messages Regarding Keyboard and Mouse 9 6 Index Index 1 Contents vii Figures Figure 2 1 SAS SATA Drive Arrangement for Installation 2 9 Figure 5 1 CPU I O Enclosures Front Panel with Drive Slots Fully Populated 5 3 Figure 7 1 ftServer Enclosures Locations of Major Enumerated Devices Front View 7 8 Figure 7 2 ftServer Enclosures Locations of Major Enumerated Devices Rear View 7 9 Figure 8 1 AgentX Enabled Extensions and Subagents 8 17 Figure 8 2 Operational State Management on ftServer Systems 8 25 viii Express5800 ftServer System Administrator s Guide for the Linux Operating System Tables Table 2 1 CD ROMs Included With ftServer Systems 2 2 Table 5 2 Ethernet Devices in ftServer CPU I O Enclosures 5 27 Table 7 1 Device Paths of ftServer Devices 7 5 Table 8 1 Meaning of Duplex for ftServer System Components 8 26 Table 8 2 Operation State Values Names and Definitions 8 30 Table 8
61. MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206141314 504766 300 RFC1213 MIB sysUpTime 0 Timeticks 5546 0 00 55 46 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB SRA ftLinux MIB SRA ftLinux MIB SRA ftLinux MIB SRA ftLinux MIB Timeticks OID 4223 0 00 42 23 STRING ftcTrapGenName 0 STRING ftcTrapGenDetailInfo 0 ftcTrapDevicePathID 0 ftcTrapAlertType 0 ftcTrapGenericInformationTrap 0 STRING wy OPSTATE CHANGE OFFLINE STRING NONE 8 38 Express5800 ftServer System Administrator s Guide for the Linux Operating System Trap Filtering SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206141315 508290 300 RFC1213 MIB sysUpTime 0 Timeticks 5746 0 00 57 46 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 40 1 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206141317 514636 300 RFC1213 MIB sysUpTime 0 T
62. Managing RAID on Linux Derek Vadala O Reilly amp Associates 2003 http www oreilly com catalog mraidlinux Setting Up the ftServer System 5 33 Additional Documentation and Resources 5 34 Express5800 ftServer System Administrator s Guide for the Linux Operating System Chapter 6 Managing Data Storage Devices This chapter discusses the following topics e CD ROM Drives e USB Storage Devices e Additional Resources Chapter 5 explains basic storage device definition and the configuration and management of the internal disk drives embedded in CPU I O enclosures This chapter briefly discusses other data storage devices that are included with or can be optionally attached to the system CD ROM Drives An ftServer system supports two CD ROM or DVD RW drives which may appear as the following names on the system depending on whether a CD ROM or DVD writer is in the drive e In CPU O I O 10 dev cdrom dev dvdwriter dev hde e In CPU 1 I O 11 dev cdrom1 dev dvdwriter1 dev hdi USB Storage Devices USB storage devices including floppy disk and solid state storage are supported through the SCSI driver These devices appear as SCSI devices You can get information about these devices by examining the file proc scsi scsi by running the command lsusb or by examining the system log var log messages When you connect a USB device to the USB bus the SCSI driver scans it once and assigns
63. Reinstall GRUB in the master boot record MBR on the problem disk To manually run GRUB 1 Boot the system from your Linux operating system bootable CD by performing the procedures in System Log Messages on page 9 11 Follow the prompts instructing you to run the chroot command Run GRUB as follows sbin grub grub gt grub gt grub gt grub gt grub gt grub gt grub gt device hd0 dev sda root hd0 0 setup hd0 device hd0 dev sdb root hd0 0 setup hd0 quit Both system disks are now bootable 4 Shutdown the system and eject the CD Incorrect GRUB parameters can also cause problems in booting Do not change the parameters from the defaults set when the operating system was installed 9 4 Express5800 ftServer System Administrator s Guide for the Linux Operating System System Boot Problems AN CAUTION In particular specifying the GRUB noapic option can make the operating system unbootable RAID Problem If a RAID 1 array has one type Oxfd Linux RAID autodetect mirror and one 0x83 Linux mirror at boot the RAID array is started in degraded mode using the type Oxfd mirror and the type 0x83 mirror is not automatically added You can add the mirror with mdadm To fix this problem just change the partition type with fdisk The system supports RAID 1 arrays that consist of type 0x83 mirrors You can create partitions of type 0x83 create RAID 1 arrays with them and then create
64. SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 120 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING OFFLINE SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206135204 763945 300 Example 8 2 shows some traps that can occur when CPU I O enclosure 11 is brought down and trap filtering is on Example 8 2 Traps That Can Occur for I O Element 11 when Trap Filtering Is On RFC1213 MIB sysUpTime 0 SNMPv2 MIB snmpTrapOID 0 SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 0 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206141302 277883 300 RFC1213 MIB sysUpTime 0 Timeticks 5445 0 00 54 45 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PRIMARY SRA ftLinux
65. System Disk on page 4 7 you must create its partitions use the mdadm command to add it to a RAID 1 array and run the grub command NOTE Do not perform these procedures if you are replacing a failed disk See instead Replacing a Failed Disk on page 5 19 e Replacing Defective Disks Interactively e Replacing Defective Disks Manually Setting Up the ftServer System 5 19 Administering RAID Arrays Replacing Defective Disks Interactively To replace a defective disk perform the following procedures Remove the defective disk and insert a spare disk Run the duplex_blank_disk command see The duplex blank disk Command on page 5 25 NOTE Replacement disks can be new factory fresh disks or disks recycled from other systems Care must be taken with recycled disks The partition table and RAID superblocks that exist on the disk can confuse the system Replacing Defective Disks Manually To replace a defective disk by manually issuing commands for each step of the process perform the following procedures Remove the defective disk and insert a spare disk Verify that the spare disk is not in use Zero the spare disk Partition the spare disk to match the running disk Add partitions on the spare disk to RAID 1 arrays Run the GRUB boot loader only if the running disk is the system disk NOTE Replacement disks can be new factory fresh disks or disks recycled from other systems
66. TES 1 The RAID arrays that are started are started as described above regardless of what kernel was booted or which disk contained the kernel 2 The recommended configuration of sda and sdd system disks results in the expected boot sequencing The kernel is found in sda or sdd and the system file systems are also on sda and sdd e Later in the boot sequence etc rc sysinit runs It finds and starts any RAID arrays in etc mdadm conf that were not already started and that are required by mounts in etc fstab 9 2 Express5800 ftServer System Administrator s Guide for the Linux Operating System System Boot Problems NOTE If a RAID array fails to start the boot stops and enters a debug shell This is almost always because of a configuration error in etc fstab or etc mdadm conf Exiting the shell forces a reboot Depending on your system s RAID configuration you may see one or more error messages similar to the following md could not bd claim sdarl md error md import device returned 16 These messages indicate that md is refusing to start an array that has already been started You can safely ignore them The recommended configuration has all RAID 1 arrays marked as type Oxfd Linux RAID autodetect so they start early and all RAID 0 arrays in etc mdadm conf so they start later e The operating system checks the file systems NOTE In the case of crash recovery the file check fsck may t
67. a remote management system running snmptrapd by adding lines as follows trapcommunity public trapsink ip_address public trap2sink ip_address public In this example ip_address is the IP address of the remote management system 3 Start or restart SNMP using the etc opt ft snmp scripts start snmp or etc opt ft snmp scripts restart snmp command for changes to take effect 4 From the remote management system you can view the traps as they are generated by tailing var log messages or wherever the remote management system is configured to log snmptrapd messages tail f var log messages Pulling CPU I O enclosures and or pulling Ethernet cables on the ftServer system will generate traps as will exercising the system using the ftsmaint command Managing SNMP The following sections discuss how to manage SNMP on your system e Testing Your SNMP Configuration e Managing ftServer Hardware Components e Testing Ethernet Ports NOTE The sample command lines in the remainder of this chapter are for general guidance only Some of the command line details and command output shown for 8 18 Express5800 ftServer System Administrator s Guide for the Linux Operating System Managing SNMP instance PCI adapter device names may differ from what is applicable to your system Testing Your SNMP Configuration The following are some Net SNMP commands that you can use to test or exercise MIBs If yo
68. ake a long time and it may fail If it fails the boot stops and enters a debug shell The administrator must manually repair the problem file systems Exiting the shell forces a reboot Possible Boot Problems A problem in booting the system may be associated with missing or corrupt fault tolerant drivers the GRUB boot loader or RAID Missing Drivers Prevent Booting If required fault tolerant drivers are not present at boot time and if the system s fault tolerant policy is set to prevent booting when drivers are missing the default setting the following prompt appears at the console This system is not fault tolerant because reason Type NON FT BOOT to allow login for repair Troubleshooting ftServer Systems 9 3 System Boot Problems In the output above reason is one of the following e ERROR building Stratus kernel objects see logfile e ERROR missing Stratus kernel objects see logfile e ERROR incorporating Stratus kernel objects see logfile In the output above logfile is the name of a file that contains relevant details To override the system s fault tolerant policy and allow the system to boot to a non fault tolerant state at the console type NON FT BOOT and press ENTER If you provide any other response three times the system starts the boot process again GRUB Problem If the system boots and hangs before the operating system is loaded it may be a problem with the GRUB boot loader
69. all PTR div of Pearson Education Inc http vig prenhall com catalog academic product 0 4096 0130462101 00 html This volume is a basic reference for common open source TCP IP network administration utilities and technologies including SNMP tools and methods UNIX Systems Advanced Administration and Management Handbook Bruce H and Karen B Hunter copyright 1996 Pearson Education Although this book is an older book that covers only the UNIX operating system the advice and wisdom packaged here for administrators of UNIX type operating systems and TCP IP networks is usually available only through intensive experience Introduction to ftServer System Administration 1 5 Additional Documentation and Resources 1 6 Express5800 ftServer System Administrator s Guide for the Linux Operating System Chapter 2 Installing the Operating System and Express5800 ftServer System Software This chapter discusses the following topics e Installation Overview e Separately Released and Optional Distribution Components e Installation Interfaces e Supported Hardware and Firmware e Pre Installation Checklist e Initial Linux Operating System and Express Builder Installation or Default Reinstallation e Post Installation Tasks and Considerations e Performing an Installation Without a Kickstart File e Additional Documentation and Resources The procedures described in this chapte
70. anual removal of the Net SNMP and ftISNMP packages Use the standard Linux operating system and Express Builder installation and upgrade procedures to manage Net SNMP and ftISNMP installations and upgrades This section documents the manual removal process First stop all server SNMP processes etc opt ft snmp scripts stop snmp This stops the SNMP subagents followed by the master agent Any errors or warnings resulting are written to the syslog or stderr To remove the installed binary RPM enter rpm e lsb ft snmp Simple Network Management Using Net SNMP and ftISNMP 8 29 Initial SNMP Testing OpState State Definitions Table 8 2 lists operation state OpState names SRA ftLinux MIB codes and definitions for ftServer systems running a supported Linux distribution together with Express Builder Table 8 2 Operation State Values Names and Definitions Page 1 of 2 1 UNKNOWN The state of a component could not be determined 2 EMPTY The component slot does not have a component present or the component does not have power 3 REMOVED A component is present in the slot but main power is not turned on and the component is out of service 4 SHOT A component has an error and was taken out of service by system logic When in this state the component is electrically isolated from the rest of the system 5 BROKEN A component has a problem an associated reason see OpState Reason describes the problem This is a termi
71. ar View 2 3 1 AS Flo 7 ae f 23 o j o H 000 Sante L J ro GA l cis Pas VaR m voon C Le cal J Ok re A i 88 000 Tes 3 co b E 9192 PCI H 122 A 6 5 4 asys077a Callout Device ID Component 1 10 9 Slot 1 low profile PCI Slot 9 2 10 10 Slot 2 full height PCI Slot 10 3 10 11 Slot 3 full height PCI Slot 11 4 11 9 Slot 1 low profile PCI Slot 9 5 11 10 Slot 2 full height PCI Slot 10 6 11 11 Slot 3 full height PCI Slot 11 Using ftServer Fault Tolerant Utilities and Software 7 9 The ftsmaint Command ftsmaint Examples The following sections provide examples of how to use the ftsmaint command e Displaying System Status e Bringing System Components Down and Up e Removing a PCI Adapter From Service and Bringing It Into Service Displaying System Status To display the status of the fault tolerant devices and subsystems in your ftServer system issue the following command ftsmaint ls Example 7 1 Displaying System Status with the tsmaint Command root lstlinux14 14
72. authorized NEC service representative to confirm that you have the correct BIOS version To disable hyperthreading you enter the ftServer Setup utility To disable hyperthreading 1 oa fF ND Turn on or restart your system When the Express5800 ftServer logo screen appears press F2 to enter your system s BIOS setup utility The BIOS setup utility s Main menu appears after the system completes more of the POST power on self test process On the Main menu use the right arrow key to select the Advanced tab Use the down arrow key to select Advanced Processor Options and press Enter Select Hyper Threading Technology and press Enter Change the value from Enabled to Disabled and press Enter Press Esc then select Exit Saving Changes and press Enter The system resumes booting Setting Up the ftServer System 5 31 Other System Configuration Information Configuring the System Video Display Your ftServer system s video comes configured by default There is normally no need to change the video displays settings and the system is strictly limited in some of its parameters For instance the screen resolution is limited to 1024x768 pixels However it is possible though not advisable to change the video configuration AN CAUTION Using other means to configure the video including any of those available from a Red Hat icon may result in loss of system fault tolerance or may cause the system to boot only in text
73. ble or disable SELinux If you choose to enable SELinux and then disable SELinux at a later time the command to mount the CD ROM device fails unless you edit the etc fstab file to remove a parameter that is added to the file when you disable SELinux To allow the command to mount a CD ROM to function properly edit the etc fstab file On the dev hda media cdrecorder line replace the following text fscontext system u object r removable t with the following fscontext system_u object_r removable t exec noauto managed 0 0 When you disable SELinux this parameter is added to the etc fstab file which causes the command to mount the CD ROM to fail Reinstalling Express Builder After a Failed Installation If the attempt to install Express Builder fails before again trying to install it perform the following steps to uninstall the requisite files and attempt to reinstall Express Builder in the correct sequence 1 Remove all NECAM packages except eula_display with this command rpm e nodeps allmatches rpm qa grep lsb ft grep v eula display 2 Remove eula display with this command rpm e nodeps lsb ft eula display 3 Reinstall Express Builder by mounting the CD and issuing the install sh command If the system does not then reboot automatically reboot the system manually Booting in Linux Rescue Mode If the OS installation fails whether or not Express Builder installed successfully attempt to boot the system
74. can ARRAY dev md2 level raidl num devices 2 UUID 5ddb14c7 d5e0b2d6 ad80086d 8db2a245 devices dev sda2 ARRAY dev md1 level raidl num devices 2 UUID 3838df6e 60caf7e6 695d0f62 de94e821 devices dev sda3 ARRAY dev md0 level raidl num devices 2 UUID 3e4ad330 c8ee5dfc f48bd88a 401ada25 devices dev sdal 5 22 Express5800 ftServer System Administrator s Guide for the Linux Operating System Administering RAID Arrays 2 Add each partition on the spare disk to the RAID 1 array containing the corresponding partition on the running disk with commands like the following mdadm a dev md0 dev sddl mdadm hot added dev sdd1 mdadm a dev md1l dev sdd3 mdadm hot added dev sdd3 mdadm a dev md2 dev sdd2 mdadm hot added dev sdd2 Perform the following procedure only if the running disk is the system disk To run GRUB If the running disk is the system disk run the GRUB boot loader on the boot partition of the spare disk after resynchronization is complete on that partition Example 5 4 shows a typical use of GRUB Example 5 4 Running GRUB sbin grub GNU GRUB version 0 95 640K lower 3072K upper memory Minimal BASH like line editing is supported For the first word TAB lists possible command completions Anywhere else TAB lists the possible completions of a device filename grub gt device hd0 dev sdd grub gt root hd0 0 Filesystem type is ext2fs partition type 0xfd grub gt setup
75. ce from the system and then insert it back in the system USB storage devices are not bootable devices Managing Data Storage Devices 6 3 USB Storage Devices Most floppy disks and solid state devices come with a virtual file allocation table VFAT file system You can create ext 2 or other file systems on the device as well You can mount them on a convenient mount point for example mkdir mnt floppy mount dev sdgl mnt floppy USB Floppy Drives The USB floppy drive appears as follows in proc scsi scsi Host scsi0 Channel 00 Id 00 Lun 00 Vendor ATA Model ST380013AS Rev 3 00 Type Direct Access ANSI SCSI revision 05 Host scsi4 Channel 00 Id 00 Lun 00 Vendor ATA Model ST380013AS Rev 3 00 Type Direct Access ANSI SCSI revision 05 The system log shows details about the device scsi4 SCSI emulation for USB Mass Storage devices Vendor NEC Model USB UF000x Rev 1 50 Type Direct Access ANSI SCSI revision 02 SCSI device sdaz 2880 512 byte hdwr sectors 1 MB sdaz Write Protect is on Write Protect is on indicates that the disk is read only USB Solid State Devices The following is an example of the proc scsi scsi display for a solid state device Host scsi5 Channel 00 Id 00 Lun 00 Vendor LEXAR Model JUMPDRIVE SECURE Rev 3000 Type Direct Access ANSI SCSI revision 02 6 4 Express5800 ftServer System Administrator s Guide for the Linux Operating System Additional Resources
76. codes For example when you bring down an I O element the display controller with the device path 10 0 or 11 0 will change state from DUPLEX to OFFLINE with the reason code of PARENT_EMPTYV If you are not interested in this type of trap use the trap filtering capability Activating and Deactivating Trap Filtering To activate trap filtering specify the following configuration line in the etc opt ft snmp ftltrapsubagent conf file sraTrapFiltering on When you activate trap filtering traps with the following operational states are filtered out e DUMPING e DIAGNOSTICS e DIAGNOSTICS PASSED e INITIALIZING e SYNCING e FIRMWARE_UPDATE e FIRMWARE_UPDATE_COMPLETE e UNKNOWN Simple Network Management Using Net SNMP and ftISNMP 8 33 Trap Filtering Traps with the following reason codes are also filtered out e PARENT_EMPTY e PARENT_BROKEN To deactivate the trap filtering capability change the above configuration line as follows sraTrapFiltering off By default trap filtering is turned off that is sraTrapFiltering is set to off in the configuration file Trap Filtering Examples Example 8 1 shows some traps that can occur when I O element 11 is brought down and trap filtering is off NOTE The following examples show sample data only Data from your system may be different Example 8 1 Traps that Can Occur for I O Element 11 When Trap Filtering Is Off RFC1213 MIB sysUpTime 0 Timeticks 11829 0
77. ction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206141320 697382 300 Simple Network Management Using Net SNMP and ftISNMP 8 39 Trap Filtering RFC1213 MIB sysUpTime 0 Timeticks 6845 0 01 08 45 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 120 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PRIMARY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206141328 502684 300 RFC1213 MIB sysUpTime 0 Timeticks 8545 0 01 25 45 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 120 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING OFFLINE SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206141345 505798 300 RFC1213 MIB sysUpTime 0 Timeticks 8846 0 01 28 46 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB f
78. d by path Using ftServer Fault Tolerant Utilities and Software 7 3 The ftsmaint Command NOTE Do not use this feature to retain a faulty or degraded device in service It may be useful if the MTBF for a device has been degraded by testing or configuration error e ftsmaint runDiag path This command starts diagnostics on the CPU element or I O element specified by path e ftsmaint setPriority level path This command sets the priority level of the CPU element specified by path to the value in the level argument e ftsmaint setMtbfThresh value path This command sets the MTBF threshold to value of the CPU element I O element or CPU I O enclosure slot specified by path e ftsmaint setMtbfType policy path This command sets the MTBF type to policy on the CPU element I O element or CPU I O enclosure slot The policy argument can take one of the following values e useThreshold e neverRestart e alwaysRestart e ftsmaint setSensorThresh th_name value path This command sets the threshold specified by th_name on the sensor device specified by path to value The th_name argument can take one of the following values e uf upper fatal e uc upper critical e unc upper noncritical e 1 lower fatal e 1c lower critical e Inc lower noncritical 7 4 Express5800 ftServer System Administrator s Guide for the Linux Operating System The ftsmaint Command The opstates for the sensors are as follows e FATAL abo
79. d prompt type the following lines to make both system disks bootable sbin grub device hd0 dev sda root hd0 0 setup hd0 device hd0 dev sdb root hd0 0 setup hd0 quit e Make sure that the system is running the SMP kernel e You must install all required software packages You may have to manually resolve package dependency failures when installing Express Builder e Make sure that the kernel command line does not contain rhgb e Make sure that services like kudzu haldaemon and microcode cil are disabled e Make sure that partitions are set up correctly Additional Documentation and Resources e The GNU GRUB Web page Free Software Foundation http www gnu org software grub grub en html e GRUB GRand Unified Bootloader version 0 5 original documentation Web site http www uruk org orig grub e GRUB file system syntax and semantics document by Eric Bolyn primary author of GRUB http www uruk org orig grub filesystem txt 2 18 Express5800 ftServer System Administrator s Guide for the Linux Operating System Chapter 3 Updating ftServer System Firmware This chapter discusses the following topics e Updating the System BIOS e Updating BMC Firmware Consult the Release Notes Express5800 ftServer for the Linux Operating System for the Express5800 ftServer System Software for the Linux Operating System Express Builder version you have or will upgrade to to determine what f
80. ded so that you can if necessary reinstall your ftServer system s software If you ordered a supported Linux distribution from a vendor other than NECAM the vendor can also provide you with distribution CDs for installing or reinstalling the system software You need to use the Express Builder CD with the CDs for the supported Linux distribution to achieve a fault tolerant ftServer system Before you install your ftServer system s software you must prepare your ftServer system by following the hardware installation instructions provided in the installation guide for your system See Chapter 2 and Chapter 4 for software installation and update procedures Updating ftServer System Firmware Specific firmware versions are required for a given release of Express Builder The firmware in an ftServer system delivered from the factory does not require updating However subsequent updates to Express Builder may require firmware updates When updating Express Builder use the versions of firmware and software that are supplied on the ExpressBuilder for Linux CD Chapter 3 provides procedures for updating your system BIOS and Baseboard Management Controller BMC firmware 1 2 Express5800 ftServer System Administrator s Guide for the Linux Operating System Additional Documentation and Resources Configuring Your ftServer System After installing the Linux operating system and Express Builder you must configure your system See Chap
81. e Express5800 ftServer system running the Linux operating system and the Net SNMP and ftISNMP packages To receive traps configure the etc opt ft snmp snmpd conf file on the ftServer system adding trapsink entries pointing to the host server running snmptrapd The configuration lines should look something like this trapcommunity public trapsink 192 468 3375 public trap2sink 192 168 33 75 public In this sample the community is public This would not be the usual case on an internet accessible system 8 16 Express5800 ftServer System Administrator s Guide for the Linux Operating System Installing Remote Network Management Services Configuring SNMP for Remote Service Management The procedure for configuring Net SNMP is very similar to Configuring SNMP for Service Management on page 8 134 which describes enabling remote services by adding SNMP users and groups If you are using a network management station you may have some other procedure provided with your software Deploying SNMP Agents and Subagents snmpd log The basic SNMP model has a master agent on the SNMP server system with behavior configured by MIBs The master agent manages one or more subagents Typically a single subagent is used per system including the SNMP server system However with AgentX extensions multiple agents can be deployed on a system performing different tasks under control of the master agent Even a MIB II subagent can be extended
82. e under permission from Stratus Technologies Bermuda Ltd The NEC Corporation of America logo Express5800 ftServer and the Express5800 ftServer logo are trademarks of NEC Corporation of America ActiveService Network is a trademark of Stratus Technologies Bermuda Ltd All other trademarks and trade names are the property of their respective owners The registered trademark Linux is used pursuant to a sublicense from the Linux Mark Institute the exclusive licensee of Linus Torvalds owner of the mark on a worldwide basis NEBS is a trademark of Telcordia Technologies Inc Red Hat Red Hat Linux and Red Hat Enterprise Linux are registered trademarks of Red Hat Inc in the US and other countries UNIX is a registered trademark of X Open Company Ltd in the U S A and other countries Manual Name Express5800 ftServer System Administrator s Guide for the Linux Operating System Part Number 456 01730 000 Revision Number 04 Publication Date January 2008 NEC Corporation of America 6535 N State Highway 161 Irving Texas 75038 2402 2006 NEC Corporation of America All rights reserved Contents Preface xi 1 Introduction to ftServer System Administration 1 1 ftServer System Terminology 1 2 System and Network Administration Overview 1 2 Installing and Updating So ftware on ftServer Systems 1 2 Updating ftServer System Firmware 1 2 Configuring Your ftServer System 1 3 Managing Data Storage Devices 1 3 Using t
83. e Network Management Services for extensible SNMP agents and then defines master agents and subagents as processing daemons An AgentX protocol is defined for communication between an AgentX capable master agent and subagents RFC2741 also defines elements of procedure for an AgentX daemon to process SNMP protocol messages Traditional CMU SNMP management utilities are modestly refined and enhanced in Net SNMP Most of the ftISNMP extensions to Net SNMP come through added MIB definitions and correspondingly augmented configuration files for managing SNMP agent and subagent daemons Installing Remote Network Management Services The remote host system must have an SNMP system installed that supports SNMPv3 an example is the Red Hat Net SNMP distribution which the following sample procedure assumes Transfer the SRA ftLinux MIB txt file to the host system and install it in opt ft mibs SRA ftLinux MIB txt or wherever the host system stores its MIBs You can use ftp to transfer the MIB file from the ftServer system to the host system as long as the two systems can communicate with each other over a network Run these commands to set up MIB path environment variables and reinitialize Net SNMP export MIBDIRS usr share snmp mibs opt ft mibs export MIBS ALL service snmpd stop service snmptrapd stop service snmpd start service snmptrapd start The host SNMP manager now can execute commands managing a network accessibl
84. e master boot record at track 0 makes both drives in the set bootable using GRUB configuration data stored in the boot partition Storage is allocated as shown in Table 5 1 e The boot directory is installed as an ext2 file system on dev sda1 and dev sdd1 on a partition of at least 256 MB This partition contains the GRUB boot loader configuration file and GRUB restricted shell as well as the Linux kernel and initial RAM disk files e A swap partition provides 2048 MB of swap space NOTE Regardless of installed system memory the current Linux kernel can only use about 2 GB of swap space per swap partition The kernel can address swap partitions on more than one drive so it is possible to improve swap performance on multi drive systems by using swap partitions on each installed drive pair However for fault tolerance always use mirrored swap partitions See Setting Up Internal Disk Storage on page 5 2 for more information e The root partition comprises 32 GB e The remainder of the space is an extended partition that includes 36 GB or more of unused space This extended partition can be further divided by logical partitions All internal drives used in mirrored RAID arrays in the internal hot swap drive bays must have firmware that meets ftServer specifications Do not update your internal drives with firmware from sources other than NEC Corporation of America Contact your NEC Corporation of America Customer Support C
85. ecklist on page 2 7 If you are upgrading to a new version of Express Builder skip this step and go to step 4 If you want to reinstall the same Express Builder version that is currently installed for example to ensure that it is not corrupted you must remove the currently installed Express Builder packages by typing the following commands 4 6 Express5800 ftServer System Administrator s Guide for the Linux Operating System Creating a Backup System Disk 4 a Remove all NECAM packages except eula_display with this command rpm e nodeps allmatches rpm qa grep lsb ft grep v eula display b Remove eula display with this command rpm e nodeps lsb ft eula display Follow the instructions in Installing Express Builder for Fault Tolerance on page 2 13 Your system now has a new version of Express Builder installed and the upgrade is complete If the upgrade failed or you want to return the system to the previous Express Builder version see Recovering from a Failed Software Upgrade on page 4 8 Creating a Backup System Disk To create a backup system disk 1 Shut down the system with the command halt p Remove the system disk from the bottom slot of the CPU I O enclosure whose power switch is not lit and set it aside as a backup disk To boot the system lift the switch cover of the CPU I O enclosure whose power switch is lit green and press the switch momentarily After the syste
86. ed uname r 25 65 97 34 EL serial idesmp To check the installed Express Builder distribution release level using the rpm command enter rpm q lsb ft eula display lsb ft eula_display 4 0 65 Storage Default Settings A CAUTION Prepare for installation by installing disk drives of identical size and geometry into the internal storage in the bottom slot the two slots labeled 1 sda and sdd of each CPU I O enclosure Remove all internal drives from the other four slots sdb sdc sde and sdf The installation process pairs the two installed drives NOTES 1 Do not install SATA disk drives in slots sda and sdd For performance reasons these drives are not supported as boot disks but can be used freely as data disks 2 Express5800 ftServer Dual Core and Quad Core systems may be mounted in a rack or in a pedestal The terms upper top lower and bottom assume a rack horizontal installation In pedestals systems are rotated 90 degrees counterclockwise from their rack mounted position so upper becomes left lower becomes middle or right and bottom becomes right 2 4 Express5800 ftServer System Administrator s Guide for the Linux Operating System Installation Overview The installation process creates a disk drive RAID array pairing sda and sdd drives as a mirrored set that holds the entire installed Linux software distribution and Express Builder On this mirrored drive set the GRUB bootloader on th
87. ed and mirrored e Networking Duplexed network components maintain network connectivity When the operating system detects a malfunction in the primary member of a duplexed pair it automatically fails over to the secondary member Express Service architecture built into ftServer systems supports these features with self checking hardware and onboard diagnostics to detect isolate and report potential problems before they affect server operation offering complete hardware diagnostics and alarms Introduction to ftServer System Administration 1 1 ftServer System Terminology ftServer System Terminology Each ftServer system houses two CPU O enclosures Each CPU I O enclosure includes a CPU element and an I O element as follows e CPU element 0 and I O element 10 The upper enclosure also referred to as CPU O 1 0 10 e CPU element 1 and I O element 11 The lower enclosure also referred to as CPU 1 1 0 11 System and Network Administration Overview Most examples and discussions in this guide assume that you are acting with root user or superuser privileges They do not always specify when you should or should not be acting as root Installing and Updating Software on ftServer Systems If you ordered the operating system from NECAM the supported Linux distribution and Express Builder were installed at the factory You also received distribution CDs containing Express Builder and the operating system These CDs are provi
88. emons need to exit before shutting the system down It is particularly important that you shut down SNMP before performing an Express Builder upgrade SNMP Foundations and Concepts The Net SNMP and ftISNMP packages support the SNMP protocol and many of the capabilities of SNMP for managing network objects using protocols and interface features described in numerous Internet Engineering Task Force IETF documents Net SNMP and ftISNMP are packages for network administration that compatibly support Express5800 ftServer fault tolerant operations using standard network communications There are few network administration tools available that readily support fault tolerant capabilities of networked systems and devices ftISNMP allows Express5800 ftServer systems to be monitored and managed by any remote networked system running SNMP based network management software 8 8 Express5800 ftServer System Administrator s Guide for the Linux Operating System SNMP Foundations and Concepts Net SNMP provides a functional network administration package for use on ftServer systems to meet identified customer needs ftISNMP is a unique extension of Net SNMP that provides the SRA ftLinux MIB to define manageable systems and components of ftServer Linux based systems ftServer subagents and MIB provide SNMP support and services for fault tolerant operations ftISNMP Management Commands The following commands are provided to start and stop the SNMP ma
89. enName 0 STRING OFFLINE SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PARENT EMPTY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN 8 36 Express5800 ftServer System Administrator s Guide for the Linux Operating System Trap Filtering SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206135032 759280 300 RFC1213 MIB sysUpTime 0 Timeticks 18744 0 03 07 44 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 6 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING OFFLINE SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PARENT EMPTY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206135047 759920 300 RFC1213 MIB sysUpTime 0 Timeticks 20244 0 03 22 44 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 6 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE
90. enter or your authorized NEC service representative for your internal SAS SATA drive support requirements Ethernet Default Configuration At installation the 10 100 1000 Megabits per second Mbps embedded Ethernet adapters of each CPU I O enclosure are configured For more information about configuring Ethernet interfaces see Ethernet Devices on page 5 26 USB and RS232C Default Settings Only the USB keyboard and optionally a USB mouse should be attached to the system during an initial installation of the operating system Installing the Operating System and Express5800 ftServer System Software 2 5 Separately Released and Optional Distribution Components In most cases attached devices are recognized and addressable on installation as is a standard USB keyboard for example although hot plugged devices may not be USB 2 0 interface specifications are supported After installation you may need to set serial port flow control and data rate characteristics for attaching serial data communications equipment or data terminal equipment such as an asynchronous terminal a printer or attached modem Note that the Linux operating system and most application software treat data communications equipment somewhat differently from data terminal equipment Default System Initialization and Run Level Control Default system initialization and run level controlled process configuration are basic and not tailored specifically for ser
91. erformed Verify that your system BIOS and BMC firmware versions are compatible with the Express Builder release you are about to install You can obtain required versions of firmware from the Express Builder CD ROM If system BIOS or BMC firmware updates are needed you must update the firmware before you begin the installation or upgrade process You can obtain the required firmware from the Express Builder CD that comes with the distribution See Chapter 3 for details Use the CD ROM drive in the top CPU I O enclosure during the installation Do not use the drive in the lower CPU I O enclosure Verify that the top enclosure is the active enclosure Ensure that the keyboard mouse and console are attached to the system The installer uses a graphical user interface on the ftServer system console which consists of a monitor attached to an SVGA port and an attached USB keyboard and mouse With the ftServer system halted detach all peripheral devices from the system This includes unsupported adapters USB and other serial devices No USB devices other than the keyboard and mouse should be attached to the ftServer system No external modem or other devices should be attached to the serial ports of the ftServer system The bottom slots slots sda and sdd in the internal storage enclosures must contain a matched pair of supported disk drives having the same model firmware level and geometry These must not be SATA drives All other d
92. es in a heterogeneous network environment Net SNMP has been ported to Linux UNIX and other operating systems such as Windows NT and Stratus VOS Note however that the Net SNMP and ftISNMP combined packages provide support only for ftServer systems running a supported Linux distribution together with Express Builder The Net SNMP and ftISNMP packages encourage deployment of distributed SNMP services on heterogeneous networks featuring both ftServer systems running a supported Linux distribution together with Express Builder and Express5800 ftServer hosts running other supported operating systems The Net SNMP and ftISNMP packages interact with and manage networked objects defined in MIB files The ftISNMP package includes the SRA ftLinux MIB file SRA ftLinux MIB txt to support fault tolerant Express5800 ftServer systems Net SNMP also supports MIB II and Host Resource MIB features The ftISNMP package follows the SNMP master agent daemon management model and extends the basic model using AgentX subagents This allows the subagents associated with different MIBs to be kept separate so that failure of one does not bring down the others Also the t ltrapsubagent was kept separate from the ft lsubagent to avoid blocking on serious traps AgentX extensions are defined in RFC2741 Agent Extensibility AgentX Protocol version 1 RFC2741 defines a standardized framework Simple Network Management Using Net SNMP and ftISNMP 8 15 Installing Remot
93. ever you cannot as a result of this discrepancy issue a command that would take the system offline See ftsmaint bringDown To verify the actual state of the device check the opstate of its LED ftsmaint lsLong This command displays the status of all fault tolerant devices on the ftServer system This command also returns the status of empty devices such as unpopulated PCI slots This command is useful to study the addressable fault tolerant devices that can be queried or controlled with Express Builder software ftsmaint lsPeriph This command displays information about peripheral devices such as CD ROMs DVDs and modems ftsmaint 1sVND This command displays the status of the Ethernet channel bonding interfaces in the ftServer system ftsmaint acSwitch 10 11 If you do not provide an enumerated hardware specification this command toggles the active compatibility of the I O elements between I O element 10 and I O element 11 If you do specify an I O element it forces the enumerated enclosure to active status ftsmaint bringDown path This command removes from service the CPU element I O element or CPU I O enclosure slot specified by path No other devices are supported When you bring down a device the effect on the system is the same as physically removing CPU 0 1 0 10 7 2 Express5800 ftServer System Administrator s Guide for the Linux Operating System The ftsmaint Command NOTE The ftsmaint bringDown
94. fault value 8924 Command m for help 8 Enter the t command Command m for help t Selected partition 1 Hex code type L to list codes 9 Press ENTER to accept the default ID 83 Enter d if you want the partition system ID to be Linux RAID autodetect Hex code type L to list codes fd Changed system type of partition 1 to fd Linux RAID autodetect Command m for help 10 Enter the w command to write the partition table to the disk and exit fdisk Command m for help w 5 6 Express5800 ftServer System Administrator s Guide for the Linux Operating System Setting Up Internal Disk Storage Default Internal Disk Configuration for a Newly Installed System The Linux operating system is installed on the sda sdd pair of SAS SATA disks See User Guide Setup for configuration of internal disk NOTE The RAID array is not deterministic Checking the Current State of the Internal Disk Subsystem The proc scsi scsi file displays the current state of the internal disk subsystem It shows all of the disks and their names and states as well as additional information This example shows a system with two SAS SATA disks Setting Up the ftServer System 5 7 Setting Up RAID Arrays Example 5 1 Checking the Current State of the Internal Storage Subsystem cat proc scsi scsi Attached devices Host scsi0 Channel 00 Id 00 Lun 00 Vendor ATA Model ST380013AS Rev 3 00 Type Direct Access ANSI
95. ftISNMP can retrieve operation state data on these devices The contents of SRA ftLinux MIB provide useful remarks about objects that can be managed 8 24 Express5800 ftServer System Administrator s Guide for the Linux Operating System SNMP and MIBS Device Enumeration See Table 7 1 for information on the enumeration of hardware components for ftServer systems running a supported Linux distribution together with Express Builder ftServer System Operation State Management Figure 8 2 illustrates the operational states and state changes in an ftServer system Figure 8 2 Operational State Management on ftServer Systems Initializing Diagnostics Diagnostics_passed Syncing The device physical states can be Empty or an online state such as Simplex or Duplex A device can be administratively Removed which the system then treats as an Empty state However a Removed device can be restarted by administrative command while an Empty device is restarted by physical removal if actually present and insertion of the device into a system slot The assigned state Shot is an automatically determined or managed state based ona failure threshold or MTBF A Shot state can be transitioned to an operational state by bringing the device up automatically or transitioned to Broken through a defined system administrative process A Broken device requires intervention an administrator can choose to return the device to service or may
96. ftware knows what has been written while the mirror was missing If the fast resync is not possible for example when a clean new disk is added a full resync is performed The full resync synchronizes all of the mirror that is in use If nothing was written while the mirror was missing no resync is needed Synchronization begins automatically when the mirror is added to a RAID array RAID software limits one resync per disk at a time so that for example on the system disks there may be one active resync and several delayed resyncs Example 5 3 provides an example of resynchronization Example 5 3 Resynchronization cat proc mdstat Personalities raid1 md21 active raidl sde1 0 sdf1 1 104320 blocks 2 1 U_ resync DELAYED md20 active raidl sdc1 2 sdd1 0 33640000 blocks 2 1 U_ PS SSS ia awa dndew aoe wd ered recovery 19 0 6400000 33640000 finish 2 1min speed 206451K sec md1l active raidl sdb2 1 sda2 0 2096384 blocks 2 2 UU Setting Up the ftServer System 5 17 Administering RAID Arrays md2 active raidl sdb3 1 sda3 0 31647936 blocks 2 2 UU md0 active raidl sdb1 1 sda1 0 104320 blocks 2 2 UU unused devices lt none gt NOTE The device names displayed in proc mdstat are the kernel names for each device These are different from the user device names displayed by the mdadm command As long as there is a missing mirror or a resynchronization in process RA
97. h most significant Express Builder upgrades Search for any prior ftISNMP with the following command rpm qa grep lsb ft snmp Run the following command to uninstall any conflicting SNMP package installation that is found rpm e lsb ft snmp Before you upgrade make sure the prerequisite Net SNMP packages are installed To install or upgrade ftISNMP from the distribution RPM enter the following command rpm Uvh pathllsb ft snmp 4 0 nnnn x86_64 rpm The string path is the directory containing the RPM and nnnn is the Express Builder build number The executables will be placed in opt ft sbin ftltrapsubagent and opt ft sbin ftlsubagent A plain text file representing SRA ftLinux MIB will be placed in the opt ft mibs directory Simple Network Management Using Net SNMP and fISNMP 8 3 Installing and Configuring fISNMP Enter the following commands and optionally add them to the login user s profile for example etc bash_ profile export MIBDIRS usr share snmp mibs opt ft mibs export MIBS ALL This installs or upgrades the MIBs and subagents ftISNMP Prerequisites ftISNMP requires Express Builder and the following Net SNMP packages to be installed Note that n n n nrepresents the current supported Net SNMP release number e net snmp libs n n n n e net snmp n n n n e net snmp utils n n n n e net snmp perl n n n n SNMP Configuration File Updates To configure ftISNMP edit the following files e etc
98. hat CPU element 0 is online and duplexed so it is safe to remove if from service To bring down CPU element 0 by invoking the ftcCpubdInitiateBringDown command use the whole numeric OID for that command 1 3 6 1 4 1 458 107 1 2 1 2 3 1 13 plus the CPU element 0 index 1 as the final octet Thus the complete OID is dM 36 Sl etl Ab BLO Al 21 2 Slee e The following example assumes that the community string private has been defined in etc opt ft snmp snmpd conf snmpset v 1 c private localhost 1 3 6 1 4 1 458 107 1 2 1 2 3 1 13 1 s test SRA ftLinux MIB ftcCpubdInitiateBringDown 3 STRING test Simple Network Management Using Net SNMP and ftISNMP 8 21 Managing SNMP To check that the CPU I O enclosure status has changed opt ft bin ftsmaint ls 0 H W Path 0 Description Combined CPU IO State OFFLINE Op State REMOVED FROM SERVICE Reason OK FOR BRINGUP LED State RED To bring CPU element 0 back up by invoking the ftcCpubdInitiateBringUp command use the numeric OID see SRA ftLinux MIB OID Values and Properties on page 8 160 for that command 1 3 6 1 4 1 458 107 1 2 1 2 3 1 11 again followed by CPU element 0 s index snmpset v 1 c private localhost 1 3 6 1 4 1 458 107 1 2 1 2 3 1 11 1 s test SRA ftLinux MIB ftcCpubdInitiateBringUp 3 STRING test If you check CPU element 0 s status immediately you can see that it has started initializing opt ft bin ftsma
99. he Express5800 ftServer Fault Tolerant Express Builder 1 3 Network Management and Reporting 1 3 Troubleshooting ftServer Systems 1 3 Additional Documentation and Resources 1 3 Red Hat Enterprise Linux 1 4 Express5800 ftServer Documentation 1 4 Linux and UNIX Documents 1 4 2 Installing the Operating System and Express5800 ftServer System Software 2 1 Installation Overview 2 2 Boot Media 2 3 Default System Setup 2 3 Linux Version Information 2 4 Storage Default Settings 2 4 Ethernet Default Configuration 2 5 USB and RS232C Default Settings 2 5 Default System Initialization and Run Level Control 2 6 Default User Environments Shells and Access Control 2 6 System Indicators and Switches 2 6 Separately Released and Optional Distribution Components 2 6 Installation Interfaces 2 7 Supported Hardware and Firmware 2 7 Contents iii Contents Pre Installation Checklist 2 7 Initial Linux Operating System and Express Builder Installation or Default Reinstallation 2 10 Booting the Operating System 2 10 Installing the Operating System 2 11 Installing Express Builder for Fault Tolerance 2 13 Avoiding CD ROM Mount Command Failure After Disabling SELinux 2 14 Reinstalling Express Builder After a Failed Installation 2 14 Booting in Linux Rescue Mode 2 15 Post Installation Tasks and Considerations 2 17 Default Configuration Notes 2 17 Configuring the Network 2 17 Adding Fault Tolerant Utilities to PATH 2 18 Performing an Installation
100. imeticks 5847 0 00 58 47 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 40 1 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING ONLINE SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206141318 517971 300 RFC1213 MIB sysUpTime 0 Timeticks 5948 0 00 59 48 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 6 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206141319 534567 300 RFC1213 MIB sysUpTime 0 Timeticks 6064 0 01 00 64 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 10 5 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING SIMPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenA
101. in rescue mode Use the Red Hat Enterprise Linux CD ROM 1 to boot in rescue mode 2 14 Express5800 ftServer System Administrator s Guide for the Linux Operating System Initial Linux Operating System and Express Builder Installation or Default Reinstallation To boot in rescue mode 1 Disconnect any floppy disk drive attached to the system s USB port NOTE If a floppy drive is connected when you boot in rescue mode the system will be unable to find the internal storage drives 2 Insert Red Hat Enterprise Linux CD ROM 1 into the CD ROM drive in the upper CPU I O enclosure The system boots from this CD 3 Disable the Boot Monitoring function in CMOS for detailed instructions see the User Guide Setup 4 After the system boots from the CD and as soon as the boot prompt appears type the following line at the boot prompt and press ENTER boot linux rescue NOTE You must type something at least one character on the boot prompt line before its timeout period expires Otherwise the boot will proceed with incorrect parameters and the keyboard will be disabled If this happens you can recover by power cycling the system and booting again 5 After several minutes the Language prompt appears Follow the prompts 6 Issue the following command chroot mnt sysimage You may need to remove and re apply system power before you can reboot from the disk If the disks are not detected and automounted co
102. ing For detailed information about controlling trap filtering see Trap Filtering on page 8 161 To perform initial testing of ft ltrapsubagent determine which system enclosure can be safely brought down Simple Network Management Using Net SNMP and ftISNMP 8 27 Initial SNMP Testing NOTES 1 Do not use this procedure on a deployed network host 2 Before continuing read ftsmaint 8 for information on single digit device path IDs and ftServer System Device Path Enumeration on page 7 5 if you have not already done so Select an enclosure that can be safely brought down To get a listing of options type opt ft bin ftsmaint 1s To get a report on a single device see ft smaint Examples on page 7 10 for an explanation of hw_path type opt ft bin ftsmaint ls hw path The status display must indicate DUPLEXED and ONLINE for a system enclosure that can be safely brought down NOTE The tsmaint command allows you to bring down an enclosure only if it has an operational duplexed partner When an enclosure is operating duplexed the simple duplex LED is lit steady white The simplex duplex LED is the bottom LED of the three LEDs on the left side of the front of the enclosure and is the right most LED on the rear of the enclosure You can also use ftsmaint to see if the enclosures are duplexed Suppose that CPU 1 I O 11 is duplexed You can bring it down leaving its partner CPU 0 I O 10 fu
103. ing System Kernel Memory Dump File Management For example the first command below brings down the bottom I O element the second command brings it back up opt ft bin ftsmaint bringDown 11 opt ft bin ftsmaint bringUp 11 NOTE Before removing an essential component like an I O element from service first verify that its partner is running When you issue the bringUp command the system should automatically synchronize the RAID array drives should update and become mirrored and the system should resume duplex operation Removing a PCI Adapter From Service and Bringing It Into Service You can also use the ftsmaint command to remove a PCI adapter from service For example use the following command to remove the PCI adapter in slot 9 of I O element 10 from service opt ft bin ftsmaint bringDown 10 9 You can bring that PCI adapter back into service by typing the following command opt ft bin ftsmaint bringUp 10 9 Kernel Memory Dump File Management A CAUTION To ensure that the ftServer dumping mechanism works successfully do not enable Diskdump or Netdump Diskdump or Netdump can interfere with the completion of an ftServer dump By default the supported Linux operating system is installed with kernel memory dump enabled If the system CPU I O enclosures are duplexed at the time of a crash the system creates a memory dump and stores the dump in the memory of one CPU I O enclosure The system re
104. instances to check for when pulling cables State changes include DUPLEX SIMPLEX BROKEN and of course various counters such as frames and collisions See OpState State Definitions on page 8 158 for state change identification 3 Run snmpwalk on the ftcEtherState OID before and after pulling each cable snmpwalk v 1 c private t 40 localhost ftcEtherState SRA ftLinux MIB ftcEtherState SRA ftLinux MIB ftcEtherState SRA ftLinux MIB ftcEtherState SRA ftLinux MIB ftcEtherState SRA ftLinux MIB ftcEtherState SRA ftLinux MIB ftcEtherState INTEGER triplex 22 INTEGER device ready 13 INTEGER simplex 20 INTEGER device ready 13 INTEGER simplex 20 INTEGER device ready 13 Nu amp UNE Il In practice you will actually redirect your snmpwalk output to files for before and after diff comparison For example in your work area run snmpwalk for the entire SRA ftLinux MIB file and dump that data to a file Pull the cable then run snmpwalk again and dump it to another file Finally run diff on the two files to see all Express5800 ftServer objects that have changed because of the fault insertion You may want to put these commands into a shell script for easier testing SNMP and MIBS The SRA ftLinux MIB file maps ftServer device definitions for management by Net SNMP and ftISNMP These device definitions map to addressable devices in the proc virtual file system
105. int ls 0 H W Path 0 Description Combined CPU IO State INRESET Op State INITIALIZING Reason NONE LED State RED After a while it is fully back up again opt ft bin ftsmaint ls 0 H W Path 0 Description Combined CPU IO State ONLINE Op State DUPLEX Reason PRIMARY LED State GREEN 8 22 Express5800 ftServer System Administrator s Guide for the Linux Operating System Managing SNMP Testing Ethernet Ports You can test Ethernet ports for proper traps and changes to OIDs On an ftServer system running a supported Linux distribution together with Express Builder Ethernet ports are uniquely identified When testing cable pulls or bringdowns the system should generate traps and the data that Express5800 ftServer MIB objects returned should reflect these changes One approach is to set up an snmpt rapd on a Linux system to verify the traps as they are generated see Verifying Traps on page 8 146 and to run snmpwalk on the SRA ftLinux MIB file before and after a fault insertion to verify object data changes A diff of these two walks will reveal changes that Express5800 ftServer MIB objects return Example Testing Ethernet Ports The following example demonstrates how to test Ethernet ports The example assumes dual port 10 100 1000 Mbps Ethernet PCI adapters installed in slot 1 of both CPU I O enclosures First determine the instance name of the Ethernet device and which slot it is in
106. interfaces are set to operate in active backup mode mode 1 with Dynamic Host Configuration Protocol DHCP enabled In many cases no additional configuration is necessary However you may want to change the default configuration to better meet your particular networking requirements You configure and administer the Ethernet interfaces on your ftServer system just as you would on any standard Linux system Additionally you can use the ft smaint command to obtain information about the fault tolerant status of the interfaces This section discusses the following topics e Monitoring Channel Bonding Interfaces e Configuring Channel Bonding Interfaces e Determining Interface Device Names Monitoring Channel Bonding Interfaces You can monitor the fault tolerant status of channel bonding interfaces by using the ftsmaint command Example 5 6 shows the default configuration of the embedded Ethernet devices Example 5 6 Default Configuration of Embedded Ethernet Devices opt ft bin ftsmaint 1sVnd Virtual Network Device VND Groups Group Name Status Inet Address RX Errors TX Errors Collisions bondo ONLINE 134 111 78 103 0 0 0 bond1 ONLINE 192 168 4 10 0 0 0 bond2 OFFLINE 0 0 0 bond3 OFFLINE 0 0 0 bond4 OFFLINE 0 0 0 VND Group Members Member Group Name Status Interface Link State Link Speed eth000010 bondo DUPLEX UP eth000011 bond1 DUPLEX UP Setting Up the ftServer Syste
107. introduces or defines new terms For example ftServer systems use replicated fault tolerant hardware to eliminate single points of failure and protect data integrity e The bold font emphasizes words in text For example Update the BIOS before you install or upgrade Express Builder e The monospace font represents text that would appear on your display screen The monospace bold font represents text you must type in examples that contain both user input and system output The monospace italic font represents terms in command lines that are to be replaced by literal values For example To display the state of a CPU enclosure type a command in the following format opt ft bin ftsmaint 1s n If you type opt ft bin ftsmaint 1s 0 atthe prompt the following output appears H W Path 0 Description CPU Node Assembly e The percent sign and the number sign are standard default prompt signs that have a specific meaning at a command prompt Although a prompt is sometimes shown at the beginning of a command line as it would appear on the screen you do not type it Express5800 ftServer System Administrator s Guide for the Linux Operating System Preface e indicates you are logged in to a user account and are subject to certain access limitations e indicates you are logged in to the system administrator account and have superuser access Users of this account are referred to as root The prompt sign used in an examp
108. ion Configuring Internal Disks The six internal storage disks are persistently named based on the slot that they occupy As shown in Figure 5 1 in the upper CPU I O enclosure the disks are dev sda dev sdb and dev sdc from bottom to top In the lower CPU I O enclosure they are dev sdd dev sde and dev sdf from bottom to top The name is associated with the slot not the disk Figure 5 1 CPU I O Enclosures Front Panel with Drive Slots Fully Populated 43 Area 6 sdc o le 2 sdb dle ll i 1 sda oT P Al l T Ja li pst tel il 2 sde AL I z P a sdd Ig LE ow asys076 The Linux operating system allows many possible configurations of these six disks To simplify administration and reduce confusion this section presents recommended configurations For convenience the disks are used in pairs based on vertical grouping of the disks in the CPU I O enclosures sda is paired with sdd sdb with sde and sdc with sdf RAID 1 arrays are created by placing one mirror on each dis
109. ion Information 2 Determine the interface device name for each physical Ethernet interface on this newly installed adapter See Determining Interface Device Names on page 5 30 for details You must add this device name to the physical interface s configuration file see step 5 3 Repeat steps 1 and 2 for the second adapter in the corresponding slot in the second paired CPU I O enclosure 4 Create a new bond file for example ifcf g bondz2 in the etc sysconfig network scripts directory Use the contents of an existing bondn file as a guide 5 Create two new physical interface configuration files for the two new physical interfaces Use the contents of an existing if cf g eth file as a guide Be sure to use the device names of the newly installed adapters see step 2 6 Type the following command to bring up the new interface ifup bond2 MAC Addresses You can use the ifconfig command to determine the current MAC address of an Ethernet interface Alternatively you can examine the interface s address file in the sys class net interfacename directory For the embedded Ethernet adapter interfaces Ethernet MAC addresses are algorithmically generated from a base MAC address assigned to the machine as a whole Each physical device receives a different MAC address Because of this a channel bonding interface and all of its physical devices may get a different MAC address from one reboot to the next depending on
110. ion and Express Builder and for system and application programmers who develop tools and scripts for use on these systems Background knowledge of Linux or UNIX shells tools and systems and Linux or UNIX and TCP IP network server and network administration technologies is assumed Revision Information This document is a revision This revision incorporates the following changes It documents the Express5800 ftServer Dual Core and Quad Core hardware platforms It documents the changed system software installation procedures It documents the internal storage SAS SATA disk drives It documents added functionality of the ftsmaint command It includes various updates and corrections Notation Conventions This document uses the notation conventions described in this section Warnings Cautions and Notes Warnings cautions and notes provide special information and have the following meanings A WARNING A warning indicates a situation where failure to take or avoid a specified action could cause bodily harm or loss of life Preface xi Preface AN CAUTION A caution indicates a situation where failure to take or avoid a specified action could damage a hardware device program system or data NOTE A note provides important information about the operation of an ftServer system Typographical Conventions xii The following typographical conventions are used in this document e The italic font
111. irmware version numbers are required A CAUTION Update your system firmware only if that version is compatible with your current Express Builder installation or if you will immediately update to operating system or Express Builder releases that are supported by the updated firmware Updating the System BIOS Make sure that the BIOS you intend to install is compatible with the Express Builder release level that you have or that you will install immediately after performing the BIOS upgrade Verify in the applicable version of the Release Notes Express5800 ftServer for the Linux Operating System that the BIOS is compatible and also that the Release Notes do not specify a different procedure or sequence for performing the BIOS update To check the BIOS version type the following command opt ft bin ftsmaint 1s 0 Firmware Rev 20 0 Updating ftServer System Firmware 3 1 Updating the System BIOS The preceding example displays a BIOS version number of 20 0 for the top CPU I O enclosure see Table 7 1 for a list of system device IDs Express5800 ftServer BIOS updates are image files that you must transfer from removable media or download from a network accessible archive Take care when updating firmware It is a necessary failover characteristic for the CPU I O enclosures to be paired in duplexed operation On system boot and before duplexing if the ftServer system detects differing firmware between the system CPU e
112. is case sd1usb you can use udevinfo to find the internal name in this case sde udevinfo q path n dev sdlusb block sde For more information about the udevinfo command see udevinfo 8 A CAUTION Before unplugging the device make sure that it is not being used the usage count is 0 If a file system is mounted unmount it and make sure the umount command completes before unplugging the device The umount command flushes any buffered pages back to the device so failing to wait for umount to complete can cause data corruption NOTES 1 An important consequence of the fact that the SCSI subsystem scans USB devices only on connection is that simply removing a floppy disk from the floppy drive or inserting a disk does not cause a rescan You must unplug the floppy drive and plug it back in to cause a rescan 2 During failovers access to USB storage devices is not robust 3 If an AC switch occurs some USB devices are not properly reset and disappear from ft smaint output To reset these devices you must unplug them and then plug them back in 4 When you unplug a device and later plug it back in the internal name may change if the SCSI subsystem has added other devices while it was removed The name assigned by the udev command however does not change If the active CPU I O enclosure fails over to the other enclosure a mounted USB disk drive device may become unusable If this happens remove the devi
113. is section discusses the following topics e Physical Device Naming e Monitoring and Configuring Channel Bonding Interfaces e MAC Addresses Physical Device Naming On many Linux systems Ethernet devices are normally assigned names based on the order of discovery at system startup The names begin with the letters eth followed by a number starting with 0 and counting up This is convenient because the first and often only device on a host is predictably named eth0 and can be configured without detailed knowledge of the device type On an ftServer system configuration may change dynamically when hardware failures occur repairs are made or when an administrator adds or removes components Creating new Ethernet device names when new hardware is installed tracking the name of an device while it is removed and replaced and matching it up again or deleting the name would be difficult and the results confusing Instead Express Builder assigns to network devices names that are derived from their physical location in the system Table 5 2 shows the names of the embedded Ethernet devices in ftServer CPU I O enclosures Setting Up the ftServer System 5 25 Ethernet Devices Table 5 2 Ethernet Devices in ftServer CPU I O Enclosures Device Embedded 10 100 1000 Mbps Ethernet PCI adapter Embedded 10 100 1000 Mbps Ethernet PCI adapter Embedded 10 100 1000 Mbps Ethernet PCI adapter Embedded 10 100 1000 Mbp
114. it Initializing USB mouse failed Mar 15 07 15 07 ftlx ifup SIOCSIFADDR No such device Mar 15 07 15 51 ftlx kernel Keyboard timed out 1 kernel keyboard timed out 1 Mar 15 07 15 52 ftlx kernel keyboard Timeout AT keyboard not present f4 9 6 Express5800 ftServer System Administrator s Guide for the Linux Operating System B backing up a system 4 7 5 26 creating disk 4 7 BIOS changing settings of Setup program 9 5 firmware 2 7 3 1 updating 3 1 BMC firmware 2 7 updating 3 5 bonding See channel bonding interfaces boot media 2 3 monitoring disabling 2 11 9 5 enabling 2 13 problems 9 1 booting the system disabling Boot Monitoring 9 5 error message 9 3 Linux rescue mode 2 15 normal sequence 9 2 problems 9 1 9 3 bring down safe 8 28 build number determining 7 5 Cc CD ROMs Red Hat Linux 2 2 with Express Builder distribution 1 2 2 2 with Linux distribution 1 2 2 2 channel bonding interfaces 5 28 configuring 5 29 determining device names 5 30 monitoring 5 28 checklists pre installation 2 7 pre upgrade 4 3 Index restoration 4 3 clock 5 33 configuring DNS resolution 5 32 Ethernet devices 2 5 ftISNMP 8 2 IP adress for bond interfaces 5 32 Linux operating system 1 3 RAID arrays 5 9 static routes 5 32 system host name 5 32 system time zone 5 32 console log 5 2 controlling system devices 7 12 Coordinated Universal Time UTC 5 33 CPU frequency
115. k of the pair For example RAID array dev mdo occupies sda1 and sdd1 Each SAS SATA disk in a pair must be the same size In the CPU I O enclosure internal disk slots insert only hard drives provided by NECAM Inserting any other type of device may cause data loss or system failure Managing Partitions You can use the fdisk or sfdisk utilities to display and change a disk s partition table and geometry see fdisk 8 and sfdisk 8 for details During the Linux operating system installation all of the mirrored boot partitions are created as type Oxfd Linux RAID Setting Up the ftServer System 5 3 Setting Up Internal Disk Storage autodetect After the installation you use the fdisk utility to add data disks with the type 0x83 Linux Both disks of a RAID pair must have the same geometry partition table and type You can use the disk command to manage disk partitions The following example uses the internal storage enclosure disk sdb To display the partition table 1 Enter the fdisk command fdisk dev sdb The number of cylinders for this disk is set to 17849 There is nothing wrong with that but this is larger than 1024 and could in certain setups cause problems with 1 software that runs at boot time e g old versions of LILO 2 booting and partitioning software from other OSs e g DOS FDISK OS 2 FDISK Command m for help 2 Enter the p argument of the fdisk command Command m for help
116. kts ifInNUCastPkts The sum of transmitted interface packets ifOutUCastPkts ifOutNUCastPkts The number of IP input packets ipInReceives 0 The number of IP output packets ipOutRequests 0 8 10 Express5800 ftServer System Administrator s Guide for the Linux Operating System SNMP Foundations and Concepts MIBs e snmptranslate This command converts object ID values into more easily understood forms See snmptranslate 1 e snmptable This command repeatedly uses SNMP GETNEXT or GETBULK requests to get information on a network entity which is specified as and must be mapped by a table See snmptable 1 e snmpset This command uses the SNMP SET request to control or set information on a network entity See snmpset 1 e snmptrap This command uses the SNMP TRAP operation to send information to a network manager when a trigger condition is met See snmptrap 8 e snmpinform This command essentially works like snmpt rap but uses a different form of signal and can require a response in order to suppress resending See snmptrap 1 e snmptest This command is a flexible test utility that can send a variety of signals and retrieve a variety of information It is best used within shell scripts that can hide its complexity and focus on particular test queries See snmptest 1 e snmpnetstat This command is a powerful data retrieval tool to query a remote system and retrieve a variety
117. l bonding interfaces Perform the following procedure to activate configuration file modifications To activate configuration file modifications 1 Type the ifdown command to stop all network interfaces 2 Type the rmmod command to unload the bonding kernel module This disables network access 3 Type the modprobe command to reload the bonding module This enables network access with the newly designated number of channel bonding interfaces 4 If necessary type an ifup bondN command for each bonded interface you wish to restart All channel bonding interfaces must operate in the same mode If you want to change the mode from the default setting mode 1 active backup mode modify the options line in the etc modprob d ft network conf file so that mode x where xis the desired mode of operation To activate configuration file modifications perform the preceding procedure 5 28 Express5800 ftServer System Administrator s Guide for the Linux Operating System Ethernet Devices NOTES 1 There must be at least one alias for an active bond in the etc modprobe d ft network conf file or bonding cannot occur 2 The etc modprobe a directory should contain no more than one ft network conf file Determining Interface Device Names When you add a PCI Ethernet adapter to a system you must determine the device names of the physical interfaces on the adapter before you can configure it See Figure 7 1 and Figure 7 2 for
118. le indicates the command can only be issued by root Syntax Notation This document uses the following format conventions for documenting commands e Square brackets enclose command argument choices that are optional For example cflow r ix i d num files e The vertical bar separates mutually exclusive arguments from which you choose one For example the following shows two mutually exclusive but optional argurments command argl arg2 The following example shows two mutually exclusive argurments one of which is required command argi arg2 In either case you may use either arg1 or arg2 when you type the command e Ellipsis indicates that you can specify the preceding argument as many times as you need to on a single command line For example command argl arg2 arg3 NOTE Dots brackets and braces are not literal characters you should not type them Any list or set of arguments can contain more than two elements Brackets and braces are sometimes nested Getting Help If your Express5800 ftServer system is covered by a service agreement please contact NEC Corporation of America Technical Support 866 269 1239 Notices e All regulatory notices are provided in the site planning guide for your system Preface xiii Preface e VTM is not available with Express5800 ftServer for Linux systems e Express Service Network is not available with Linux systems e Although this g
119. lements the firmware from the CPU I O enclosure that is booting is replicated to the other enclosure automatically allowing the enclosures to synchronize NOTE If an ftServer system boots from the system enclosure that has not been upgraded the upgraded system firmware will be overwritten with older firmware in order to synchronize to duplex This defeats the intended upgrade N CAUTION Do not interrupt a BIOS firmware update while a burn that is a write to EEPROM is in progress Interrupting power to a CPU I O enclosure during a burn can result in EEPROM damage that prevents it from storing the firmware successfully Correcting this problem may require that you obtain a replacement CPU I O enclosure Read this entire procedure to review your options before beginning To update the system BIOS 1 Insert the ExpressBuilder for Linux CD in the CD ROM drive in the CPU I O enclosure whose power button is lit 2 Log on to the system as root 3 Mount the CD ROM drive by typing the following command mount media cdrecorder 3 2 Express5800 ftServer System Administrator s Guide for the Linux Operating System Updating the System BIOS Use the ftsmaint command to verify that you are starting from a known good state At this point both CPU I O enclosures should be operating duplexed opt ft bin ftsmaint ls 0 H W Path Description State Op State Reason LED Green LED Yellow LED White Modelx
120. m 5 27 Ethernet Devices eth080010 bondo DUPLEX UP eth080011 bond1 DUPLEX UP In Example 5 6 there are two online channel bonding interfaces masters bond0 and bond1 each composed of two physical interfaces slaves The output shows the four physical slave interfaces in the system and also shows their status and the name ofthe bond to which they belong Note that three other channel bonding interfaces are defined by default but they are not configured and are therefore offline You can monitor additional information about the currently installed channel bonding and physical interfaces by running the Linux ifconfig tool Configuring Channel Bonding Interfaces You configure and administer channel bonding interfaces using standard Linux procedures Configuration of channel bonding interfaces is controlled by the ifcfg files in the etc sysconfig network scripts directory You modify existing channel bond configurations by editing the bond s ifcfg bondn file or the bond s slaves interface ifcfg eth files Additionally you use standard Linux network utilities like ifdown ifup service ifconfig ip and route By default the system supports five channel bonding interfaces In the unlikely event that you must configure more than five bonding master interfaces up to a maximum of 10 modify the options line in the etc modprobe d ft network conf file so that max_bonds x where x is the desired number of channe
121. m has booted insert a spare disk in the bottom slot of the CPU I O enclosure whose power switch is not lit Perform the procedures described in Manually Creating Partitions on Blank Disks and Adding to RAID 1 Arrays on page 5 20 Updating the Operating System and Express5800 ftServer System Software 4 7 Recovering from a Failed Software Upgrade NOTE Backup disks can be new factory fresh disks or disks recycled from other systems However care must be taken with recycled disks The partition table and RAID superblocks that exist on a recycled disk can confuse the system Recovering from a Failed Software Upgrade Use this procedure if an upgrade procedure failed or if you want to go back to the software versions installed before an upgrade procedure was performed To recover from a failed software upgrade procedure 1 Shut down the system with the command halt p 2 Remove the system disks from the bottom slots of both CPU I O enclosures and set them aside 3 Insert the backup system disk you prepared before performing the upgrade in the bottom slot of the CPU I O enclosure whose power switch is lit green 4 Boot the system by lifting the switch cover of the core CPU I O enclosure whose power switch is lit green and by pressing the switch momentarily 5 After the system has booted insert the one of the disks you set aside in step 2 in the bottom slot of the CPU I O enclosure whose power switch is n
122. ministrator s Guide for the Linux Operating System Upgrading or Restoring the Linux Operating System e To preserve your changes incorporate the updates into the files you have modified Compare files in the etc OPT ft network scripts ARCHIVE directory that have a rpmnew extension to your modified files and copy the updates from the rpmnew file to your modified file For example if you added a port to the bond defined in the etc sysconfig network scripts ifcfg bondo file and if the RPM file for the upgrade contains an update to the ifcfg bondo file from the earlier RPM file the upgrade process copies the updated ifcfg bondo file from the new RPM as etc OPT t network scripts ARCHIVE ifcfg bond0 rpmnew After the upgrade modify your etc sysconfig network scripts ifcfg bondo file with any differences you note in the etc OPT t network scripts ARCHIVE ifcfg bond0 rpmnew file NOTES 1 Resolve the differences in these files immediately or back up the rpmsave or rpmnew files to another location If the same rpmsave or rpmnew files are generated by uninstalling Express Builder or by an upgrade or reinstallation the previous archived versions could be overwritten 2 If you uninstall Express5800 ftServer system software any of these network files that you modified are saved to the etc OPT ft network scripts ARCHIVE directory and given a rpmsave extension Upgrading or Restoring the Linux Operating System U
123. mode or may return an error message If you do alter video settings change them only by using a text editor to change the entries in the etc X11 xorg conf file Use one of the configured video modes that the Express Builder installation program installed in the xorg conf file To do this select one of the available modes for the pixel depth you are using by putting the desired mode first in the list for that depth Changes made to xorg conf file are preserved during an upgrade Managing the System Clock 5 32 You may see the following message after the system boots or after you attempt to use the system config time utility Cannot access the Hardware Clock via any known method Use the debug option to see the details of our search for an access method The message occurs when you are trying to use Coordinated Universal Time UTC Instead use the Network Time Protocol NTP The following message does not indicate a problem with the system The clock will be properly reset and you can safely ignore the message Losing some ticks checking if CPU frequency changed Express5800 ftServer System Administrator s Guide for the Linux Operating System Additional Documentation and Resources Additional Documentation and Resources Linux System Administrator s Guide v0 8 Linux Documentation Project http www ibiblio org pub Linux docs linux doc project system admin guide http unthought net Software RAID HOWTO
124. ms This chapter discusses the following topics e LED and Visual Diagnostics e LED and Visual Diagnostics e System Log Messages This chapter provides information that will help you use available ftServer system and Linux operating system features to diagnose system problems In many cases you will be able to identify the source of the problem If you cannot contact the NEC Corporation of America Customer Support Center or your authorized NEC service representative LED and Visual Diagnostics Express5800 ftServer systems have a number of light emitting diodes LEDs and indicator lamps that can provide information of diagnostic value For a complete explanation of the location and interpretation of LEDs in your system see the troubleshooting chapter in the operation and maintenance guide for your system System Boot Problems If you experience problems in booting the system the following information may help you diagnose the problem e Normal Boot Sequence e Possible Boot Problems e Error and Log Messages Regarding Keyboard and Mouse Also refer to Booting in Linux Rescue Mode on page 2 15 for a procedure to boot the operating system after a failed installation Troubleshooting ftServer Systems 9 1 System Boot Problems Normal Boot Sequence The active CPU I O enclosure that is the primary enclosure whose power switch is lit green initiates the boot by starting the BIOS The
125. nal state some user action must occur to change this state User actions that cause a transition out of the BROKEN state include bringing the component up or down or removing the component DUMPING A CPU I O enclosure is recovering crash dump information DIAGNOSTICS A component is running diagnostics DIAGNOSTICS_PASSED A component has passed diagnostics INITIALIZING Software is preparing a device to be brought online KE FIRMWARE _UPDATE Board firmware code is being updated FIRMWARE UPDATE COMPLETE Board firmware code is updated Eg OFFLINE The unit has been brought down STOPPED The driver has stopped the component the component is no longer running 8 30 Express5800 ftServer System Administrator s Guide for the Linux Operating System Initial SNMP Testing Table 8 2 Operation State Values Names and Definitions Page 2 of 2 ONLINE The unit can be communicated with 20 SIMPLEX A component is online and has no partner it is not safe to remove this component Applies to components that can be partnered 21 DUPLEX The component is online and has a partner component that is running in lockstep mirrored or available for failover depending on the type of component This component is safe to remove Applies to components that can be partnered OpState Reason Definitions Table 8 3 lists reason names SRA ftLinux MIB codes and definitions for ftServer systems running a supported Linux distribution toge
126. nclosure in top I O enclosure 11 40 Internal disk slot 1 maps to sdd 11 40 1 Internal disk slot 2 sde 11 40 2 Internal disk slot 3 sdf 11 40 3 BMC 11 120 Fan speed sensor 11 140 Voltage sensors 11 150 11 162 t IDs in the format nn nn n indicate PCI bus slot and function These numbers may change as a result of certain system events and are provided here as representative sample data only Using ftServer Fault Tolerant Utilities and Software 7 7 The ftsmaint Command Figure 7 1 and Figure 7 2 show the locations of the major enumerated devices Figure 7 1 ftServer Enclosures Locations of Major Enumerated Devices Front View 5 eo PL 4 310 EL 3 1 2 F e yA 5 Ui Hf 6 2 asys076a Callout Device ID Component 1 0 CPU 0 1 0 10 2 10 40 1 Internal disk drive 1 sda 3 10 40 2 Internal disk drive 2 sdb 4 10 40 3 Internal disk drive 3 sdc 5 1 CPU 1 1 0 11 6 11 40 1 Internal disk drive 1 sdd 7 11 40 2 Internal disk drive 2 sde 8 11 40 3 Internal disk drive 3 sdf 7 8 Express5800 ftServer System Administrator s Guide for the Linux Operating System The ftsmaint Command Figure 7 2 ftServer Enclosures Locations of Major Enumerated Devices Re
127. nctioning with the following commands opt ft bin ftsmaint bringDown 11 opt ft bin ftsmaint bringDown 1 NOTE The term DIAGNOSTICS may appear when remotely bringing down an CPU I O enclosure This term is typically returned if a diagnostic test is in progress without regard to whether the test will succeed or fail DIAGNOSTICS is a transient state 8 28 Express5800 ftServer System Administrator s Guide for the Linux Operating System Initial SNMP Testing Use the following command to bring the CPU I O enclosure up again opt ft bin ftsmaint bringUp 11 opt ft bin ftsmaint bringUp 1 Initial Testing of Et lsubagent Use the snmpwalk tool to perform a get next operation on a system where an SNMP master agent is running See snmpwalk 1 For example for the ftcPcidevenf table snmpwalk Os c public v 1 t 40 localhost 1 3 6 1 4 1 458 107 1 2 5 2 1 ftcPcidevcnfMasterDataParityError iso 3 6 1 4 1 458 107 1 2 5 2 1 14 0 INTEGER 2 iso 3 6 1 4 1 458 107 1 2 5 2 1 14 1 INTEGER 2 ftcPcidevcnfSignaledSERR iso 3 6 1 4 1 458 107 1 2 5 2 1 15 0 INTEGER 2 iso 3 6 1 4 1 458 107 1 2 5 2 1 15 1 INTEGER 2 ftcPcidevcnfDetectedParityError iso 3 6 1 4 1 458 107 1 2 5 2 1 16 iso 3 6 1 4 1 458 107 1 2 5 2 1 16 1 INTEGER 2 INTEGER 2 Notice that the snmpwalk tool can provide symbolic decoding of absolute numbers OIDs Removing ftISNMP Whenever possible avoid m
128. new user 1 To create a new user configuration enter the following command net snmp config create snmpv3 user 2 When prompted enter the user name and password you want to assign Simple Network Management Using Net SNMP and fISNMP 8 7 SNMP Foundations and Concepts 3 Edit etc opt ft snmp snmpd conf and add the user to the VACM using a current group and view or creating new ones The following example lines add a new user paul to the current view and group in snmd conf by inserting the highlighted line for paul group v3group usm admin group v3group usm v3user group v3group usm paul view v3view included 1 3 6 1 access v3group usm authNoPriv exact v3view v3view v3view Note that this example shows that you previously ran snmpusm commands to create the admin user Start SNMP and run the following two commands to clone a new user and change the password snmpusm v3 u admin n 1l authNoPriv a MD5 A your_passwd localhost create paul admin snmpusm v3 u paul n 1 authNoPriv a MD5 A your passwd localhost passwd old_passwd new_passwd When you run start_snmp you will be able to use this user ID and password in SNMPv3 snmpwalk and snmpget commands for example snmpwalk v 3 t 40 1 authNoPriv u paul A new_passwd localhost 1 3 6 1 4 1 458 Unless you plan to use SNMP to monitor the local server you should configure SNMP to shut down when entering runlevel 1 At any rate SNMP da
129. ng ftServer Systems 9 5 System Log Messages N CAUTION Boot monitoring is one of the fault tolerant features of your ftServer system You must reenable it for full fault tolerance System Log Messages System log messages contain information on the operation state of the system The file var log messages contains system log messages You can find logs that are specific to ftServer systems in the directory var opt ft log See OpState State Definitions on page 8 158 and OpState Reason Definitions on page 8 159 for explanations of some of the terminology you may see in these messages Error and Log Messages Regarding Keyboard and Mouse In the system log or at system boot you may see stderr messages such as those shown in Example 9 1 These and similar messages may occur multiple times They are not a cause for concern if the system boots without undue delay and when the operating system presents you with the logon prompt it is appropriately presented on your display device and your input devices are supported to interact with the system The messages are an unavoidable result of the order in which drivers need to be loaded during the Linux operating system distribution boot process Example 9 1 Possible Keyboard and Mouse Error Messages at Boot Time Mar 15 07 15 00 ftix rc sysinit Initializing USB keyboard failed Mar 15 07 15 00 ftlx modprobe modprobe Can t locate module mousedev Mar 15 07 15 00 ftix rc sysin
130. nstalling the Express Builder you have the option of installing ftISNMP If you chose not to install ftISNMP at that time you can install it later by rerunning the Express5800 ftServer System Software for the Linux Operating System Express Builder installation program and selecting to install ftISNMP Installing ftISNMP this way will not reinstall packages already installed on the system and therefore not affect unrelated configurations On ftServer systems running a supported Linux distribution together with SNMP is composed of two separate packages e Net SNMP net snmp which is installed during the Linux operating system installation SNMP is a widely used protocol for monitoring network equipment for example routers computer equipment and devices such as uninterruptible Simple Network Management Using Net SNMP and ftISNMP 8 1 Installing and Configuring fISNMP power supplies UPS Net SNMP is a suite of applications used to implement SNMP v1 SNMP v2c and SNMP v3 using both IPv4 and IPV6 This suite includes Various command line applications for retrieving manipulating converting and displaying information Adaemon application for receiving SNMP notifications An extensible agent for responding to SNMP queries for management information A library for developing new SNMP applications See www net snmp org for more information about Net SNMP e ftISNMP Isb ft snmp which is preinstalled with Expre
131. ntact the NEC Corporation of America Customer Support Center or your authorized NEC service representative for assistance Installing the Operating System and Express5800 ftServer System Software 2 15 Post Installation Tasks and Considerations Post Installation Tasks and Considerations After installing the operating system and Express Builder consider the following topics e Default Configuration Notes e Configuring the Network e Adding Fault Tolerant Utilities to PATH Default Configuration Notes After installation the default installed system should appear as described in Default System Setup on page 2 3 The following notes apply to the default system configuration NOTES 1 After you upgrade or restore the Linux operating system and Express Builder distributions as described in this chapter you may also need to separately install optional packages using the rpm command 2 The system disk pair was created as a RAID 1 mirrored drive set on drives sda and sdd Each drive is bootable and configured identically to the other 3 After installation the kudzu haldaemon and microcode_ctl services are disabled Do not enable these services Configuring the Network Perform the following tasks to configure the system for operating on a network 1 Issue the following command to launch the Linux graphical network configuration program system config network 2 Inthe graphical network c
132. nted as user commands man 1 you should treat SNMP utilities as the administrative tools they are and closely limit privileges to execute these commands e snmpwalk This command uses SNMP GETNEXT requests to query a network entity for a tree of information that maps the managed objects by object ID hierarchically See snmpwalk 1 While this can return much information take care not to use this command on a heavily loaded net since it can add significantly to traffic e snmpget This command queries a single SNMP object using an SNMP GET request See snmpget 1 e snmpgetnext This command uses GETNEXT requests to query network entities for information e snmpgetbulk This command uses the SNMP GETBULK request to query a network entity for quantities of information See snmpgetbulk 1 e snmpdf This command replicates df command functionality on a network accessible drive The snmpdf command checks disk space on the remote machine by examining the system s HOST RESOURCES MIB hrStorageTable ora UCD SNMP MIB s dskTable value See snmpdf 1 e snmpstatus This command queries a network entity to retrieve significant information about a communicating object See snmpstatus 1 The following information is retrieved The IP address of the entity A textual description of the entity sysDescr 0 The uptime of the entity s SNMP agent sysUpTime 0 The sum of received packets on interfaces if InUCastP
133. nting Simple Network Management Protocol SNMP for managing network objects The ftISNMP package is typically installed with Express Builder Troubleshooting ftServer Systems Problem identification system and application diagnostics and system configuration to resolve problems with ftServer systems are essential troubleshooting tasks Chapter 9 discusses system features and procedures to assist you in troubleshooting ftServer systems Additional Documentation and Resources The following resources provide additional information that may be helpful to you in administering your ftServer system Introduction to ftServer System Administration 1 3 Additional Documentation and Resources Red Hat Enterprise Linux Documentation for the Red Hat Linux operating system is available at http www redhat com docs Express5800 ftServer Documentation The ExpressBuilder CD ROM provided with your system contains all of the system documentation for ftServer systems that run the Linux operating system It is provided in Adobe Acrobat Portable Document Format PDF for viewing and printing This manual occasionally refers to other documentation that is specific to your particular ftServer system Linux and UNIX Documents The following sources provide further information about the Linux and UNIX operating systems The Linux Documentation Project http www tldp org The System Administrator s Guide LDP v0 8 and the Network Admini
134. o the RAID array 5 18 Express5800 ftServer System Administrator s Guide for the Linux Operating System Administering RAID Arrays NOTE If the running member of the RAID array was a system disk the bootloader grub is added to the newly inserted disk To replace a failed disk 1 While the system and RAID array are running remove the failed disk 2 Insert a blank disk The blank disk is automatically added to the array Configuring Safe Mode By default the OSM configuration file opt ft osm config xml configures this automatic pairing of disks in safe mode In safe mode a newly inserted disk is considered blank if it has no valid partition table or RAID superblocks Otherwise a newly inserted disk is considered blank even if it has a valid partition table or RAID superblocks as long as it does not belong to a running RAID array The following entry in the OSM configuration file configures safe mode lt entry key blankDiskSafeMode value true gt To specify that automatic pairing of disks not operate in safe mode replace the word true with false so that the file contains the following entry lt entry key blankDiskSafeMode value false gt Because OSM preferences are only read at start up time you must reboot the system for a change to take effect Manually Creating Partitions on Blank Disks and Adding to RAID 1 Arrays When you create a backup system disk which is described in Creating a Backup
135. of information about communications objects See snmpnetstat 1 e snmpdelta This command is a tool used to monitor values of a network object over time and respond if the values deviate from established parameters See snmpdelta 1 A management information base MIB uses ISO Abstract Syntax Notation 1 to assign a unique object identifier to any object to be managed by SNMP This syntax is a hierarchical model that is intended to provide unique object identification Under this notation there is conceptually only one true MIB everything fits within it To the extent that developers observe syntactic standards various MIB definitions will not conflict because any MIB used should use only unique identifiers Thus Net SNMP and other SNMP implementations allow a large number of MIBs to be loaded from various paths at initialization under the presumption that all identifications are unique In current standard SNMP implementations at least the IETF MIB II definitions supporting RFC1213 must be used RFC1213 Management Information Base for Network Management of TCP IP based internets MIB II can be downloaded from the IETF Web site SNMPv1 MIBs supported only strings of data but later MIBs typically use a columnar layout of information that can be easily manipulated by a scripting language that handles textual data such as Perl The supplied MIBs are stored in the opt ft mibs and usr share snmp mibs directories in the default search path
136. on The following requirements must be met to ensure that these modules will be rebuilt e Express5800 ftServer kernel modules are only built for SMP kernels Make sure that you only use SMP kernels e The kernel smp devel package must be installed The version of this package must match the version of the kernel smp in use e The boot file system must be mounted when Express5800 ftServer kernel modules are rebuilt so the initrd can be re created with these modules boot is mounted by etc fstab e The directory opt ft must reside within the root file system Immediately after rebuilding kernel modules at boot time the system automatically reboots to place these rebuilt modules into use If the system cannot successfully build all the required Express5800 ftServer kernel modules a policy of whether a non fault tolerant boot is allowed or prohibited takes effect This policy is set by the etc opt ft modules policy file and affected by the etc opt ft non ft boot sh script Execute the install sh installation script to review or change the policy Upgrading the Linux Operating System Use the Red Hat Update Agent up2date to install a new Linux operating system U update Before you install a new Linux operating system U update for example U3 make sure to do the following e See the Release Notes Express5800 ftServer for the Linux Operating System or check with the NEC Corporation of America Customer Support Center or your au
137. on discusses the following topics e To Stop a RAID Array and Move It to Another System e Errors and Faulty Mirrors e Removing a Faulty Mirror e Resynchronization e Replacing a Failed Disk e Manually Creating Partitions on Blank Disks and Adding to RAID 1 Arrays You can use the mdadm command to administer RAID arrays The following sections provide examples of how to perform some common administrative procedures using mdadm NOTE Never remove both member disks of a RAID 1 array The Linux operating system does not support that operation To Stop a RAID Array and Move It to Another System You can stop RAID 0 and RAID 1 arrays if they are not in use Unmount the file system if one is mounted and stop the array as shown in the following example umount dev md30 mdadm S dev md30 Before physically removing the disks from the system check that the RAID array no longer appears in proc mdstat Edit the etc fstab and etc mdadm conf files on the current system to delete it Edit the information into the files in the new system If the new system already has the device in use you cannot start the RAID array Setting Up the ftServer System 5 15 Administering RAID Arrays You can start a RAID array when it is stopped Use the following command to start a RAID array that was already configured in etc mdadm conf mdadm A dev md30 Errors and Faulty Mirrors When an error is
138. on of a supported Linux operating system and Express Builder If you need to upgrade an existing Express Builder release see Chapter 4 LI Check that you have current release notes and installation guides for your distribution To restore your installation make sure you have the appropriate version of the release level that you will restore The Release Notes Express5800 ftServer for the Linux Operating System in PDF format on the ExpressBuilder for Linux CD provides the information that you need Also check the system hardware against the release notes The I O elements and installed devices must comply with any identified configuration requirements and Installing the Operating System and Express5800 ftServer System Software 2 7 Pre Installation Checklist 2 8 support restrictions on hardware that apply either to this installation procedure or to the current Express Builder release generally The installation CDs ask that you read and accept end user license agreements EULAs You should not perform the installation if you cannot accept the EULAs or are not authorized to accept them Installation terminates without completion if you decline a required EULA You can read the text of the required EULAs in the installation guide for your system If you are reinstalling a release back up all data files and prepare backup files to reconfigure the system for security and network operation after the installation procedure has been p
139. onfiguration tool specify the network hostname for your system See the section on adding hosts in the Red Hat Enterprise Linux 4 System Administration Guide for detailed information about configuring networks 2 16 Express5800 ftServer System Administrator s Guide for the Linux Operating System Performing an Installation Without a Kickstart File Adding Fault Tolerant Utilities to PATH Express5800 ftServer fault tolerant utilities like ftsmaint and ASNConfig reside in the opt ft bin and opt ft sbin directories Consider setting your PATH to include these directories Performing an Installation Without a Kickstart File 1 After installation while the system is booting the GRUB menu must supply linux reboot warm nmi_watchdog 0 i 8042 noaux At the boot prompt You must type something at least one character on the boot prompt line before its timeout period expires Otherwise the boot will proceed with incorrect parameters and the keyboard will be disabled If this happens you can recover by power cycling the system and booting again This command is necessary for your Express5800 ftServer to be fault tolerant You can find additional information about the kickstart files in the Red Hat Linux OS system administration documentation Installing the Operating System and Express5800 ftServer System Software 2 17 Additional Documentation and Resources 2 Manually make the second disk a bootable disk At the comman
140. opt ft snmp snmpd conf see The snmpd conf File on page 8 133 and snmpd conf 5 NOTE The Express Builder installation automatically creates snmpd conf in the etc opt ft snmp directory while Net SNMP creates it in the etc snmp directory You should use the etc opt ft snmp version because it is the directory that the ftISNMP scripts reference e etc opt ft snmp ftlsubagent conf see The ft lsubagent conf and ftltrapsubagent conf Files on page 8 133 e etc opt ft snmp ftltrapsubagent conf see The ftlsubagent conf and ftltrapsubagent conf Files on page 8 133 These files are created when you install or upgrade ftISNMP provided that they do not already exist If they already exist installing or upgrading ftLSNMP does not overwrite them Use the information in the template files for example etc opt ft snmp snmpd conf template as a guide when editing the files However values of parameters to be set in these files are system network and SNMP manager specific 8 4 Express5800 ftServer System Administrator s Guide for the Linux Operating System Installing and Configuring fISNMP The snmpd conf File A CAUTION Use SNMPv3 when the manager and master agent are separated on a public network The following is an example only Failure to use SNMPv3 when communicating over a public network is a server and network security risk SNMP V3 includes true authentication and encryption The th
141. ot lit The drive spins up the system adds it to the RAID array and resynchronizes it so that it mirrors the backup system disk You can monitor the mirroring process of drives sda and sdd by using the tools described in Chapter 5 or by monitoring the proc mdstat file 6 If you upgraded the BIOS and BMC firmware during the upgrade process you must restore the firmware to its previous version See Chapter 3 for details The system is now restored to the pre upgrade state Related Information and Resources 4 8 Release Notes Express5800 ftServer for the Linux Operating System Express5800 ftServer System Administrator s Guide for the Linux Operating System Chapter 5 Setting Up the ftServer System This chapter discusses the following topics e Setting Up Internal Disk Storage e Setting Up RAID Arrays e Removing and Replacing Disks e Administering RAID Arrays e System Backup and Disaster Recovery e Ethernet Devices e Other System Configuration Information e Additional Documentation and Resources At system startup the operating system autoprobes hardware for legacy devices and attached devices that are not already configured for use on the system Often the device is recognized and automatically supported requiring no direct configuration While the operating system may recognize legacy devices Express5800 ftServer System Software for the Linux Operating System
142. ow control of serial ports 2 6 ftISNMP 8 1 agents 8 17 configuration files 8 4 configuring 8 2 for remote service management 8 17 for Service Management 8 6 description 8 2 extensions 8 15 GET operations 8 32 installing 8 1 8 2 8 3 inventory 8 2 management commands 8 9 managing 8 18 MIBs 8 11 8 24 objects defined by 8 12 prerequisites 8 4 removing 8 29 SET operations 8 32 SRA ftLinux MIB 8 26 subagents 8 17 testing the configuration 8 19 uninstalling 8 29 upgrading 8 3 verifying traps 8 18 ftlsubagent description 8 2 initial testing 8 29 ftltrapsubagent description 8 2 initial testing 8 27 ftServer systems documentation 1 4 firmware 3 1 system administration 1 1 troubleshooting 1 3 9 1 Index 2 Express5800 ftServer System Administrator s Guide for the Linux Operating System Index ftsmaint command 7 1 acSwitch 7 2 bringDown path 7 2 bringUp path 7 3 burnProm fw file path 7 3 clearMtbf path 7 3 dump path 7 3 examples 7 10 ls path 7 2 lsLong 7 2 lsPeriph 7 2 IsVND 7 2 powerOff modem 7 3 powerOn modem 7 3 reset modem 7 3 resetMtbf path 7 3 runDiag path 7 4 setMtbfThresh value path 7 4 setMtbfType policy path 7 4 setPriority level path 7 4 setSensorThresh th name value path 7 4 task arguments 7 1 version 7 5 G GRUB boot loader 9 2 documentation 2 19 problems 9 4 shell 2 6 H hardware fault tolerant 1 1 managing 8
143. p Disk dev sdb 146 8 GB 146815737856 bytes 255 heads 63 sectors track 17849 cylinders Units cylinders of 16065 512 8225280 bytes Device Boot Start End Blocks Id System dev sdb1 1 17849 143372061 83 Linux Command m for help 3 In this example there is one partition sdb1 that is 143 372 061 1K blocks long Note the geometry 255 heads 63 sectors track This is the information required when you need to adjust the geometry of a replacement partner disk to match see Manually Creating Partitions on Blank Disks and Adding to RAID 1 Arrays on page 5 20 Enter the q command to quit or continue with other commands as required The following example creates a new partition table and adds a primary partition sdb1 of type Oxfd on sdb 5 4 Express5800 ftServer System Administrator s Guide for the Linux Operating System Setting Up Internal Disk Storage To create a new partition table and add a partition 1 If disk is not already running enter the fdisk command fdisk dev sdb The number of cylinders for this disk is set to 17849 There is nothing wrong with that but this is larger than 1024 and could in certain setups cause problems with 1 software that runs at boot time e g old versions of LILO 2 booting and partitioning software from other OSs e g DOS FDISK OS 2 FDISK Command m for help Enter the o command to create a new empty DOS partition table Note the caution displayed by the
144. pdate that modifies the kernel By default the Update Agent only updates RPMs that are already installed on the system If the operating system upgrade failed or you want to return the system to the previous operating system version see Recovering from a Failed Software Upgrade on page 4 8 Restoring the Linux Operating System 1 Make sure that you have a backup system disk in case the reinstallation fails or you want to return to the previous version of the Linux operating system See Creating a Backup System Disk on page 4 7 for details 2 Shut down the ftServer system with the command halt p 3 See the User Guide Setup to ensure that your system is configured for reinstallation Updating the Operating System and Express5800 ftServer System Software 4 5 Upgrading or Restoring Express Builder If the operating system reinstallation failed or you want to return the system to the previous operating system version see Recovering from a Failed Software Upgrade on page 4 8 Your system should now have the same version of operating system software installed as it had previously But since it has no Express Builder software its fault tolerant features are not operational so you must upgrade or restore Express Builder on your system See Upgrading or Restoring Express Builder on page 4 6 Upgrading or Restoring Express Builder Prepare for the Express Builder upgrade using this checklist 0 O
145. press5800 ftServer System Administrator s Guide for the Linux Operating System Installing and Configuring ftISNMP The ftISNMP package is preinstalled with the default Express Builder installation and only requires configuration and deployment Files in the ftISNMP package are located in the following directories e etc opt ft snmp Contains the fault tolerant subagent configuration templates and the Net SNMP master agent configuration template e etc opt ft snmp scripts Contains the start stop and restart scripts e opt ft doc lsb ft snmp 4 0 Contains the README file e opt t mibs Contains the SRA ftLinux MIB file e opt ft sbin Contains thefault tolerant subagents ftlsubagent and ftltrapsubagent e opt ft share man en mans Contains the fault tolerant subagent man pages Manually Installing and Upgrading the ftISNMP RPM Whenever possible avoid manual installation of ftISNMP Use the standard Express Builder installation and upgrade procedures to manage ftISNMP installations and upgrades Remove earlier versions of ftISNMP before you install a later ftISNMP release Whenever the Express Builder is upgraded the ftISNMP distribution may require updating as well Among other reasons changes in the proc virtual file system in supported hardware configurations and in device support may need to be reflected in the SRA ftLinux MIB file and possibly in supporting scripts You can expect ftISNMP to be upgraded wit
146. pulled the RAID 1 mirrors are broken the other CPU I O enclosure becomes simplex and its disk status LED becomes amber Pulling the CRU would cause all disk LEDs in the remaining CRU to go amber Pulling one disk would cause only its partner s LED to go amber An amber LED indicates that the device is no longer safe to pull The Opstate Manager OSM administratively removes the missing mirrors from their RAID 1 arrays When the CPU I O enclosure is re inserted the SAS SATA driver spins up the disk s and the OSM administratively adds the mirrors back into their RAID arrays The other CPU I O enclosure remains simplex until all mirror synchronization completes The RAID 1 fast resync feature can greatly reduce resync time SAS SATA RAID DM multipath and OSM plug ins are independent modules that interact with each other The Console Log and the var log messages File The system console displays messages from the internal storage subsystem This includes messages when disks are inserted and removed and when disk errors occur The console messages are also in the system log var log messages Tailing the messages file while configuring disks is very helpful tail f var log messages 5 2 Express5800 ftServer System Administrator s Guide for the Linux Operating System Setting Up Internal Disk Storage Since some disk configuration operations produce considerable console output it can be helpful to log on to another sess
147. r System Administrator s Guide for the Linux Operating System When the command completes the RAID array is up and running You can use the mdadm command to see the status of the new RAID array mdadm Q detail dev md20 dev md20 Version Creation Time Raid Level Array Size Device Size Raid Devices Total Devices Preferred Minor Persistence Update Time State Active Devices Working Devices Failed Devices Spare Devices Number 0 8 1 8 UUID Events Major 00 90 01 Wed Sep 28 15 20 08 2005 raidl 143371968 136 73 GiB 146 81 GB 143371968 136 73 GiB 146 81 GB 2 2 20 Superblock is persistent Wed Sep 28 15 20 58 2005 clean 2 2 0 0 Minor RaidDevice State 17 0 active sync 49 1 0 3 Setting Up RAID Arrays active sync 866e4ecd 12657190 79293b72 6b774c0d Creating a RAID 0 Array When the desired file system is larger than a single disk use RAID 0 to combine multiple RAID 1 arrays into a single RAID array This example assumes that two RAID 1 arrays have been created md20 consisting of sdb1 and sde1 and md21 consisting of sdc1 and sdf1 on the sdc sdf pair of disks To create a RAID 0 array 1 Select a pair of RAID 1 arrays Setting Up the ftServer System 5 11 Setting Up RAID Arrays 2 Editthe etc mdadm conf file so that the new RAID O array starts each time the system boots a Use an existing ARRAY line as a model Copy it to the bottom of the file b Edit
148. r are for a fresh installation or complete reinstallation of a supported Linux operating system and the Express5800 ftServer System Software for the Linux Operating System Express Builder on ftServer systems Chapter 4 describes an upgrade procedure and system and package restoration procedures for upgrading a recent distribution to the current software distribution level NOTE Be sure to read the accompanying Release Notes Express5800 ftServer for the Linux Operating System document before you undertake an initial installation or a reinstallation of Linux and Express Builder Installing the Operating System and Express5800 ftServer System Software 2 1 Installation Overview Installation Overview An installable distribution CD ROM CD set is provided Table 2 1 lists the CDs included in this distribution Table 2 1 CD ROMs Which May Be Included With ftServer Systems ExpressBuilder for Linux CD Express5800 ftServer fault tolerant system software ftControl Software Update Updated ftServer fault tolerant system software Express Builder Debug Info Includes debuginfo RPMs Set of Red Hat Enterprise Linux Red Hat Enterprise Linux operating operating system CDs system related packages and documentation If your site did not purchase the Linux operating system distribution from NECAM you must perform the complete initial Linux operating system installation However if you purchased an ftServer system and the Linux
149. rating system and Express Builder It is assumed that no valuable data exists on the target system disks The installation and upgrade installation procedures will destroy existing data on the drives in the bottom slot labeled 1 of each CPU I O enclosure If you need to repair a corrupted system or update the system to a new release determine whether the upgrade procedure in Chapter 4 will meet your requirements before doing a full installation Boot Media The supported Linux operating system is provided on a set of distribution CDs available from NECAM or the Linux operating system vendor Default System Setup This section provides an overview of the default setup that is provided on initial installation or complete reinstallation of the Linux operating system distribution It does not reflect installation of optional packages Installing the Operating System and Express5800 ftServer System Software 2 3 Installation Overview NOTE The Linux operating system installer program does not anticipate customer added and unknown hardware Any such hardware should be added and the system configured as required to support it only after installation procedures have been completed and the system has been determined to function as expected Linux Version Information You can check the installed version of the Linux operating system on your system using the uname command The r specifies that the kernel release level be return
150. ree authentication models are NoAuthnoPriv authNoPriv and authPriv Note that you must have auth status for encryption An SNMP engine identifier takes the first IP address as the default that identifies the agent in the device Each device must have a user login account for the device SNMPv3 also has concepts of groups views and privileges for access control These are referred to as the view based access control model VACM and user based security model USM You must keep the following two lines in the snmpd conf file for the master agent to function properly master agentx agentxTimeout 60 ftISNMP requires agentX services To avoid timeouts when the subagents are running under abnormal system stress for example 80 CPU usage and disks heavily stressed raise the value of agent xTimeout If a timeout occurs there will be a short delay while the subagents reinitialize their communications The tlsubagent conf and ftltrapsubagent conf Files These files require no editing for default operation but you may want to adjust logging You can change the trace level from off to brief or verbose as desired or as suggested by NECAM to aid in diagnosing any problems Debugging information will be logged These files contain the following configuration lines sraTraceLevel off sraTraceLog var opt ft log ftlsubagents log Simple Network Management Using Net SNMP and fISNMP 8 5 Installing and Configuring ftISNMP With
151. rives must be removed from the system See the drive arrangement in Figure 2 1 For information about the supported disk drives see the operation and maintenance guide for your system During the installation the two installed drives will be paired and configured using RAID 1 mirroring A Linux boot partition a swap partition and a root partition will be installed on the paired drives Express5800 ftServer System Administrator s Guide for the Linux Operating System Initial Linux Operating System and Express Builder Installation or Default Reinstallation Figure 2 1 SAS SATA Drive Arrangement for Installation SE RCE eat 2 sdb 21 CT O 2 sda Ig LE i 92 Or 3 sdf 3 BI 3B PL P le III 2 sde olzy nil a sdd d he amp asys076 O Make sure that the system and monitor power connections are secure and firmly plugged in before beginning an installation procedure Power cabling should be guarded against inadvertent disconnection during the installation process The monitor may use a separate power
152. s Ethernet PCI adapter Ethernet PCI adapter Ethernet PCI adapter Ethernet PCI adapter Ethernet PCI adapter Ethernet PCI adapter Ethernet PCI adapter Ethernet PCI adapter Ethernet PCI adapter Ethernet PCI adapter Ethernet PCI adapter Ethernet PCI adapter Ethernet PCI adapter Location CPU O 1 0 10 slot 5 port 0 CPU 1 1 0 10 slot 5 port 1 CPU 0 1 0 11 slot 5 port 0 CPU 1 1 0 11 slot 5 port 1 CPU O 1 0 10 PCI slot 9 port 0 CPU 1 1 0 10 PCI slot 9 port 1 CPU 0 1 0 10 PCI slot 10 port 0 CPU 1 1 0 10 PCI slot 10 port 1 CPU 0 1 0 10 PCI slot 11 port 0 CPU 1 1 0 10 PCI slot 11 port 1 CPU O 1 0 11 PCI slot 9 port 0 CPU 1 1 0 11 PCI slot 9 port 1 CPU 0 1 0 11 PCI slot 10 port 0 CPU 1 1 0 11 PCI slot 10 port 1 CPU 1 1 0 11 PCI slot 11 port 0 CPU 1 1 0 11 PCI slot 11 port 1 Ethernet Interface Device Name eth000010 eth000011 eth080010 eth080011 eth000008 eth000009 eth000218 eth000219 eth000220 eth000221 eth080008 eth080009 eth080218 eth080219 eth080220 eth080221 5 26 Express5800 ftServer System Administrator s Guide for the Linux Operating System Ethernet Devices Monitoring and Configuring Channel Bonding Interfaces By default the physical Ethernet interfaces listed in Table 5 2 are bound together into two channel bonding interfaces called bondO and bond1 The two channel bonding
153. s disk names 5 3 management partitions of disks 5 4 storage allocation 5 8 supported SCSI disks 5 2 system console messages 5 2 SCSI subsystem errors 5 17 Index 4 Express5800 ftServer System Administrator s Guide for the Linux Operating System Index serial ports data rate 2 6 flow control 2 6 setting up the Linux operating system 2 3 shells debug 9 3 GRUB 2 6 simplex 8 31 LED indicator 8 28 SNMP Simple Network Management Protocol 8 1 See also ftISNMP and Net SNMP basic Net SNMP commands 8 10 concepts 8 8 configuring for remote service management 8 17 configuring for service management 8 6 configuring to start at system initialization 8 6 fault tolerant operation 8 15 initial testing 8 27 managing 8 18 managing hardware 8 20 testing a configuration 8 19 testing Ethernet ports 8 23 traps See traps view of a network 8 14 snmpset command 8 20 snmptrapd 8 18 snmpusm command 8 7 snmpwalk 8 19 software optional components 2 6 SRA ftLinux MIB 8 26 description 8 2 GET operations 8 32 OID values and properties 8 32 SET operations 8 32 ftsmaint command identify 7 3 storage devices definition 5 9 subagents log files 8 17 system administration 1 1 1 2 backup 5 26 boot Linux rescue mode 2 15 normal sequence 9 2 problems 9 1 9 3 devices controlling 7 12 initialization 2 6 log messages 9 6 pedestal and rack orientation 2 4 system clock 5 33 system dumps
154. s CPU 1 up runs diagnostics on it automatically burns the new BIOS from CPU 0 onto it reruns diagnostics and resynchronizes CPU 1 with CPU 0 NOTES At this point reboot only if you need to change BIOS configuration settings 7 Repeat step 4 to verify that the CPU I O enclosures are again duplexed If the new BIOS did not perform as you expected first verify that you do not also need to perform an Express Builder upgrade to use the new BIOS If so proceed to step 9 returning here if the upgrade procedure does not initiate or follow the documented and expected steps If your operating system is fully up to date it is likely that the BIOS image file was not the correct firmware file for your system or the EEPROM that holds the BIOS did not properly capture the BIOS This occurs rarely but it can happen In that case repeating the burn procedure usually works 9 Double check that you have a good BIOS image file before attempting the BIOS upgrade again It should not matter whether you retry the burn on the upper CPU I O enclosure or the lower CPU I O enclosure but you should note which option you choose in case troubleshooting is required Remember that image files are easily corrupted during file transfer if copied from one format to another as when written out as a regular file rather than stored as an image or by transfer as 3 4 Express5800 ftServer System Administrator s Guide for the Linux Operating System Updating BMC
155. se this checklist to prepare for the Linux operating system upgrade or reinstallation D Before performing an upgrade to a system in use perform a complete file system backup D Carefully read the current Release Notes Express5800 ftServer for the Linux Operating System document at http support necam com for this release Distribution Release Notes may have been updated with information about restrictions or problems and workarounds software updates and document corrections not found in the Release Notes from earlier distribution CDs Verify that the new Linux operating system version you are about to install is intended for installation on your ftServer system A The system that you wish to upgrade must be configured to load from a bootable system disk that is in the ftServer system s boot path Updating the Operating System and Express5800 ftServer System Software 4 3 Upgrading or Restoring the Linux Operating System The following topics apply when upgrading or restoring the Linux operating system e Express5800 ftServer Kernel Modules e Upgrading or Restoring the Linux Operating System Express5800 ftServer Kernel Modules Whenever the Linux operating system is upgraded a new Linux kernel is installed Whenever Express Builder is installed or upgraded the fault tolerant Express5800 ftServer kernel modules are automatically rebuilt at the next boot time These modules must be present for fault tolerant operati
156. shows some traps that can occur when CPU 1 I O 11 2 brought up and trap filtering is on Example 8 4 Traps That Can Occur When Trap Filtering Is On RFC1213 MIB sysUpTime 0 Timeticks 10411 0 01 44 11 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 0 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE Simple Network Management Using Net SNMP and ftISNMP 8 41 Trap Filtering SRA ftLinux MIB ftcTrapGenName 0 STRING DUPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING PRIMARY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051207144108 553784 300 RFC1213 MIB sysUpTime 0 Timeticks 10511 0 01 45 11 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 2 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING DUPLEX SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING SECONDARY SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051207144109 557164 300 8 42 Express5800 ftServer System Administrator s Guide for the Linux Operating System Chapter 9 Troubleshooting ftServer Syste
157. ss Builder The ftISNMP package consists of the following components SRA ftLinux MIB This MIB supports NECAM s fault tolerant hardware ftlsubagent This subagent supports SRA ftLinux MIB SNMP GET and SET operations ftltrapsubagent This subagent supports SRA ftLinux MIB traps Various startup restart and shutdown scripts Man pages The ftISNMP package is currently built against a particular Net SNMP version Review the Release Notes Express5800 ftServer for the Linux Operating System for compatibility information During most Express Builder upgrades ftISNMP is also upgraded Install ftISNMP upgrades with the provided installer or upgrade script If a script is not provided use rpm to upgrade with the Express5800 ftServer binary ftISNMP distribution package This keeps the RPM database correct and provides a simple package management approach to maintaining ftISNMP ftISNMP Inventory This ftISNMP distribution is provided as a single binary RPM package on the Express Builder distribution called 1sb ft snmp 4 0 nnnn x86 64 rpm The number nnnn is the Express Builder build number When binary RPMs are installed the rpm database will track the revision level of your installed distribution ftISNMP binaries are built with performance optimizations for use with the Linux operating system as installed on ftServer systems See also the Release Notes Express5800 ftServer for the Linux Operating System 8 2 Ex
158. ss Builder distribution Express Service Network and ftISNMP packages are installed as options and require additional configuration before they can be used See Kernel Memory Dump File Management on page 7 13 and Chapter 8 for information on configuring and using these utilities Installation Interfaces The installation process has two parts First you install the Linux operating system and then you perform the Express Builder installation You must connect a supported monitor to the VGA port on the rear of the system and a supported USB mouse and keyboard to a USB port on the rear of the system The Linux operating system and Express Builder installation process use an attached LCD or SVGA capable monitor attached to the SVGA connector at the ftServer back panel and a USB mouse and keyboard attached to a USB port Supported Hardware and Firmware The Linux operating system combined with Express Builder can be installed only on supported Express5800 ftServer Dual Core and Quad Core systems Do not attempt to install this software combination on an ftServer system that does not support it Express Builder interfaces with ftServer firmware that has been tuned to support fault tolerance Specific Express Builder releases may require corresponding updates to the system BIOS or baseboard management controller BMC firmware Pre Installation Checklist The following checklist is provided for an initial installation or a full reinstallati
159. starts with the remaining CPU I O enclosure After the system restarts the dump is automatically written in a compressed format to a disk file in var crash YYYY MM DD HH mm vmcore You must send the dump file to theNEC Corporation of America Customer Support Center or your authorized NEC service representative for analysis Using ftServer Fault Tolerant Utilities and Software 7 13 Kernel Memory Dump File Management It is important that you monitor and maintain the size of the var crash directory Back up old crash dump data before deleting it 7 14 Express5800 320Ma System Administrator s Guide for the Linux Operating System NROO3L Chapter 8 Simple Network Management Using Net SNMP and ftISNMP This chapter discusses the following topics e Installing and Configuring ftISNMP e SNMP Foundations and Concepts e Installing Remote Network Management Services e Managing SNMP e SNMP and MIBS e SNMP Network Management Station Considerations e Initial SNMP Testing e Trap Filtering If you are reading this chapter for the first time be sure you first read Release Notes Express5800 ftServer for the Linux Operating System as well Because Simple Network Management Protocol SNMP is new to some administrators this chapter provides a conceptual introduction following the immediate discussions of installing and configuring ftISNMP Installing and Configuring ftISNMP When i
160. ster agent daemon and subagents on the ftServer host system e start snmp stop snmp restart_snmp These commands start stop and restart stop then restart the two subagents ft lsubagent and ftltrapsubagent and the master agent Use the preceding commands for most situations A CAUTION The following commands are also available to deal with special situations However you should not normally use them except under the guidance of the NEC Corporation of America Customer Support Center or your authorized NEC service representative as starting and stopping agents in untested order can have unforeseen consequences e start all subagents stop all subagents restart all subagents These commands start stop and restart only the subagents e start ftlsubagent stop ftlsubagent restart ftlsubagent These commands start stop and restart only ft lsubagent e start ftltrapsubagent stop ftltrapsubagent restart _ftltrapsubagent These commands start stop and restart only ftltrapsubagent e start snmp daemon stop snmp daemon restart snmp daemon These commands start stop and restart only snmp_ daemon the master agent Simple Network Management Using Net SNMP and ftISNMP 8 9 SNMP Foundations and Concepts The Basic Net SNMP Commands These tools provide a basic set of features for exercising and managing objects using a standard command syntax and core functionality NOTE Although these commands are docume
161. strators Guide LDP v2 0 are two freely redistributable publications available from the Linux Documentation Project Web site that you can use to supplement this Express5800 ftServer System Administrator s Guide for the Linux Operating System The LDP also provides other online manuals how to documents and links to additional online accessible data e Sunsite Web page University of North Carolina provides well ordered archives and links to many resources ftp sunsite unc edu pub Linux e Linux Administration Handbook by Nemeth Snyder and Hein copyright 2002 Prentice Hall PTR div of Pearson Education Inc 1 4 Express5800 ftServer System Administrator s Guide for the Linux Operating System Additional Documentation and Resources http vig prenhall com catalog academic product 0 4096 0130084662 00 html This volume is a reference manual for both system and network administration of the Linux operating system It focuses on available at time of publication open source tools but incorporates in depth knowledge of UNIX administration utilities and network management practices Linux in a Nutshell A Desktop Quick Reference 4th Ed by Siever Figgens and Weber copyright 2003 O Reilly amp Associates Inc This book can help you assess available tools and assemble an effective toolkit for managing servers and networks for example Open Source Network Administration by James Kretchmar copyright 2004 Prentice H
162. tcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 11 40 1 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING REMOVED SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051206141348 510919 300 Example 8 3 shows some traps that can occur when CPU 1 I O 11 is brought up and trap filtering is off Example 8 3 Traps That Can Occur for When Trap Filtering Is Off RFC1213 MIB sysUpTime 0 Timeticks 461810 1 16 58 10 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 2 SRA ftLinux MIB ftcTrapAlertType 0 STRING OPSTATE CHANGE SRA ftLinux MIB ftcTrapGenName 0 STRING DIAGNOSTICS SRA ftLinux MIB ftcTrapGenDetailInfo 0 STRING NONE 8 40 Express5800 ftServer System Administrator s Guide for the Linux Operating System Trap Filtering SRA ftLinux MIB ftcTrapGenAction 0 STRING UNKNOWN SRA ftLinux MIB ftcTrapGenEventTimeStampWithOffsetFromUTC 0 STRING 20051207143640 794126 300 RFC1213 MIB sysUpTime 0 Timeticks 466169 1 17 41 69 SNMPv2 MIB snmpTrapOID 0 OID SRA ftLinux MIB ftcTrapGenericInformationTrap 0 SRA ftLinux MIB ftcTrapDevicePathID 0 STRING 2 SRA ftLinux
163. ter 5 for configuration information Managing Data Storage Devices In addition to the SAS SATA disk storage discussed in Chapter 5 your system supports CD ROM drives and USB storage devices Chapter 6 provides a discussion of these devices and the information needed to manage them Using the Express5800 ftServer Fault Tolerant Express Builder While you can use standard Linux tools to perform many system administration tasks on your ftServer systems some tasks on fault tolerant systems require specialized supporting utilities The Express Builder includes libraries and utilities to support fault tolerant system administration tasks Chapter 7 discusses management tasks and utilities to manage fault tolerant features and supporting applications of your system They include the Express Service Network package that provides support for Express Service Network access using an attached modem This package allows the NEC Corporation of America Customer Support Center or your authorized NEC service representative to provide remote support for your system Your system comes with Express Service Network installed Network Management and Reporting Express Builder includes optional utilities to allow remote support of your ftServer system These include an extensible network administration framework and a server monitoring utility that provides notification services Chapter 8 discusses the configuration and use of the optional ftISNMP package impleme
164. the hardware device path of PCI Ethernet adapters and then refer to Table 5 2 to determine the device names When installing a PCI adapter you can use the output of the lspci command to confirm its Ethernet interface device name By comparing the output of 1spci before and after installing the adapter you can identify the newly displayed output that corresponds to the newly installed adapter The output might look like the following 04 02 0 Ethernet controller Intel Corp 82546EB Gigabit Ethernet Controller Copper rev 01 The device names for the two physical interfaces on the newly added dual Ethernet adapter shown above are eth000008 and eth000009 AN CAUTION Issuing an lspci command with the xxx option may temporarily interfere with the fault tolerant operation of the system To avoid this problem never type the lspci command with the xxx option To add two physical interfaces and configure a new channel bonding interface 1 Install the first Ethernet PCI adapter in a supported slot in one CPU I O enclosure NOTE When adding a pair of Ethernet PCI adapters to the system be sure to install one adapter in the upper CPU I O enclosure and the other in the lower CPU I O enclosure in same numbered slots This is necessary to maintain fault tolerance If you install both devices in the same enclosure and that enclosure fails you will lose connectivity Setting Up the ftServer System 5 29 Other System Configurat
165. ther with Express Builder Table 8 3 Reason Codes Names and Definitions Page 1 of 2 e CT ee specified for this component HARDWARE_INCOMPATIBLE The component hardware is incompatible with the online system hardware HOLDING_DUMP Bring up failed for dump is in process EAEL Simplex state was entered because a cable was unplugged 10 FIRMWARE_BURN_FAIL Failed to update the enclosure s BIOS or firmware 11 FIRMWARE_FILE_NOT_FOUND The entered firmware file path is either incorrect or the file does not exist FIRMWARE_FILE_ERROR There was an error in the firmware image on disk Simple Network Management Using Net SNMP and ftISNMP 8 31 Initial SNMP Testing Table 8 3 Reason Codes Names and Definitions Page 2 of 2 FIRMWARE_PROM_ERROR Could not write to the firmware PROM AUTOBURN_DISABLED Cannot match a new enclosure s BIOS or firmware with that of the existing enclosure 16 PRIMARY With duplex devices this indicates that the specific device is primary in the pair 17 SECONDARY With duplex devices this indicates that the specific device is secondary in the pair GET and SET Operations for ftISNMP MIB Objects See the SRA ftLinux MIB txt file for information on objects that have GET and SET operations Table 8 4 lists the operations Table 8 4 Set Operations Currently Implemented in ftISNMP ftcIobdInitiateBringUp ftclobdSetMTBFThreshold ftcEtherSetMTBFThreshold fccithersettbitype
166. thorized NEC service representative to make sure there is an Express Builder version available tosupport the Linux operating system update e Make sure the system is registered with Red Hat and has entitlement on the RHN server 4 4 Express5800 ftServer System Administrator s Guide for the Linux Operating System Upgrading or Restoring the Linux Operating System By default the Update Agent on a system running Express Builder is configured to access the following servers e The Red Hat RHN server for Linux OS patches To upgrade the Linux operating system 1 Start the Red Hat Update Agent from the graphical desktop or by running the up2date command on the command line The Update Agent does the following e Queries the RHN server for new versions of RPMs that are already installed on your system e Lists all available updates 2 Select the updates you want to install e By default all available OS updates that do not affect the kernel are selected for installation e By default all available OS updates that do require kernel modification are not selected To install these updates explicitly select their check boxes e By default all available Express Builder updates are selected for installation 3 The Update Agent downloads the updates and prompts you to install them 4 To install a new version of Express Builder see Upgrading or Restoring Express Builder 5 Reboot the system only if you manually selected an u
167. to provide new functionality using AgentX Although logging is flexible on some SNMP systems logging is simply merged with syslog output In the default ftISNMP configuration logging is configured as shown in Figure 8 1 Figure 8 1 AgentX Enabled Extensions and Subagents Net SNMP AgentX Communications SRA T ftLinux Subagent Master a MIE MIBs Agent bd ft SRA Trap ftLinux Subagent MIB ftlsubagent log ftltrapsubagent log By default snmpd log ft lsubagent 1log and fttrapsubagent log are located in var opt ft log You can relocate the subagent logs by modifying etc opt ft snmp ftlsubagent conf and etc opt ft snmp ftltrapsubagent conf see The ftlsubagent conf and ft ltrapsubagent conf Files on page 8 133 If you have sraTraceLevel set to brief or verbose in these files you may want to relocate the logs to a file system with ample space Simple Network Management Using Net SNMP and ftISNMP 8 17 Managing SNMP Verifying Traps You can easily verify traps using snmptrapd on a remote Linux system with Net SNMP installed 1 On the remote Linux system set up Net SNMP to autostart and verify it using the chkconfig command or manually start Net SNMP 2 On the ftServer system with the Linux operating system Express Builder Net SNMP and ftISNMP installed configure etc opt ft snmp snmpd conf with trapsink entries that point to the IP address of
168. u run these remotely the target name and IP address will differ To walk the SRA ftLinux MIB file snmpwalk v 1 c public t 120 localhost 1 3 6 1 4 1 458 snmpwalk v 2c c public localhost 1 3 6 1 4 1 458 To walk the Express5800 ftServer ftcPcidevenf table snmpwalk v 2c c public localhost 1 36 1 4 1 458 107 1 2 5 2 21L To walk the UCDAVIS MIB snmpwalk v snmpwalk v iso org dod snmpwalk v 2c cC publ 2c c publ lic localhost 1 3 6 1 4 1 2021 2 lic localhost internet private enterprises ucdavis memory 2c c publ lic localhost ucdavis Simple Network Management Using Net SNMP and ftISNMP 8 19 Managing SNMP To use SNMPv3 with snmpwalk snmpwalk v 3 1 authNoPriv u v3user A new_passwd localhost ucdavis snmpwalk v 3 1 authNoPriv u v3user A new_passwd localhost system snmpwalk v 3 1 authNoPriv u v3user A new_passwd localhost 1 3 6 1 4 1 458 snmpwalk v 3 t 40 l authNoPriv u v3user A new_passwd localhost 1 3 6 1 4 1 458 In these command examples v3user and new_passwd are the user name and password set up in Configuring SNMP for Service Management on page 8 134 Managing ftServer Hardware Components 8 20 In the following command examples v3user and new_passwdare the user name and password set up in Configuring SNMP for Service Management on page 8 134 To bring down a CPU element Note that a final octet 1 identifies CPU
169. uide may document modem functionality modems are not available for all systems Ask your sales representative about modem availability xiv Express5800 ftServer System Administrator s Guide for the Linux Operating System Chapter 1 Introduction to ftServer System Administration This chapter discusses the following topics e ftServer System Terminology e System and Network Administration Overview e Additional Documentation and Resources ftServer systems running a supported Linux distribution together with Express5800 ftServer System Software for the Linux Operating System Express Builder operate as fault tolerant servers The supported server models are the Express5800 ftServer Dual Core and Quad Core systems Every Express5800 ftServer system uses replicated fault tolerant hardware to eliminate single points of failure and protect data integrity in all areas of data handling including e Processing Replicated fault tolerant processing components process the same instructions at the same time In the event of a component malfunction the partner component acts as an active spare that continues normal operation preventing system downtime and data loss e Internal Storage Mirrored RAID 1 Redundant arrays of inexpensive disks arrays Serial attached SCSI and Serial Advanced Technology Attachment SAS SATA disks prevent single disk failures from causing data loss Replacement disks are automatically recogniz
170. v md20 r dev sdb1 In the preceding example the mirror dev sdb1 is removed from the RAID array dev md20 After running this command the proc mdstat file shows the RAID array without the mirror You cannot remove a mirror that is not faulty Before a disk is pulled all of the mirrors on it must be marked faulty either by the operating system or with the mdadm command and removed with the mdadm r command before the system can 5 16 Express5800 ftServer System Administrator s Guide for the Linux Operating System Administering RAID Arrays completely remove the disk the OSM storage plugin automates these tasks This means that until all mirrors are removed a replacement disk inserted in the same slot will not spin up You can use the mdadm command to add a mirror into a running RAID array The following example shows how to do this mdadm dev md20 a dev sdb1 In the preceding example dev md20 is the RAID array and dev sdb1 is the mirror After running this command the proc mdstat file shows the RAID array with the new mirror as a spare The command starts resynchronization of the mirror Resynchronization can take a while depending on how much data must be written to the new mirror Resynchronization must finish before the add operation is complete Resynchronization There are two classes of resynchronization fast and full A fast resync is done when the mirror was recently part of the array and RAID so
171. ve It to Another System 5 16 Errors and Faulty Mirrors 5 17 Removing a Faulty Mirror 5 17 Resynchronization 5 18 Replacing a Failed Disk 5 19 Configuring Safe Mode 5 20 Manually Creating Partitions on Blank Disks and Adding to RAID 1 Arrays 5 20 Replacing Defective Disks Interactively 5 21 Replacing Defective Disks Manually 5 21 The duplex blank disk Command 5 25 System Backup and Disaster Recovery 5 26 Ethernet Devices 5 26 Physical Device Naming 5 26 Monitoring and Configuring Channel Bonding Interfaces 5 28 Monitoring Channel Bonding Interfaces 5 28 Configuring Channel Bonding Interfaces 5 29 Determining Interface Device Names 5 30 MAC Addresses 5 31 Other System Configuration Information 5 31 Disabling Hyperthreading 5 32 Configuring the System Video Display 5 33 Managing the System Clock 5 33 Additional Documentation and Resources 5 34 6 Managing Data Storage Devices 6 1 CD ROM Drives 6 1 USB Storage Devices 6 1 USB Floppy Drives 6 4 USB Solid State Devices 6 4 Additional Resources 6 5 Contents v Contents vi 7 Using ftServer Fault Tolerant Utilities and Software 7 1 The tsmaint Command 7 1 Device Path Enumeration 7 5 ftServer System Device Path Enumeration 7 5 ftsmaint Examples 7 10 Displaying System Status 7 10 Bringing System Components Down and Up 7 12 Removing a PCI Adapter From Service and Bringing It Into Service 7 13 Kernel Memory Dump File Management 7 13 8 Simple Network Management Using Net SNMP
172. ve uf or below 1f e CRITICAL above uc or below 1c e WARNING above unc or below unc e NORMAL default e ftsmaint version This command returns the build number of the ftsmaint command on your system This number coincides with the build number of Express Builder installed on the system Device Path Enumeration Some subsystems and components of the ftServer system are addressable by device path IDs Device path enumerators uniquely identify the various devices in an ftServer system ftServer System Device Path Enumeration Table 7 1 lists the device paths for devices in an ftServer system Table 7 1 Device Paths of ftServer Devices Page 1 of 3 Location Device Path Top CPU element Top CPU element 0 DIMMs addressed by slot 0 0 0 7 Processors 0 20 0 23 CPU internal temperature sensor 0 20 130 0 23 130 CPU 12v sensors 0 20 150 0 23 150 Ambient air temperature sensor 0 130 Fan sensors 0 140 0 141 Voltage sensors 0 150 0 152 Using ftServer Fault Tolerant Utilities and Software 7 5 The ftsmaint Command Table 7 1 Device Paths of ftServer Devices Page 2 of 3 Location Device Path Bottom CPU Bottom CPU element 1 element DIMMs addressed by slot 1 0 1 7 Processors 1 20 1 23 CPU internal temperature sensor 1 20 130 1 23 130 CPU 12v sensors 1 20 150 1 23 150 Ambient air temperature sensor 1 130 Fan sensors 1 140 1 141
173. ver operations You will need to configure the system as required for your specific application Default User Environments Shells and Access Control The GRUB bootloader package supplied with the Linux distribution includes a restricted GRUB shell that can be entered at system boot for boot loader configuration and boot recovery operations See grub 1 for a discussion of this feature Make sure that GRUB requires root privilege and password protect this shell for system security System Indicators and Switches See the User Guide and User Guide Setup for your system for information about the LED indicators and switches of your ftServer system Separately Released and Optional Distribution Components You can install and use provided optional tools You can install optional packages by using the rpm command to select and install packages from a mounted CD ROM drive N CAUTION Some installed and optional utilities may depend on specific versions of other packages required by your system for fault tolerant operation Always use the software packages that are provided with the Express Builder distribution unless you have good reason to replace a package Before updating a distribution package use rpm 2 6 Express5800 ftServer System Administrator s Guide for the Linux Operating System Installation Interfaces to check dependencies Note that rpm does not always reveal specific release level dependencies From the Expre
174. y the IETF as the Internet Standard Management Framework RFC3410 SNMPv3 incorporates elements of SNMPv1 and SNMPvz2 and shares the same basic modular architecture This framework consists of four structures a data definition language SMlv1 a management information base MIB defining management information a separately defined communication protocol and security and administration applications and engines Features of SNMP version 1 SNMP version 2 and SNMP version 3 are not mutually exclusive IETF Best Current Practices 74 BCP74 describes how to implement these protocols compatibly on networks and on internetworked environments so that objects can be managed using the least sophisticated protocol required In this way networked and internetworked objects may be managed using SNMPv1 for example without becoming obsoleted if the network is commingled into a larger network where objects are managed using SNMPv2 or SNMPv3 This is necessary because the SNMP schema treats all networks as potentially a single network providing for addressing every object uniquely with a single MIB Accordingly implementing conformant extensions to SNMP should not cause interoperability conflicts with existing standards conforming SNMP implementations In the SNMP network universe any number of SNMP servers can exist and they can manage the objects they know about using SNMPv1 SNMPv2 SNMPv3 and with confidence that any SNMPv4 or subsequent protocol that
175. ystem as root 3 Mount the CD ROM drive by typing the following command mount media cdrecorder Updating ftServer System Firmware 3 5 Updating BMC Firmware 4 Obtain the latest BMC image for the Express5800 ftServer and copy it to the ftServer tmp folder NOTE All ftServers running a supported Linux distribution and Express5800 ftServer system software for the Linux Operating System Express Builder use the same BMC firmware 5 Type the following commands to update the BMC firmware on each I O element opt f t bin ftsmaint burnProm path and filename for the BMC firmware Updated firmware on the device at path 10 120 opt ft bin ftsmaint burnPROM path and filename for the BMC firmware Updated firmware on the device at path 11 120 6 Type the following commands to verify that your BMC firmware is duplexed be sure that the opstate is DUPLEX opt ft bin ftsmaint 1s 10 120 opt ft bin ftsmaint 1s 11 120 7 Type the following commands to verify that the I O elements are duplexed be sure that the opstate is DUPLEX opt ft bin ftsmaint 1s 10 opt ft bin ftsmaint 1s 11 3 6 Express5800 ftServer System Administrator s Guide for the Linux Operating System Chapter 4 Updating the Operating System and Express5800 ftServer System Software This chapter documents how to upgrade the Linux operating system and the Express5800 ftServer System Software for the Linux Operating System Express

NEC Express5800/320Fc Administration Manual

Contents

Download Pdf Manuals

Related Search

Related Contents