IBM Netfinity Cluster Enabler Hardware and Software Installation
Contents
1. X 09000000000 KKKA K 0 00 0000 09900000000000 dna aoo 0 0 0 0 00 00000 O H row soe 090000000000000 KOLKKO 00000 000 00000000 OO O 0 em e 800 boo bo vo ooo o 890000000 0 0 000 0009000000000 oo 00 000 0 0000000 0900000000 No COOOL 0 00 KKO o oo o 0 0 000000000 Ogm 000000000 00000000 00000 CO 00 DO 00 DO 00000 O 6 ewe 090 0 0 0 0 0 0 00 0 00 0 0 0 000000 Cp 0 90 00000000000900 oo KAA 0 o0 000000000000 000000000 000000 0 0 0 000000000000000000000 ooo ov DGD ODDO o 9900000 90 0 0000000 a vo 00 2000000000 0 0 0 000000000 Dea 006 0 900000 KKKA 0 0000 000 0 00 NO NINO 000000 XXS 20 00 9 0000 o opa a 0 0 0900000000000000000000 DUO 0 0000000000 000000 000000 oa ao 0 0 0 o o oo 200000 DO 9 0000000000 0 0 0000 000000000 0000000000000000 o Po BO OBD 0000 0000000000 000000 0 0 0000 989000000000000000000000000000 0 Po 6 o 0000 000000000 00000 989 0 000008000000000000000000000000 o do oo o0 0 OD 000000000 0000000000 00 0890000000000000000000 00 IO Oo Pp 0 000000000000000000000000000000 EEB o o o o o oo ob BD BB 000 0000000000 p 0 0 00 0 090 90 0 0 0 09090 9 0 0 09 0 0 0 0 0 0 Pao o ooo o o o o o o e eco a a 0000000000 0 0000000 000000000 0 9000000 XX 0000 ooo 0 000 00 G 0000 pa IBM 8271 712 Ethernet2. Important N TN ee Noe o Keyboard Mouse Power Switch Connect the monitor cable to the selector switch monitor connector Connect the keyboard cable to the selector switch keyboard connector Connect the mouse cable to the selector switch mouse connector Using the Console Server Selector Switch Cable Sets one set for each server connect one end of the monitor keyboard and mouse cable to the monitor keyboard and mouse ports on the first server installed in rack number one Looking below the label marked One on the rear of the Console Server Selector Switch connect the other end of the Console Server Selector Switch Cable Set keyboard and mouse cable to the Console Server Selector Switch connectors marked K and M respectively Connect the Console Server Selector Switch Cable Set monitor cable to the Console Server Selector Switch connector marked One Repeat steps 4 through steps 6 connecting one end of each of the cables in the Console Server Selector Switch Cable Set to its respective port at the rear of each server and the other end to the appropriate port on the rear of the Console Server Selector Switch Refer to IBM Netfinity Rack Console Server Selector Switch Installation Instructions for safety information and instructions on how to use the selector switch Step 8 Connect the IBM 10 100 EtherJet PCI Adapters to the IBM 8271 712 Ethernet LAN Switch
3. Note Label the cables from the Ethernet LAN Switch to the EtherJet PCI Adapter for future reference 1 2 For each server in your configuration connect one end of the Category 5 Ethernet cable to the IBM 10 100 EtherJet PCI Adapter Connect the other end of the Category 5 Ethernet cable to any port on the front of the IBM 8271 712 Ethernet LAN Switch 12 IBM Netfinity Cluster Enabler Step 9 Connect the Fibre Channel Cables to the IBM Netfinity Fibre Channel Hub Notes 1 The ports on the back of the IBM Netfinity Fibre Channel Hub are labeled 0 through 6 and can receive fibre channel cables from any of the installed server Fibre Channel PCI Adapters 2 Label all Fibre Channel cables for future reference 3 Do not bend crimp or coil the fibre channel cables smaller than a 4 inch diameter Doing so could cause data loss Extra GBICs may have to be added to the Fibre Channel Hubs depending upon your configuration The RAID Controller and Failsafe RAID Controller are each sold with one Media Interface Adapter MIA For each server in your configuration connect one end of the fibre channel cable to a IBM Netfinity Fibre Channel PCI Adapter that is installed in the server Connect the other end of the IBM fibre channel cable to one of the GBICs on the back of one of the IBM Netfinity Fibre Channel Hubs Connect the second fibre channel cable to the second IBM Netfinity Fibr
4. b Cc Double click Image Name to order the processes alphabetically Click Start Settings Control Panel Click Services and scroll down the selection list until you see IBMCoreClusterService under the Service heading and highlight the service Click Start to start the service After IBMCoreClusterService has started a list of started processes should be under the Task Manager Process List IBM Netfinity Cluster Enabler g If those started processes whose names begin with CS begin to fail and disappear from the Task Manager Process List refer to 5 and then contact IBM Support e OraclePGMSService From the Windows NT Task Manager click Processes a b Double click Image Name to order the processes alphabetically O Click Start Settings gt Control Panel d Click Services and scroll down the selection list until you see OraclePGMSService under the Service heading and highlight the service e Click Start to start the service From the Task Manager Process List a process named CMSRVR EXE should be listed after the OraclePGMSService has started g If this service is not started ping this node by host name to ensure network connectivity and verify that this server node can access shared DASD e Reinstalling or Reconfiguring the IBM Netfinity Cluster Enabler Software If you must reinstall or reconfigure the IBM Netfinity Cluster Enabler Software then you are generally
5. which could last from 2 to 3 minutes the IBM Core Cluster Service is still in the started state but some of the processes might not be fully restarted and running If the installation of the IBM Netfinity Cluster Enabler OSD software is interrupted for example by a power failure you will not be able to uninstall the files None The server will restart the services automatically Delete the UNINST ISU file located in the installation directory and reinstall the IBM Netfinity Cluster Enabler OSD software Shared drives show up first under Microsoft Disk Administrator instead of the local drives 28 To implement failover capability with dual IBM Netfinity Fibre Channel RAID Controller Units for Windows NT 4 0 an IBM Netfinity Fibre Channel Class device driver must be loaded prior to loading the Windows NT SCSI Class device driver This implementation will cause the shared disks to be assigned prior to the local disks with Windows NT Disk Administrator If a RAID controller fails the class device driver is required for the shared disks to failover to the alternate RAID Controller If a shared disk has failed correct the failed shared disks and recover the LUN disks before running SETLINKS or edit the ORAnnn TBL file to reflect the current disk configuration Copyright IBM Corp 1998 Problem Explanation Action The command Net Stop IBMCoreClusterService does not successf
6. IBM Commercial Desktop U S http www pc ibm com us intellistation IBM IntelliStation U S http www pc ibm com us netfinity IBM Netfinity and PC Servers U S http www pc ibm com us options IBM Options U S Help is also available from bulletin boards and online services as well as by fax and telephone Services available and telephone numbers listed are subject to change without notice Solving Problems Many computer problems can be solved without outside assistance by using the online help or by looking in the online or printed documentation that comes with your computer or software Also be sure to read the information in any README files that come with your software Most computers operating systems and application programs come with documentation that contains troubleshooting procedures and explanations of error messages The documentation that comes with your computer also contains information about the diagnostic tests you can perform 34 O Copyright IBM Corp 1998 Appendix A Supplementary Instructions for Installing a Six Node Oracle Parallel Server Configuration The Oracle Parallel Server Getting Started Guide provides instructions on how to set up the first four servers nodes of the cluster If you want to set up a six server node cluster use the instructions in this section along with the instructions in the Oracle Parallel Server Getting Started Guide These supplementary instructions are in addi
7. LAN Switch S oo o O o JC ES amp Installing the Hardware 7 Step 1 Unpack and Set Up the Racks Unpack and set up each rack Refer to the BM 9306 Model 900 PC Server Rack Enclosure Caution and Danger Information safety booklet to ensure the safe installation of your rack Notes 1 For individual racks be sure to install the stabilizer brackets 2 Refer to the IBM Netfinity Rack Configurator Program and the BM 9306 Model 900 Netfinity Rack Planning and Installation Guide for detailed installation instructions 3 Do not install the monitor keyboard and mouse at this time Step 2 Install the Required Upgrades into each IBM Netfinity 7000 M10 Server 1 Refer to the IBM Netfinity 7000 M10 Server Library documentation Legal and Safety Information section for safety concerns while installing these options 2 Install additional memory for a total of 1 GB in each IBM Netfinity 7000 M10 server Refer to the documentation that comes with the memory for detailed installation instructions and then proceed with installing the microprocessors 3 Install the microprocessors for a total of 4 in each IBM Netfinity 7000 M10 server Refer to the documentation that comes with the microprocess
8. desired directory name Important Be sure to use the same fully qualified directory path for each server c Click OK d Click Next IBM Netfinity Cluster Enabler Setup verifies all the necessary information before copying the IBM Netfinity Cluster Enabler files to the desired directory 6 Click Next When the IBM Netfinity Cluster Enabler finishes copying the files into the desired directory the README TXT file will appear When you close the README TXT the following screen appears Setup Complete Setup has finished copying files to your computer Before you can use the program you must restart Windows or your computer No will restart my computer later Remove any disks from their drives and then click Finish to complete setup 7 Select Yes want to restart my computer now 8 Click Finish Note The server must be restarted for the settings to take effect 9 Repeat steps 2 on page 18 through step 8 and install the IBM Netfinity Cluster Enabler files on each server Step 8 Create a Shared Partition for the IBM Cluster Enabler Heartbeat Disk The IBM Netfinity Cluster Enabler software uses a special logical partition on a shared disk for communication in the event of a failure From a disk subsystem the information in this logical partition referred to as the heartbeat disk is used by all servers to determine the correct configuration for continued operation Therefo
9. new services and their respective port numbers must be added to the Windows NT SERVICES file in each server The IBM Netfinity Cluster Enabler software in each server uses these files to communicate with the software in the other servers The port numbers are used to direct IP messages to the correct program in each server Notes 1 Certain editors such as Notepad will corrupt the network SERVICES file Use Edit that is shipped with Windows NT to change your network SERVICES file 2 The above port numbers are examples only 3 The protocol for csgsr is udp while the protocol for the other services is tcp Any port number that complies with the following rules can be used 1 Each of the four services must have a different port number 2 Each port number can be any 4 digit number that is not already used in the SERVICES list and that is not already being used by any other program 3 The port numbers for a particular service must be the same on all servers nodes in the cluster To add port numbers in the SERVICES file in each server do the following 1 Open a command prompt window 2 Type edit SYSTEMROOT SYSTEM32 DRIVERS ETCISERVICES 3 Add the five new services and their port numbers Services Port Protocol csts 1234 tcp cstsr 1235 tcp csgs 1236 tcp csqll 1237 tcp csgsr 1238 udp 4 Repeat steps 2 and 3 for each server The IBM Netfinity Cluster Enabler OSD software installation is now complete and Ora
10. required to stop the Oracle database and its associated services on all servers nodes If this is the first time that you are bringing up the OPS or if you are running the OPS on a test configuration and not a production configuration then reinstall and reconfigure the IBM Netfinity Cluster Enabler Software using the steps beginning at Step 7 Install the IBM Netfinity Cluster Enabler OSD Software on page 18 in this publication If the server node exhibiting the problem has been successfully running in a production mode or the OPS has been successfully running elsewhere on this cluster then you might be able to resolve the problem without shutting down the entire cluster For this situation refer to 5 on page 32 in this publication and then contact IBM Support Troubleshooting IBM Netfinity Cluster Enabler Configuration 33 Getting Help Service and Information If you need help service technical assistance or just want more information about IBM products you will find a wide variety of sources available from IBM to assist you For example IBM maintains pages on the World Wide Web where you can get information about IBM products and services find the latest technical information and download device drivers and updates Some of these pages are http www ibm com Main IBM home page http www pc ibm com IBM Personal Computing http www pc ibm com support IBM Personal Computing Support http www pc ibm com us desktop
11. the ServeRAID EtherJet PCI and Fibre Channel Adapters into each Netfinity 7000 M10 server Step 4 Install the hardware UPS units servers keyboard hubs monitor and Console Selector Switch into rack 1 Step 5 Install the hardware UPS units servers EXP 15 Storage Units into rack 2 Step 6 Install the hardware UPS units servers RAID Controller Unit and EXP 15 Storage Units into rack 3 Step 7 Connect the monitor keyboard and mouse cables to the Console Server Selector Switch and each Server Step 8 Connect the IBM 100 10 EtherJet PCI Adapter in each server to the EtherJet LAN Switch Step 9 Connect the Fibre Channel cables from the Netfinity Fibre Channel PCI Adapter in each server to the Netfinity Fibre Channel Hub Step 10 Connect the SCSI cables from the Netfinity EXP 15 to the Netfinity Fibre Channel RAID Controller Unit Step 11 Connect the power cables from all components to the APC Smart UPS Step 12 Power up the hardware according to the prescribed sequence in this document Step 13 Your hardware installation and cabling is complete Cable Management and Labeling 5 Front and Back View of the Hardware These illustrations contain detailed front and rear views of the hardware components illustrated in this overview The illustrations in this overview might look slightly different from your hardware Fro
12. 0 Configure the IBM Netfinity Cluster Enabler OSD Software 22 Step 11 Set Port Numbers in Windows NT Services File 27 Troubleshooting IBM Netfinity Cluster Enabler Configuration 28 Isolating Problems 30 iii iv Getting Help Service and Information 34 Solving Problems e 34 Appendix A Supplementary Instructions for Installing a Six Node Oracle Parallel Server Configuration n 35 Step 1 Assign Symbolic Links to Each Logical Drive 35 Step 2 Configure the IBM Netfinity Cluster Enabler OSD Software 36 Step 3 Create Services 2 2 2 36 Step 4 Add the ORACLE SID Entry to the Registry 36 Step 5 Configure the Network 00 0000 0 eee eee 36 Step 6 Create an Oracle Parallel Server Database 36 Appendix B Product Notices 38 Trademarks 6 2 2453 Puma SAAT be Beg ane Spey be dT n 38 IBM Netfinity Cluster Enabler About This Book This book contains instructions to assist you with the setup of your IBM Netfinity Cluster Enabler hardware and with the installation of the IBM Netfinity Cluster Enabler Operating System Dependent OSD software that is used to build an IBM Netfinity server cluster The OSD software is required for the proper operation of the Oracle Parallel Server OPS The hardware setup consists of two four or six IBM Netfinity 7000 M10 servers nodes with a fibre chann
13. 900 Netfinity Rack Planning and Installation Guide This book contains information about the IBM 9306 Model 900 Netfinity Rack It also contains a product description a list of available options option and equipment installation guidelines and a description of the IBM Netfinity Rack Configurator program IBM Netfinity 7000 M10 Server Library This book contains detailed instructions for installing and setting up your IBM Netfinity 7000 M10 servers IBM Netfinity EXP15 Installation and User s Handbook This book provides instructions for installing and replacing options and provides information on troubleshooting the IBM Netfinity EXP15 Storage Expansion Unit IBM Netfinity Fibre Channel PCI Adapter Hardware Installation Handbook This book contains detailed instructions for installing the IBM Netfinity Fibre Channel PCI Host Adapter IBM Netfinity Fibre Channel RAID Controller Unit User s Handbook This book contains detailed instructions for installing a rack mounted IBM Netfinity Fibre Channel RAID Controller Unit IBM Netfinity Fixed Shelf Installation Instructions This book contains detailed instructions for installing fixed shelves in the rack IBM Netfinity Rack Attachment Kit Installation Instructions This book contains detailed instructions for connecting two or more racks together to form a suite of racks IBM Netfinity Rack Monitor Compartment Installation Instructions This book contains detailed instructions for installing
14. IBM Netfinity Cluster Enabler Hardware and Software Installation Guide for Oracle Parallel Server IBM Netfinity Cluster Enabler Hardware and Software Installation Guide for Oracle Parallel Server Note Before using this information and the product it supports be sure to read the general information under Appendix B Product Notices on page 38 First Edition November 1998 The following paragraph does not apply to the United Kingdom or any country where such provisions are inconsistent with local law INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This publication could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements and or changes in the product s and or the program s described in this publication at any time This publication was developed for products and services offered in the United States of America IBM may not offer the products services or features discussed in this document in other countries and the in
15. Run the ASCII input file for each ORALINKn TBL through the SETLINKS application as shown below Copyright IBM Corp 1998 35 C gt CD ORACLE_HOME OPS C ORACLE_HOME OPS gt SETLINKS F ORALINKn TBL where x is one less than the number of the node where F is the drive where the ORALINKn TBL file exists Refer to the instructions in the Oracle Parallel Getting Started Guide for more information and then proceed to Step 2 Step 2 Configure the IBM Netfinity Cluster Enabler OSD Software Refer to Step 10 Configure the IBM Netfinity Cluster Enabler OSD Software on page 22 for instructions and then proceed to Step 3 Step 3 Create Services Refer to the instructions in the Oracle Parallel Server Getting Started Guide and then proceed to Step 4 Step 4 Add the ORACLE SID Entry to the Registry Refer to the instructions in the Oracle Parallel Server Getting Started Guide and then proceed to Step 5 Step 5 Configure the Network Refer to the instructions in the Oracle Parallel Server Getting Started Guide and then proceed to Step 6 Step 6 Create an Oracle Parallel Server Database 1 Ensure that you have copied the following files to the ORACLE_HOME OPS directory C_RBS5 SQL C THR5 SOL C RBS6 SOL C THR6 SQL 2 To connect to the instance and run the OPSALL SQL script to create the database type the following C gt CD ORACLE HOMEVOPS C ORACLE_HOME OPS gt SVRMGR30 SVRMGR gt OPSALL SQL Note I
16. are into Rack 2 n 10 Step 6 Install the Hardware into Rack 3 L 10 Step 7 Connect the Monitor Keyboard and Mouse Cables to the Console Server Selector Switch e 12 Step 8 Connect the IBM 10 100 EtherJet PCI Adapters to the IBM 8271 712 Ethernet LAN Switch aaa aaa 12 Step 9 Connect the Fibre Channel Cables to the IBM Netfinity Fibre Channel UD ss a e rk Se ee en A Se m t E Kandi 1 13 Step 10 Connect the IBM Netfinity EXP15 Drive Module Cables to the IBM Netfinity Fibre Channel RAID Controller Unit 13 Step 11 Connect the Power Cords to the APC Smart UPS Power Supply Units 13 Step 12 Power On the Hardware 0 000000000 14 Installing the Software n 15 Step 1 Configure the ServeRAID 3L Adapter 16 Step 2 Install Microsoft Windows NT Server Version 4 0 Enterprise Edition 16 Step 3 Install the IBM Netfinity Fibre Channel Adapter Device Drivers 16 Step 4 Install IBM Netfinity SYMplicity Storage Manager 17 Step 5 Verify and Test Interprocess Communication Connectivity 17 Step 6 Install Oracle Parallel Server OPS 17 Step 7 Install the IBM Netfinity Cluster Enabler OSD Software 18 Step 8 Create a Shared Partition for the IBM Cluster Enabler Heartbeat Disk 21 Step 9 Assign Symbolic Links for the IBM Netfinity Cluster Enabler Heartbeat DISK ein k laa Hn ae ce Rir he Peds Ba SA et TALE Ae Bee Rate BAe ated 22 Step 1
17. cable connections between your servers disk subsystems multiple network connections power configurations and video subsystems Examples include fibre channel configurations server clusters multiple unique clusters located in the same rack or across multiple racks and clusters where components might not be physically located in the same room building or site The necessity for detailed cable management and labeling is due to the complexity of today s configurations and the potential distances between components Benefits from more detailed cable management and labeling include e Ease of installation e Ongoing cluster and systems management e Increased serviceability Be sure to label your cables Refer to the IBM support Web site at http www pc ibm com support 1 Click IBM Server Support on the left side of the Web page 2 Click Hints and Tips on the right side of the Web page 3 Click Configuration on the right side of the Web page 4 Click Servers Cable management and labeling for solutions utilizing racks n node Clustering or Fiber Channel for detailed instructions on how to label your IBM Netfinity Cluster IBM Netfinity Cluster Enabler Installing the Hardware This section guides you through installing a six server node configuration in three racks Step 1 Unpack and set up the racks Step 2 Install the memory and processor upgrades into each Netfinity 7000 M10 server Step 3 Install
18. cle Parallel Server is now ready to be configured Installing the Software 27 Troubleshooting IBM Netfinity Cluster Enabler Configuration The following table lists some problems that might occur The first column contains a description of the problem The second column contains a brief explanation of the problem The last column provides a course of action to resolve the problem Problem IBM Core Cluster Service error appears after the first restart Explanation When you first restart a server after installing the IBM Netfinity Cluster Enabler you might get an error message indicating that the services have failed to start This message is not truly an error Once all configuration steps are complete the services will start correctly Action Restart the IBM Core Cluster Service The command Net Start OraclePGMSService fails with a System Error 1067 and the process is terminated Fully qualified domain names FQDN are not supported Rerun IBM Netfinity Cluster Enabler configuration using non fully qualified host names IBM Core Cluster Service appears to hang while in a started state IBM Netfinity Cluster Enabler OSD files cannot be uninstalled When the IBM Core Cluster Service is started from the Windows NT Services menu a set of processes is initiated If any of these processes fail a recovery action occurs that automatically restarts all the processes During the restart period
19. dd to the cluster Note You must verify the IP address used for each server node for the cluster to interconnect 9 Click OK You are returned to the IBM OSD Configuration IBM Netfinity Cluster Enabler window 2 BM OSD Configuration x Cluster D atabase Name OPS Cluster nodes information 0 nodea 203 3 6 101 1 nodeb 203 3 6 102 2 nodec 203 3 6 103 3 noded 203 3 6 104 4 nodee 203 3 6 105 5 nodef 203 3 6 106 4 Add Deste Edt The Cluster name must be the same as the Database name you intend to use 10 Repeat steps 7 on page 24 through 9 until all server node names and IP addresses are entered in the cluster 11 Click OK Installing the Software 25 Note This screen can also be used to Edit the nodes in the cluster list or to Delete items from the cluster list by clicking the list items wanted and then clicking the respective button The IBM Netfinity Cluster Enabler configuration program creates CSCLUSTER CFG and CSCOMPUTER CFG configuration files copies these files to each server listed in the IBM OSD Configuration window and updates the Windows NT Registry for each server The IBM Netfinity Cluster Enabler configuration files are located in subdirectory Your_Installation_Dir CONFIG 26 IBM Netfinity Cluster Enabler Step 11 Set Port Numbers in Windows NT Services File As part of the configuration process for the IBM Netfinity Cluster Enabler software four
20. e Channel PCI Adapter that is installed in each server Connect the other end of the second IBM fibre channel cable to one of the GBICs on the back of the second IBM Netfinity Fibre Channel Hub Connect an IBM fibre channel cable to the media adapter attached on the IBM Netfinity Fibre Channel RAID Controller Unit Refer to the IBM Netfinity Fibre Channel RAID Controller Unit User s Handbook for specific instructions when connecting the cable to the front connector Connect the other end of the IBM fibre channel cable to one of the GBICs on the back of the IBM Netfinity Fibre Channel Hub Step 10 Connect the IBM Netfinity EXP15 Drive Module Cables to the IBM Netfinity Fibre Channel RAID Controller Unit Refer to the BM Netfinity Fibre Channel RAID Controller Unit User s Handbook for detailed cabling instructions for your configuration and then proceed to Step 11 Step 11 Connect the Power Cords to the APC Smart UPS Power Supply Units Refer to APC Smart UPS Uninterruptible Power Supply for power requirements Do not power on the servers at this time Connect the power cords for the IBM Console Selector Switch IBM 8271 712 Ethernet LAN Switch and the IBM Netfinity EXP15 Drive Module to one available APC Smart UPS Installing the Hardware 13 Connect the power cords for the IBM Netfinity Fibre Channel RAID Controller Unit and the display monitor to another available APC Smart UPS power supply Connect the pow
21. el external storage system as the shared hard disk device The software setup consists of the Microsoft Windows NT operating system Oracle Enterprise Edition for Windows NT IBM Netfinity clustering software also know as Operating System Dependent OSD software and IBM Netfinity SYMplicity Storage Manager This book is intended for the following audience e Network administrator the individual responsible for configuring managing maintaining and troubleshooting a network or LAN e Hardware administrator the individual responsible for configuring maintaining and supporting an IBM Netfinity Server e Database administrator the individual responsible for configuring and maintaining the Oracle Parallel Server software and database e Customer project administrator the individual who is the focal point for all IBM communications related to the cluster project How This Book Copyright IBM Corp 1998 Is Organized This book has the following major sections e Introduction on page 1 contains a description of the IBM Netfinity Cluster Enabler e Installing the Hardware on page 5 contains procedures for installing and cabling the hardware in a rack This section also contains information about cable management and cable labeling for the IBM Netfinity Cluster Enabler e Installing the Software on page 15 contains procedures for installing the necessary software to configure the cluster e Tro
22. ep 1 Step 2 Step 3 Validating Device Drivers and Microcode Verify that the server node in the cluster is operating with the appropriate version of the controller microcode adapter microcode system BIOS and device drivers A list of supported versions can be found at the Web site http www pc ibm com support Isolating the Disk Subsystem The following is a list of problems related to the disk subsystem e Cannot recognize the RAID controllers e Cannot recognize shared storage disk drives from the Windows NT Disk Administrator e Errors from IBM Netfinity SYMplicity Storage Manager e Amber lights on hard disk drives e SYMArray messages in the system event log of the Windows NT event log If any of the above conditions or messages occur do the following a Use the recovery steps in the IBM Netfinity SYMplicity Storage Manager to correct the condition or message b Refer to BM Netfinity SYMplicity Storage Manager for Windows NT Installation and User s Handbook for further problem analysis c Contact IBM support If none of the conditions or messages occur continue with step 3 Isolating the IBM Netfinity Server Hardware IBM Netfinity Server hardware problems appear as one of the following e POST error messages e Error messages in the Windows NT event log e Trap errors before you are prompted to log on If any of these conditions or messages occur do the following a Consult the hardware documentation for the component tha
23. er cords from each server to an available APC Smart UPS Power Supply Important To ensure high availability between the clusters distribute the power cords from the servers equally among the UPS units Step 12 Power On the Hardware Be sure to follow the power on sequence outlined in this section to ensure that all hardware units are recognized by the servers and other controller units 1 Ensure that the APC Smart UPS power supplies have charged for 2 5 hours before use Refer to APC Smart UPS Uninterruptible Power Supply for more information 2 Power on each APC Smart UPS power supply by pushing the test button Refer to APC Smart UPS Uninterruptible Power Supply for more information 3 Power on the Ethernet LAN Switch IBM Netfinity Fibre Channel Hubs and Console Selector Switch 4 Power on the IBM Netfinity Fibre Channel RAID Controller Unit and IBM Netfinity EXP15 Drive Modules simultaneously 5 Power on each server one at a time and wait until each server is fully operational before powering on the next server Important a Ensure that the IBM Netfinity Fibre Channel RAID Controller Unit and IBM Netfinity EXP15 drive modules have powered on and started successfully wait approximately one minute after the disk appears to be idle before continuing to the next step b Do not start or stop the servers nodes in rapid succession Wait a minimum of 30 seconds between starting or stopping the servers node
24. f you are setting up a 3rd 4th 5th or 6th server node in the cluster type the following for each node 36 IBM Netfinity Cluster Enabler C gt CD ORACLE_HOME OPS C ORACLE_HOME OPS gt SVRMGR30 SURMGR gt CONNECT INTERNAL ORACLE SVRMGR gt STARTUP SVRMGR gt C_THRn SQL SVRMGR gt C_RBSn SQL where n equals the number of the node 3 Refer to the Oracle Parallel Getting Started Guide and the README TXT file in the C IBMOSD CONFIG for further details to enable the cluster Appendix A Supplementary Instructions for Installing a Six Node Oracle Parallel Server Configuration 37 Appendix B Product Notices References in this publication to IBM products programs or services do not imply that IBM intends to make these available in all countries in which IBM operates Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Subject to IBM s valid intellectual property or other legally protectable rights any functionally equivalent product program or service may be used instead of the IBM product program or service The evaluation and verification of operation in conjunction with other products except those expressly designated by IBM are the responsibility of the user IBM may have patents or pending patent applications covering subject matter in this document The furnishing of this document does not give you any license to these pate
25. finity 7000 M10 servers configured with various hardware components to form a system capable of delivering a single database to a maximum number of clients for high availability and scalability The system can be configured according to the demands of the operating environment whether to maximize the number of clients or to configure the system with storage redundancy to safeguard the information contained within the database O Copyright IBM Corp 1998 1 Hardware Installation Requirements The following items are required to complete a typical two four or six server node hardware configuration However hardware requirements might vary depending on your configuration Hardware Requirements ses as ee IBM Netfinity Racks 2 2 3 3 4 IBM Netfinity 7000 M10 Servers 2 4 6 IBM Console Server Selector Switch 1 1 1 IBM ServeRAID 3L Adapters 1 per server 2 4 6 IBM 10 100 EtherJet PCI Adapters for interconnect only 2 4 6 IBM Console Cable Sets 2 4 6 IBM 8271 712 Ethernet LAN Switch 1 1 1 Hard disk drives for the server 2 per server 4 8 12 IBM Netfinity Fibre Channel RAID Controller Unit 1 1 1 IBM Netfinity Fibre Channel RAID Controller for the IBM Netfinity Fibre Channel RAID Controller Unit IBM Netfinity Fibre Channel Failsafe RAID Controller for the IBM Netfinity Fibre Channel RAID Controller Unit Rack mounts for the IBM Netfinity Fibre Channel RAID Controller Unit IBM Netfinity EXP15 S
26. formation is subject to change without notice Consult your local IBM representative for information on the products services and features available in your area Requests for technical information about IBM products should be made to your IBM reseller or IBM marketing representative Copyright International Business Machines Corporation 1998 All rights reserved Note to U S Government Users Documentation related to restricted rights Use duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp Contents Copyright IBM Corp 1998 About This Book siso sas e 4 34 04 a iv How This Book Is Organized e iv Publications Required for the Installation lt v Hardware Publications v Software Publications L vi INTKOAUCTION assassinos tm a SAE OR AJ O AR AA Aate BGS 1 Hardware Installation Requirements se 2 Software Installation Requirements 3 Tool Requirements 2 00002 ee ee 3 Cable Management and Labeling 4 4 Installing the Hardware 2 2 00 0002s 5 Front and Back View of the Hardware 2 6 Step 1 Unpack and Set Up the Racks L 8 Step 2 Install the Required Upgrades into each IBM Netfinity 7000 M10 Server 8 Step 3 Install the ServeRAID 3L EtherJet PCI and Fibre Channel Adapters into each IBM Netfinity 7000 M10 Server L 8 Step 4 Install the Hardware into Rack Hl L 9 Step 5 Install the Hardw
27. ir is present and its path points to the OSD directory and to the same path on each node If this environment variable is not present or does not show the correct path do not attempt to change the environment variable reinstall the IBM Cluster Enabler software on that node and reconfigure the IBM Cluster Enabler Software on the cluster Validate that the server node IDs and IP addresses are filled in and correct in the CSCLUSTER CFG file Validate that the second line of file CSCOMPUTER CFG identifies the ID number of the server node this file came from Check the Windows NT event log for hardware errors and correct any hardware errors as necessary Restart the server node If you determine that you must reinstall or reconfigure the IBM Cluster Enabler Software follow these steps 1 Stop the above listed four services on all nodes 2 Install the IBM Cluster Enabler Software on the required nodes 3 Configure the IBM Cluster Enabler Software on the required nodes If none of the previous actions corrects the problem do the following 1 Reinstall the IBM Cluster Enabler software Troubleshooting IBM Netfinity Cluster Enabler Configuration 31 2 If the problems are ORA xxxxx or Oracle Database errors continue with step 5 on page 32 Oracle Support 3 Continue with step 5 Step 5 Contacting Support To assist with problem resolution have the following information ready before contacting suppor
28. ler failover therefore it is necessary to install IBM Netfinity SYMplicity Storage Manager on each server node Important IBM Cluster Enabler supports active active mode for the RAID controller Refer to the IBM Networked SYMplicity Storage Manager Installation and User s Handbook for detailed information on how to set up active active mode Install IBM Netfinity SYMplicity Storage Manager on each server node Step 5 Verify and Test Interprocess Communication Connectivity Important This TCP IP connectivity is the EtherJet to Ethernet interconnect Not a LAN connection Before you configure the IBM Netfinity Cluster Enabler files you must verify and test the interconnect connectivity To verify and test the connectivity ping each server by host name and verify the resolution of the IP addresses to the proper host names If your cluster is connected to a LAN verify and test the network connectivity on each server Step 6 Install Oracle Parallel Server OPS Install the Oracle Parallel Server OPS Refer to Oracle Parallel Server Getting Started for Windows NT Handbook for detailed installation instructions Installing the Software 17 Step 7 Install the IBM Netfinity Cluster Enabler OSD Software You must install the IBM Netfinity Cluster Enabler Operating System Dependent OSD software on each server to provide low level cluster functions for the Oracle Parallel Server To install the IBM Netfini
29. mber three next to the EtherJet PCI Adapter 5 Install the IBM Netfinity Fibre Channel PCI Adapter in each IBM Netfinity 7000 M10 server Refer to IBM Netfinity Fibre Channel PCI Adapter Hardware Installation Handbook for detailed installation instructions and then proceed to install the hardware into the first rack The following instructions describe the installation procedures and layout for a typical hardware set up using three racks one ServeRAID Controller Unit and six EXP15 Storage Expansion Units Refer to the IBM Netfinity Rack Configurator Program for assistance in configuring the hardware that you selected Step 4 Install the Hardware into Rack 1 Important Install the APC Smart UPS and the servers in the bottom of the rack because they are heavy 1 Refer to the IBM 9306 Model 900 PC Server Rack Enclosure Caution and Danger Information safety booklet to ensure the safe installation of devices into the rack 2 Install two APC Smart UPS in the bottom of the first rack Refer to the IBM Netfinity Rack Configurator program and BM 9306 Model 900 Netfinity Rack Planning and Installation Guide for detailed installation instructions and then proceed to installing the first server 3 Install the first IBM Netfinity 7000 M10 server into the rack Refer to IBM Netfinity 7000 M10 Server Library for detailed installation instructions and then proceed to installing the keyboard tray 4 Install the keyboard tray i
30. mily click Fibre Channel Solutions 2 Download the device drivers e IBM Netfinity Rack Configurator Program Note The IBM Netfinity Rack Configurator Program might be needed for you to accurately configure your hardware You can obtain the latest version of the Rack Configurator Program on the World Wide Web at http www pc ibm com support Click IBM Server Suppori 2 Under Technical Information click Downloadable files 3 Click Other 4 Click Servers IBM Netfinity Rack Configurator v1 8B 5 Download the executable file to your server e Oracle Enterprise Edition Version 8 0 5 n for Windows NT e IBM Netfinity Cluster Enabler Version 1 1 you can download the software from the World Wide Web at http www pc ibm com us netfinity parallel server html 1 Click Download the Netfinity Cluster Enabler version 1 1 here 2 Complete the Netfinity Cluster Enabler Download Registration form 3 Click Submit Tool Requirements The following tools are required for the installation of this kit e Small flat blade screwdriver e Phillips head screwdriver e 3 16 inch socket wrench and nut driver e 5 mm Allen wrench e 8 mm socket wrench e 9 mm socket wrench e 10 mm socket wrench Introduction 3 Cable Management and Labeling 4 Cable management and labeling are increasingly important Cable management and labeling needs have expanded from the traditional labeling of network connections to labeling most
31. nt View Back View 000000000000000000000000000000000000000 8 fofo o ojo foto o o o ooo foto oro oo oo To o o o oro foto jo ro ooo To o o oo 00 000000 000 0000000 00 0000000 do 0000000 000 0000000 000 000000 olo o a lefjafjefja p INN O JO D JO yoy 0 0 Of 0 0 oy O O o L 6 g 2 A E Ay 8 E 8 B B E 8 B E B 8 B g o U 19 U UU U 9 19 9 U JU edodoijo jo o o o of o o 0 S E o ET o Ko IBM Netfinity 7000 M10 Server APC Smart UPS IBM Netfinity Fibre Channel Hub IBM Console Server Selector Switch 6 IBM Netfinity Cluster Enabler Front View Back View
32. nto the rack Refer to IBM Netfinity Rack Keyboard Tray Installation Instructions for detailed installation instructions and then proceed to install the second server 5 Install the second IBM Netfinity 7000 M10 server into the rack Refer to the BM Netfinity 7000 M10 Server Library for detailed installation instructions and then proceed to install the Ethernet LAN Switch 6 Install the bracket and IBM 8271 712 Ethernet LAN Switch into the rack Refer to the 8271 Ethernet LAN Switch Model 712 User s Guide for detailed installation instructions and then proceed to install the Fibre Channel Hubs Installing the Hardware 9 Note Leave at least 1 U of open air space between the fixed shelf and the IBM Netfinity Fibre Channel Hubs 7 Install a rack fixed shelf over the IBM 8271 712 Ethernet Switch and place the two IBM Netfinity Fibre Channel Hubs side by side on top of the shelf Refer to the BM Netfinity Fixed Shelf Installation Instructions for detailed installation instructions and then proceed to install the monitor 8 Install the monitor compartment and monitor into the rack Refer to the BM Netfinity Rack Monitor Compartment Guide for detailed installation instructions and then proceed to install the Console Selector Switch 9 Install the IBM 9306 Model 900 Netfinity Rack Console Server Selector Switch into the rack Refer to the IBM Netfinity Rack Console Server Selector Switch Installation Instructions for de
33. nts You can send license inquiries in writing to IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504 1785 U S A Licensees of this program who wish to have information about it for the purpose of enabling i the exchange of information between independently created programs and other programs including this one and ii the mutual use of the information which has been exchanged should contact IBM Corporation Department 80D P O Box 12195 3039 Cornwallis Research Triangle Park NC 27709 U S A Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee Any references in this publication to non IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk Trademarks The following terms are trademarks of the IBM Corporation in the United States or other countries EtherJet ServeRAID IBM Netfinity Microsoft Windows and Windows NT are trademarks or registered trademarks of the Microsoft Corporation Other company product and service names may be trademarks or service marks of others 38 Copyright IBM Corp 1998 Part Number 33L3619 Printed in U S A
34. odel 900 PC Server Rack Enclosure Caution and Danger Information safety booklet to ensure the safe installation of devices into the rack Install two APC Smart UPS in the bottom of the third rack Refer to IBM 9306 Model 900 Netfinity Rack Planning and Installation Guide for detailed instructions and then proceed to install the fifth server Install the fifth IBM Netfinity 7000 M10 server into the rack Refer to IBM Netfinity 7000 M10 Server Library for detailed instructions and then proceed to install the sixth server Install the sixth IBM Netfinity 7000 M10 server into the rack Refer to IBM Netfinity 7000 M10 Server Library for detailed instructions and then proceed to install the RAID Controller Unit Install the bracket and the IBM Netfinity Fibre Channel RAID Controller Unit into the rack Refer to IBM Netfinity Fibre Channel RAID Controller Unit User s Handbook for detailed installation instructions and then proceed to install three EXP15 Units Install the brackets and the three IBM Netfinity EXP15 Storage Expansion Units into the rack Refer to IBM Netfinity EXP15 Installation and User s Handbook for detailed instructions and then proceed to Step 7 Installing the Hardware 11 Step 7 Connect the Monitor Keyboard and Mouse Cables to the Console Server Selector Switch k Power Cord Monitor Connector Aux K M fn SE o Information
35. of the partition you created for the heartbeat disk for the IBM Netfinity Cluster Enabler Notes a The symbolic link name must be OPS_CMDISK b The IBM Netfinity Cluster Enabler uses raw disk I O routines to access the OPS_CMDISK drive so the disk drive does not need to be formatted c Do not run the SETLINKS utility until after you have defined symbolic links for the Oracle Parallel Server 3 Run the SETLINKS utility 4 Refer to Oracle Parallel Server Getting Started documentation for information about the SETLINKS utility Step 10 Configure the IBM Netfinity Cluster Enabler OSD Software Important 1 IBM Netfinity Cluster Enabler OSD Software does not support fully qualified domain names FQDN Use NodeA not NodeA X Y Z 2 You must edit the ORALINKn TBL file to add any other nodes when you have determined the architecture of your database 3 When configuring the IBM Netfinity Cluster Enabler OSD Software at least one 22 IBM Netfinity Cluster Enabler free drive letter must exist on the server node from which the IBMGSCFG EXE configuration utility is run Perform the following steps only once since the data is propagated to all servers in the configuration To configure the IBM Netfinity Cluster Enabler do the following 1 Ensure that all servers are up and running and network connectivity has been verified 2 To ensure that proper network connectivity has been established perform the f
36. ollowing a Ping each server by host name and verify the resolution of the IP addresses to the host names b Click Start Settings Control Panel c Click Services and scroll down the selection list until you see Server under the Service heading d Under the Startup heading the category should read Automatic e Close the Services window 3 Click Start Run 4 In the Open box type Your_Installation_Dir BIN IBMGSCFG EXE 5 Click OK The IBM OSD Configuration IBM Netfinity Cluster Enabler window appears 2 BM OSD Configuration x Cluster Database Name OPS Cluster nodes information m Add Delete Edit The Cluster name must be the same as the Database name you intend to use 6 Type the OPS cluster or database name in the this panel This is the name of the database you will use when configuring the OPS For the Oracle Parallel Server the cluster name and the database name are the same The default Installing the Software 23 name as it appears on this panel is OPS Make sure that you use the same database name when configuring the Oracle Parallel Server 7 Click Add 24 IBM Netfinity Cluster Enabler The Editing cluster node information screen appears Editing cluster node information Ea Enter Mode Mame Computer ID jo Mode name rodea IF address 203 3 6 1 01 cancel 8 Type the server node name and the IP address for the server you want to a
37. or for detailed installation instructions and then proceed with Step 3 Step 3 Install the ServeRAID 3L EtherJet PCI and Fibre Channel Adapters into each IBM Netfinity 7000 M10 Server 1 Refer to the BM Netfinity 7000 M10 Server Library documentation Legal and Safety Information section for safety concerns while installing these options 2 Install a ServeRAID 3L Adapter that will control your startup drives in slot 1 of each IBM Netfinity 7000 M10 server Refer to IBM Netfinity ServeRAID 3H and ServeRAID 3L Ultra2 SCSI Adapters Installation and User s Guide for detailed installation instructions and then proceed with cabling the adapter 3 Cable the ServeRAID 3L Adapter in each IBM Netfinity 7000 M10 server Refer to the Working with Adapters section in the BM Netfinity 7000 M10 Server Library for detailed installation instructions and then proceed to install the EtherJet Adapter 4 Install the EtherJet PCI Adapter in each IBM Netfinity 7000 M10 server install the EtherJet PCI Adapter in PCI slot number two of each server 8 IBM Netfinity Cluster Enabler Refer to the documentation that comes with the IBM 10 100 EtherJet PCI Adapter for detailed installation instructions and then proceed to install the Fibre Channel Adapter Important a Install the EtherJet PCI Adapter card for the interconnect in PCI slot number two in each server b If you add a LAN network card to each server install it in PCI slot nu
38. racle program files reside on nonshared storage on each server Create a RAID Level 1 array and a logical drive in each server using the ServeRAID configuration utility Refer to the IBM Netfinity ServeRAID 3H and ServeRAID 3L Ultra2 SCSI Adapters Installation and User s Guide that comes with the adapter for detailed installation instructions and then proceed to Step 2 Step 2 Install Microsoft Windows NT Server Version 4 0 Enterprise Edition Install Microsoft Windows NT Server Version 4 0 Enterprise Edition with Service Pack 4 on each server Refer to the documentation that comes with Windows NT Version 4 0 for detailed installation instructions and then proceed to Step 3 Step 3 Install the IBM Netfinity Fibre Channel Adapter Device Drivers 1 Install the IBM Netfinity Fibre Channel Adapter device drivers in each server Refer to the documentation that comes with the IBM Netfinity Fibre Channel Device Drivers for detailed installation instructions and then proceed to add the driver Click Start Settings Control Panel Click SCSI Adapters Click Drivers Add the Fibre Channel Device Drivers When Prompted shut down and restart the server O oa E ON 16 IBM Netfinity Cluster Enabler Step 4 Install IBM Netfinity SYMplicity Storage Manager IBM Netfinity SYMplicity Storage Manager configures monitors and controls the state of the RAID controllers and disk arrays It also provides RAID control
39. re the IBM Netfinity Cluster Enabler heartbeat disk must be configured as a logical drive on a shared disk partition 1 Shutdown all servers except one and then create your logical unit under the IBM Netfinity SYMplicity Storage Manager Refer to the BM Netfinity Storage Manager User s Handbook and Oracle Parallel Server Getting Started documentation for more information and then proceed to powering on the servers Installing the Software 21 2 Power on all servers one at a time Allow each server to fully start before powering on another server Note Record the disk and partition numbers created in Step 8 Create a Shared Partition for the IBM Cluster Enabler Heartbeat Disk on page 21 and used by the OPS for later reference during the Oracle software installation Refer to the Performing the Pre Installation Tasks section in the Oracle Parallel Server Getting Started documentation for detailed instructions for creating the shared disk partition and setting up symbolic links Step 9 Assign Symbolic Links for the IBM Netfinity Cluster Enabler Heartbeat Disk To assign a symbolic link for the heartbeat disk that was created in Step 8 1 First create the ORALINKn TBL file where n is the number of nodes in the cluster 2 Edit the ORALINKn TBL file to add the following OPS CMDISK Device HarddiskxX Partition Y where OPS CMDISK is the symbolic link name X is the physical drive number Y is the partition number
40. s in a cluster 14 IBM Netfinity Cluster Enabler Installing the Software This section guides you through the tasks for installing the software Step 1 Configure the ServeRAID Adapters on all servers Step 2 Install Microsoft Windows NT Server Enterprise Edition with Service Pack 4 on all servers Step 3 Install the device drivers and verify firmware levels for the Fibre Channel adapters on all servers Step 4 Install the IBM Netfinity SYMplicity Storage Manager on all servers Step 5 Test and verify connectivity of the interconnect on all servers Step 6 Install Oracle Parallel Server OPS Ver 8 0 5 n on all servers Step 7 Install the IBM Netfinity Cluster Enabler files OSD software Ver 1 1 on all servers Step 8 Create a 2MB extended partition on the shared storage for the Heartbeat Disk perform only once Step 9 Assign symbolic links for the IBM Netfinity Cluster Enabler Heartbeat Disk on all servers Step 10 Configure the IBM Netfinity Cluster Enabler perform only once Step 11 Set port numbers in Microsoft Windows NT Services file on all servers Step 12 U Your cluster is now ready for Oracle configuration XA Installing the Hardware 15 Step 1 Configure the ServeRAID 3L Adapter Note Oracle Parallel Server OPS requires that the operating system and the O
41. t e IBM Support a Machine type model and serial numbers of the servers in the cluster b Operating system service pack and IBM Cluster Enabler OSD software level BIOS microcode and device driver levels for the server and IBM Netfinity EXP15 Fibre Channel disk subsystem and all adapter cards Configuration of IBM Netfinity EXP15 Fibre Channel disk subsystem Number of servers nodes in the cluster showing similar symptoms Oracle version and service patch level Name and version of IBM Netfinity SYMplicity Storage Manager software h Model number of RAID Controller Unit and serial number on the IBM Netfinity EXP15 Fibre Channel Drive Module e Oracle Support a o O Oracle customer number Operating system version number Oracle DataBase version number ORA xxxxx error information if any Error or log file information resulting from the problem Notes 1 The Oracle Guide to Customer Support publication included with your Oracle software will provide the most current information on customer support 2 Refer to the Oracle web site at http www oracle com support Step 6 Advanced Software Problem Isolation 32 This section describes steps to isolate problems where a service appears to start correctly and then aborts shortly thereafter or where a service appears to start correctly and then hangs e IBMCoreClusterService a From the Windows NT Task Manager click Processes
42. t appears to be failing b Check the IBM online technical tips at http www pc ibm com support c Contact IBM support Note Search for text relating to the component that appears to be failing 30 IBM Netfinity Cluster Enabler Step 4 Isolating Start Up and Software Errors Check the following items in the order shown a Validate that the critical Windows NT services are running to ensure the OPS solution is functioning Close and restart Windows NT services if it is already open Validate that the following four services are present and in a started state IBMCoreClusterService OracleTNSListener80 OraclePGMSService OracleServiceS D where the number digit in the S D is one number higher than the node number If any of the above services are not in a running state then restart that service and all subsequent services in the order shown in the list above Note Each service might take from 1 to 2 minutes to start Validate that all servers nodes are using the same port number assigned in Step 11 Set Port Numbers in Windows NT Services File on page 27 Verify that the following three files are located in your_installation_di IBMOSD CONFIG If any of these files are not present reinstall the OSD software on that specific server node and reconfigure the OSD software on that cluster CSMETA CFG CSCLUSTER CFG CSCOMPUTER CFG Verify that the environment variable CS Install D
43. tailed installation instructions and then proceed to Step 5 Step 5 Install the Hardware into Rack 2 Important Install the APC Smart UPS and the servers in the bottom of the rack because they are heavy 1 Refer to the BM 9306 Model 900 PC Server Rack Enclosure Caution and Danger Information safety booklet to ensure the safe installation of devices into the rack 2 Install two APC Smart UPS in the bottom of the second rack Refer to the IBM 9306 Model 900 Netfinity Rack Planning and Installation Guide for detailed instructions and then proceed to install the third server 3 Install the third IBM Netfinity 7000 M10 server into the rack Refer to the BM Netfinity 7000 M10 Server Library for detailed installation instructions and then proceed to install the fourth server 4 Install the fourth IBM Netfinity 7000 M10 server into the rack Refer to the BM Netfinity 7000 M10 Server Library for detailed installation instructions and then proceed to install three EXP15 Storage Units 5 Install the brackets and the three IBM Netfinity EXP15 Storage Expansion Units into the rack Refer to IBM Netfinity EXP15 Installation and User s Handbook for detailed installation instructions and then proceed to Step 6 Step 6 Install the Hardware into Rack 3 Important Install the APC Smart UPS and the servers in the bottom of the rack because they are heavy 10 IBM Netfinity Cluster Enabler Refer to the IBM 9306 M
44. the monitor compartment in a rack IBM Netfinity ServeRAID 3H and ServeRAID 3L Ultra SCSI Adapters Installation and User s Guide This book contains information and detailed instructions for using the IBM ServeRAID 3H and the IBM ServeRAID 3L Adapters IBM Netfinity Cluster Enabler IBM PC Server 10 100 EtherJet Adapter User s Manual This book contains detailed instructions for installing the IBM 10 100 PCI EtherJet adapter in the server Software Publications IBM Netfinity Networked SYMplicity Storage Manager Installation and User s Handbook This book contains installation and setup information for the SYMplicity Storage Manager software in a networked Microsoft Windows NT environment IBM Netfinity SYMplicity Storage Manager for Windows NT Installation and User s Handbook This book contains detailed instructions for installing the SYMplicity Storage Manager software with the Windows NT operating system IBM Netfinity SYMplicity Storage Manager User s Handbook This book contains complete instructions for installing the SYMplicity Storage Manager software to perform various storage management tasks Microsoft Windows NT Installation and User s Guide This book contains instructions for installing and using Microsoft Windows NT Oracle Parallel Server Getting Started for Windows NT Version 8 0 5 1 This book contains operating system specific information on using Oracle Parallel Server for Windows NT Oracle Parallel Server Ge
45. tion to and are supplementary to the instructions in the Oracle Parallel Server Getting Started Guide The following files and additional information provide instructions on how to set up a six node Oracle Parallel Server configuration The files are not provided by Oracle on the Oracle8 Enterprise Edition for Windows NT Cd You can find the files needed to complete this task in the Your Installation DiNCONFIG directory provided with the IBM Cluster Enabler OSD software Copy the following files to the ORACLE HOME OPS directory from the Your Installation DiNCONFIG directory e Database initialization files INITOPS5 ORA INITOPS6 0RA e SETLINKS input files ORALINK4 TBL ORALINKS TBL e Database creation files C RBS5 SOL C THR5 SOL C RBS6 SOL C THR6 SOL Step 1 Assign Symbolic Links to Each Logical Drive If you are setting up a sixth node in the cluster input files are created for SETLINKS that contain information for each node These files are ORALINK4 TBL for fifth node ORALINKS TBL for sixth node 1 Ensure that you have copied the ORALINK4 TBL and ORALINKS TBL files to the ORACLE_HOME OPS directory from the Your_Installation_DiNCONFIG directory to all servers 2 Edit each ORALINKn TBL file where n is one less than the number of the node 3 Refer to the Oracle Parallel Server Getting Started Guide for instructions regarding updating each ORALINKn TBL file and then proceed to run the SETLINKS application 4
46. torage Expansion Units 1 6 1 6 1 6 Hard disk drives for the IBM Netfinity EXP15 Storage Expansion Unit per expansion unit Rack mounts for the IBM Netfinity EXP15 Storage Expansion Unit 1 per EXP15 18 i ne IBM Netfinity Fibre Channel Hub 2 2 2 IBM Netfinity Fibre Channel PCI Adapters 2 per server 4 8 12 Microprocessor Upgrades 3 per server 6 12 18 1 GB Memory Option Kit 1 per server refer to the documentation that comes with your server for memory 2 4 6 configuration APC Smart UPS Model 3000 2 4 6 Standard Category 5 Ethernet cables with RJ45 2 4 6 connectors Shortwave Fibre Optic cables 6 10 14 Fibre Optic Gigabit Interface Converters each IBM Netfinity Fibre Channel Hub comes with 4 GBICs table 6 10 14 lists total needed Display monitor keyboard and mouse one of each 1 1 1 Monitor compartment for the IBM Netfinity Enclosure 1 1 1 Fixed shelf for the IBM Netfinity Rack 1 1 1 Keyboard tray for the IBM Netfinity Enclosure 1 1 1 2 IBM Netfinity Cluster Enabler Software Installation Requirements The following software is required to complete the IBM Netfinity Cluster Enabler e Microsoft Windows NT Server Version 4 0 Enterprise Edition with Service Pack 4 e IBM Netfinity Fibre Channel RAID Support Version 6 2 or later You can obtain the latest Fibre Channel device drivers on the World Wide Web at http www pc ibm com support 1 Under select fa
47. tting Started Guide This book contains instructions for setting up servers nodes in a cluster About This Book vii viii IBM Netfinity Cluster Enabler Introduction Welcome to the IBM Netfinity Cluster Enabler Hardware and Software Installation Guide for Oracle Parallel Server This book will provide a high level overview of the process of assembling the IBM Netfinity Cluster Server hardware and installing your software for the proper operation of the Oracle Parallel Server OPS Version 8 0 5 n The IBM Netfinity Cluster Enabler has been certified by Oracle to fulfill the Oracle Parallel Server OPS requirements for vendor supplied Operating System Dependent OSD software Interconnect Ethernet Hub r N W yt N N p Server node Server node Server node Server node Server node Server node IBM Netfinity IBM Netfinity IBM Netfinity IBM Netfinity IBM Netfinity IBM Netfinity 7000 M10 7000 M10 7000 M10 7000 M10 7000 M10 7000 M10 RE Aa Ra Fibre Channel IBM Netfinity Fibre Channel RAID Controller Unit and Hubs N ar A K X Sa na Saa JA r N Sa a A K a Data Data Data Data Data Data IBM Netfinity IBM Netfinity IBM Netfinity IBM Netfinity IBM Netfinity IBM Netfinity EXP 15 EXP 15 EXP 15 EXP 15 EXP 15 EXP 15 Six Node Cluster Configuration The IBM Netfinity Cluster Enabler hardware consists of two four or six IBM Net
48. ty Cluster Enabler OSD do the following 1 Copy the IBM Netfinity Cluster Enabler OSD Software to each server that was downloaded from the World Wide Web in Software Installation Requirements on page 3 2 Run SETUP a Click Start Run b In the Open box type the full directory path and NCES EXE c Click OK A Welcome screen appears 3 Click Next 18 IBM Netfinity Cluster Enabler Information Make sure you follow the instructions Before the installation of IBM OSD Components stop IBMCoreClusterS ervice first For all cluster nodes use the same full directory path to install IBM OSD components If you are installing the IBM Netfinity Cluster Enabler software for the first time continue with step 4 If you are not installing the IBM Netfinity Cluster Enabler software for the first time continue here Note You must stop the IBM Core Cluster Service if you have installed the IBM Netfinity Cluster Enabler software and are running its services a Click Start Settings Control Panel to stop the service b Double click Services c Highlight IBMCoreClusterService then click Stop d Close the Services window 4 Click Next Installing the Software 19 20 5 Select a Destination Folder to install the IBM Netfinity Cluster Enabler software Choose Destination Location a To accept the default path click Next b To select a different directory click Browse and select the
49. ubleshooting IBM Netfinity Cluster Enabler Configuration on page 28 contains information on how to isolate and recover from some potential problems that might occur e Getting Help Service and Information on page 34 contains instructions on how and where to obtain service e Appendix A Supplementary Instructions for Installing a Six Node Oracle Parallel Server Configuration on page 35 contains instructions on how to set up a sixth node in the cluster e Appendix B Product Notices on page 38 contains notices and trademarks Publications Required for the Installation During the installation of the IBM Netfinity Cluster Enabler you might need to refer to one or more of the following publications Hardware Publications vi 8271 Ethernet LAN Switch Model 712 User s Guide This book contains information for installing and configuring an 8271 712 switch APC Smart UPS Uninterruptible Power Supply Model 1400 3000 This book contains information that can help you optimize the performance and service life from your UPS it also contains a description of the inner workings of the UPS IBM Netfinity Rack Console Server Selector Switch Installation Instructions This book contains detailed instructions for installing and using the selector switch IBM 9306 Model 900 Netfinity Rack Enclosure Unpacking Instruction This book contains detailed instructions for unpacking the PC Server Rack Enclosure IBM 9306 Model
50. ully complete if other Oracle services are running The IBM Core Cluster Service and other Oracle services are interdependent due to timing considerations If you stop IBM Core Cluster Service while OraclePGMSService and OracleServiceOPSn are running you will be prompted to confirm that you want to stop these other services The Oracle services might be stopped but the IBM Core Cluster Service might not When the Oracle services have stopped issue Net Stop IBMCoreClusterService a second time Or perform one of the following Stop each service in the correct order 1 OracleServiceOPSn 2 OraclePGMSService 3 IBMCoreClusterService Use the Windows NT Services window to stop IBMCoreClusterService Troubleshooting IBM Netfinity Cluster Enabler Configuration 29 Isolating Problems The following section is designed to help you isolate problems if they occur To expedite problem resolution use the following problem determination sequence It can help you resolve problems and help identify to whom you should go for support These steps can assist you with the isolation and resolution of problems with the hardware the IBM Cluster Enabler OSD software and the Oracle Parallel Server OPS software The error you receive might be application generated or application related If you determine that an error is application generated or application related contact the appropriate vendor for that application St
Download Pdf Manuals
Related Search