Home

working draft

1. Figure 94 Screenshot Example search for Oracle update After locating the patch number you proceed to the Patches amp Updates tab and select a simple search Enter the patch number that you found in the table and hit Go see Figure 95 146 SSS Figure 95 Screenshot Oracle Patches amp Updates Simple Search Window That will bring you to the actual download screen Figure 96 147 Figure 96 Screenshot Oracle Patches amp Updates Patch 7631956 Download Window You should be sure to view the readme file It will tell you which version of OPatch is required to install this patch and other prerequisites It will also give you step by step instructions for installing the patch 148 APPENDIX ORACLE INSTALLATION Oracle installation consists of two elements installation of the Oracle software and the creation of the instances Installation of Oracle to a new version occurs every 3 to 5 years as Oracle support expires for named version or hardware is replaced Creation of new instances typically occurs by cloning Creation of special purpose instances or the original instance of a new version occurs much less frequently This section documents the installation process for version 12c and the process of creating an instance with tools availabl
2. wee SE ven le T umi os IE a m id Ecc B igneus EX EDIT Se ee Figure 113 Screenshot Bi weekly tape rotation 2015 update QUARTERLY TAPE ROTATION SCHEDULE The quarterly tape rotation schedule includes AIMS and archival backups These backups are done the first week of February May August and November These backups require at least two 500GB tapes and two days to complete since verification is included in the process The folders backed up include Insert tape rotation calendar here Figure 114 Screenshot Quarterly tape rotation schedule INPUTS TO BACKUP POLICIES This material is derived from the FHWA July 2009 backup procedures document This system does not include indefinite archives Indefinite archives are covered in the annual data submission and the NARA submission The frequency of backup to cartridge by server drive has been established based on drive usage The inclusion of incremental backups for drives backed up weekly to cartridge was initiated when additional external storage was installed Table 21 Database server frequencies Drive Partition Name Total GB Update Cycle Backup Letter Frequency C 08 20 Continuous Weekly D Working 170 Continuous Tncremental 8T Storage Week AMS TIU
3. Figure 46 Screenshot Folders on server for storing backup reports The folder Backup has three sub folders Backup logs IDR and Offsite container IDR Intelligent Disaster Recovery is a folder used by BE Offsite_container contains files on the original system for storing cartridges off site It will be removed January 2011 after all cartridges have cycled into the new backup and container schedules Backup logs holds all the copies of the backup job set ups histories and logs 80 There are six sub folders under Backup logs 0_Incremental_db 0 Incremental traffic 0 Incremental working 1 Monthly 2 SDR 3 Weekly 3 Periodic 0 Incremental db incremental backup of the LTPP Oracle databases G 0 Incremental traffic incremental backup of the traffic drive 0 Incremental working incremental backup of the working storage drive D 1 Monthly full backups that are stored off site for either 13 weeks or 2 years 2 SDR backups of the most recent SDR instance 3 Weekly full backups that are retained on site 3 Periodic Quarterly backups folder contains three types of pdfs job setups job histories and job logs Job setups are the final step of the process in creating a job The 0_ folders have a Last week subfolder to store prior weeks pdfs Job histories and Job logs are created as follows Select the Job Monitor tab Select a job in the Job Histor
4. 168 169 ommum meum 15 ey ee tie ck Crate o creat ee To da fs ae pcs Bom in ep fe ape cage vag Cem Se ole oe 170 APPENDIX REMOTE ACCESS Remote access to the central server from outside TFHRC is managed by FHWA IT at TFHRC over a VPN Individuals holding DOT credentials are eligible to use the system Individuals needing remote access must also have accounts on the LTPP servers The System Administrator and Backup System Administrator identified in Appendix A Roles and Responsibilities TFHRC Server have remote access privileges The TFHRC Help Desk also has remote access privileges through the internal TFHRC network to the Remote access is also used to work on the servers while at TFHRC Due to noise concerns limited time is actually spent in the server room The TFHRC servers have multiple network cards One card on each server is connected to a HP Procurve switch for use on an internal network A non COE machine located outside the server room is also connected to this internal network The Dell 2900 is at 192 168 2 25 The Dell R515 is at 192 168 2 26 The non COE machine is at 192 168 2 37 Additional connections are available for LTPP FHWA staff but are not currently in use m APPENDIX
5. prr puo acere oe Dit sere n prr EPA ta sar Ca Ced Cl Update SQL Developer trom Help oL 139 APPENDIX N SOFTWARE APPLICATIONS The server was delivered with the Windows Server 2008 R2 Operating System This was then configured to comply with NIST 800 53 Revision 2 Annex 1 since this is a low impact system After configuration anti virus software was installed The applications installed were Oracle 12c Oracle 11g Client Symantec Backup and various utilities Once Oracle was installed the production database instance IMSProd was created Then IMSTest and IMSDev were cloned The following is a list of software on the central server Software Renewal Date Renewed by Symantec Backup 2015 Tan 2017 FHWA maintenance Oracle 12 NIA SQLDeveloper 4x Freeware Winzip 19 Tndefinite DOT Adobe Acrobat 7 N A DOT TextPad Shareware Notepad Freeware PowerDesk 9 Tndefinite LTPP Team Leader and COR have licensing information On at least a monthly basis the server should be connected to the Internet for operating system and anti virus software updates Use the Windows Update service to keep the operating system current on patches Use Symantec Live Update for updates to the anti virus and back up software Patching of the Oracle software is done in coordination with patches to the DPW OPER
6. ROLE Organization with Individuals in Role ADEP Power User AMEC ESCINC Fugro NCE SAIC Stantec PPDB Forms Edit AMEC ESCINC Fugro NCE SAIC Stantec LTAS Fugro NCE SAIC Stantec SSH Users Fugro NCE SAIC Stantec Technical Support Services Contractor SAIC Information System Owner FHWA Customer Service ESCING Authorizing Official FHWA Other Designated Contacts SAIC Assignment of Security Responsibility SAIC FHWA See LTPP Team listing http www fhwa dot gov research tfhro programs infrastructure pavements Itpp whoswh Table 12 DPW Organizational Contact Information Organization Contact Information Long Term Pavement Program Team FHWA DOT HRDI 30 LTPP 6300 Georgetown Pike McLean VA 22101 SAIC TSI Lafayette Drive Oak Ridge TN 37830 AMEC Environment amp Infrastructure 12000 Indian Creek Court Suite F AMEC Beltsville MD 20705 301 210 5105 175 Organization Contact Information Fugro Consultants Inc Fugro 8613 Cross Park Drive Austin Texas 78754 512 977 1800 NCE 1885 5 Arlington Ave Suite Reno NV 89509 775 329 4955 1000 Young Street Suite 470 Tonawanda NY 14150 716 632 0804 Engineering amp Software Consultants Inc 14123 Robert Paris Court ESCINC Chantilly VA 20151 176 APPENDIX R CONTINUITY OF OPERATIONS DPW PURPOSE The Continu
7. ecovey neds this wizard fnds the rieligert Disaster Recovery action pine aorar is ooo or select oder edie aaar on weich tro IOR osten taled You or eae covey medi recorder sever 1 Figure 53 Screenshot IDR Preparation Wizard Opening Sereen 87 IDR Preparation very Preparation wizard Create IDR Boot Media este new boot media that wil be usod case rover Soc the of macta you to preparo aure that the cordato be elected can be ates er We type ef bool meds sco ca ik Ces SO Rotate ae nage or wihi hotare lepe Ce Norcerabie desserreccvey CD inane copy Disterecoverifomatin dies lt Net Cane Figure 54 Screenshot IDR Boot Media Options ry Preparation writable CD Starting CD Image Creation The waard guides you trough crear a bootable CD mage cen be used wth he rot backup c recover compe ie werden P Windows CD CD Rr COM bum the bocble CD mace Betere umma ts an un a ful aca ofthe computer beng pected Yeu mat bum the CD image fo CD using id cer software To bum a CD you mst have isle CD deuis 85 Figure 55 Screenshot IDR CD Creation Instructions ntetigent osaster Recovery Preparation Wizard
8. de Figure 92 That will bring you to the actual download screen see Figure 93 shot Oracle Patches amp Updates Simple Search Window 144 3 Fete Om Od 0 Sie OC s 2 MB e cr aon aro teret ey ORACLE IE IE YES Trata me trata rre pen inti ferien res enon EI EE cere pes Figure 93 Screenshot Oracle Patches amp Updates Patch 8559467 Download Window Be sure to view the readme file It will tell which version of OPatch is required to install this patch and other prerequisites It will also give step by step instructions for installing the patch Oracle patches can be downloaded from My Oracle Support formerly Metalink The general procedure is to perform a knowledge base search for the update that you are interested For example you could search for Critical Patch Update CPU July 2009 Oracle Products In the Patch Availability document search for the table listing the critical patch update availability for Oracle Database Then find the patch number for version 10 2 0 3 on Windows 32 Bit In the example below Figure 94 you will notice that the patch is listed as N A This is because support for 10 2 0 3 ended on February 22 145 2009 If you go back to the January 2009 Critical Patch Update you will see that patch 7631965 is the last CPU available
9. EL aT a Eom Volume H USB Drive pem OE Volume Full ADEP Repository oer Backup 4 RD1000 EXT Cartridges Stored OfFSite pe a Figure 97 Schematic Backup process overview Drive Backup Strategy The RD1000 mounted on drive J is used to write backups for off site storage The system uses 500GB cartridges for working off site backups 1 TB cartridges are used for backup of AIMS material A 1 TB WD1000 external hard drive mounted on Drive M is used for incremental backups 179 There are multiple types of backups taking place First there are the drive level backups Symantec Backup Exec System Recovery 2010 is used to back up drive C to the staging area H SymantecRecoveryPoints The backup of drive C starts with a full image backup on Sunday at 10 30 PM with differential backups created at 10 30 PM Monday through Saturday The staging area maintains two full image differential backup sets The Backup Exec software also takes care of copying the current week s backup files to drive J to be moved off site This copy occurs as the full or differential backup completes The backup of drive D is handled differently since the backup media is not large enough to contain an image backup Drive D is backed up on a file by file basis to H SymantecRecoveryPoints LTPP
10. Sable nace for uee War 50 9060 tete nh mapa cose Nextel denter emen CD rage Cony sone meow oration fen Figure 157 Sereenshi Selecting the Disaster Recovery File Option in the IDR Preparation Wizard capy Ta cm m m Figure 158 Screenshot Identifying computer and location for dr File 267 Disaster Recovery formaton Fes Copy smien teret Beade merere Hes ene le eo covey ema fae Ub Bac d gt kien cm uem eme Figure 159 Screenshot Completion of creation of copy of dr file Disaster recovery media updated Ya ame non may recovery the evert aman corper dente e win ach cun Fatedealysovethecorpusrapcic dade eco Cony Grane rematan SS Gee Figure 160 Screenshot Completion of disaster recovery preparation RESTORING A FILE 268 APPENDIX AL ORACLE OPERATIONS DELL 2900 Basic Oracle syntax does not vary in a material fashion between the two servers The primary differences between the two TFHRC servers are file locations for Oracle databases and software This section discusses the Oracle 11g database actions on the Dell 2900 The Dell 2900 has SQLDeveloper installed When the central server s version is updated copy is made to the Dell
11. ori Golo Fi Ria rie c ee ener eren stent orta Vy ar reget daaar mates compu a DR opten fous 1 mde Figure 148 Screenshot IDR Preparation Wizard Opening Screen IDR Preparation 262 Create IDR Boot teow mods hat s dior ec co the npe oi mede yeu wart to separ Make aure ht the coepere a ica mods out able ace free w h CO Wea 50 65 222100 p 54 Disaster recovery reason yen Exp Figure 149 Screenshot IDR Boot Media Options Starting CD Image Creation Dre wd aie ou br amati CD nase lerares covera conous Tre zeae Aiae COSCE RI feku eee CO Tam Beers vr rr enc te coroner ous burte CD irage to a CD a Does Ta man a CD ya merd hane a tete Fae CD dos e Figure 150 Screenshot IDR CD Creation Instructions Select Computers to Protect Si rtr rane ho boe mada a bang juste Sees anode tos ee Figure 151 Screenshot Selecting a computer for disaster recovery preparation Select Location tor CD An 1S0 mage tia wl sso artera path who tha CO mane bs Hen CD Wc rolar tt image be or the reme
12. Figure 90 Screenshot Example search for OS updates Each patch that has not been applied is downloaded to a flash drive or other portable media The media containing the patches is then mounted on the server and the patches are applied while logged in as an administrator ORACLE Oracle patches can be found using My Oracle Support https support oracle com Oracle patches can be downloaded itpi wsvw oracle com technetworkltopicsfsecurity alers 08686L html from My Oracle Support formerly Metalink The general procedure is to perform a knowledge base search for the update that you are interested For example you could search for Critical Patch Update July 2009 Oracle Products In the Patch Availability document search for the table listing the critical patch update availability for Oracle Database Then find the patch number for version 102 04 on Windows x86 64 In the example below this is patch 8559467 see Figure 91 htp f eww oracle com technetworkltopics security alerts 86861 html 142 Figure 91 Screenshot Example search for Oracle update After locating the patch number you proceed to the Patches amp Updates tab and select a simple search Enter the patch number that you found in the table and hit Go see Figure 92 143 poa Single Search EI c nd 0 4 Man bem
13. tsar on Orc Na iter Eiter me rame afe tene yeu nario cea Lustenername mmn race Hors mstatea wi a Crac ome User ora Tae Windows saraca for tite wil on run unde ie Orsi Hore User actor Ooo 4 ome en 2 nv me protects you nesa You can congue hclner fo connecons rone or poco Sua which you want congu or e Vener Keep your as as posse by Sch TOPP should stanar uss Th port ural selecta nat be used ay omer soon mis enous Use me standard potnumner 521 C Useanonerpennumeer ET outa vou ma to corngure anamer stener stoner contin CEES 158 Wacom to tye rad Nat Configuration Assistant Tris taat tak aig me common arguo steps Ito teow T contguston T toca nat sarace Name T Disce Usage Conkgueon ses Due gt Helme to ha Orc Na Corihguration Assistant ie tia you rouge coran ster hte etc Chiese e eu wou o de F Listener comnaurazon connouranon E Loca niat Saraca renom usage Tu access an Case database or aher sere aces ulus a nei rare na Ora He Connguranon assistant lis ouo work wih n
14. Run LoudUpload emd in a DOS window using the following command LoadUpload dbaUN dbaPW database upload date where upload date is the official upload date in yyyymmdd format 2 Review the import lg file Compare the import files to the related export files Any discrepancies must be investigated Run FinishUploadProcess in a DOS window using the following command FinishUploadProcess dbaUN dbaPW G database upload date where upload date is the official upload date in yyyymmdd format 121 Review the output file for errors Scroll through the entire output file and verify that all constraints were enabled successfully Table altered Instance Review Database synchronization and cloning is dependent on correct assignment of Oracle objects to schemas to reduce loss of work in progress or duplication of object names There are basically three classes of users in the LTPP instances operational users data release users and working users Operational users are LTPPDBA and the TREDBA These are the two schemas under which all of the tables associated with basic LTPP data operations are created The tables and materialized for these schemas are typically listed in the LTPPTD table or the SDR_Table_Inclusion_List Data release users include CUSTSUPP and IPC The tables associated with these schemas are not separately catalogued in LTPP created tables Working users consist of all the other named users with tab
15. Data Update via Scripts modifi not exist for table modifications jons to data using SQL scripts when forms do LTAS data or software issues for the LTAS application PPDB QC programs loaders LDEP forms data entry issues Directive I 155 or the most recent version of it discusses how SPR information is entered for the PPDB and LTAS and the priorities assigned Software Change Notice SCN Report Deployment of software updates is documented in Redmine in licu of the previous Software Change Notice report The changes deployed can be found by filtering the records in Redmine oftware Deployment DPW Deployment of changes to the PPDB forms and utilities takes place on a monthly basis Changes which are critical to continuous regional data entry operations may be deployed between the monthly updates Deployment of LTAS changes is done on an as needed basis Major changes to functionality are deployed after regional testing but typically no more than twice a year Action Items Most LTPP team meetings and many teleconferences result in action items for follow up These items are tracked using Redmine under Issues SPRs and the Operations project The Action Items subproject contains the tracking mechanism for these items 4 RECURRING SERVER ACTIVITIES Recurring server activities consist of backups off site data storage software updates and system security Bi weekly backups of electronic
16. Select Computers to Protect eect carer names er which he bie berg cated The sontes m ctberunna the same veion ofthe Winsous operate Stem Aralatie Compton Compton Computer Nene Syst Compete Name Topsang MROLIPPSERVER Windows Sem 1 FOER sed Figure 56 Screenshot Selecting a Computer for Disaster Recovery Preparation Replace new server name Wizard Soloct Location for CD imaga ho bocas mage urti you bum a C Jn ISO 99600 mace fle wil be reed Pesos erter a path where this CD irage be oed es CO sonare products ei the rege to be onthe same corer ts revert Undown ho bm Arecormerded locaton wel be a shar onthe computer nhers the CD Vier rated tera path tere CD des one 89 Figure 57 Screenshot Location Selection for CD Image Preparation Vizard a Path to Windows Operating System Installation Fico Tre mage Tho Yirdeme cporang eor on tho ination pd muat match tre verson and A ranged the conte berg preci Browse ortinethe paths the keaton the Windows operating system Creating Disaster Recovery Image 1 re boil baaga ie The nizan cresting the bootable ima
17. System Administrator Information Systems Owner FHWAIT Delegation of Authority Essential Equipment and Systems PowerEdge R515 TFHRC Server 2 MD1200 40TB storage units Seagate Backup Plus External Hard Drives Connection to the internet Installation Media Continuity Facilities Resumption of operations in the event of a catastrophic failure at the TFHRC facility will follow the plan for the facility as a whole Continuity Communications There is not another connection to the internet designated for resuming operations in the event of a catastrophic event at the TFHRC facility Communications between personnel will be accomplished using a combination of phone and email Phase IV Reconstruction Operations The plan is to repair current systems to restore service This will be accomplished using vendor support such as the Dell on site service contract along with THFRC systems personnel to restore the THFRC server with the least data loss possible When the THFRC is ready to be used for data dissemination a notice will be sent to the contacts listed in the Alert and Notification section al APPENDIX C DISASTER RECOVERY TFHRC SERVER INTRODUCTION Purpose The purpose of the disaster recovery plan is to define a set of potential problems and their likely solution Planning Assumptions The basic assumption is that there is a three day window to recover from a disaster without causing
18. Qee 072 024 00 EI Figure 107 Hlustration DLA 2200 Smart UPS front panel AVR trim indicates that the UPS is compensating for high utility voltage AVR boost in contrast is compensation for low utility voltage Online means that the UPS is supplying utility power connected to the outlet and not using battery power The indicators between the on off buttons and the column of lights on the right side are from top to bottom Overload On Battery and Replace Battery Battery Disconnected Overload when lit indicates that the connected loads are drawing more power than the UPS power rating and requires use of the troubleshooting section of the manual On battery indicates that utility power is probably off and that shutdown of the attached equipment is recommended The unit has sufficient power to handle the Dell 2900 and the two MD1000 units for five to fifteen minutes Replace battery battery disconnected need to be verified by the user To check for a user serviceable disconnection turn the UPS off and remove the front cover of the unit The battery plugs into the front of the unit and the connection can be unplugged and plugged in If the battery isn t holding a charge obtaining a replacement needs to be discussed with the COR Any more rigorous battery maintenance needs to be handled with the assistance of the TFHRC IT Desk and concurrence of the COR Additional troubleshooting information is con
19. OracleServiceIMSDEV This is the development instance of Oracle Restart this service if Oracle is not responding This service typically takes about two minutes to restart This service not restart after completion of backups restarts following Windows software updates Itis likely that problem requiring the restart of this service may also require applying redo logs to bring the instance to a consistent state OracleServicelMSPROD This isthe production instance of Oracle Restart this service if Oracle is not responding This service typically takes about two minutes to restart This service may not restart after completion of backups or restarts following Windows software updates It is likely that a problem requiring the restart of this service may also require applying redo logs to bring the instance to a consistent state OracleServiceIMSTEST This is the test instance of Oracle Restart this service if Oracle is not responding This service typically takes about two minutes to restart This service may not restart after completion of backups or restarts following Windows software updates It is likely that problem requiring the restart of this service may also require applying redo logs to bring the instance to a consistent state Single Hard Drive Failure on the RAID 1 Array 186 Description A single drive failure on the RAID 1 array will not be evident other than by us
20. package of stored procedures has been developed to assign the correct CN to each data record by comparing the date of data collection to the date each CN was assigned to a given section The CN assign procedure must be run before QC checks are performed on a module of data The procedure can be run on demand from the Utilities menu in the Forms Selection section CN Assign is also run nightly on the database 5 Perform QC Checks When new data has been entered or loaded into the LTPP database a series of Quality Checks QC programs are run to check the reasonableness of many of the data elements Three types of checks are performed on the data Level C Checks These are checks to identify critical fields that contain a null value Level D Checks These are range checks on the validity and reasonableness of values entered in a field Level E Checks These checks are relational checks between data stored in one field and data stored in other related fields or tables The RECORD_STATUS field in each table stores the quality status of each record Records passing all level C checks are upgraded to RECORD_STATUS Records passing level D checks are upgraded to D and records passing level E checks are upgraded to E Not all data records will make it to level E Some records will stay at a lower level until a revision to a QC program is distributed Some records will not be upgraded because there is a
21. section b Delete everything above the STARTUP NOMOUNT line Find the ALTER TABLESPACE line and delete everything below it d Find the line with the se colon after CHARACTER SET US7ASCII i Remove all lines down to ALTER DATABASE OPEN RESETLOGS ii Remove all commented lines lines beginning iii Remove the line RECOVER DATABASE USING BACKUP CONTROLFILE Change all occurrences of IMSPROD to IMSNew the new instance name Change path if necessary Change the CREATE CONTROLFILE REUSE DATABASE line to CREATE CONTROLFILE REUSE SET DATABASE 4 Save the file m The new instance is started by Executing the script to create the control file from the SQLPlus prompt Changing user passwords Deleting LTPP created users not used in the instance Adding the instance specific users This is best done with scripts if object level privileges are granted Creating a SPFILE and restart the database instance using it The sequence of commands from the command prompt is typically sqlplus sys ayspud imsnew as Followed by a series of command executed at the SQLPlus prompt f connected to an idle instance Shutdown abort BD VLTPP Database VCreateIMSTestControlPile sq Alter user ltppdba identified by replace oldpwd cloned db Alter user trfdba identified by newpwd replace oldpwd Alter user custsupp identified by newpwd replace oldpwd clo
22. Figure 130 Figure 131 Figure 132 Figure 133 Figure 134 Figure 135 Figure 136 Figure 137 Figure 138 Figure 139 Figure 140 Figure 141 Figure 142 Figure 143 Figure 144 Figure 145 Figure 146 Figure 147 Figure 148 Figure 149 Figure 150 Figure 151 Figure 152 Figure 153 Figure 154 Figure 155 Figure 156 Figure 157 Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Screenshot Preparation Wizard Figure 158 Figure 159 Figure 160 Screenshot Screenshot Screenshot Merge selection options Selecting Resource Order Resource Credential Testing Log on Priority Selection Defaults Selecting a Backup Device Picking General Settings Backup Method Picking General settings Compression Type Selection of Advanced Options Defaults Setting Advanced Open File Options defaults Advanced Open File Options Used Using Defaults for Network and Security Identifying Pre and Post Commands Archive Method Selection Job functions accessible through Job Setup screen Delete Confirmation dialog box Initial Properties
23. Management System IMS The LTPP IMS is comprised of all the information collected as part of the LTPP Program This includes electronic data and data on paper datasheets in addition to documents raw data files videos meeting minutes software reports products and much more that is not easily captured in a relational database The two major IMS components are the Pavement Performance Database PPDB and the Ancillary Information Management System AIMS The electronic data and data collected using paper forms are loaded processed and stored in the Pavement Performance Database PPDB The electronic files paper data forms and other types of LTPP information are considered part of the ancillary information management system AIMS figure 1 The PPDB is a relational database that contains data elements that are easily organized into relational tables Included in the same database instance are the tables used by the LTPP Traffic Analysis Software LTAS to generate information for the PPDB s traffic module Figure 1 Schematic LTPP Information Management System IMS There are multiple copies of the PPDB and AIMS associated with different functions of the IMS Central Database The LTPP database and AIMS are maintained at Turner Fairbank Highway Research Center TFHRC as part of the Technical Support Services Contract TSSC This database serves as the record or archival copy It may be referred to in some do
24. Purpose Planning Assumptions Objectives Concept of Operations SYSTEMS OVERVIEW Risk Identification and Mitigation 42 Complete Loss of Facilities at TFHRC 46 APPENDIX D HARDWARE MAINTENANCE SERVER Server status Dell Support Server ATTACHED STORAGE Dell Support Storage UPS DRAC USAGE APPENDIX E BACKUP POLICIES BI WEEKLY DRIVE ROTATION SCHEDULE QUARTERLY TAPE ROTATION SCHEDULE INPUTS TO BACKUP POLICIE APPENDIX F BACKUP SOFTWARE 59 61 61 63 SYMANTEC ADMINISTRATION AND TROUBLESHOOTING 63 Passwords 63 Symantec User Accounts 63 APPENDIX G BACKUPS AN ILLUSTRATED HOW TO CREATING A JOB SETTING UP RECURRIN MANAGING JOBS REVIEWING JOB HISTORY SAVING BACKUP REPORT PREPARING FOR RECOVERY IDR Preparation dr Files WORKING WITH MEDIA AND DEVIC Media Devices APPENDIX H OFF SITE BACKUP PROCESS FIRST FEDERAL CORPORATION Logging in Checking Inventory Preparing a Batch Getting boxes back Ordering boxes and labels OFF SITE BOX ROTATION APPENDIX 1 SERVER RECOVERY TFHRC SERVER INTELLIGENT DISASTER RECOVERY RESTORING A FILE APPENDIX J ORACLE DBA QUICK REFERENCE ORACLE ERROR MESSAGES 18 80 80 DATABASE MAINTENANCE Determine Characteristics of Tablespaces 106 Increase Tablespace Size Stop Start the Database Recover a Database TABLE MAINTENANCE Add Remove Modily Field 109 Modify Keys Constraint
25. The file naming conventions for the various folders are as follows Incremental weekday I_work_hist log_yyyymmdd fail 0 Incremental db I db hislog yyyymmdd fail 0 Incremental trafic _traffic_histlog_yyyyymmdd _fail 1 Monthly cdgfNN histlog yyyymmd fail 2 SDR gt gt gt 3 Weekly cdgfNN_hisvlog_yyyymmdd _fail 3 Periodic AIMS tapes Click on Save Figure 49 Screenshot Selecting a printer Sere as troe Hes p gern pres Figure 50 Screenshot Locations and names for job logs Verify the full expanded history was printed when it comes up in Acrobat Click on Job Log to ensure it is in front Figure 51 Click on Expand upper right comer Click on Print lower sight comer The printer selection menu will come up Figure 49 Adobe PDF should be the default printer If not select it and then click on Print The filename that comes up is that assigned by BE Figure 50 ne pu E Figure 51 Screenshot Job Log screen Summary Form Enter the file name 85 Type the first letter of the file name The file name in this case where the Job Log tab is forward will be the log version The dates in the files are the date the backup being printed was STARTED not the date it finished If the backup failed a printout is still made to know which backups do and do not exist The History or the Log may be missing
26. sever En uc aruit Figure 22 Screenshot Making Folder and File Selections for Backup On the following screen click on Resource credentials E 9 Fac Due pol Figure 23 Screenshot Verifying Access to Drives Identified in Backup Selections Ifthe logon account does fot exist on the server it must be changed using Change Change will bring up a list of users Select the appropriate user and click ok immani Ueer Yos Cormon HADLTFPSER LTFP DBM HADLIFPSew HEDATEPSER eer arin HAD LTPPSe estes HED LTEPSER Syer MRDLTPESER Figure 24 Screenshot Picking a Logon Account to Run a Backup Job Figure 25 Screenshot Preparing to Test Access for Backups Test the Selected or All depending on the number of accounts Figure 26 Screenshot Successful Access Test successful test should be accepted by clicking on OK If it is not successful troubleshoot based on error message displayed APPENDIX G BACKUPS AN ILLUSTRATED HOW TO CREATING A JOB job is created from the Job Setup screen shown in figure 27 Either the New Job or New job using wizard option from the Backup Tasks box on the left hand side of the screen can be used I CEP EE e Figure 27 Screenshot Symantec BE job set up screen The upper left box
27. sys syspwd insprod as sysdba eate pfile from spfile alter database backup controlfile to trace shutdown immediate Shut down the destination instance if not newly created using the syntax in the next three lines The first line calls SQLPlus and the remaining lines run at the SQLPlus prompt sqiplus sys syspud imsnew sysdba shutdown immediate Delete the files from the destination directories Log on as Administrator to do this without having to check folder permissions The syntax provided is run at the command prompt rather than using Windows Explorer to do the same thing Check for location of trace files del G LTPP_Database IMsNew ONLINELOG del G LTPP_Database copies IMSNew ONLINELOG del F LTPP_Database IMSNew ONLINELOG 126 the source instance to the destination copy D VLTPP Database MIMSProdl D LTPP_Database IMSTest Copy the online logs from the originating database locations to the new database The original locations will typically be G LTPP_Database IMSProd ONLINELOG G LTPP_Database IMSTest ONLINELOG Copy G LTPP_Database copies INSProd ONLINELOG TPP_Database copies IMSTest ONLINELOG Copy F LTPP_Database IMSProd ONLINELOG F LTPP_Database IMSProd ONLINELOG Copy the original control files from the originating database locations to the new database py_G LTPP_Database IMSProdVCONTROLETEBV G LTPP_Databas
28. APPENDIX AG SYMANTEC 2010 DOING BACKUPS CREATING A JOB 242 SETTING UP A RECURRIN MANAGING JOBS REVIEWING JOB HISTORY SAVING BACKUP REPORTS APPENDIX AH RECOVERY FROM BACKUP DELL 2900 INTELLIGENT DISASTER RECOVERY PREPARING FOR RECOVERY IDR Preparation Files RESTORING A FILE APPENDIX AI ORACLE OPERATIONS BEFORE CLONING TO A NEW INS CLONING APPENDIX AJ ROBOCOPY APPENDIX BA WORKSTATION ACTIVITIES BACKUPS AND ARCHIVES WORKSTATION SOFTWARI LIST OF FIGURES Figure 1 Schematic LTPP Information Management System IMS 2 Figure 2 Schematic LDEP Application provides access to PPDB 3 Figure 3 Screenshot Home screen for Backup Exec 2010 17 Figure 4 Flow Chart Regional data flow diagram replace 24 Figure 5 Screenshot Example of data filter in PPDB 26 Figure 6 Screenshot Selecting a job to check 27 Figure 7 Screenshot Sample PPDB form 27 Figure 8 Schematic Backup Process Overview Replace with TFHRC equivalent or reference to backups section of backups and archives Figure 9 Schematic TFHRC Server Access Diagram Figure 10 Illustration Front panel features Figure 11 Illustration Hard drive carrier Figure 12 Illustration R515 back panel Figure 13 Illustration Location of redundant power supply indi Figure 14 Illustration Front view of Dell PowerVault MD 1200 Figure 15 Illustration Rear of MD1200 unit 55 Figure 16 Elements
29. EIS HDD The specified hard drivehas See Troubleshooting a Hard Drive If the Abe experienced rebuild abor problem persists see your RAID documentation E1812 HDD The specified hard drive has been Information only Removed removed from the system E1913 CPU amp The BMC firmware doesnot Update to the latest BMC firmware See Firmware suppor the CPU the BMC User s Guide for more Mismatch information on setup and use of BMC PBD Pwr The power distribution board Ensure that the power distribution board Cable power cable is unseated missing power cable is seated properly If the 27 problem persists replace the power istibution board power cable See Installing the Power Distribution Board SAS Cable SAS cable Ais unseated missing Check the cable connection to the SAS or bad backplane See Cabling the SAS Backplane Boards EIAIS SAS Cable B SAS cable B is unseated missing Check the cable connection to the SAS backplane See Cabling the SAS Backplane Boards 215 Code Text Causes Corrective Actions Cable bay SAS cable is unseated Check the cable connection to the SAS FB missing or bad backplane See Cabling the SAS Backplane Boards LAI Pwr Cable FB Flex bay power cable is unseated Check the power cable connection to the missing or bad flex bay back
30. General Tasks on the screen has several functions that may be useful Delete Properties Run now and Test run Copy is a between server function and not applicable to the LTPP system These are discussed in Managing jobs This discussion goes through the process using the New job option Start by clicking on New Job This will bring up figure 28 The first activity is to select the items to be backed up If this is the first time this collection of folders files has been backed up enter a descriptive name for the list of items under Selection list name to replace the Backup NNNNN naming Below the name write some text about the materials in the backup 68 Do not select show file details unless a by directory listing is absolutely necessary The list generated in the expanded job log will run to hundreds of pages Select the folders and files from each drive 8 The right hand selection block starting with Domains is not applicable for LTPP 9 Ib is recommended to save time and space that the recycle bin not be backed 10 System state and Shadow Copy Components should generally be included Figure 28 Screenshot Starting a new job selections If this is or variant of an existing backup incremental version of a full backup for example click on the arrow beside Backup NNNNN to get a list of existing selection lists figure 29 Clicking on the preferred list will popul
31. function If the problem persists the system board is The system BIOS has reported PCI parity error on a component that resides in PCI slot faulty See Getting Help 24 Code Causes Corrective Actions E1712 PCISERR The system BIOS has eporid a Remove and recat the PCI expansion DIM FHA PCL system error on a component cards If the problem persists see that resides in PCI configuration Troubleshooting Expansion Cards PCLSERR space at buss device Slot function 1 the problem persists the system board is faulty See Getting Heip The system BIOS has reported a PCI system error on a component that resides in slot E1714 Unknown Err The system BIOS has determined See Gening Help that there has been an error in the system but is unable o determine its origin ITIP PCIE Faal The system BIOS has reported a Remove and reseat the PCI expansion Em BA Di PC fatal error on a component cards If the problem persists see pe that resides in PCI configuration Troubleshooting Expansion Cards space at bus device 8 PCIE Fatal funcion Ifthe problem persists the system board is Er So faulty See Getting The system BIOS has reported a PCIe fatal error on a component that resides in slo EISIO HDD amp amp Fault The SAS subsystem has See Troubleshooting Hard Drive determined that hard drive amp has lexperienced a fault
32. move and delete data from AIMS using Tortoise SVN LTPP Traffic Analysis Software The LTAS application uses a set of tables within the PPDB The outputs of the software support data and computed parameters tables in the PPDB Prior to the implementation of LTAS version 2 0 the tables were stored separately from the PPDB The LTAS application is the user s interface to the traffic analysis tables The raw hourly and vehicle traffic data goes through automated checks as itis loaded and summarized to daily data records Then the daily records go through another series of automated checks and are summarized to monthly records Finally the monthly records are checked and summarized to annual records The LTAS application provides tools to analyze the data including running the QC process purging data graphing data sets and reporting The LTAS Application is documented in the following volumes of the LTPP Traffic Software Materials Reference Volume 1 LTAS Users Guide Volume 2 LTAS Graphics Specifications Volume 3 LTAS Oracle Table Specifications Codes and QC Volume 4 LTAS Functional Specifications Volume 5 LTAS Program Design Specifications More information on LTAS documentation is included Appendix T Reference Documents Pavement Performance Database The Pavement Performance Database PPDB is an Oracle database structured for efficient data storage that contains many types of pavement performance data
33. sz 1 Change Storege Management pse 2016 21 Change Read tts fume 21 Change Offire orine 41 Drop Tablespace fasz 1246 WON BIS 0 rue Figure 83 Screenshot Actions dropdown Figure 84 Screenshot Edit Tablespace dialog box SQL Developer 132 E mwe 3 OS Wesen RO e gee Lim een nes Figure 85 Creating a new tablespace USING SCRIPTS SETTING UP CONNECTIONS MANAGING USERS Users can be created dropped and modified within this utility by users with sufficient privileges Once a user is connected to an instance the other users can be identified by clicking on the Other Users folder at the bottom of the users folder list Right clicking on the Other Users label brings up options to refresh filer clear filter create user or drop In creating user the user name and password is assigned at a minimum In addition the user should be assigned User_Data as a default table space and Temporary_TS as the temporary Locking a user with SQL Developer is done after connecting using a DBA account If the connection is made through the Connections window the users may be found under Other Users the last option on the list under the DBA objects If the connection is made through the DBA window Users be found under the Security options as shown in figure 86 Figure 86 Screenshot SQL Developer DBA locatio
34. 11 2 0 LTPP database initimsnew o 269 The value of syspwd should be recorded for reference Loss of the SYS password may require deleting and recreating the instance Edit CAoracle produci 1 1 2 0 LTPP network admin tnsnames ora Duplicate the IMSProd entry and change IMSProd to IMSNew in the copy Edit C oracle product 2 0 LTPP network admin Listener ora Duplicate the IMSProd entry and change IMSProd to IMSNew in the copy Restart the listener logged in to the Administrator account using the syntax of the next two lines at the command prompt snr lan 1 stop CLONING Cloning is the duplication of a database Throughout this discussion IMSProd is the source instance being cloned IMSNew is the instance that is getting and exact duplicate clone of IMSProd Shut down the source instance to be cloned using the following syntax where cach line is separate entry The initial line is executed at the command prompt SQLPlus is called in the first line The remaining lines are executed at the SQLPlus prompt sqlplus sys syspud imsprod as syadba create pfile from alter database backup controlfile to trace shutdown immediate Shut down the destination instance if not newly created using the syntax in the next three lines The first line calls SQLPlus and the remaining lines run at the SQLPlus prompt sqlplus sys syspud imsnew as sysdba Shutdown immediate Delete the files from
35. 93 93 94 95 96 96 97 98 99 100 100 101 101 102 102 102 103 105 131 131 132 132 133 Figure 86 Screenshot SQL Developer DBA location for users to manager user properties 134 Figure 87 Screenshot General user information in SQL Developer 134 Figure 88 Screenshot Simple user edits in SQL Developer 135 Figure 89 Screenshot Example search for OS updates 141 Figure 90 Screenshot Example search for OS updates 142 Figure 91 Screenshot Example search for Oracle update 143 Figure 92 Screenshot Oracle Patches amp Updates Simple Search Window 144 Figure 93 Screenshot Oracle Patches amp Updates Patch 8559467 Download Window 145 Figure 94 Screenshot Example search for Oracle update 146 Figure 95 Screenshot Oracle Patches amp Updates Simple Search Window 147 Figure 96 Screenshot Oracle Patches amp Updates Patch 7631956 Download Window 148 Figure 97 Schematic Backup Process Overview Replace with TFHRC equivalent or reference to backups section of backups and archvies 179 Figure 98 Schematic DPW server access diagram 184 Figure 99 Schematic Backup Process Overview Replace with TFHRC equivalent or reference to backups section of backups and archives 198 Figure 100 Schematic TFHRC Server Access Diagram 203 Figure 101 Illustration Drive Status Indicators 211 Figure 102 Illustration Location of redundant power supply indicators 212 Figure 103 Illustration
36. All equipment was purchased with Dell Support The equipment is out of warranty and no extended warranty has been purchased If support is needed the TFHRC Help Desk is the first point of contact If Dell needs to be contacted the following information may be useful Model Dell PowerEdge 2900 Service Tag Express Code The contact information for Dell was www dell con ProSupport Federal Government under suppor if doing troubleshooting or requesting help Phone 1 800 945 3355 Service Tag will be needed when calling Customer Acct 652071937 Customer Purchase Order 4 DTFH6110F000232 Dell Purchase ID 2001802580973 Order Number 137363466 137363680 1 for each unit Storage Status Additional storage was added using two Dell PowerVault MD1000 units with fifteen 2TB SATA hard drives in each Documentation is stored in 7 dellcom Manalall productiesuprt ser stor neuesuprt powervaulipowervaut sd L000_Ovwner i27s620Manual_en us pd accessed 5 7 2015 218 5 enclosure tas LED 7 2 ve actity LED 55 spit mode LED 3 enclosure mode sich Figure 103 Illustration Front view of Dell PowerVault MD1000 NE Figure 104 Illustration Hard drive carrier LEDs for MD1000 1 pomar Eat 3 powersuopiyeaning tan Figure 105 Illustrat n Rear of MD1000 unit http fdownloads dell com Manuals all products esuprt ser stor neV esuprt powervaultpowerv
37. Assumptions The LTPP Team has agreed that three days of downtime would not have an adverse effect on data delivery It has also been decided that if a hardware problem occurs with the server the server will be repaired Therefore there is not a failover system standing by As explained later we will rely on the Dell server maintenance and support agreement to repair the system in the event of a hardware failure Objectives The objective of this plan is to outline the steps necessary to minimize disruption to LTPP data delivery in case of a system failure CONCEPT OF OPERATIONS Phase I Readiness and Preparedness Knowing that system failures are inevitable there are certain steps that can be taken ahead of time to ensure that services can be restored in a timely manner Backups The server contains two Redundant Array of Independent Disks RAID volumes The first volume is configured as RAID 1 array consisting of two 1 TB Serial Attached 35 Small Computer System Interface SAS drives This volume is the C drive and contains the operating system as well as a copy of the database backups source code repository and archive logs This drive also hosts the bulk of the web applications The RAID 1 array can withstand a single drive failure and remain operational These drives are not hot swappable The second volume is configured as a RAID 5 array consisting of seven 2 TB SAS drives There is an eighth 2 TB SAS drive sta
38. DBA Role Role Privs discussed later in this appendix There is no way to determine what a user s password is Lost passwords must be reset script user review is stored in the synchronization template for use in finding the users in any instance particularly the cloned ones It is run iteratively to both identify users and to set the passwords for the cloned instance Manage Users Users are created using the CREATE USER command and given privileges with the GRANT command A quick way to create a user who can create tables in the database s default tablespace and default temporary space is to use the GRANT command For example D BY neupwd space temporary tablespace temporary This creates a user who can connect to the database as newuser with newpwd as the password The specification of a user tablespace puts all objects the user creates in a tablespace other than the SYSTEM tablespace Typically the user tablespace name for LTPP users is User Data 11g or Users 12c The temporary tablespace name is Temporary is 11g or Temp 12 After creating a user privileges with at the system or object level can be granted Privileges may be granted to individual users or to roles and the roles granted to users LTPP practice is to grant privileges to roles and then grant roles to users All users are granted the select only role which allows connection to the database and the ability 10 query any table The com
39. Each user of Symantec Backup must be identified to the software by name and password The users in Symantec have accounts whose names may or may not match those of their server login accounts The passwords for the user logon accounts much match those of the server Symantec User Accounts For the purposes of backup the user for logon to the server and the Symantec software is zsymantec user If this user name changes several items must be changed in the Symantec software to continue to use the system One is the ownership of the selection lists On the menu bar pick edit manage selection lists at which the following dialog box comes Bebe alit edt delete 3 E ppr o De o Tate n Woody CDS stot maling soge dss Bep ee de Figure 115 Screenshot Picking a selection list to create a job Each list must be checked via edit for a valid logon account When Edit is selected the screen in figure 116 will appear Figure 116 Screenshot Confirming changes to a selection list with impacts identified Click on Yes and then on the screen shown in Figure 117 click on Resource Credentials in the left most column under Source Figure 117 Screenshot Making folder and file selections for backup 238 coy pm perm en Figure 118 Screenshot Verifying access to drives identified in backup selections 1E the logon account
40. Front view of Dell PowerVault MD 1000 219 Figure 104 Illustration Hard drive carrier LEDs for MD1000 219 Figure 105 Illustration Rear of MD1000 unit 220 Figure 106 Photos Smart UPS 2200 rack mount UPS front and back DLA model Figure 107 Illustration DLA 2200 Smart UPS front panel Figure 108 Screenshot Weekly tape rotation for 2010 Figure 109 Screenshot Bi weekly tape rotation 2011 Figure 110 Screenshot Bi weekly tape rotation 2012 Figure 111 Screenshot Bi weekly tape rotation 2013 Figure 112 Screenshot Bi weekly tape rotation 2014 Figure 113 Screenshot Bi weekly tape rotation 2015 update Figure 114 Screenshot Quarterly tape rotation schedule Figure 115 Screenshot Picking a selection list to create a job Figure 116 Screenshot Confirming changes to a selection list with impacts identified 238 Figure 117 Screenshot Making folder and file selections for backup 238 Figure 118 Screenshot Verifying access to drives identified in backup selections 239 Figure 119 Screenshot Picking a Logon Account to Run Backup Job 240 Figure 120 Screenshot Preparing to Test Access for Backups 240 Figure 121 Screenshot Successful Access Test 241 Figure 122 Screenshot Symantec BE job set up screen 242 Figure 123 Screenshot Starting a new job Selections 243 Figure 124 Screenshot Picking from existing selection lists 244 Figure 125 Figure 126 Figure 127 Figure 128 Figure 129
41. Information System Owner The information system owneris the agency official responsible for the overall procurement development integration modification operation and maintenance of the information system Authorizing Official The authorizing is a senior management official or executive with the authority to formally assume responsibility for operating an information system at an acceptable level of risk to agency operations agency assets individuals Assignment of Security Responsibility The assignment of security responsibility role is responsible for the overall security of the DPW Other Designated Contacts The other designated contacts role designates key contact personnel who can address inquiries regarding system characteristics and operation FHWA The FHWA role provides oversight to the LTPP program Customer Service The Customer Service role is responsible for answering customer inquiries including distributing data Customer Service is also responsible for maintaining some centrally processed data Table 11 DPW Role assignments ROLE Organization with Individuals in Role System Administrator Backup System Administrator E DPW Database Administrator Backup DPW Database Administrator AMEC Regional Database Administrator Fugro NCE Stantec Regional Program Manager Fugro NCE Stantec 174
42. It has two network controller cards The second is attached to a switch for use on an internal LTPP network with three devices the central server the Dell 2900 and a personal computer Server maintenance and troubleshooting is discussed in Appendix D Hardware Maintenance Operating system updates are acquired automatically and applied manually on notification of availability Anti virus updates are done automatically Oracle updates and patches are done on an as needed basis The upgrade from Oracle 118 to Oracle 12c was done in summer 2014 The system has a 64 bit version of Oracle 12c with 32 bit Oracle 11g and 12c clients to run LTPP specific software Instances in the Oracle installation include IMSProd IMSTest IMSDev and SDR29 The SDR29 instance is a static copy of the information provided for public distribution via InfoPave Data Processing Workstation DPW The Data Processing Workstation is the development machine for updating PPDB data as well as the development environment for IMS and Traffic Analysis software It is a Dell PowerEdge R515 purchased in 2013 as a replacement for a Dell PowerEdge 2900 server The DPW is located off site It has two Dell PowerVault MD 1200 storage units The Dell 2900 is available as a failover machine It has two 1 86 GHz Xenon 5150 dual processors It also has 2 GB of RAM and a RAID 5 array consisting of seven 10 000 RPM 146 GB SAS disks For backups it h
43. LTPP users for instances on the TFHRC server are LTPPDBA CustSupp TRFDBA Traffic Consolidated IPC InstanceUser and DataCheck LTPPDBA is the primary DBA and the schema under which the PPDB tables are created TRFDBA is the DBA under which the LTAS schema was developed when LTAS and the PPDB were in separate instances CustSupp is the schema originally associated with all customer support functions Its use has been reduced to creation of a schema associated with database extractions for public dissemination Traffic is a user associated with early versions of LTAS It has been retained in lieu of recoding the software to ignore the tables in the Traffic schema Consolidated is a schema used for materialized views that consolidate information from multiple tables IPC is the schema used for objects that exist primarily of the use of the InfoPave application InstanceUser is user name prefaced with the instance short form name This user can do most manipulations on most common objects within the database and is the preferred user for general work in an instance Datacheck is a user that can look at but not alter objects in the instance Other users may be created on an as needed basis Identifying Users The users for an instance can be identified using the Users view Select from AIL Users order Hj Ugerlane Additional information about the roles and privileges granted users may be obtained using Role Privs SYS Privs and
44. Q ROLES AND RESPONSIBILITIES DPW The Data Processing Workstation Disaster Recovery DR and Continuity of Operations Plan COOP along with the Data Processing Workstation IT Security Plan for the Long Term Pavement Performance Program require that persons be identified to fulfill various roles This section describes the roles required and the organizations providing individuals for those roles The assignment of specific individuals by role and their contact information is provided quarterly to the COR who is the Information System Owner This information is provided separately from this document to remove personally identifiable information PII and prevent obsolescence of this document The role assignments for this system are reviewed monthly The organizations with individuals assigned to each role are documented in table 11 Contact information for each organization is provided in table 12 System Administrator The system administrator is responsible for maintaining the DPW in good working order This includes the operating system access control installed software backups and hardware Backup System Administrator The backup system administrator is responsible for maintaining the DPW under the direction of the system administrator and or when the system administrator is unavailable This includes the operating system access control installed software backups and hardware DPW Databas
45. The backups are done the 1 Wednesday of February May August and November of each year and run for more than 24 hours due to the inclusion of the verify option in Symantec Backup Exec 2015 Recovery Media The DVDs required to rebuild the server are located in server room in the top right drawer of the credenza Return of off site backups can be requested from First Federal Corporation The information system owner would be the primary person to interface with First Federal Corporation The system administrator would be the secondary contact Dell Service The TFHRC server is a Dell PowerEdge R515 with service tag SXTSIO2 It is covered by the Gold or ProSupport with Mission Critical and the 4 Hour On Site Service plans until 5 10 2019 These warranty plans ensure that the TFHRC server will be repaired quickly The system administrator would be the primary person to interface with Dell Support The information system owner would be the secondary contac The two MD1200 40 TB storage units with service tags J3BVFZI and 2SFVLO2 are under warranty until 5 17 2017 Phase II Activation Decision Process The decision to implement will begin with the initial notification that the server is not functioning This will typically be on direct observation by the system administrator Upon identification the severity of the problem will be assessed If it is determined that the problem can likely be resolved by a restart of services the
46. allow recovery of the last transaction committed before the failure The RMAN backups are managed by two Windows Task Scheduler jobs The RMAN Level 0 Backup runs every Tuesday at 6 00 AM This job produces a full backup of the instance and removes old archive logs and backup sets that are no longer necessary The RMAN Backup runs every day at 10 00 PM It produces a differential backup of changes to the data base instances The drive C backup captures these backups for off site storage AIMS Repository and Historical Files The AIMS Repository and historical files are too big to fit onto a single cartridge As a complete copy of the AIMS exists on an external hard drive in the possession of Customer Service in addition to the copy on the DPW the decision has been made not to backup video files A full backup strategy has been implemented for all other files The 180 AIMS repositories reside in K Historical_ AIMS cYYYY gt where lt YYYY gt is year of the update The backups are done the 1 Wednesday of February May August and November of each year and run for more than 24 hours due to the inclusion of the verify option in Symantec Backup Exec 2010 Recovery Media The CDs required to rebuild the server are located in server room in the top right drawer of the credenza They br cordion folder labeled DPW inside i Banker s Box that is in the center section of the eredenza Return of off site backups can req
47. also been decided that if a hardware problem occurs with the server server repairs will be contingent on the expected remaining time for parallel processing and cost Therefore there is not failover system standing by As explained later we will rely on TFHRC s Help Desk to support repair of the system in the event of a hardware failure Objectives The objective of this plan is to outline the steps necessary to minimize disruption to LTPP data delivery in case of a system failure CONCEPT OF OPERATIONS Phase I Readiness and Preparedness Knowing that system failures are inevitable there are certain steps that can be taken ahead of time to ensure that services can be restored in a timely manner Backups 196 The server contains two Redundant Array of Independent Disks RAID volumes The first volume is configured as RAID 1 array consisting of two 1 TB Serial Attached Small Computer System Interface SAS drives This volume is the C drive and contains the operating system as well as a copy of the database backups source code repository and archive logs This drive also hosts the bulk of the web applications The RAID 1 array can withstand a single drive failure and remain operational These drives are not hot swappable The second volume is configured as a RAID 5 array consisting of seven 2 TB SAS drives There is an eighth 2 TB SAS drive standing by as a hot spare This array can withstand a single drive failure and the ho
48. are no workarounds to allow the user access to the server Resolution Notify FHWA IT that the VPN network is down This step is typically unnecessary since they have automated notifications when the BPN network has a problem Wait for service to be restored DPW Software Services Become Unresponsive Description Software services becoming unresponsive typically results in inability to access an Oracle instance Effect The effect is that users are prevented from using the affected Oracle instance Mitigation No mitigation is necessary Resolution First use Process Explorer procexp exe to see if there are any processes with high resource utilization The resources to check are CPU Disk VO and Memory In normal operation the CPU rarely exceeds 15 the Commit Charge is rarely over 20 GB and the Disk Bytes is typically under 100 KB s If a process is found to be causing the server to exceed these limits determine if it needs to be killed or restarted If resources were not the problem check the following table to determine which service needs to be restarted On rare occasions a restart of the DPW may be required 185 Table 13 DPW service diagnostics Service Description This listens for connections to the Oracle database instances Restart this service if having Oracle connection problems Restarting this service will typically not impact users greatly Restart takes about 30 seconds
49. back to its role as a standby There may be a few hours of degraded performance while the replacement drive is being rebuilt Failure of the RAID 1 Array Description A failure of the RAID 1 array C would take down the server The system will not boot Effect Because the operating system is on C the system would not be able to continue operations and it would be impossible to reboot the server due to the lack of a boot drive Mitigation A copy of the latest drive C backup is kept on an attached USB drive well as removable RD1000 cartridges 45 Resolution Contact Dell and have them determine the cause of the failed array After the failed drives and or controllers are replaced install the OS anti virus and Symantec Backup Exec 2015 From there you can restore the latest usable backup The Oracle instances will likely need recovery after drive C is restored due to the sudden crash of the operating system The expected downtime would be 1102 days Failure of the RAID 6 Array Complete Loss of Description failure of the RAID 5 Array D would leave the server running but the portals Redmine LTAS APEX and ADEP would be unavailable because they depend on database repositories D The code repositories would still be accessible if needed Effect Files on D will be unavailable There will also be a loss of data due to some files not being backed to stay within the limitations of our backup media T
50. backups The modification has been designed to avoid to the extent possible swapping backup cartridges in and out of storage boxes at the time of pick up The 13 week cycle implemented uses the following logic minimum of 13 weeks exists between reuse of a cartridge The cartridge for the 4 week of every month cartridge used is stored off site Cartridges from the 4 week of the 1 and 2 months in a quarter are stored off site for a 13 week period Cartridges 4 week of the 3 months in a quarter are stored off site for a 2 year period Off site storage occurs the 3 week of the following month new cartridge is used for each backup stored off site for 2 years for the 1 two years This is cartridge 12 for the 1 2 year backup and A G for the next 7 in the cycle The following has been designated the ideal quarterly cycle for backups using the following logic minimum of a year exists between reuse of a ridge Cartridges from the first three backups in a calendar year are stored off site for a year Off site storage occurs the quarter after the backup occurs 235 recycled cartridge from off site is used the quarter after it returns from storage The last backup of a calendar year is stored off site for 2 years before recycling None of the quarterly backups are retained indefinitely 236 APPENDIX AF SYMANTEC BACKUP EXEC 2010 SYMANTEC ADMINISTRATION AND TROUBLESHOOTING Passwords
51. daily SOFTWARE LTPP uses a combination of off the shelf and custom software Both systems use Windows Server 2008 SP 2 on the Dell R515s The DPW software includes Redmine Apache Tomcat and other packages not used on the central server The following is a list of key software packages that are loaded on the LTPP machines to facilitate operations Windows Server 2008 Operating system patches are applied through Windows update on monthly basis The process is set for manual rather than automatic updates even though the computer is typically on and connected to the Internet Updates are also applied as requested by the TFHRC Help Desk Symantec Anti Virus Symantec Endpoint Protection 11 0 2000 1567 is being used on the central server This is the FHWA required antivirus program It requires a manual update to ensure all elements are current Symantec Anti Virus is being run on the DPW Symantec Backup Exec 2010 Symantec Backup Exec 2010 is being used to perform backups to RD1000 disk cartridges It requires manual updates to remain current The product on the Dell Poweredge 2900 was supported through February 2014 10 Symantec Backup Exec 2015 Upgrade from Symantec Backup Exec 2014 for the central server 4 Oracle Oracle was chosen early in the development of the IMS At the time it was one of the only tools which could handle large databases across multiple hardware platforms Oracle also had a form
52. dialog box Folders on Server for Storing Backup Reports Selecting a Job to Print Job History screen Selecting a Printer Locations and names for job logs Job Log screen Summary Form IDR Wizard selection IDR Preparation Wizard Opening Screen IDR Boot Media Options IDR CD Creation Instructions Selecting a computer for disaster recovery preparation Location selection for CD image Identifying Windows OS installation file location Image creation messages Outcome of disaster recovery preparation Identification of image file name and location for Selecting the Disaster Recovery File Option in the IDR Identifying computer and location for File Completion of creation of copy of file Completion of disaster recovery preparation LIST OF TABLES Table 1 PPDB Data modules and submodules Table 2 Filter CN and QC programs and chapters for active modules Table 3 Filter CN and QC programs and chapters for inactive modules Table 4 server role assignments Table 5 Server Organizational Contact Information Table TFHRC server Oracle service diagnostics Table 7 Hard drive indicator patterns Table 8 Function of redundant power supply indicators Table 9 Diagnostic light codes Dell R515 Table 10 Database Server Frequencies Table 11 DPW Role assignments Table 12 DPW Organizational Contact Information Table 13 DPW service diagnostics Table 14
53. does not exist on the server it must be changed using Change Change will bring up a list of users Select the appropriate user and click ok 239 immani Ueer SymetosUsx Yos Cormon HEDLIFPSER LTFP DBM HADLIFPSew HEDATEPSER eer arin HAD LTPPSe estes HED LTEPSER Syer MRDLTPESER Figure 120 Screenshot Preparing to Test Access for Backups Test the Selected or All depending on the number of accounts Figure 121 Screenshot Successful Access Test successful test should be accepted by clicking on OK If it is not successful troubleshoot based on error message displayed 241 APPENDIX AG SYMANTEC 2010 DOING BACKUPS CREATING A JOB job is created from the Job Setup screen shown in Figure 122 Either the New Job or New job using wizard option from the Backup Tasks box on the left hand side of the screen can be used SS aor ae DEL Emo ligure 122 Screenshot Symantec BE job set up screen The upper left box General Tasks on the screen has several functions that may be useful Delete Properties Run now and Test run Copy is a between server function and not applicable to the LTPP system These are discussed in Managing jobs This discussion goes through the process using the New job option Start by clicking on New Job This will bring up figure 123 The first activity is to select the items to be back
54. en b product support praduc powervanlt ml1200manuals accessed 5 7 2015 Image taken from Dell PowerVaull MD1200 and MD1220 Storage Enclosures Hardware Owner s Manual 54 1 Power supply cooling fan module 2 Primary enclosure management module EMM 3 Secondary enclosure management 4 Power switches 2 module EMM 5 Power supply cooling fan module Figure 154 Illustration Rear of MD1200 unit Comments on status indicators good and bad 3 4 5 6 7 1 System status indicator 2 Debug port 3 SAS In 4 Tn port link status 5 SAS pon Out 6 port link status 1 EMM status LED Figure 164 Elements of Enclosure Management Module 55 Dell Support Storage All equipment was purchased with Dell Support The equipment is under warranty If Dell needs to be contacted the following information may be useful Model Dell PowerVault MD1200 Upper unit Service Tag Express Code 41560205725 Lower unit Service tag 2SFVLO2 Express Code 6073285394 The contact information for Dell is www delLcom ProSupport Federal Government under support if doing troubleshooting or requesting help Phone 1 800 945 3355 A Service Tag will be needed when calling Customer Acct 20080702 Customer Purchase Order DTFAWA 11 D00004 Dell Purchase ID Order Number 1200337526 Maintenance and support are as follows Dell Hardware Warra
55. id I accessed 12 17 201 3 0C2211 APEX Version 4 2 5 00 8 was current on the DPW as of 5 7 2015 For the most current documentation see http www oracle com echnetwork developer tools apex documentation index html which may require a free account on Oracle Technology Network hitp docs oracle com edE17556_01 doc user 40 e15517 pdf Oracle Application Express Application Builder User s Guide Release 4 0 15517 02 September 2010 accessed 12 17 2013 SQL Developer htip www oracle com technetwork developer tools sql developer documentation documentation index 152579 himl accessed 5 7 2015 is the index to all SQL Developer documentation Version installed as of 5 7 2015 is 4 0 3 on the TFHRC server and on the Dell 2900 Notepad notepad plus plus org accessed 5 7 2015 There is no user documentation se but references and links to support Long Term Pavement Performance Data entry Portal LDEP User Guide November 2012 Source Code Inventory This document was created to give an inventory of all source code developed at SAIC in the course of working on the LTPP contracts since 1989 The source code is kept in a configuration management system PVCS that is organized by logical groupings of programs The Table of Contents mirrors this directory structure Individual programs are listed in the body of the document in its respective group For example programs are written in Pro C so they are in
56. in the box next to Associate with Click on Yes on the dialog box lower left corner to confirm the selected media set The media will be removed from the current list and show up on the media list selected In this case the list will be retired The file has not been moved on the server cartridge it has just been made permanently unavailable for backup use media set can be defined by using New media set under Media Set Tasks or Media set wizard under Media Wizards on the left side of the screen Gereral Vat Ru Name Meda st vernteprolectionpetod Wace d peiod WARNING Pry mada can be cveneren f the cuente Protection veli Nona Tho overarto protection laval setto Pario wth taer prompt Gick Help forrereifomaion mcm RN Figure 69 Screenshot Setting media set properties In creating a media set three things are established its name the overwrite protection period and the append period LTPP does not currently use the Vault options The name reflects the content and frequency of the backup being included in the set The overwrite period is set to 1 week less than the backup retention period This value must be integer This is set so that in the event backup failure occurs and the backup is run later than anticipated the media will be available when the next backup is expected The append period is set to Infinite since either the c
57. may be installed on three machines The license covers both active servers This copy of the software is not being updated Spiceworks This is third party freeware for network management The software was installed to generate a list of installed applications for IT Security documentation None of the other functionality is used It requires access to the internet to function 226 APPENDIX AE BACKUP POLICIES DELL 2900 BI WEEKLY TAPE ROTATION SCHEDULE The tape rotation for weekly backups is the 13 week schedule discussed in Inputs to Backup Policies In figure 108 the schedule for 2010 taken from the offsite_rotations spreadsheet is provided The underlined dates are the weeks that off site pickup and delivery occurs The Week column is the number of the week in the quarter The Date value is the Tuesday backup date The Tape number is the label on the box and the tape of the tape to be used When a tape other than the 2 year archive is taken out of the rotation it is replaced by a tape with the same number and an alpha character The 8 tape was used to store 2010 annual submission for indefinite archive Its replacement is 8A The off site column indicates whether or not the tape is sent off site for storage It is sent out the following month the indicated Thursday date The Recycle date is the date that a tape will be reused It has only been filled in for tapes sent off site so that the date it needs to be returned from
58. oeroude iore ute staan terr the Wier ae ears rth ge Figure 152 Screenshot Location selection for CD image to Windows Operating System Installation Fier re inde nem nation fs oe snes bot Tre Weve vein estem onthe nato path rusi racha veto and d mune he erpi serg etai Bones oryps the path tatha boaton tha pron nate fs Sassi E Om Figure 153 Screenshot Identifying Windows OS installation file location Tha wari ono mage Seana pening sem ia Bap niat Racovy rain es Pease wan Franar te togra voses Pense wat Figure 154 Screenshot Image creation messages Figure 155 Screenshot Outcome of disaster recovery preparation CD Image created successfully au mr tumtha lonng mag 50 s CD eda Woran o Ferd sor the carpere eser con omen by cavi he cad ad editae Copy ste coe aso CO rags hide CD Page fer wh CD Vitas Tun Le ERR E Figure 156 Screenshot Identification of image file name and location for iso Files 266 Oeste IDF Root te yet you sure tthe conto be canto toad enia ped usc ra You tae
59. of output to document the action di Comment out the import line in the import batch file if it is table specific or all tables in an export dmp file have been dropped c Editimport batch file to reflect existence of data and files to reload in the event of cloning an instance Electronic Files Management 123 APPENDIX L CLO THE DATABASE This must done under the Administrator password on the central server to have the necessary privileges BEFORE CLONING FOR THE FIRST TIME If the destination instance is brand new a service will need to be created for the instance Create a new PFILE for the instance by copying an existing copy For example copy initimsprod ora to initimstest ora Copy app ora_12c product 12 1 0 LTPP database initoBNane Ora c app ora_12e product 12 1 0 LTPP database initNewDaName ora For the remainder of this discussion it will be assumed that the initDBName is IMSPROD and the initNewDBName is IMSTEST If the pfile does not exist create it from the spfile Edit the new PFILE and change IMSPROD to IMSTEST everywhere using a text editor Notepad or Notepad Create the directories for the new instance md g LTPP_Dat abase O8 ame nd 9 ma G LTeP_batabase bB_namie CONTROLEILE _ Database DB_name DATAFILE LTPP_Database DB name VONLINELOG q NLTPP Batabag copies VbB name md giVUTPP Database copies DB name CONTROLFILE md g LTP
60. of Enclosure Management Module 55 Figure 17 Photos APC Smart UPS 2200 rack mount UPS front and back SMT model 57 Figure 18 Illustration SMT 2200 Smart UPS front panel 58 Figure 19 Screenshot Bi weekly tape rotation for 2015 60 Figure 20 Screenshot Picking a Selection List to Create a Job 63 Figure 21 Screenshot Confirming Changes to a Selection List with Impacts Identified 64 Figure 22 Screenshot Making Folder and File Selections for Backup 65 Figure 23 Screenshot Verifying Access to Drives Identified in Backup Selections 65 Figure 24 Screenshot Picking a Logon Account to Run a Backup Job 66 Figure 25 Screenshot Preparing to Test Access for Backups 66 Figure 26 Screenshot Successful Access Test 67 Figure 27 Screenshot Symantec BE job set up screen 68 Figure 28 Screenshot Starting new job Selections 69 Figure 29 Screenshot Picking from existing selection lists 70 Figure 30 Screenshot Merge selection options n Figure 31 Screenshot Selecting Resource Order n Figure 32 Screenshot Resource Credential Testing Log on 72 Figure 33 Screenshot Priority Selection Defaults 72 Figure 34 Screenshot Selecting a Backup Device 73 Figure 35 Screenshot Picking General Settings Backup Method 73 Figure 36 Screenshot Picking General settings Compression Type 74 Figure 37 Screenshot Selection of Advanced Options Defaults 74 Figure 38 Screenshot Setting Advanced Open File Options defaults 75 Fig
61. of Oracle s Enterprise Manager on the Dell 2900 Limited documentation on the more common uses of this tool for a DBA is found in Appendix M SQL Developer Notes Abbreviated documentation is also found in the database engineer s guide on data review and manipulation This software is updated when there is a major version change by copying over the new installation package SQLPlus Worksheet The SQLPlus Worksheet is an Oracle client tool that allows the user to type SQL Plus commands in an input window and see results in an output window Commands are executed by pressing or CTRL Enter Previous commands can be accessed by pressing CTRL p and next commands by pressing CTRL n Other shortcuts are available in this tool This tool has been replaced by SQL Developer in daily use Command Line SQLPlus and Oracle utilities can be executed from a Command window by users with permissions to access the server directly For example a sqlplus file sql can be executed with the following syntax sqlplus connectstring Geglfile sqi The Oracle data export command can be executed as follows exp connectstring parameters par where the parameters par file has all input parameters The same list of parameters can be included on the command line PowerDesk7 PowerDesk is file management software similar to Windows Explorer advantage over Windows Explorer is that the results of directory searches may be save to text x
62. of in service pavement sections across the United States and Canada These pavement sections have been constructed using highway agency specifications and contractors and have been subjected to real life traffic loading Pavement sections that are part of the LTPP program are categorized as General Pavement Studies GPS and or Specific Pavement Studies SPS GPS consist of a series of studies on nearly 800 in service pavement test sections throughout North America SPS are intensive studies of specific variables involving new construction maintenance treatments and rehabilitation activities Refer to the QC Manual listed in Appendix T Reference Documents for a list of GPS and SPS experiments LTPP DATA The majority of LTPP data has been collected by four Regional Support Contractors RSCS Each RSC is responsible for data collection in a region of North America The RSCs coordinate with state and provincial highway agencies SHAs in their regions to collect many types of data including details of maintenance and rehabilitation activities coring and sampling activities collection of site specific weather data drainage and traffic data RSCs also collect deflection FWD distress friction longitudinal profile and transverse profile data Regional date entry and validation are conducted using the LTPP Data Entry Portal LDEP and AIMS Data Entry Portal ADEP web enabled systems linked to the production instance LTPP Informai
63. quality checks For example TST represents TST Some SPS programs have additional divisions such as DI and Figure 5 TST_D and TST instead of just C programs Archive Processed Data Instructions for data archival can be found in the directives In most cases both processed data and raw data should be archived Filter Data into Database Table 2 and table 3 list the filter programs used for each module The PPDB application includes forms from which each filter can be executed An example is shown in Figure 5 The outcome of the processing can be reviewed by identifying the job from a list of background jobs from the left hand of the PPDB screen as shown in Figure Screenshot Example of data filter in PPDB Background Jone Userstort TimeEnd TimetiretneyConsoe Output gt reser tuv pln D rator Yom D eG Figure 6 Screenshot Selecting a job to check b Paper Sheet Data Most modules have some sheet data to enter into the database The exceptions are the CLM DLR and Longitudinal and Transverse Profile data modules Sheet data comes in the form of paper data sheets that may have been filled out by one of the state or provincial agencies or by regional contractor on site during construction or rehabilitation of an LTPP section Some data modules are composed entirely of sheet data INV TST and others have mostly electronic data with only one or two sheets of
64. range and Problems If the problem persists ensure has halted operation thatthe microprocessor heat sinks are properly installed See Troubleshooting the Microprocessors NOTE The LCD continues to display this message until the system s power cord is disconnected and reconnected to the AC power source or the SEL is cleared using Either Server Assistant or the BMC Management Utility See the Dell OpenManage Baseboard Management Controller User s Guide for information about these utilities Elis CPUS Specified processor is missing or Troubleshooting the Microprocessors Presence bad and the system is in an unsupported configuration CPU Processors are in a configuration Ensure that your processors match and Mismatch unsupported by Dell conform to the type described inthe Code Text Causes Corrective Actions Microprocessor Technical Specifications outlined in your system s Started Guide EIIF CPU Protocol The system BIOS has reported a See Genting Help processor protocol error 1420 CPU Bus The system BIOS has reporteda See Getting Help PERR processor bus parity errar E1421 CPU Init The system BIOS has reporteda See Getting Help processor initialization errar E1422 CPU Machine The system BIOS has reporteda See Getting Help machine check error E1610 PS Missing poweris available from the See T
65. schedule which has been linked to the tape rotation schedule A pop up calendar exists for date selection Add another box volser volume serial number if needed A separate batch must be created for each container size The post upload pickup in August or September will generally need multiple batches to cover all the boxes a Figure 74 Screenshot Preparing to add batch for pickup by FFC VBScript CMSNET re yeti gong ts create a net atch Figure 75 Screenshot Confirmation to start add batch operation Click Yes 101 Afterall data has been entered click on Dataset on the right panel to enter a note which applies to the content of the box This makes it possible to review the inventory without having any of the tape rotation materials on hand to flag a memory of their content Then click on Post Batch on the right panel to post the first container Repeat for additional container sizes Microsoft Internet Explorer ef Perino ses cea Figure 76 Screenshot Confirmation of successful hatch posting as pending Click OK After all batches have been posted click Verify on the right panel to make sure Return dates are future dates et VBScript CMSNET Are you ware ta very thatch Click Yes Click on Post Batch to send an email to FFC VBScript CMSNET Are yc you wart to post the serves at Figure 78 Screenshot Confirmat
66. serious disruption in data dissemination capabilities Given that a hot standby is not required it has been decided that the most cost effective solution is to repair whatever is wrong with the current system and resume operations We are counting on the Dell ProSupport Mission Critical Option that specifies 4 hour on site service with 6 hour parts availability to resolve any hardware problems with the TFHRC server We are also counting on First Federal Corporation to be able to deliver backups from off site storage within three hours of an emergency request Objectives The main objective is the restore the TFHRC to full operational status as soon as possible with minimal data loss A secondary objective is to keep key personnel at the FHWA and the TSSC informed about the problem and its resolution When possible limited functionality will be made available while permanent repairs are underway Concept of Operations SYSTEMS OVERVIEW The TFHRC server is a databases and files housed on a single server that is accessible either on location or via VPN As shown in Figure 9 a keyboard connected via a KVN switch and VPN are the primary entry points into the system Insert figure here Figure 9 Schematic TFHRC Server Access Diagram Risk Identification and Mitigation Internet Connection Failure Description An internet connection failure will first become apparent due to failure to access the system via VPN or access to
67. supporting information FWD i Review Data Sheets An engineer familiar with the LTPP section should review all data sheets submitted by an agency before the data is entered into the database Questionable values or blank items should be researched and corrected ii Enter Data with PPDB Forms Basic forms instructions are included in the LDEP User s Guide For specific information about each data entry form refer to the Forms Index in the User s Guides section of the LTPP Operations Center Forms are grouped by module and in some cases by submodule a Figure 7 Screenshot Sample PPDB form Much LTPP data has been collected on paper datasheets In most cases one datasheet represents one entry form Entry forms were designed to look like the datasheets to avoid mismatches of data being entered Database tables were also designed to keep related data together Most of the data from a particular datasheet is stored together in one table Figure 7 shows a blank form for Traffic Data Sheet 16 entry located by clicking on Traffic under Forms Selection and then picking the Sheet 16 option Operations on a form are controlled by the permissions granted individual users Details on a form including content format codes dependencies minimum data checks Level C QC range checks Level D QC and intra modular checks selected Level E QC are available through the Field Definitions button at the bottom of cach f
68. the Archive Logs in CALTPP_Darabase lt instance gt ArchiveLogs If drive fails drive D contains all of the files necessary to recover the database If drive D fails the RMAN backups from drive C s Flash Recovery Area will need to be restored and the logs applied in order to recover the database Redo logs and archive logs are managed by the Oracle instance and allow recovery of the last transaction committed before the failure The RMAN backups are managed by two Windows Task Scheduler jobs The RMAN Level 0 Backup runs every Tuesday at 6 00 AM This job produces a full backup of the instance and removes old archive logs and backup sets that are no longer necessary The RMAN Backup runs every day at 10 00 It produces a differential backup of changes to the data base instances The drive C backup captures these backups for off site storage AIMS Repository and Historical Files The AIMS Repository and historical files are too big to fit onto a single cartridge As a complete copy of the AIMS exists on an external hard drive in the possession Customer Service in addition to the copy on the DPW the decision has been made not to backup video files A full backup strategy has been implemented for all other files The 199 AIMS repositories reside in K Historical_ AIMS cYYYY gt where lt YYYY gt is year of the update The backups are done the 1 Wednesday of February May August and November of each year and run for
69. the destination directories Log on as Administrator to do this without having to check folder permissions The syntax provided is run at the command prompt rather than using Windows Explorer to do the same thing Check for location of trace files VLTPP Database VIMSNew VLTPP Database VIMSNew Modi fications D VLIPP Database VIMSNew Trace VLTPP Database VIMSNew Tablesps 270 the source instance to the destination copy 0 LTPi copy D LTP tabase IMSProd D LTPP_Da tabase IMSProd Tablespaces DIALIPP Database IMSNew Tablespaces Create a SQL script to generate a new control file that identifies the clone database name and points to the new file location Note that once this script is created these steps only need to be repeated when new data files are added to the database It is however good practice to create new script every time a database is cloned A script to generate a control file is created with the following steps 1 Find the trace file with the same timestamp as the alter database backup controlfile to trace command This will be located in the source database s trace directory An example would be DALTPP_DatabaseIMSProd Trace find correct directory msprod ora 2976 1 2 Save the file as DALTPP_Database IMSNew Modifications CreateIMSNewControlFile sql 3 Edit the trace file with a text editor Le Notepad or Notepad Find the Set 2 RESETLOGS case
70. 0 W 10 XJ XF bak tmp XD Srecycle bin LOG dADr 1 DB Backup 20100216105 ed ja mkdir D_Working RoboCopy DA Working MIR NP R 10 W 10 X XF pagefile sys bak tmp XD IMS Training Miriam 2009 11 30 Srecycle bin system volume information JLOG dADr D Work Backup 20100216102 ed jn mkdir C_OS RoboCopy CA JAC OS MIR NP R 10 W 10 X XF BCD NTUSER USRCLASS tmp XD Srecycle bin system volume information LOG dADr C OS Backup 2010021610 D Net start OracleServicelMSProd Net start OracleServicelMSDev Net start OracleServicelMSTest Net start OracleServiceTREProd Net start OracleserviceTREDev Net start OracleServiceTAPTest Net start Oraclese 278 APPENDIX BA WORKSTATION ACTIVITIES The work station in F 212B is a Windows XP machine that may no longer be attached to a TFHRC network This equipment is on the inventory of a FHWA LTPP team member replacement will be available at some time in the future BACKUPS AND ARCHIVES Backups of the principal workstation attached to the TFHRC server are intended to have a second on site copy of scripts documents and other data related working materials that used in LTPP database activities In the event of a computer failure itis expected that re establishing the software will be done by the TFHRC Help Desk as is customary for all none COE computers complete copy of the files on the system was ma
71. 0 seconds OracleServiceIMSDEV This is the development instance of Oracle Restart this service if Oracle is not responding This service typically takes about two minutes to restart This service may not restart after completion of backups or restarts following Windows software updates It is likely that a problem requiring the restart of this service may also require applying redo logs to bring the instance to a consistent state OracleServicelMSPROD This is the production instance of Oracle Restart this service if Oracle is not responding This service typically takes about two minutes to restart This service may not restart after completion of backups or restarts following Windows software updates It is likely that a problem requiring the restart of this service may also require applying redo logs to bring the instance to a consistent state OracleServiceIMSTEST This is the test instance of Oracle Restart this service if Oracle is not responding This service typically takes about two minutes to restart This service may not restart after completion of backups or restarts following Windows software updates It is likely that problem requiring the restart of this service may also require applying redo logs to bring the instance to a consistent state Single Hard Drive Failure on the RAID I Array Description single drive failure on the RAID 1 array will not
72. 012 McLean VA 22101 2296 November 2013 14 Sponsoring Agency Code TS Supplementary Nove Contracting Officer s Representative COR Yan Jane Jiang HRDI 30 Abstract This document provides information about the Long Term Pavement Performance LTPP Programs database operations This document provides specific information about maintaining and upgrading the LTPP database servers operating systems and software performing DBA functions tracking change requests and producing the annual Standard Data Release t provides information about using the LDEP Application including running QC programs data loaders and entering data the Pavement Performance Database PPDB with data entry forms References to the Traffic Analysis Software LTAS are included In addition it provides an overview of regional activities hat relate tothe database and data entry applications TI Key Words Ti Disinibution Statement Database database server general pavement studies LTPP pavement performance specific pavement studies LTAS operating system AIMS central database server maintenance anti virus backup software relational database database tools SDR standard data release DBA functions user administration SPR software release output data loaders data entry forms QC Manual 19 Security Clasif ol this epon 20 Security Classi otis page Price Unclassified Unclassified Page
73. 0505 ES NTFS C Q X d backuplcr ixt E ed ja mkdir AIMS_2008_06 RoboCopy E AIMS_2008_06 J AIMS_2008_06 MIR INP R 10 W 10 X XF bak amp XD Srecycle bin LOG dADr AIMSI2008 Backup 20100505 1og 275 mkdir AIMS_TRACKER RoboCopy EAAIMS TRACKER J AIMS_TRACKER MIR NP R 10 W 10 XI XF bak tmp XD Srecycle bin LOG dADr AIMSITRKR Backup 20100505 log ed jn mkdir SDR24 RoboCopy EASDR24 JASDR24 MIR NP R 10 W 10 XJ XF bak tmp XD Srecycle bin LOG dADr AIMSISDR24 Backup 20100505 log mkdir regionaluploads RoboCopy E regionaluploads J regionaluploads MIR INP R 10 W 10 XJ XF bak amp XD Srecycle bin LOG dADr AIMS lregup_Backup_20100505 log mkdir RoboCopy EACTDB JACTDB MIR NP R 10 W 10 XI XF bak tmp XD Srecycle bin LOG dADr AIMSICTDB Backup 20100505 1og D BackupAIMS2yyyyMM emd Format J V AIMS2100506 FS NTFS C Q X lt d backupleraxt E mkdir Installation RoboCopy EAnstallation JAInstallation MIR INP R 10 W 10 XJ XF bak tmp IXD Srecycle bin LOG dADr AIMS2Inst Backup 20100506 log mkdir 5 2009 09 RoboCopy 2009 09 5 2009 09 MIR NP R 10 W 10 XJ XF bak mp XD Srecycle bin LOG dADr_AIMS2_Backup_20100506 l0g mkdir ims uploads 276 RoboCopy E ims_uploads J ims_uploads MIR NP R 10 W 10 XJ XF bak tmp IXD Srecycle bin LOG dADr AIMS2IMSup Backup 20100506
74. 102 mkdir Installation_200708 RoboCopy EAInstallation 200708 J Installation 200708 MIR NP R 10 W 10 XJ bak tmp XD Srecycle bin LOG d Dr_AIMS2Inst07_Backup_20100506 log mkdir old installation RoboCopy E old installation JAold installation MIR NP R 10 W 10 X XF bak mp XD Srecycle bin LOG dADr AIMS2oldInst Backup 20100506 cds mkdir old server RoboCopy E old server I old server MIR NP R 10 W 10 XJ XF bak tmp XD Srecycle bin LOG d Dr_AIMS2oldsvr_Backup_20100506 log D The weekly backup was done with the file backupweekly cmd Its contents follow The structure is to stop the Oracle instances format the cartridge to be used and name it in the process create the folder into which the files will be copied make the copy write out a log on the folder copied and go on to the next folder Afterall folders were copied the Oracle instance were restarted The folders copied were operating system D working storage and Database The Recycle Bin and selected files were omitted from the copies for each folder use rod TREProd Net stop Oracleservi Net stop OracleSerw Net stop Oracleservi Net stop Oraclesery Net stop OracleServ Net stop OracleServiceTRFTe Net stop OracleServiceTrf9Aug Format J V CDG100202 FS NTFS Q X lt d backupler txt E mkdir LTPP Database m RoboCopy G J G_LTPP_Database MIR NP R 1
75. 2900 to use in updating Consistency is only an issue for major version updates i e 3 x to 4 x so that LTPP documentation updates can be simplified BEFORE CLONING TO A NEW INSTANCE This must be done under the Administrator password on the Dell 2900 to have the necessary privileges Since the destination instance did not previously exist a new service will need to be created for it Create a new PFILE for the instance by copying an existing pfile For example copy initimsprod ora to initimstest ora where IMSProd is the existing instance and IMSNew is the new instance copy C loracle product t 1 2 0 LTPP database initimsprod ora CAoracle product LI 2 0 LTPP databasetinitimsnew ora If the pile does not exist create it from the spfile for the existing instance Edit the new PFILE and change IMSPROD to IMSNEW everywhere using a text editor Notepad or Notepad Create the directories for the new instance as indicated with the following syntax ad the command prompt md D LTPP_Database INSNew md D NUTPP Database VIMSNes V It may be located in d VLIPP Databasediag md D LTPP_Dat abase IMSNew Tal D VUTPP Database VIMSNex Hoc md Kio oracle flash recovery Mlg new verify that trace is located here dbms insnew trace Create the service for IMSNew at the command prompt using the syntax oradim new sid IMSNew sys4imstest maxusers 50 startmode auto pfile C oracle product
76. 37830 AMEC Environment amp Infrastructure 12000 Indian Creek Court Suite F AMEC Beltsville MD 20705 301 210 5105 Engineering amp Software Consultants Inc 14123 Robert Paris Court ESCINC Chantilly VA 20151 195 APPENDIX AB CONTINUITY OF OPERATIONS DELL 2900 PURPOSE The Continuity of Operations Plan COOP describes how the Long Term Pavement Performance LTPP Program will continue to function when the Dell 2900 server is unable to sustain normal operations The outage may be partial or total and may be the result of operator error software problems hardware problems or network connectivity problems Scope The scope of this COOP is the Dell 2900 Server Situation Overview The Dell 2900 is used for parallel processing of the central archival of data gathered by the regional contractors It provides a back up for data disseminated to the public through demand requests Original copies of the data are maintained on the DPW and within the regions Because the original data is not lost it is possible for the Dell 2900 server to be unavailable for a period of time without causing all data dissemination activity to stop The primary risk to the system being unavailable is that failure of both servers may halt cusstom data dissemination Planning Assumptions The LTPP Team has agreed that two weeks of downtime would not have an adverse effect on data delivery for this parallel system It has
77. 9 Screenshot Bi weekly tape rotation 2011 229 M m SS IgE bes A imer E DRM S BRUN i a 2 ER mop SE as eta EE P i a E aen M EE sac tan ct as tope dy Figure 110 Screenshot Bi weekly tape rotation 2012 230 ab ig an Twe tate nde m Te ote en i E 2 2 soan B mme een i pos usc ssp ccc E ot Tren Ego Cae nan sen io mas nana unen ES c Ee Rag me emen em ME omen B i Rath amet I LAN LO Stl s mme Dee mb i ied pore E gt mE ees 55 Mos sso mee Dae prg eso 13an aes tH Figure 111 Screenshot Bi weekly tape rotation 2013 231 ee im gt Metas Ra eee tn neq ae ten cn ant sett ma na ua ies 1 mo rt rat gem mm Tem moos mam prem diis mm Dd coe aen Bu de Eo 2 o Fara woen 2 Figure 112 Screenshot Bi weekly tape rotation 2014 222 fk ffm cf
78. ATING SYSTEM Operating system updates can also applied as individual patches downloaded from Microsoft on a separate computer These patches are located by going to Jntp Avww microsoft com technetsccurity currentaspx and performing a Microsoft Security Bulletin Search We are interested in searching for all bulletins for Windows Server 2008 x64 SP2 An example search is shown below in Figure 89 140 TA aa Microsoft Security Bulletin Figure 89 Screenshot Example search for OS updates Each patch that has not been applied is downloaded to a flash drive or other portable media The media containing the patches is then mounted on the server and the patches are applied while logged in as an administrator Operating System Updates Operating system updates are applied as individual patches downloaded from Microsoft a separate computer since internet access is not available from the server These patches are located by going to https technet microsoft comv en us security bulletin and performing Microsoft Security Bulletin Search We are interested in searching for all bulletins for Windows Server 2008 SP2 An example search is shown in Figure 90 141 aS DE Bears ss mee Search by buletin KR oF CVE miner ORF butletin by product ar component
79. And as a general check the LCD panel in front should be lit with a blue light If anything goes wrong on the system the LCD panel will have an amber light which means further action is necessary Hard Drive Inspection Figure 101 shows where the hard drive carriers may be on the front of the Dell 2900 The LTPP unit only has eight horizontal drives The vertical drives in the illustration and all of the drives for removable media except for the DVD drive at the far left of the unit are not part of the LTPP system The hard drive carriers have two indicators the drive status indicator and the drive activity indicator Figure 101 Each hard drive should have a green drive status indicator dell comisupporvhome uslen 19 product support product poweredge 2900 manuals accessed 5 7 2015 1 drive status indicator green and amber green drive activity indicator Figure 101 Illustration Drive Status Indicators Table 17 lists the drive indicator patterns Different patterns are displayed as drive events occur in the system For example if a hard drive fails the drive failed pattern appears After the drive is selected for removal the drive being prepared for removal pattern appears followed by the drive ready for insertion or removal pattern After the replacement drive is installed the drive being prepared for operation pattern appears followed by the drive online pattern Tabl
80. Annual Last and first quarter of year T20 Incremental 3 Quarterly Gr Database 1 670 Annual for core data Tneremental 2 uploads Incremental 4 Continuous for analysis Weekl Hr Recovery 3 Never Dell recovery arca Never var USB port NIA drives connected for upload of data T Backup device N A backup cartridges Ki Archive 30 000 Topic dependent Quarterly Some materials have external drive backup on a topic specific basis Using following logic results in the ideal 13 week cycle A minimum of 13 weeks exists between reuse of a cartridge Every 4 cartridge used is stored off site for a 13 week period Off site storage occurs 2 weeks after the backup occurs A recycled cartridge from off site is used the week after it returns from storage The last backup of a calendar quarter is stored off site for 2 years before recycling A new cartridge is used for each backup stored off site for 2 years for the I two years This is cartridge 12 for the 1 2 year backup and A G for the next 7 in the cycle The ideal cycle has been modified because cartridge pickups are a fixed week of the month and not a fixed interval A spreadsheet offsite_rotationsNN xls saved in G_Task Order L NTask D IMS Hardware Softwarell Database Operation a_Operating the Database Backup Procedures was used to develop the annual calendar and container rotation schedules for the various
81. CHARACTER SET WESMSWWIN1252 i Remove all lines down to ALTER DATABASE OPEN RESETLOGS ii Remove all commented lines lines beginning _ iii Remove the line RECOVER DATABASE USING BACKUP CONTROLFILE e Change all occurrences of IMSPROD to IMSNew the new instance name f Change path if necessary Change the CREATE CONTROLFILE REUSE DATABASE line to CREATE CONTROLFILE REUSE SET DATABASE 3 Save the file Execute the script to create the control file Execute the script to create the control file from the SQLPlus prompt and start the database For the script to work the 12 user must have read write permissions on the included directories Permission should be given at the LTPP_Database level The new instance is started by Executing the script to create the control file from the SQLPlus prompt e Changing user passwords Deleting LTPP created users not used in the instance Adding the instance specific users This is best done with scripts privileges are granted f object level Creating a SPFILE and restart the database instance using it The sequence of commands from the command prompt is typically sqlplus sys syspud imsnew as sysdba 128 Followed by a series of command executed at the SQLPlus prompt f connected to an idle instance Shutdown abort LTPP_Database IMSNew Modi Ficat ions CreateIMsNewCont rolFile sql Alter Alter alter Alter Alter
82. DPW using Symantec Backup Exec 2010 There several directories that are excluded from the backups such as the ADEP repositories and the LTPP database instances This backup is scheduled to run every day at 11 30 PM The backup job does not automatically copy files to drive J for off site storage when complete There is a Windows Task Scheduler job CopylncrementalFileBackupToCartridge that copies the H SymantecRecoveryPoints LTPPDPW directory to JALTPPDPW at 6 00 PM every day Application Backup Strategy Because it may be desirable to recover application data at finer grained intervals than the recovery of an entire drive many applications have their own backups being produced Oracle Database Instances The control files and redo logs are written to drives D and KA in ALTPP_Database cinstance gt where instance is IMSProd IMSTest or IMSDev The control files contain the latest structure of the database files and the redo logs contain the latest transactions Drive K contains the Flash Recovery Area in KALTPP_Database lt instance gt FlashRecoveryArea and the Archive Logs in CALTPP_Database lt instance gt ArchiveLogs If drive C fails drive D contains all of the files necessary to recover the database If drive D fails the RMAN backups from drive C s Flash Recovery Area will need to be restored and the logs applied in order to recover the database Redo logs and archive logs are managed by the Oracle instance and
83. Dell PowerEdge RS15 Systems Hardware Owner s Manual 49 1 drive status indicator green and amber 2 _ green drive activity indicator Figure 11 Illustration Hard drive carrier Table 7 lists the drive indicator patterns Different patterns are displayed as drive events occur in the system For example if a hard drive fails the drive failed pattern appears After the drive is selected for removal the drive being prepared for removal pattern appears followed by the drive ready for insertion or removal pattern After the replacement drive is installed the drive being prepared for operation pattern appears followed by the drive online pattern Table 7 Hard drive indicator patterns Condition Drive Status Indicator Pattern Identify drive preparing for removal Blinks green two times per second Drive ready for insertion or removal OH Drive predicted failure Blinks green amber and off Drive failed Blinks amber four times per second Drive rebuilding Blinks green slowly Drive online Steady green Rebuild aboried Blinks green three seconds amber three seconds and off six seconds Back Panel Inspection 50 1 Serial connector 2 Video connector 3 IDRAC6 Enterprise port optional 4 VFlash media slot optional 5 USB connectors 2 USB 2 0 compliant 6 Ethernet connection 2 10 100 1000 7 PCI expansion slot
84. Long Term Pavement Performance Information Management System Operations Manual VR 2 17 Publication No TBD 22015 US Department of Transportation Federal Highway Administration Research Development and Technology Tumer Fairbank Highway Research Center 6300 Georgetown Pike McLean Virginia 22101 2296 FOREWORD The FHWA Office of Infrastructure R amp D conducts and oversees research and development programs and projects that address critical highway infrastructure needs and priorities of national importance Studies focus on the design materials construction operation and preservation of highway pavements bridges culverts tunnels and other structures In addition the Office of Infrastructure R amp D provides expert technical assistance to other FHWA offices other Federal agencies State and local transportation organizations industry and academia This document describes the processes used to manage the LTPP Information Management System IMS and its component hardware and software Jorge Pagiin Ortiz Director Office of Infrastructure Research and Development Notice This document is disseminated under the sponsorship of the U S Department of Transportation in the interest of information exchange The U S Government assumes no liability for the use Of the information contained in this document This report does not constitute a standard specification or regulation The U S Governme
85. OFTWARE SYMANTEC ADMINISTRATION AND TROUBLESHOOTING Passwords Symantec User Accounts The users in Symantec have accounts whose names may or may not match those of their server login accounts The passwords for the user logon accounts much match those of the server For the purposes of backup the user for logon to the server and the Symantec software is zsymantec user If this user name changes several items must be changed in the Symantec software to continue to use the system One is the ownership of the sclection lists On the menu bar pick edit manage selection lists at which the following dialog box comes up Manage Selection Lists Select a istto edit or delete Nae Te E 08 reromarial Backup siete Weekly seit Backup woking storage De Figure 20 Screenshot Picking a Selection List to Create a Job Each list must be checked via edit for a valid logon account When edit is selected the following screen will appear 63 uw 1i pou change ts selection ist al associated pbs willbe aec Do yeu edt AMS Figure 21 Screenshot Confirming Changes to a Selection List with Impacts Identified Click on yes and then LA nx mm E psum m fae oto na F et nce pou E Vestes onsen 809 Wis AST mer e dac noe saan sua Sage Mr Ete
86. Operations Plan COOP appendices along with the Data Processing Workstation IT Security Plan for the Long Term Pavement Performance Program require that persons be identified to fulfill various roles This section describes the roles required and the organizations providing individuals for those roles The assignment of individuals by role and their contact information is provided by quarterly memo to the COR who is the Information System Owner The contact information for the various organizations and individuals is also include The information is provided separately from this document to remove personally identifiable information PII and prevent obsolescence of this document The role assignments for this system are reviewed quarterly The organizations with people assigned to each role are documented in Table 4 Contact information for each organization is provided in Table 5 System Administrator The system administrator is responsible for maintaining the TFHRC server in good working order This includes the operating system access control installed software backups and hardware Backup System Administrator The backup system administrator is responsible for maintaining the TFHRC server under the direction of the system administrator and or when the system administrator is unavailable This includes the operating system access control installed software backups and hardware TFHRC Database Administrator The dat
87. PLpatabase copies DB_name ONLINELOG md FrMUrPP DatabaseVcopiesVDB name md VUEPP Database copies VD name CONTROLFLE m P VUTPP Database copies DB name VONLINELOG Where G is the primary database location and F is the backup file location Create the service for IMSTest oradim new sid IMSTest syspwd sysdimstest maxusers 50 startmode auto pfile C app ora_12c produci t 12 1 0 LTPP database initNewDBNane ora 125 The value of syspwd should be recorded for reference Loss of the SYS password may require deleting and recreating the instance Enter the Oracle service user password when requested Edit CA applora_12c product 12 1 0 LTPP network admin tnsnames ora Duplicate the IMSProd entries IMSProd and Listener_IMSProd Change IMSProd to IMSTest in the copy Edit CA applora_12c product 12 1 0 LTPP network admin Listener ora Duplicate the IMSProd entry and change IMSProd to IMSTest in the copy Restart the listener lanrctl stop CLONING Cloning is the duplication of a database Throughout this discussion IMSProd is the source instance being cloned IMSNew is the instance that is getting and exact duplicate clone of IMSProd Shut down the source instance to be cloned using the following syntax where each line is separate entry The initial line is executed at the command prompt SQLPlus is called in the first line The remaining lines are executed at the SQLPlus prompt sqlplus
88. RFDBA but IMPEXPDBA given the DBA privilege which has been deprecated Importing the data requires the following command Imp impexpdba imxpexppudPinstance file filenane dmp f muser schemal touser schanal ignore y log imp filename YVYYMMDD 10g al will be successively LTPPDBA TRFDBA CUSTSUPP Enable constraints 2 Run record counts 3 Clone record database using the instructions in the NIMS Ops Manual 4 Return hard drive for use in next synchronization Work flow 1 Verify the working folder GA 1 Synchronizationyyyymm has copies ofthe tools to be used LoxiDuacmd TST Sample Basic RefreshData cmd LTESQL DCVSQL Refresh Dan s Reprocess_code_asgn_date sql parameters amp 2 Create current quarter s folder on the server K Wistorical_Database VYYY Syne MM_ Month 3 On receipt of hard drive off logs and other supporting export information to folder on GA 118 4 Do a record count of existing instance tables 5 Compare pre load record count to post load record count from last synchronization If the counts do not match either determine the difference or export the tables for later analysis overwrite of imported tables or submittal for use on the DPW The action will be determined by the table affected and the reason for the difference 6 Compare pre load record count to export log to verify tables expected and ident
89. TB External Hard Drive Connection to the internet Installation Media Continuity Facilities Resumption of operations in the event of a catastrophic failure at the TFHRC facility will follow the plan for the facility as a whole Continuity Communications There is not another connection to the internet designated for resuming operations in the event of a catastrophic event at the TFHRC facility Communications between personnel will be accomplished using a combination of phone and email Phase IV Reconstruction Operations The plan is to repair current systems to restore service This will be accomplished using vendor support such as the Dell on site service contract along with THFRC systems personnel to restore the server with the least data loss possible When the THFRC is ready to be used for data dissemination a notice will be sent to the contacts listed in the Alert and Notification section 183 APPENDIX 8 DISASTER RECOVERY DPW INTRODUCTION Purpose The purpose of the disaster recovery plan is to define a set of potential problems and their likely solution Planning Assumptions The basic assumption is that there is a three day window to recover from a disaster without causing serious disruption in data dissemination capabilities Given that a hot standby is not required it has been decided that the most cost effective solution is to repair whatever is wrong with the current system and res
90. TFHRC Server Role Assignments Table 15 Dell 2900 Server Organizational Contact Information Table 16 Dell 2900 Service Diagnostics Table 17 Hard drive indicator patterns Table 18 Fonction of redundant power supply indicators Table 19 Dell Poweredge 2900 LCD status message key Table 20 Dell 2900 COTS software Table 21 Database server frequencies Acronym ADEP AIMS CN CPU CSSC CTDB DPW FHWA GB IMS KB LDEP LTAS LTO LTPP MB PDE PPDB SAS SATA SDR SPR TFHRC TSSC TSSC UPS VA LIST OF ACRONYMS Definition AIMS Data Entry Portal Auxiliary Information Management System construction number critical patch update Customer Support Services Contractor Central Traffic Database Data Processing Workstation Federal Highway Administration gigabyte information management system kilobyte LTPP data entry portal LTPP Traffic Analysis Software Linear Tape Open Long Term Pavement Performance megabyte Public data extraction Pavement Performance Database serial attached SCSI serial ATA standard data release software performance report terabyte Turner Fairbank Highway Research Center Technical Support Services Contract Technical Support Services Contractor Uninterruptible Power Supply volt amp INTRODUCTION This manual is intended to support server and database operations for the LTPP Information Management System IMS The LTPP server is located at Turner Fairbank Rese
91. abase administrator maintains the database instances on the TFHRC server This includes database backups modification and storage management hema Backup TFHRC Database Administrator The backup database administrator maintains the database instances on the DPW This includes database backups schema modification and storage management under the direction of the database administrator and or when the database administrator is unavailable TSSC Program Manager The TSSC program manager is responsible for the oversight of the LTPP server activities at TFHRC Technical Support Services Staff The technical support services staff role provides support for TFHRC server activities on site under supervision of the TFHRC database administrator Customer Service The Customer Service role is responsible for answering customer inquiries including distributing data Customer Service is also responsible for maintaining some centrally processed data and maintaining copies of ADEP on the TFHRC server TFHRC IT The TFHRC IT role is responsible for granting VPN access through the FHWA TFHRC Internet 2 firewall and maintenance of FHWA licensed software Information System Owner The information system owneris the agency official responsible for the overall procurement development integration modification operation and maintenance of the information system Authorizing Official The authorizing is a senior management offic
92. aging area maintains two full image differential backup sets The Backup Exec software also takes care of copying the current week s backup files to drive J to be moved off site This copy occurs as the full or differential backup completes The backup of drive D is handled differently since the backup media is not large enough to contain an image backup Drive D is backed up on a file by file basis to H SymantecRecoveryPoints LTPPDPW using Symantec Backup Exec 2010 There are several directories that are excluded from the backups such as the ADEP repositories and the LTPP database instances This backup is scheduled to run every day at 11 30 PM The backup job does not automatically copy files to drive J for off site storage when complete There is a Windows Task Scheduler job CopylncrementalFileBackupToCartridge that copies the HSymantecRecoveryPoints LTPPDPW directory to INLTPPDPW at 6 00 PM every day Application Backup Strategy There are no applications on the Dell 2900 beyond the Oracle databases that require application specific backups Oracle Database Instances The control files and redo logs are written to drives D and KA in LTPP_Database lt instance gt where instance is IMSProd IMSTest or IMSDev The control files contain the latest structure of the database files and the redo logs contain the latest transactions Drive contains the Flash Recovery Area in KALTPP_Database lt instance gt FlashRecoveryArea and
93. all when the courier is on site The courier is met at the front desk of the Turner building with any boxes being sent A TFHRC individual with FFC identification must be present to log boxes in and out Setting up unscheduled visits Unscheduled visits should only be required in the event of catastrophic failures flood fire equipment failure and inability to use current on site backups to restore files While the cost is not large these visits must be cleared through the COR Obtaining First Federal credentials The COR will contact First Federal with the necessary information to put the LTPP IMS DB Engineer on the list of approved FFC contacts to pick up and deliver media An electronic ID RFID is provided It is needed for confirmation of pickup and delivery as well as for access to the FHWA account on line for scheduling HARDWARE CHECKS The system is checked on weekdays to verify that no yellow orange lights show on the front of any item Any light that is not green or blue requires investigation The steps to be taken are specific to the equipment and should be identified by using the relevant manual s On a weekly basis the Dell Server Manager tool is used to check internal elements of the server and external storage including batteries power supply and potential hard drive failures This check is generally done when the backup logs are saved AIL LTPP equipment is under warranty through either the LTPP program sp
94. arch Center TFHRC It is referred to in this document as the central server Other machines also under the control of FHWA s LTPP team are also addressed The LTPP IMS includes the Pavement Performance Database PPDB tables for the LTPP Traffic Analysis Software LTAS and the Ancillary Information Management System AIMS Applications that may be run on the central server will be discussed in this manual either explicitly by reference to other documents identified in Appendix Reference Documents Some of these programs were utilities written to accomplish a task and were not intended for general distribution However the majority of the programs are part of the LTPP data entry portal LDEP AIMS data entry portal ADEP or ETAS applications Information for each principal computer provided by FHWA including hardware warranties maintenance activities and software is included A limited discussion of regional operations is provided The discussion is not intended to be an exhaustive list of regional activities merely an overview of the elements that affect IMS operations LTPP SYSTEM OVERVIEW LTPP PROGRAM The LTPP program was established as part of the Strategic Highway Research Program SHRP in 1987 and has been managed by the Federal Highway Administration FHWA since 1992 LTPP was designed as a partnership with the States and Canadian Provinces The LTPP program is an on going study of the performance
95. ardware and OS issues We are also counting on First Federal Corporation to be able to deliver backups from off site storage within forty eight hours of an emergency request Objectives The main objective is the restore the Dell 2900 to full operational status as soon as possible with minimal data loss A secondary objective is to keep key personnel at the FHWA and the TSSC informed about the problem and its resolution When possible limited functionality will be made available while permanent repairs are underway Concept of Operations SYSTEMS OVERVIEW The Dell 2900 has databases and files housed on a single server that is accessible either on location or via VPN As shown in Figure 100 a keyboard connected via KVN switch and VPN are the primary entry points into the system Insert figure here Server Access Diagram Risk Identification and Mitigation Internet Connection Failure Not applicable This unit is not longer on the list of machines approved by TFHRC IT for internet access Dell 2900 Software Services Become Unresponsive 203 Description Software services becoming unresponsive typically results in inability to access an Oracle instance Effect The effect is that users are prevented from using the affected Oracle instance Mitigation No mitigation is necessary Resolution First use Process Explorer procexp exe to see if there are any processes with high resource utilization T
96. artridge will be removed before the next backup requiring a cartridge or the amount of time to fill the media is unknown 97 Devices There are two devices for the LTPP server a DELL RD1000 cartridge drive and the TB storage unit The TB storage unit has a folder 0_Incremental_backups in which all the devices exist Each named folder is treated as a separate device Multiple folders are strongly suggested so that if one of a set of nightly backups fails it does not cause the other backups to be missed because the device is unavailable due to the failure Devices can be found and their contents inventoried under the Devices tab in BE as shown in Figure 70 Figure 70 Screenshot Device Options Moving files media in Windows Explorer or other tools does NOT have any impact on where BE things the files arc If files are moved between Back up to Disk Folders using Windows Explorer each of the affected folders should be inventoried using the Inventory option after all the moves are completed This also includes after deletion of retired media See the Media section for a discussion of retired media Its recommended that cartridges be Ejected using BE rather than using Windows Explorer so that BE will track on line and off line data correctly 98 APPENDIX H OFF SITE BACKUP PROCESS FIRST FEDERAL CORPORATION FFC All activities are done through the FFC web site They consist of logging in checking
97. as a PowerVault RD1000 which accepts 1 TB hard disk cartridges All of this is protected by 2200 VA UPS The Dell R515 was received with Windows 2008 R2 installed As a test of recovery procedures the OS was removed and a bare metal recovery done of the OS and the applications on the Dell 2900 Following a successful recovery all existing applications and data were transferred to the new server The two systems were operated in parallel for a month to verify that all applications were working correctly Backups The original central server and DPW use Symantec Backup Exec and RD1000 500GB and RD1000 1 TB Disk cartridges for backups The replacement units use 4 TB hard drives for large backups both annual submissions and AIMS While the backup software versions and devices vary both machines use a similar backup strategy That is regular backups of the Oracle database to removable media Due to the fact that most information entered into the database comes on either paper forms or electronic data sets the risk of data loss is low Therefore a bi weekly backup to removable media has been chosen as the proper balance between risk and cost for the central server A weekly backup to removable media is done for the DPW since the data and software are until continual revision Incremental backup policies vary by location Due to the less frequent updating of the central server incremental backups are performed on alternate days rather than
98. as current at the time this document was last revised Managing Roles The Create Role Alter Role And Drop Role commands are used to create and remove roles Information about these commands is available in the Oracle Database SQL Reference 10g Release 2 manual This section will concentrate on maintenance of the existing PPDB roles Whenever a table is added or removed from PPDB the corresponding privileges of the SELECT ONLY and POWER USER roles need to be updated When a table is added the roles need to be updated to grant privileges to those users assigned these roles GRANT SELECT ON newtable TO S GRANT INSERT UPDATE TE ON newtable TO PONER U These statements grant privileges to the roles instead of individual users Any user who has the role will automatically get the privilege This is similar for revoking privileges When a table is deleted the privileges should be revoked before it is deleted REVOKE S REVOKE INS etedtable FROM SELECT ONLY DATE DELETE ON fedtable FROM POWER_USER 14 EARLIER ORACLE VERSIONS LTPP has been using Oracle software since version 5 at the beginning of the program LTPP servers managed by the TSSC have versions 10g 115 and 12c installed This section discusses the applications and references requiring use of 10g or 11g Several LTPP standalone applications that run against the database for producing the standard data release and other deliverab
99. atabase objects tables views indices tablespaces users etc This software is installed on the server when the Oracle RDBMS is installed and led on the workstation when the Oracle Client is installed This tool is no longer commonly used This software has been removed and replaced with Oracle SQL Developer SQLPlus Worksheet The SQLPlus Worksheet is an Oracle client tool that allows the user to type SQL Plus commands in an input window and see results in an output window Commands are executed by pressing or CTRL Enter Previous commands can be accessed by pressing CTRL p and next commands by pressing CTRL n Other shortcuts are available in this tool This tool has been replaced by SQL Developer and Apex in daily use Command Prompt SQLPlus and Oracle utilities can be executed from a Command window by users with permissions to access the server directly For example a sqlplus file sql can be executed with the following syntax sqlplus connectstring sqifile sqi The Oracle data export command can be executed as follows exp connect string parameters par where the parameters par file has all input parameters The same list of parameters can be included on the command line Work on the central server has not migrated to Power Shell although it is available PowerDesk Central Server PowerDesk is file management software similar to Windows Explorer Its advantage over Windows Explorer i
100. ate all the selections as shown in figure 30 The right slash indicates only some of the items are being included A check mark indicates all items are being included Uncheck show file details when making a replacement copy 69 1E the selection list is going to be a combination of new items and existing lists use the Load selections from existing list option This will bring up figure 31 or an equivalent list to pick from The Include Exclude option is not currently being used If the server were having files permanently deleted or only a specific period was desired in a backup or a differential backup was desired those capabilities can be executed through this selection option Figure 29 Screenshot Picking from existing selection lists 0 EGF 0008 tachi Drie ceases D rare tated Bahus ant DD Wek con edidic TAE pet reina sange de Dee nel corner ured Figure 31 Screenshot Selecting Resource Order Figure 33 Screenshot Priority Selection Defaults cp Ep Figure 34 Screenshot Selecting a Backup Device ES CoO T SS Figure 35 Screenshot Picking General Settings Backup Method 73 Figure 37 Screenshot Selection of Advanced Options Defaults Figure 39 Screensh
101. ault md1000_Owner 27s 20Manual_en us pdf Dell Support Storage All equipment was purchased with Dell Support The equipment is out of warranty and extended warranty has been purchased If support is needed the TFHRC Help Desk is the first point of contact If Dell needs to be contacted the following information may be useful lt Model Dell PowerVault MD100 Upper unit Service Tag located on upper right front of unit outside of enclosure 8825XL1 Express Code located below Service Tag 1911094765 Lower unit Service tag BROSXLI Express Code 25580828485 The contact information for Dell was wwe delLcom ProSupport Federal Government under suppor if doing troubleshooting or requesting help Phone 1 800 945 3355 A Service Tag will be needed when calling 220 Customer Acct 65207193 Customer Purchase Order DTFH6110F00023 Dell Purchase ID 2001802580973 Order Number 137363466 137363680 1 for each unit Order Number 137363466 137363680 1 for each unit Hard Drive Replacement Hard drive failure is identified by yellow lights in the affected drive bay New drives are no longer available from Dell nor are refurbished drives A single replacement drive is on hand for the next drive failure in the server or the attached storage Replacement drives can be obtained from xByte Technologeis http www xByte com The server and the storage units use 3 5 2TB SATA hard drives 7200RPM or slower disk speeds ar
102. b History to ensure it is in front Click on Expand upper right comer Click on Print lower right comer The printer selection menu will come up Figure 49 Adobe PDF should be the default printer If not select it and then click on Print The filename that comes up is that assigned by BE Figure 50 Before changing the file name verify that the correct directory Figure 46 in G Backup Backup_logs has been selected Enter the file name Using the file naming convention type the first letter of the file name It can be simpler to edit an existing file name than to type one from scratch The file name in this case where the Job History tab is forward will be the hist version The dates in the files are the date the backup being printed was STARTED not the date it finished If the backup failed a printout is still made to know which backups do and do not exist The History or the Log may be missing depending on the type of failure The file naming conventions for the various folders are as follows 0 Incremental weekday 1 work histlog yyyymmdd fail Incremental db 1 db histlog yyyymmdd fail 0 Incremental traffic I_traffic_hist log_yyyyymmdd _fail 1 Monthly edgiNN_hist log_yyyymmdd _fail 2 SDR gt gt gt 3 Weekly cdgfNN hislog yyyymmdd fail 3 Periodic AIMS_tape _mmyyyy Click on Save Figure 145 Screenshot Locations and names for job logs 258 Verify the fll ex
103. be evident other than by using Dell s Server Administrator or seeing status lights on the Effect There is no effect on server operations Mitigation No mitigation is possible Resolution Contact Dell Support and declare this a severity 1 critical situation because if the other internal drive fails before the bad drive is replaced and had time to rebuild the entire system volume is lost This will cause the server to be unavailable until the drives are replaced and reloaded from backups 30 minute to 1 hour down time will be required to replace the failed drive when the replacement arrives After the replacement there could be a period of several hours where performance is degraded due to the mirror being rebuilt Single Hard Drive Failure on the RAID 6 Array Description A single drive failure on the RAID 6 array D will not be evident other than by using Dell s Server Administrator or seeing status lights on the Effect There is no effect on server operations Mitigation The hot spare will take the place of the failed drive There are a few hours while the hot spare is being built where server performance may be degraded and the loss of an additional drive could cause the loss of the array Resolution Contaet Dell and have them ship a replacement drive under warranty When the replacement arrives hot swap the failed drive and the replacement The replacement drive will become part of the array and the hot spare will go
104. bility to change the PPDB data in addition to reading it The Power User Role has the Select_Only Role A user granted Power User does not need to be granted Select Only also The privileges granted to the Power User include Insert any Table Update any Table and Delete any Table This user may modify data in tables but not make major changes to the instance This use of any table functionality is to simplify the user s access to non standard tables in an LTPP instance The Junior role provides basic functionality in manipulating the database beyond the capabilities of the Power_User role The privileges include Comment Any Table Create Database Link Create Materialized View Create Procedure Create Public Database Link Create Public Synonym Create Sequence Create Session Create Synonym Create Table Create Trigger Create View Drop Public Database Link Drop Public Synonym Execute Any Procedure Export Full Database Force Transaction Global Query Rewrite Import Full Database On Commit Refresh Query Rewrite and Unlimited Tablespace The privileges remain limited to manipulating data and making individual rather than global changes to the database The Senior role provides the ability to manipulate or manage any object in the database including the tablespaces In addition to the Junior role the Senior role is granted the privileges of Alter Any Table Alter Tablespace Alter User Create Any Cluster Create Any In
105. ces are found through The names of these services all begin with OracleService and end with the instance name Stopping the service will stop the instance and starting the service will start the instance Starting and stopping can be done using the Windows control panel or using the net start and net stop commands in a command Window When Windows starts the services are automatically started and therefore the databases are running when the server starts When a Windows shut down is required during a power failure for example the services will shut down which will cleanly shut down the databases The other method of starting and stopping the databases is to use SQL Plus This is a good method to use for performing cold backups because it gives providesthe chance to perform some other housekeeping activities like creating backups of the parameters and control file Shutting down the database only requires the SHUTDOWN IMMEDIATI command but the commands below can make cloning and recovery easier SOLPLUS AS SYS CREATE PFILE BackuplnitlMSProd ora FROM ALTER DATABASE BACKUP CONTROLFILE TO TRACE SHUTDOWN IMMEDIATE Starting the database from SQL Plus is equally easy Just use STARTUP SOLPLUS IMsProd AS SYSDBA STARTUP 107 Database Database recovery can be a complex task It tends to be needed when a cloning or database upgrade is done incorrectly Somet
106. cess Data Electronic Data 432 Sheet Data j 431 1 Backup Raw Data 4312 Review DataRun Proprocessas 4323 Review Data Shoots 43 13 Archive Processed Database nata RIMS 4BManvally Upgrade Data 44Run CN Assign 45 Perform aC Checks ie Data Conect 48 Correct Data 47 Research Data Anomalies 4 12 Populate Folders Data Sheets Figure 4 Flow Chart Regional Scan Paper 1 45 Review QC Output i 49 Run Record Counts i 410 Upload Data to flow diagram replace Table 2 Filter CN and QC programs and chapters for active modules ata Module ilter Program d 2 Program Data Modul Filter Prog QC Prog Chapter Administration NA N Chapter 2 Admin QC Auto Disc CROSSSLOPELOAD Chapter 7 Padiast2 PADIASA2 DISPAD42 Climate CLMLOAD CimOwsde Chapter 12 ClmVsQC Deflection FWDOSLOD Chapter 5 FWDCOMM PDDXLOAD Friction NA FRICTION Chapters Maintenance NA Y mnro Chapter 9 Manual Distress JpecFault DIS Chapter 7 Profile PROFLOAD Rehabilitation NA Y uu Chapter 10 Tes Materials ape Y mre Cha
107. cle Database Administrator s Guide hip idocs oracle com database 121 ADMINitoc him Oracle 12 Oracle Database Reference on line hip docs oracle comm database 121 REFRNoe him 106 running the Alter_db_size_auto sq located in the synchronization directory This script identifies all tablespaces that are at 90 of capacity and generates the scripts to extend the associated databases Insufficient space may also be discovered on loading data during database synchronization Discuss the script that identifies 90 full tablespaces Add information on how failure may be found on loading data The syntax to increase tablespace size is ath and database name DBP AUTOEXTEND ON NEXT 256 MAXSIZE size in bytes KB MB Note that the name of the data file being extended is fully qualified and enclosed in single quotes Typically the LTPP databases extend data files rather than add them when expanding tablespace sizes With SDR 29 some tablespaces have reached the maximum allowable database file size and must be extended by use of multiple files As of SDR 29 the Undo tablespace could not be enlarged by expanding the initial database and additional files would need to be added to address storage issues Stop Start the Database There are two methods of starting and stopping the Oracle database instances One method is to start and stop the corresponding Windows service A second method is to use SQL Plus The servi
108. cluding video items of historical value and supplemental databases from outside sources Incremental backups to the external 2 TB drive while the standard data release SDR is being prepared are done Three days a week Monday Wednesday Thursday for DA LTPP Database FA Traffic and GA Working Storage Incremental backups when the standard data release is not being processed 1 Three days a week Monday Wednesday Thursday for DA LTPP Database 7 2 Two days a week Monday Thursday am for Traffic 3 Two days a week Wednesday Sunday am for GA Working Storage The SDR processing period is defined to include the week prior to the receipt of data for the SDR through the week after the shipment of the SDR or other copy of the data to FHWA for permanent storage Work station backups are done as a part of GA drive backups using Windows backup capability to copy the material to a shared drive on the server Work Flow The following lists the recurring activities associated with server backups The number of incremental backups to be verified is the only thing that changes depending on whether a release is being processed Daily verify that the scheduled backups have run Bi weekly Tuesday prior to full backup Run preparation for disaster recovery create dr file Preparing for Recovery Weekly Wednesday mornings 1 Verify the bi weekly backup ran for odd Tuesdays in the quarter 2 Prin
109. cuments as a data repository The data in the central database is updated quarterly from the production instance The AIMS files are updated annually and on the completion of regional contracts Data provided by outside contractors and not through the RSCs is typically loaded by the TSSC to the LDEP rather than directly to the central server LTPP Data Entry Portal A database production instance is maintained off site for data entry by all LTPP contractors Modifications to the database using development and test instances are also off site but also accessible to LTPP regional contractors via the web The server used is referred to as the Data Processing Workstation DPW The LTPP Data Entry Portal LDEP application includes data entry forms data loaders Quality Checks QC programs and utilities see figure 2 This application is the user s interface with the regional data The user can enter review edit and delete data from the database with this application Data entry and revision to tables not supported by forms or loaders is done via SQL scripts PPDB Data pube QC Programs Data Loaders gt Utilities 50 Figure 2 Schematic LDEP Application provides access to PPDB AIMS Data Entry Portal The AIMS Data Entry Portal ADEP application is a directory structure on the DPW to store AIMS data by region in Subversion environment The user can add
110. d for local testing and development They are not synchronized with the LDEP instances of the same name PPDB Data Modules The PPDB contains approximately 15 000 data elements that are organized into more than 600 tables These tables are grouped into data modules by data type For example the Rehabilitation Module contains 54 tables that contain data about various rehabilitation events that have taken place on each section Table 1 lists the modules and submodules that comprise the PPDB With the exception of the tables in the Administration module the first three letters of the table name Table Prefix identify the module to which a particular table belongs Tables resulting from data compilation views are excluded Tables used by the LTAS application are not included For more detailed information about each module and the data it contains refer to the Long Term Pavement Performance Information Management System User Guide IMS User s Guide listed in Appendix T Reference Documents Table 1 PPDB Data modules and submodules Data Module Table Prefix Descri Administration None structure of the database LTPPDD LTPPTD and coded values used CODES CODETYPES REGIONS It also contains the master test section control table EXPERIMENT SECTION the section location table SECTION_COORDINATES the section layering table SECTION LAYER STRUCTURE regional lookup table REGIONS and a table of
111. d is shown in figure 80 The SQL Language Reference and Administrator s Guide are two of the more frequently used manuals Also included further down the list under Supporting Documentation is an Error Message document PDF versions of the documents can be downloaded Copies are kept on the server in the Software Downloads folder under Oracle Oracle Database Online Documentation 12 Release 1 12 1 Database irato Work the database may be done at the command prompt in SQLPlus or using the DBA window in Oracle SQL Developer In the sections which follow examples are limited to use of SQLPIus The SQL Developer alternatives are discussed in the SQL Developer appendix The included SQL scripts were taken from Oracle SQL Developer The Oracle SQL Developer version used for this document was 4 0 3 16 for Java 170 55 Oracle IDE 4 031684 ORACLE ERROR MESSAGES There are multiple ways to interpret Oracle error messages A good starting point is using cither the Oracle Error Messages document or to go directly to ttps docs oracle com database 12 ERRMG toc htm for Oracle 12c or htp docs oracle com ed EL 1882 OL server 112 e17766 10c htm for Oralce 11g This will give you a basic understanding of what the error is and its possible causes Another technique in diagnosing errors is to look in the database s trace directory The alert log and trace files located there may provide extra deta
112. d of contractor facilities The server is also designed to be somewhat fault tolerant It features multiple power supplies and hot swappable hard drives While the uptime requirements of the LTPP server are not that high this redundancy helps to keep the systems running while waiting 8 parts to arrive On the down side without someone checking the system for faults on a regular basis the servers can keep running for a long time with failed parts This can lead to loss of data if for example one hard drive has already failed and another fails before the first failure is repaired The hard drives on the Dell R515 are joined together in a single RAID 1 array They are associated with the PERC 7000 The attached storage units controlled by a PERC H800 control as RAID 6 with 2 arrays of 32 TB capacity each The hard drives are partitioned into volumes and these volumes are set aside for different purposes Microsoft Windows Server is installed on the C volume The Oracle database files are stored on the D volume The server was delivered with the Windows Server 2008 R2 Operating System This was then configured to comply with NIST 800 53 Revision 2 Annex 1 since this is a low system When setting up the server various utilities such as the backup software led Then the Oracle database software was installed and the database instances created This server is run on an internal 12 TFHRC network and is not a domain controller
113. d outside of TFHRC Resolution The latest off site backups would be shipped to TFHRC where the external RD 1000 drive could be used to apply the backups to a Windows 2008 R2 server builtin the Amazon cloud The expected downtime would be approximately a week This largely depends on how long it takes to set up an LTPP account on Amazon and how long it takes to transfer the backup data into the cloud storage 207 APPENDIX AD HARDWARE SOFTWARE AND MAINTENANCE DELL 2900 The original central server was purchased in April of 2009 The central server was purchased with the vision to be a central repository located at a FHWA facility instead of contractor facilities It is located at FHWA s research facility TFHRC in McLean VA It is a Dell 2900 running Windows Server 2008 This server became the central server when operations were centralized at FHWA in the fall of 2009 Additional storage in the form of two MD1000 units was added in 2011 The server was replaced 2015 but not inactivated The server is also designed to be somewhat fault tolerant It features multiple power supplies and hot swappable hard drives While the uptime requirements of the LTPP server are not that high this redundancy helps to keep the systems running while waiting parts to arrive On the down side without someone checking the system for faults on a regular basis the servers can keep running for a long time with failed parts This can lead to a l
114. databases consist of a full backup of working directori done on alternate Tuesdays Incremental backups of the database instances and other folders are done during the week Quarterly backups of AIMS folders are done the 1 Wednesday of February May August and November The outline of the process is included in Appendix G Backups An Illustrated How To Instructions on how to use and maintain various elements of the backup software are contained in appendices of this document The software currently used for backups is Symantec Backup Exec 2015 for Windows Servers It is LTPP licensed software Updates must be manually identified on the software s website and manually transferred to the server as long as the system is not connected to the Internet The process for updates is discussed in SYMANTEC ADMINISTRATION AND TROUBLESHOOTING FHWA has contracted with First Federal for off site storage and a monthly delivery and pickup of storage boxes for electronic media Changes to the schedule must be done by the COR Additional information on this activity is found in Appendix Off site Backup Process Software updates are done on an informal schedule No updates are done automatically Microsoft software is updated monthly unless a critical update is flagged by the DOT Help Desk Anti virus software is updated at the same time as Microsoft software Oracle updates are done when a critical patch addresses a major security flaw or an operationa
115. de See Gating Help on page 187 Possible processor farre Troubleshooting Procesor on m Possible expansion card failure out eshooting Expansion Cards on page 172 Memory Taare See Troubleshooting Syster Memory on page 16 Possible video File See Gelling Help on page 187 land deve failure Ensure tat hc diskette drive and hard drive properly connected See Hard Daves on page WT Tor information on the drives installed in your system Possible USB failure Sec Troubleshooting a USE Device on page 158 Norway mailer deca See Troubleshooting Sytem Mene is Sys bean lur Help on page TET Mesas contigua eor S Troubleshooting System Memo va Code Causes Corrective Action 6000 Saree GG reset aT eee0 Your Sytem on page 157 forte apropia sale our system oem peni Gening Help on Dell Support Server All equipment was purchased with Dell Support Terms and information needed to obtain support follow Model Dell PowerEdge R515 Service Tag SXTSHO2 Express Code 121928399634 The contact information for Del is wow dell comProSupport Federal Government under support if doing troubleshooting or requesting help Phone 1 800 945 3355 A Service Tag will be needed when calling Custome
116. de to a hard drive when the system was taken off the network Selected files associated with LTPP deliverables were transferred to the new server for archival purposes WORKSTATION SOFTWARE The software on the workstation had all of the appropriate licenses when network ac discontinued in 2013 The work station software includes the standard distribution from FHWA Windows XP MS Office Winzip Roxio Creator Adobe Acrobat 9 and Symantec Endpoint Protection in addition to LTPP specific items The LTPP items include Oracle 10g and 11g clients TextPad Notepad The Oracle software is licensed by FHWA Notepad is freeware 29
117. depending on the type of failure Click on Save Verify the full expanded log was printed when it comes up in Acrobat Click on Close Return to Symantec and the next Job to be printed When all jobs have been printed delete them from the Job History section of Job Monitor Add notes on tape replacement printouts which may not occur File management discussion saving length to hold length to recycle PREPARING FOR RECOVERY at least a monthly basis after an off site tape is created a copy of both the disaster recovery file and a bootable image of the primary drive should be updated The former is referred to as dr file in Symantec BE terms The latter is referenced as an iso file an image to be copied to DVD in order to be used The creation of both items is done through the Intelligent Disaster Recovery Preparation Wizard Select Prepare for Intelligent Disaster Recovery in the Job Setup screen once to prepare the iso file and a second time to create the dr file There may be a User Account control to validate continuing If so Click on Continue The Wizard will come up The IDR option has been screen can remain unchecked before clicking on Next on the server so the block on the 86 ae BT Figure 52 Screenshot IDR Wizard replace 2010 Disaster Recovery Preparation Welcome to the IDR Preparation Wizard This wizen gues you rough
118. dex Create Any Materialized View Create Any Procedure Create Any Sequence Create Any Synonym Create Any Table Create Any View Create Role Create Tablespace Create User Drop Any Index Drop Any Materialized View Drop Any Procedure Drop Any Role Drop Any Synonym Drop Any Table Drop Any View Grant Any Role Lock Any Table Redefine Any Table Select Any Dictionary and Select Any Sequence The LDBA role contains most of the higher level privileges of the DBA role that may be needed to work with an LTPP instance The LDBA is assigned the Senior roll and the privileges Administer Database Trigger Alter Any Sequence Alter Any Trigger Alter Database Alter Profile Alter Rollback Segment Alter Session Alter System Analyze Any Analyze Any Dictionary Audit Any Audit System Backup Any Table Become User Create Any Indextype Create Any Job Create Any Trigger Create Cluster Create Indextype Create Job Create Profile Create Rollback Segment Debug Any Procedure Debug Connect Session Drop Any Operator Drop Any Sequence Drop Any Trigger Drop Profile Drop Rollback Segment Drop Tablespace Drop User Execute Any Program Flashback Any Table Flashback Archive Administer Force Any Transaction Grant Any Object Privilege Grant Any Privilege Manage Tablespace Merge Any View Restricted Session Resumable and Select Any Transaction Privileges assigned to roles can be changed over time so the previous listing should be taken
119. e IMSProd tdbaUN is the TRFDBA schema s DBA username tdbapwd is the password for that user sync date is the year and month for the current synchronization in the mm format used in the directory structure and instance abbr is the name of the folder for the specific instance being checked This will create two files in the 1_Synchronization yyyy_mum instance directory Instance ByTableL TPPDBA lis and Instance ByTable TRFDBA is The files should be imported into an Excel spreadsheet for record count comparisons with previous synchronizations Compare pre load record count to post load record count from last synchronization Compare pre load record count to the incoming export log to verify tables expected and identify any expected count decreases Clone the current production instance to K Historical_Database Y YY Y Sync MM_Month for temporary storage The local test and development instances may be cloned as well After the initial instance is prepped IMSTest by preference Verify a good backup of the current production instance Run RefreshDataTrucate sql at the SQLPlus prompt to lock tables and disable constraints triggers truncate clusters and truncate tables o Start SqlPlus Connect as the DBA to the relevant instance Type GRefieshDataTruncate sql to start the script Load data The 12c databases have been set up with single DBA user that is not the LTPPDBA or the T
120. e IMSTEst CONTROLFILE Copy G LTPP_Database copies IMSProd CONTROLEILE G VTPP Database copies IMSTestACONTROLFILE Copy F VLTPP Database VIMSP rod CONTROLF ILE V F LTPP_Database IMSTest CONTROLE TLE the data files from the originating database location to the new database py_G LTPP_batabade rmsPy6d DATAF LrPP Database INSTest OATAF ILE Create a SQL script to generate a new control file that identifies the clone database name and points to the new file location Note that once this script is created these steps only need to be repeated when new data files are added to the database It is however good practice to create a new script every time a database is cloned script to generate a control file is created with the following steps Find the trace file with the same timestamp as the alter database backup controlfile to trace command This will be located in the source database s trace directory An example would be G ALTPP_database diag rdbms IMSPROD MSProd trace imsprod_ora_4124 tre 1 Save the file as GALTPP_Database IMSTest Modifications CreateI MSNewControlFile sql 2 Edit the trace file with a text editor i e Notepad or Notepad 127 a Find the Set 2 RESETLOGS case section b Delete everything above the STARTUP NOMOUNT line c Find the ALTER TABLESPACE line and delete everything below it d Find the line with the semi colon after
121. e 17 Hard drive indicator patterns Condition Drive Status Indicator Pattern Identify drive preparing for removal Blinks green two times per second Drive ready for insertion or removal lom Drive predicted failure Blinks green amber and off Drive failed Blinks amber four times per second Drive rebuilding Blinks green slowly Drive online Steady green Rebuild aborted Blinks green three seconds amber three seconds and off six seconds Power Supply Inspection The power button on the front panel controls the power input to the system s power supplies The power indicator lights green when the system is on The indicators on the optional redundant power supplies show whether power is present whether a power fault has occurred see table 18 and figure 102 Table 18 Function of redundant power supply indicators Indicator Function Power supply status Green indicates that the power supply is operational Power supply fault Amber indicates a problem with the power supply AC line status Green indicates that valid AC source is connected to the power supply Image taken from Dell PowerBdge 2000 Systems Hardware Owner s Manual 2 T power supply satus power supply fut AC ne Figure 102 Illustration Location of redundant power supply indicators LCD Status Messages The system s c
122. e 18 The system status can be determined by checking the LEDs and reviewing the menus on the display panel There are both Standard and Advanced menus Under the Standard menue are Status Configuraiton Test and Diagnostics and an About Function The up and down keys are used to scroll through the menus Pressing enter permits viewing the submenus DRAC USAGE The DRAC or Dell Remote Access Card provides a visual interface to check on hardware batteries power supplies controllers hard drives and fans components includir from Operation Manual Smart UPS Uninterruptible Power supply Rack Mount 2U 780000500 VA 120 230 Vac 2200 VA 120 Vac 3000 VA 100 120 208 230 Vac com productsresourcelincludehechspee index efim hase sku DLA2200RM2UA amp tabsdocu mentation accessed 5 7 2015 58 APPENDIX E BACKUP POLICIES BI WEEKLY DRIVE ROTATION SCHEDULE The drive rotation for weekly backups is the 13 week schedule discussed in Inputs to Backup Policies In figure 19 the schedule for 2015 taken from the offsite_rotations spreadsheet is provided The underlined dates are the weeks that offsite pickup and delivery occurs The Week column is the number of the week in the quarter The Date value is the Tuesday backup date The Tape number is the label on the box and the tape of he tape to be used When a tape other than the 2 year archive is taken out of the rotation it is replaced by tape with the sam
123. e Administrator The database administrator maintains the database instances on the DPW This includes database backups schema modification and storage management 4 DPW Database Administrator The backup database administrator maintains the database instances on the DPW This includes database backups schema modification and storage management under the direction of the database administrator and or when the database administrator is unavailable Regional Database Administrator The regional database administrator oversees the maintenance of their regions data They have the ability to modify their region s data using SQL scripts after TSSC review Regional Program Manager The regional program manager is responsible for the oversight of the data collection activities within the region ADEP Power User The ADEP power user is responsible for maintaining the files in the regional ADEP repository PPDB Forms Edit The PPDB forms edit role is responsible for data entry and maintenance using the PPDB forms 173 LTAS The LTAS role allows the maintenance of traffic data using SSH Users The SSH users role allows users with a firewall account to use SSH to establish an encrypted tunnel to the database instances Technical Support Services Contractor The technical support services contractor role provides oversight for data collection and processing at the program level
124. e Backup Letter Frequency lt os 20 Continuous Weekly D Working 1710 Continuous Tncremental Storage Week E 1710 Quarterly Last and first quarter of year E T210 Annual Quarterly 6 Database 1670 Annual Tor core data Incremental 82 uploads Weekly Continuous for analysis Hr Recovery 3 Never Dell recovery arca Never var USB port NIA drives connected for upload of data X Backup device NIA backup cartridges Archive 50000 N A primarily backup Some materials have external drive backup on a topic specific basis The following is the ideal 13 week cycle using following logic A minimum of 13 weeks exists between reuse of a cartridge Every 4 cartridge used is stored off site for a 13 week period Off site storage occurs 2 weeks after the backup occurs recycled cartridge from off site is used the week after it returns from storage The last backup of a calendar quarter is stored off site for 2 years before recycling new cartridge is used for each backup stored off site for 2 years for the 1 two years This is cartridge 12 for the 1 2 year backup and A G for the next 7 in the cycle The ideal cycle has been modified because cartridge pickups are a fixed week of the month and not a fixed interval A spreadsheet offsite_rotationsNN xls saved in G_Task Order L NTask D IMS Hardware Softwarell Database Operation a_Operating the Database 0_Bac
125. e acceptable Smaller storage amounts are not If Dell drives are not available through xByte compatible replacements may be obtained based on a discussion of drives compatible with the MD1000 The two suitable drives for the LTPP system Hitachi Ultrastar A7K2000 SATA 2TB 7 2K RPM HUA722020ALA330 4 Western Digital 500 SATA 2TB 5 4K RPM WD2002FYPS I8UIB0 The drives are effectively plug and play They may be received as bare drives or with carriers If the drive is received without a carrier removed the failed drive and swap the drives in the carrier Replace the drive in the rack Open the DRAC and start the rebuilding process It will take several hours to fully restore the system UPS flagshiptech 1 0Q0HDSupportMatrx pa accessed 5 6 2015 Access through bup lstorelagship com and select PowerVault storage MD 1000 15 newegg com Produc Prodict aspx Item N82E 16822145310 accessed 5 6 2015 and identified as Hitachi GST Ultastar A7K2000 0F10452 2TB 7200 RPM 32MB Cache SATA 3 0Gb s 35 Internal Hard Drive Bare Drive hup newegg com Produc Prodict aspx Item N82E1 6822136365 accessed 5 6 2015 and identified as Western Digital WD RES GP WD2002FYPS 2TB 64MB Cache SATA 34IGb s 3 5 Hard Drive Bare Drive 21 The UPS with the Dell 2900 is an APC Smart UPS 2200 Error Reference source not found rack mount which has been discontinued by the manufacturer It is the lower
126. e data to compare and contrast once the new data has been loaded See Section Error Reference source not found of the NIMS Ops Manual for details of the cloning process Verify Media and Organize Upload Data When the public data extraction is received it is important to verify the contents of each upload and that all media are readable Once database staff has verified that all data files are readable uploaded files should be copied into the standard synchronization structure Review Selected Upload Files CN output files and record count output files can indicate data problems These files should be reviewed by the regions but can also be reviewed centrally if time allows Check for the following when reviewing CN output files Output for each table will include either a list of sections that had changes or a statement that No sections had changes Output for each table will include total number of records number of records with correct CN number of records with wrong CN number of records that had CN updated number of records with wrong CN that did not have CN updated The total number of records should equal either the number of records with correct CN or the number of records with correct CN plus the number of records with updated CN Other combinations of counts need to be investigated Contact regions with questions When reviewing record count output Look for tables that have many records at RS lt E Look for tables
127. e in INSTALLING ORACLE 12C Oracle comes with universal installer Once the software is downloaded and unzipped to a working directory clicking on the setup application begins the process sert Updates 149 C Domin ud cars po x D eme uem 150 151 essct uiti NERIS md d 152 Info on Oracle Home User Windows http docs oracle com database 121 NTQRF oh_usr htm hitp docs oracle com database 12 1 NTDBl install htm amp NTDBIO441 http idocs oracle com database 121 NTQRF oh_usr htm NTORE673 Erico inr dix See preuve enc eco Orao aban e cre Ponte corfu en ne 1 1 a Is ium earal Oracle net configuration asst to me rade nit Coaguraton Assistant is et ar ou rou e commen staps ses t hones te conan ou wouid ita tod Listener contauaton Naming tero s corquaton Locale Name restr Usage wey cnnsctons our Orada you must configu s rade elena The rade Net Contguratan Asis atowe outa recomngura rename tana Select umstyou wartta de as comnactens ose mace to crac
128. e number and an alpha character The off site column indicates whether or not the tape is sent off site for storage It is sent out the following month the indicated Thursday date The Recycle date is the date that tape will be reused It has only been filled in for tapes sent offsite so that the date it needs to be returned from storage can be identified The recycle date in all cases is the Date value on which the tape is used the next quarter Tapes identified with an asterisk are on a 2 year archive schedule and the year for recycling is 2012 for the 1 3 quarters and 2013 for the 4 quarter 59 Figure 19 Screenshot Bi weekly tape rotation for 2015 QUARTERLY TAPE ROTATION SCHEDULE The quarterly tape rotation schedule addresses the backups of AIMS material annual submissions and the indefinite archives INPUTS TO BACKUP POLICIES This material is derived from the FHWA July 2009 backup procedures document This system does not include indefinite archives Indefinite archives are covered in the annual data submission and the NARA submission The frequency of backup to cartridge by server drive has been established based on drive usage The inclusion of incremental backups for drives backed up weekly to cartridge was initiated when additional external storage was installed Table 10 Database Server Frequencies Drive Partition Name Total GB Update Cycl
129. e the change That SQL can be a good starting point in the generation of scripts Find tables and tablespace and owner select owner table name tablespace name from dba tables Find table ddl Set long 5000 need to have space to write out the answer Select sys dbms metadata get ddl Table tablename schema from dual 108 Add Remove Modity Field See ALTER TABLE in the Oracle Database SQL Reference 11 Release 2 manual for syntax options for this command Some examples are as follows ALTER TABLE tablename ADD neucolumn NUMBER 3 1 ALTER TABLE tablename DROP COLUMN oldcolumn ALTER TABLE tablename MODIFY smallnumber NUME The first statement would add a column called NEWCOLUMN as datatype NUMBER 3 The second statement would remove OLDCOLUMN from the table The third statement could be used to change SMALLNUMBER from a NUMBERG 1 to a NUMBER 4 1 Modify Keys Constraints See ALTER TABLE in the Oracle Database SQL Reference 1 Le Release 2 manual for syntax options for this command Some examples are as follows ALTER TABLE tablename ADD CONSTRAINT pk newtable PRIMARY KEY atate code shrp id USING INDEX TABLESPACE index tabi ALTER TABLE tablename DROP CONSTRAINT pk newtable ALTER TABLE tablename disable CONSTRAINT fi cab The first statement would add a constraint to the table This could be any type of constraint such as NOT NULL The second statement drops the constraint
130. ecifically or the FHWA IT unit Repairs if needed are to be coordinated with the COR Only items that are hot swappable are repaired by the LTPP Database Engineer Electronic copies of hardware manuals are located in Hardcopy hardware manuals and CDs with users manuals are kept in the right hand drawers of the desk with hutch in the server room These should not be needed The basic instructions on items to check before calling for maintenance are outlined in Appendices D Hardware Maintenance and Hardware Software and Maintenance Dell 2900 The following manuals are present for the hardware Dell 2900 Server MD Powervault storage APC Smart UPS 2200 original unit HP Procurve switch REGIONAL OPERATIONS Regional office operations are driven by the flow of data through the process shown in figure 4 All data entered into the LTPP Pavement Performance Database PPDB whether electronic or on paper forms goes through the same basic process Data coming into a regional office is reviewed often with the help of preprocessing software and loaded into the database by filter or by entry form using the PPDB element of the LDEP Itis then checked by the Quality Checks QC software Based on the results of the QC software the data may go through a correction or upgrade process Data is subsequently released to data analysts and researchers via InfoPave and the SDR more complete discussion of the process is includ
131. ed export and import lines in the export and import batch files 122 3 Check the script s output file to verify that a parameter file exists for each working a Ifa parameter file docs not exist because there are too few files to warrant a parameter file verify that an export line exists in the batch file to run the parameter files non standard_parfiles_instancename bat b Ifthe parameter file does not exist because the user has been added since the last synchronization create a parameter file and add a line for it to the export batch file or add an export command line to the export batch file Add the appropriate line to the import batch file 4 Check the script s output to verify that no unexpected files are associated with the operations users a If unexpected files exist create a parameter file s to transfer the files to the appropriate user i Add the parameter file s to the export batch file di Add the exported file s to the import batch file b Iexpected files exist that are not part of the designated list verify that they are listed in the appropriate parameter file for export 5 Run the export batch file non standard_parfiles_instancename bat 6 Review all log files from export a If any exports fail re run them at the command prompt and edit the batch file to have the working syntax b Hfany tables have 0 rows exported i Drop the empty tables Drop the tables with a script and log
132. ed in the Long Term Pavement Performance Data Entry Portal LDEP User Guide November 2012 The steps in the process are 1 Collect Data Many resource documents exist to support data collection activities Most collection activities are specified in directives from FHWA For example information on site selection site preparation and data collection for AWS sites can be found in directive AWS 01 2 Process Data The type and format of the data determines how the data is processed Some data is collected in electronic data files and some data is collected on a paper data sheet 3 Data may be electronic or provided on paper forms a Electronic Data Many data modules have at least some electronic data to process Table 2 and Table 3 lists the data modules and the filter program s used with each module as well as other module speciic information i Backup Raw Data Raw data files should be copied to ensure that mo data is lost in transit from field to office ii Review Data Run Preprocessors The header portion of electronic data files should be reviewed manually for required information Automated preprocessing software checks much of the electronic data for correct format and reasonable values For example FWD data is preprocessed with the FWDSCAN software Longitudinal and Transverse Profile data is preprocessed with the PROQUAL software This excludes the dipstick data 42 Collect Dala 43 Pro
133. ed in the appropriate column Level E output lists key fields for each error record and then an error message for each error encountered Error messages are numbered to facilitate looking up the corresponding check in the QC Manual The 29 QC Manual chapter corresponding to each data module is listed in table 2 and table 3 Research Data Anomalies QC output is used to identify records that have data anomalies Sometimes a data element is merely overlooked mistyped or can be found or verified on the paper data sheet or in other documentation d Correct or Manually Upgrade Data A record that is listed in the level C QC output has not been upgraded to RECORD STATUS C by the program This indicates that the record is missing one or more required fields However regional personnel may conclude that the missing data cannot be determined but that the remaining data is still useful In this case the failing record can be manually with SQL upgraded to level C Once all possible missing data is entered or records are manually upgraded the level C QC can be rerun and the new output can be examined Once the data manager is satisfied with the level C QC output level D QC can be run All records with RECORD STATUS that pass the level D checks will be upgraded to RECORD STATUS D Level D output will be used to make corrections to the records that did not pass level D checks In this case a data value may have b
134. ed up If this is the first time this collection of folders files has been backed up enter a descriptive name for the list of items under Selection list name to replace the Backup NNNNN naming Below the name write some text about the materials in the backup 242 Do not select show file details unless a by directory listing is absolutely necessary The list generated in the expanded job log will run to hundreds of pages Select the folders and files from each drive 4 The right hand selection block starting with Domains is not applicable for LTPP 5 It is recommended to save time and space that the recycle bin not be backed 6 System state and Shadow Copy Components should generally be included Egg eee Joe d Figure 123 Screenshot Starting a new job Selections If this is a copy or variant of an existing backup incremental version of a full backup for example click on the arrow beside Backup NNNNN to get a list of existing selection lists figure 123 Clicking on the preferred list will populate all the selections as shown figure 30 The right slash indicates only some of the items are being included A check mark indicates all items are being included If the selection list is going to be a combination of new items and existing lists use the Load selections from existing list option This will bring up figure 31 or an equivalent list to pick from The Include Exclude option is not currently b
135. ed use Backup mode corv copyfiag s 3 what to COPY for files default is COPY DAT DeData AeAttributes ToTinestanps SsSecurity NTFS ACLs info UaUditing info ncobY T COPY Directory Timestamps SEC copy files with security equivalent to COPY DATS COPYALL COPY ALL file info equivalent to COPY DATSOU NOCOPY i COPY NO file info useful with PURGE pa tory alent to E plus PURGE files delete ts RASHCNET 2 add the given Attributes to copied files 3 2 RASHCNET CREATE CREATE directory tree and zero length files only 1256 Off very long path gt 256 characters supp 273 monin norum iis bnt fer dris su source run again when more than n changes seen Monitor source Fun again m minutes Time if changed Bun Hours times when new copies may be started Check run hours on Per File not per pass basis Inter Packet Gap ms to free bandwidth on slow Lines copy symbolic Links versus the target RASHCNETO RASHCNETO tile airs 91 8 P jx js ar duxi pen pere re E E vost options E jum E E Legging Options a D copy only files with the Archive attribute set Include only files with of the given Attributes set exclude files wit
136. een mistyped or it may need to be removed because it is obviously wrong However if the data correct and is just outside the expected range the record can be manually upgraded to level D Level D QC can be rerun to recheck and upgrade corrected records the data manager is satisfied with level D QC output evel E QC is All records with RECORD STATUS that pass the level E checks will be upgraded to level E Records that do not pass will be listed in the level E output and that will be used to make corrections and upgrades Level E QC can be rerun to recheck and upgrade corrected records Manual upgrades are performed using the Browser Application and QC output files to generate manual upgrade scripts for qualifying records Refer to Browser documentation for more information on how to manually upgrade qualifying records 6 Run Record Counts The record count program rec cntexe can be run on the PPDB at the regional or program level to create a report showing how many records in each data table have advanced to a record status 7 Populate AIMS Folders Directive GO I 170 or the most recent versions also provides instructions on electronic format file naming conventions and directory 30 structure for the submission of files to the Ancillary Information Management System AIMS a1 APPENDIX A ROLES AND RESPONSIBILITIES TFHRC SERVER The Disaster Recovery DR and Continuity of
137. eing used If the server were having files permanently deleted or only a specific period was desired in a backup or a differential backup was desired those capabilities can be executed through this selection option Figure 124 Screenshot Picking from existing selection lists EDGR 2year etter en qiy Pe Coenen bach Drie ceases D rare tated Bahus acorn DD vest con pet Cie nel corner ured rel Figure 126 Screenshot Selecting Resource Order 245 Figure 128 Screenshot Priority Selection Defaults 246 cp Ep Figure 129 Screenshot Selecting a Backup Device S Oo T wae Figure 130 Screenshot Picking General Settings Backup Method 247 Figure 132 Screenshot Selection of Advanced Options Defaults 248 Figure 134 Screenshot Advanced Open File Options Used 249 Figure 136 Screenshot Identifying Pre and Post Commands 250 ifi ig init Figure 137 Screenshot Archive Method Selection 251 SETTING UP A RECURRING JOB set up a recurring job the same steps of the job set up process as followed with the exception of the schedule selection MANAGING JOBS The upper left box General Tasks on the Job Setup screen has several functions that may be useful Delete Propert
138. el sonic ramas econ al Selec whatyou want f C ones Each Oval or senice nas sence ame An Oracle Saas s sence name 1a normals datas name Entar ta sonca name of atas or aiher sca mu want 160 To communicate wine astanase across a near use Saladi tne proc vse or e database ou aniio acess lec 4 Te communicate wi he using TCPAP acl database computers nostna Etere ost ferta conputr he cles Hostname ATCPIPpotumberis also raqured In most aces th standard used Usetne standard pot umber 82t 1 Car 161 Wat while ha oral Nat Cenfguralon Assistant ies to usina einmal ou d o Enleranams tormis net serie name Theoret Conourton stet has 4 gt 162 Would ou ike to contigurs anotar net name 4 e gt ORACLE 12 SHY ee 12 gem aco bat 167
139. em Cooling Ambient of acceptable range Problems El Temp Memory has exceeded acceptable See Troubleshooting System Cooling Memory temperature and has been disabled Problems to prevent damage tothe components E1210 CMOS Bat CMOS battery is missing orthe See Troubleshooting the System Battery voltage is of acceptable range E1211 ROMB Bat RAID batery is either missing Reseat the RAID battery Sce Replacing bad or unable to recharge due to SAS RAID Controller Daughter Card thermal issues Banery and Troubleshooting System Cooling Problems Ell XX PwrGd Specified voltage epulator has Getting Help filed E1229 CPUS Processor VCORE voltage See Getting Help VCORE regulator has failed E1310 RPM Fan RPM of specified cooling fanis See Troubleshooting System Cooling oat of acceptable operating range Problems E1313 Fan The system is no longer fan Check control panel LCD for additional Redundancy redundant Another fan failure will scrolling messages See Troubleshooting put the system at sk System Cooling Problems heating E1410 CPUAIERR Specified microprocessor is Sce your system s Information Update reporting system error Tech Sheet located on supportdelLcom forthe most curent system information I the problem persists sce Getting Help CPUS Specified microprocessor is of See Troubleshooting System Cooling acceptable temperature
140. er to establish extraction and delivery dates At the beginning of the month a synchronization is due Confirm delivery with DPW DBA lt Verify the location of the current copy of the record count program and edit batch files that reference it appropriately Verify create current quarter s folders for the server Server GAL Synchronization yyyymm working copy with subfolders for each instance Server K Historical_Database YYYY Syne MM_Month archive 116 Verify GA folders have all necessary scripts which include the core listed above and any scripts parameter files or batch files used in the prior synchronization by instance receipt of files Copy off logs and other supporting export information to synchronization folders in Gand KA Verify that all tables views and materialized views are in the correct schema See the section on Instance Review for details of the process to be completed Doa record count of existing instance tables To run the record count program on the existing data for the LTPPDBA and TREDBA schemas Ensure no tables exist by running purge recyclebin and committing it Open a command prompt window in the directory 1 Synchronization and type the following command Recct syne dbaUN dbapud db tabi tdbapwded gt ync date instance abbr Where dbaUN is the LTPPDBA schema s DBA username dbapwd is the password for that user db is the database instanc
141. ery Preparation Mizar Copy Disaster Recover intonation Fics E rates eec se ve ient the dette fide Succeed copia tho deser ecovey Hfomtenfleeto rere DR Gk Nent to cone ets ces Figure 64 Screenshot Completion of Creation of Copy of dr File Disaster Recovery Disaster recovery media updated You ee row ready to recover your compiterin the evert ofa the compute pectcsaderrecovey updated wie oach backup Ferodcaly save conpuerepecfe disaster raraton by unra ine ware are Copy Disaster foin fles ion ce Figure 65 Screenshot Completion of Disaster Recovery Preparation WORKING WITH MEDIA AND DEVICES Symantec uses media and devices in a way that is specific to the software Media refers to the files created in making backup Devices may be hardware or simply folder locations on a hard drive Media Media are accessed through the Media tab A media set is the collection of files media related 10 specific backup activity For the LTPP server media sets have been defined to correspond to the various backups There are other media sets Imported Retired Scratch On line Off line that are native to 94 E Peer 100 L Figure 66 Screenshot Main Media screen Each file that is part of a media set i
142. es are made quarterly and the other modifications are typically queries rather than modifications a bi weekly backup to removable media has been chosen as the proper balance between risk and cost for the central server Incremental backups are performed on alternate days rather than daily The following guidelines are provided to assist in checking the Dell 2900 for problems before they result in data loss It is recommended that these checks be performed on a weekly basis concurrent with the server backup process Server status Software inspection can be accomplished by using the Dell Remote Access Card DRAC discuss basics Since the physical inspection outlined in the following sections is usually sufficient there is really no need to perform an inspection using the DRAC All of the relevant messages will appear on the LCD panel The Hard Drives Power Supplies and the messages on the LCD panel should be checked once a week A convenient time to perform this inspection is during backups since someone has to be physically at the server to mount the backup cartridge 209 The following sections contain excerpts from the Dell PowerEdge 2900 Hardware Owner s Manual A copy of the manual is included in the reference documents folder accompanying this document The basic procedure is to look for lights that are amber Each power supply should have a green power supply status and a green AC status indicator
143. foPave AIMS items combined with the PPDB database are vital to the program s mission because they represent the LTPP legacy HARDWARE The LTPP Program has provided servers and workstations to key contracts at various points over the life of the program Currently the only program provided hardware is the servers at TFHRC and the DPW This version of the document addresses four systems They are the central server the old central server the DPW and the workstation The central server is located at TFHRC and is the archival server for the LTPP program references to TFHRC server central server the software hardware or related activities are associated with the server that was purchased and installed at TFHRC in 2014 It may also be referenced as the Dell R515 or Dell PowerEdge R515 its model name 10 10 10 34 its IP address on the internal TFHRC 12 network or 192 168 2 26 its address on the LTPP network Documentation on itis contained in the main body of this document and in various single alpha labeled appendices Appendix A Roles and Responsibilities TFHRC Server references to the old TFHRC server refer to the server purchased in 2009 intended to be the original central server and archival location for LTPP data It may also be referenced as the Dell 2900 or Dell PowerEdge 2900 which reflects the model name 12 LTPP its name on the internal TFHRC 12 network 10 10 10 29 its IP address on the
144. ge uie meda and charae Woe bes E aang seran fe ira Disaster Recovery fies siang pocas Piense wai Figure 59 Screenshot Image Creation Messages CD Image created successfully You must bu tne flowing mage 50 S660 to eng ii paty sitae Te rage Forodcaly zave ho comouterepocfie daaar roosvoy Pomaton by unning te ard ser the Tep Disaster coven rfomalon fies optom Note Fharnare 5 5 crvere cange on the beng morale CD irage by seecing Eecabe CD mage for use wih CD Vies om War 91 Figure 61 Screenshot Identification of Image File Name and Location vdr Files inteligent Disaster Recovery Preparation wizard Croate IDR Boot Moda Taza now ot odatrat i be used dung a dear eco Figure 62 Screenshot Selecting the Disaster Recovery File Option in the IDR Preparation Copy Disaster Hocovery information Flos m gt Copy syta epoche fleo to the coxinston oido IDR oomp tr epoce diaasterracoveyntornaton fora Window computor sher a T lbacapispefamed Seect he DR es you wert a copy PSERVER DE HED TRESERVER Vindar 41 1 SSC Figure 63 Screenshot Identifying Computer and Location for File Recov
145. general section comments COMMENTS GENERAL Automated AWS This module contains data collected by the Weather Station LTPP program from automated weather stations installed on some SPS projects Climate This module contains data collected from offsite weather stations that are used to compute a simulated virtual weather station for LTPP test sections or project sites Data in this module are updated at 5 year intervals and was last updated in 2013 with data through 2012 MERRA This module contains data based on Modern Era Retrospective Analysis for Research and Applications MERRA a model developed by the National Aeronautics and Space Administration Data Module Table Prefix Dynamic Load Response DLR This module contains dynamic load response instrumentation data from SPS test sections located in North Carolina and Ohio Ground Penetrating Radar GPR This module contains Ground Penetrating Radar GPR measurements performed on a subset of LTPP sections which provide an estimate of layer thickness variations within the monitoring portion of the test section Inventory INV This module contains inventory information for all GPS test sections and for SPS sections originally classified in maintenance and rehabilitation experiments Maintenance MNT This module contains information on maintenance type treatments reported by highway agency
146. h any of the given Attributes set exclude Files matching given names paths uildcards Exclude Directories matching given names paths exclude Changed files MAXimum file size exclude files bigger then n bytes Maximum file AGE exclude files older than n days date MINimun file AGE exclude files never than n days date Vf n lt 1900 then days else YYYYMMDD date exclude Junction points normally included by default assume FAT File Times 2 second granularity number of Retries on failed copies default 1 million Save Rin and in the Registry as default settings walt for sharenanes To Be Defined retry error 67 List only don t copy timestamp or delete any files report all extra files not just those selected produce Verbose output showing skipped files include Pull Pathnane of files in the output 24 i output status to LOG file append to UNILOG file 2 output status to LOG file as UNICODE overwrite existing E95 output status to LOG file as UNICODE append to existing Ee AIMS backup were made using the following two batch files BackupAIMS lyyyyMM cmd and BackupAIMS2yyyyMM cmd Each batch file was structured to format and name the cartridge and then copy folders so that the maximum amount of space would be used on the cartridge A log file was created for each folder backed up Back pAIMS lyyyyMM cmd Format J V AIMS1 10
147. he files that will be lost are not critical to operations Mitigation A copy of the latest drive D backup is kept on an attached USB drive as well as removable RD1000 cartridges Resolution Contact Dell and have them determine the cause of the failed array After the failed drives and or controllers are replaced copy the files from the latest backup onto the drive It will also be necessary to have the four cartridges that make up ADEP backup returned from Iron Mountain off site storage The contents of those cartridges will need to be copied onto D and then the incremental backups from drive will need to be applied to restore ADEP to point that is within 1 day of the crash The Oracle database instances will need 10 be restored from the RMAN backups on C and the logs applied to roll the databases forward until the point that the Array failed If for some reason the MySQL instance does not come up it can be recreated using the backups located in CNbackup The expected downtime would be 2 to 3 days Most services could be restored by the second day with ADEP being the last service to come back online ities at TFHRC Description This would be a catastrophic event that completely destroys the facilities at TFHRC along with the server and on site backups Effect No services provided by the TFHRC server would be available Mitigation Off site backups stored outside of TFHRC Resolution Retorign the server would foll
148. he resources to check are CPU Disk 10 and Memory In normal operation the CPU rarely exceeds 15 the Commit Charge is rarely over 20 GB and the Disk Bytes is typically under 100 KB s If a process is found to be causing the server to exceed these limits determine if it needs to be killed or restarted If resources were not the problem check the following table to determine which service needs to be restarted On rare occasions a restart of the Dell 2900 may be required Table 16 Dell 2900 Service Diagnostics Service Description OraclcOraDbI Tg home2TNSListencr This listens for connections to the Oracle database instances Restart this service if having Oracle connection problems Restarting this service will typically not impact users greatly Restart takes about 30 seconds OracleServiceIMSDEV This is the development instance of Oracle Restart this service if Oracle is not responding This service typically takes about two minutes to restart Thi not restart after completion of backups or restarts following Windows software updates It is likely that a problem requiring the restart of this service may also require applying redo logs to bring the instance to a consistent state OracleServicelMSPROD This is the production instance of Oracle Restart this service if Oracle is not responding This service typically takes about two minutes to restart Thi not restart after complet
149. he table to be referenced by any user as if it were in their schema remove synonym the following command is used DROP PUBLIC SYNONYM delete bie USER ADMINISTRATION User administration in Oracle involves privileges passwords and storage It is important to understand the term user and schema are interchangeable Privileges control whether not the user can create tables or even connect to the database Passwords should be unique for each user in an instance and between instances as well Passwords for these users should be provided to the COR when an instance is established or the passwords are updated Storage controls amount of space user can take in their default tablespace Oracle Users The users in an instance can be identified with the following command Select username from allu The core Oracle users are SYS and SYSTEM Except for database creation cloning or repair no work should be done with these user names Other Oracle users may be created when an instance is created LTPP practice is to lock all other accounts for security purposes The accounts can be checked and locked using SQL Developer To manually lock an account use Alter user username password expire account lock This command also sets the password to expired so that even if the account is subsequently unlocked the user must know the old password and supply a new one to use the account LTPP Users Standard
150. ial or executive with the authority to formally assume responsibility for operating an information system at an acceptable level of risk to agency operations agency assets or individuals Assignment Of Security Responsibility The assignment of security responsibility role is responsible for the overall security of the central server Other Designated Contacts The other designated contacts role designates key contact personnel who can address inquiries regarding system characteristics and operation LTPP The LTPP role provides FHWA oversight of the LTPP program Table 4 TFHRC server role assignments Role Organization with Individuals in the Role System Administrator AMEC Backup System Administrator SAIC Database Administrator AMEC Backup TFHRC Database Administrator SAIC TSSC Program Manager Customer Service User ESCINC Technical Support Services Staff AMEC Tnformation System Owner FHWA Authorizing Official FHWA Assignment Of Security Responsibility Other Designated Contacts None LTPP See Team listing Help Desk FHWAIT http www fhwa dot gov researchytfhre programs infrastructure pavements Itpp whoswh 3 Table 5 TFHRC Server Organizational Contact Information Organization Contact Information Long Term Pavement Program Team LTPP FHWA DOT HRDI 30 6300 Georgetown Pi
151. ical Pavement Distress Analysis System PADIAS method Cross slope data is included in this submodule Rehabilitation This module contains information on rehabilitation treatments Seasonal 5 This module contains SMP specific data such Monitoring as the onsite air temperature and precipitation Program data subsurface temperature and moisture content data and frost related measurements Specific SPS This module contains SPS specific general and Pavement construction information for the SPS 1 through Studies 9 experiments Traffic TRF This module contains traffic load classification and volume data Test TST This module contains field and laboratory materials testing data A key table in this module is TST_LOSB which contains layer thickness and composition information based on measurements from the test section site Ancillary Information Management System The AIMS is a collection of information not contained in the PPDB such as raw profile and deflection data distress photographs and images reference documents experiment guidelines test protocols output files etc LTPP began to catalogue and archive data using the AIMS metadata with the intent to provide the public with information about the availability of the AIMS online so that they may request this data through the LTPP Customer Support Services Center A majority of the AIMS data is available through In
152. ies Run now and Test run Copy is a between server function and not applicable to the LTPP system nta 9 uem Figure 138 Screenshot Job functions accessible through Job Setup screen Delete does exactly what it says removes a job from the list Highlight a job and click on it The response will be the dialog box in figure 44 The selection list option exists to remove the information about what is being backed up Unless the list is used only for the job or is not anticipated to be used in the future it should not be deleted 352 you stre you want ta delete he fb Quarery AIMS aswel as alic te sehed oceurences Mete ol dating tha job you can pespone iby places ton had T7 Ifthe section bst used by tis oie re longe nute delete Figure 139 Screenshot Delete Confirmati dialog box Selection of a job and clicking on Properties provides the capabilities to edit an existing Job The dialog box in Figure 140 will appear after the selection and click Any item on the list may be checked and changed if necessary To save any changes click on Submit The revised properties should be saved for reference by creating a pdf in the associated backup log folder rem x p Dowie oa A emote deg tars tee ee Cs Figure 140 Screenshot Initial Properties dialog box The Run now option star
153. ify expected count decreases 7 Clone the current production instance to the local test and development instances 8 Run LoadData batch file to update the database 9 Run any materialized view scripts 10 Update any administrative and computed parameters tables that were received out of date or empty 11 Run RefreshData cmd 12 Run record counts 13 Clone record database to development and tes instances 14 Make copy of files from GA in current 15 Erase prior year s synchronization files for the same quarter 16 Return hard drive for use in next synchronization Process is standard import to dev then test then prod Clones not done unless working files from each instance have been exported for import Alter table for sizing Check counts log of import for failures Check Mview status refresh and run statistics as needed PROD as clean cop Confirm good backup of production database Once each backup is complete the system administrator SA should review the backup logs and verify that the backup completed successfully and that there were no errors that would cause problems restoring the database Before continuing with the data upload database staff should check with the SA to make sure a recent good backup of the database is available Clone the IMSProd Database to IMSTest 119 Cloning the production instance at this point in the process allows database staff to go back to a pre upload copy of th
154. ils about the error Internet options include searches and My Oracle Support which requires registration and a license for Oracle support for most activities A general web search may or may not provide additional information to resolve a problem DATABASE MAINTENANCE Database maintenance is associated with sizing and running the database DBA level maintenance can be done in SQL Developer using the DBA window This panel is the DBA option under View on the toolbar for SQL Developer Once connection exists a list of options will appear for the instance Install database Record of database upgrade Add tablespace Listener issues including services starting Determine Characteristics of Tablespaces Tablespace data dictionary views can be used to identify characteristics of tablespaces A list of the views can be found in the Oracle Database Administrator s Guide More detailed information on each of them can be found in the Oracle Database Reference The most useful views are 4 VStablespace names Files files associated with each tablespace Increase Tablespace Size When a tablespace fills up it will usually need to be increased in size to accommodate more data Original tablespace sizes were estimated and a maximum size was dictated to avoid tablespaces growing very large without the administrator s knowledge Determining which tablespaces have are approaching the maximum size may be done by Oracle 12 Ora
155. imes all it takes is the command RECOVER DATABASE When it works the Database Recovered message appears SOLPLUS rNserod AS sY RECOVER DATABASE Other times a slightly more complex version of the command is required That is RECOVER DATABASE UNTIL CANCEL In this case each of the three redo logs are tried until the one that was in use when the database crashed is encountered SQLPLUS 8IMSProd AS SYSDBA RECOVER DATABASE USING BACKUP CONTROL G NUTPP Database INsProd redo log G LTPP_Dat abase IMSProd redo02 log G LTPP_Dat abase INsProd redo03 log CAN of those two procedures will most likely recover the database If they do not an Oracle service request may need to be created to have them help in recovering the database or a recent backup may need to be restored TABLE MAINTENANCE Due to the requirement to keep several databases in sync it is not recommended that any tools other than SQL scripts be used to change the schema SQL scripts provide a repeatable means of making changes to the schema That does not mean that GUI tools such as Oracle Enterprise Manager OEM and SQL Developer are not useful Both OEM and SQL Developer have tools to show the SQL statements necessary to recreate an object For example it is possible to make a change to an object using the GUI and click Show SQL instead of Apply A window will pop up showing the SQL that OEM would use to mak
156. including materials testing monitoring maintenance and rehabilitation It also includes the tables for the LTAS application Production Instance The production instance is used for data entry validation and computations for Each region entering data has access only to the data for that region The and Customer Support Services CSSC have access to all data for validation and rights to center data collected or computed by others Data validation by the TSSC is done on a periodic basis with all active modules checked at least annually In the fall of 2009 the archival copy of the production instance moved to the Federal Highway Administration FHWA Headquarters at the Turner Fairbanks Highway Research Center TFHRC in McLean VA It is housed on a Dell R515 server with attached storage using Windows 2008R2 Server and an Oracle 12c database Development Test Instances Both a development database instance IMSDev and a test database instance IMSTest are included with the LDEP These instances or versions of the database are used to develop and test updates to the production system They are accessible to all users to test data entry via scripts Validated updates and data changes are subsequently applied to the production instance development database instance IMSDev and a test database instance IMSTest are also housed with the production database at TFHRC in McLean These instances of the database are use
157. ing Dell s Server Administrator or seeing status lights on the Effect There is no effect on server operations Mitigation No mitigation is possible Resolution Contact Dell Support and declare this a severity 1 critical situation because if the other internal drive fails before the bad drive is replaced and had time to rebuild the entire system volume is lost This will cause the server to be unavailable until the drives are replaced and reloaded from backups 30 minute to 1 hour down time will be required to replace the failed drive when the replacement arrives After the replacement there could be a period of several hours where performance is degraded due to the mirror being rebuilt Single Hard Drive Failure on the RAID 5 Array Description single drive failure on the RAID 5 array D will not be evident other than by using Dell s Server Administrator or seeing status lights on the Effect There is no effect on server operations Mitigation The hot spare will take the place of the failed drive There are a few hours while the hot spare is being built where server performance may be degraded and the loss of an additional drive could cause the loss of the array Resolution Contact Dell and have them ship a replacement drive under warranty When the replacement arrives hot swap the failed drive and the replacement The replacement drive will become part of the array and the hot spare will go bac
158. internal TFHRC 12 network or 198 162 2 25 its address on the LTPP internal network This server is not connected to the I2 network without prior notification to the TFHRC Help Desk Documentation specific to this server which is still operation is found in appendices with an Aa labeled appendix i e Appendix AC Disaster Recovery Dell 2900 The Data Processing Workstation DPW is a server maintained off site for production work It contains the working version of the database for use by LTPP s regional contractors This document contains limited information on the hardware and software elements of that system It only addresses the current hardware installed in 2014 Information on the system installed in 2009 included in earlier versions of this document has been removed The old system is inactive and retained off site for disaster recovery purposes This document does contain the security information continuity and disa recovery documentation for the DPW The information on hardware maintenance and troubleshooting is the same as for the central server Central Server TFHRC The central server was purchased in September 2013 and is located at FHWA headquarters at TFHRC in McLean VA It is Dell PowerEdge R515 running Windows Server 2008 R2 Standard SP 1 Two Dell PowerVault MD1200 storage units are attached The central server was purchased with the vision to be a central repository located at FHWA facility instea
159. ion of backups or restarts following Windows software updates It is likely that a problem requiring the restart of this service may also require applying redo logs to bring the instance to a consistent state OracleServiceIMSTEST This is the test instance of Oracle Restart this service if Oracle is not responding This service typically takes about two minutes to restart This service may not restart after completion of backups or restarts following Windows software updates It is likely that problem requiring the restart of this service may also require applying redo logs to bring the instance to a consistent state Single Hard Drive Failure on the RAID I Array 205 Description A single drive failure on the RAID 1 array will not be evident other than by using Dell s Server Administrator or seeing status lights on the Effect There is no effect on server operations Mitigation No mitigation is possible Resolution Replace the hard drive immediately using the drive on hand If a drive must be purchased failure of the second drive may result in having to do a complete rebuild from backups A 30 minute to 1 hour down time will be required to replace the failed drive when the replacement arrives After the replacement there could be a period of several hours where performance is degraded due to the mirror being rebuilt Single Hard Drive Failure on the RAID 5 Array Description A sing
160. ion query to post a batch as final The date in the message will be the pickup date Click Yes The response is 102 Microsoft Intornet Explorer Werte tested on pte Figure 79 Screenshot Receipt Confirming Batch Posting Click OK Getting boxes back Ordering boxes and labels OFF SITE BOX ROTATION While a general summary of content is included in the data stored on the site a detailed listing of any given box s contents is printed for inclusion in the container Locks combinations APPENDIX 1 SERVER RECOVERY TFHRC SERVER INTELLIGENT DISASTER RECOVERY Intelligent Disaster Recovery is the method included in Symantec BE to support file restoration It has two components a bootable disk from which the computer can be restarted or a new one brought up as a replacement and a tracking system The bootable disk should be created on at least a yearly basis or after a major system patch The tracking file should be updated weekly prior to the weekly backup RESTORING A FILE APPENDIX J ORACLE DBA QUICK REFERENCE This section includes instructions on a few functions that the DBA may encounter in normal operations There is no way to document all possible situations The on line reference manuals for Oracle 12c are located through https docs oracle com en database database html or at hups docs oracle com database 12 naviportal_5 hum A screenshot with the Database Administration section highlighte
161. ions Drive K contains the Flash Recovery Area in KALTPP_Database lt instance gt FlashRecoveryArea and the Archive Logs CALTPP_Darabase lt instance gt ArchiveLogs If drive C fails drive D contains all of the files necessary to recover the database If drive D fails the RMAN backups from drive C s Flash Recovery Area will need to be restored and the logs applied in order to recover the database Redo logs and archive logs are managed by the Oracle instance and allow recovery of the last transaction committed before the failure The RMAN backups are managed by two Windows Task Scheduler jobs The RMAN Level 0 Backup runs every Tuesday at 6 00 AM This job produces a full backup of the instance and removes old archive logs and backup sets that are no longer necessary The RMAN Backup runs every day at 10 00 PM It produces a differential backup of changes to the data base instances The drive C backup captures these backups for off site storage AIMS Repository and Historical Files The AIMS Repository and historical files are too big to fit onto a single cartridge As a complete copy of the AIMS exists on an external hard drive in the possession of Customer Service in addition to the copy on the DPW the decision has been made not to backup 38 video files A full backup strategy has been implemented for all other files The AIMS repositories reside in K Historical_AIMS lt YYYY gt where lt YYYY gt is year of the update
162. ious roles This section describes the roles required and the organizations providing individuals for those roles The assignment of individuals by role and their contact information is provided by quarterly memo to the COR who is the Information System Owner The contact information for the various organizations and individuals is also included The information is provided separately from this document to remove personally identifiable information and prevent obsolescence of this document The role assignments for this system are reviewed quarterly The organizations with people assigned to each role are documented in table 14 Contact information for each organization is provided in table 15 System Administrator The system administrator is responsible for maintaining the Dell 2900 in good working order This includes the operating system access control installed software backups and hardware Backup System Administrator The backup system administrator is responsible for maintaining the Dell 2900 under the direction of the system administrator and or when the system administrator is unavailable This includes the operating system access control installed software backups and hardware Dell 2900 Database Administrator The database administrator maintains the database instances on the Dell 2900 This includes database backups schema modification and storage management Backup Dell 2900 Database Adminis
163. isruption to LTPP data delivery in case of a system failure CONCEPT OF OPERATIONS Phase I Readiness and Preparedness Knowing that system failures are inevitable there are certain steps that can be taken ahead of time to ensure that services can be restored in a timely manner Backups The server contains two Redundant Array of Independent Disks RAID volumes The first volume is configured as a RAID 1 array consisting of two 1 TB Serial Attached Small Computer System Interface SAS drives This volume is the C drive and contains m the operating system as well as a copy of the database backups source code repository and archive logs This drive also hosts the bulk of the web applications The RAID 1 array can withstand a single drive failure and remain operational These drives are not hot swappable The second volume is configured as a RAID 5 array consisting of seven 2 TB SAS drives There is an eighth 2 TB SAS drive standing by as a hot spare This array can withstand a single drive failure and the hot spare will automatically take the place of the failed drive providing additional protection until the failed drive can be replaced These drives are hot swappable This array is the D drive and contains the database instances repositories and backup staging areas The diagram below shows the highlights of the backup process 178 Volume C Fea
164. itigation A copy of the latest drive D backup is kept on an attached USB drive as well as removable RD1000 cartridges Resolution Contact Dell and have them determine the cause of the failed array After the failed drives and or controllers are replaced copy the files from the latest backup onto the D drive It will also be necessary to have the four cartridges that make up ADEP backup returned from Iron Mountain off site storage The contents of those cartridges will need to be copied onto D and then the incremental backups from drive C will need to be applied to restore ADEP to a point that is within 1 day of the crash The Oracle database instances will need to be restored from the RMAN backups on C and the logs applied to roll the databases forward until the point that the Array failed If for some reason the MySQL instance does not come up it can be recreated using the backups located in C backup The expected downtime would be 2 to 3 days Most services could be restored by the second day with ADEP being the last service to come back online at Oak Ridge Description This would be a catastrophic event that completely destroys the facilities at Oak Ridge along with the server and on site backups Effect No services provided by the DPW server would be available Mitigation Off site backups stored outside of Oak Ridge Resolution The latest off site backups would be shipped to Oak Ridge where the e
165. ity of Operations Plan COOP describes how the Long Term Pavement Performance LTPP Program will continue to function when the DPW is unable to sustain normal operations The outage may be partial or total and may be the result of operator error software problems hardware problems or network connectivity problems Scope The scope of this COOP is the DPW Situation Overview The DPW is used for the central archival of data gathered by the regional contractors It provides the source of data disseminated to the public through standard media formats and on demand requests Original copies of the data are maintained on the DPW and within the regions Because the original data is not lost it is possible for the THFRC server to be unavailable for a period of time without causing all data dissemination activity to stop The primary risk to the system being unavailable is that delivery of data may be delayed beyond scheduled dates Planning Assumptions The LTPP Team has agreed that three days of downtime would not have an adverse effect on data delivery It has also been decided that if a hardware problem occurs with the server the server will be repaired Therefore there is not a failover system standing by As explained later we will rely on the Dell server maintenance and support agreement to repair the system in the event of a hardware failure Objectives The objective of this plan is to outline the steps necessary to minimize d
166. k to its role as a standby There may be a few hours of degraded performance while the replacement drive is being rebuilt Failure of the RAID I Array Description A failure of the RAID 1 Array would take down the server The system will not boot Effect Because the operating system is on C the system would not be able to continue operations and it would be impossible to reboot the server due to the lack of a boot drive e Mitigation A copy of the latest di backup isk attached USB de Resolution Contact Dell and have them determine the cause of the failed array Alter the failed drives and or controllers are replaced install the OS and Symantec Backup Exec 2010 From there you can restore the latest usable backup The Oracle instances will likely need recovery after drive C is restored due to the sudden crash of the operating system The expected downtime would be 1 to 2 days Failure of the RAID 5 Array Complete Loss of Description A failure of the RAID 5 Array D would leave the server running but the portals Redmine LTAS APEX and ADEP would be unavailable because they depend on database repositories on D The code repositories would still be ible if needed Effect Files on D will be unavailable There will also be a loss of data due to some files not being backed to stay within the limitations of our backup media The files that will be lost are not critical to operations M
167. ke McLean VA 22101 TFHRCIT FHWA DOT 1 6300 Georgetown Pike McLean VA 22101 SAIC TSI Lafayette Drive Oak Ridge TN 37830 AMEC Environment amp Infrastructure AMEC 12000 Indian Creek Court Suite F Beltsville MD 20705 301 210 5105 Engineering amp Software Consultants Inc ESCINC 14123 Robert Paris Court Chantilly VA 20151 APPENDIX B CONTINUITY OF OPERATIONS TFHRC SERVER PURPOSE The Continuity of Operations Plan COOP describes how the Long Term Pavement Performance LTPP Program will continue to function when the TFHRC server is unable to sustain normal operations The outage may be partial or total and may be the result of operator error software problems hardware problems or network connectivity problems Scope The scope of this COOP is the TFHRC Server Situation Overview The THFRC server is used for the central archival of data gathered by the regional contractors It provides the source of data disseminated to the public through standard media formats and on demand requests Original copies of the data are maintained on the DPW and within the regions Because the original data is not lost it is possible for the THFRC server to be unavailable for a period of time without causing all data dissemination activity to stop The primary risk to the system being unavailable is that delivery of data may be delayed beyond scheduled dates Planning
168. kup_Procedures was used to develop the annual calendar and container rotation schedules for the various backups The modification has been designed to avoid to the extent possible swapping backup cartridges in and out of storage boxes at time of pick up The 13 week cycle implemented uses the following logic A minimum of 13 weeks exists between reuse of a cartridge The cartridge for the 4 week of every month cartridge used is stored off site Cartridges from the 4 week of the 1 and 2 months in a quarter are stored off site for a 13 weck period Cartridges 4 week of the 3 months in a quarter are stored off site for a 2 year period Off site storage occurs the 3 week of the following month new cartridge is used foreach backup stored off site for 2 years for the 1 two years This is cartridge 12 for the 1 2 year backup and A G for the next 7 in the cycle The following has been designated the ideal quarterly cycle for backups using the following logic A minimum of a year exists between reuse of a cartridge Cartridges from the first three backups in a calendar year are stored off te for a year Off site storage occurs the quarter after the backup occurs A recycled cartridge from off site is used the quarter after it returns from storage The last backup ofa calendar year is stored off site for 2 years before recycling None of the quarterly backups are retained indefinitely APPENDIX BACKUP S
169. l issue that is preventing efficient LTPP operations All other software is checked quarterly for updates The software for security firewall etc is Symantec Endpoint Protection It is FHWA licensed software Itis not possible to update this software without an Internet connection The systems used by the LTPP program are expected to be FISMA compliant Elements of the security plans for both the central server and the DPW are included with this document All users of the systems are required to take DOT IT security training annually have their accounts locked Access to either system is subject to approval by the COR of the TSSC contract 15 Backups Frequency Specified by data type and hardware Tasking Task D Leader References Symantec Backup Exec Administrator s Guide pdf available with software saved in G Working_space Backup References on server Filing Task_D l_Database Operation a_Operating the Database 0_Backup_procedures schedule spreadsheet Task_D I_Database Operation a_Operating the Databasell First Federal off site schedule database Task Database Operation a_Operating the Database Sym ntec software used for notes references e mails and other administrative matters On the server GABackup References G3BackupiBackup logs pdf copies of the log files from all backups by type incremental weekly monthly periodic and backup selection criteria CAProg
170. le view and materialized view creation privileges These are the users whose tables vary over time and can be easily lost on cloning a database Prior to updating any instance changes in these schemas need to be identified and non standard tables exported for import after cloning The process to clean up schemas and preserve working tables has several steps identification of users identification of objects that do not belong to operations users in their schemas creation and editing of parameter files by user to either transfer files to another schema or save them for later import editing a batch file to run the exports and doing the exports Details of the process follow The scripts that should exist for each instance include schema_check_eabbrsql non standard_parfil s instanceabbr bat export batch file Instanceabbr_exp_schema_yyyynmpar Post_clone_load_instanceabby bat import batch file 1 Run the script schema check jnstancename sql This identifies all users and objects that do not belong to operations users It lists the files associated with data release users It also counts objects belonging to other users 2 Check the script s output file to see if users have been added or dropped based on the previous synchronization s script output Ifusers have been added add counts for that user to the count section and re run the script before proceeding b Hfusers have been dropped comment out the associat
171. le drive failure on the RAID 5 array D will not be evident other than by using Dell s Server Administrator or seeing status lights on the Effect There is no effect on server operations Mitigation The hot spare will take the place of the failed drive There are a few hours while the hot spare is being built where server performance may be degraded and the loss of an additional drive could cause the loss of the array Resolution Purchase a replacement drive When the replacement arrives hot swap the failed drive and the replacement The replacement drive will become of the array and the hot spare will go back to its role as a standby There may be a few hours of degraded performance while the replacement drive is being rebuilt Failure of the RAID I Array Description A failure of the RAID 1 Array C would take down the server The system will not boot Effect Because the operating system is on C the system would not be able to continue operations and it would be impossible to reboot the server due to the lack of a boot drive Mitigation A copy of the latest drive C backup is kept on an attached USB drive as well as removable RD1000 cartridges Resolution Contact TFHRC IT and have them determine the cause of the failed array After the failed drives and or controllers are replaced install the OS and 206 Symantec Backup Exec 2010 From there restore the latest usable backup The Oracle instances will like
172. les are compiled against Oracle 10g or 11g dlls As of SDR 29 the 2015 release most applications have been ported to 112 The affected applications include rec cntexe ExtractStandardDataRelease exe FHWASubmittal and the QC programs The central server has a copy of the Oracle 1 1g client so that the QC programs can be run The only QC program that needs to be run at TFHRC is the ESALCalcQC exe All other QC is run on the DPW FHWASubmittal exe has not been modified since it was updated for the Dell 2900 on which the latest Oracle version is 11g ExtractStandardDataRelease exe was modified for other reasons for SDR 29 and runs under Oracle 12c The Oracle 11g documentation can be viewed at http does oracle comv ed E11882_01 nav portal_4 him The layout is similar to that for Oracle 12c APPENDIX K DATABASE SYNCHRONIZATION The recurring task is database synchronization Frequency Quarterly January April July October Tasking PM2PUCo PL References Oracle import export functionality Oracle datapump Software Oracle Notepad or other text editor rec_ct exe LTPP utility Microsoft Excel Scripts alter db size autosql RefrshDataTruncate sql RefreshDataEnableConstrints sl User_review sql RecCtSync emd Filing Server GAI_Synchronizaton yyyymm working fles Server KHistorical Database YYYY Syne MM_Month record copy Coordination The PM PI will review the schedule at the beginning of the task ord
173. ltppiba identified newped replace oldpud cloned db trfdba identified by newpwd replace oldpwd custeupp identified by newpwd replace oldpwd cloned db system identified by newpwd replace aldpwd cloned db Watacheck identified by newpwd replace oldpud Drop OriginatingInstanceUser on delete cascade Grant power_user to Newlnstance ser identified by userpudy Grant select_only to datacheck identified by datacheckpwdj shutdown immediate create spfile from pfilej startup Connect to the database from which the clone was made and start it Connect sys syspwdBimsprod as sysdba Startup open Exit 1E the listener does not know of the service when the attempt is made to log on do the following at the command prompt Lsnreti stop Lsnretl start 129 APPENDIX M SQL DEVELOPER NOTES EXPORTING DATA DATABASE MANAGEMENT Tablespace Modificat With a connection to the appropriate database in the DBA window select the Storage option and then Tablespaces as shown in Error Reference source not found Under Tablespaces a list of all tablespaces will appear Selecting tablespace will bring up details on it list of data files free space information a list of objects in it a pie chart for usage and the SQL to recreate the tablespace A example with a partial list of objects is shown in Error Reference source not found Changing the tablespace size is done by selecting the Edit option on the Actions dropdown
174. ly need recovery after drive C is restored due to the sudden crash of the operating system The expected downtime would be 1 to 2 days Failure of the RAID 5 Array Complete Loss of Description A failure of the RAID 5 Array D would leave the server running but the portals Redmine LTAS APEX and ADEP would be unavailable because they depend on database repositories D The code repositories would still be accessible if needed Effect Files on D will be unavailable There will also be a loss of data due to some files not being backed to stay within the limitations of our backup media The files that will be lost are not critical to operations Mitigation A copy of the latest drive D backup is kept on an attached USB drive as well as removable RD1000 cartridges Resolution Contact TFHRC IT to determine the cause of the failed array After the failed drives and or controllers are replaced copy the files from the latest backup onto the D drive The Oracle database instances will need to be restored from the RMAN backups on C and the logs applied to roll the databases forward until the point that the Array failed The expected downtime would be 2 to 3 days ilities at TFHRC Description This would be a catastrophic event that completely destroys the facilities at TFHRC along with the server and on site backups Effect No services provided by the Dell 2900 would be available Mitigation Off site backups store
175. mand is GRANT Sele only to newuser Other roles are discussed in a later section Removing users is accomplished with the DROP USER command For example DROP USER newuser CASCADE will remove the user newuser from the database along with any tables owned by newuser The word CASCADE can be left off if the tables will be deleted manually before dropping the user This can prevent accidentally dropping tables owned by newuser and used by others Resetting a password can be accomplished with the ALTER USER command ALTER USER username IDENTIFIED BY newpas3y This changes username s password to newpass How to find out the tablespace assigned DBA Users Default tablespace temporary_tablespace How to find roles and privileges assigned user tab privs user user role privs Select grantee table name from tab privs where grantee Select from Sys Privs where grantee DATACHECK Select from DBA Role Privs where grantee CUSTSUPP How to find which users exist Roles are a way to group system and object privileges to control user access and security of a database It is more efficient to use roles when assigning privileges to users instead of individual privileges Privileges on objects can be assigned to roles This makes it simpler to assign the ability to select data from a new table by assigning that privilege to a role rather than assigning that privilege to every user in the insta
176. more than 24 hours due to the inclusion of the verify option in Symantec Backup Exec 2010 Recovery Media The CDs required to rebuild the server are located in server room in the top right drawer of the credenza They are in a brown accordion folder labeled DPW inside of a white Banker s Box that is in the center section of the credenza Return of off site backups can be requested from First Federal Corporation The information system owner would be the primary person to interface with First Federal Corporation The system administrator would be the secondary contact Dell Service The Dell PowerEdge 2900 is out of warranty The two MD1000 25 TB storage units are of warranty Necessary repairs are done by the system administrator with support from FHWA and TFHRC IT support Phase II Activation Decision Process The decision to implement will begin with the initial notification that the Dell 2900 is not functioning This will typically be on direct observation by the system administrator Upon identification the severity of the problem will be assessed If it is determined that the problem can likely be resolved by a restart of services then those actions will be taken before implementing this plan Alert and Notification IF itis determined that actions beyond a quick fix the information system owner and the TSSC program manager will be notified in person by phone or by e mail as appropriate This notification will incl
177. n for users to manager user properties Left clicking on a user name brings up a screen like figure 87 where account status can be checked In the example the account status is Open Right clicking on a user name or the Actions drop down brings up the list in figure 88 that allows editing properties for a user Selecting Lock User is the simplest way to lock an account Clicking on Apply on the next screen to appear will make the change Figure 87 Screenshot General user information in SQL Developer General Roles Sys rvs Object Prive Quotas Cn Create Le Es trop Use pre Paver lame rata Gran Figure 88 Screenshot Simple user edits in SQL Developer Managing Roles male meer 50L pow E p Granted aie 7 9 Jon pus eae Te er mcam rong ere nre bumeran 1 mae a 25 nmm E H aa H B D H paca H 4 um Grate TI Bn H 1 H is Is E H 8 t H H B Is H H fal Is St ES pL i H H H H 8 H H 136 up Sites 70 INSTALLING SQL DEVELOPER Enlerthe ful patra Ier java sre p TENE Ec Erter te tul pathname fr java ne ox 138 e de me deme Res Tose dm
178. n the Reference Documents subdirectory Accessing LTPP_Data_2013 pdf LTAS User Guide Bookshelf The 2010 version of the LTAS User Guide can be found in the Reference Documents subdirectory Analysis Vol 1 0 2010 05 26 pdr In addition a description of other LTAS documentation is included in the LTAS Bookshelf Traffic Bookshelf 2006 05 25 pdf Current versions of these documents can be requested from LTPP Customer Support OPERATIONS IT Security Data Repository Server IT Security Plan for the Long Term Pavement Performance Program December 2013 Hardware Manuals Dell PowerEdge R515 http www dell com supporv home usfen 9 product supportservicetag SXTSH02 manuals accessed 5 6 2015 DELL PowerEdge 2900 htp www dell com supporuhome us en 19 product support product poweredge 2900 manuals accessed 5 4 2015 190 DELL PowerVault MD1000 http www dell com supporv home us en 04 product support produc powervault md1000 manuals accessed 5 8 2015 DELL PowerVault MD1200 http www dell com supporv home us en 04 product support produc powervault md1200 manuals accessed 5 7 2015 Smart UPS 2200 DLA Model Smart UPS 2200 SMT Model http vww apemedia com salestools MMIS SHUQQZ MMIS SHUQQZ R2_EN pdi sdirect true accessed 5 7 2015 als oftware Symantec Backup Exec 2015 Symantec Backup Exec 2010 R2 htpj www symantec com business supportindex page content amp
179. n those actions will be taken before implementing this plan Alert and Notification If it is determined that actions beyond a quick fix the information system owner and the TSSC program manager will be notified in person by phone or by e mail as appropriate This notification will include any details that are known about the outage along with an expected duration if it is known This notice should be followed up with additional emails details become available Finally notice should be sent when the system is available for use Phase III Continuity Operations Essential Functions 39 The essential function of data archival can continue with only a slight degradation This is because the there are at least two other off site copies of the information on the TFHRC server Data extraction for general dissemination occurs during the March through April time frame which is the most critical period for on line functionality Down time primarily affects schedules in the extraction and review process Data extraction on request is a process that occurs in low volumes sporadically throughout the year Most responses are associated with material not available through InfoPave The few requests that cannot be handled with the SDR require access to AIMS and can be addressed with the copy maintained by Customer Support Downtime primarily affects turnaround time a response Essential Personnel Order of Succession
180. nce that needs access to that table Identifying Roles The roles that exi in an instance can be listed by using Select from dba roles The system privileges associated with a role can be listed by using Select from role sys priva The object privileges that are assigned to roles can be listed using from role The roles that are assigned to other roles can be listed using Select from role role priva The ability to assign roles to other roles allows for a hierarchy of privileges and permits assigning different groups of privileges to different users Three useful roles are DBA Select Only and Power User The DBA role is defined by Oracle and provides the ability to manage the database It is being deprecated To replace DBA on the TFHRC server three roles have been created Junior Senior and LDBA They include most but not all of the roles assigned to DBA The Select_Only and Power_User roles are maintained as part of the DPW They are included on the TFHRC server for compatibility in loading DPW files The role definitions on the two systems not the same The roles and privileges discussion in this document is limited to the TFHRC server The Select_Only role provides access to the database Create Session and the privileges necessary to read PPDB tables Select any Table It does not provide the ability to change any of these tables The Power_User role provides the a
181. nding by as a hot spare This array can withstand a single drive failure and the hot spare will automatically take the place of the failed drive providing additional protection until the failed drive can be replaced These drives are hot swappable This array is the D drive and contains the database instances ADEP repositories and backup staging areas The diagram below shows the highlights of the backup process 36 Volume H USB Drive pem OE Volume Full ADEP Repository oer Backup 4 RD1000 EX Cartridges Stored OfFSite Ss EE Figure 8 Schematic Backup Process Overview Replace with TFHRC equivalent or reference to backups section of backups and a Drive Backup Strategy The Seagate Backup Plus mounted on drive J is used to write backups for off site storage The system uses ATB drives for working off site backups A 4TB external hard drive mounted on Drive is used for incremental backups There are multiple types of backups taking place First there are the drive level backups Symantec Backup Exec System Recovery 2010 is used to back up drive C to the staging area H SymantecRecoveryPoints The backup of drive C starts with a full image backup Sunday at 10 30 PM with differential backups created at 10 30 PM Monday through Saturday The staging area maintai
182. ned db Alter user system identified by newpwd replace oldpwd cloned db Alter user datacheck identified by newpsd replace oldpw Drop originatingInstanceUser on delete cascade Grant power user to Newnstance ser Grant select only to datacheck identified by datacheckpud shutdown immediate create spfile from startup Connect to the database from which the clone was made and start it Connect sys syspudBimsprod as sysdba Startup open Exit 1E the listener does not know of the service when the attempt is made to log on do the following at the command prompt Lsnreti stop Lsnretl start 272 APPENDIX AJ ROBOCOPY SYNTAX Prior to acquisition of Backup Exce the Dell 2900 used Robocopy as the backup tool Since the annual submissions saved in that format have not been verified as replaced on the central server the syntax is provided here for easy reference Robocopy is a Windows utility for making copies The following lists all of the switches and their actions for the utility Usage 5 ROBOCOPY source destination file file options source sr Source Directory drive path Jer shaze path destination i Destination Di path ehaze patn file 12 File s to names uildcards default is S copy subdi es but not empty ones copy files in restartable mode 1 copy files in Backup mode i use restertable made Lf access deni
183. ns two full image differential backup sets The Backup Exec software also takes care of copying the current week s backup files to drive J to be moved off site This copy occurs as the full or differential backup completes The backup of drive D is handled differently since the backup media is not large enough to contain image backup Drive D is backed up on file by file basis to H SymantecRecoveryPoints LTPPDPW using Symantec Backup Exec 2010 There are several directories that are excluded from the backups such as the ADEP repositories and the LTPP database instances This backup is scheduled to run every day at 11 30 PM The backup job does not automatically copy files to drive J for off site storage when complete There is a Windows Task Scheduler job CopyIncrementalFileBackupToCartridge that copies the H SymantecRecoveryPoints LTPPDPW directory to JNLTPPDPW at 6 00 PM every day Application Backup Strategy The majority of the application on the server are COTS and installed on CA LTPP Specific applications are not developed on the server The backup period for LTPP applications run on the server is linked to the Oracle backup cycle Oracle Database Instances The control files and redo logs are written to drives D and KA in ALTPP_Database cinstance gt where instance is IMSProd IMSTest or IMSDev The control files contain the latest structure of the database files and the redo logs contain the latest transact
184. nt does not endorse products or manufacturers Trademarks or manufacturers names appear in this report only because they are considered essential to the objective of the document Quality Assurance Statement The Federal Highway Administration FHWA provides high quality information to serve Government industry and the public in a manner that promotes public understanding Standards and policies are used to ensure and maximize the quality objectivity utility and integrity of its information FHWA periodically reviews quality issues and adjusts its programs and processes to censure continuous quality improvement Technical Report Documentation Page 1 Report No 2 Government Accession No Y Recipient s Catalog No Sue S Report Date PAVEMENT PERFORMANCE May 2010 INFORMATION MANAGEMENT SYSTEM OPERATIONS MANUAL 6 Performing Organization Code Performing Organization Report Miriam Pitz and Tommy Clark revised Barbara Ostrom No Performing Organization Name and Address TO Work Unit No TRAIS Scientific Applications Intemational Corporation SAIC 151 Lafayette Dr Oak Ridge TN 37830 TI Contract or Grant No DTFH61 00 F 00046 T2 Sponsoring Agency Name and Address 13 Type af Report and Period of Infrastructure Research and Development Covered Highway Administration Operations Manual May 2009 May 6300 Georgetown Pike 2010 ev May 2010 November 2
185. nt one as of the pickup date is the one sent off site The most recent backup is retained on site For Customer Service at First Federal Corp Mr Matt Kough 301 548 9676 Office 301 963 8974 Fax Email Operations fidera com username 700 password FHA700 Badge ID the middle three letters of the user s FFC ID Setup date 1020 09 Work flow A batch in FFC is a collection of boxes of the same type that will be sent to FFC Returns automatically generated based on the inventory information at FFC 20 The second Friday of the month Set up batch es with the box es to be sent to FFC that are at THFRC Verify that all the boxes that should be returned have return dates in Inventory for the third Thursday Change the inventory if necessary The third Wednesday of the month send an e mail to TFHRC Security to notify them of the courier s visit The courier s name is not required FFC or First Federal Courier is sufficient The third Thursday of the month Add a batch if needed for a box that is being returned that day Mark up the box receipts with their label names to match against inventory after tape transfer The third Friday of the month verify FFC inventory versus receipts Regular visits Visits for return and pick up of media occur on the 3 Thursday of every month except when that date is holiday The expected pickup time is between 9 30 and 11 30 Either TFHRC Security or the courier will c
186. nty Plus Onsite Service Initial Year Dell Hardware Warranty Plus Onsite Service Extended Year s ProSupport Next Business Day Initial Year ProSupport Next Business Day 2 Year Extended 3 26 2015 to 3 26 2017 UPS The UPS for the Dell R515 is an APC Smart UPS 2200 figure 17 rack mount It is the upper of the two UPS units in the server rack Figure 17 Photos APC Smart UPS 2200 rack mount UPS front and back SMT model The model number is SMT2200RM2U The serial number is AS 1403141091 The unit has a three year repair and replace on all elements except for the battery There is a two year warranty on the battery The effective date of the warranty is the manufacturing date January 14 2014 In the event of battery failure a single replacement battery model RBC43 will be required From hitp vwwape comiproductsresourcelincludehtechspee index efm base sku SMT23UDRMQUAtotl wat 15 50 accessed 5 7 2015 57 1 Online LED 2 UPS On Off key 3 Battery LED 4 Site Wiring Fault LED 5 Replace Battery LED 6 Display interface 7 Up Down arrow keys Enter key 9 Escape key Figure 18 lustration SMT 2200 Smart UPS front panel Documentation is stored in the APC subdirectory of the Da ScratchSpace Software_Downloads folder A second copy is stored on the incremental backups hard drive in the folder Setup Manuals The front display panel is shown in figur
187. of the two UPS units in the server rack Figure 106 Photos APC Smart UPS 2200 rack mount UPS front and back DLA model The model number is DLA2200RM2U The serial number is 180838025983 The unit is out of warranty In the event of battery failure a single replacement battery model RBC43 will be required Documentation is stored in For routine maintenance look at the front of the UPS to verify that the UPS indicates a full charge and that itis not overloaded Additionally check the online LED and verify that the light is steady Any other condition requires checking the troubleshooting section of the user s manual The front display panel is shown in figure 107 The line of lights green on the left side of the panel indicates the amount of load on the unit The light of lights also green of the right side of the panel indicates the level of battery charge In the center of the panel are two buttons a test power on button on top and the power off button on the bottom The indicators between the left hand column of lights and the on off buttons are from top to bottom AVR Trim Online and AVR Boost The indicators between the on off buttons and the column of lights on the right side are from top to bottom Overload On Battery and Replace Battery Battery Disconnected Prom sup ape com productsresourcefincludehechspec_ index base_sku DLA2200RM2U amp tab doce mentation accessed 5 7 2015 Front Display Pane
188. on and click Any item on the list may be checked and changed if necessary To save any changes click on Submit The revised properties should be saved for reference by creating a pdf in the associated backup log folder Heese AIS mme eu 3 n GEL em E zo Quen peer EI Meier rie tide besar device d ael des Figure 45 Screenshot Initial Properties dialog box The Test run function checks that all of the pieces are present drives accessible to the user command files for pre and post job actions readable and so forth It will execute all of the elements of the backup job except the actual backup 79 The Run now option starts a job immediately rather than waiting for a scheduled time or setting up a schedule REVIEWING JOB HISTORY SAVING BACKUP REPORTS Printed copies of backup reports are created once a week to have a permanent record of backups made and their content Printed copies are made with Adobe Acrobat Job reports can be stored indefinitely in BE These guidelines have been established to set limits on the amount of information stored Figure 46 shows a series of folders on drive F Working Storage of the server that apply to backups Included are the folders Backup_logs and IDR E DES 9 pr nimm nerd one El mcs Mop pn
189. ontrol panel LCD provides status messages to signify when the system is operating correctly or when the system needs attention The LCD lights blue to indicate a normal operating condition and lights amber to indicate an error condition The LCD scrolls a message that includes a status code followed by descriptive text The codes can be interpreted in more detail using Table 1 6 page 18 of the Dell PowerEdge 2900 Systems Hardware Owner s Manual Server Troubleshooting Troubleshooting for the Dell Poweredge 2900 consists of checking that all hard drives are working and knowledge of the possible error conditions for the system The corrective actions references in table 19 apply to the sections of the hardware user s guide Table 19 Dell Poweredge 2900 LCD status message key Code Text Causes Corrective Actions SYSTEM 62 character string This message is for information only can be defined by the user in You can change the system string in the System Setup program Sce Using the System Setup Program the System Setup program The SYSTEM NAME displays under the following conditions The system is powered on The power is off and active POST errors are displayed E1000 FAILSAFE See Getting Help Call Support Code Text Causes Corrective Actions Temp Ambient system temperature is See Troubleshooting Syst
190. orks Central Server This is third party freeware for network management The software was installed to generate list of installed applications for IT Security documentation None of the other functionality is used Redmine DPW Redmine is a project management web application It is open source software distributed under the GNU General Public License v2 The software is used for track issues with data and software and track action items and operational activities TRACKING PROCESSES The DPW workstation contains the mechanisms to track data and programming issues with both the PPDB and LTAS and action items from various meetings Access to the various tracking areas varies as do permissions to make entries and modifications to those entries Issues Software Performance Reports SPRs Software Performance Reports SPRs have been used to document questions about and issues with the PPDB and all of its components since the beginning of the LTPP program In addition SPRs have been used to document new development for the LTPP applications SPRs can be submitted by regional or central users of the database system SPRs are recorded in the Redmine application at http portal tpp org under the Issues SPRs tab Users log into both the web site and the Redmine application After selecting the Issues SPRs tab and logging into Redmine the user selects a subproject in the LTPP Database System project The options for SPRs
191. orm iii Scan Paper Data Sheets Once paper data sheets have been reviewed and processed they are scanned and filed in accordance with Directive I 170 or the most current version 4 Assign Construction Number CN to Data Records Each highway section included in the LTPP study has a documented pavement structure when it accepted into the Program This structure is assigned a construction number CONSTRUCTION NO CN of 1 maintenance and rehabilitation activities are performed on the section the construction number is incremented and the changes are documented in the database These changes are identified by the new construction number CONSTRUCTION NO is a Key field in the EXPERIMENT SECTION table which contains one record for cach STATE CODE SHRP ID and CONSTRUCTION NO combination Occasionally rehabilitation event causes a section to move to a different experimental study or to go out of study For example an AC overlay placed on a section in the GPS 1 study could move the section to the GPS 6B study ince the pavement structure is critical to many types of pavement performance analysis each data collection activity must be associated with the correct structure and the correct Most data tables contain the CONSTRUCTION NO field As new maintenance or rehabilitation activities are added to the database this field can get out of sync with the correct pavement structure for that data record Therefore a
192. ort file restoration It has two components a bootable disk from which the computer can be restarted or a new one brought up as a replacement and a tracking system The bootable disk should be created on at least a yearly basis or after a major system patch The tracking file should be updated weekly prior to the weekly backup PREPARING FOR RECOVERY at least a monthly basis after an off site tape is created a copy of both the disaster recovery file and a bootable image of the primary drive should be updated The former is referred to as a dr file in Symantec BE terms The latter is referenced as an iso file an image to be copied to DVD in order to be used The Dell 2900 has a DVD R drive The creation of both items is done through the Intelligent Disaster Recovery Preparation Wizard Both items need to be done to be prepared for recovery Select Prepare for Intelligent Disaster Recovery in the Job Setup screen once to prepare the iso file and a second time to create the dr file There may be User Account control to validate continuing If so Click on Continue The Wizard will come up The IDR option has been installed on the server so the block the screen can remain unchecked before clicking on Next 261 ning pee Welcome to the IDR Preparation Wizard ys s repens bole Diar wa tha Wait
193. oss of data if for example one hard drive has already failed and another fails before the first failure is repaired The hard drives are joined together in a single RAID 5 array The hard drives are partitioned into volumes and these volumes are set aside for different purposes Microsoft Windows Server is installed on the C volume The Oracle database files are stored on the D volume When setting up the server the operating system and various utilities such as the backup software were installed Then the Oracle database software was installed and the database instances created The Dell 2900 server has had Microsoft Office loaded in order to be able to use Microsoft Access to process standard data releases In addition Symantec Anti virus Symantec Backup Exec 2010 Adobe Acrobat SQL Developer Notepad and Tortoise SVN have been installed HARDWARE The original central server is Dell PowerEdge 2900 with two 3 GHz Xenon E3450 quad core processors This server also has 32 GB of RAM an internal RAID 5 Array consisting of eight 7 200 RPM 1 TB SATA disks and an external RAID 5 Array consisting of thirty 5 400 RPM 2 SATA disks For backups it has a PowerVault RD1000 which accepts 1 TB hard disk cartridges and a for incremental backups of this is protected by 2200 VA UPS Server Setup The server setup for the Dell 2900 is detailed in HRD LTPPServerSetup200906 1 doc see Appendix Reference Document
194. ot Advanced Open File Options Used 15 Figure 41 Screenshot Identifying Pre and Post Commands 76 SETTING UP A RECURRING JOB To set up a recurring job the same steps of the job set up process as followed with the exception of the schedule selection To set various schedule options MANAGING JOBS The upper left box General Tasks on the Job Setup screen has several functions that may be useful Delete Properties Run now and Test run Copy is a between server function and not applicable to the LTPP system n T Figure 43 Screenshot Job functions accessible through Job Setup screen Delete does exactly what it says removes a job from the list Highlight a job and click on it The response will be the dialog box in figure 44 The selection list option exists to remove the information about what is being backed up Unless the list is used only for the job or is not anticipated to be used in the future it should not be deleted 78 sure you want ta delete he pb Quarery AIMS as wel as alic te sehadet occurences Mete ol dating tha job you can postpone iby place ton had he section bst used s bis ro longer ue delete Figure 44 Screenshot Delete Confirmation log box Selection of a job and clicking on Properties provides the capabilities to edit an existing job The dialog box in Figure 45 will appear after the selecti
195. ow the facility plan for restoring THFRC facilities The latest off site backups would be shipped to FHWA designated facility where the external USB drives could be used to apply the 46 backups to a Windows 2008 R2 server built in the Amazon cloud The expected downtime would be approximately a week This largely depends on FHWA TFHRC s recovery procedures 4 APPENDIX D HARDWARE MAINTENANCE The following guidelines are provided to assist in checking the TFHRC server for problems before they result in data loss It is recommended that the visual checks be performed on at least a weekly basis concurrent with the server backup process Checking the operational status of various server components can be done using the DRAC as discussed at the end of this appendix This should be done at least monthly The server is designed to be somewhat fault tolerant It features multiple power supplies and hot swappable hard drives While the uptime requirements of the central server are not that high this redundancy helps to keep the systems running while waiting on parts to arrive On the down side without someone checking the system for faults on a regular basis the server can keep running for a long time with failed parts This can lead to a loss of data if for example one hard drive has already failed and another fails before the first failure is repaired Hardware status is determined on a gross level by visual inspection as discu
196. panded history was printed when it comes up in Acrobat Click on Job Log to ensure it isin front Figure 51 Click on Expand upper right comer Click on Print lower right comer The printer selection menu will come up Figure 49 Adobe PDF should be the default printer If not select it and then click on Print The filename that comes up is that assigned by BE Figure 50 cote tem apd te e CE Figure 146 Screenshot Job Log screen Summary Form Enter the file name Type the first letter of the file name The file name in this case where the Job Log tab is forward will be the log version The dates in the files are the date the backup being printed was STARTED not the date it finished If the backup failed a printout is still made to know which backups do and do not exist The History or the Log may be missing depending on the type of failure Click on Save Verify the full expanded log was printed when it comes up in Acrobat Click on Close Return to Symantec and the next Job to be printed When all jobs have been printed delete them from the Job History section of Job Monitor Add notes on tape replacement printouts which may not occur File management discussion saving length to hold length to recycle APPENDIX AH RECOVERY FROM BACKUP DELL 2900 INTELLIGENT DISASTER RECOVERY Intelligent Disaster Recovery is the method included in Symantec BE to supp
197. plane See Installing the 1x2 Fiex Bay Drive Bracket PDB Cl The power distribution board Ensure that the power distribution board Cable control cable is unseated missing control cable is seated properly If the or bad problem persists replace the power distribution board control cable See Installing the Power Distribution Board E2010 No Memory No memory is installed inthe Install memory See Memory system E2011 Mem Config Memory detected but is not Sce Troubleshooting System Memory Er configurable Error detected during memory configuration E2012 Unusable Memory is configured but not Troubleshooting System Memory Memory usable Memory subsystem failure E2013 Shadow The system BIOS failed to copy its See Troubleshooting System Memory BIOS Fail image into memory E2014 CMOS Fail CMOS failure CMOS RAM not See Getting Help functioning properly E2015 DMA DMA controller failure See Getting Help Controller E2016 Int Controller Interrupt controller failure See Getting Help E2017 Timer Fail Timer refresh failure See Getting Help E2018 Prog Timer Programmable interval imererror_ See Getting Help E2019 Parity Error Party error Sce Getting Help E201A SIO Er SIO failure See Getting Help EXUB Keyboard controller failure See Getting Help Controller EXC SMI Init System management interrupt See Getting Help SMD initialization failure E2OID Shu
198. pter 3 MatAetLoud BstarQc MatLoad ba boa PGBinderQC UnboundSpecGravQc Transverse Profile TPRFLOAD TPROF Caper CrossSlopeLoad CrossSlopeQC TPRECLOD The asterisk in the OC program name Widicacs that there isa separate program for Tevel C Tevel D and level E QC quality checks For example TST represents TS1 C TST_D and TST E programs Table 3 Filter CN and QC programs and chapters for inactive modules ata Module ilter Program d gt Program Data Modul Filter Prog QC Progi Chapter Auto Distress PADIAS DISPADIO Chapter Automated Weather AWSLOAD aws Chapter 23 Station Monitoring NA MonDraisQc Chapter 27 Drainage Dynamic Load DIRLOAD AC Chapter Response DLRLOAD_PCC Ground Pener Radar Chapter 28 GPRLOAD M Manual Data Module Filter Program QC Program chats Inventory NA N Chapter 2 INV QC Seasonal Monitoring SMPLOAD Y Chapter 22 Program TDRMANUALLOAD T ManalQC SMPFRAUT Specific Pavement Chs 13 2L and Studies SINIT 4 125673 Chapter 26 SPS misc TRAF N Chapel TDSHEETS TrafficAnalysisQC TDSHEETS TDSHEET TDSHEETS TRELOAD TRE 5 5 The asterisk 1 the OC program name indicates that here isa separale program for evel C level D and level QC
199. question about the quality or correctness of the data QC programs are intended to run one level of QC checks at a time Once level C ha been run the level C output is checked for errors i e records that do not pass the checks Records are corrected or upgraded as necessary and level C QC is run again Once the user is satisfied with the level C output level D QC is run Level D output is reviewed corrections and upgrades are made and level E QC is run Once level E output bas been reviewed and the user is satisfied with the results the QC process is complete Execute QC Programs b Review QC Output Each time a QC program is executed an output file is generated listing each table and any records that did not pass the checks on that table QC output files can be reviewed on screen or can be printed Files can be very large and should be checked for size before printing All QC output has some general header information at the top of the file including date and time of run For each database table with QC checks the level C output has the table name and a column for cach key field and for each required field If a record is missing one or more required fields the key fields are filled in and an R is placed in the columns for the missing field s See for a sample QC output files Level D QC output is formatted much like level C except data elements that do not fall within the expected range are print
200. r Acct 20080702 Customer Purchase Order DTFAWA 11 D00004 Dell Purchase ID Order Number 1200337526 Server maintenance and support are as follows Dell Hardware Warranty Plus Onsite Service Initial Year Dell Hardware Warranty Plus Onsite Service Extended Year s ProSupportPlus www dell com prosupport regionalcontacts April 9 2014 April 10 2019 4 7x24 On site service with Emergency Dispatch Initial Year 4 hour 7x24 On site service with Emergency Dispatch 4 Year Extended April 10 2015 April 10 2019 ATTACHED STORAGE There are two attached external twelve bay MD1200 storage units discussed separately Each unit is populated with ten 4TB 7 200 RPM Near line SAS 3 5 inch hot pluggable drives Documentation is stored in the Dell subdirectory of the D ScratchSpace Software_Downloads folder A second copy is stored on the incremental backups hard drive in the folder Setup Manuals Figure 14 shows the elements and indicators on the front of the storage units The enclosure status LED should be blue indicating normal operation If the light is amber after the unit is turned on an error exists The drives are hot swappable and should be lit with blue lights under normal operation Enclosure status LED 2 Power LED 3 Split mode LED 4 Sync entiation bation E Tad aves Raine mode lich Figure 14 Ilustration Front view of Dell PowerVault MD1200 del com syppor home
201. ram Files Symantec Backup Exec IDR Data disaster recovery alternate path External Hard Drive MABackupMDR 2 ISO image for data recovery Data dr data recovery files MABackuplIDRUDR_ dr data recovery files MABackup DRUDR_iso 150 image for data recovery 4 Miles for incremental backups and the associated ind Server backups are done with Symantec Backup Exec 2010 R2 BE as of April 1 2011 Backups are done with zSymantec User as the user name This user has the privileges needed to create backups and restore files from backups if required 16 soars Figure 3 Screenshot Home screen for Backup Exec 2010 Cartridge backups are tracked using a spreadsheet which has the various schedules offsite_rotationsvN xlsx This file is maintained on the FHWA provided computer in the Task_D_IMS_Hardware Software directory The subfolder 0 Backup procedures is located under a Operating the database in Database Operation The adopted backup schedule for the TFHRC server was derived from a document written by SAIC in July 2009 and implemented to reflect operations at TFHRC and the off site storage schedule Full backups to removable media are done excluding the contents of recycle bins Bi weekly on Tuesday evenings for the CA and DA drives and selected folders on the FA and GA drives Quarterly on the first Wednesday of February May August and November for AIMS material ex
202. rd Figure 63 Screenshot Identifying Computer and Location for dr File Figure 64 Screenshot Completion of Creation of Copy of File Figure 65 Screenshot Completion of Disaster Recovery Preparation Figure 66 Screenshot Main Media Screen Figure 67 Screenshot Retiring Media Figure 68 Screenshot Select a New Media Set Association Figure 69 Screenshot Setting Media Set Properties Figure 70 Screenshot Device Options Figure 71 Screenshot Login Screen for First Federal Corporation Off site Storage Figure 72 Screenshot Secondary Login for First Federal Figure 73 Screenshot Error Message on Login to First Federal Figure 74 Screenshot Preparing to Add a Batch for Pickup by Figure 75 Screenshot Confirmation to Start Add Batch Operation Figure 76 Screenshot Confirmation of Successful Batch Posting as Pending Figure 77 Screenshot Query to Confirm Batch Verification Figure 78 Screenshot Confirmation Query to Post a Batch as Final Figure 79 Screenshot Receipt Confirming Batch Posting Figure 80 Screenshot Online documentation for Oracle 12c Database Administration Figure 81 Screenshot SQL Developer tablespace management Figure 82 Screenshot Object list under Tablespace in SQL Developer DBA window Figure 83 Screenshot Actions dropdown Figure 84 Screenshot Edit Tablespace dialog box SQL Developer Figure 85 Creating a new tablespace 89 91 91 92 92
203. resents the DIMM pair implicated by the BIOS E2113 Mem Mirror They system BIOS has disabled See Troubleshooting System Memory DIMM amp memory mirroring because it has e determined that one half of the mirror has too many errors 4 amp 8 represents the DIMM pair implicated by the BIOS E2118 Pall NB One of the connections in the Fully See Troubleshooting System Memory MemCRC Buffered DIMM FBD memory subsystem link on the Northbound side has failed E2119 Fatal SB One of the connections in the FED See Troubleshooting System Memory Mem CRC memory subsystem link on the Southbound side has failed 11910 Intrusion System cover has been removed Information only gt 3 ERRs Chk LCD overflow message Check the SEL for details on the events Log A maximum of three error messages can display sequentially on the LCD The fourth message displays as the standard overflow message 217 Code Text Causes Corrective Actions 11912 SEL Pall System Event Log is full of evens Clear the log by deleting event entries and is unable to log any more WI228 ROMB lt Warns predictively that the RAID Replace RAID battery See Replacing the ES battery has less than 24 hours of SAS RAID Controller Daughter Card charge lft Banery NOTE For the full name of an abbreviation or acronym used in this table see the Glossary Dell Support Server
204. roubleshooting Power Supplies specified power supply specified power supply is improperly installed faulty E1614 PS Status No poweris available from the See Troubleshooting Power Supplies specified power supply specified power supply is improperly installed or faulty Power supply voltage s out of Troubleshooting Power Supplies Predictive acceptable range specified power supply is improperly installed or faulty EIGIC PS Input Power source for specified power Check the AC power source for the Lost supply is unavailable oF out of specified power supply t problem acceptable range persists sce Troubleshooting Power Supplies E1620 PS Input Power source for specified power Check the AC power source for the Range supply is unavailable or out of specified power supply t problem acceptable range persists see Troubleshooting Power Supplies E1624 The power supply subsystem is Sce Troubleshooting Power Supplies Redundancy longer redundant If the last supply fails the system will go down E1710 Channel Thc system BIOS has reported an See Getting Help channel check error E1711 PCIPERR The system BIOS has eported a Remove and reseat the PCI expansion PCI parity error on a component cards I the problem persists see that resides in PCL configuration Troubleshooting Expansion Cards PCIPERR spice at bus amp amp device Slot
205. rts in Symantec BE after verifying pdfs exis 7 Insert cartridge forthe next bi weekly backup 8 Inventory the cartridge and verify it has space for next backup 9 Erase folders on cartridge NOT created with BE if necessary Windows Explorer 10 Format a new cartridge if necessary Incremental Backups Incremental backups are done using a folder on the external drive Incremental backups are retained for two weeks before being overwritten BE keeps track of the actual file retention Bi weekly Backups 19 Since the ORACLE instances must be shut down prior to backup this may be done by including a pre backup command script in the Symantec schedule and a post backup command script to bring them backup up Quarterly Backups Drives that only change with a release or a data transfer are backed up on a quarterly basis Off site Storage Frequency Monthly 3 Thursday established by COR Tasking Task Leader Task D References On line Help for First Federal software Filing Task D M Database Operationta Operating the DatabaseMl First Federal off site FHWA LTPP has a contract with First Federal Corporation FFC to provide for off site storage of electronic media The contract is managed by the COR of the LTPP contract changes to the contract scope or activities must be done through and by the COR Bi weekly backups and quarterly backups are both stored off site The backup created prior to the most curre
206. rus program It requires a manual update while connected to the internet to ensure all elements are current The LTPP Administrator does not have administrative rights to change the backup frequencies or folders for this software Symantec Backup Exec 2010 Symantec Backup Exec 2010 is being used to perform backups It requires manual updates to remain current Maintenance support ended in February 2014 No updates are being applied No upgrade will be installed Oracle Oracle was chosen early in the development of the IMS At the time it was one of the only tools which could handle large databases across multiple hardware platforms Oracle also had a forms and reports package for application development which was able to run on multiple platforms without recoding 24 The server is on an isolated network with access limited to LTPP personnel Oracle software updates have generally only been applied during major system upgrades The Oracle Database Enterprise Edition version 11 2 0 1 0 along with the administrative tools is installed for use with the PPDB on the Dell 2900 Add reference on location of documentation locally and on line SQL Developer This is Oracle software that provides Graphical User Interface GUD with which to view and manage database objects tables views indices tablespaces users etc This software is installed on the server when the Oracle RDBMS is installed This software replaces the use
207. s Form DOT F 1700 7 072 keprotuction of completed page sator SI MODERN METRIC CONVERSION FACTORS APPROXMATE CONVERSIONS TO STUNTS Spes Wiener Rew Wun By ooo iiem ENSE To a 3 nea E ES E x VOLUME EE ee 5 p TEMPERATURE eat das mo FORCE and PRESSURE or STRESS CONVERSTONS FROM STUNTS When Yor Kaw EL d TABLE OF CONTENTS LTPP SYSTEM OVERVIEW 1 LTPP PROGRAM 1 LTPP DATA 1 LTPP Information Management System IMS 1 Pavement Performance Database 4 Ancillary Information Management System 7 HARDWARE 8 Central Server TFHRC 8 Data Processing Workstation DPW 9 Backups SOFTWARE TRACKING PROCESSES Issues Software Performance Reports SPRS B Software Change Notice SCN Report B Software Deployment DPW B Action Items B RECURRING SERVER ACTIVITIES Backups Off site Storage HARDWARE CHECKS REGIONAL OPERATIONS APPENDIX A ROLES AND RESPONSIBILITIES TFHRC SERVER APPENDIX B CONTINUITY OF OPERATIONS TFHRC SERVER PURPOSE Scope Situation Overview Planning Assumptions Objectives CONCEPT OF OPERATIONS Phase I Readiness and Preparedness 35 Backups 35 Dell Service 39 Phase II Activation 39 Phase III Continuity Operations 39 Phase IV Reconstruction Operations 40 APPENDIX C DISASTER RECOVERY TFHRC SERVER INTRODUCTION
208. s This document describes in detail how the RAID Array operating system and Oracle databases were configured The highlights of the server setup are as follows 208 The server was delivered with the Windows Server 2008 Operating System This was then configured to comply with NIST 800 53 Revision 2 Annex 1 since this is a low impact system The server was originally made the DHCP and DNS server for the private LAN to which itis attached There is no domain controller on this network When the server wa connected to the TFHRC Internet 2 network through IP6 it was no longer on a private network The DHCP and DNS options were disabled Oracle 10g 10 2 0 4 which was the first version that handles 64 bit databases under Windows Server 2008 was originally installed on the server When the DPW was activated the central server was upgraded to Oracle 115 11 2 0 Oracle was installed the production database instances IMSProd and TRFProd were created After the production instances were created two test instances IMSTest TRFTest and two development instances IMSDev TRFDev were cloned The TRF instances were shut down after the instances were migrated to the PPDB Backups The original central server uses Symantec Backup Exec and RD1000 500GB and RD1000 ITB Disk cartridges for backups The backup strategy is regular backups of the Oracle database to removable media Due to the fact that major data chang
209. s 109 Add Remove Table 109 Creating Public Synonyms R ADMINISTRATION Oracle Users LTPP Users Identifying Users Manage Users Roles EARLIER ORACLE VERSIO APPENDIX K DATABASE SYNCHRONIZATION Work flow Instance Review Electronic Files Management APPENDIX L CLONING THE DATABASE BEFORE CLONING FOR THE FIRST TIME CLONING APPENDIX M SQL DEVELOPER NOTES IMPORTING DATA EXPORTING DATA DATABASE MANAGEMENT Tablespace Modifications USING SCRIPTS SETTING UP CONNECTIONS MANAGING USERS Managing Roles INSTALLING SQL DEVELOPER APPENDIX N SOFTWARE APPLICATION OPERATING SYSTEM ORACLE APPENDIX ORACLE INSTALLATION INSTALLING ORACLE 12C CREATING AN INSTANCE APPENDIX P REMOTE ACCES APPENDIX Q ROLES AND RESPONSIBILITIES DPW APPENDIX R CONTINUITY OF OPERATIONS DPW PURPOSE Scope Situation Overview Planning Assumptions Objectives CONCEPT OF OPERATIONS Phase I Readiness and Preparedness 177 Backups 177 Dell Service 181 Phase II Activation 181 Phase Continuity Operations 181 Phase IV Reconstruction Operations 183 APPENDIX S DISASTER RECOVERY DPW INTRODUCTION Purpose Planning Assumptions Objectives Concept of Operations SYSTEMS OVERVIEW Risk Identification and Mitigation Complete Loss of Facilities at Oak Ridge APPENDIX REFERENCE DOCUMENTS DATA USER GUIDES LTPP IMS User Guide QC Manual Accessing LTPP Da
210. s or individuals Assignment Of Security Responsibility The assignment of securi responsibility role is responsible for the overall security of the Dell 2900 Other Designated Contacts The other designated contacts role designates key contact personnel who can address inquiries regarding system characteristics and operation LTPP The LTPP role provides FHWA oversight of the LTPP program Table 14 TFHRC Server Role Assignments Role Organization with Individuals in the Role System Administrator AMEC Backup System Administrator SAIC Database Administrator AMEC Backup TFHRC Database Administrator SAIC TSSC Program Manager AMEC Customer Service User ESCING Technical Support Services Staff AMEC Information System Owner FHWA Authorizing Official FHWA Assignment Of Security Responsibility Other Designated Contacts None LTPP See Team listing FHWA IT Help Desk ipis Twa dol gov research Tar JprogransTinlrastucture pavementsipp shos who cim Table 15 Dell 2900 Server Organizational Contact Informati Organization Contact Information Tong Term Pavement Program Team FHWAIDOT HRDI 30 LTPP 6300 Georgetown Pike McLean VA 22101 194 Organization Contact Information TFHRCIT FHWA DOT HRRM T 6300 Georgetown Pike McLean VA 22101 SAIC 151 Lafayette Drive Oak Ridge TN
211. s and reports package for application development which was able to run on multiple platforms without recoding Oracle software updates have generally only been applied during major system upgrades The DPW which houses a central repository is on an isolated network with access limited to LTPP personnel The central server is on an isolated network with access limited to LTPP personnel as well The Oracle Database Enterprise Edition version 11 2 0 1 0 along with the administrative tools is installed for use with the PPDB on the Dell 2900 Add reference on location of documentation locally and on line The Oracle Database Enterprise Edition version 12 along with is installed for use with the PPDB on the Dell R515 Add reference on location of documentation locally and on line APEX This is an Oracle provided web based SQL application that is used on the DPW It provides the ability to run SQL scripts and get limited types of output Scripts may be edited uploaded and stored in the application SQL Developer SQL Developer is a GUI provided by Oracle that allows manipulation of both data and the databases This software replaces the use of Enterprise Manager on the TFHRC server Limited documentation on the more common LTPP uses of this tool is found in Appendix SQL Developer Notes Enterprise Manager This is Oracle software that provides a Graphical User Interface GUI with which to view and manage d
212. s listed in the right hand box when that media set is selected from the folder list in the center of the screen Media have unique labels assigned by the software The color coding associated with the label indicates whether the file can currently be overwritten reused appended or not overwritten Blue indicates media available to be overwritten A media icon with a circled red X in the lower right corner is off line Media sets may be on line or off line An on line set is one that is physically on the computer on a hard drive or cartridge An off line set is one that is on a cartridge not in the cartridge drive Media may be moved between media sets This may occur when a different folder structure is selected to store files on a hard drive or when media fail A media failure is typically identified when a backup up fails The most common error message is Not appendable End marker unreadable In this case the media are moved to the Retired media set To move media select the media and right click on the name A menu will come up as shown in the lower right portion of Figure 67 95 Sry Backup Enee 125 or Win a OC SR Figure 68 Screenshot Seleet a new media set association 96 Click on the arrow next to the block labeled Associate with when the next dialog box comes up This will bring up a list similar to that shown in Select and Click on Retired Media It will then appear
213. s that the results of directory searches may be save to text 1 0 or comma separated value csv files This product requires a license Notepad Central Server This is text editor a free source code editor distributed under the GPL license It is similar to Notepad or Wordpad Its advantages are that it can do columnar cut and paste it has basic syntax checking capabilities for SQL and other languages and files can be opened in more than one tool using a text editor and they will be synchronized The last capability makes it possible edit scripts in Notepad and after saving them run them in SQLDeveloper while having them open in both applications Microsoft Office Dell 2900 Microsoft Office has been installed on this machine primarily to provide Microsoft Access for extracting tables for LTPP applications requiring tables in this format It is possible to update this software using the same procedure outlined in the preceding Operating System Updates section Just choose Microsoft Access as the product Office has been installed in order to process data releases efficiently Since data releases are provided in Microsoft Access format it is very important to have Office tools available The software is updated through the Windows Update function at the same time operating system updates are done Software upgrades no longer require the actual installation of MS Office and itis not installed on the Del R515 Spicew
214. s using riser card 8 System identification connector System status indicator 10 System identification button Power supply 2 12 Power supply Figure 12 Illustration R515 back panel The indicators on the optional redundant power supplies show whether power is present whether a power fault has occurred see table and figure 13 Table 8 Function of redundant power supply indicators Color Condition Not tit AC power not connected Green In standby mode indicates that a valid AC source is connected to the power supply and that the power supply is operational Amber Indicates a problem with the power supply amber Alternating green and Mismatched power supplies when hot adding a power supply Figure 13 Ilustratio Diagnostic Light Codes Location of redundant power supply indicators The system s front panel displays error code during startup The diagnostic lights are not lit after the system successfully boots to the operating system Table 9 provides a list of codes and the location in the owner s manual for further troubleshooting Table 9 Diagnostic light codes Dell R315 Code Causes Corrective Action The system iim anormal oF condition possible pre BIOS failure has occurred Plug the system into a working electrical outlet and press the power baton checksum Talore decet system isin recovery mo
215. shown in Error Reference source not found The dialog box that appears to edit the table space Error Reference source not found includes the name of the data file that will be expanded and the option to set the maximum size Max Size in kilobytes megabytes gigabytes or terabytes Selecting the DDL tab in the Edit Tablespace dialog with the Update option chosen will display the SQL statement to make the changes made in the File Specifications window A sample statement would be ALTER DATABASE 256 MAXSIZE 90 Where the new maximum size is shown in bytes 90177536 This can be edited to the appropriate multiple rather than computing the value in bytes if done manually Note that the name of the data file being extended is fully qualified and enclosed in single quotes Typically the LTPP databases extend data files rather than add them when expanding tablespace sizes 130 EET H Contguraton statue B oaro 8 Perfomance 1E O RMAN Bacup Recovery suco Mae S sume Arhive Loss e Tablesoace Grouse dba Figure 81 Screenshot SQL Developer tablespace management et ese Figure 82 Screenshot Object list under Tablespace in SQL Developer DBA window 131 Detaies Fre Spece Objects Usage Chart SOL
216. sites for update of the OS and anti virus software Effect The effect is that the server appears to be down for external access There will be no remote connection capability On the server side everything is functioning normally and access will return as soon as the internet connection is restored Mitigation There are no workarounds to allow the user access to the server Resolution Notify TFHRC IT that the VPN network is down for external access failures either inbound or outbound This step is typically unnecessary since they have automated notifications when the VPN network has a problem Wait for service to be restored TFHRC Software Services Become Unresponsive Description Software services becoming unresponsive typically results in inability to access an Oracle instance Effect The effect is that users are prevented from using the affected Oracle instance Mitigation No mitigation is necessary Resolution If resources were not the problem check table 6 to determine which service needs to be restarted On rare occasions a restart of the central server may be required 43 Table 6 TFHRC server Oracle service diagnostics Service Description OracleOraDbI1g_home2TNSListener This listens for connections to the Oracle database instances Restart this service if having Oracle connection problems Restarting this service will typically not impact users greatly Restart takes about 3
217. ssed in the following sections The hard drives power supplies and the codes on the LCD panel should be checked at least once a week The hard drives are joined together in a single RAID 6 array The hard drives are partitioned into volumes and these volumes are set aside for different purposes Microsoft Windows Server is installed on the C volume The Oracle database files are stored on the D volume SERVER The central server is a Dell PowerEdge R515 with two 3 GHz Xenon E5450 quad core processors Itis a twelve bay system eight of which are populated with 7 200 RPM near line SAS hot plug drives This server also has 32 GB of RAM an internal RAID 5 Array consisting of eight 7 200 RPM 1 SATA disks There are two altached external twelve bay MD1200 storage units discussed separately For backups it has 4TB external USB 3 0 hard drives of this is protected by a 2200 VA UPS The server setup for the Dell R515 is detailed in doc see Appendix T Reference Documents This document describes in detail how the RAID Array operating system and Oracle databases were configured The Dell PowerEdge R515 Systems Hardware Owner s Manual is stored in the Dell subdirectory of the D ScratchSpace Software_Downloads folder A second copy is stored on the incremental backups hard drive in the folder Setup Manuals Server status The basic procedure is to look for lights that are not green blue on the front of the ser
218. storage can be identified The recycle date in all cases is the Date value on which the tape is used the next quarter Tapes identified with an asterisk are on 2 year archive schedule and the year for recycling is 2012 for the 1 3 quarters and 2013 for the 4 quarter Figure 109 is the 2011 schedule when tape backups shifted from weekly to every other week bi weekly The change was made due to costs associated with media required for both weekly and quarterly backups as the amount of data expanded The schedule was establish with a rotation cycle of four quarters with the extended retention periods for the last tape of each quarter Describe figure 98 Figure 110 through figure 112 show the bi weekly schedules used for 2012 to 2014 In 2015 the Dell 2900 became the backup instead of the primary server and tape backup frequency was reduced to monthly following the SDR and retention of parallel processing capability as shown in figure 113 2 Figure 108 Screenshot Weekly tape rotation for 2010 sera mjanja oe EE Ede eo pa m H El cse cue i pod mel a n Sows Bowl p DRE E ioco oc mos NEM nan caen v onem E fee sate mea dak aloe rests te nara arrastrar sn arose Want Figure 10
219. t Installation Media Continuity Facilities Resumption of operations in the event of a catastrophic failure at the TFHRC facility will follow the plan for the facility as a whole Continuity Communications There is not another connection to the internet designated for resuming operations in the event of a catastrophic event at the TFHRC facility Communications between personnel will be accomplished using a combination of phone and email Phase IV Reconstruction Operations The plan is to repair the Dell 2900 system to the extent needed to support parallel processing This will be accomplished using THFRC systems personnel to restore the Dell 2900 with the least data loss possible When the Dell 2900 is ready to be used for 201 data dissemination a notice will be sent to the contacts listed in the Alert and Notification section 202 APPENDIX DISASTER RECOVERY DELL 2900 INTRODUCTION Purpose The purpose of the disaster recovery plan is to define a set of potential problems and their likely solution Planning Assumptions The basic assumption is that there is a two week window to recover from a disaster without causing serious disruption in data dissemination capabilities Given that hot standby is not required it has been decided that the most cost effective solution is to repair whatever is wrong with the Dell 2900 and resume operations We are counting on internal TFHRC resources to resolve h
220. t or comma separated value csv files This product requires a license 225 This product has not been updated since installation Notepad This is a text editor It is a free source code editor distributed under the GPL license It is similar to Notepad or Wordpad Its advantages are that it can do columnar cut and paste it has basic syntax checking capabilities for SQL and other languages and files can be opened in more than one tool using a text editor and they will be synchronized The last capability makes it possible edit scripts in Notepad and after saving them run them in SQLDeveloper while having them open in both applications This product is updated when new stable version is available Microsoft Office Office has been installed in order to process data releases efficiently The tools include Word Excel Outlook Powerpoint and Access Microsoft Office has been installed on this machine primarily to provide Microsoft Access for reviewing tables used by LTPP applications in this format This software was updated using Windows Live Update in manual mode This software is longer being updated with removal of the Dell 2900 from the internet Winzip Winzip is used for file compression The original installation on the Dell 2900 was Winzipl 1 No intermediate upgrades were obtained In 2014 to maintain compatibility with the DPW a new version was purchased As commercial license the software
221. t pdf copies of all job logs and histories since the last weekly backup including incremental backups i expanded format 3 Move prior week s incremental pdfs to Last_ week folder 4 Erase weekly pf forthe same tape name 5 Erase incremental backup pdfs more than 2 weeks old 6 Eject the Current cartridge 7 Eras all job reports in Symantec BE after printing S Insert the tape to be used for the next backup typically for the following bi weekly backup 9 Inventory the cartridge and verify it has space for next backup 10 Erase folders on cartridge NOT created with BE if necessary Windows Explorer 11 Format a new cartridge if necessary 18 Fourth Wednesday of the month Put cartridge for off site storage in FFC container or in desk drawer if container is off site Quarterly 1 First Wednesday morning of second month 2 Take AIMS backup job off hold 3 Verify selections for backup 4 pdf of the backup conditions if modified 5 Print paper copy of the current AIMS backup properties for storage with the cartridges 6 Inventory ITB cartridge to verify space available overwrite conditions First Thursday morning of second month or Wednesday pm 1 Put AIMS backup job on hold on completion 2 Print pdf copies of job log and history 3 Print paper copy of the backup log for storage with the cartridges 4 Delete last pdf of backups to sume cartsidge s 5 Eject the current cartridge 6 Erase job repo
222. t spare will automatically take the place of the failed drive providing additional protection until the failed drive can be replaced These drives are hot swappable This array is the D drive and contains the database instances ADEP repositories and backup staging areas The schematic in Figure 100 below shows the highlights of the backup process 197 pe 8 Ware Vli ME ERR Eee m TUS VERUS 7 prx cmm Volume USB Drive pene Volume J RD1000 Full ADEP Repository Tee Backup 4 RD1000 E Cartridges Stored Off Site pene m Figure 99 Schematic Backup Process Overview Replace with Dell 2900 equivalent Drive Backup Strategy The RD1000 mounted on drive J is used to write backups for off site storage The system uses 500GB cartridges for working off site backups 1 TB cartridges are used for backup of AIMS material A 1 TB WD1000 external hard drive mounted on Drive is used for incremental backups 198 There are multiple types of backups taking place First there are the drive level backups Symantec Backup Exec System Recovery 2010 is used to back up drive C to the staging area H SymantecRecoveryPoints The backup of drive C starts with a full image backup on Sunday at 10 30 PM with differential backups created at 10 30 PM Monday through Saturday The st
223. ta LTAS User Guide Bookshelf OPERATIONS IT Security Hardware Manuals Software Manuals Source Code Inventory SPR Database Directive 1 170 APPENDIX AA ROLES AND RESPONSIBILITIES DELL 2900 APPENDIX AB CONTINUITY OF OPERATIONS DELL 2900 PURPOSE Scope Situation Overview Planning Assumptions Objectives CONCEPT OF OPERATIONS Phase I Readiness and Preparedness Backups Dell Service Phase II Activation 200 Phase III Continuity Operations 200 Phase IV Reconstruction Operations 201 APPENDIX AC DISASTER RECOVERY DELL 2900 INTRODUCTION Purpose Planning Assumptions Objectives Concept of Operations SYSTEMS OVERVIEW Risk Identification and Mitigation Complete Loss of Facilities at TFHRC APPENDIX AD HARDWARE SOFTWARE AND MAINTENANCE DELL 2900 Server Setup Backups Server status Power Supply Inspection Server Troubleshooting Dell Support Server Storage Status Dell Support Storage Hard Drive Replacement UPS SOFTWARE Windows Server 2008 Symantec Anti Virus Symantec Backup Exec 2010 Oracle SQL Developer SQLPlus Worksheet Command Line PowerDesk7 Notepad Microsoft Office Winzip Spiceworks APPENDIX AE BACKUP POLICIES DELL 2900 BI WEEKLY TAPE ROTATION SCHEDULI QUARTERLY TAPE ROTATION SCHEDULE INPUTS TO BACKUP POLICIES APPENDIX AF SYMANTEC BACKUP EXEC 2010 SYMANTEC ADMINISTRATION AND TROUBLESHOOTING Passwords Symantec User Accounts 237
224. tage along with an expected duration if it is known This notice should be followed up with additional emails as details become available Finally notice should be sent when the system is available for use Phase III Continuity Operations Essential Functions 181 The essential function of data archival can continue with only a slight degradation This is because the there are at least two other off site copies of the information on the server Data extraction for general dissemination occurs during the August to November time frame which is the most critical period for on line functionality Down time primarily affects schedules in the extraction and review process Data extraction on request is a process that occurs in low volumes sporadically throughout the year Most responses are based on the Access version of the database standard data release SDR copied in quantity to USBs as part of the dissemination process and will not be impacted by downtime The few requests that cannot be handled with the SDR require access to AIMS and can be addressed with the copy maintained by Customer Support Downtime primarily affects turnaround time on a response Essential Personnel Order of Succession System Administrator Information Systems Owner o FHWAIT Delegation of Authority Essential Equipment and Systems PowerEdge RSIS 2 MD1200 4078 storage units RD1000 External Cartridge Drive WD1
225. tained in the documentation for the unit SOFTWARE The following is a list of licensed software associated with the Dell 2900 Generate a formal list with versioning using Spiceworks over acquisition update and uses Table 20 Dell 2900 COTS software Software Renewal Date Renewed by Symantec Backup Exec 2010 Jan 20l4 FHWA w maintenance 19 lostations taken from APC User s Guide sup pe com productsresource includehechspec_index cim base_sku DLA2200RM2U amp tab docu mentation accessed 5 7 2015 Oracle NA SQLDeveloper x Freeware MS Office 2007 N A DOT MS Access 2003 N A DOT Winzip DOT Adobe Acrobat 9 NA DOT TextPad Shareware Notepad Freeware PowerDesk 7 Purchase of upgrades only LTPP Spiceworks Freeware Team Leader and COR have licensing information Not renewed because operations shifted to new server 10 10 10 34 Windows Server 2008 This is the operating system for Dell 2900 Operating system patches are applied through Windows update sporadically with the removal of the server from the internet The Windows update process was set for manual rather than automatic updates when the computer was connected to the internet Updates are also applied as requested by the TFHRC Help Desk Symantec Anti Virus Symantec Endpoint Protection 12 1 5337 5000 is the current version as of May 2015 This is the FHWA provided antivi
226. tdown BION shutdown test failure See Getting Help Test POST Mem 8105 POST memory tes failure See Troubleshooting System Memory If Test problem persists see Getting Help E201F DRAC Dell remote access controller Check screen for specifie error messages Config DRAC configuration failure 216 Code Causes Corrective Actions Ensure that DRAC cables and connectors properly seated If problem persists sec your DRAC documentation E2020 Config CPU configuration failure Check sereen for specific error messages E2021 Memory Incorrect memory configuration Check sereen for specific error messages Population Memory population order See Troubleshooting System Memory E2022 POST Fail General failure after video Check sereen fr specifie error messages E2110 MBEDIMM One of the DIMMS in the set Sce Troubleshooting System Memory begat implicated by H amp hada memory multi bit error MBE E2111 SBELeg The system BIOS has disabled Troubleshooting System Memory Disable memory single bit error SBE DIMM A logging and will not resume logging further SBEs until the system is rebooted 4 represents the DIMM implicated by the BIOS E2112 Mem Spare The system BIOS has spared the See Troubleshooting System Memory DIMM memory because it has determined thatthe memory had too many 4 amp 8 rep
227. te that the Vault number 700 is the default for the FHWA LTPP Account The Type is Distribution to First Federal The other type is Retum from First Federal and only used for unscheduled returns The Media option selected in this example is Container Small There are a number of options the default being 4mm When selecting the media for an LTPP batch the option is determined by the fourth and fifth characters of the label 700 CXNN XXXXX as underlined For CS use Container Small For CM use Container Medium and for CL use Container Large Each media type requires a separate batch Under Volser type in the label of the container 700 CXNN XXXXX Only boxes which at FHWA or not in inventory when the batch is created can be added If a box on hand is being sent and a box of the same size is being received and sent back the same day they 100 will be in different batches The box on hand is in the batch created on the 3 Monday The box being received and sent back the same day will need a batch created on the day of delivery Thursday morning Failure to do so will result in a discrepancy e mail from FFC and a need to update the inventory to account for the unscheduled return Under A R Archive Return leave the selection Return This indicates that the box will come back on the return date specified Under return date enter the 3 Thursday of the month the box should come back to FHWA This date can be found from the box rotation
228. that was just added The third statement temporarily removes a constraint so that data can be added without considering parent child relationships Add Remove Table See CREATE TABLE and DROP TABLE in the Oracle Database SQL Reference 11g Release 2 manual syntax options for this command Examples of the basic statement are as follows CREATE TABLE newtable atate code NUMBER 2 0 NOT NULL shrp id VARCHAR2 4 NOT NULL data NUMBER 6 2 CONSTRAINT pk newtabi PRIMARY KEY atate code shrp id USING INDEX TABLESPACE inde TABLESPACE tab DROP TABLE newtable 109 The first command creates a table in the tablespace table_tablespace with STATE_CODE and SHRP_ID as the primary key with one column containing numeric data The second statement removes the table that was just created Creating Public Synonyms Public synonyms allow tables in other schemas to be referenced without specifying the schema The full syntax of the Create Synonym and Drop Synonym commands is available in the Oracle Database SQL Reference 10g Release 2 manual LTPP instances generally work on the assumption that tables are referenced by public synonyms which are identical to the table name Therefore anytime a table is added removed the corresponding public synonym should be created or deleted The basic syntax to create a public synonym is as follows CREATE PUBLIC SYNONYM newtable FOR newtable This allows t
229. that were applied to a test section Monitoring MON This module contains pavement performance monitoring data and it is the largest module in the database It is divided into submodules by data type Deflection MON_DEFL This submodule contains data from FWD tests BAKCAL This submodule contains backcalculation results from deflection data available through Distress MON DIS This submodule contains distress survey data from both manual and film based PADIAS Drainage MON DRAIN This submodule contains information on the inspection of drainage features Friction FRICTION This submodule contains friction measurements taken by participating highway agencies Profile MON_HSS This submodule contains longitudinal profile data collected by an automated profiler or by manual dipstick measurements Beginning with SDR 29 texture measurements are also included Data Module Table Prefix Rut MON_RUT This submodule contains rutting data measured using a 1 2 m 4 ft straightedge These data tables are superseded by the rutting indices located within the Transverse Profile module Note Straightedge rut measurements were not taken all test sections Transverse MON T PROF This submodule contains transverse profile Profile data and computed transverse profile distortion indices rut depth from manual dipstick measurements or the opt
230. the backup job set ups histories and logs 254 There are six sub folders under Backup logs 0 Incremental db 0 Incremental traffic 0 Incremental working 1 Monthly 2 SDR 3 Weekly 3 Periodic 0 Incremental db incremental backup of the LTPP Oracle databases 02 0_Incremental traffic incremental backup of the traffic drive F 0_Incremental_working incremental backup of the working storage drive D L Monthly full backups that are stored off site for either 13 weeks or 2 years 2 SDR full backups Weekly full backups that are retained on site 3 Periodic Quarterly backups A folder contains three types of pdfs job setups job histories and job logs Job setups are the final step of the process in Creating a job The 0 folders have a Last week subfolder to store prior weeks pdfs Job histories and Job logs are created as follows Select the Job Monitor tab Select a job in the Job History section Right click on the job to bring up the menu box as shown in Figure 47 Select the earliest if multiple jobs of the same type have run 255 Figure 142 Screenshot Selecting a Job to Print Click on Properties The Properties selection brings up the Job history screen with its two tabs Job History and Job Log Click on Job History The condensed version comes up as shown in Figure as 256 pu Figure 143 Screenshot Job History screen Click on Jo
231. the inventory creating batches and ordering additional boxes and labels as needed Logging in Go to hitps emsnet03 federal com User ID 700 Password fha700 Figure 71 Screenshot Login Screen for First Federal Corporation Off site Storage Type in the middle three characters of the code on the FFC security badge web Pagevialog 9 Please Enter Badge ene federa p Figure 72 Screenshot Secondary Login for First Federal You may get an error message as shown below IE Component Installation Failure You donet ave the inal thi ccrtc You ray the et shown below smal ta DOT at SonteeDcsh cch aov oral HELP to ecued broese SOURCE Fipe Peerless core CONT AGL CHSNET CAG CLASGID BFROCAGOET40 496 8692 DECCOAZEFDET VERSICN 1000 senicedesk dol Figure 73 Screenshot Error message on login to First Federal This error will have no impact on processing It has been discussed with IT and they won t make any changes Checking Inventory The boxes stored at First Federal and their content can be checked on line Preparing a Batch On the left panel ofthe screen and under Table of Contents click on Manual Entry Then on the right panel click on Start New Empty Batch A batch is a set of media boxes that are all the same type in FFC terms No
232. the IMS PC QC directory The last hard copy of this document was created in 2010 The current inventory is kept in Tortoise 191 SVN a type of version control software The version of each software package and utility developed by the LTPP program current and in use at the time of the SDR is provided for storage and backup at TFHRC under after the submission of the SDR on the Dell 2900 The same files are stored on the TFHRC server under SPR Database The SPR Database is an Access database that has all SPRs from mid 1993 to August 2011 Earlier SPRs were kept in hardcopy and are now in electronic form for eas searching The SPR Database includes dates that the SPR was received referred and completed it includes a description and a resolution and it includes the originating organization Several reports are available from the Access Reports list SPRs since that date are maintained in a Redmine project accessible to users of the DPW Directive 1 170 This document provides information about AIMS electronic data format and submission standards It is located in the Reference Documents directory 1 170 pdf along with the first amendment Dates for AIMS updates are contained in directive or its replacements 192 APPENDIX AA ROLES AND RESPONSIBILITIES DELL 2900 The Disaster Recovery DR and Continuity of Operations Plan COOP appendices along with the require that persons be identified to fulfill var
233. trator The backup database administrator maintains the database instances on the Dell 2900 This includes database backups schema modification and storage management under the direction of the database administrator and or when the database administrator is unavailable TSSC Program Manager The TSSC program manager is responsible for the oversight of the LTPP server activities at TFHRC Technical Support Services Staff The technical support services staff role provides support for LTPP server activities on site under supervision of the Dell 2900 database administrator Customer Service The Customer Service role is responsible for answering customer inquiries including distributing data Customer Service is also responsible for maintaining some centrally processed data and maintaining copies of ADEP on the Dell 2900 server TFHRC IT The TFHRC IT role is responsible for maintenance of FHWA supplied operating and anti virus software and supporting hardware maintenance activities Information System Owner The information system owneris the agency official responsible for the overall procurement development integration modification operation and maintenance of the system e Authorizing Official The authorizing is senior management official or executive with the authority to formally assume responsibility for operating an information system at an acceptable level of risk to agency operations agency asset
234. ts a job immediately rather than waiting for a scheduled time or setting up a schedule The Test run function checks that all of the pieces are present drives accessible to the user command files for pre and post job actions readable and so forth It will execute all of the elements of the backup job except the actual backup REVIEWING JOB HISTORY SAVING BACKUP REPORTS Printed copies of backup reports are created once a week to have a permanent record of backups made and their content Printed copies are made with Adobe Acrobat Job reports can be stored indefinitely in BE These guidelines have been established to set limits on the amount of information stored Figure 141 shows a series of folders on drive Working Storage of the server that apply to backups Included are the folders Backup_logs and Symantec Backup Source Files The latter is exactly what it says and is the location for the original software and any patches applied manually NE DC Meme S IE aan e pcr jer Pine ae EIE Figure 141 Screenshot Folders on server for storing backup reports The folder Backup has three sub folders Backup_logs IDR and Offsite_container IDR Intelligent Disaster Recovery is a folder used by BE Offsite container contains files on the original system for storing cartridges off site Backup_logs holds all the copies of
235. ude any details that are known about the outage along with an expected duration if it is known This notice should be followed up with additional emails as details become available Finally notice should be sent when the system is available for use Phase III Continuity Operations Essential Functions The essential function of data archival can continue with only a slight degradation This is because the there are at least two other off site copies of the information on the Dell 2900 Data extraction for general dissemination occurs during the March and April time frame which is the most critical period for on line functionality Down time primarily affects schedules in the extraction and review process Data extraction on request is a 200 process that occurs in low volumes sporadically throughout the year Most responses are based on the Access version of the database standard data release SDR or use of Infopave The few requests that cannot be handled with the SDR require access to AIMS and can be addressed with the copy maintained by Customer Support Downtime primarily affects turnaround time on a response Essential Personnel Order of Succe System Administrator Information Systems Owner FHWAIT o Delegation of Authority Essential Equipment and Systems PowerEdge 2900 TFHRC Server 2MDI000 25 TB storage units RD1000 External Cartridge Drive WD 1 TB External Hard Drive Connection to the interne
236. uested from First Federal Corporation The information system owner would be the primary person to interface with First Federal Corporation The system administrator would be the secondary contact Dell Service The DPW server is a R515 with service tag 2WENPQA It is covered by the Gold or ProSupport with Mission Critical and the 4 Hour On Site Service plans until 5113 2014 These warranty plans ensure that the DPW will be repaired quickly The system administrator would be the primary person to interface with Dell Support The information system owner would be the secondary contact The two MD1000 25 TB storage units with service tags And out of warranty Necessary repairs are done by the system administrator Phase II Activation Decision Process The decision to implement will begin with the initial notification that the server is not functioning This will typically on direct observation by the system administrator Upon identification the severity of the problem will be assessed If it is determined that the problem can likely be resolved by a restart of services then those actions will be taken before implementing this plan Alert and Notification If it is determined that actions beyond a quick fix the information system owner and the TSSC program manager will be notified in person by phone or by e mail as appropriate This notification will include any details that are known about the ou
237. ume operations We are counting on the Dell ProSupport Mission Critical Option that specifies 4 hour on site service with 6 hour parts availability to resolve any hardware problems with the DPW We are also counting on Iron Mountain to be able to deliver backups from off site storage within three hours of an emergency request Objectives The main objective is the restore the DPW to full operational status as soon as possible with minimal data loss A secondary objective is to keep key personnel at the FHWA and the TSSC informed about the problem and its resolution When possible limited functionality will be made available while permanent repairs are underway Concept of Operations SYSTEMS OVERVIEW The DPW has databases and files housed on a single server that is accessible either on location or via VPN As shown in figure 98 a keyboard connected via and VPN are the primary entry points into the system Figure 98 Schematic DPW server access diagram Risk Identifi ion and Mitigation Internet Connection Failure Description An internet connection failure will first become apparent due to failure to access the system via VPN 184 Effect The effect is that the server appears to be down for external access There will be no remote connection capability On the server side everything is functioning normally and access will return as soon as the internet connection is restored Mitigation There
238. ure 39 Screenshot Advanced Open File Options Used 35 Figure 40 Screenshot Using Defaults for Network and Security 76 Figure 41 Screenshot Identifying Pre and Post Commands 76 Figure 42 Screenshot Archive Method Selection 77 Figure 43 Screenshot Job functions accessible through Job Setup screen 78 Figure 44 Screenshot Delete Confirmation dialog box Figure 45 Screenshot Initial Properties dialog box Figure 46 Screenshot Folders on Server for Storing Backup Reports Figure 47 Screenshot Selecting a Job to Print Figure 48 Screenshot Job History screen Figure 49 Screenshot Selecting a Printer Figure 50 Screenshot Locations and names for job logs Figure 51 Screenshot Job Log screen Summary Form Figure 52 Screenshot IDR Wizard replace 2010 Figure 53 Screenshot IDR Preparation Wizard Opening Screen Figure 54 Screenshot IDR Boot Media Options Figure 55 Screenshot IDR CD Creation Instructions Figure 56 Screenshot Selecting a Computer for Disaster Recovery Preparation Replace new server name Figure 57 Screenshot Location Selection for CD Image Figure 58 Screenshot Identifying Windows OS Installation File Location Figure 59 Screenshot Image Creation Messages Figure 60 Screenshot Outcome of Disaster Recovery Preparation Figure 61 Screenshot Identification of Image File Name and Location Figure 62 Screenshot Selecting the Disaster Recovery File Option in the IDR Preparation Wiza
239. ver Each hard drive should have a green drive status indicator Each power supply should have a green power supply status and a green AC status indicator And as a general 48 check the LCD panel should be blank If anything goes wrong on the system the LCD panel will have one or more lit numbers indicating what further action is necessary The back of the server should be checked to verify that all of the lights associated with the storage to server connections are on and green The following sections contain excerpts from the Dell PowerEdge R515 Systems Hardware Owner s Front Panel Inspection Figure 10 shows the front panel features of the Dell R515 The LTPP unit uses only eight bays 1 System identification bution 2 Diagnostic lights 3 Power on indicator powerbution 4 NMI button 5 Hard drives 6 Video connector 7 USB Connector 2 0 compliant 8 System identification panel Figure 10 lustration Front panel features The power button on the front panel controls the power input to the system s power supplies The power indicator lights green when the system is on The hard drive carriers have two indicators the drive status indicator and the drive activity indicator Figure 11 Each existing hard drive should have a green drive status indicator hut wll com supportiomelus en 1 product supportservietag SNTAMOMmanuals accessed 5 6 2015 Image taken from
240. with records Contact regions with questions 4 Review the export logs Check each module for extraction of all requested tables generally all Verify that there were no errors on exporting Update Upload Processing Files Run the commented out portion that counts constraints to determine if any constraints have been added or removed Document the value found from running the count in the script with the date run 120 Disable any new constraints foreign keys that could cause trouble while loading data Disable database triggers that might fire when data loaded into database Comment out truncate statements for tables that are not being uploaded Add truncate statement for any new tables being uploaded Enable constraints and triggers disabled in the StartUploadProcess sq file Process Upload Once these procedures have been updated they will be executed as Follows Run StartUploadProcess from the synchronization directory in a DOS window using the following command StartUploadProcess dbaUN dbaPW G database upload date where upload date is the official upload date in yyyymmdd format Review the output file for errors Scroll through the entire output file and verify that all constraints were disabled successfully Table altered Verify that all triggers were disabled Trigger altered Verify that the cluster Cluster truncated and all tables were truncated successfully Table truncated
241. xternal RD 1000 drive could be used to apply the backups to a Windows 2008 R2 188 server builtin the Amazon cloud The Internet Protocol IP address of the portals would be redirected to the IP address of the Amazon hosted server The expected downtime would be approximately a week This largely depends on how long it takes to set up an LTPP account on Amazon and how long it takes to transfer the backup data into the cloud storage 189 APPENDIX REFERENCE DOCUMENTS DATA USER GUIDES LTPP IMS User Guide The Long Term Pavement Performance Information Management System User Guide User Guide gives the user an in depth look at all of the data modules and most of the tables in the Pavement Performance Database As of SDR 28 the document included information on LTAS and AIMS This document should be the data analyst s best friend The version distributed with SDR 29 is located in the Reference Document subdirectory User_Guide_2013 pdf QC Manual This manual documents the QC checks to which the LTPP data is subject through the QC programs The last distributed version is dated September 2013 and can be found in the Reference Documents subdirectory QCMan201 3 pdf Accessing LTPP Data This document is intended to provide tips to Microsoft Access users on how to effectively use Access to analyze LTPP data It includes Access tips and tricks and tidbits about the LTPP data The most recent version of this document is i
242. y section Right click on the job to bring up the menu box as shown in Figure 47 Select the earliest if multiple jobs of the same type have run Figure 47 Screenshot Selecting a Job to Print Click on Properties The Properties selection brings up the Job history screen with its two tabs Job History and Job Log Click on Job History The condensed version comes up as shown in Figure 48 pum Figure 48 Screenshot Job History sereen Click on Job History to ensure it is in front Click on Expand All upper right corner Click on Print lower right corner The printer selection menu will come up Figure 49 Adobe PDF should be the default printer If not select it and then click on Print The filename that comes up is that assigned by BE Figure 50 Before changing the fle name verify thatthe correct directory Figure 46 in GABackup Buckup_logs has been selected Enter the file name Using the file naming convention type the first letter of the file name It can be simpler to edit an existing file name than to type one from scratch The file name in this case where the Job History tab is forward will be the hist version The dates in the files are the date the backup being printed was STARTED not the date it finished If the backup failed a printout is still made to know which backups do and do not exist The History or the Log may be missing depending on the type of failure 83

working draft

Contents

Download Pdf Manuals

Related Search

Related Contents