Home

ADC Software Troubleshooting Guide

image

Contents

1. file adc log any any match adc Alternatively you can configure the syslog server See your J uniper Networks documentation for more information on the syslog Viewing the Syslog File You can view the syslog file using one of the following methods Note The syslog filename adc log is a sample filename used for these examples Your filename may be different For more information see Configuring the Syslog File on page 20 20 Document ID RDWR RSLB V1 4R0 0_TSG0711 re p H radwa re ADC Software Troubleshooting Guide Troubleshooting Fundamentals ei To use the show command to view the syslog file user host gt show log adc log ei To set a separate connection that constantly shows the new messages in the syslog file Accessing the Syslog File The syslog file itself is located in the directory var log You can access the file using FTP to connect to the device and then navigating to var log Old files are compressed zipped in the same location using the filename adc log gz Note The syslog filename adc log is a sample filename used for these examples Your filename may be different For more information see Configuring the Syslog File on page 20 Configuring the Trace File The trace file is configured in the ADC software using the Junos OS conventions for trace configuration ei To configure the trace file edit extensions adc traceoptions file dynamic txt si
2. Commit Script Automatic Configuration The ADC software uses the commit script to configure static automatic configuration ei To view the automatic configuration set up using the commit script user host gt show configuration display commit script Note The ADC automatic configuration added by the commit script is marked using the apply macro attribute Document ID RDWR RSLB V1 4R0 0 TSG0711 41 j 1 WI ADC software Troubleshooting Guide sr adw are Feature Specific Troubleshooting gt Example user host gt show configuration interfaces display commit scripts ge 0 2 0 unit 0 family inet filter apply macro Input was added by automatic script input list rdwr default accept address 100 0 0 1 16 ge 0 3 0 unit 0 family inet filter apply macro Input was added by automatic script input list rdwr lbl be filt rdwr default accept address 200 0 0 1 16 ms 1 0 0 unit 0 family inet unit 10000 family inet filter apply macro This term was added by a commit script input rdwr ms 100 filt 42 Document ID RDWR RSLB V1 4R0 0_TSG0711 re p H radwa re ADC Software Troubleshooting Guide Feature Specific Troubleshooting Junos OS SDK API Automatic Configuration The ADC software uses the dynamic J unos OS SDK API for automatic configuration A firewall filter is added to the client facing interfaces using the J unos OS SDK API This firewa
3. instance lt adc name gt Document ID RDWR RSLB V1 4R0 0 TSG0711 49 ADC Software Troubleshooting Guide Feature Specific Troubleshooting License Information radware The ADC software must be licensed in order to run on your Multiservices DPC The license is set per chassis serial number and determines the maximum number of Multiservices DPC NPUs that can run the ADC software ei To verify the chassis serial number userehost gt show chassis hardware Hardware inventory Item Version Chassis Midplane REV 07 FPM Board REV 04 PEM 1 Rev 01 240V AC in PEM 2 01 240V AC in Routing Engine 0 09 CB 0 07 FPC 0 18 10GE X CPU 03 PIC 0 BUILTIN PIG 1 BUILTIN Xcvr 740 011613 Xcvr 740 011613 Xcvr NON JNPR Xcvr NON JNPR PIC 2 BUILTIN Xcvr 740 014289 PIG 3 BUILTIN Xcvr 740 014289 FPG 2 750 024064 CPU 710 013713 PIC 0 BUILTIN PIG T BUILTIN FPC 2 750 024064 CPU 710 013713 PIC 0 BUILTIN PIG 1 BUILTIN Fan Tray 0 710 030216 ei To verify the license installed in your chassis Serial number JN1111111111 ABAA8 888 YB2459 QCS1002C0A1 Part number 760 021404 760 021392 740 022697 740 022697 OCS1002C02E 740 015113 710 021523 750 022766 9009015040 YC1029 XY4672 710 022351 XX1139 BUILTIN BUILTIN PH25T3E PH25T3Y PT5342883 A0507085508 BUILTIN CA05BQ035 BUILTIN CA05BQ039 XT0653 XT4129 BUILTIN BUILTIN XR6311 XR6176 BUILTIN BUILTIN XV8381 D va Hew bi lt lt lt re lt
4. AD fi Troubleshooti j ee C So ae roubleshooting Guide er adw are Introduction Table 1 Acronyms Transmission Control Protocol User Datagram Protocol 16 Document ID RDWR RSLB V1 4R0 0_TSG0711 WI p p H radwa re ADC Software Troubleshooting Guide Troubleshooting Fundamentals Chapter 2 Troubleshooting Fundamentals This chapter provides conceptual information about the methods and tools used for troubleshooting and isolating problems in the Juniper Networks Application Delivery Controller ADC software The types of problems that typically occur with networks are connectivity and performance The ADC software and a Juniper Networks MX3D Unviersal Edge Router support a diverse range of network architectures and protocols some of which are used to maintain and monitor connectivity and isolate the connectivity faults This chapter includes the following topics e Port Mirroring on page 17 e Interface Statistics on page 19 e System Log and Trace File on page 19 Port Mirroring Port mirroring involves sending a copy of a data packet from a routing platform to an external host This section includes the following topics e Junges OS Port Mirroring on page 17 e ADC Software Port Mirroring on page 17 e Extended Port Mirroring Overview on page 18 e Port Mirroring Configuration on page 18 Junos OS Port Mirroring You can send a copy of an Internet Protocol version 4 IPv4 or Internet Protocol vers
5. ADC MGMT Daemon 11 1R1 14 1 4R0 0 int092 ADC Control Component 11 1R1 14 1 4R0 0 int092 ADC Dataplane Component 11 1R1 14 1 4R0 0 int092 26 Document ID RDWR RSLB V1 4R0 0_TSG0711 ra dw are ADC Software Troubleshooting Guide Initial Troubleshooting ei To view additional information about the software version Note The adc internal software version command displays detailed information about the current ADC software version userehost gt show extensions adc internal software version Software Build Information Software Version 1 4R0 0 Build ID 092 Build time FW VERSION 2 Wed Apr 20 13 36 58 IDT 2011 Saving Technical Support Information You can save the technical support file and send it to your technical Support personnel for debugging purposes The file contains device information statistics and configuration The output of the support command is directed to the console screen You can transfer the output into a file in order to Save it ei To print the maintenance information to the console screen user host gt request extensions radware maintenance information ei To save the maintenance information to a file userehost gt request extensions radware maintenance information save lt filename gt Viewing System Maintenance Information The maintenance command displays information about the maximum and currently enabled capacity for the connection table as well as various counters from
6. Dieser Code wird hiermit allgemein zuganglich gemacht Dieses Produkt enthalt einen vom OpenBSD Projekt entwickelten Code Copyright c 1983 1990 1992 1993 1995 The Regents of the University of California Alle Rechte vorbehalten Die Verbreitung und Verwendung in Quell und bin rem Format mit oder ohne Ver nderungen sind unter folgenden Bedingungen erlaubt 1 Die Verbreitung von Quellcodes muss den voranstehenden Copyrightvermerk diese Liste von Bedingungen und den folgenden Haftungsausschluss beibehalten 2 Die Verbreitung in bin rem Format muss den voranstehenden Copyrightvermerk diese Liste von Bedingungen und den folgenden Haftungsausschluss in der Dokumentation und oder andere Materialien die mit verteilt werden reproduzieren 3 Weder der Name der Universit t noch die Namen der Beitragenden d rfen ohne ausdr ckliche vorherige schriftliche Genehmigung verwendet werden um von dieser Software abgeleitete Produkte zu empfehlen oder zu bewerben Dieses Produkt enth lt von Markus Friedl entwickelte Software Dieses Produkt enth lt von Theo de Raadt entwickelte Software Dieses Produkt enth lt von Niels Provos entwickelte Software Dieses Produkt enth lt von Dug Song entwickelte Software Dieses Produkt enth lt von Aaron Campbell entwickelte Software Dieses Produkt enth lt von Damien Miller entwickelte Software Dieses Produkt enth lt von Kevin Steves entwickelte Software Dieses Produkt enth lt von Daniel Kouril entwick
7. radware availability performance security ADC Software Troubleshooting Guide Software Version 1 4R0 0 Document ID RDWR RSLB V1 4R0 0 TSG0711 July 2011 ADC Software Troubleshooting Guide 2 r adw are 2 Document ID RDWR RSLB V1 4R0 0_TSG0711 radware Important Notices The following important notices are presented in English French and German Important Notices This guide is delivered subject to the following conditions and restrictions Copyright Radware Ltd 2006 2011 All rights reserved The copyright and all other intellectual property rights and trade secrets included in this guide are owned by Radware Ltd The guide is provided to Radware customers for the sole purpose of obtaining information with respect to the installation and use of the Radware products described in this document and may not be used for any other purpose The information contained in this guide is proprietary to Radware and must be kept in strict confidence It is strictly forbidden to copy duplicate reproduce or disclose this guide or any part thereof without the prior written consent of Radware Notice importante Ce guide est sujet aux conditions et restrictions suivantes Copyright Radware Ltd 2006 2011 Tous droits r serv s Le copyright ainsi que tout autre droit li la propri t intellectuelle et aux secrets industriels contenus dans ce guide sont la propri t de Radware Ltd Ce guide d informations est four
8. Disconn License Licensed Document ID RDWR RSLB V1 4R0 0 TSG0711 47 j 1 WI ADC software Troubleshooting Guide sr adw are Feature Specific Troubleshooting 4 Verify that the routes in the front end routing instance point to the RMS and not to the Multiservices DPCs Look for the virtual server routes in the rdwr lt adc instance name gt fe ri The following example uses the adc instance name demo and the virtual server address 6 2 2 50 gt Example userehost gt show route all rdwr demo fe ri inet 0 6 destinations 7 routes 6 active 0 holddown 0 hidden Active Route Last Active Both 2 0 24 Direct 0 22 49 28 gt via fe 0 0 1 0 1 32 Local 0 22 49 28 Local via fe 0 0 1 0 0 24 Direct 0 22 49 28 gt via fe 0 0 3 0 Direct 0 22 49 28 gt via fe 0 0 3 0 Local 0 22 49 28 Local via fe 0 0 3 0 Local 0 22 49 28 Local via fe 0 0 3 0 Static 1 14 57 40 gt via rms0 0 5 Verify the RMS filter counter is defined and catching traffic Note The counters are defined only when the debug enable command is set For more information see The debug enable Command on page 43 user host gt show firewall filter rdwr rms0 filt Filter rdwr rms0 filt Counters Name Packets all 0 48 Document ID RDWR RSLB V1 4R0 0_TSG0711 re 2 FA ef adw are ADC Software Troubleshooting GUIR Feature Specific Troubleshooting Connection Synchronization Troubleshooting Use
9. Layer 3 to Layer 7 per Multiservices DPC NPU ei To view system maintenance information userehost gt show extensions adc internal maintenance lt Multiservices DPC PIC name gt Document ID RDWR RSLB V1 4R0 0 TSG0711 27 H i WI ABG Software Troubleshooting Guide sr adw are Initial Troubleshooting Providing a Network Topology Map Every network is designed differently You should maintain a detailed and accurate topology diagram of your network showing the nodes and connections This visual depiction of your network is very helpful to technical support personnel when they assess your problem Documenting System and Network Changes When troubleshooting a problem verify if anything has changed in the network recently The following are a few questions that help you analyze and document changes to your system network e Have you recently changed or upgraded your system your network or a custom application For example has any configuration or code been changed e When were these changes made Provide the date and time e Who made these changes Were the changes made by a partner or customer Provide the names of the individuals who made the changes e Which events such as an upgrade a LAN change increased traffic or new hardware can be identified prior to the trouble occurring Assessing Panic An ADC software crash is also called a panic When a panic happens the core dump file saves the ADC software status
10. RDWR RSLB V1 4R0 0_TSG0711 17 H H WI ADC software Troubleshooting Guide R radwa re Troubleshooting Fundamentals Caution If you apply the port mirroring filter to the input side of an IFL which is set up in the ADC software configuration one of the functions either the ADC software or the mirroring will not work on the traffic You should not use this type of configuration Extended Port Mirroring Overview This section discusses how the server facing and client facing interfaces perform port mirroring This section includes the following topics e Server Facing Interfaces on page 18 e Client Facing Interfaces on page 18 Server Facing Interfaces The ADC software uses input lists to add filters to the server facing interfaces If you configure a port mirroring filter it will be first in the input list as the rest of the filters are configured by commit script after the user configuration This implies that the ADC software will not see the traffic from these ports Client Facing Interfaces The ADC software is using the J unos OS SDK API to apply input filters on the client facing logical interfaces When using the API to set filters on an IFL the API filter has higher priority over user configured filters This implies that if the port mirroring filter is applied to a client facing interface input it will not see the traffic that is sent to the ADC software Port Mirroring Configuration The following example il
11. Radware upon request A copy of the license can be viewed at http www gnu org licenses old licenses gpl 2 0 htm This code is hereby placed in the public domain This product contains code developed by the OpenBSD Project Copyright c 1983 1990 1992 1993 1995 The Regents of the University of California All rights reserved Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met 1 Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer 2 Redistributions in binary form must reproduce the above copyright notice this list of conditions and the following disclaimer in the documentation and or other materials provided with the distribution 3 Neither the name of the University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission This product includes software developed by Markus Friedl This product includes software developed by Theo de Raadt This product includes software developed by Niels Provos This product includes software developed by Dug Song This product includes software developed by Aaron Campbell This product includes software developed by Damien Miller This product includes software developed by Kevin Steves 4 Document ID RDWR RSLB V1 4R0 0_TSG0711 WI p H rad
12. Redundant Multiservices PIC RMS for high availability As part of the high availability solution the ADC software passes the connection table data from the active Multiservices DPC NPU in the RMS to the backup Multiservices DPC NPU This process is called connection synchronization This section explains how to troubleshoot the RMS and connection synchronization mechanisms This section includes the following topics e RMS Troubleshooting on page 47 e Connection Synchronization Troubleshooting on page 49 46 Document ID RDWR RSLB V1 4R0 0_TSG0711 WI 2 FA sr adw are ADC Software Troubleshooting GUIR Feature Specific Troubleshooting RMS Troubleshooting Use the following procedure to troubleshoot RMS issues ei To troubleshoot high availability for the ADC software 1 Check that the RMS configuration is set up correctly in Junos OS user host gt show configuration interfaces men redundancy options primary ms 0 2 0 secondary ms 0 3 0 hot standby 2 Check that the adc instance is configured to use the RMS and not the Multiservices DPCs user host gt show configuration extensions adc adc instance demo router interfaces ms interfaces rms0 3 Verify that the RMS appears in the ADC status command as an active interface userehost gt show extensions adc status SLB daemon status Up Up Since Tue Mar 15 18 56 33 2011 adc instance kobis Interface Status Control Daemon Data Daemons up
13. THIS SOFTWARE EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Notice traitant du copyright Ce produit renferme des codes d velopp s dans le cadre du projet OpenSSL Ce produit inclut un logiciel d velopp dans le cadre du projet OpenSSL Pour un usage dans la boite a outils OpenSSL http www openssl org Copyright c 1998 2005 Le projet OpenSSL Tous droits r serv s Ce produit inclut la cat gorie de chiffre Rijndael L impl mentation de Rijindael par Vincent Rijmen Antoon Bosselaers et Paulo Barreto est du domaine public et distribu e sous les termes de la licence suivante version 3 0 D cembre 2000 Code ANSI C code pour Rijndael actuellement AES author Vincent Rijmen lt vincent rij men esat kuleuven ac be gt author Antoon Bosselaers lt antoon bosselaers esat kuleuven ac be gt author Paulo Barreto lt paulo barreto terra com br gt Le commutateur OnDemand peut utiliser les composants logiciels sous licence en vertu des termes de la licence GNU General Public License Agreement Version 2 GPL v 2 y compris les projets a source ouverte LinuxBios et Filo Le code source de LinuxBios et Filo est disponible sur demande aupr s de Radware Une copie de la licence est r pertori e sur http www gnu org licenses old licenses gpl 2 0 htm Ce code est galement plac dans le domaine public Ce produit renferme des codes d velopp s dans le cadre du projet OpenSSL Copyright c 1983 1990 1992 199
14. Tools This section explains how to use specific troubleshooting tools available in the ADC software This section includes the following topics e Filter Log Action on page 38 e Server Operational Status on page 39 e Connection Table Troubleshooting on page 39 Filter Log Action When working with filters transparent load balancing you can add a log action to a filter This will cause each packet that matched the filter conditions send a syslog message The log action can be configured in parallel with other filter actions Caution Using the log action on filters has a severe impact on performance as it sends a syslog message for every relevant packet ei To configure filter log action edit extensions adc adc instance lt name gt filters term lt name gt user host set then log 38 Document ID RDWR RSLB V1 4R0 0 TSG0711 WI p H radwa re ADC Software Troubleshooting Guide Extended Troubleshooting Server Operational Status While debugging you may want to disable or enable a server This is called changing the server operational status Changing the server operational status also lets you replace a server or put it into a temporary down state Note The disable and enable states are not saved after reboot After reboot all servers are considered enabled If you want the server not to participate in load balancing after reboot you should deactivate the server in configuration mode Ste
15. in the file system prior to the panic This is useful in backtracing the events which led to the panic The information is held in core dump files according to the process that had the panic The files are numbered starting from 0 for each process The files are created under the directory var tmp in the router The ADC software running in the routing engine adc mgmt core dump file is called adc mgmt core tarball X tgz The Multiservices DPC NPU core dump file is called adc ctrl core msXX X gz AND OR adc data core msXX X gz In many cases both are created Caution After the panic happens the core dump file is created It can take up to 20 minutes before the file is ready Collecting System Statistics In each Multiservices DPC NPU there is 1 control processor CP and 21 data processors DP While in the Junos OS these numbers are configurable the ADC software only uses these specific numbers 1 control core and 7 data cores are translated into 1 control processor and 21 data processors This section explains in detail how to capture these statistics for troubleshooting It includes the following topics e Control Processor CPU Use on page 29 e Switch Processor on page 29 28 Document ID RDWR RSLB V1 4R0 0_TSG0711 33 AD fi Troubleshooti j sr adw are C Software roub eshooting Suge Initial Troubleshooting Control Processor CPU Use You can capture Controller Processor CP CPU use by using the cpu comma
16. 1 4R0 0_TSG0711 radware Table of Contents Important Nolices eu ee 3 PINNE 4 Document Conventions is 9 Chapter 1 IHWoduchon in naeh 15 PIErEQU SIE sen ee rer 15 PETE eek 15 Chapter 2 Troubleshooting FundamentalsS sssvnnnnnnnnnnvnnnnnnnnnnnnnnnnnnnnnnnnnnvnnnnnr 17 Pogues een een 17 Junos OS Port Mirroring iii 17 ADC Software Port Mirroring iii 17 Extended Port Mirroring Overview iii 18 Port Mirroring Configuration sis 18 Interface Statistics E 19 System Log and Trace File u a 19 Configuring the Syslog File sisi 20 Viewing The Syslog File ur sen annee eat 20 Accessing the Syslog File iii 21 Configuring the Trace File iii 21 Viewing the Trace File iii 21 Accessing the Trac File ee ee lande nn nee 22 Chapter 3 Initial Troubleshooting es 23 Gathering Information sic cusses den ea 23 Viewing Chassis Information sir 23 Viewing General System Information ss 24 Viewing the Software Version iii 26 Saving Technical Support Information reenrennnnvrnnonvrnrnnrrnnrnnnnnrnrrrnnnnvnnrnrrreennnnerenrreennnnnseenn 27 Viewing System Maintenance Information eannvnnnonrrnnnnrnnnonvnnnnnnvnnnnnrnnnnrrnnnnnnnnrnnrnenennnnnnn 27 Providing a Network Topology Map ss 28 Documenting System and Network Changes 44s4s440nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnann 28 le H 28 Collecting System Statistics u men 28 Control Processor CPU Use iii 2
17. 3 1995 Les membres du conseil de l Universit de Californie Tous droits r serv s Document ID RDWR RSLB V1 4R0 0 TSG0711 5 ji I WI ADC Software Troubleshooting Guide sr adw are La distribution et l usage sous une forme source et binaire avec ou sans modifications est autoris e pour autant que les conditions suivantes soient remplies 1 La distribution d un code source doit inclure la notice de copyright mentionn e ci dessus cette liste de conditions et l avis de non responsabilit suivant 2 La distribution sous une forme binaire doit reproduire dans la documentation et ou dans tout autre mat riel fourni la notice de copyright mentionn e ci dessus cette liste de conditions et l avis de non responsabilit suivant 3 Le nom de l universit ainsi que le nom des contributeurs ne seront en aucun cas utilis s pour approuver ou promouvoir un produit d riv de ce programme sans l obtention pr alable d une autorisation crite Ce produit inclut un logiciel d velopp par Markus Friedl Ce produit inclut un logiciel d velopp par Theo de Raadt Ce produit inclut un logiciel d velopp par Niels Provos Ce produit inclut un logiciel d velopp par Dug Song Ce produit inclut un logiciel d velopp par Aaron Campbell Ce produit inclut un logiciel d velopp par Damien Miller Ce produit inclut un logiciel d velopp par Kevin Steves Ce produit inclut un logiciel d velopp par Daniel Kouril Ce produit inclut
18. 58 illustrates the emergency recovery tree for switch panics Figure 5 Switch Panic Recovery Diagram Restart the adc Where is the mgmt service by Panic Multiservice PIC Make the Multiservice PIC offline and online by the request chassis pic command Reinstall the ADC software Collect maintenance and core dump information plus any information available such as traces connection table print taken before the crash etc Is service up Restart the router 58 Document ID RDWR RSLB V1 4R0 0_TSG0711 a er adw are ADC Software Troubleshooting Guide Emergency Recovery Tree Real Server Down Figure 6 on page 59 through Figure 9 on page 62 illustrate the emergency recovery tree to follow when a real server becomes unavailable Figure 6 Real Server Down Recovery Diagram 1 Handle port down situation not an ADC issue Real server is reachable using ping Handle next hop down not an ADC issue Document ID RDWR RSLB V1 4R0 0 TSG0711 59 ADC Software Troubleshooting Guide 4 r adw are Emergency Recovery Tree Figure 7 Real Server Down Recovery Diagram 2 60 Document ID RDWR RSLB V1 4R0 0_TSG0711 LA 1 Ti 1 h D D Ku radware ADC Software Troubleshooting Guide Emergency Recovery Tree Figure 8 Real Server Down Recovery Diagram 3 Document ID RDWR RSLB V1 4R0 0 TSG0711 61 ADC Software Troubleshooting Guide gt KM r adw are Em
19. 9 Switch tee TEE 29 Chapter 4 Extended Troubleshooting es 33 System and Hardware Troubleshooting mrrrnrrrnnnnnvrrnnnnonvrrnnnnnnnnnnrnrrrrnnnnnrnrrnnsnnnnnnnnnn 33 System Troubleshooting sis 33 Document ID RDWR RSLB V1 4R0 0_TSG0711 11 H i ve ADC Software Troubleshooting Guide ef adw are Table of Contents Hardware Troubleshooting sise 34 Multiservices DPC NPU Troubleshooting 34 Troubleshooting OSI layers a ee ae 34 Layer and Layer2 nette einen 34 Ler 35 Layer EE 36 ENE EE EE 38 Troubleshooting Tools sera es Dr 38 Filter Log Action zn a la aa 38 Server Operational Status is 39 Connection Table Troubleshooting 44440srsnnonnnnnnennnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnenn 39 Chapter 5 Feature Specific Troubleshooting rrrrrrrrrnvnvnnnnnnnnnnnnnnnnnnnnnnevenenvvennr al ADC Automatic Configuration EE 41 Commit Script Automatic Configuration urrnnvnnrorvrrnnnvnnnonvnnrnnrnvnnnnnnnnnrrennnnnnnrnrrenannnnnennnne 41 Junos OS SDK API Automatic Configuration reernrrnnnnnvnnrnrrrnnnnrnnnrnrrnnnnrrnnrnnrnnrrrrneennnnerenn 43 The debug enable Command is 43 N T IP Addresses a un er EE Ce 45 High Availability iii 46 RMS Troubleshooting iii 47 Connection Synchronization Troubleshooting rrrnserrrrnnnrrnnnrrrvnnnnrnnnenvnnnnnnrnnrrnrenennnnnnennnen 49 License Information en en 50 Chapter 6 Emergency Recovery Tree mmssvvvr
20. Description MX240 MX240 Backplane Front Panel Display PS 1 2 1 7kW 100 PS 1 2 1 7kW 100 RE S 1300 MX SCB DPCE 20x 1GE 2x DPC PMB 10x 1GE LAN 10x 1GE LAN SFP SX FP SX FP SX FP SX x 10GE LAN WAN FP 10G SR x 10GE LAN WAN FP 10G SR MS DPC DPC PMB MS DPC PIC MS DPC PIC MS DPC DPC PMB MS DPC PIC MS DPC PIC Enhanced Fan Tray user host gt show system license 50 Document ID RDWR RSLB V1 4R0 0_TSG0711 WI p p H radwa re ADC Software Troubleshooting Guide Feature Specific Troubleshooting ei To verify the license defined and Multiservices PI Cs allowed by the license userehost gt show extensions adc license info Document ID RDWR RSLB V1 4R0 0 TSG0711 51 AD fi Troubleshooti j H C So ware roubles ooting Guide ra dw are Feature Specific Troubleshooting 52 Document ID RDWR RSLB V1 4R0 0_TSG0711 s AD fi Troubleshooti j H radware C Software Troubleshooting Guide Emergency Recovery Tree Chapter 6 Emergency Recovery Tree This chapter outlines the procedures for recovering as quickly as possible from field outages It includes the following topics e VIP Is Not Working on page 53 e ADC Software Panic on page 58 e Real Server Down on page 59 VIP Is Not Working Figure 1 on page 54 through Figure 4 on page 57 illustrate the emergency recovery tree to follow when the VIP is not working Document ID RDWR RSLB V1 4R0 0_TSG0711 53 ADC Software Tr
21. HUNGEN GLEICH WIE SIE ENTSTANDEN SIND UND F R JEGLICHE ART VON HAFTUNG SEI ES VERTR GE Document ID RDWR RSLB V1 4R0 0_TSG0711 7 ji I LL ADC Software Troubleshooting Guide ef adw are GEF HRDUNGSHAFTUNG ODER DELIKTISCHE HAFTUNG EINSCHLIESSLICH FAHRL SSIGKEIT ODER ANDERE DIE IN JEGLICHER FORM FOLGE DER BENUTZUNG DIESER SOFTWARE IST SELBST WENN AUF DIE M GLICHKEIT EINES SOLCHEN SCHADENS HINGEWIESEN WURDE 8 Document ID RDWR RSLB V1 4R0 0_TSG0711 WI p H radwa re ADC Software Troubleshooting Guide Document Conventions The following describes the conventions and symbols that this guide uses Item Description Description French Beschreibung German gt An example scenario Un sc nario d exemple Ein Beispielszenarium Example Possible damage to Endommagement M gliche Sch den an equipment software or possible de l quipement Ger t Software oder data des donn es ou du Daten logiciel Additional information Informations Zus tzliche compl mentaires Informationen A statement and R f rences et Eine Erkl rung und instructions instructions Anweisungen A suggestion or Une suggestion ou Ein Vorschlag oder eine workaround solution Umgehung Possible physical harm to Blessure possible de Verletzungsgefahr des the operator l op rateur Bedieners Warning Document ID RDWR RSLB V1 4R0 0_TSG0711 9 ji I WI ADC Software Troubleshooting Guide gr adw are 10 Document ID RDWR RSLB V
22. TE DOMMAGEABLE Y COMPRIS LA N GLIGENCE OU AUTRE D COULANT DE QUELLE QUE FA ON QUE CE SOIT DE L USAGE DE CE LOGICIEL M ME S IL A T AVERTI DE LA POSSIBILIT D UN TEL DOMMAGE Copyrightvermerke Dieses Produkt enth lt einen vom OpenSSL Projekt entwickelten Code Dieses Produkt enth lt vom OpenSSL Projekt entwickelte Software Zur Verwendung im OpenSSL Toolkit http www openssl org Copyright c 1998 2005 The OpenSSL Project Alle Rechte vorbehalten Dieses Produkt enthalt die Rijndael cipher Die Rijndael Implementierung von Vincent Rijndael Anton Bosselaers und Paulo Barreto ist ffentlich zug nglich und wird unter folgender Lizenz vertrieben version 3 0 December 2000 6 Document ID RDWR RSLB V1 4R0 0_TSG0711 WI p H radwa re ADC Software Troubleshooting Guide Optimierter ANSI C Code f r den Rijndael cipher jetzt AES author Vincent Rijmen lt vincent rij men esat kuleuven ac be gt author Antoon Bosselaers lt antoon bosselaers esat kuleuven ac be gt author Paulo Barreto lt paulo barreto terra com br gt Der OnDemand Switch verwendet m glicherweise Software die im Rahmen der DNU Allgemeine Offentliche Lizenzvereinbarung Version 2 GPL v 2 lizensiert sind einschlieRlich LinuxBios und Filo Open Source Projekte Der Quellcode von LinuxBios und Filo ist bei Radware auf Anfrage erhaltlich Eine Kopie dieser Lizenz kann eingesehen werden unter http www gnu org licenses old licenses gpl 2 0 html
23. Troubleshooting on page 34 e Multiservices DPC NPU Troubleshooting on page 34 System Troubleshooting Use the following commands to troubleshoot system issues ei To verify general system related issues userehost gt show hardware ei To ensure no alarms are raised for your device userehost gt show chassis alarms Document ID RDWR RSLB V1 4R0 0 TSG0711 33 j 1 WI ADC Software Troubleshooting Guide er adw are Extended Troubleshooting Hardware Troubleshooting Use the following command to troubleshoot hardware issues ei To verify chassis and hardware issues Multiservices DPC NPU Troubleshooting Use the following commands to troubleshoot issues with the Multiservices DPC NPU ei To verify the Multiservices DPC status user host gt show chassis fpc pic status Note This command may show a Multiservices DPC as being up but that does not mean that it is ready to run ADC traffic It may mean that it is in the process of loading the ei ADC software and is not yet ready to process traffic Use the adc status command to verify the status of the ADC software To change the Multiservices DPC status between offline and online user host gt request chassis pic offline fpc slot lt slot gt pic slot lt pic gt OR user host gt request chassis pic online fpc slot lt slot gt pic slot lt pic gt Troubleshooting OSI layers This section explains troubleshooting using the Open Systems In
24. X SFP SX SFP SX 1x 10GE LAN WAN XFP 10G SR 1x 10GE LAN WAN XFP 10G SR MS DPC DPC PMB MS DPC PIC MS DPC PIC MS DPC DPC PMB MS DPC PIC MS DPC PIC Enhanced Fan Tray The adc status command also has a detail option that shows more information on the adc mgmt daemon Note After the Multiservices DPC is online the Multiservices DPC is starting to load the ADC software This process takes some time If the Multiservices DPC is online but appears as down in the adc status command it is probably loading the software 24 Document ID RDWR RSLB V1 4R0 0_TSG0711 re p p H radwa re ADC Software Troubleshooting Guide Initial Troubleshooting ei To view the general system information user host gt show extensions adc status SLB daemon status Up Up Since Wed Mar 14 12 45 22 2012 adc instance 1b1l Interface Status Control Daemon Data Daemons up Disconnections License ms 1 0 0 Licensed OR user host gt show extensions adc status detail SLB daemon status Up Up Since Wed Mar 14 12 45 22 2012 SLB daemon to SDK Services daemon SSD connection current status Up number of disconnections 0 SLB daemon to Dynamic Firewall Filters Daemon DFWD connection current status Up number of disconnections 0 Interface status Interfaces running adc ms 1 0 0 ms 1 1 0 Interfaces that appear in the configuration ms 1 0 0 adc instance 1b1l Interface Status Control Daemon Dat
25. a Daemons up Disconnections License ms 1 0 0 Licensed Unattached Interface Status Document ID RDWR RSLB V1 4R0 0 TSG0711 25 ADG Software Troubieshooing Guide Ha r adw are Initial Troubleshooting Viewing the Software Version The following examples illustrate how to view the software version on the device ei To view the software version Note The show version command displays the software version for all software running on the device userehost gt show version Hostname host Model mx240 JUNOS Base OS boot 11 1R1 14 JUNOS Base OS Software Suite 11 1R1 14 JUNOS Kernel Software Suite 11 1R1 14 JUNOS Packet Forwarding Engine Support M T Common 11 1R1 14 JUNOS Packet Forwarding Engine Support MX Common 11 1R1 14 JUNOS Online Documentation 11 1R1 14 JUNOS Voice Services Container package 11 1R1 14 JUNOS Border Gateway Function package 11 1R1 14 JUNOS Services AACL Container package 11 1R1 14 JUNOS Services LL PDF Container package 11 1R1 14 JUNOS Services PTSP Container package 11 1R1 14 JUNOS Services Stateful Firewall 11 1R1 14 JUNOS Services NAT 11 1R1 14 JUNOS Services Application Level Gateways 11 1R1 14 JUNOS Services Captive Portal and Content Delivery Container package 11 1R1 14 JUNOS Services RPM 11 1R1 14 JUNOS AppId Services 11 1R1 14 JUNOS IDP Services 11 1R1 14 JUNOS Runtime Software Suite 11 1R1 14 JUNOS Routing Software Suite 11 1R1 14
26. are Copyright Notices The following copyright notices are presented in English French and German Copyright Notices This product contains work derived from the RSA Data Security Inc MD5 Message Digest Algorithm RSA Data Security Inc makes no representations concerning either the merchantability of the MD5 Message Digest Algorithm or the suitability of the MD5 Message Digest Algorithm for any particular purpose Itis provided as is without expressed or implied warranty of any kind This product contains code developed by the OpenSSL Project This product includes software developed by the OpenSSL Project For use in the OpenSSL Toolkit http www openssl org Copyright c 1998 2005 The OpenSSL Project All rights reserved This product contains the Rijndael cipher The Rijndael implementation by Vincent Rijmen Antoon Bosselaers and Paulo Barreto is in the public domain and distributed with the following license version 3 0 December 2000 Optimized ANSI C code for the Rijndael cipher now AES author Vincent Rijmen lt vincent riimen esat kuleuven ac be gt author Antoon Bosselaers lt antoon bosselaers esat kuleuven ac be gt author Paulo Barreto lt paulo barreto terra com br gt The OnDemand Switch may use software components licensed under the GNU General Public License Agreement Version 2 GPL v 2 including LinuxBios and Filo open source projects The source code of the LinuxBios and Filo is available from
27. are command to show this information ei To view the chassis information userehost gt show chassis hardware Hardware inventory Version Item Chassis Midplane FPM Board PEM 1 240V AC in PEM 2 240V AC in Routing Engine 0 CB 0 FPC 0 10GE X CPU PIC 0 PIC 1 Xcvr Xcvr Xcvr Xcvr PIC 2 Xcvr PIC 3 Xcvr FPC 1 CPU PIC PIC FPC CPU PIC PIC Fan Tray 0 REV 07 REV 04 01 01 09 07 18 03 Part number 760 021404 760 021392 740 022697 740 022697 740 015113 710 021523 750 022766 710 022351 BUILTIN BUILTIN 740 011613 740 011613 NON JNPR NON JNPR BUILTIN 740 014289 BUILTIN 740 014289 750 024064 710 013713 BUILTIN BUILTIN 750 024064 710 013713 BUILTIN BUILTIN 710 030216 Viewing General System Information The adc status command displays technical information about the ADC software status When troubleshooting a problem use this command to obtain useful information about the ADC software Serial number JN1111111111 ABAA8 888 YB2459 QCS1002C0A1 QCS1002C02E 9009015040 YC1029 XY4672 XX1139 BUILTIN BUILTIN PH25T3E PH25T3Y PT5342883 A0507085508 BUILTIN CA05BQ035 BUILTIN CA05BQ039 XT0653 XT4129 BUILTIN BUILTIN XR6311 XR6176 BUILTIN BUILTIN XV8381 radware Description MX240 MX240 Backplane Front Panel Display PS 1 2 1 7kW 100 PS 1 2 1 7kW 100 RE S 1300 MX SCB DPCE 20x 1GE 2x DPC PMB 10x 1GE LAN 10x 1GE LAN SFP SX SFP S
28. e statistics which is from the router point of view The input direction is for packets coming from the Multiservices DPC to the router The output direction is for packets coming from the router to the Multiservices DPC The interface statistics can be shown by using the following command System Log and Trace File You can use the system log syslog and trace file to discover more information about the ADC software The syslog is used by the Multiservices DPC to log messages such as a real server coming up or going down a virtual server becoming available and so on The trace file is used by the ADC software running in the routing engine adc mgmt to print state and debug information Document ID RDWR RSLB V1 4R0 0 TSG0711 19 H H WI ADC software Troubleshooting Guide R radwa re Troubleshooting Fundamentals This section includes the following topics e Configuring the Syslog File on page 20 e Viewing the Syslog File on page 20 e Accessing the Syslog File on page 21 e Configuring the Trace File on page 21 e Viewing the Trace File on page 21 e Accessing the Trace File on page 22 Configuring the Syslog File The following example illustrates how to correctly set up your syslog for your device ei To configure the syslog for the device Note In this example the syslog file is configured globally for the router under the system hierarchy edit system syslog file device log any any
29. elte Software Dieses Produkt enth lt von Wesley Griffin entwickelte Software Dieses Produkt enth lt von Per Allansson entwickelte Software Dieses Produkt enth lt von Nils Nordman entwickelte Software Dieses Produkt enth lt von Simon Wilkinson entwickelte Software Die Verbreitung und Verwendung in Quell und bin rem Format mit oder ohne Ver nderungen sind unter folgenden Bedingungen erlaubt 1 Die Verbreitung von Quellcodes muss den voranstehenden Copyrightvermerk diese Liste von Bedingungen und den folgenden Haftungsausschluss beibehalten 2 Die Verbreitung in bin rem Format muss den voranstehenden Copyrightvermerk diese Liste von Bedingungen und den folgenden Haftungsausschluss in der Dokumentation und oder andere Materialien die mit verteilt werden reproduzieren S MTLICHE VORGENANNTE SOFTWARE WIRD VOM AUTOR IM IST ZUSTAND AS IS BEREITGESTELLT JEGLICHE AUSDR CKLICHEN ODER IMPLIZITEN GARANTIEN EINSCHLIESSLICH DOCH NICHT BESCHR NKT AUF DIE IMPLIZIERTEN GARANTIEN DER MARKTG NGIGKEIT UND DER ANWENDBARKEIT F R EINEN BESTIMMTEN ZWECK SIND AUSGESCHLOSSEN UNTER KEINEN UMST NDEN HAFTET DER AUTOR F R DIREKTE ODER INDIREKTE SCH DEN F R BEI VERTRAGSERF LLUNG ENTSTANDENE SCH DEN F R BESONDERE SCH DEN F R SCHADENSERSATZ MIT STRAFCHARAKTER ODER F R FOLGESCH DEN EINSCHLIESSLICH DOCH NICHT BESCHR NKT AUF ERWERB VON ERSATZG TERN ODER ERSATZLEISTUNGEN VERLUST AN NUTZUNG DATEN ODER GEWINN ODER GESCH FTSUNTERBREC
30. ergency Recovery Tree Figure 9 Real Server Down Recovery Diagram 4 62 Document ID RDWR RSLB V1 4R0 0_TSG0711 radware Index A accessing the syslog file 21 accessing the trace file 22 acronyms 15 ADC automatic configuration 41 ADC software panic 58 ADC software port mirroring 17 assessing panic 28 C chassis information viewing 23 client facing interfaces 18 commit script automatic configuration 41 configuring the syslog 20 configuring the trace file 21 connection synchronization 49 connection table 39 control processor CPU use 29 cp CPU use 29 CPU use control processor 29 cp 29 crash panic 28 D debug enable flag 43 documenting network changes 28 system changes 28 down emergency recovery real server down 59 DP CPU statistics 30 DP maintenance statistics 31 E emergency recovery ADC software panic 58 VIP not working 53 emergency recovery tree 53 extended discussion client facing interfaces 18 server facing interfaces 18 extended port mirroring discussion 18 extended troubleshooting 33 hardware 33 system 33 Document ID RDWR RSLB V1 4R0 0 TSG0711 F feature specific troubleshooting 41 filter log action 38 G gathering information 23 chassis information 23 general system information 24 software version 26 system and network changes 28 system maintenance 27 gathing information network topology map 28 general system information viewing 24 H hardware troubleshootin
31. filtered by each of the following parameters e destination address e destination port e destination address and port e filter term e nat address e real server e source address e source port ei To delete an entry in the connection table user host gt clear extensions adc connection entry lt instance name gt lt destination address gt lt destination port gt lt source address gt lt source port gt lt tcp udp gt To delete the entire connection table for an ADC instance user host gt clear extensions adc connection table lt instance name gt 40 Document ID RDWR RSLB V1 4R0 0_TSG0711 WI p H radware ADC Software Troubleshooting Guide Feature Specific Troubleshooting Chapter 5 Feature Specific Troubleshooting This chapter describes the feature specific troubleshooting tools available in the ADC software It includes the following topics e ADC Automatic Configuration on page 41 e NAT IP Addresses on page 45 e High Availability on page 46 e License Information on page 50 ADC Automatic Configuration The ADC software uses automatic configuration to handle packet receiving and sending to and from the device This section describes how to troubleshoot the automatic configuration and how to use it for further troubleshooting This section includes the following topics e Commit Script Automatic Configuration on page 41 e Junges OS SDK API Automatic Configuration on page 43
32. g 33 34 high availability 46 connection synchronization 49 RMS troubleshooting 47 I initial troubleshooting 23 assessing panic 28 gathering information 23 system statistics 28 technical support information 27 interface statistics 19 introduction 15 J Junos OS port mirroring 17 L layer 1 34 layer 2 34 layer 3 35 layer 4 36 layer 7 38 license information 50 M map network topology 28 63 radware multiservices dpc npu troubleshooting 34 N NAT IP address 45 network topology map 28 O OSI layers 34 layer 1 34 layer 2 34 layer 3 35 layer 4 36 layer 7 38 P panic assessing 28 crash 28 port mirroring 17 ADC software 17 configuration 18 extended discussion 18 Junos OS 17 port mirroring configuration 18 prerequisites 15 R real server down 59 RMS troubleshooting 47 S saving technical support information 27 SDK API automatic configuration 43 server operational status 39 server facing interfaces 18 software version viewing 26 statistics DP CPU 30 dp maintenance 31 switch processor 29 dp cpu 30 dp maintenance statistics 31 syslog Document ID RDWR RSLB V1 4R0 0_TSG0711 accessing 21 configuring 20 viewing 20 system log 19 system maintenance information viewing 27 system statistics collecting 28 control processor cpu use 29 switch processor 29 system troubleshooting 33 T technical support information saving 27 tools 38 filter log action 38 server operational s
33. ices DPC NPUs running the ADC software ei To configure the debug enable flag edit extensions adc user host set debug enable Document ID RDWR RSLB V1 4R0 0 TSG0711 43 j 1 WI ADC software Troubleshooting Guide sr adw are Feature Specific Troubleshooting Since the commit script uses an input list to bind filters to the device interfaces the counter names for these filters are changed by the Junos OS The relevant counter name is always the interface name followed by an i string gt Example A To view the firewall filters counters configured by the commit script for fe 1 3 3 user host gt show firewall filter fe 1 3 3 0 i Filter fe 1 3 3 0 i Counters Name default accept fe 1 3 3 0 i demo accept to serv if fe 1 3 3 0 i demo first fragment fe 1 3 3 0 i demo icmp fe 1 3 3 0 i demo non first fragment fe 1 3 3 0 i rdwr ms 100 be counter fe 1 3 3 0 i 13398625 B To view the counters for the firewall filters configured by the J unos OS SDK API user host gt show extensions adc internal generated configuration Front End firewall Filter name rdwr fe filt Related interfaces ge 0 2 0 0 Adc instance 1b1 Active VIPs 150 0 0 120 routes added Non active VIPs None Routing instance rdwr lbl fe ri Packet counter 10 Total Bytes 1244 44 Document ID RDWR RSLB V1 4R0 0_TSG0711 re p H radwa re ADC Software Troubleshooting Guide Feature Specific Troubleshooting NAT IP Addresses Use the f
34. if To disable a server from an ADC instance user host gt request extensions adc disable real server lt server name gt if To enable a server from an ADC instance Ste user host gt request extensions adc enable real server lt server name gt if To disable all servers from an ADC instance ho user host gt request extensions adc disable adc instance lt adc name gt if To enable all servers from an ADC instance Ste userehost gt request extensions adc enable adc instance lt adc name gt Connection Table Troubleshooting The connection table contains all tracked connections in the ADC software Each connection that is not part of a per packet load balance service or filter will appear in the connection table Information in the connection table can help you determine if a session is recognized by the ADC software Each connection table entry displays information on which server was chosen for the connection and if the connection is involved in extended processing such as NAT or persistency if To show connection table counters for an open connections Ste userehost gt show extensions adc connection table Document ID RDWR RSLB V1 4R0 0 TSG0711 3 AD fi Troubleshooti j se C Software Troub es ooting Guide fra dw are Extended Troubleshooting ei To show the connection table entries user host gt show extensions adc connection table extensive adc instance 1bl Connection table entries can be
35. in bytes Current memory in bytes allocs frees alloc failures bytes hiwait Document ID RDWR RSLB V1 4R0 0_TSG0711 31 H i WI ADG Software Troubleshooting Guide gr adw are Initial Troubleshooting 32 Document ID RDWR RSLB V1 4R0 0 TSG0711 3 AD fi Troubleshooti j radware C Software Troubleshooting Sue Extended Troubleshooting Chapter 4 Extended Troubleshooting This chapter explains extended troubleshooting using a step by step model to systematically identify and rule out problems with hardware software layers and different software features Therefore it is recommended to follow the procedures in this chapter in the order they appear This approach helps you to maximize your time by minimizing the chance that the problem you are attempting to resolve resides in a different troubleshooting layer than the layer you are currently analyzing This chapter includes the following topics e System and Hardware Troubleshooting on page 33 e Troubleshooting OSI layers on page 34 e Troubleshooting Tools on page 38 System and Hardware Troubleshooting This section explains troubleshooting that is related to the device system and the device hardware This section is only a small portion of the possible troubleshooting actions for such issues For more information see your Juniper Networks troubleshooting documentation This section includes the following topics e System Troubleshooting on page 33 e Hardware
36. ined for each static route and that the routes for dynamic networks appear in the routing table Use the following commands to diagnosis issues with Layer 3 u Document ID RDWR RSLB V1 4R0 0_TSG0711 3 j 1 WI ADC Software Troubleshooting Guide er adw are Extended Troubleshooting ei To verify the interfaces configuration and status user host gt show configuration interface AND user host gt show interface terse ei To verify that the routing table uses the correct next hops and interfaces ei To display information on the routes in the device userehost gt show route table lt all gt AND user host gt show route forwarding table ei To verify VRRP issues userehost gt show vrrp Layer 4 Check the following to diagnose issues in Layer 4 e The correct interfaces are configured as client and server interfaces e There are failed or blocked services Blocked services indicate that another real service is failing a health check within the same group e Real servers are up e Real servers are part of the desired group e The correct Layer 4 configuration client server NAT hot standby and filtering is configured e Verify the virtual server and service status If the status of the services is down then make sure that the associated real server status is up and functioning e Verify the virtual server routes are added to the front end routing instance fe ri e Verify if the real
37. ion 6 IPV6 packet from the routing platform to an external host address or a packet analyzer for analysis This is known as port mirroring Port mirroring is different from traffic sampling In traffic sampling a sampling key based on the packet header is sent to the routing engine There the key can be placed in a file or certain packets based on the key can be sent to a flow analysis server In port mirroring the entire packet is copied and sent out through a next hop interface For more information on port mirroring see the Juniper Networks documentation on port mirroring ADC Software Port Mirroring The ADC software accomplishes port mirroring using port mirroring filters that are applied to interfaces IFLs The port mirroring filters can be applied to the input or output side of the IFL However the port mirroring filters have an implied accept action that accepts all traffic that is mirrored thus making it impossible for other filters on the same IFL to monitor or see traffic Because the ADC software uses filters to catch traffic when the ADC software works with the IFL the port mirroring filters are applied only to the output side of the IFL You should fully understand applying port mirroring to the output side of an IFL the mirrored traffic does not include the requests from this IFL Rather the mirrored traffic only includes the responses from the IFL usually after being treated in the ADC software Document ID
38. least half DPs cross this threshold 1 second 4seconds 64seconds 10 13 35 35 47 47 41 41 42 42 39 39 41 41 41 42 50 50 36 37 39 39 36 36 48 48 39 39 45 45 38 38 42 42 31 31 34 34 39 39 45 45 41 41 3 4 4 4 3 4 Homo au NS H WU WW UT arn oO NO w A J 5 W H k H H Fi H H FA H H H WO WN HUF WN H Ww W H Lo AJ On UI amp DA M Ho gt N N N 30 Document ID RDWR RSLB V1 4R0 0_TSG0711 ve fi ii gr adw are ADC Software Troubleshooung GUDE Initial Troubleshooting DP Maintenance Statistics The maintenance command displays DP maintenance statistics per DP ei To display DP maintenance statistics user host gt show extensions adc internal maintenance ms 1 0 0 dp 12 Maintenance statistics for Interface ms 1 0 0 dp 12 Success from RCV Errors from SND Success to SND failures to 2414337 1354977 2502375 2415755 1275781 2504818 2381364 1655359 2207492 2350546 0 2200316 2347841 1300620 2248642 1981460 1674814 2030104 1896073 1639463 0 1660339 1167307 985151 1040409 1102901 1304601 2028115 982410 1094261 802970 0 1532456 1404239 742807 1701184 660126 478408 1119825 1112334 770817 0 CO OO OO OO OO OO OO OO OO OO OO c CO OO OO ee OO OO OO OO OO OO OO CH learn err noddw resolve err noddw age mp noddw delete miss pfdb free empty tcp discards udp discards Dynamic Memory Statistics Total memory
39. ll filter is dynamic since it only catches from container traffic with destination IP as VIP and only when the VIP is marked as up in the ADC software ei To view the firewall filter that is added on the client facing interfaces user host gt show extensions adc internal generated configuration Front End firewall Filter name rdwr fe filt Related interfaces ge 0 2 0 0 Adc instance 1b1 Active VIPs 150 0 0 120 routes added Non active VIPS None Routing instance rdwr lbl fe ri Packet counter 0 Total Bytes 0 The ADC software also adds routes to the routing instances on the device These routes can be displayed as well as all other routes using the appropriate command ei To view the routes on the device userehost gt show route all The debug enable Command The debug enable command is a hidden debug command under the ADC hierarchy When the debug enable command is set the ADC software does two additional activities for debugging purposes e The ADC software sends more syslog messages with debug information on the internal ADC software processes e The ADC software configures counters per firewall filter term to count the traffic that is caught by each term This configuration is performed both for the commit script filters and for the Junos OS SDK API filters As a result of the above additional activities the debug enable command results in a performance penalty and reduces the capacity of the Multiserv
40. lustrates how to correctly set up port mirroring for your device ei To correctly set up your port mirroring configuration 1 Define the port mirroring parameters forwarding options port mirroring input rate 1 copy all packets run length 0 copy all packets family inet output interface ge 0 0 0 0 logical interface the packets go out of next hop 192 168 1 233 next hop for the packets 18 Document ID RDWR RSLB V1 4R0 0_TSG0711 3 AD fi Troubleshooti j H radware C Software Den eshooting Guide Troubleshooting Fundamentals 2 Define the port mirroring filter firewall filter mirroring filter term terml then port mirror accept 3 Define the filter on the output side of an IFL assuming this IFL is configured in the ADC configuration as a server or client facing interface interfaces fe 1 3 0 unit 0 family inet filter output mirroring filter Interface Statistics The interface statistics command lets you display information on received and transmitted packets at the ports The command can be used on the client and server facing interfaces Another option is to use the interface statistics command on the Network Processing Unit NPU in the Multiservices DPC Like any other interface each NPU interface in the Multiservices DPC also has statistics However when viewing statistics of the NPU interfaces you should be aware of the direction of thes
41. nd ei To capture CP CPU information user host gt show extensions adc cpu CPU Utilization for interface ms 1 0 0 CP average for last 64 seconds 15 DP average for last 64 seconds 40 max 50 CPU Utilization for interface ms 2 0 0 CP average for last 64 seconds 2 DP average for last 64 seconds 41 max 60 CPU Utilization for interface ms 2 1 0 CP average for last 64 seconds 2 DP average for last 64 seconds 44 max 60 Switch Processor DP CPU is the main data path processor for the ADC software There are 21 DPs per Multiservices DPC NPU This section explains how to view DP statistics and trace the reasons for high DP CPU use High DP CPU use is usually due to high traffic volume or a denial of service DoS attack to client facing interfaces To verify that check the interface statistics This section includes the following topics e DP CPU Statistics on page 30 e DP Maintenance Statistics on page 31 Document ID RDWR RSLB V1 4R0 0 TSG0711 29 H i WI ADG Software Troubleshooting Guide sr adw are Initial Troubleshooting DP CPU Statistics You can display statistics for all DP CPUs in a certain Multiservices DPC NPU using the cpu command ei To display DP CPU statistics userehost gt show extensions adc cpu ms 1 0 0 CPU Utilization for interface ms 1 0 0 DP average for last 64 seconds DP max for last 64 seconds DP min for last 64 seconds cpu threshold 85 Alert is triggered when at
42. ni nos clients dans le cadre de l installation et de l usage des produits de Radware d crits dans ce document et ne pourra tre utilis dans un but autre que celui pour lequel il a t con u Les informations r pertori es dans ce document restent la propri t de Radware et doivent tre conserv es de mani re confidentielle Il est strictement interdit de copier reproduire ou divulguer des informations contenues dans ce manuel sans avoir obtenu le consentement pr alable crit de Radware Wichtige Anmerkung Dieses Handbuch wird vorbehaltlich folgender Bedingungen und Einschr nkungen ausgeliefert Copyright Radware Ltd 2006 2011 Alle Rechte vorbehalten Das Urheberrecht und alle anderen in diesem Handbuch enthaltenen Eigentumsrechte und Gesch ftsgeheimnisse sind Eigentum von Radware Ltd Dieses Handbuch wird Kunden von Radware mit dem ausschlie lichen Zweck ausgeh ndigt Informationen zu Montage und Benutzung der in diesem Dokument beschriebene Produkte von Radware bereitzustellen Es darf f r keinen anderen Zweck verwendet werden Die in diesem Handbuch enthaltenen Informationen sind Eigentum von Radware und m ssen streng vertraulich behandelt werden Es ist streng verboten dieses Handbuch oder Teile daraus ohne vorherige schriftliche Zustimmung von Radware zu kopieren vervielf ltigen reproduzieren oder offen zu legen Document ID RDWR RSLB V1 4R0 0_TSG0711 3 ij i WI ADC Software Troubleshooting Guide sr adw
43. ns adc internal generated configuration N Document ID RDWR RSLB V1 4R0 0_TSG0711 3 AD fi Troubleshooti j ee C Software Troub es ooting Guide fra dw are Extended Troubleshooting Layer 7 Verify Layer 7 to diagnose issues in Secure Sockets Layer ID SSL ID URL parsing scripted and content based health check and Domain Name System DNS load balancing e Check SSL ID persistency Sniff the connection and check the server logs for the SSL IDs Correlate this information For example if a client connects to the device check the initial request to the real server coming on a 0 valued Session ID The server issues an SSL ID back to the client As long as the client uses the same SSL ID for incoming connections the session is bound to the same real server If there is a change in SSL ID due to a client or server issue it is easily captured in the sniffer traces e Check URL parsing Configure content match strings Map the requests and verify them using a sniffer to ensure that the correct content is specified e Perform script and content based health checks Check if there is any configuration error in the script or content Verify the script using a telnet client to ensure that you are getting a 200 OK or the expected response in the script back from the server e Check DNS based load balancing Verify the string associated with the DNS name specified e Check URL hashing and header hash load balancing Troubleshooting
44. ollowing procedure to troubleshoot NAT addresses in the ADC software ei To troubleshoot NAT addresses for the ADC software 1 Check that the client nat parameter is enabled for the servers user host gt show configuration extensions adc adc instance lbl real servers ri address 200 client nat address 200 client nat Note The client nat parameter must be enabled on the servers 2 For filters with action load balancing client nat should be configured per each filter user host gt show configuration extensions adc adc instance lt adc name gt filters term lt name gt then load balance group gl client nat Document ID RDWR RSLB V1 4R0 0 TSG0711 45 j 1 WI ADC software Troubleshooting Guide sr adw are Feature Specific Troubleshooting 3 Check the number of NAT addresses configured on all Multiservices DPC CPUs At least one NAT address must be configured per NPU user host gt show configuration extensions adc adc instance lbl router interfaces ms interfaces ms 1 0 0 unit 0 family inet nat address 1 1 1 1 ms 1 0 0 unit 0 family inet nat address range 1 1 1 2 to 1 1 1 5 Note The configured NAT address cannot be the same as to any other IP address defined in the configuration such as a real server address virtual server address and so on 4 Check NAT statistics High Availability The ADC software uses the Junos OS based high availability mechanism called
45. oubleshooting Guide gt KM r adw are Emergency Recovery Tree Figure 1 VIP Not Working Recovery Diagram 1 54 Document ID RDWR RSLB V1 4R0 0_TSG0711 D r adw are ADC Software Troubleshooting Guide Emergency Recovery Tree Figure 2 VIP Not Working Recovery Diagram 2 Handle port down situation not an ADC issue Real server is reachable using ping Handle next hop down not an ADC issue VIP working Document ID RDWR RSLB V1 4R0 0 TSG0711 55 ADC Software Troubleshooting Guide D r adw are Emergency Recovery Tree Figure 3 VIP Not Working Recovery Diagram 3 56 Document ID RDWR RSLB V1 4R0 0_TSG0711 ADC Software Troubleshooting Guide Emergency Recovery Tree radware Figure 4 VIP Not Working Recovery Diagram 4 the traffic from real server to the clients ingresses the router on the correct interface VIP working Check if the virtual service port is the same port as used by the real server to listen to connections telnet lt server address gt port lt port num gt Configure either server listening port under virtual service or listening port under real server to match the right port VIP working number in the second case server listening port under virtual service should 0 Document ID RDWR RSLB V1 4R0 0 TSG0711 57 ADC Software Troubleshooting Guide 2 r adw are Emergency Recovery Tree ADC Software Panic Figure 5 on page
46. port This involves gathering information assessing the situation and collecting certain system statistics that will be helpful to technical support personnel This chapter includes the following topics e Gathering Information on page 23 e Assessing Panic on page 28 e Collecting System Statistics on page 28 Gathering Information Before contacting technical support gather information that can help support personnel when troubleshooting This includes the following information e ADC software status e Software version e Saving technical support dump e System capacity e System configuration details e Session entry e System log e Topology of the network e A list of any changes made prior to the issue such as software hardware and upgrades This section includes the following topics e Viewing Chassis Information on page 23 e Viewing General System Information on page 24 e Viewing the Software Version on page 26 e Saving Technical Support Information on page 27 e Viewing System Maintenance Information on page 27 e Providing a Network Topology Map on page 28 e Documenting System and Network Changes on page 28 Viewing Chassis Information The first thing you will need to gather is the chassis information which chassis is used what are the PICs inside it and so on Document ID RDWR RSLB V1 4R0 0 TSG0711 23 ADC Software Troubleshooting Guide Initial Troubleshooting Use the hardw
47. rnnnnnnnnnnnennnnnnnnnnennnnnnnnnnnnnnnnnnnnnnnnnen 53 VIPIS NO Working ee 53 ADC Software Panic een dre en dan eee nt etienne 58 Real Server Dawn nee ee 59 MAG OX AP 63 12 Document ID RDWR RSLB V1 4R0 0_TSG0711 radware Table of Figures VIP Not Working Recovery Diagram 1 an en a 54 VIP Not Working Recovery Diagram 2 en eeeeieg 55 VIP Not Working Recovery Diagramme nettes 56 VIP Not Working Recovery Diagram E 57 Switch Panic Recovery Diagram una a 58 Real Server Down Recovery Diagram 1 aus 59 Real Server Down Recovery Diagram 2 une 60 Real Server Down Recovery Diagram 3 1 2 na 61 Real server Down Recovery Diagram 4 anne 62 Document ID RDWR RSLB V1 4R0 0_TSG0711 13 H i ve ADC Software Troubleshooting Guide sr adw are Table of Figures 14 Document ID RDWR RSLB V1 4R0 0_TSG0711 3 fi Troubleshooti j er adw are ADC Software Troubleshooting Guide Introduction Chapter 1 Introduction Juniper Networks Application Delivery Controller ADC for the MX Series 3D Universal Edge Router offers advanced router integrated ADC functions that enables service providers and enterprises to efficiently scale service capacity and increase service performance Routers are already ubiquitously deployed throughout the network at the network edge in the network core and in the data center Integrating the advanced ADC with the carrier grade MX3D router promotes network consolidation and reduces the number of network elements that pro
48. servers are passing health checks If the health check fails then do the following Confirm that the appropriate gateway is on the real server The gateway needs to be the device interface or next hop toward the device 36 Document ID RDWR RSLB V1 4R0 0_TSG0711 3 AD fi Troubleshooti j radware C Software Troubleshooting Sun Extended Troubleshooting Confirm that health check replies are forwarded by the appropriate interface for a multihomed real server Confirm that there are no routing loops between the real server and the router Use the following commands to diagnose issues with Layer 4 if To verify correct interfaces are configured as client and server interfaces Ste user host gt show configuration extensions adc adc instance lt name gt router interfaces if To verify real server and failed or blocked services per server name user host gt show extensions adc real server lt server name gt if To verify group status and traffic distribution Ste user host gt show extensions adc group group name if To verify the virtual server status Ste userehost gt show extensions adc virtual server virtual server name if To verify the virtual service status and statistics Ste user host gt show extensions adc lt protocol gt virtual server lt virtual server name gt if To verify the virtual server routes and automatic configuration user host gt show extensio
49. tatus 39 trace file 19 accessing 22 configuring 21 viewing 21 troubleshooting ADC automatic configuration 41 commit script automatic configuration 41 connection synchronization 49 connection table 39 debug enable flag 43 feature specific 41 high availability 46 license information 50 multiservices dpc npu 34 NAT IP address 45 OSI layers 34 SDK API automatic configuration 43 tools 38 troubleshooting fundamentals 17 interface statistics 19 port mirroring 17 system log 19 trace file 19 V viewing chassis information 23 viewing the syslog file 20 viewing the trace file 21 VIP not working 53 64
50. terconnection OSI model to systematically identify and rule out problems at Layers 1 through 4 and Layer 7 This section includes the following topics e Layer 1 and Layer 2 on page 34 e Layer 3 on page 35 e Layer 4 on page 36 e Layer 7 on page 38 Layer 1 and Layer 2 Use the following commands to diagnose the Layer 1 and Layer 2 issues 34 Document ID RDWR RSLB V1 4R0 0_TSG0711 re p radwa re ADC Software Troubleshooting Guide Extended Troubleshooting To verify which interfaces exist in the device and what their status is Ste user host gt show interface terse if To verify packet statistics on a specific interface ho user host gt show interface lt interface name gt if To verify errors and extensive statistics on a specific interface Ste userehost gt show interfaces extensive lt interface name gt if To verify spanning tree issues mo user host gt show spanning tree if To verify link aggregation and LACP issues Ste user host gt show lacp Layer 3 Check the following to diagnose issues in Layer 3 e The interfaces are up and assigned to the correct VLAN Local networks are defined for any route or interface Check if they are defined for all interfaces and each static route Otherwise all non local net defined traffic is sent to the default gateway The routing table uses the correct next hops and interfaces Also ensure that local networks are def
51. the following procedure to troubleshoot issues relating to connection synchronization ei To troubleshoot connection synchronization for the ADC software 1 When using virtual server load balancing verify the connection synchronization is configured for your virtual service user host show extensions adc adc instance lt adc name gt virtual server lt name gt http virtual service lt name gt sync connections 2 When using filter load balancing verify the sync connections parameter is configured for your filter user host show extensions adc adc instance lt adc name gt filters term lt name gt then load balance sync connections 3 Verify connection synchronization is not disabled for the entire ADC instance Do this by ensuring the no connections sync parameter is not set in the adc instance Caution If the no connections sync parameter is used at the adc instance level the sync connections parameter at any other level of the hierarchy will not function user host show extensions adc adc instance lt name gt no connections sync 4 Use the hidden passive switch to find the connection table that is synchronized When using this switch with the connection table the ADC software takes the connection table information from the backup Multiservices DPC NPU inside the RMS This lets you view the synchronized data user host gt show extensions adc connection table extensive passive adc
52. un logiciel d velopp par Wesley Griffin Ce produit inclut un logiciel d velopp par Per Allansson Ce produit inclut un logiciel d velopp par Nils Nordman Ce produit inclut un logiciel d velopp par Simon Wilkinson La distribution et l usage sous une forme source et binaire avec ou sans modifications est autoris e pour autant que les conditions suivantes soient remplies 1 La distribution d un code source doit inclure la notice de copyright mentionn e ci dessus cette liste de conditions et l avis de non responsabilit suivant 2 La distribution sous une forme binaire doit reproduire dans la documentation et ou dans tout autre mat riel fourni la notice de copyright mentionn e ci dessus cette liste de conditions et l avis de non responsabilit suivant LE LOGICIEL MENTIONN CI DESSUS EST FOURNI TEL QUEL PAR LE D VELOPPEUR ET TOUTE GARANTIE EXPLICITE OU IMPLICITE Y COMPRIS MAIS SANS S Y LIMITER TOUTE GARANTI E IMPLICITE DE QUALITE MARCHANDE ET D ADEQUATION A UN USAGE PARTICULIER EST EXCLUE EN AUCUN CAS L AUTEUR NE POURRA TRE TENU RESPONSABLE DES DOMMAGES DIRECTS INDIRECTS ACCESSOIRES SP CIAUX EXEMPLAIRES OU CONS CUTIFS Y COMPRIS MAIS SANS S Y LIMITER L ACQUISITION DE BIENS OU DE SERVICES DE REMPLACEMENT LA PERTE D USAGE DE DONN ES OU DE PROFITS OU L INTERRUPTION DES AFFAIRES QUELLE QU EN SOIT LA CAUSE ET LA TH ORIE DE RESPONSABILIT QU IL S AGISSE D UN CONTRAT DE RESPONSABILIT STRICTE OU D UN AC
53. viders must rack power cool maintain and upgrade Furthermore the ADC software which is optionally licensed improves service resiliency by monitoring server and application health and by automatically bypassing failures This guide describes the diagnostic tools available for the ADC software using the command line interface CLI For more information on learning the basic structure and operation of the CLI see the ADC Software Reference Guide e e For more information on troubleshooting the J uniper Networks router see the J unos OS Baseline Operations Guide available at the following link https www juniper net techpubs en_US junos10 4 information products topic collections nog baseline index html Prerequisites This guide is intended for network administrators with the following background Basic knowledge of networks Ethernet bridging and IP routing Familiarity with networking concepts and terminology e Basic knowledge of network topologies e Basic knowledge of J unos OS Acronyms Table 1 on page 15 shows the acronyms used in this guide Table 1 Acronyms Description CLI command line interface CPU central processing unit DNS Domain Name System FTP File Transfer Protocol Lightweight Directory Access Protocol MAC media access control NAT Network Address Translation Open Systems Interconnection Simple Network Management Protocol Document ID RDWR RSLB V1 4R0 0 TSG0711 15
54. wa re ADC Software Troubleshooting Guide This product includes software developed by Daniel Kouril This product includes software developed by Wesley Griffin This product includes software developed by Per Allansson This product includes software developed by Nils Nordman This product includes software developed by Simon Wilkinson Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met 1 Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer 2 Redistributions in binary form must reproduce the above copyright notice this list of conditions and the following disclaimer in the documentation and or other materials provided with the distribution ALL THE SOFTWARE MENTIONED ABOVE IS PROVIDED BY THE AUTHOR AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT INDIRECT INCIDENTAL SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES INCLUDING BUT NOT LIMITED TO PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY OUT OF THE USE OF
55. ze 500000 world readable flag all Viewing the Trace File You can view the trace file using one of the following methods Note The trace filename dynamic txt is a sample filename used for these examples Your filename may be different For more information see Configuring the Trace File on page 21 Document ID RDWR RSLB V1 4R0 0 TSG0711 21 AD fi Troubleshooti j ee C So tere roubleshooting Guide er adw are Troubleshooting Fundamentals ei To view the trace file using the show command user host gt show log ext radware dynamic txt ei To set a separate connection that constantly shows the new messages in the trace file user host gt monitor start ext radware dynamic txt Accessing the Trace File The trace file itself is located in var log ext radware You can access the file using FTP to connect to the device and then navigating to var log ext radware Old files are compressed zipped in the same location using the filename dynamic txt gz Note The trace filename dynamic txt is a sample filename used for these examples Your filename may be different For more information see Configuring the Trace File on page 21 22 Document ID RDWR RSLB V1 4R0 0_TSG0711 33 AD fi Troubleshooti j sr adw are C Software roub eshooting GUDE Initial Troubleshooting Chapter 3 Initial Troubleshooting This chapter describes the actions you should take before calling J uniper Networks Technical Sup

Download Pdf Manuals

image

Related Search

Related Contents

HEPATITIS B – HBsAg  HD-SDI Water-proof IR Camera - Hi-view  Stratos®Pro A2... PH Mode d`emploi  International Travel Adapter  minidrop  Easy-Master - Support  SUB U-COM  USER`S MANUAL/PARTS LIST  Mi 96 HD USER MANUAL  

Copyright © All rights reserved.
Failed to retrieve file