Home

NightTrace Manual

1. Sai to Is the text Use a display text constant GridLabel or graphics Use a Graphics sal DataBox Create a l re ES Use a Column Ruler Ruler Do you want to graph states or events Yes Do you want to graph argument or expression values Use a StateGraph or Event Graph Use a DataGraph 7 11 NightTrace Manual GridLabel DataBox 7 12 Figure 7 6 Display Object Use Flowchart fe ed p Tra Sih hg Gal Available For aves 2 Quick Loading Samples From Distribution 6 Figure 7 7 GridLabel Examples Clicking on Create I GridLabel lets you draw or create a GridLabel display object on the grid A GridLabel is a rectangle that contains a string of text This text usually is a title or description of an adjacent display object on the grid and makes the display page easier to interpret GridLabels can appear anywhere on the grid but they cannot go inside a Column You can put several GridLabels on a grid If the text is too long to fit into the GridLabel the lower right corner of the box is filled in If this occurs you should resize the GridLabel This is described in Resizing Display Objects on page 7 7 A newly created label contains the word label See GridLabel on page 8 12 for more information GridLabels are static display objects That is a GridLabel does
2. SYNTAX C int trace flush int trace trigger Fortran integer function trace flush integer function trace trigger Ada procedures procedure trace flush procedure trace trigger functions function trace flush return ntrace error function trace trigger return ntrace error DESCRIPTION When the user daemon is idle it sleeps The process of copying trace events from the shared memory buffer to a trace event file is called flushing the buffer The user daemon wakes up and flushes the buffer when any of these conditions exist the user daemon s sleep interval elapses The buffer full cutoff percentage is exceeded The shared memory buffer is full of unwritten trace events e Your application calls trace flush trace trigger or trace end No event has been logged in a period of time in which the lower 32 bits of the timestamp source would roll over It is important to detect this rollover so that proper ordering of trace events is maintained User daemon options let you set limits for the first three conditions above When you invoke a user daemon with one of these options and it detects the corresponding condition it automatically flushes the buffer There is one key way that trace flush and trace trigger differ from the flush control the user daemon provides with trace flush and trace trigger you decide when to asynchronously flush the shared memory buffer based on your program
3. User Events Event Start o Event Count 8 Event End 8 Zoom Factor 2 0 Increment 25 002 Current Time 1 2065472s Apply Reset Center Mark Zoom Region Zoom in Zoom out Refresh Time Start 0 0000000s Time Length 4 8261836s Time End 4 8261896s pa Figure 7 1 Display Page with Display Objects Figure 7 1 shows what a display page may look like when you invoke ntrace and specify the default display page The default display page contains display objects See Default Page on page 5 28 Display objects filter process and display the information in the trace event file These display objects are created with the display page and then viewed on the display page You may want to create your own set of display objects to view your trace event file To do this follow the steps below 1 Read The Display Page on page 7 2 to learn about the various parts of a display page 2 Read The Display Page on page 7 2 which describes the different modes a display page can be in Edit and View 7 1 NightTrace Manual 3 Put the display in Edit mode 4 Read Display Objects on page 7 8 which explains what a display object is and what the different types of display objects you can put on your display page are 5 Read Operations on Display Objects on page 7 4 which explains how to perform various operations creating selecting moving and resizing on display objects 6
4. 000 cece eee eens 9 32 event LADO cack ewes cee SI AS nee Na 9 32 event matches cios a 9 33 State Functions 200 eru A aera 9 34 Start Functions ss coo e bee bes 9 34 start dd yrs ares rte XI ERR RAN OR NUR 9 35 statt arg a coL o eee eek en eck aae INR 9 35 11 NightTrace Manual 12 start are dbl ce Deum Sara wee eee RES 9 36 Start num args eee RP ER nERAR P BER RAL 9 37 start pidii a an tede duce et ds 9 37 Start Taw Pld uos eer Cere eb TO e e SIS Re des 9 38 start Iwpld 51s iR LUE IER RE 9 39 start thread ad iiu tige e DER E Penes 9 39 start task Idse ea e e egg et E RAE AVES 9 40 statt tid ccce a RD RUDI UG ERGO IN RUE 9 41 SIaEESCDU Soo Ad ASE i eth tk Bd eor 9 41 start Offset rial aia 9 42 start time ceeds dene Pade oe d corr ees 9 42 start hode AO nec ERU DRAIN S SUE ues 9 43 start pid table name 0 0c eee eee ee eee 9 44 start_tid_table_name 0 0 0 0 ccc ccc ee eee eee 9 44 start_node_name 0 ccc cee eee ee eens 9 45 End Puntos cas it a Rees ER S MAECEN 9 45 end 1d iR O Ee a UE 9 46 end satg Oi cs has ans Dok e eni ed ep ee 9 47 end arg dbl eurer dene nda aeta 9 47 end num args sueco ee Rie Rees OH POUR ers 9 48 end pid o 645 Wa ee uu hubo bere eee bI 9 48 end raw pid ir seb REID a a a e apes 9 49 O O eed RES en arse e 9 50 end thread id sor ii stn keds eb dere ROS CEN AA 9 51 end task idi ile ERREUR EXER 9 51 end tid d
5. Permission to lock the text and data pages of the NightTrace library routines was denied If the user is not privileged to lock pages see your system administrator or set ntc_lock_pages to FALSI Gl This can occur if the size of the shared memory buffer exceeds the system limit SHMMAX or the shared memory buffer already exists but the size required by ntc_buffer_size which is roughly ntc_buffer_size sizeof ntevent_t exceeds the current size Related routines include trace_open_thread trace_end trace_open_thread Adding Library Calls to Your Application The trace open thread routine prepares the current process C thread or Ada task for trace event logging SYNTAX C Fortran int trace_open_thread char thread_name integer function trace_open_thread thread_name character thread name Ada function trace open thread l thread_name string return ntrace_error PARAMETERS thread_name In NightTrace every thread of execution to be traced whether a sepa rate process or a C thread or Ada task within a process must be associated with a name thread_name which may be null Night Trace s graphical displays and textual summary information show which threads logged trace events If the trace open thread thread name is null the ntrace display utility uses the global thread identifier TID as a label in these displays For more information on global thread identifier
6. 0 0 0 0 cece eee ee eee 2 10 Table 11 1 Example Logical CPU Mapping lessen 2 10 Table 11 2 Non Device Related Interrupt Reference 2 19 Table 11 3 Device Related Interrupt Reference 0 005 2 20 Table 11 4 Exception Reference 20 0 cece cece ee 2 21 Table B 1 Meanings of Common Subobjects and Attributes B 2 Table B 2 Suggested Colors for X Resources 00 0 0 eese B 5 Glossary Index 17 NightTrace Manual 18 Overview What is NightTrace 1 Introduction User and Kernel Tracing 0 0 0 eee cee eee Timestamp Source Selection 0 0 0 cece eee eee Trace Point Placement ehec ienei cece eee een eens Languages Supported 0 0 0 cece eee eee Processes and CPUS eee bans tat Ane dato oa ears e lice Information Displayed 0 0 cece eh Searches and Summaries 0 0 cece cece eee eee eee Logging and Analysis The User Trace Event Logging Procedure 00 0 00 0 ree The Kernel Trace Event Logging Procedure 00 0000 ee eee ee The Trace Event Analysis Procedure 0 0 0 0 cece cence eee eee Recommended Reading 1 1 1 1 1 2 1 2 1 2 1 3 1 3 1 3 1 3 1 3 1 4 1 5 1 6 1 7 NightTrace Manual Overview 1 Introduction This chapter provides an overview of NightTrace steps involved in using the toolset and recommended readings
7. 0 0 ccc cece o 10 2 Search Form Push Buttons 0 0 0 ccc cee ene teenies 10 3 Search Form Fields eo sch ck otk e URS 10 4 Summarizing Statistical Information 0 00 0 c eee eee eee ee 10 5 Summarize Form Radio Buttons 0 ccc eee nee 10 6 Summarize Form Fields o ooooooooooo ooo 10 6 Summarize Form Push Buttons 0 0 0 000 eee eens 10 8 Menu Bat ui A ED mS AERE 10 9 Fil Operations ere id Samos Be 10 9 Save Text eos dece eh ee whee ea eae ae DEC OR ER 10 10 Save Text AS s lebe duae eid daw X PER UIS 10 10 eh CMM 10 10 Summary Display Area 0 cee eh 10 10 Event SUMMANES AA A NE ee OR ENIM 10 11 State Summarles csi A eho BY EE US ERES ERIT 10 12 Exercise Using the Search Tool 0 0 0 eee eee 10 14 Exercise Using the Summarize Tool 0 00 0 c ce eee eee 10 15 NightTrace Manual 10 Using the Built In Tools Overview ntrace comes with a set of built in tools available in View mode These tools make it easier for you to pinpoint important trace events and numerically analyze aspects of your trace session This chapter covers the following built in tools Search Locates interesting parts of your trace session Summarize Summarizes statistics about trace events or states Figure 10 1 shows the display page menu that gives you access to these tools Figure 10 1 Tools Menu Searching for Points of Interest Clicking on Tools I S
8. NOTE The Daemon Name is merely a label to aid the user in identify ing specific daemons with a session It has no external meaning and is unrelated to the NightTrace API Target The name of the system on which the associated daemon is running The target system is specified in the Target System field on the General page of the Daemon Definition dialog see Target System on page 3 29 3 18 State The state of the daemon Attached Logging Not Executing Paused Pausing Resuming Starting Stopping NightTrace Session Manager indicates the daemon is currently capturing events indicates the daemon has not been started indicates the daemon is started but is not capturing events While paused attempts to log events from user applica tions or via the operating system kernel are discarded Note that these are not considered lost events see Lost on page 3 20 indicates the daemon is going from a Logging state to a Paused state indicates the daemon is going from a Paused state to a Logging state indicates the daemon is going from a Not Executing state to a Logging state indicates the daemon is going from a Paused or Log ging state to a Not Executing state The number of user application threads or processes that are associated with the dae mon Streaming Indicates whether or not data from this daemon is being streamed to the NightTrace display buffer This is specified by the setting
9. See Policies in the section below for a list of applicable NUMA policies Policies Each of the above flags can be set to one of the following Global Specifies that the designated pages should be placed in global memory Soft Local Specifies that the designated pages be placed in local memory if space is available otherwise they should be placed in global memory 3 44 NightTrace Session Manager Hard Local Specifies that the designated pages must be placed in local memory Default Specifies that the default NUMA policy on the target system should be used Streaming The Streaming page of the Daemon Definition dialog see Daemon Definition Dia log on page 3 25 allows the user to specify advanced settings with respect to the transfer of trace data from the daemon to the NightTrace display buffer Daemon Definition 1048576 14096 Figure 3 25 Daemon Definition dialog Streaming Streaming Options 3 45 NightTrace Manual Stream Buffer Size The number of bytes for the buffer that the NightTrace Session Manager uses to hold data from the daemon before sending it to the NightTrace display buffer NOTE This is an internal buffer You should not need to adjust the size of this buffer unless NightTrace finds that it cannot transfer data quickly enough between the daemon and the NightTrace analyzer In such a circumstance the daemon is forced into a Paused state see State on page
10. Therefore if you have made any changes to the summary display area that you want to keep you must perform a Save Text or Save Text As before you do a Close Summary Display Area 10 10 After you click on Summarize ntrace appends statistics to the end of the scrolling summary display area It automatically scrolls this area so the newest statistics are visible Every line in this area has a unique number A blank line separates sets of statistics If you want new statistics to appear alone in the summary display area click on Clear before you click on Summarize See Summarize Form Push Buttons on page 10 8 for more information By default ntrace displays 14 lines in the summary display area You can alter this number by changing the size of the Summarize Form To change the Summarize Using the Built In Tools Form size vertically resize your window by using features of your window manager It is not necessary to resize a window to see lines 15 and higher you can scroll through all lines by using the scroll bar Sometimes the statistical information exceeds the width of the summary display area In this case you must horizontally resize your window The summary display area of the Summarize Form presents different information depending on your Summary Expression and whether your summary type is Event or State Event Summaries Configuring event summaries is similar to configuring DataBox display objects The configurati
11. start cpu start_id start_arg Using Expressions start offset start time e start node id start pid table name start tid table name start node name DESCRIPTION The start id function returns the trace event ID of the start event of the most recent instance of a state SYNTAX start id QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO idQ on page 9 19 end id on page 9 46 and offset id on page 9 60 DESCRIPTION The start arg function returns the value of a particular trace event argument associated with the start event of the most recent instance of a state SYNTAX start_arg N QS 9 35 NightTrace Manual PARAMETERS N OS RETURN TYPE integer SEE ALSO Specifies the Mth argument logged with the start event Defaults to 1 A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 argO on page 9 20 start arg dbl on page 9 36 start num args on pag
12. 2 20 Adding Library Calls to Your Application flow and with certain options the user daemon flushes the shared memory buffer automatically If the shared memory buffer becomes full of trace events trace events may be lost To keep this situation from occurring configure the user daemon to flush the buffer regularly This is particularly good to do if your application will soon be busy Waking the user daemon to flush the buffer takes time and this overhead can distort trace event timings Therefore call trace_flush and trace_trigger only in parts of your application where time is not critical TIP trace_trigger is identical to trace_flush except trace_trigger works only in buffer wraparound mode Call trace_trigger instead of trace_flush so that only buffer wrap around s performance is affected When you run in buffer wraparound mode you are telling NightTrace to intention ally discard older or less vital trace events when the shared memory buffer gets full In buffer wraparound mode you must explicitly call trace_flush or trace trigger Only then does the user daemon copy the remaining trace events from the shared memory buffer to the trace event file However do not call trace flush or trace_trigger too often or you will reduce the effec tiveness of this mode See Option to Establish Buffer Wraparound Mode buffer wrap on page 4 13 for more information on buffer wraparound mode RETURN V
13. Deselect All Delete Editing operations are enabled only when the display page is in Edit mode which is selected by clicking on the radio button labeled Edit in the upper left hand corner of the display page Select All selects every display object on the grid This is useful when you want to perform some operation on every display object on the grid for example moving or deleting every display object Deselect All deselects every selected display object on the grid Delete deletes the selected display object s File Operations Save 7 18 Figure 7 15 File Menu The file operations are accessed through the File operations menu shown in Figure 7 15 Save saves the current display page including all local macros qualified events and qualified states to the configuration file you opened Thus any changes you have made since the last Save operation will be saved You can continue editing or viewing the display after this operation The Save operation is disabled dimmed if this is a new Creating Display Objects display page or you have not made any changes since the last time the display page was saved Instead use Save As Save As Save As saves the current display page to a file other than the one you opened You can continue editing or viewing the display after this operation Save As uses a File Selection Dialog Box to prompt you for a file name See The File Selection Dialog Box
14. PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO tid on page 9 25 end tid on page 9 52 and offset tid on page 9 65 DESCRIPTION The start cpu function returns the logical CPU number associated with the start event of the most recent instance of a state CPUs are logically numbered start ing at 0 and monotonically increase thereafter NOTE This function is only valid when applied to events from Night Trace kernel trace event files SYNTAX start cpu QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that 9 41 NightTrace Manual start_offset start_time 9 42 state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO cpu on page 9 26 end_cpu on page 9 52 and offset_cpu on page 9 66 DESCRIPTION The start_offset function returns the ordinal number offset of the start event of the most recent instance of a state SYNTAX start offset 105 PARAMETERS QS A user defined qualified state If supplied it
15. The name of the user application containing trace_ calls on the remote system User The user who invoked the Program on the remote system NightTrace Manual General 3 28 Key File The filename which is used to calculate the shared memory segment identifier asso ciated with the logging of user trace events See Key File on page 3 31 for more information The following buttons appear at the bottom of the Import Daemon Definition dialog and have the specified meaning OK Imports daemon attributes into the current daemon definition from the user applica tion selected in the list Refresh Queries the specified target system for user applications making trace related calls Cancel This button closes the Import Daemon Definition dialog without importing any daemon attributes from any of the listed applications Help Brings up online help for this dialog The General page of the Daemon Definition dialog see Daemon Definition Dialog on page 3 25 contains information such as the name of the daemon configuration the tar get system on which the daemon will run the user s login on that system and settings specifying whether kernel or user application tracing will be performed Items related to trace events output such as the names of output and key files and settings such as whether or not streaming will be performed by this daemon are found on this page as well NightTrace Session Manager Figure 3 20 Daem
16. Trace event file size Indefinite filewrap bytes Trace events enabled for log All disable ID and ging enable D Page fault handling Page locking lockdisable Preemption control Modify IPL register ipldisable PowerMAX OS Rescheduling variables RedHawk Linux resched ntraceud Modes 4 4 NightTrace can operate in three modes expansive default file wraparound and buffer wraparound As the following two tables show these modes meet different needs and have different characteristics They differ mainly by their handling of the shared memory buffer and the trace event file on disk By default NightTrace operates in expansive mode NightTrace operates in file wrap around mode when you specify the filewrap option on the nt raceud invocation line The ntraceud bufferwrap option puts NightTrace in buffer wraparound mode See Option to Establish File Wraparound Mode filewrap on page 4 12 and Option to Establish Buffer Wraparound Mode bufferwrap on page 4 13 for more information on these options It is not possible to combine expansive mode with either file wraparound or buffer wrap around mode Although you can mix file wraparound and buffer wraparound modes it is not recommended Table 4 2 provides some guidelines to help you decide which mode to use Table 4 2 Mode Selection Guidelines Generating Trace Event Logs with ntraceud Constraint Expansive MODE File Wraparound Buffer Wrapar
17. Using the Data Recording Window in the NightProbe User s Guide 0890480 Timing Source By default an architecture specific clock is used to timestamp trace events On NightHawk 6000 Series machines the interval timer is used on Power Hawk and PowerStack systems it is the PowerPC Time Base Register on iHawk systems the Intel Time Stamp Counter is used NightTrace can also specify the Real Time Clock and Interrupt Module RCIM as a timestamp source see Timestamp Source Selection on page 1 2 for more infor mation This is most useful when concurrent traces running on multiple systems are desired Using the RCIM as a timing device allows NightTrace to present the user with a synchronized view of concurrent activities on those systems Default Specifies that the architecture specific clock will be used to timestamp trace events On NightHawk 6000 Series machines the interval timer is used on 3 30 NightTrace Session Manager Power Hawk and PowerStack systems it is the PowerPC Time Base Register on iHawk systems the Intel Time Stamp Counter is used RCIM Tick Specifies that the Real Time Clock and Interrupt Module RCIM tick clock will be used to timestamp trace events NOTE Use of this option requires that an RCIM board is installed and configured on the target system Trace Events Output Stream When checked this specifies that streaming is in effect so that the output trace events will go di
18. ntraceud memsize count cutoff percent trace file There are several times when you may want to use the following invocation Usually this invocation is appropriate if you are using trace_flush calls to debug a fault in your application or to reduce the number of logged trace events so the nt race display is more readable ntraceud bufferwrap trace file The following invocation is also useful on several occasions One example is if you want to conserve disk space ntraceud filewrap bytes trace file The following invocation waits for all user applications associated with the running ntraceud daemon to terminate flushes remaining trace events to the trace event file closes the file removes the shared memory buffer then terminates the running ntra ceud ntraceud quit trace_file Similarly the following invocation immediately flushes remaning trace events to the trace file closes the file and terminates the running ntraceud daemon User applications can continue to run and make NightTrace API calls but no trace events will be logged Subse quently a new user daemon can be initiated and trace events will start beging logged again Generating Trace Event Logs with ntraceud ntraceud quit trace_file At this point you can begin data analysis Exercise Logging Trace Events The following exercise has you log trace events It is a continuation of Exercise Instru menting Code on page 2 25 1 Compile and l
19. 2 6 The Interval Scroll Bar 0 0 0 ee eee 2 7 The Interval Push Buttons 0 0 0 eee eee eee 2 8 The Interval Control Area 0 0 0 ee eee 2 11 Amount of Scrolling Due to Increment Value 2 14 Display Page with Display Objects 00 00 0000 2 1 Elements of a Display Page 0 cece ee eee ee 2 2 Edit and View Mode Buttons 0 00 00 eee eee 2 3 Button Functions on a Mouse 0 0 0 0 eee eee eee 2 5 Create Display Objects Menu 0 0 0 0 0 eee eee 2 8 Display Object Use Flowchart 0 0 0 0 00 eee ee eee 2 12 GridLabel Examples 0 00 errereen errer 2 12 DataBox Examples i 3 5 4 A AC RE end a a 2 12 Column Example 0 0 0 cece cee eee 2 13 StateGraph Example 0 0 00 cee eee eee ee 2 14 EventGraph Example 0 0 0 cee eee eee 2 15 DataGraph Examples 0 00 cece eee eee 2 16 Ruler Example uti ook HE eek RR AR hte techie 2 17 Edit Menu nseries ps 2 17 Ele Med o 2 18 Configure Command Menu eese eee ee 2 1 Left Center and Right Justified Text 000005 2 10 Top vs Bottom Gravity 0 0 0 ee eee 2 10 Configuration Form Push Buttons 0 0005 2 10 GridLabel Configuration Form 0 0 00 ee eee ee 2 12 DataBox Configuration Form 0 0 00 eee eee eee 2 13 StateGraph Configurati
20. Do you wish to save them Figure 3 5 Unsaved Changes dialog Yes When the user presses this button the current configuration is saved to the current session configuration file The name of the current session configuration file appears in the Session Configuration File Name Area see Session Configuration File Name Area on page 3 17 If the current configuration has not yet been saved the Save Session dialog see Save Session on page 3 6 will be presented to allow the user to navigate to the desired directory in which to save the session configuration file No When the user presses this button any unsaved changes are discarded and the termi nation of the session completes Cancel Cancels the termination action Help Provides online help for this dialog NightTrace Session Manager Daemons The Daemons menu provides functionality for configuring new and existing daemon definitions as well as attaching to and detaching from running daemons The Daemons menu appears on the NightTrace Session Manager menu bar see Night Trace Session Manager Menu Bar on page 3 2 Figure 3 6 Daemons Menu New Opens the Daemon Definition dialog see Daemon Definition Dialog on page 3 25 allowing the user to configure a new daemon definition Edit Opens the Daemon Definition dialog see Daemon Definition Dialog on page 3 25 for the daemon definition currently selected in the Daemon
21. Listed node IDs are listened to Acomma separated list of host names The name of a node host Spaces are not allowed Foreground The colors your X server supports as spec The color used by the display object to Color ified in the rgb txt file draw text and graphics in the foreground Background The colors your X server supports as spec The color in the background that any text Color ified in the rgb txt file and graphics are drawn over Font The fonts your X server supports or are The style of text characters that the display installed are in the directory object uses to display text usr lib X11 fonts Left Text is justified on the left side of the display object Text Justify Center Text is horizontally centered in the display object Right Text is justified on the right side of the display object Default Same as Left unless a different default is specified in an X resource 8 3 NightTrace Manual Table 8 1 Common Configuration Parameters Cont Parameter Name Possible Values Meaning Bottom Text sinks to the bottom of the display object Text Gravity Center Text is vertically centered in the display object Top Text floats to the top of the display object Default Same as Bottom unless a different default is specified in an X resource Display Object Name The Display Object Name is the field at the top of the configuration form This field is not titled in the configu
22. NOTE The Node List field appears in this dialog only when NightTrace is configured to use an RCIM to timestamp events The If Expression of an EventGraph determines whether a trace event should be graphed If the If Expression is true then a vertical line is drawn at the point in time that the trace event occurred 8 16 Configuring Display Objects DataGraph interrupt 000000 000000 QI o gt 4 TR_INTERRUPT_ENTRY TR_INTERRUPT_EXIT TRUE O D F Figure 8 9 DataGraph Configuration Form The configuration form for the DataGraph is shown in Figure 8 9 NOTE The Node List field appears in this dialog only when NightTrace is configured to use an RCIM to timestamp events The Fill Style parameter determines the style of DataGraph created The possible choices are None or Solid If None is chosen then a vertical line is drawn only at the time of a trace event If Solid is chosen then all space to the right of a trace event will be filled until the next trace event is encountered Figure 8 10 shows the difference between Solid and None 8 17 NightTrace Manual 8 18 Figure 8 10 Solid vs No Fill The Maximum parameter determines what data value corresponds to the top of the Data Graph The possible values are integers or CALC If an integer is specified as the maximum any data that is equal to or greater than that value results in a line or bar that goes to the top of the DataGraph If CALC is spe
23. SYNTAX offset_task_id offset_expr PARAMETERS offset_expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE integer SEE ALSO task id on page 9 25 start_task_id on page 9 40 and end task id on page 9 51 DESCRIPTION The of fset_tid function returns the internally assigned NightTrace thread identifier TID from which the ordinal trace event offset was logged SYNTAX offset tid offset expr PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE integer SEE ALSO tid on page 9 25 start tid on page 9 41 and end tid on page 9 52 9 65 NightTrace Manual offset_cpu DESCRIPTION The of fset_cpu function returns the logical CPU number on which the ordinal trace event offset occurred CPUs are logically numbered starting at 0 and mono tonically increase thereafter NOTE This function is only valid when applied to events from Night Trace kernel trace event files SYNTAX offset_cpu offset_expr PARAMETERS offset_expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE integer SEE ALSO cpu on page 9 26 start_cpu on page 9 41 and end_cpu on page 9 52 offset_time DESCRIPTION The offset time function returns the time in seconds between the
24. Specifies which clock to use as a timing source This value must be one of NT_USE_ARCHITECTURE_CLOCK or NT_USE_RCIM_TICK_CLOCK The user daemon default value is NT_USE_ARCHITECTURE_CLOCK ntc_shmid_perm Specifies the permissions to use when creating the shared memory segment The user daemon default value is 0666 ntc_daemon_preferred DESCRIPTION Specifies that if a user daemon already exists and the configuration settings differ from these configuration settings that the user daemon settings are preferred and these values are ignored although the value of ntc_buffer_size specified to this routine must not be larger than the size set by the daemon The trace_begin routine performs the following operations Verifies that the version of the NightTrace library linked with the application is compatible with the version used by the user daemon if it is already running Adding Library Calls to Your Application Verifies the supplied configuration settings are not in conflict with a pre existing daemon or defines the configuration with these settings if the user daemon does not yet exist Verifies that the RCIM synchronized tick clock is counting if it was selected as the timestamp source Attaches the shared memory buffer after creating it if needed nitialized the preemption control mechanism Locks critical NightTrace library routine pages in memory nitializes trace eve
25. event pid tid boolean name_pid name_tid node_name pid_nodename tid_nodename vector syscall device vector_nodename syscall_nodename device_nodename event_summary event_arg_summary event_arg_dbl_summary state_summary The results are undefined if you supply your own version of these tables For more information on pre defined tables see Pre Defined String Tables on page 5 14 Pre Defined Format Tables on page 5 20 and Kernel String Tables on page 11 16 TIP Put tables in separate configuration files from display pages This way tables do not get redefined if you close and reopen a display page during a single nt race session If you define a string table or format table more than once in a configuration file ntrace merges the two tables if there are duplicate entries values come from the last definition You can log a trace event with one or more numeric arguments Sometimes these arguments can take on a nearly fixed set of values A string table associates an integer value with a character string Labeling numeric values with text can make the values eas ier to interpret The syntax for a string table is string_table table_name item int const str const default item str const y You do not need to separate the parts tokens of the string table with white space String table tokens are indivisible although these tokens need not break on the lines shown they mu
26. function trace enable ID event type return ntrace error function trace enable ID low event type ID high event type return ntrace error function trace enable all return ntrace error function trace disable ID event type return ntrace error function trace disable ID low event type ID high event type return ntrace error function trace disable all return ntrace error Each trace event has a user defined trace event ID ID This ID is a valid integer in the range reserved for user trace event IDs 0 4095 inclusive See trace_event and Its Variants on page 2 10 for more information It is possible to manipulate groups of trace event IDs by specifying a range of trace event IDs D low is the smallest trace event ID in the range It is possible to manipulate groups of trace event IDs by specifying a range of trace event IDs D high is the largest trace event ID in the range The enable and disable library routines allow you to select which trace events are enabled and which are disabled for logging A discussion of disabling trace events appears first because initially all trace events are enabled Sometimes so many trace events that it is hard to understand the nt race display Occasionally you know that a particular trace event or trace event range is not inter esting at certain times but is interesting at others When either of these conditions exist it is useful to
27. g filewrap bytes bufferwrap memsize count timeout seconds cutoff percent clock source reset Fons ty stats disable ID ID enable ID ID trace file General options help Write this message to standard output version Write the current ntraceud version stamp to standard output Options for a new ntraceud daemon ipldisable Disable use of the IPL register lockdisable Disable use of page locking filewrap bytes Use file wraparound mode with max trace file size in bytes bufferwrap Use shared memory buffer wraparound mode memsize count Set shared memory buffer size to specified event count timeout seconds Set the ntraceud timeout to specified seconds cutoff percent Flush events to disk at specified cutoff level clock source Specify source of event time stamps Valid values for source are default Use the default system clock E camesEm ok Use the RCIM synchronized tick clock Options for an existing ntraceud daemon reset Reset the ntraceud daemon and the trace file quit Quit running ntraceud stats Write statistics resource environment to standard output Options for new and existing ntraceud daemons disable ID ID Disable a specific event ID or ID range from logging enable ID ID Enable a specific event ID or ID range to log Files trace_file Holds events logged by your application and ntraceud P Screen 4 1 Sample Output from the ntraceud help Option 4 7 Night
28. get item get format and summary functions appears later in this chapter NightTrace expressions are comprised of a combination of operators and operands A description of these operators and operands appears in the following sections Operators Operators in NightTrace expressions include e Arithmetic operators modulo unary Shift operators lt lt gt gt Bitwise operators not amp and exclusive or or Logical operators not amp amp and or Relational operators lt lt gt gt equivalence non equivalence 9 4 Operands Constants Conditional operator expr true value false value Using Expressions Unary casts to data types where the parentheses are required e g int NightTrace operators follow the operator precedence rules of the C programming lan guage Operand types are largely based on the C programming language and include integer double precision floating point character string boolean Operands include constants see Constants on page 9 5 macro calls see Macros on page 9 6 function calls see Functions on page 9 9 qualified events in functions only see Qualified Events on page 9 81 qualified states in functions only see Qualified States on page 9 83 Constants are one type of operand that may be used in NightTrace expressions
29. instrumented code interrupt interrupt graph The region of the display page filled with parallel rows and columns of dots that holds display objects A display object that displays constant textual information See graphical user interface A menu item that presents the online manual using the HyperHelp viewer The system on which the NightTrace GUI is running The small graphical image and or text label that represents a window or window family when the window is minimized The text label is either the window title or an abbreviated form of the title Iconified windows are still active See trace event ID Source code after you have put calls to NightTrace library routines into it An event external to the currently executing process an interrupt stops the current execution stream to begin execution of a higher priority execution stream There are device related and software generated interrupts Interrupts have an associated priority known as the interrupt priority level IPL which allows an interrupt to interrupt the execution stream of a lower IPL interrupt A DataGraph on a kernel display page It displays states representing interrupts executing on the associated CPU interrupt priority level IPL register Glossary 8 A system register than can be used by the NightTrace library to prevent reschedul ing and interrupts during trace event logging interval interval control area interval timer Kernel Tr
30. on page 5 30 for more information about this menu item Click on the Refresh push button on the display page Notice that the current trace event is now identified by its symbolic tag because that trace event has an entry in the map file This exercise continues in Exercise Using the Search Tool on page 10 14 For practice customizing the graphical user interface read Appendix B and try Exercise Customizing Display Colors on page B 5 6 Viewing Trace Event Logs with ntrace OVA be ree A Wa Ies Re 6 1 Mouse Button Operations 0 0 eee teens 6 3 The Grid e e O IC 6 4 VIEWING Strate sy uos Cetus is cacy IA A AA Br 6 5 The Interval Scroll Bar 2 0 0 ccc eee een eens 6 7 The Interval Push Buttons leeeeeeeeeee eee 6 8 The Interval Control Area 1 eee hn 6 11 Reading Fields orisni EXE AD ee ue NUR TRO ROV RIP A US 6 11 Editing Single Fields ocres e EE cee eee 6 12 Editing Multiple Fields llle 6 14 Field Editing eed eee ee teer RP cH es 6 16 Editing Text Fields occ eee tee dale op SRS Ste WEG EN PP EG 6 16 Positioning Within Text Fields llle eee 6 18 NightTrace Manual Overview 6 Viewing Trace Event Logs with ntrace ntrace s display page has two modes Edit mode and View mode The words Edit and View pertain to the operations you can perform on the graphical display not the text fields or scroll bar This chapter discusses View mode the mode that displays t
31. on page 9 50 and offset Iwpid on page 9 63 thread id DESCRIPTION The thread id function returns the thread identifier associated with a trace event NOTE See the thr self 3thread man page for more information SYNTAX thread id QEp PARAMETERS QE A user defined qualified event If supplied the function returns the thread identifier of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the thread identifier of the current trace event For more information see Qualified Events on page 9 81 RETURN TYPE integer 9 24 Using Expressions SEE ALSO start_thread_id on page 9 39 end_thread_id on page 9 51 and offset_thread_id on page 9 64 task_id DESCRIPTION The task_id function returns the Ada task identifier associated with a trace event NOTE This function is only meaningful for trace events logged by Ada tasking programs SYNTAX task_id QE PARAMETERS QE A user defined qualified event If supplied the function returns the Ada task identifier of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the Ada task identifier of the current trace event For more information see Qualified Events on page 9 81 RETURN TYPE integer SEE ALSO start_task_id on page 9 40
32. when you click on an arrowhead of the interval scroll bar or between an arrowhead and the slider on the interval scroll bar This field may contain either a percentage or an absolute amount of time in seconds The default is 25 Editing Single Fields 6 12 Changing the interval control area fields allows you to examine different intervals in your trace session Usually you modify fields in the interval control area when you already know something about your trace events and their distribution When you press lt Enter gt or click on the Apply push button at the end of your editing ntrace validates the data in each field you modified and takes appropriate action If ntrace detects an invalid value it restores the affected field to its previous value For more information on the Apply push button see The Interval Push Buttons on page 6 8 nt race displays all times in the interval control area in seconds with the s suffix You can enter times into time related fields in the following ways Numeric time nt race assumes that the time unit is seconds Numeric time in seconds with a s suffix 6699 Numeric time in microseconds with a u suffix The following text explains what constitutes a valid field change and describes the effects of changing a single field For general information on field editing see Field Editing on page 6 16 Time Start A valid change keeps Time Start less than the endi
33. 5 9 5 15 5 31 5 34 8 2 11 6 eventmap file 5 11 5 14 11 6 Exception 5 5 5 7 11 3 11 13 11 17 11 18 11 20 e e e e e e e e e e e e 6 e e e Index 4 graph 11 13 reference 11 21 resumption 11 3 11 13 suspension 11 3 11 13 exec 2 service 2 7 2 10 Exit menu item 5 32 Expansive mode 4 2 4 4 4 5 4 12 4 13 4 14 A 1 A 4 Expression configuration parameter 9 4 9 7 Expression field 9 7 Expressions 9 4 constant string literals 5 18 9 13 9 77 functions 9 9 macros 5 9 5 11 5 27 5 30 7 18 8 2 operands 9 5 operators 9 4 qualified events 5 9 5 10 5 11 5 27 5 30 7 18 8 4 9 81 qualified states 5 9 5 11 5 27 5 30 7 18 8 4 9 83 Expressions menu 9 1 Field Background Color 8 3 CPU List 8 2 9 81 9 83 Glossary 14 Current Time 6 7 6 11 6 13 editing operations 6 16 End Events 8 15 9 7 9 85 End Expression 8 15 9 4 9 7 9 83 10 7 Glossary 14 Event Color 8 15 Event Count 6 12 6 13 Event End 6 7 6 12 6 13 Event List 8 2 9 83 9 85 10 5 10 7 Event Start 6 7 6 11 6 13 Expression 9 7 File Name 5 33 5 34 Filter 5 35 Filter Expression 9 4 10 7 Font 8 3 Foreground Color 8 3 If Expression 8 2 9 4 9 7 9 81 9 83 9 85 10 7 10 8 Glossary 14 Increment 6 7 6 12 6 14 Lost Event Color 8 19 MacroDefinition 9 7 Mark Color 8 19 Maximum 8 18 Minimum 8 18 Name 8 2 No Event List 10 5 Node List 8 3 9 81 9 83 Glossary 14 PID List 8 3 9 81
34. CPU Bias Selection of a specific CPU or set of CPUs can be advantageous to prevent event loss reduce daemon instruction on the rest of the system and for careful configura tion to allow for relaxed Spin Lock Protection see Spin Lock Protection on page 3 34 All CPUs Selects all CPUs on the target system On platforms belonging to the local global remote subclass of non uniform memory access NUMA architectures primary memory is divided into global and local memories Global memory is located on a memory board where it is equally distant in terms of access time from all of the CPUs in the system All CPUs share a single data path to glo bal memory known as the system bus Local memory is located on a CPU board where it is closer in terms of access time to the co resident CPUs The path between a CPU and its local memory does not include the system bus Local memory usage improves the throughput of the system in two ways smaller access times for the co resident CPUs and less system bus contention for the remaining CPUs Applications can influence the page placement decisions made by the kernel by selecting NUMA policies for different parts of their address spaces NUMA policies specify where data should reside in the local global remote hierarchy NOTE These settings are ignored for non NUMA target systems archi tectures such as PowerHawk PowerStack and iHawk series machines NightTrace Manual NOTE These
35. Create the various display objects you want and place them on the display page Move or resize any display objects necessary to improve the layout of the page The Display Page Menu Bar Mode Buttons Message Display Area The Grid Thread threadi User Events d Interval en Scroll a el Bar Time Start o o000000s Time Length 4 5008572s Time End 4 5008572s Event Start o Event Count s Event End 4 Interval Zoom Factor 2 0 Increment 25 002 Current Time 1 1252140s Control Apply Reset Center Mark Zoom Region Zoom In Zoom Out Refresh Area Interval Push Buttons Figure 7 2 Elements of a Display Page 7 2 Creating Display Objects A display page lets you view the trace event data in the trace event file Figure 7 2 shows an example of a display page and points out the portions of the display page Following is a brief description of the portions of a display page Menu bar Contains menu items When you click on a menu item in the menu bar a pull down menu appears with a list of related menu entries You can then initiate an operation on the menu Mode buttons Are radio buttons that control whether the display page is in Edit or View mode and allow you to switch between modes by clicking on them Message display area Displays error and status messages It has a scroll bar so you can view previous or current messages Grid Contains display objects Figure 7 2 shows a grid bef
36. Node Box Figure 11 7 shows a node box The node box simply identifies which node the displayed data corresponds to NOTE The node information is displayed only when NightTrace is con figured to use an RCIM to timestamp events The node box is a GridLabel display object See Chapter 7 and Chapter 8 for more infor mation on creating and configuring GridLabels Context Switch Information Context Switch Lines Current Time Line Figure 11 9 Context Switch Lines Figure 11 9 shows an example of several context switch lines Context switch lines are superimposed on the exception and syscall graphs They indicate that the kernel has switched out the process that was previously running on the CPU and switched in a new process There is a direct correlation between context switch lines and the running process box the running process box shows the process associated with the context switch line that immediately precedes the current time line 1 11 NightTrace Manual Interrupt Information Last Interrupt Name Current Time Line ho resched strsched Interrupts Nested Interrupts Figure 11 10 Last Interrupt Box and Interrupt Graph 1 12 Figure 11 10 shows a last interrupt box and an interrupt graph The interrupt graph dis plays a state that is drawn whenever an interrupt is executing on the associated CPU Inter rupts can be interrupted while executing and the interrupt graph shows this interrupt n
37. The of fset_pid function returns the global process identifier PID from which the ordinal trace event offset was logged NOTE A global process identifier does not have the same meaning as the typical operating system definition of pid A PID within Night Trace is a 32 bit integer value that contains the operating system process identifier raw PID in the upper 16 bits and the light weight process identifier LWPID in the lower 16 bits Consult the 1wp global self 2 man page for more information SYNTAX offset pid offset expr PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE integer SEE ALSO pid on page 9 22 raw pid on page 9 23 Iwpid on page 9 23 start pid on page 9 37 and end pid on page 9 48 Using Expressions offset raw pid DESCRIPTION The offset raw pid function returns the process identifier raw PID from which the ordinal trace event offset was logged NOTE A NightTrace raw PID has the same meaning as the typical oper ating system definition of pid See the getpid 2 man page for more information SYNTAX offset raw pid offset expr PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE integer SEE ALSO pid on page 9 22 raw pid on page 9 23 Iwpid on page 9 23 start_
38. They operate on either the qualified event specified to that function or if unspecified the current trace event Trace event functions include multi event functions An integer that identifies a trace event User trace event IDs are in the range 0 4095 inclusive Kernel trace event IDs are in the range 4100 4300 inclusive A symbolic name mapped to a numeric trace event ID in an event map file A place of interest in the source code In user tracing at each trace point in your application you call a trace event logging routine to log a trace event possibly with additional data describing part of your program s state at that time Kernel trace points and trace events are already defined and embedded in the kernel source The graphic part of a scroll bar that holds the slider A pre defined dynamically generated string table in the vectors file This string table contains the interrupt and exception vector names associated with the system on which the kernel tracing was performed The display page mode that allows you to see search for and summarize trace event information in the message display area the summary display area and display objects on the grid create edit and configure macros qualified events and qualified states The other display page mode is Edit mode A window component for example a scroll bar or push button A rectangular screen area that permits the display and or entry of data The Night Trace display
39. You can control the interval by zooming in or out scrolling searching for specific trace events or jumping to portions of the trace event file Display summary information This information may be about your entire trace session or the characteristics of particular trace events and states in this trace session Introduction Recommended Reading Referenced publications appear in the front of this manual Related text books that are useful resources for general background information follow X Window System User s Guide This text book by Valerie Quercia and Tim O Reilly is published by O Reilly amp Associates Inc It is available under publication number 0890300 This text book introduces X terminology and concepts It also discusses several popular window managers the xterm terminal emulator X resources and X desk accessories OSF Motif Style Guide This text book is published by Prentice Hall Inc It and its companion books OSF Motif User s Guide and OSF Motif Programmer s Guide are packaged together under publication number 0890380 This text book introduces Motif terminology and concepts It also provides information about Motif features 1 7 NightTrace Manual 1 8 2 Adding Library Calls to Your Application OVA alate ea tte d t ae ERR eaten e 2 1 Language Specific Source Considerationfcs 0 0 00 eee eee eee 2 1 Cs EE EUM TM AM a 2 1 POT ls coh A eg aoe ead va ae etu eoe ce E SA S RS da 2
40. arg3 get string device simulator4 3604484 get item device controller rtc The pid string table is also used by the kernel display pages For more information on the pid string table see Pre Defined String Tables on page 5 14 For examples of function calls with these tables see Table 8 3 Kernel Reference The following sections provide a brief reference to the most common interrupts exceptions and syscalls 11 18 Interrupts Tracing the Kernel There are many different types of interrupts that can be logged by the kernel The possible types are listed in the system dependent vector string table in the vectors file There are two main categories of interrupts Non device related interrupts Device related interrupts The members of these two categories are described in the following two sections Non Device Related Interrupts Table 11 2 provides an alphabetical list of the most common non device related inter rupts Table 11 2 Non Device Related Interrupt Reference Interrupt Description callout int console wake int on no int power fail rescheduling softclock spurious int sysfault int user int xcall int A real time clock interrupt that is used internally by the kernel An interrupt caused by the console wakeup button An interrupt that occurs during the processing of another interrupt A power fail interrupt A rescheduling interrupt use
41. end_task_id on page 9 51 and offset task id on page 9 64 tid DESCRIPTION The tid function returns the internally assigned NightTrace thread identifier TID associated with a trace event SYNTAX tid 10ED 9 25 NightTrace Manual PARAMETERS OE RETURN TYPE integer SEE ALSO A user defined qualified event If supplied the function returns the NightTrace thread identifier of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the NightTrace thread identifier of the current trace event For more information see Qualified Events on page 9 81 TID List on page 8 8 start_tid on page 9 41 end tid on page 9 52 and offset tid on page 9 65 cpu DESCRIPTION The cpu function returns the logical CPU number associated with a trace event CPUs are logically numbered starting at 0 and monotonically increase thereafter NOTE This function is only valid when applied to events from Night Trace kernel trace event files SYNTAX cpu QE PARAMETERS OE RETURN TYPE integer 9 26 A user defined qualified event If supplied the function returns the logical CPU number of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the logical CPU number of the current trace event For more inform
42. interval control area See Field Editing on page 6 16 for more information Note that you can type default or just a space in a field to get the default value Many of the display objects share common configuration parameters These common configuration parameters are summarized in Table 8 1 and discussed in the following sections For more information about configuration parameters refer to the sections on configuring the object you are interested in Table 8 1 Common Configuration Parameters Parameter Name Possible Values Meaning Display Object Any alphanumeric string beginning witha The name of the display object Name letter Underscores are also allowed Spaces are not allowed Any meaningful combination of the fol lowing ALL All trace events are caught ALLUSER Alluser trace events are caught Event List ALLKERNEL All kernel trace events are caught NONE No trace events are caught 8 70 lu Ixus 409b Listed user trace events are caught 4100 4101 4102 4300 Listed kernel trace events are caught A comma separated list of alphanu meric strings beginning with letters Underscores are also allowed Spaces are not allowed The tags of trace events as specified in an event map file are caught See Un derstanding Event Map Files on page 5 9 for more information If Expression Boolean expression Expression is any valid boolean C like expr
43. is configured to use an RCIM to timestamp events See Node List on page 8 9 for more information about this field For information about other configuration parameters see Chapter 8 especially Com mon Configuration Parameters on page 8 1 Configuring qualified events is similar to configuring DataBox display objects The configuration parameters for a qualified event are identical to those that are used to configure a DataBox display object See DataBox on page 8 13 for information on how to configure a DataBox EXAMPLE Qualified events can be useful when you are interested in seeing a trace event or state that occurs within a certain amount of time after another trace event Given the following qualified event configuration Using Expressions QualifiedEvent fire Event List FIRE CPU List 2 an EventGraph can be configured to show only BAR trace events that happen within 100 microseconds of a FIRE trace event on CPU 2 Event List BAR If Expression time time fire lt 100us Note The BAR trace events themselves can happen on any CPU and as long as they occur with 100 microseconds of a FIRE trace event on CPU 2 they will be graphed Qualified States A qualified state is a user defined named state configuration that consists of a set of one or more states possibly restricted by a Start Expression End Expression CPU List TID List PID List and Node List Qualified stat
44. maximum of 5 seconds The ntraceud timeout option lets you establish the maximum number of seconds seconds that the nt raceud daemon sleeps Waking the nt raceud daemon incurs overhead that can distort trace event timings decreasing the timeout parameter makes it more likely that the daemon will be awake when needed You can also decrease the timeout parameter to prevent trace event loss Note If your application does not log events frequently you can increase the timeout to reduce the time the daemon runs and consumes CPU cycles If you use the timeout option you must start up ntraceud with it If you invoke ntraceud with both the timeout and bufferwrap options ntraceud ignores the timeout option ntraceud does not sleep for the full period if Your application executes a call to trace flush trace trigger ortrace end Your application logs a trace event that causes shared memory to become full or your buffer full cutoff percentage to be reached You specify a timeout parameter which exceeds the time in which the lower 32 bits of the timestamp source would roll over This rollover time varies from architecture to architecture with a minimum value of 257 69803 seconds and is calculated by ntraceud as part of its initialization It is important to detect this rollover so that proper ordering of trace events is maintained If you specify a timeout inter val which exceeds the rollover time ntraceud uses the rollover t
45. message in the message display area on the display page For more information on the Search push button see Search Form Push Buttons on page 10 3 When you make field changes and click on Search ntrace temporarily saves your field settings in the search history in memory You can step through these groups of set tings by clicking on the Prev and Next push buttons Clicking on the Close push button erases all but the last group of field settings from the search history For more information on these push buttons see Search Form Push Buttons on page 10 3 See Chapter 8 for a definition of each field all its possible values and its default value There is only one difference between the ntrace behavior described there and the behavior of the Search Form on the Search Form ntrace searches for but does not display data that meets the criteria The search stops when ntrace finds a suitable value or runs out of trace events The No Event List field is the only field that is unique to the Search Form This field lets you decide which trace event s to ignore in a search The possible values are the same as those in the Event List field It is not meaningful to put the same value in the Event List and in the No Event List NOTE The Node List field appears in this dialog only when NightTrace is configured to use an RCIM to timestamp events Summarizing Statistical Information Clicking on Tools I Summarize on the display page
46. 1 shows the display page menu that lets you define macros qualified events and qualified states For more information about display pages see Understanding Page Con figuration Files on page 5 11 Figure 9 1 Expressions Menu Selecting any of these menu entries makes an Expression Dialog Box appear 9 1 NightTrace Manual Expression Dialog Boxes In the following text expr stands for macro qualified event and qualified state Selecting any of the entries from the Expressions menu of the display page makes a dialog box like the one in Figure 9 2 appear Because all exprs are user defined the list of exprs is empty at first Figure 9 2 Macro Dialog Box The caption and the list presented are suitably different for each of the expr dialog boxes The push buttons in the dialog boxes perform the following functions Add Create a new expr on the current display page The initial name of an expr is type_ where type is macro event or state and is a three digit number beginning with 001 Delete Remove the selected expr Configure default Reconfigure or rename the selected expr Close Close the dialog box Add Delete and Close need no further explanation Selecting Configure makes an expr Configuration Form appear Expression Configuration Forms In the following text expr stands for macro qualified event and qualified state The Configuration Forms for exprs are similar Common features are describe
47. 10 When you click on File I New Page on the Global Window a display page with a dotted but otherwise empty grid appears This menu item is useful if you want to create a display page from scratch Most regular features of a display page are immediately available on a new page However you must create and configure display objects before you can Examine trace events states and trace event arguments using different display objects Graphically depict your trace session NOTE The new display page comes up in Edit mode In Edit mode the field labels scroll bar and push buttons on the display page are disabled dimmed 5 27 NightTrace Manual Default Page 5 28 Tine Sartjo 0 0000000s es engin 0 0006400s Time Eng 0 0006400s Event Start o 3n Event End o Zoor Factor 2 0 5 002 fae urrent Time 0 0001518s as Av Reset enter Man Ze ion Zoomin Zoom ut Retesh Figure 5 4 New Display Page When you click on File Default Page on the Global Window an automatically generated display page appears The purpose of the default display page is to save you time configuring display objects if this page does not exactly meet your needs you can customize it or you can create a new display page ntrace brings up this page in View mode All regular features of a display page are immediately available on a default page Before ntrace displays a default display page it counts the number of processes in your tra
48. 11 has three numeric fields that list the beginning end and current time for the time interval displayed in the Column A Ruler display object however displays this information in a graphical format on the grid Like their physical counterparts Ruler display objects have major and minor hash marks to mark divisions but the units are of time not distance They represent the amount of time since the first trace event was logged Usually the first trace event is logged by the trace_open_thread call You can create a Ruler by clicking on Create 1 Ruler and drawing on the grid In addition to hash marks and numbers Rulers can also have lost data indicators and a mark The lost data indicator is a reverse video L and indicates the location in time where NightTrace lost some data For more information on trace event loss see Prevent ing Trace Events Loss on page A 1 Marks are explained in The Interval Push Buttons on page 6 8 Rulers are static display objects That is they do not change their appearance or contents depending on the trace event data They do change their appearance however to reflect the current interval being displayed A Ruler should be at least three grid dots high In addition to determining the size of the Ruler you also specify other aspects of the Ruler See Ruler on page 8 19 for more information Editing Operations Figure 7 14 Edit Menu 7 17 NightTrace Manual Select All
49. 16 11 Tracing the Kernel OVIEDO a 11 1 Default Kernel Trace Points ooo oooooooooccocoro cee ee 11 1 Context Switch Trace Event eee eee 11 2 Interrupt Trace Evehts at O Ral eee els ee 11 2 Exception Trace Events e eee cee es 11 3 Syscall Trace Events sx cti pad Bae eee 11 4 Kernel Trace Points Not Enabled By Default o ooooo oocoococooooo 11 5 Page Fault Evento tse SEES AS RONDE UY SPAN 11 5 Protection Fault Event ose esee ienei ei eee eh 11 5 Viewing Kernel Trace Event Files llle eese 11 6 Kernel Display Pages 11 6 RCIM Default Kernel Display Page 00 00 0000 00 o 11 7 CPU Informal o boat dd cea 11 9 Running Process Information 0 00 00 c eee eee eee eee 11 10 Node Information 0 0 0 ccc eee eee 11 11 Context Switch Information 0 0 0 0 eee ee eee eee 11 11 Interrupt Information 0 eect tenes 11 12 Exception Information enres ensa enal ne cece eee eee 11 13 Syscall Information 0 00 cece eee eee 11 15 Color Information ne s cue ie ewe ede eae BRAGS eS eee 11 16 Kernel String Tables 2 0 eee cee ee nce Rh nn 11 16 Kernel Referente ia E Neee a a a a onde Badal 11 18 nterr pts s taa da e Stee aan tao E RASE A N 11 19 Non Device Related Interrupts 0 0 2 eese 11 19 Device Related Interrupts 0 0 eee eee eee 11 20 Exceptions sic ea oles nih Laewain A Sects 11 20 Syscall seis
50. 19 MacroDefinition 9 7 Mark Color 8 19 Maximum 8 18 Minimum 8 18 Name 8 2 Node List 8 3 8 9 PID List 8 3 8 7 QualifiedEvent 9 82 9 83 QualifiedState 9 84 9 85 Start Events 8 15 Start Expression 8 15 Text 8 12 Text Gravity 8 4 8 10 Text Justify 8 3 8 9 Then Expression 8 2 8 5 9 4 9 76 9 78 9 79 9 80 TID List 8 3 8 8 Configure menu 8 1 Configure push button 9 2 Configuring display object 8 1 Conserving disk space 4 28 A 1 A 4 Constant string literals 5 18 9 13 9 77 Constant times 9 6 Content menu item 8 1 Context switch 11 1 11 4 lines 11 11 11 13 11 15 Counters 7 13 8 13 Index 2 cpp 1 command 2 24 CPU box 11 10 cpu function 8 5 9 26 CPU List configuration parameter 8 2 8 6 CPU List field 8 2 9 81 9 83 Glossary 14 CPU number logical 8 6 11 10 physical 11 10 Create menu 7 6 7 8 Create mouse operation 7 6 crossref trace_flush_and_trace_trigger 4 17 Current time 6 2 Current Time field 6 7 6 11 6 13 Current time line 6 2 6 2 11 9 11 12 11 13 centering 6 9 manipulating 6 9 Cursor see Text field cursor Cutoff 4 4 4 5 4 18 A 3 C 1 D DataBox 5 29 7 12 8 13 9 79 9 82 10 11 11 10 11 12 11 13 11 16 B 4 configuration form 8 13 using as a counter 7 13 8 13 DataGraph 5 30 6 4 7 16 8 17 11 12 B 4 configuration form 8 17 Fill Style configuration parameter 8 17 Maximum configuration parameter 8 18 Minimum configuration parameter 8 18 Debugger N
51. 19 10 10 Content 8 1 Default Kernel Page 5 27 11 6 Default Page 5 27 5 28 Delete 7 18 Deselect All 7 18 desensitized 5 27 7 19 dimmed see Menu item disabled disabled 5 27 7 4 7 18 10 10 Exit 5 32 New Page 5 27 Open Config File 5 12 5 27 5 30 5 36 Read Event Map File 5 10 5 31 5 36 Save 7 18 Save As 7 19 Save Text 10 10 Save Text As 10 10 Search 10 1 Select All 7 18 Summarize 10 5 Message display area 5 10 5 25 6 1 6 4 6 14 7 3 7 7 7 15 7 16 statistics 6 9 min function 9 71 10 7 10 8 min_offset function 9 73 10 7 10 8 Minimum configuration parameter 8 18 Minimum field 8 18 Minimum value 8 18 9 71 9 73 Mode buffer wraparound 2 21 4 4 4 13 4 17 4 18 A 1 A 4 Edit 5 27 6 1 7 4 expansive 4 2 4 4 4 5 4 12 4 13 4 14 A 1 A 4 file wraparound 4 4 4 5 4 12 A 1 A 4 radio buttons 7 3 View 5 28 6 1 6 6 7 4 9 7 10 1 11 6 Monochrome display 5 2 7 14 B 1 B 4 Motif 1 7 5 2 Mouse pointer shapes 5 22 Mouse button 1 5 21 6 2 6 3 6 8 6 9 6 16 7 5 7 6 2 6 3 6 4 6 8 6 9 6 16 7 5 7 7 7 15 7 16 3 6 3 6 4 6 10 6 16 7 5 7 7 7 16 Mouse operation create 7 6 move 7 7 resize 7 7 select 7 6 Move mouse operation 7 7 mpadvise 3C routine A 4 Multi event functions 9 32 Multi state functions 9 56 Name configuration parameter 8 2 Name field 8 2 name_pid table 5 16 11 17 name_tid table 5 16 New Page menu item 5 27 Next push button 10 3 10 5 NFS file
52. 2 AGE eb MEI iig eu Ice EHEROIOE MELOS eh eS 2 2 Inter Process Communication and Library Routines lesse eese 2 3 Understanding NightTrace Library Calls 0 0 0 0 eee eese 2 3 trace begin ses Peer Rx Re IN UR e a ce T A Res S 2 5 trace open thread ui REIR RR Re 2 9 trace event and Its Variants 0 0 0 ccc cette 2 10 trace_enable trace_disable and Their Variants ooooooo o o 2 16 trace flush and trace_trigger 0 0 2 cece ee eee 2 20 trace close thread 0 cc eee en o 2 22 t cita oles rh atte bashes a is Saeed 2 23 Disabling Tracing uenerat eed chee 2 24 Compiling and Linking 0 0 0 0 e a E e 2 24 CLERAMple sas rt ot eee el a a re il er Mar 2 25 Fortran Examples vinotinto PO MESA ENS 2 25 Ada Example op e exempt du 2 25 Exercise Instrumenting Code ossos senes omiani ne eee ee 2 25 NightTrace Manual 2 Adding Library Calls to Your Application Overview This chapter describes language specific considerations for using NightTrace with user applications CAUTION Do not call clock_settime from your application This sys tem call can corrupt both the system interval timer and Time Base Register which NightTrace uses for trace event timings Language Specific Source Considerationfcs NightTrace applications must be written in C Fortran or Ada For your applications to trace events you must edit your source code and insert NightTrace library routin
53. 4 DataGraph 5 30 6 4 7 16 8 17 11 12 B 4 EventGraph 5 30 6 4 7 15 8 16 9 7 B 4 GridLabel 5 29 7 12 8 12 11 10 11 11 B 4 loading 7 6 moving 5 22 7 5 7 7 overlapping 7 8 placement 5 22 7 6 properties 7 9 resizing 5 22 7 5 7 7 Ruler 5 30 6 9 7 17 8 19 A 1 B 4 selecting 7 5 7 6 StateGraph 5 28 5 29 6 4 7 14 8 14 9 7 9 84 10 12 11 13 11 16 B 4 usage flowchart 7 12 Display object configuration parameters Background Color 8 3 8 9 common 8 1 CPU List 8 2 8 6 Index End Events 8 15 End Expression 8 15 Event Color 8 15 Event List 8 2 8 4 Fill Style 8 17 Font 8 3 8 9 Foreground Color 8 3 8 9 If Expression 8 2 8 4 Lost Event Color 8 19 Mark Color 8 19 Maximum 8 18 Minimum 8 18 Name 8 2 Node List 8 3 8 9 PID List 8 3 8 7 Start Events 8 15 Start Expression 8 15 Text 8 12 Text Gravity 8 4 8 10 Text Justify 8 3 8 9 Then Expression 8 2 8 5 9 4 9 76 9 78 9 79 9 80 TID List 8 3 8 8 Display page 5 27 6 2 7 2 11 6 B 2 X resources B 2 Display page area grid 6 4 B 2 interval control area 6 11 interval push buttons 6 8 interval scroll bar 6 7 C 1 message display area 6 1 6 4 6 9 6 14 7 3 7 7 7 15 7 16 Do Not Move Current Time radio button 10 3 Dotted area see Grid Duration state 9 57 Edit menu 7 17 Edit mode 5 27 6 1 7 4 Editing operation delete character 6 16 insert character 6 16 positioning 6 18 replace character 6 17 restore the default 6 17 undo 6 17 Enabli
54. 6 10 9 10 10 Summarize form radio buttons Event 10 6 Interval 10 6 Region 10 6 State 10 6 Summary Range 10 6 Summary Type 10 6 Trace Event File 10 6 Summarize push button 10 6 10 9 10 10 Summarizing trace event 1 3 10 5 trace session 1 6 Summary event 10 11 matches 9 74 10 8 state 10 12 Summary display area 10 7 10 9 10 10 10 11 10 12 Summary functions 9 4 9 70 Summary Range radio buttons 10 6 Summary Type radio buttons 10 6 summary_matches function 9 74 10 8 Summary Expression field 9 4 9 81 10 7 10 8 10 11 10 12 Syscall 11 4 11 15 11 17 graph 11 15 resumption 11 4 suspension 11 4 11 15 Index 13 NightTrace Manual syscall table 5 17 8 6 11 4 11 17 11 21 syscall_nodename table 5 17 11 18 System call 11 4 11 15 11 17 Table boolean 5 16 8 6 device 5 17 11 4 11 17 device_nodename 5 17 11 18 event 5 14 8 5 event_arg_dbl_summary 5 20 8 6 event_arg_summary 5 20 8 6 10 8 event_summary 5 20 8 6 10 7 format 5 9 5 17 9 79 functions 9 75 name_pid 5 16 11 17 name_tid 5 16 node_name 5 16 11 17 pid 5 15 8 5 11 18 pid_nodename 5 16 11 17 pre defined 5 14 5 20 11 4 11 16 state_summary 5 20 8 6 10 7 string 5 9 5 13 9 75 9 77 syscall 5 17 8 6 11 4 11 17 11 21 syscall_nodename 5 17 11 18 tid 5 15 8 6 tid_nodename 5 16 vector 5 17 8 6 11 2 11 3 11 17 11 19 vector_nodename 5 17 11 18 tables file 5 14 5 16 5 20 5 20 9 77 9 78 9 80 10 8 Task name 9 31 ordinal tra
55. 9 83 Glossary 14 QualifiedEvent 9 82 QualifiedState 9 84 Start Events 8 15 9 7 9 85 Start Expression 8 15 9 4 9 7 9 83 10 8 Summary Expression 9 4 9 81 10 7 10 8 10 11 10 12 Text 8 12 Then Expression 8 2 TID List 9 81 9 83 Glossary 14 Time End 6 7 6 11 6 12 10 3 10 6 Time Length 6 11 6 13 Time Start 6 7 6 11 6 12 10 3 10 6 Zoom Factor 6 12 6 13 Field editing 6 16 multiple fields 6 14 single fields 6 12 Field label dimmed 5 27 7 4 File Xdefaults 5 2 5 8 B 1 B 5 Xresources 5 2 5 8 dev spl A 3 etc conf mtune d trace A 2 usr bin ntrace 5 3 usr bin ntraceud 4 2 usr include ntrace h 2 1 2 24 4 23 usr lib libntrace a 2 24 usr lib NightTrace eventmap 5 11 5 14 11 6 asr lib NightTrace examples entry_exit c 2 25 usr lib NightTrace examples entry exit map 5 35 usr lib NightTrace examples entry exit page 5 35 usr lib NightTrace tables 5 14 5 16 5 20 5 20 9 77 9 78 9 80 10 8 configuration 5 9 5 11 event map 2 13 5 4 5 9 5 9 5 15 5 31 5 34 8 2 11 6 filter 5 35 NightTrace kernel trace event 5 5 5 7 5 27 rgb txt 5 8 8 3 8 15 8 19 B 1 B 5 trace event 1 5 2 5 4 1 4 12 4 13 5 9 A 4 trace see trace event vectors 5 5 5 15 11 2 11 17 11 18 File menu 5 26 7 18 10 9 File Name field 5 33 5 34 File Selection Dialog Box 5 31 5 32 5 33 7 19 10 10 B 3 File system NFS A 2 A 3 C 2 File wraparound mode 4 4 4 5 4 12 A 1 A 4 Fill Style configuration parameter 8 17 F
56. Close Rather than duplicating functionality in both a menu entry and push button ntrace supports the menu entry A radio button is a graphic labeled diamond shape that represents a mutually exclusive selection from related radio buttons Related radio buttons usually appear on the same panel a panel between the menu bar and the push buttons The first radio button is the default one The selected radio button has a different color than the others Scroll regions and scroll bars Text fields A scroll region appears between the menu bar and the push buttons The scroll bar is immediately below or to the right of the scroll region A text field appears between the menu bar and the push buttons While a text field is being edited it contains a blinking vertical bar called the text cursor The text cursor shows your current edit posi tion within the field Figure 5 1 illustrates these window components 5 23 NightTrace Manual Menu Bar Scroll Region Radio v Edit and Buttons o View Y Scroll Bar offset 3 id 1562 argl 0 Thread threadi fgarcia 2059 0 Ihread threadi cap 815 0 User Events Scroll Bar Time Start 0 0000000s Time Length 4 8261996s Time End 4 8261896s T Event Start o Event Count 3 Event End 8 ext Push Zoom Factor 2 0 Increment 25 007 Current Time 1 2065472s Fields Buttons Apply Reset Center Mark Zoom Region Zoomin Zoom Out Refresh Fig
57. EA Ee 2 9 trace event and Its Variants 0 0 0 ec eee 2 10 trace_enable trace_disable and Their Variants oooooooo o 2 16 trace flush and trace_trigger 2 2 eee eee eens 2 20 trace close thread 4 x cae a PERO we 2 22 tace end 5 casei a Sb dE Ct ie 2 23 Disabling Tracing eee eneret m A UR E SACS 2 24 Compiling and Linking llleeeeeeeeeee Ih 2 24 GXExample vest a ee RET DERE Se a 2 25 Fortran Example s eU e esc e mS 2 25 Ada Example 2 ede Ree Ir ES US bg ee eon eo VR 2 25 Exercise Instrumenting Code 0 0 cece cee eee 2 25 Chapter 3 NightTrace Session Manager Using the NightTrace Session Manager 0 0 cece eee 3 1 NightTrace Session Manager Menu Bar 0 0 00 00 eee 3 2 NigbtTrace ea eee Cere ce ir 3 2 Open Session obe eR REI ed ee be eee 3 4 NightTrace Manual S veSesslOD 4c Soh oe eis tae eee ee EN 3 6 Unsaved Changes serie rec eA eA WR sens E RUN 3 8 DAEMON CERE 3 9 IPM IP 3 10 Enter Password iso eI ee de ee ace 3 11 Attach Da MOS ic cele bh ERE pU E a OPE PE 3 12 Options cose pe E ta tesque be ied 3 14 Refresh Interval 5st che ea E RUE IRR CREAR DES 3 14 TOOLS aisha Etpe OW UR GIOI ERES 3 14 Help yt lia Phe ENS RU REEF EE 3 16 Session Configuration File Name Area lsel eese 3 17 Daemon Details ATea cocer A OU CU EAE ER 3 17 Daemon Control Area 2 cette re 3 20 Enable Disable Trace Events 3 22
58. For dragged over text fields NightTrace displays that text in reverse video A line that groups related window components or menu components A collection of one or more daemon definitions The intermediate destination of trace events before nt raceud copies them to the trace event file on disk The graphic part of a scroll bar that you move in the trough to change the display This component is sometimes called a thumb A device used to protect a resource for example the shared memory buffer A state function that provides information about the start event of the most recent instance of a state The state to which the start function applies is either the quali fied state specified to the function or the state being currently defined Thus if a qualfied state is not specified start functions are only meaningful when used in expressions associated within a state definition In addition start functions should not be used in a recursive manner in a Start Expression a start function should not be specified in a Start Expression that applies to the state definition contain ing that Start Expression Conversely an End Expression may include start functions that apply to the state definition containing that End Expression A state is bounded by two trace events a start event and an end event An instance of a state is the period of time between the start event and end event including the start and end events themselves Additiona
59. Name Pattern If you know only some of the characters in the file name use the steps below to find and open it 1 Type a file name pattern possibly with leading directory name s and appropriately placed asterisk s into the Filter text field Each asterisk in this field represents zero or more characters at this position 2 If you mistype the field name pattern see Field Editing on page 6 16 and correct the problem 3 Press lt Enter gt This causes ntrace to replace the contents of the scroll region with subdirectory names and file names that match your pattern To locate your file in the scroll region and open it see the Scrolling Through Existing File Names on page 5 34 If you caused ntrace to bring up this File Selection Dialog Box again all fields contain the same values as when you left this dialog box Exercise Displaying Trace Events The following exercise has you graphically display the trace events you logged in Exer cise Logging Trace Events on page 4 29 Copy the usr lib NightTrace examples entry exit page configuration file to your directory and call it page See Understanding Page Configuration Files on page 5 11 for more information about configuration files cp usr lib NightTrace examples entry exit page page Copy the usr lib NightTrace examples entry exit mapevent map file to your directory and call it map See Understanding Event Map Files on page 5 9 for more inform
60. Open As an alternative to these two steps you could double click quickly on the file name If you have read permission to this file and it is of the right type ntrace opens the file If it is a configuration file ntrace puts up all display page s from this file If it is an event map file ntrace adds those trace event tags and trace event argument formats to its internal list If opening the file was not successful ntrace puts up a Warning Dialog Box The warning message in the dialog box differs depending on the problem When you have read the warning click on OK As a result ntrace removes the dialog box If you do not find the file in the list 1 Click on the directory name under which it resides This selects and high lights the directory name 2 Click on Open As an alternative to these two steps you could double click quickly on the directory name This causes ntrace to e Put the selected directory s name in the File Name text field Change to that directory cd Display the file and directory names under that directory Invoking the ntrace Display Utility You can repeat the steps in this method until you find the file If you cause nt race to bring up this File Selection Dialog Box again all fields contain the same values as when you left this dialog box TIP Clicking on the directory causes the scrolled list to be filled with the contents of the parent directory Typing in a Filter File
61. Option to Select Timestamp Source clock The ntraceud clock option allows you to select which timing source will be used to timestamp events SYNTAX ntraceud clock source trace file DESCRIPTION The ntraceud clock option lets you select the timing source used to timestamp trace events Valid source values are default the interval timer NightHawk 6000 Series or the Time Base Register Power Hawk PowerStack rcim tick the RCIM synchronized tick clock If you invoke ntraceud with the clock option you must supply a value for the source If rcim tick is specified for the source and the system on which you are tracing does not have an RCIM installed or configured or if the RCIM synchronized tick clock on the system on which you are tracing is stopped the NightTrace daemon and library initialization routine exit with errors If the clock option is not specified the interval timer NightHawk 6000 Series or the Time Base Register Power Hawk PowerStack is used to timestamp trace events 4 19 NightTrace Manual Option to Reset the ntraceud Daemon reset The nt raceud reset option resets a running nt raceud daemon process SYNTAX ntraceud reset trace_file DESCRIPTION Running ntraceud daemons are located using the shared memory identifier keyed by the trace event file name trace file By default ntraceud overwrites the trace event file if it is not currently in use In contrast the ntraceud reset optio
62. Summary Display Area 0 cece teens 10 10 Event SUMMA eii vL ER SEAS BARODA ded RIWGG x 10 11 State SUMMA 10 12 Exercise Using the Search Tool cee 10 14 Exercise Using the Summarize Tool 0 00 0 c eee eee eee 10 15 Chapter 11 Tracing the Kernel OVEIVICW e AA RO A ane RO ER 11 1 Default Kernel Trace Points oocoo oocoocoocorcor rh 11 1 Context Switch Trace Event 0 0 0 cece eee 11 2 Interrupt Trace Events ia e ORE De ES 11 2 Exception Trace Events zesire pp Wa Aa ee Ge ees 11 3 Syscall Trace Events sidere eR Pe REGUM ERR ES 11 4 Kernel Trace Points Not Enabled By Default o oooooocoocooco momo 11 5 Page Fault Event oce eed eg cand oo oe oe ee Dr e 11 5 Protection Fault Event 0 cece eee e 11 5 13 NightTrace Manual Viewing Kernel Trace Event Files lees 11 6 Kernel Display Pages ocur cose 4 ja oa ie ge hele sales epe y Ree 11 6 RCIM Default Kernel Display Page 00 00 0005 11 7 CPU Information ct A A eas 11 9 Running Process Information 0 0 0 0 cece eee eee 11 10 Node Information eessen editarea te p aeea a e eE 11 11 Context Switch Information 00 0 0 2c eee eee eee 11 11 Interrupt Information sse e 11 12 Exception Information 00 00 cece eee eee 11 13 Syscall Information ve ccc he esa teran eee bebe babe eae e es 11 15 Color Information 0 0 cece eee 11 16 Kernel Ste
63. Table 9 3 NightTrace Functions Syntax Return Type id QE start_id QS end id QS offset id offset expr The integer trace event ID arglN QE start_arg N QS end_arglM QS offset arg N offset_expr The integer trace event argument arg N db1 QE start arg N db1 QS end arg N db1 QS offset arg N dbl offset expr The double precision floating point trace event argument num args QE start num args QS end num args QS offset num args offset expr The number of arguments associated with a trace event pid QE start pid QS end pid QS offset pid offset expr The integer global process identifier PID associated with a trace event raw pid QE start raw pid QS end raw pid QS offset raw pid offset expr The integer process identifier raw PID associated with a trace event lwpid QE start lwpid QS end lwpid QS offset lwpid offset expr The integer lightweight process identifier LWPID associated with a trace event thread id QE start thread id QOS end thread id QS offset thread id offset expr The integer thread identifier thread ID associated with a trace event task id QE start task id QOS end task id QS offset task id offset expr The integer Ada task id
64. This button applies changes made and closes the Daemon Definition dialog Apply This button applies changes made but leaves the Daemon Definition dialog open Reset This button restores the values of all items to the previously applied values and leaves the Daemon Definition dialog open Import Presents the Import Daemon Definition dialog see Import Daemon Defini tion on page 3 27 allowing the user to define daemon attributes based on a user application running on a remote system The Import Daemon Definition dialog is presented following user authentication see Login on page 3 10 and Enter Password on page 3 11 Import Daemon NightTrace Session Manager Cancel This button restores the values of all items to the previously applied values and closes the Daemon Definition dialog Help This button brings up the help topic for this page Definition This dialog allows the user to define daemon attributes based on a running user applica tion containing NightTrace API calls The Import Daemon Definition dialog is pre sented following user authentication see Login on page 3 10 and Enter Password on page 3 11 The user may select an application running on the specified target system from which they wish to import trace related attributes Import Daemon Definition Figure 3 19 Import Daemon Definition dialog Program ID The process ID PID of the Program on the remote system Program
65. about nt race options see ntrace Options on page 5 3 A 5 NightTrace Manual A 6 Overview B GUI Customization The graphical user interface GUI for nt race is based on OSF Motif nt race runs in the environment of the X Window System Your X terminal vendor supplies you with vendor specific directories and files that pertain to colors and fonts The file that contains available colors is called rgb txt The fonts that your X server supports are in the usr 1ib X11 fonts directory ntrace has default values for X resources These resources include fonts some push button names window titles window component dimensions and colors You can over ride the following default X resource settings by providing new values in the following places e In your Xdefaults file Onthe ntrace invocation line e Inaresource file that the xrdb 1 X resource database manager reads If you specify the same X resource on the ntrace invocation line and in your Xdefaults file the setting on the invocation line overrides the one in the file An X resource line has the following format object subobject subobject attribute value where object Is the name of the X client program Nt race subobject Is a level in the widget window component hierarchy with the most general level first this always begins on an upper case letter In ntrace the first subobject is often Color for color displays or Mono for monochrome displays Th
66. also be a hand edited table file con taining definition of string tables and or format tables The NightTrace form that allows you to define a display object s data content constraints and graphic attributes the value of a macro or the constraints of a qualified event or qualified state A push button that reconfigures and renames the selected macro qualified event or qualified state An action that occurs inside the kernel Its functions are to save the state of the process that is currently executing to initialize the state of the process to be run and to begin execution of the new process A vertical line superimposed on an exception graph or a syscall graph on a kernel display page It indicates that the kernel has switched out the process that was previously running on the CPU and switched in a new process See mouse button push button and radio button CPU box current instance of a state current time current time line current trace event cursor daemon definition DataBox DataGraph Default Kernel Page Glossary A GridLabel on a kernel display page It identifies which logical central processing unit the displayed data corresponds to Logical CPU numbers are related to but not necessarily identical to physical CPU numbers The instance of a state which has begun but has not yet completed Thus the cur rent time line would be positioned within the region from the start event up to but not inc
67. analysis tool that is part of the Night Star tool kit It consists of the ntraceud daemon NightTrace library routines and the nt race display utility This product allows you to log trace events and data from applications written in C Fortran or Ada these applications may be composed of one or more processes running on one or more CPUs You can then examine these trace events and those from the kernel through the nt race display utility A process thread or Ada task or a set of any combination of these that is associated with a uniquely named trace context The thread name is derived from the argument specified to the trace open thread function See TID Glossary 11 NightTrace Manual NightView node node box node ID node name node_name table node PID table node TID table NT_ASSOC_PID Glossary 12 A symbolic debugger that is part of the NightStar tool kit It lets you debug C and Fortran applications these applications may be composed of one or more processes running on one or more CPUs Among other things NightView can automatically patch trace event logging routines into your executable application A system from which a trace event file can come from If the RCIM synchronized tick clock is used to timestamp events this is a GridLabel on akernel display page It identifies which node to which the displayed data corre sponds A unique identifier internally assigned by NightTrace to every node th
68. and Library Routines Your application logs trace events to the shared memory buffer Later a user daemon cop ies trace events from the shared memory buffer to the trace event file The relationship between your application and the user daemon and the sequence of library calls needed to maintain this relationship appears in Figure 2 1 Understanding NightTrace Library Calls There is a C Fortran and Ada version of each NightTrace library routine These routines perform the following functions Initialize a trace Open the current thread for trace event logging Log trace events to shared memory Enable and disable specified trace events Copy trace events from shared memory to disk Close the current thread for trace event logging Terminate a trace See the NightTrace Pocket Reference card for a syntax summary of these routines The next sections describe these routines in detail 2 3 NightTrace Manual Parent processes follow this sequence Child processes follow this sequence e trace_begin e trace_open_thread trace_open_thread e log trace events e log trace events e trace_close_thread e trace close thread e trace end Thread 1 Process A Thread 2 Child of B Shared Memory user Child of B Buffer daemon Process C Trace Event File An application written in C can log trace events using e trace event e trace event arg e trace event flt e trac
69. and the end event of the instance immediately preceeding it or zero if there was no previous instance SYNTAX state gap QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE double precision floating point SEE ALSO start time on page 9 42 end time on page 9 54 event gap on page 9 32 and state dur on page 9 57 DESCRIPTION The state dur function returns the time in seconds between the start event and the end event of the last completed instance of a state Thus if the current time line occurs within an instance of the state but before it has ended state dur returns the duration of the previous instance or zero if there was no previous instance SYNTAX state dur QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may 9 57 NightTrace Manual state_matches state_status 9 58 only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE double precision floating point SEE ALSO state_gap on page 9 57 DESCRIPTION The state_matches function returns the n
70. are as fast as the application s logging rate Increase the shared memory buffer size memsize Decrease the shared memory buffer full cutoff percentage cutoff Decrease the ntraceud timeout interval timeout e Call trace flush ortrace trigger appropriately For more information on the memsize cutoff and timeout options and trace flush see respectively Option to Define Shared Memory Buffer Size memsize on page 4 16 Option to Set the Buffer Full Cutoff Percentage cutoff on page 4 18 Option to Set Timeout Interval timeout on page 4 17 and trace flush and trace trigger on page 2 20 Optimizing File System and CPU Usage Different systems may share files via the Network File System NFS however accessing an NFS mounted file takes longer than accessing a local file You get the best NightTrace and KernelTrace performance if you avoid NFS accesses put your trace event file on the same system where both the daemons and your application run To determine whether your disk is local to your system verify that it is mounted on dev and not on another host You can do this by running the d 1 command and looking for a colon in the Filesystem column A single system may have more than one CPU Consider assigning the daemon and your application to different CPUs on the same system this way the daemos will not interfere with your application A 3 NightTrace Manual You can use t
71. asterisk next to a node in the Available Nodes list indi cates that the particular node has already been selected through the Node Selection Dialog You may select the nodes you wish to be included on the kernel display page you are building by either double clicking each node name in the Available Nodes list or by selecting a node from that list and using the right arrow button to add it to the list of Selected Nodes When the list of Selected Nodes contains all the nodes you wish to display on your kernel display page you may press the Build button As each node is added to the list of Selected Nodes ntrace figures out how much vertical real estate the grid needs based upon the number of nodes you wish to display and how many CPUs each node has If the required vertical space does not exceed the maximum grid height ntrace will allow the page to be created Otherwise ntrace will pop up a warning dialog window and will not allow the page to be created Tracing the Kernel Warning Dialog f Adding this node would exceed maximum display page height Figure 11 4 Node Selection Warning Dialog Figure 11 5 shows the display of information for a CPU on a particular node on a dynami cally built kernel page NOTE The node information is displayed only when NightTrace is con figured to use an RCIM to timestamp events Node Last Interrupt Interrupt Graph Last Exception Last Syscall Exception Graph Running Proce
72. beginning of the trace run and the ordinal trace event offset SYNTAX offset_time offset_expr PARAMETERS offset_expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE double precision floating point 9 66 Using Expressions SEE ALSO time on page 9 27 start_time on page 9 42 and end_time on page 9 54 offset_node_id DESCRIPTION The offset node id function returns the internally assigned node identifier from which the ordinal trace event offset was logged SYNTAX offset node id offset expr PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE integer SEE ALSO node id on page 9 28 start node id on page 9 43 and end node id on page 9 54 offset pid table name DESCRIPTION The offset pid table name function returns the name of the inter nally assigned NightTrace process identifier table PID table for the ordinal trace event offset SYNTAX offset pid table name offset expr PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE string 9 67 NightTrace Manual SEE ALSO pid_table_name on page 9 29 start_pid_table_name on page 9 44 and end_pid_table_name on page 9 55 offset_tid_table_name offset_node
73. being discarded in favor of new ones are not included in the Lost count Daemon Control Area The area located at the bottom of the NightTrace Session Manager contains a number of buttons which control the daemons currently selected in the Daemon Details Area Stan Stop Pause Resume Flush Display Trace Events A Figure 3 16 Daemon Control Area Start Starts execution of the daemon s currently selected in the Daemon Details Area NOTE Starting a daemon does not imply that the daemon begins to col lect events 3 20 NightTrace Session Manager Start operations are time consuming and involve possibly connecting to a target sys tem user authentication etc Once the daemon is started it is more efficient to uti lize the Pause and Resume operations which require less time and resources Stop Stops execution of the daemon s currently selected in the Daemon Details Area The connection to the target system is terminated by this operation Once the dae mon is started it may be more efficient to utilize the Pause and Resume opera tions Pause Pauses the execution of the daemon s currently selected in the Daemon Details Area NOTE When a daemon is paused incoming trace events are discarded without notice Resume Resumes execution of the daemon s currently selected in the Daemon Details Area Once resumed incoming events are placed into the daemon buffer for subsequent processing by the daemon Fl
74. caption The descriptive text within a window In the following X resource strings default values are shown where they exist The resource strings for the global window message display area dimensions and window title are Nt race GlobalWindow TextScrollbox defaultLines 20 Nt race GlobalWindow TextScrollbox defaultChars 80 Nt race GlobalWindow name NightTrace Nt race GlobalWindow geometry The resource strings for the line count of the display page message area follow Note minimumLines must be less than or equal to defaultLines and defaultLines must be less than or equal to maximumLines Nt race DisplayPage TextScrollbox defaultLines 3 Nt race DisplayPage TextScrollbox maximumLines 3 Nt race DisplayPage TextScrollbox minimumLines 3 The resource strings for grid attributes follow ntrace uses the defaultDotsHigh and defaultDotsWide attributes only for new display pages Note if defaultDotsWide is too narrow to accommodate all the display page push buttons nt race overrides this setting Nt race Grid foreground Nt race Grid background Nt race Grid font Nt race Grid defaultDotsHigh Nt race Grid defaultDotsWide 30 60 GUI Customization The resource strings for the File Selection Dialog Box width window titles push buttons and prompt strings follow A File Selection Dialog Box is the type of window ntrace uses to prompt for file names for example configuration file names to open and save Nt race F
75. configuration file can contain any number of the following definitions Macro qualified event and qualified state definitions See Chapter 9 String table definitions See String Tables on page 5 13 Format table definitions See Format Tables on page 5 17 Display page definitions 5 11 NightTrace Manual ntrace Tables 5 12 NOTE The components of a configuration file are often interrelated For example display pages may reference user defined tables nt race generates an error message if your configuration file refers to a table before you define it To avoid this problem create your configuration files so that a table definition precedes its reference s There is no similar problem for macros qualified events and qualified states If you accidentally define a macro qualified event or qualified state more than once in a configuration file ntrace flags subsequent definitions as errors If you define a string table or format table more than once in a configuration file ntrace merges the two tables if there are duplicate entries values come from the last definition Results can be unpredictable if multiple users simultaneously modify a configuration file Similarly ntrace may behave strangely if you edit your configuration file so that any of your display objects overlap You can create modify save and load configuration files from within ntrace however you must use a text editor to create and
76. current time line would be positioned either on or after the end event for a state most recent instance If the current time line is positioned within a current instance of a state then it is that instance of the state Otherwise it is the last completed instance of a state Figure 9 5 illustrates some of these concepts with a StateGraph Event Gap State Duration 1 6308s 6303 1 6310s State Gap Line Figure 9 5 Function Terminology Illustrated 9 14 Using Expressions A more detailed example is illustrated in the following figure gt gt e x Sd SON v SS SS y y e 1 6318s time line z time line y time line x Figure 9 6 States and Events The following discusses the terminology with respect to time line x time line y and time line z Assuming the current time line was positioned at time line x in Figure 9 6 the various instances would be defined as current instance No current instance is defined since the cur rent time line is not positioned within any instance of a state last completed instance Instance B most recent instance Instance B Since the current time line is not positioned within any instance of a state the most recent instance is the last completed instance 9 15 NightTrace Manual The table below indicates the information returned by various NightTrace functions assuming the current time line was positioned at time line x in
77. differently when you use it e On ntraceud start up On later ntraceud invocations If you start up nt raceud with the enable option the specified trace event s are the only one s enabled all other trace events are disabled For example if the fol lowing invocation starts up nt raceud then only trace event 18 is enabled ntraceud e18 traceout When you use the enable option on non startup nt raceud invocations Night Trace adds the specified trace event s to the list of enabled trace events Further more attempting to enable an already enabled trace event has no effect For example assume that you invoke ntraceud four times sequentially before your application terminates and that ntraceud has not logged to the nt output file before ntraceud ntoutput all trace events enabled ntraceud d4 d7 ntoutput all except 4 and 7 enabled ntraceud e4 ntoutput all except 7 enabled ntraceud e4 ntoutput no effect all except 7 enabled The following two sequences show how important timing can be when you use the enable option The same steps appear in both sequences but their order differs When the first sequence ends nothing has been logged and all trace events are enabled In contrast when the second sequence ends trace event 52 has been logged once and is now disabled Table 4 6 ntraceud Enable Sequence 1 From the Shell From the Application Comments 1 Invoke ntraceud All trace events enabl
78. disable the extraneous trace events You can disable trace events 2 17 NightTrace Manual 2 18 temporarily where you disable and later re enable them You can also disable them permanently where you disable them at the beginning of the process or at a later point and never re enable them NOTE These routines enable and disable trace events in all processes that rely on the same user daemon to log to the same trace event file All disable library routines make your application start ignoring requests to log trace event s to the shared memory buffer The disable routines differ by how many trace events they disable trace disable disables one trace event ID trace_disable_range disables a range of trace event IDs including both range endpoints trace_disable_all disables all trace events Disabling an already disabled trace event has no effect All enable library routines let you re enable a trace event that you disabled with a disable library routine or user daemon The effect is that your application resumes noticing requests to log the specified trace event to the shared memory buffer The enable routines differ by how many trace events they enable trace_enable enables one trace event ID trace enable range enables a range of trace event IDs including both range endpoints trace enable all enables all trace events Enabling an already enabled trace event has no effect TIP Consider invoking the user daemon
79. display area Each vertical line in a DataGraph represents a trace event argument If you click on a data value with mouse button 3 ntrace writes information about the data value in the message display area For information about StateGraphs EventGraphs and DataGraphs see StateGraph on page 7 14 EventGraph on page 7 15 and DataGraph on page 7 16 If your grid has a Column and you have not already positioned your interval somewhere else ntrace displays in the Column the earliest 5 percent of your trace session Usually this information is uninteresting and you want to see other parts of your trace session The following list shows the ways you can get ntrace to locate interesting parts of your trace session Scroll through the interval using the interval scroll bar Zoom in or zoom out using interval push buttons Change the parameters defining the interval by editing its fields Viewing Trace Event Logs with ntrace Use the Tools I Search menu item to search for a specific trace event or condition See Chapter 10 for more information Viewing Strategy ntrace is a flexible tool Depending on your personal preferences and how much you know about your trace events there are several ways to locate intervals of interest The following flowchart provides information to help you decide what to do next in View mode NightTrace Manual Look at the grid Is the displayed information i
80. events In this way the different trace contexts of multiple processes threads and tasks can be easily distinguished from each other For more information on threads see Programming with the Threads Library in the PowerMAX OS Programming Guide A process that results from the exec 2 system service does not inherit a trace mechanism Therefore if that process is to log trace events it must call both trace begin andtrace open thread RETURN VALUES The trace open thread routine returns a zero value NTNOERROR on successful completion Otherwise it returns a non zero value to identify the error condition A list of trace open thread error codes follows NTINIT The NightTrace library routines were not initialized or they were initialized but no user daemon has yet been initiated Ensure a trace begin call precedes this call If the pre ceding trace begin call returned NTLISTEN then a value of NTINIT is not a failure condition and once a user dae mon is started subsequent attempts at logging events will suc ceed NTINVALID An invalid thread name was specified Solution Choose a thread name that meets the requirements mentioned earlier NTRESOURCE There are not enough resources to open this thread Solution Ask your system administrator to increase the size of the pro cess table NTPGLOCK Permission to lock the text and data pages of the NightTrace library routines was denied If
81. events or pre defined kernel events Timestamp Source Selection By default an architecture specific timing source is utilized For Intel based machines the Intel Time Stamp Counter TSC register for Night Hawk 6000 series machine the interval timer for PowerHawk and PowerStack series machines the Time Base Register TBR However the Real Time Clock and Interrupt Module RCIM can be also used as a timestamp source The RCIM is an optional hardware module attached to a single board computer SBC which contains a tick clock that can be synchronized between several SBCs by way of an interconnection cable This synchronized tick clock can be used as a common time base for both kernel level tracing and user level tracing across multiple SBCs NightTrace sup ports using the RCIM synchronized tick clock to timestamp trace events and also supports displaying trace data generated on multiple SBCs having the common time base The RCIM also contains a POSIX clock However the POSIX clock is not supported as a timestamp source by NightTrace For more information about the RCIM please see the clock_synchronize 1M rcim 7 rcimconfig 1M and sync_clock 7 man pages Trace Point Placement 1 2 A trace point is a place of interest in the source code At each user trace point you make your application log some user specified information along with a timestamp and some additional system information This logged information is collective
82. in your configuration file string table fruit item 3 apple item 4 orange item 5 cherry item 6 banana default item Unknown y Assume that you make the following call in the Then Expression of a DataBox get_item fruit orange In this example the fruit string table associates specific numeric codes with a corresponding fruit name string it associates all other numeric codes with the string Unknown When NightTrace evaluates the Then Expression of this DataBox it 1 Calls get item 2 Uses the string orange as an index into the fruit string table 3 Retrieves the first associated index 4 4 Returns the index number 4 RETURN TYPES On successful completion get item returns an item number from a string table If several item lines within the string table have the same string value as str const get item returns the first item number from one of these item lines If table name is not found NightTrace issues an error message and the results are undefined If str const is not found and there is no default item line the results are undefined Because get item returns an integer you can use it anywhere an integer expression can be used For more information on string tables see String Tables on page 5 13 and the usr lib NightTrace tables file For more examples of function calls with pre defined string tables see Table 8 3 9 78 Using Expressio
83. information about interrupts are also red By default the word Exception is green and all display objects that display information about exceptions are also green By default the word Syscall is blue and all display objects that display information about syscalls are also blue The default colors of the different groups of kernel objects can be controlled with X resources The colors are specified on a per CPU basis The default resources for logical CPU 0 are Ntrace Color GridObject interruptO foreground red Nt race Color GridObject exception0 foreground green Nt race Color GridObject syscall0 foreground blue See Appendix B for more information on X resources Kernel String Tables 11 16 There are seven kernel related pre defined string tables They are vector syscall device name_pid node_name pid_nodename Tracing the Kernel This string table contains the interrupt and exception vector names associated with the system that the kernel tracing was performed on It is contained in the vectors file This table is indexed by an exception interrupt vector number or an exception interrupt vector name Examples of using this table are get_string vector arg3 get_string vector 15 get_item vector ncr_intr This string table contains the names of all the possible syscalls that can occur on the system It is contained in the vectors file For brief descriptions of the en
84. initialized global data mean ing these are reserved physical pages Performance Tuning Ensuring Accurate Timings If you lack the privilege to lock your pages in memory P_PLOCK you must invoke ntraceud with the 1ockdisable option If your application lacks read and write privilege to dev spl you must invoke ntraceud with the ipldisable option Invoking ntraceud with either the Lockdisable or ipldisable option may introduce delays and waiting within your application Use the lockdisable and ipldisable options only when necessary For more information on the lockdisable option see Option to Prevent Page Locking lockdisable on page 4 11 For more information on the ipldisable option see Option to Disable the IPL Register ipldisable on page 4 9 By default ntraceud and NightTrace library routines use page locking to prevent page faults during trace event logging NightTrace also modifies the interrupt priority level IPL register this action prevents rescheduling and interrupts during trace event logging NightTrace prevents the operating system from pre empting your trace event logging application to make itself most unobtrusive to your application If the application must wake the ntraceud daemon unexpectedly overhead can cause trace event timings to be distorted Do one or more of the following to increase the likelihood that the daemon will be awake when needed and to make sure that disk write rates
85. interesting Making argl 0x1234 amp amp arg2 arg3 gt 700 a macro would help ensure that you did not type the expression wrong in one of the fields and it would allow you to change the expressions easily even while viewing the trace run in View mode You can leave Macro Configuration Forms up while in View mode Another good use for a macro is for focusing many display objects on a specific process group For example if a Column contained several EventGraphs each of which had the following If Expression If Expression process name task 9 7 NightTrace Manual then a task macro definition of foobar would cause all of the EventGraphs to show only trace events logged by process foobar Changing the macro to bazonk would shift the focus of the EventGraphs from process foobar to process bazonk This technique can also be used in DataBoxes DataGraphs and State Graphs Functions Using Expressions Functions are pre defined NightTrace entities that may be used in an expression Night Trace defines five classes of functions Trace event functions see Trace Event Functions on page 9 19 State functions see State Functions on page 9 34 Offset functions see Offset Functions on page 9 59 e Summary functions see Summary Functions on page 9 70 Format and table functions see Format and Table Functions on page 9 75 The general syntax of al
86. memory buffer In either case trace event IDs are integers in the range 0 4095 inclusive At defined times ntraceud copies trace events from the shared memory buffer to the trace event file trace file NOTE The enable option affects all processes that rely on the same ntraceud daemon to log to the same trace event file The first format provides the same functionality as the trace enable Night Trace library routine The second format provides the same functionality as the trace enable range NightTrace library routine One advantage of using the ntraceud option instead of the library routine is that you do not have to re edit recompile and relink your application For more information on enable library routines see trace enable trace disable and Their Variants on page 2 16 In the following text the names of the trace event files are varied for interest Unless otherwise stated all the following examples describe the results of a non startup nt raceud invocation There may be any number of enable e options on an ntraceud invocation line The following example illustrates this fact ntraceud e10 e15 mytrace trace events 10 and 15 enabled You may specify a hyphenated trace event range on the ntraceud invocation line The following example depicts this case ntraceud e23 25 traceoutput trace events 23 24 amp 25 enabled Generating Trace Event Logs with ntraceud The enable option acts
87. modify tables in a configuration file ntrace ignores blank lines and treats text between a and a as comments in configuration files however saving a configuration file removes your comments To load an existing configuration file do one of the following Start an ntrace session with the file name as an argument Click on File 1 Open Config File on the Global Window For more information on manipulating configuration files see NightTrace Menu Item on page 5 26 The configuration file may contain two types of tables string tables and format tables Both types of table can improve the readability of your ntrace displays The only way to put tables into your configuration file is by text editing the file before you invoke ntrace To avoid any forward reference problems define all string tables before any format tables A table lets you associate meaningful character strings with the more cryptic integer values such as trace event arguments These character strings may appear in ntrace displays For example suppose that one trace event s argument may have the decimal values 2 5 and 8 You can define e A string that appears when the value is 2 String Tables Invoking the ntrace Display Utility A different string that appears when the value is 5 A third string that appears when the value is 8 The following table names are reserved in nt race and should not be redefined in upper case or lower case
88. my event Operates on the qualified event my event end argl Operates on the last completed instance of the state being defined and can only appear within a state definition end_argl my_state Operates on the last completed instance of the qualified state my_state 9 12 Using Expressions This manual uses the following conventions for function parameters OE OS offset_expr expr table_name int_expr str_const format_string arg A user defined qualified event If supplied the function applies to the specified qualified event For more information see Qualified Events on page 9 81 A user defined qualified state If supplied the function applies to the specified qualified state For more information see Qualified States on page 9 83 An expression that evaluates to the offset or ordinal trace event number of a trace event Any valid NightTrace expression see Expressions on page 9 4 An unquoted character string that represents the name of a string table or format table An integer expression that acts as an index into the specified string table or format table int_expr must either match an identifying integer value in the table_name table or the table_name table must have a default item line A string constant literal that acts as an index into the specified string table A character string that contains literal characters and conversion specifications Conversion specifi
89. names do not have to exactly match textually but they do have to refer to the same actual pathname for example one path name may begin at your current working directory and the other may begin at the root directory For C and Fortran either a NULL pointer in which case the default settings are used or a pointer to a ntconfig t structure For Ada the individual members of the structure are supplied directly as parameters to the routine with appropriate default values Both the user application and the user daemon associated with it must agree 2 5 NightTrace Manual ntc_buffer_size ntc_use_spl on the configuration settings or indicate that the other s settings may be preferred The size of the shared memory buffer in units of trace events The user daemon default size is 1024 16 events The size must be greater or equal to 4096 Specifies that SPL preemption control will be used see spl_request 3C This feature is not available on RedHawk Linux systems and is automatically translated to ntc_use_resched The user daemon default value is TRUE for PowerMAX OS and FALSE for RedHawk Linux ntc_use_resched ntc_lock_pages ntc_clock Specifies that rescheduling variable preemption control will be used See resched_cntl 2 The user daemon default value is FALSE for PowerMAX OS and TRUE for RedHawk Linux Specifies that critical pages will be locked in memory The user dae mon default value is TRUE
90. of 5678 3 would also have been displayed by the StateGraph since the thread dae mon also existed with a TID of 56787 3 If the trace event file has multiple threads with the same name for example CHILD THREAD specifying any one of the TIDs with that thread name selects all of the TIDs with that thread name To avoid this it is recommended that all threads be given unique names If that is not possible you can isolate individual threads by including a TID restriction in the If Expression parameter For example if CHILD THREAD includes TIDs 100 1 200 1 and 300 1 and you want information only on TID 100 1 set the TID List parameter to CHILD THREAD and the If Expression to tid 100 1 For more information on thread names see trace open thread on page 2 9 For more information about the t id function see tid on page 9 25 Node List Configuring Display Objects When NightTrace processes a trace file which was timestamped by an RCIM synchro nized tick clock it internally assigns a node identifier to each node host name represented by a trace file If no trace file was generated using the tick clock this parameter is not dis played The Node List parameter is the list of node identifiers or node names from which the dis play object will accept trace events If the trace event did not occur on a node listed in this parameter the trace event is ignored If a number or name is specified that is not a valid
91. on RedHawk systems If selected for RedHawk systems it is silently translated to the Rescheduling Variables protection as described below Rescheduling Variables This level of protection is sufficient for user applications that log trace events as long as no user level interrupt handlers will be logging trace events None Selecting no protection opens up the real possibility that the user process or the daemon could preempt each other if they are allowed to operate on the same CPU However the Runtime subpage allows the user to define the CPU binding CPU Bias and priority Priority at which the daemon operates See Runtime on page 3 41 for a description of these settings Thus if the user also takes similar care to schedule their user applications then selecting None is sufficient Page Critical Locking Page locking is required to prevent preemption while holding a spin lock Without this choice it is possible that a page fault could occur while a spin lock is held allowing for a user application or daemon to spin forever Lock Pages When this option is selected the daemon and user applications associated with this daemon lock down the required pages and unlock them when the NightTrace API is terminated 3 35 NightTrace Manual Shared Memory Inheritance When the daemon starts up certain settings can be inherited from a running user application that has set up a trace definition The NightTrace API tr
92. on page 9 62 Using Expressions end_thread_id DESCRIPTION The end_thread_id function returns the thread identifier associated with the end event of the last completed instance of a state NOTE See the thr_self 3thread man page for more information SYNTAX end thread id QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO thread id on page 9 24 start thread id on page 9 39 and offset thread id on page 9 64 end task id DESCRIPTION The end task id function returns the Ada task identifier associated with the end event of the last completed instance of a state NOTE This function is only meaningful for trace events logged by Ada tasking programs SYNTAX end task id QS 9 51 NightTrace Manual PARAMETERS OS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO task_id on page 9 25 start task id on page 9 40 and offset_task_idQ on page 9 64 end tid DESCRIPTION The
93. on the grid or by the interval control area fields The right arrowhead represents the end of the entire trace session If you have not already positioned your interval somewhere else the movable slider of the interval scroll bar is adjacent to the scroll bar s left arrowhead When the slider is here the Time Start statistic in the interval control area is 0 0000000 seconds The length of the slider is proportionate to the amount of the trace session displayed in the interval By default a display page shows 5 of a trace session In the following interval scroll bar descriptions the fields in the interval control area that are affected by the interval scroll bar include Current Time Time Start Time End Event Start Event End and Increment For more information on these fields see The Interval Control Area on page 6 11 Trough Slider Left Arrowhead Right Arrowhead Figure 6 4 The Interval Scroll Bar Manipulating the interval scroll bar in the following ways has the following results 6 7 NightTrace Manual Table 6 2 Manipulating the Interval Scroll Bar Mouse Action Location Result Button Click Any Left If the interval scroll bar slider is not already at the leftmost position arrowhead Moves the slider to the left Scrolls backward Increment seconds or Increment percent of the current display interval Click Any Right If the interval scroll bar slider is not already at the rightmost position arr
94. otherwise ntraceud overwrites it Unless your umask 1 setting overrides this default ntraceud creates the file with mode 666 read and write permission to all users If you want to maximize performance use a trace event file that is local to the system where the ntraceud daemon and your application run 4 1 NightTrace Manual A single ntraceud daemon may service several running applications or processes Several ntraceud daemons can run simultaneously the system identifies them by their distinctive trace event file names The ntraceud daemon resides on your system under usr bin ntraceud Whenever the daemon is idle it sleeps You can control the sleep interval with an ntraceud option Logging a trace event may wake the daemon if the buffer full cutoff percentage is exceeded or if shared memory becomes full of trace events Flushing trace events from the shared memory buffer to disk always wakes the daemon The Default User Daemon Configuration 4 2 Invoking ntraceud with a trace event file argument and without any options will attempt to start a user daemon with the default user daemon configuration You can over ride defaults by invoking nt raceud with particular options Table 4 1 summarizes these options Later sections provide detailed descriptions of these options and operating modes However if a user application has already been initiated it may have specified a non default configuration via the trace_begin call If the c
95. particular daemon is running This dialog is opened by selecting a currently executing daemon from the Daemon Details Area and pressing the Trace Events button in the Daemon Control Area of the NightTrace Ses sion Manager see Daemon Control Area on page 3 20 Figure 3 17 Enable Disable Trace Events dialog 3 22 NightTrace Session Manager Disabled Events This is a list of user trace or kernel trace event types that are disabled Disabled events are not logged to daemon buffers and therefore are not included in event trace outputs Enabled Events This is a list of user trace or kernel trace event types that are enabled Enabled events are allowed to be placed into daemon buffers and are subsequently transferred to the output device see Trace Events Output on page 3 31 Enable gt Moves the selected items from the Disabled Events list or the Trace Event field to the Enabled Events list lt Disable Moves the selected items from the Enabled Events list or the Trace Event field to the Disabled Events list Trace Event Allows the user to enter a particular trace event type or range of trace event types and subsegently Enable gt or Disable gt it The user may use the event name associated with the event type e g TR_SYSCALL_RESUME or the numerical value of the trace event type e g 4131 The user may also enter a range of values either using the event names or their numerical valu
96. pre empt a daemon or user process in the middle of a critical section and then spin forever waiting for it to be unlocked The spin locks are held for extremely short periods of time Failure to properly select a protection level may result in a process spinning forever on a CPU in the event of unfortunate preemption Generating Trace Event Logs with ntraceud In more severe cases the system may hang this is only a problem if a user level interrupt preempts another user process or daemon at an unfortunate time and attempts to log trace events to the same trace daemon session Data corruption of the trace data will not occur in any case lt default option gt IPL protection is the safest form of preemption control as it prevents even machine interrupts from preempting the locking process This is required when a user application will be logging events in a user level interrupt han dler i e at system interrupt level This is the default preemption control mechanism for PowerMAX OS sys tems NOTE This mechanism is not available on RedHawk systems If selected for RedHawk systems it is silently translated to the resched protection as described below resched This level of protection is sufficient for user applications that log trace events as long as no user level interrupt handlers will be logging trace events This is the default preemption control setting for RedHawk Linux ipldisable Selecting no protection o
97. precision floating point SEE ALSO event_gap on page 9 32 start_time on page 9 42 end_time on page 9 54 state_gap on page 9 57 state_dur on page 9 57 and offset_time on page 9 66 node_id DESCRIPTION The node id function returns the internally assigned node identifier associated with a trace event NOTE The node id function is of limited usefulness since the node identifier is an internally assigned integer number assigned by NightTrace The node name function is more useful as it returns the name of the system from which a trace event was logged See node name on page 9 30 for more information about this function SYNTAX node id QE PARAMETERS QE RETURN TYPE integer 9 28 A user defined qualified event If supplied the function returns the node identifier of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the node identifier of the current trace event For more information see Qualified Events on page 9 81 pid_table_name tid_table_name Using Expressions SEE ALSO start_node_id on page 9 43 offset_node_id on page 9 67 and end_node_id on page 9 54 DESCRIPTION The pid_table_name function returns the name of the internally assigned NightTrace process identifier table PID table associated with a
98. raceud daemon by its trace event file name trace file By default NightTrace locks its pages in memory This capability prevents page faults during trace event logging that could distort trace event timings If you lack sufficient privileges required to lock your pages in memory your invocation of ntraceud and your application exit with errors If you still want to trace events you must invoke the ntraceud daemon with the lockdisable option This option makes ntraceud and the NightTrace library routines in your application run without locking their pages in memory If you use the 1ockdis able option you must start up ntraceud with it You must not use the 1ockdisable option if your user level interrupt routine logs trace events to the shared memory buffer 4 11 NightTrace Manual Option to Establish File Wraparound Mode filewrap 4 12 By default the trace event file can grow indefinitely With the ntraceud filewrap option you can make NightTrace operate in file wraparound mode rather than expansive mode In file wraparound mode you limit the trace event file size SYNTAX ntraceud filewrap bytes trace file DESCRIPTION The ntraceud filewrap option lets you specify the maximum byte size bytes of the trace event file trace file Specify the bytes parameter as a number of bytes or as a number with a K or M suffix to show that the bytes parameter is in kilobyte or megabyte units respectively For example 12K m
99. returned for a process ID unless the process name is not unique on that partic ular node These tables are indexed by a process identifier or a process name Examples of using these tables are 11 17 NightTrace Manual get_string pid_hal pid get_item pid_simulator odyssey syscall_nodenameThis string table contains the names of all possible system calls that can occur in trace event files for node name nodename It is con tained in the vectors file This table is indexed by a system call number or a system call name Examples of using this table are get_string syscall_systemx 31 get_string syscall_systemy arg2 get_item syscall_systemz read vector_nodename This string table contains the interrupt and exception vector names associated with trace event files for node name nodename It is con tained in the vectors file This table is indexed by an exception interrupt vector number or an exception interrupt vector name Examples of using this table are get string vector machinel arg3 get string vector machine2 585 get item vector system3 data access device nodename This string table contains the names of devices configured in the ker nel for trace event files from node name nodename It is contained in the vectors file This table is indexed by a device number or a device name Exam ples of using this table are get string device simulatorl
100. settings do not affect the memory associated with the trace buffers Kernel trace buffers are in kernel memory allocated out of the global memoy pool and user trace buffers are in shared memory allocated out of the global memory pool Text NUMA Flag This item selects the NUMA policy to apply to text code pages Text pages are those pages in private mappings that belong to a file in a file system The traditional text segment falls into this category See Policies in the section below for a list of applicable NUMA policies Private Data NUMA Flag This item selects the NUMA policy to apply to private data pages Private data pages are those pages in private mappings that do not belong to a file in a file system The traditional stack and data segments fall into this category Note that the first time that a process writes to a page in a private writable mapping to a file the page will move from the text category to the private data category See Policies in the section below for a list of applicable NUMA policies Shared Data NUMA Flag This item selects the NUMA policy to apply to shared data pages See Policies in the section below for a list of applicable NUMA policies U block NUMA Flag This item selects the NUMA policy to apply to kernel data structures that contain the stack used during system calls the register save area used during context switches and miscellaneous other bits of information about the LWP
101. solely of alphanumeric characters and underscores Keep trace event tags short other wise ntrace may be unable to display them in the limited window space available The following words are reserved in ntrace and should not be used in upper case or lower case as trace event tags NONE ALL ALLUSER ALLKERNEL TRUE FALSE CALC TIP Consider giving your trace events upper case tags in event map files and giving any corresponding qualified event the same name in lower case For more information about qualified events see Qualified Events on page 9 81 If your application logs a trace event with one or more numeric arguments by default ntrace displays these arguments in decimal integer format To override this default provide a count of argument values and one argument conversion specification or display format per argument nargs The number of arguments associated with a particular trace event If nargs is too small and you invoke ntrace with the event map file and the 1isting option ntrace shows only nargs arguments for the trace event Invoking the ntrace Display Utility conv_spec A conversion specification or display format for a trace event argument nt race uses conversion specification s to display the trace event s argument s in the designated format s There must be one conversion specification per argument Valid con version specifications for displays include the following S dsi
102. specified permissions Timestamp Heartbeat 3 36 For performance reasons NightTrace events normally include only the low 32 bits of a full 64 bit timestamp The heartbeat ensures that the daemon logs a full 64 bit timestamp before the interval of time represented by 32 bits expires The daemon automatically cal culates the heartbeat time when it determines how fast the timing source clock ticks User Event Buffer Events NightTrace Session Manager Log Heartbeat The frequency at which a full 64 bit timestamp will be generated NOTE There would be no particular benefit by setting the heartbeat to a value shorter than the automatically calcuated time unless trace time anomolies are seen because the daemon is preempted by higher priority processing and cannot otherwise log the heartbeat in time Buffer Size The number of events that can be held in memory before being written to the output device Flush Threshold The Flush Threshold expressed as a percentage of the Buffer Size is the point at which the daemon begins to transfer events from the user event buffer to the out put device see Trace Events Output on page 3 31 The threshold is important so as not to lose events if the daemon cannot respond quickly enough NOTE If events are being lost a combination of changing the Buffer Size the Flush Threshold and the runtime Priority see Pri ority on page 3 42 of the daemon is normally sufficient to p
103. specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO offset on page 9 27 and end offset on page 9 53 DESCRIPTION The start time function returns the time in seconds associated with the start event of the most recent instance of a state Times are relative to the earliest trace event from all trace data files currently in use SYNTAX start time QS start_node_id PARAMETERS QS RETURN TYPE Using Expressions A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 double precision floating point SEE ALSO time on page 9 27 end time on page 9 54 state gap on page 9 57 state dur on page 9 57 and offset time on page 9 66 DESCRIPTION The start node id function returns the internally assigned node identifier associated with the start event of the most recent instance of a state SYNTAX start node id QS PARAMETERS QS RETURN TYPE integer SEE ALSO A user defined qualified state If supplied it specifies the state to which the function applies If omitted the fun
104. summary and a StateGraph is that a state summary shows information textually and a StateGraph shows it graphically By default the Summary Expression for a state summary type displays one line for each of the following in the summary display area Using the Built In Tools The range of ordinal trace event numbers offsets summarized The number of matches summarized The minimum time gap between matches and the ordinal trace event number offset where it began The maximum time gap between matches and the ordinal trace event number offset where it began The average time gap between matches The sum of the time gaps between matches The minimum time duration of a match and the ordinal trace event number offset where it began The maximum time duration of a match and the ordinal trace event number offset where it began The average time duration of a match The sum of the time durations of matches Figure 10 5 shows a state summary 10 13 NightTrace Manual Summarizing from offset 0 thru offset 10915 3876 state 1 2 3 4 5 6 7 8 3 10 11 Figure 10 5 The State Summarize Form NOTE The Node List field appears in this dialog only when NightTrace is configured to use an RCIM to timestamp events Exercise Using the Search Tool The following exercise has you search for trace events you logged in Exercise Logging Trace Events on page 4 29 while using files you created in Exerc
105. table associated with a trace event node name QE start node name QS end node name QS offset node name offset expr The string describing the name of the sys tem from which a trace event was logged process name QE offset process name offset expr The string describing the name of the pro cess PID associated with a trace event task name QE offset task name offset expr The string describing the name of the Ada task associated with a trace event thread name QE offset thread name offset expr The string describing the name of the C thread associated with a trace event event gap QE state gap QS The double precision floating point time expresed in units of seconds between the instances of either a trace event or a state state dur QS The double precision floating point time expressed in units of seconds of an instance of a state event matches QE state matches QS summary matches 0 The integer number of instances of either a trace event or a state state status QS The boolean status of a state true if the cur rent time line is within an instance of the state false otherwise See state status on page 9 58 for important details 9 11 NightTrace Manual Table 9 3 NightTrace Functions Syntax Return Type offset QE The integer ordinal number offset of a
106. target system It does not necessarily need to be accessible from the host system the system where the NightTrace GUI is running however that can be convienient for subsequent analysis via NightTrace NOTE When the output method is NOT streaming see Stream above the Key File see Key File on page 3 31 defines the name of the Output File Browse Brings up a standard file selection dialog so that the user may navigate to the desired location of the Output File In order to browse the Target System see Target System on page 3 29 must be operational The file selection dialog invoked by that button shows files relative to the Target System File Wrap When checked allows the user to specify the Maximum File Size for the Key File Output File Maximum File Size The maximum number of bytes for the Key File Output File When the Maximum File Size is reached subsequent events will overwrite the oldest events NightTrace automatically detects this and presents events in chronological order from oldest to newest Events that are discarded due to File Wrap are NOT considered lost events see Lost on page 3 20 in statistics provided by the NightTrace Session Manager or the NightTrace ana lyzer NOTE For a daemon capturing kernel trace events the file wrap sizes that the user specifies are rounded up to a multiple of kernel buffer sizes On PowerMAX OS systems a kernel trace buffer has a fixed s
107. the contents are lines or shapes like a bar chart Display objects can be scrollable or non scrollable Scrollable means the display object acts as a movable window into the trace event file The basic types of display objects you can create are listed below and discussed in the fol lowing sections GridLabel Static textual display object that contains a user specified string of text and is used to label other display objects for clarity DataBox Dynamic display object that displays textual information such as the trace event tag or the time the trace event occurred Its main use is to display data that does not lend itself to graphical representation Column Dynamic display object that does not display data itself but holds the scrollable graphical display objects StateGraphs EventGraphs Dat aGraphs and Rulers Its purpose is to group together related graphi cal display objects It is the only non configurable display object StateGraph Dynamic scrollable graphical display object that displays a state as a bar and other trace events as a vertical line It indicates the states and trace events relative positions in time since the trace started This display object is usually used if you want to know when the application enters and exits a particular user defined state EventGraph Dynamic scrollable graphical display object that displays a trace event as a vertical line and indicates its relative position in time since
108. the result of the get_string call and replaces the second conversion specification Sd with the integer result of the numeric expression offset If arg 1 has the value 99 and offset has the value 10 then ntrace displays 5 19 NightTrace Manual Current state is Terminating current trace event is 10 For all other trace events ntrace displays Other For more information on get_string see get string on page 9 75 For more information on format tables and the get_format function see get format on page 9 79 See also the usr lib NightTrace tables file For more information about argl see arg on page 9 20 For more information about offset see offset on page 9 27 Pre Defined Format Tables The following format tables are pre defined in the usr lib NightTrace tables file state_summary Formats statistics about the state matches summarized state durations and state time gaps This table provides the default state summary output format event_summary Formats statistics about the trace event matches and trace event time gaps This table provides the default trace event summary output format event_arg_summary Formats statistics about the trace event matches and their type long trace event arguments event_arg_dbl_summary Formats statistics about the trace event matches and their type double trace event arguments For more information about summaries see Chapter 10 You
109. the top of the DataGraph Figure 8 11 Maximum vs Minimum Values Configuring Display Objects See DataGraph on page 7 16 for more information See Common Configuration Parameters on page 8 1 for descriptions of the common configuration parameters that DataGraphs use Ruler unnamed object J Ruler Junnamed_obj ect Lost Event Color Mark Color Foreground Color Background Color Font Figure 8 12 Ruler Configuration Form The configuration form for the Ruler is shown in Figure 8 12 The Lost Event Color parameter specifies the color of the reverse video L shown in Figure 8 13 that is placed on a Ruler where NightTrace lost data The possible values are the colors your X server supports as specified in the rgb txt file See Preventing Trace Events Loss on page A 1 for more information on lost data The Mark Color parameter specifies the color of the mark indicator a triangle that appears on the Ruler shown in Figure 8 13 The possible values are the colors your X server supports See The Interval Push Buttons on page 6 8 for more information about the mark 0 06621s 1 0 06624s Figure 8 13 Mark and Lost Event Markers See Ruler on page 7 17 for more information See Common Configuration Parame ters on page 8 1 for descriptions of the common configuration parameters that Rulers use 8 19 NightTrace Manual 8 20 9 Using Expressions ON 9 1
110. tite opea brane s equ see dcs ess 11 21 NightTrace Manual Overview 11 Tracing the Kernel This chapter provides a description of the trace points logged by the kernel It also discusses the steps required to produce a highly detailed picture of kernel activity with NightTrace This lets you customize the default nt race kernel display pages or combine kernel information with user application trace information Kernel trace event files are logged in raw format by the kernel trace daemon ntrace accepts files of this type as arguments When it detects such a file on the command line or when the session manager indicates such a file should be displayed it automatically fil ters the raw data file and creates two new files The first file created is the filtered data which contains trace events in a manner suitable for display within NightTrace This file is saved with a pathname constructed from the original raw kernel trace event filename with a nt suffix appended to it The second file saved is commonly referred to as the vectors file It contains tables that are specific to the actual raw data The vectors file is saved with a pathname constructed from the orignal raw kernel trace event filename with a vec suffix appended to it A more detailed description of the vectors file is given subsequently in this chapter On subsequent invocations of nt race either the raw kernel file may be specified or alternativel
111. trace event SYNTAX min expr PARAMETERS expr A numeric expression RETURN TYPE data type of expr SEE ALSO Summary Functions on page 9 70 and Summarizing Statistical Information on page 10 5 DESCRIPTION The max function returns the maximum value of all occurrences of expr within a time range When used ina Summarize Form the time range is defined by that form When used elsewhere the time range is defined as the region starting with the first trace event and ending with the current trace event SYNTAX max expr PARAMETERS expr A numeric expression RETURN TYPE data type of expr 9 71 NightTrace Manual avg sum 9 72 SEE ALSO Summary Functions on page 9 70 and Summarizing Statistical Information on page 10 5 DESCRIPTION The avg function returns the average value of all occurrences of expr within a time range When used ina Summarize Form the time range is defined by that form When used elsewhere the time range is defined as the region starting with the first trace event and ending with the current trace event SYNTAX avg expr PARAMETERS expr A numeric expression RETURN TYPE data type of expr SEE ALSO Summary Functions on page 9 70 and Summarizing Statistical Information on page 10 5 DESCRIPTION The sum function returns the sum value of all occurrences of expr within a time range When used ina Summarize
112. user daemon Click on the Stream checkbox to ensure it is now unchecked Enter the filename passed to the trace_begin routine in the text field for the Key File Click the OK button Start the user daemon by clicking on the Start button in the Session Man ager window Once the state for the daemon changes to Paused click the Resume button Run your application As NightTrace trace event logging routines execute they write trace event information into a shared memory buffer Periodi cally the user daemon copies this information to a trace event file on disk For example a out When the application completes or when you have captured sufficient data that you now wish to analyze stop the daemon in the session manager by pressing the Flush button followed by the Stop button To display the data press the Display button in the session manager The Kernel Trace Event Logging Procedure Alternatively to log and view kernel data invoke the nt race command and follow these steps ntrace mgr amp 1 Define a kernel daemon in the NightTrace Session Manager window by selecting the Daemon gt New menu item which brings up a Daemons Definition dialog Click on the Kernel radio button to define this as a kernel daemon Click on the Stream checkbox to ensure it is now unchecked Enter an output filename such as tmp kernel data in the text field for Output File Click the OK button Start the kernel daemon by cli
113. user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 end_pid_table_name end_tid_table_name Using Expressions RETURN TYPE integer SEE ALSO node_id on page 9 28 start_node_id on page 9 43 and offset_node_id on page 9 67 DESCRIPTION The end pid table name function returns the name of the inter nally assigned NightTrace process identifier table PID table associated with the end event of the last completed instance of a state SYNTAX end pid table name QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE string SEE ALSO pid table name on page 9 29 start pid table name on page 9 44 and offset pid table name on page 9 67 DESCRIPTION The end tid table name function returns the name of the inter nally assigned NightTrace thread identifier table TID table associated with the end event of the last completed instance of a state SYNTAX end tid table name OS 9 55 NightTrace Manual PARAMETERS OS A user defin
114. with events disabled instead of calling the trace enable and trace disable routines Using these options saves you from re editing recompiling and relinking your application TIP If you want to log only a few of your trace events disable all trace events with trace disable all and then selectively enable the trace events of interest RETURN VALUES The trace disable trace disable range trace disable all trace enable trace enable range and trace enable all routines return a zero value NTNOERROR on suc cessful completion Otherwise they return a non zero value to identify the error condition A list of error codes for these routines follows NTINIT The NightTrace library routines were not initialized Solution Be sure a trace begin and trace open thread call precede the call to the disable or enable routine NTINVALID An invalid trace event ID has been supplied Solution Use trace event IDs only in the range 0 4095 inclusive Adding Library Calls to Your Application SEE ALSO Related routines include trace event trace event arg trace event dbl trace event four arg 2 19 NightTrace Manual trace_flush and trace_trigger The trace flush andtrace trigger routines asynchronously wake the user and direct it to copy trace events from the shared memory buffer to the trace event file on disk Note These routines do not wait for the copy to complete
115. 0 7 8 7 9 7 16 8 5 9 20 9 21 9 35 9 36 9 37 9 47 9 48 9 60 9 61 10 9 average size 4 16 context switch 11 2 disabling 2 17 4 4 4 24 discarding 2 21 A 1 A 4 C 1 display utility 5 1 enabling 2 17 4 4 4 26 exception 11 3 file 1 5 2 5 4 1 5 9 file size 4 12 4 13 A 4 functions 9 19 ID 1 2 2 13 2 17 4 24 4 26 5 4 5 9 5 10 5 14 5 29 5 31 8 2 8 4 8 5 10 4 10 6 C 1 information 7 15 9 19 interrupt 11 2 loading 5 7 A 5 logging 1 3 1 4 4 12 4 13 4 24 4 26 A 4 C 1 loss 2 15 2 21 4 16 4 28 7 17 8 19 A 1 C 1 NightTrace kernel file 5 5 5 7 5 27 node identifer ending trace event 9 54 node identifer offset 9 67 node identifer starting trace event 9 43 node identifier 9 28 node name 9 30 Index node name ending trace event 9 56 node name ordinal trace event 9 68 node name starting trace event 9 45 NT_ASSOC_PID 6 4 NT_ASSOC_TID 6 4 NT_CONTINUE 2 9 2 13 4 16 5 6 offset 9 59 offset see Offset ordinal 9 67 9 68 9 69 9 70 ordinal number see Offset PID table name 9 29 process identifer ending trace event 9 55 process identifer offset 9 67 process identifer starting trace event 9 44 process identifier table name 9 29 process name 9 30 process name ordinal trace event 9 69 searching 1 3 1 6 6 1 6 5 10 1 10 1 summarizing 1 3 10 5 syscall 11 4 tag 5 4 5 9 5 9 5 10 5 14 5 29 5 31 10 4 10 6 11 6 task name 9 31 task name ordinal tra
116. 0 2 Do Not Move Current Time 10 3 Edit mode 7 3 7 18 Event 10 6 Forward 10 2 Global Search 10 3 Interval 10 6 Interval Manipulation 10 3 Interval Search 10 3 Region 10 6 Scroll Current Time to Event 10 3 Search Constraints 10 2 Search Direction 10 2 State 10 6 Summary Range 10 6 Summary Type 10 6 Text Gravity 8 4 Text Justify 8 3 Trace Event File 10 6 View mode 7 3 Zoom to Include Event 10 3 Raw PID 8 7 8 8 raw PID 9 10 9 23 9 38 9 49 9 63 raw_pid function 9 23 Read Event Map File menu item 5 10 5 31 5 36 Read push button 5 31 Record see Trace event Refresh push button 6 10 Region radio button 10 6 Reset push button 6 9 8 11 9 3 10 3 10 8 Resize mouse operation 7 7 Resizing display objects 5 22 7 5 7 7 windows 5 24 5 26 Restore push button 8 11 9 3 10 9 Return values 2 1 rgb txt file 5 8 8 3 8 15 8 19 B 1 B 5 Index 12 Ruler 5 30 6 9 7 17 8 19 A 1 B 4 configuration form 8 19 Lost Event Color configuration parameter 8 19 Mark Color configuration parameter 8 19 run 1 command A 4 Running process box 11 10 S Save As menu item 7 19 Save menu item 7 18 Save Text As menu item 10 10 Save Text menu item 10 10 Scroll bar 5 23 6 7 C 1 Scroll Current Time to Event radio button 10 3 Scroll region 5 23 Search menu item 10 1 Search Constraints radio buttons 10 2 Search Direction radio buttons 10 2 Search form 9 4 10 2 B 3 C 1 default search behavior 10 4 Search form push butt
117. 1 For general information on field editing see Field Editing on page 6 16 Table 6 3 Valid Multiple Field Changes Fields Result Time Start The new interval starts at Time Start and ends at Time End Time End Time Start The new interval starts at Time Start and has a length of the speci Time Length fied Time Length Time Length The new interval ends at Time End and has a length of the specified Time End Time Length Event Start The new interval starts at ordinal trace event number offset Event Event End Start and ends at ordinal trace event number offset Event End Event Start Event Count Event Count Event End Time Start Event Count Time End Event Count Event Start Time Length Event End Time Length The new interval starts at ordinal trace event number offset Event Start and includes the specified quantity of trace events The new interval ends at ordinal trace event number offset Event End and includes the specified quantity of trace events The new interval starts at Time Start and includes the specified quantity of trace events unless the Time Length forces Time Start to change The new interval ends at Time End and includes the specified quantity of trace events unless the Time Length forces Time End to change The new interval starts at ordinal trace event number offset Event Start and has a length of the specified Time Length unless the Time Length forces Event Start to change
118. 1 table and its corresponding system call name is displayed This is meaningful only for NightTrace kernel trace event files See get string on page 9 75 and Kernel String Tables on page 11 16 arg s value is looked up in the pre defined vect or table and its corresponding interrupt or exception name is displayed This is meaningful only for NightTrace kernel trace event files See get_string on page 9 75 and Ker nel String Tables on page 11 16 The formatted string from the next_state format table indexed by the integer returned by id if any Display statistics about state matches the state gaps and the state durations See get_format on page 9 79 and Pre Defined Format Tables on page 5 20 Display statistics about trace event matches and trace event gaps See get_format on page 9 79 and Pre Defined Format Tables on page 5 20 Display statistics about trace event matches and their type long third argument See get_format on page 9 79 and Pre Defined Format Tables on page 5 20 Display statistics about trace event matches and their type double first argument See get_format on page 9 79 and Pre Defined Format Tables on page 5 20 CPU List 6 6 The CPU List parameter determines from which logical central processing units CPUs the display object will process trace events Only processes that run on one of the CPUs on this lis
119. 16 get_string The get_string routine dynamically looks up a string in a string table SYNTAX get string table_namel int_expr PARAMETERS table_name int_expr table_name is an unquoted character string that represents the name of a string table To avoid possible forward reference problems try to make your get string calls refer to pre viously defined string tables The following string table names are pre defined in NightTrace event pid tid bool ean name pid name tid node name pid nodename tid nodename vector syscall device event summary event arg summary event arg dbl summary state summary For more information on these tables see Pre Defined String Tables on page 5 14 and Kernel String Tables on page 11 16 int expr is an integer expression that acts as an index into the specified string table int expr must either match an identifying integer value in the table name string table or the table name string table must have a default item line otherwise get string returns a string of int expr in decimal Often int expr is based on a NightTrace function If your table consists of only a default item line omit this parameter 9 75 NightTrace Manual DESCRIPTION The following NightTrace constructs can call get string to dynamically locate a static string in a string table A Then Expression of a display object configuration A value field of a format table For each ge
120. 20 4 4 4 5 4 13 4 14 4 17 4 28 A 2 A 3 trace open thread 2 9 2 15 2 18 2 22 6 4 7 17 8 3 trace start 2 5 2 10 2 15 2 18 2 23 4 1 4 28 C 1 trace trigger 2 20 4 4 4 17 A 2 A 3 A 4 Lightweight process identifier 8 7 9 10 9 23 9 39 Index 7 NightTrace Manual 9 50 9 63 Loading trace event 5 7 A 5 Locating see Searching Logging trace event 1 3 1 4 4 12 4 13 4 24 4 26 A 4 C 1 Loss trace event 2 15 2 21 4 16 4 28 7 17 8 19 A 1 C 1 Lost Event Color configuration parameter 8 19 Lost Event Color field 8 19 LWPID 8 7 9 10 9 23 9 39 9 50 9 63 lwpid function 9 23 Macro Configuration form 9 6 9 7 dialog box 9 6 Expression configuration parameter 9 4 9 7 MacroDefinition configuration parameter 9 7 MacroDefinition configuration parameter 9 7 MacroDefinition field 9 7 Macros 5 9 5 11 5 27 5 30 7 18 8 2 9 81 Map file see Event map file Mark inside Ruler 7 17 8 19 manipulating 6 9 push button 6 9 representation 6 9 Mark Color configuration parameter 8 19 Mark Color field 8 19 Matches event 8 6 8 13 9 33 state 8 6 9 58 summary 9 74 max function 9 71 10 7 10 8 max offset function 9 73 10 8 Maximum configuration parameter 8 18 Maximum field 8 18 Maximum value 8 18 9 71 9 73 Memory size 4 4 4 5 4 16 A 3 C 1 Menu Configure 8 1 Create 7 6 7 8 Edit 7 17 Expressions 9 1 File 5 26 7 18 10 9 Help 5 33 Tools 10 1 10 5 Menu item Index 8 Close 7
121. 3 ntrace window Configuration 8 4 8 10 8 12 8 13 8 14 8 16 8 17 8 19 9 2 9 4 Display Page 5 27 6 2 11 6 File Selection Dialog Box 5 31 5 32 5 33 7 19 10 10 B 3 Global 5 3 5 6 5 10 5 25 11 6 A 1 B 2 iconified 5 22 Macro configuration 9 6 9 7 Qualified Event configuration 9 81 Qualified State configuration 9 83 9 84 resizing 5 24 5 26 Search 9 4 10 2 B 3 C 1 Summarize 9 4 10 12 10 14 B 3 ntrace window component menu bar 5 23 5 24 pull down menu 5 23 5 24 push button 5 23 6 8 radio button 5 23 scroll bar 5 23 scroll region 5 23 text field 5 23 ntrace h 2 1 2 24 4 23 ntraceud buffer full cutoff see ntraceud cutoff cutoff 4 4 4 5 4 18 A 3 C 1 daemon 1 3 4 1 flush mechanism 4 4 help 4 7 invoking 4 28 memory size 4 4 4 5 4 16 A 3 C 1 page fault handling 4 4 performance considerations 4 1 4 17 4 18 A 1 quit running 4 21 4 28 4 29 reset 4 20 shared memory buffer size see ntraceud memory size sleep interval 4 3 4 4 4 17 statistical information 4 22 A 1 timeout interval 4 4 4 5 4 17 A 3 trace event file size 4 4 4 12 trace event logging 4 4 version information 4 8 ntraceud mode buffer wraparound 2 21 4 4 4 13 4 17 4 18 A 1 A 4 Index 10 expansive 4 2 4 4 4 5 4 12 4 13 4 14 A 1 A 4 file wraparound 4 4 4 5 4 12 A 1 A 4 ntraceud option bufferwrap buffer wraparound mode 4 4 4 13 4 17 4 18 A 1 A 4 cutoff cutoff perce
122. 3 19 3 46 4 Generating Trace Event Logs with ntraceud OVERVIEW x5 oder pl AS 4 1 The ntraceud Daemon ooooooooo e 4 1 The Default User Daemon Configuration 0 0 0 0c ee eee ee 4 2 htrace d MOdES sf usse A op Re SPT ee Pe ees 4 4 ntraceud Options i ebore ERU IURE SEU EE SEE 4 5 Option to Get Help help 0 0 00 0 0 cece ee e 4 7 Option to Get Version Information version 020 eee ee eee 4 8 Option to Disable the IPL Register ipldisable o oo oooooooo 4 9 Option to Prevent Page Locking lockdisable esee esses 4 11 Option to Establish File Wraparound Mode filewrap 4 12 Option to Establish Buffer Wraparound Mode bufferwrap 4 13 Option to Define Shared Memory Buffer Size memsize 4 16 Option to Set Timeout Interval timeout llle 4 17 Option to Set the Buffer Full Cutoff Percentage cutoff 4 18 Option to Select Timestamp Source clock oooooocooooooooooo oo 4 19 Option to Reset the ntraceud Daemon reset 000 eee eee ee 4 20 Option to Quit Running ntraceud quit 0 0 0 0 0 eene 4 21 Option to Present Statistical Information stats o oo omoooomom 4 22 Option to Disable Logging disable sees eee 4 24 Option to Enable Logging enable 00 eee esee 4 26 Invoking ntraceud motas oT RR eee Dees Shee hee RA
123. 8 start task id 9 40 start thread id 9 39 start tid 9 41 Start tid table name 9 44 Start time 9 42 state dur 9 57 state gap 9 9 9 57 state matches 9 58 S tate status 9 58 9 85 sum 9 72 summary 9 4 9 70 summary matches 9 74 10 8 table 9 75 task id 9 25 task name 9 31 thread id 9 24 thread name 9 32 tid 8 6 8 8 9 25 tid table name 9 29 time 9 27 9 83 trace event 9 19 G Gap event 9 32 state 9 57 get_format function 5 20 8 6 9 4 9 7 9 79 10 7 10 8 get_item function 9 4 9 77 get_string function 5 17 5 18 5 19 8 5 9 4 9 75 Global process identifier 7 8 8 5 8 7 9 10 9 22 9 37 9 48 9 62 10 4 10 6 11 2 Global Search radio button 10 3 Global Window area message display area 5 10 5 25 Graph data 5 30 6 4 7 16 11 12 event 5 30 6 4 7 15 9 7 exception 11 13 interrupt 11 12 State 5 28 5 29 6 4 7 14 9 7 9 84 10 12 11 13 11 16 syscall 11 15 Graphical user interface 5 1 B 1 resources 11 16 Grayed out button see Disabled button Grid 6 4 B 2 GridLabel 5 29 7 12 8 12 11 10 11 11 B 4 configuration form 8 12 Text configuration parameter 8 12 GridObject B 4 GUI see Graphical user interface Hardclock interrupts 5 7 11 12 11 20 Help ntrace 5 4 ntraceud 4 7 Help menu 5 33 h 77 1 command 2 2 Iconified window 5 22 iconnect 3C routine 11 19 id function 8 5 9 19 9 79 9 81 idtune 1M command 4 14 4 16 A 2 If Expression configuration paramete
124. 9 81 9 83 9 85 10 7 10 8 Glossary 14 Increment 6 7 6 12 6 14 Lost Event Color 8 19 MacroDefinition 9 7 Mark Color 8 19 Maximum 8 18 Minimum 8 18 Name 8 2 No Event List 10 5 Node List 8 3 9 81 9 83 Glossary 14 PID List 8 3 9 81 9 83 Glossary 14 QualifiedEvent 9 82 QualifiedState 9 84 Start Events 8 15 9 7 9 85 Start Expression 8 15 9 4 9 7 9 83 10 8 Text 8 12 Then Expression 8 2 TID List 9 81 9 83 Glossary 14 Time End 6 7 6 11 6 12 10 3 10 6 Time Length 6 11 6 13 Time Start 6 7 6 11 6 12 10 3 10 6 Zoom Factor 6 12 6 13 ntrace functions 9 9 ntrace macros 5 9 5 11 5 27 5 30 7 18 8 2 ntrace mode 7 3 Edit 5 27 6 1 7 4 radio buttons 7 3 View 5 28 6 1 6 6 7 4 9 7 10 1 11 6 ntrace option end load events before constraint 5 7 A 5 filestats list statistics and trace events 5 6 5 26 A 1 flat set flat color 5 8 help help 5 4 listing list trace events 5 4 5 10 nohardclock strip hardclock 5 7 A 5 process load process s events 5 7 A 5 start load events after constraint 5 7 A 5 version version 5 4 Xoption use X 1 options 5 8 ntrace qualified events 5 9 5 10 5 11 5 27 5 30 7 18 Index 9 NightTrace Manual 8 4 9 81 ntrace qualified states 5 9 5 11 5 27 5 30 7 18 8 4 9 13 9 35 9 36 9 37 9 38 9 39 9 40 9 41 9 42 9 43 9 44 9 45 9 46 9 47 9 48 9 49 9 50 9 51 9 52 9 53 9 54 9 55 9 56 9 57 9 58 9 59 9 8
125. 916 events saved in memory 0 trace events lost 52 4036288s time span from 0 0000000s to 52 4036288s RCIM synchronized tick clock was used to time stamp events 10916 total events read from disk plus 9863 continuation events 10916 total events saved in memory 1 events internal to ntrace 0 total trace events lost npe total time span saved in memory N Screen 5 4 Example of ntrace filestats Output Invoking the ntrace Display Utility By default when nt race starts up it reads and loads all trace events from all trace event files into memory therefore the more trace events in your trace event file s the more memory ntrace uses The nohardclock process start and end options let you prevent the loading but not the reading of certain trace events nohardclock Do not load hardclock interrupts from the kernel trace event file This option may save about 15 of the memory ntrace consumes For more information on the hardclock interrupt see Hardclock Interrupt Handling in the PowerMAX OS Real Time Guide If you invoke ntrace with the process option it loads only exceptions and system calls of processes you specify after the process this takes some extra processing time during ntrace start up You can invoke ntrace with multiple process options The possible ways to use the process option include process all process PID process name start offset start time slu star
126. A StateGraph represents an instance of a state as a solid horizontal bar that starts when the state is active and ends when the state is inactive Instances of the same state do not nest thus once a state becomes active events that might normally satisfy the conditions for the start event are ignored until the end event is encountered You can create a StateGraph by clicking on Create I StateGraph and drawing on the grid StateGraphs must be placed in a Column A StateGraph and a Ruler are shown in Figure 7 10 A StateGraph can display trace events in a manner identical to an EventGraph This can be useful for saving screen space or detecting when state start and state end trace events occur out of order For example the trace event lines can show multiple state start trace events occurring before a state end trace event Some examples of information that StateGraphs can be used to display are EventGraph Creating Display Objects The times your application is executing a particular subroutine The differences in the execution speed of parallel threads The time spent in contention for resources See StateGraph on page 8 14 for more information In View mode to find out more information about a particular trace event position the cursor on a trace event line and click once with mouse button 2 Information about that trace event is displayed in the message display area You can also click with mouse button 2 on the start and
127. ALUES The trace flush andtrace trigger routines return a zero value NTNOERROR on successful completion Otherwise they return a non zero value to identify the error condition A list of trace flush and trace trigger error codes follows NTFLUSH A failure occurred while attempting to flush the shared mem ory buffer Solution Verify the status of the user daemon if necessary restart it and rerun the trace SEE ALSO Related routines include trace event trace event arg trace event dbl trace event four arg 2 21 NightTrace Manual trace_close_thread The trace close thread routine disables trace event logging for the current thread or process SYNTAX G int trace close thread Fortran integer function trace close thread Ada function trace close thread return ntrace error DESCRIPTION A NightTrace thread can be a process C thread or Ada task Each thread that C calls trace open thread must have its own trace close thread call For more information on threads see Programming with the Threads Library in the PowerMAX OS Programming Guide RETURN VALUES The trace close thread routine returns a zero value NTNOERROR on successful completion Otherwise it returns a non zero value to identify the error condition A list of trace close thread error codes follows NTINIT The NightTrace library routines were not initialized Solution Call trace close thread only once
128. C is a trademark of International Business Machines Corp NightTrace Manual Contents Chapter 1 Introduction OVerVIe While E PA MD ANE ER 1 1 WMhatasNightlIrace ay uctor ere Ree RIDERS E eee Ei ud S ets 1 1 User and Kernel Tracing 2 0 0 eee cee eee 1 2 Timestamp Source Selection 0 00 cece 1 2 Trace Point Placement 0 0 cee eee eee 1 2 Languages Supported 2 0 0 N e eee eee eee 1 3 Proc ssesand CPUS ses e Sb er eth Sito wi Dese 1 3 Information Displayed 0 2 0 eee ce cee eee 1 3 Searches and Summaries 0 0 1 3 LEogemg and Analysis cue nccee ee esa ee ARR Se VE I RA DERE 1 3 The User Trace Event Logging Procedure 0 errereen 1 4 The Kernel Trace Event Logging Procedure 0 0004 1 5 The Trace Event Analysis Procedure 0 00 c eee eee eee eee 1 6 Recommended Reading oes cake A Mae han Des te Ree eee 1 7 Chapter 2 Adding Library Calls to Your Application OVETVIEW A ese ae A A E eI erp Ere Re Sealers cg 2 1 Language Specific Source Considerationfcs llle essel 2 1 Quo e nose A eee Ue RE M EL CEREREM EE 2 1 Fortr n i dre enne eee ae ever Aa e a LR Fe Soe Sil ALS 2 2 Ads pc A ES UR M EROR EO ERE Es 2 2 Inter Process Communication and Library Routines 008 2 3 Understanding NightTrace Library Calls 0 0 0 0 cece eee eene 2 3 O Leod sale E ke test e Rage O 2 5 trace Open thread eror a PER e YR
129. Daemon Definition Dialo8 ooooocooorccoccocor e 3 25 Import Daemon Definition eh 3 27 General ics eer Gites Ce we eG Ge beet ee ehh Eee 3 28 ERA 3 29 Trace Events Output 0 0 e 3 31 User Trace RR ODER PEE ao ees oT E Tee TIS Text 3 33 Locking Policies ii dre oe oh A 3 34 Shared MEMO so eyer prne ads 3 36 Timestamp Heartbeat o on 00 0 eee eee eee 3 36 User Event Butter oca A as 3 37 Events A A eee ak 3 37 Load Event NaS cai seas o Ada 3 39 RUNTIME 2 5 ooo lit id BS 3 41 SCA it A EIA 3 42 CPU Baas std de 3 43 NUMA oL oe tudes ias bar aa 3 43 PoliCie 25 4 uon oet tt Ue cee Ae bese ees 3 44 A sats sie oa kes wes Rk eS SAG RAR SAA OO 3 45 Streaming Options dsrin e ee ce ee ee bebe ee e 3 45 Chapter 4 Generating Trace Event Logs with ntraceud OVERVIEW Sues a tect Soh o WD ede wa tae Bee ee 4 1 The ntraceud Daemon cui A reg eb ADRESS RAE ES 4 1 The Default User Daemon Configuration 00 0 0 0 02 eee ee eee 4 2 ntraceud MOdES osi dose EIA TA 4 4 ntraceud Options ea Ss eis Gt areas Pees ba v wuchs wets estis 4 5 Option to Get Help help 0 0 00 cee eee eee 4 7 Option to Get Version Information version 0 e cece eee 4 8 Option to Disable the IPL Register ipldisable ooo oo ooo oo 4 9 Option to Prevent Page Locking lockdisable 0 0 4 11 Option to Establish File Wraparound Mode filewrap 4 12 Option to Establis
130. Details Area see Daemon Details Area on page 3 17 allowing the user to edit that particular defi nition NOTE The daemon definition may not be altered while the daemon is executing Delete Deletes the daemon definition s currently selected in the Daemon Details Area see Daemon Details Area on page 3 17 The user is prompted for confirmation before the deletion is performed 3 9 NightTrace Manual Login 3 10 Attach Allows the user to query any target system for user application trace daemons and displays the results in the Attach Daemons dialog see Attach Daemons on page 3 12 The user may then attach to the desired daemon and control it using the NightTrace Session Manager Detach Relinquishes control of the running daemon s currently selected in the Daemon Details Area see Daemon Details Area on page 3 17 Reset Flushes the contents of trace buffers for the running daemon s currently selected in the Daemon Details Area see Daemon Details Area on page 3 17 Any events in the buffer at the time of the reset are discarded Events that have already been writ ten to the output device file or stream are unaffected Pressing the Reset button also places the selected daemons in a Paused state see State on page 3 19 NOTE This option is not supported for kernel trace daemons Select All Selects all daemon definitions listed in the Daemon Details Area s
131. EFLT ADDR and TR PROTFLT ADDR Events 11 14 Note the TR PROTFLT ADDR event to the left of the current time line at t ime 9 459738 and the TR_PAGEFLT_ADDR event to the right of the current time line at t ime 9 460050 and the corresponding data in the message display area See The Display Page on page 7 2 for more information on the message display area and other elements of the display page Note also that the TR PROTFLT ADDR and TR PAGEFLT ADDR events are represented by a vertical line that only intersects the exception state graph whereas a TR SWITCHIN event see Context Switch Trace Event on page 11 2 intersects both the exception and syscall state graphs In addition TR PROTFLT ADDRand TR PAGEFLT ADDR events will only appear within a currently executing exception This can be seen in Figure 11 13 Tracing the Kernel NightTrace ntracBAAav02Jp offset 729 id TR_SWITCHIN pid in rwhod tid 166 0 cpu 1 time 4 922322s argl 10878977 offset 738 id TR_PAGEFLT_ADDR pid in rwhod tid 155 0 cpu 1 time 4 922573s argl 0xe21a0000 arg2 0x9Be c arg3 0x1 offset 751 id TR_SWITCHIN pid idle tid 0 0 cpu 1 time 4 922808s argl 0 resched strsched CPU 1 data access lid in ruhod sie sent Sia Seca Time Start 4 3222635s Time Length 0 0006062s Time End 4 3228757s Event Start 726 Event Count 24 Event End 751 Zoom Factor 2 0 Increment 25 007 Current Time 4 9225061s Apply Reset Center M
132. ES e Oh tah Sahel wrest ed de e 3 37 Load Event Name Si co a a 3 39 Runtime i t At S A eit gc ed ii 3 41 Scheduling s 243 se ha Sk te Sh aie see RID AM 3 42 CRU Bias b eH e EUER rre ee Wa eee ye Cete 3 43 NUMA 0er da A tes 3 43 Policies teen ote etx tt eer ici 3 44 SICA E ee ee eG ea DITA 3 45 Streaming Options l l E a E ES 3 45 NightTrace Manual 3 NightTrace Session Manager The NightTrace Session Manager allows users to manage user and kernel NightTrace dae mons It provides users with the ability to define a session consisting of one or more dae mon definitions which can be saved for future use These definitions include daemon col lection modes and settings daemon priorities and CPU bindings and data output formats as well as the trace event types that are logged by that particular daemon Using the NightTrace Session Manager users can manage multiple daemons simulta neously on multiple target systems from a central location The NightTrace Session Manager offers the user the ability to start stop pause and resume execution of any of the daemons under its management The user may also view statistics as trace data is being gathered as well as dynamically enable and disable events while a particular daemon is executing In addition to sending trace output to a file for later analysis the NightTrace Session Man ager also offers a streaming output method When streaming trace output is sent directly to the Nigh
133. Expressions Men coccion aa Loe LE Bel 9 1 Expression Dialog Boxes 2 0 00 0 cece eee eee 9 2 Expression Configuration Forms 0 0 0 eee 9 2 EXDIeSSIOUS uti ta ds se pace euer del fs 9 4 Operators eli er RISU NES a e e 9 4 Opetandss c2 ers Cie EN b eb E EVENISSE ee 9 5 G nstantSi cor a E LO S aes eed CER 9 5 Macros bere CERERI HERE IM a M Pd de 9 6 FUNCHONS ists skeet EB EE WE Ob dul e en ERE 9 9 Function Parameters 0 0 0 ccc cece t nee nee 9 12 Function Terminology 0 0 0 0 ee 9 13 Trace Event Functions 9 19 TAQ 56 RSs erat A aa 9 19 ATS iss Seat eu aes E Ee Gee hee ales te 9 20 ASTUR EAE ee MEN 9 21 n mcatgsQ eee A IS 9 21 PdO sad ii a weal 9 22 raw pid eere eddie es 9 23 WD eU E RH EORR E P ORE P SNR 9 23 thread 1d sii ieiee ls ok Ore X DIRE ed eke bese 9 24 task iii Ela bee is pate dulce phd ob erit 9 25 A S PED uS EE E ie 9 25 CPUC nes eae DROP ER I MUS DE RE MERI TRUE OR SURE HE 9 26 Offset oils ete ee We Ee ee ee a 9 27 A Gated ea td ee ee hl EP bats 9 27 node 1d ean ae ne id dd db e es 9 28 pid table mame A EIER US ace 9 29 tid table name ioci ee ek See Deed REDE RU ERIS 9 29 nde namel oso ncbnbherteierssertISebvrurer siu 9 30 process name License ber ue REIHE t np E enna 9 30 task name t eL EXAG x uU RICH T eee ee CN 9 31 thread name comicios e ai ueerde r3 bv 9 32 Multi Event Functions 0 0 00 cc cece eee 9 32 event BAP EA A re dee a
134. Figure 9 6 state_status false The current time line was not posi tioned within a current instance of a state state gap 0 000020 The duration of time in seconds between event b and event c The function operated the most recent instance of the state instance B and the immediately preceding instance instance A state dur 0 000090 The duration of time in seconds between event c and event d The function operated on the last com pleted instance of the state instance B state matches 2 Assuming no other instances of the state preceded those shown in the fig ure The function operated on all com pleted instances of the state which included instances A and B start time 1 631750 The time associated with event c The function operated on the most recent instance of the state instance B end_time 1 631840 The time associated with event d The function operated on the last com pleted instance of the state instance B Assuming the current time line was positioned at time line y in Figure 9 6 the various instances would be defined as current instance Instance C last completed instance Instance B most recent instance Instance C 9 16 Using Expressions The table below indicates the information returned by various NightTrace functions assuming the current time line was positioned at time line y in Figure 9 6 state status state gap state dur State matches st
135. Form the time range is defined by that form When used elsewhere the time range is defined as the region starting with the first trace event and ending with the current trace event SYNTAX sum expr PARAMETERS expr A numeric expression min offset max offset Using Expressions RETURN TYPE data type of expr SEE ALSO Summary Functions on page 9 70 and Summarizing Statistical Information on page 10 5 DESCRIPTION The min offset function returns the ordinal trace event offset where the minimum value of the parameter occurred for matches in the time range Thus if the same minimum was seen more than once the offset corresponds to the first one seen SYNTAX min offset expr PARAMETERS expr A numeric expression RETURN TYPE integer NOTE There is no function that returns the trace event ID where the minimum value of the first argument occurred for all matches in the time range You could obtain this value by nesting the functions as follows offset id min offset argl SEE ALSO Summary Functions on page 9 70 and Summarizing Statistical Information on page 10 5 DESCRIPTION The max offset function returns the ordinal trace event offset where the maximum value of the parameter occurred for matches in the time range Thus if 9 73 NightTrace Manual summary matches 9 74 the same maximum was seen more than once the offset corresponds to the first
136. If you wish to change the definintion of an expr you must select it from the list of exprs and press Configure See Expression Dialog Boxes on page 9 2 for details TIP If you want to share exprs among multiple display pages create an empty display page and put only exprs in it Any new exprs or changes to old exprs should be added to this display page It is also a good idea to place a DataBox on this page for every expr that you add to this page This way you can see the current value of all your exprs at a glance 9 3 NightTrace Manual Expressions NightTrace expressions can evaluate to numbers strings or boolean values Expressions appear in the following places in NightTrace e Start Expression and End Expression on Configuration Forms Summarize Forms e f Expression on Configuration Forms Summarize Forms Search Forms Then Expression on Configuration Forms Filter Expression and Summary Expression on the Summarize Form Expression on Macro Dialog Boxes Values in format tables e Calls to format get string get item get format and summary functions Start Expressions End Expressions If Expressions and Filter Expression must evaluate to boolean values See Chapter 8 for more information on the Configuration Form See Chapter 10 for more information on the Search and Summarize Forms See Format Tables on page 5 17 for more information on format tables Information on format get string
137. Integer literals may be expressed using typical C language notation decimal literals have no special prefix octal literals begin with a zero hexadecimal literals begin with a 0x Floating point literals are always considered to be double precision floating point literals String literals must be enclosed within double quotes to include a double quote in a con stant string literal precede the double quote with a backslash character For example possible meltdown alert E and FALSI The case insensitive boolean constants TRUI respectively E have the values 1 and 0 9 5 NightTrace Manual Table 9 1 shows units and suffixes for time constants Table 9 1 Time Units and Constant Suffixes Time Unit Suffix Seconds This is the default S Milliseconds 10e 3 seconds ms Microseconds 10e 6 seconds us Nanoseconds 10e 9 seconds ns Macros Macros are named expressions provided for flexibility and convenience Table 9 2 con trasts functions and macros Table 9 2 A Comparison of Functions and Macros Functions Macros Predefined User defined May have parameters Cannot have parameters Invoked with parentheses around the Invoked with a dollar sign before the parameter list macro name To create a macro definition select the Macros menu item from the Expressions menu see Expressions Menu on page 9 1 to open the Macro Dialog Box see Expression Dialog Boxes on
138. NightTrace Manual cum CONCURRENT 0890398 080 CORPORATION June 2002 Copyright 2002 by Concurrent Computer Corporation All rights reserved This publication or any part thereof is intended for use with Concurrent products by Concurrent personnel customers and end users It may not be reproduced in any form without the written permission of the publisher The information contained in this document is believed to be correct at the time of publication It is subject to change without notice Concurrent makes no warranties expressed or implied concerning the information contained in this document To report an error or comment on a specific portion of the manual photocopy the page in question and mark the correction or comment on the copy Mail the copy and any additional comments to Concurrent Computer Corpora tion 2881 Gateway Drive Pompano Beach FL 33069 4324 Mark the envelope Attention Publications Depart ment This publication may not be reproduced for any other reason in any form without written permission of the publisher The license management portion of this product is based on Elan License Manager Copyright 1989 1994 Elan Computer Group Inc All rights reserved NightTrace KernelTrace NightView NightStar Power Hawk RedHawk and MAX Ada are trademarks of Concurrent Computer Corporation Elan License Manager is a trademark of Elan Computer Group Inc PowerPC is a trademark of International Business Machin
139. RU OO EN cad mua a dea 6 1 Mouse Button Operations o oocococcococ ete eee 6 3 The Grid x rts ERE e Le e e etes be 6 4 Viewing Strategy s csi cell roD RH GER ed De bee bee tea 6 5 The Interyal Scroll Bate eus eere A ERU EN RA 6 7 NightTrace Manual The Interval Push Buttons 0 0 cece e 6 8 The Interval Control Area 0 0 eee e 6 11 Reading Fields creber eet wee be ele ri ee RR 6 11 Editing Single Fields co ae nee ino aR 6 12 Editing Multiple Fields llis 6 14 Field Editing o rore hos bkRUILA es 6 16 Editing Text Fields eR ERES RESET 6 16 Positioning Within Text Fields llle 6 18 Chapter 7 Creating Display Objects OVerVIe Woo desc VR Mei ates Ea scd te NES EUR e URL E De a 7 1 The Display Page REREER eb e EEUU SEU REA SE EUR 7 2 Display Page Modes coruna vda id 7 3 Edit Mode c eere EE CREMAS ER M 7 4 VIEW MOE cs cita A A A RAS MEG ED RUN 7 4 Operations on Display Objects ssleseele re 7 4 Creating Display Objects ooooocococooorcoccocr eh 7 5 Selecting Display Objects 0 0 0 cece eee 7 6 Moving Display Objects 00 cece ee 7 7 Resizing Display Objects 0 0 cee eee eee ee 7 7 Display DDEC eii REL RPESe IG BIG RE ERA Sole SU 7 8 Gridbabell ac eee LR Dette LE e 7 12 DataB Ox ices eee en A A MA 7 12 Columnist 35 aie A en Oh rE eee ea 7 13 StateGraph csm il o SUN aia 7 14 Events a a 7 15 Data IA a 7 16 Rul r rt e ied Mol dol dd D Canes 7 17 Ed
140. SERM 4 28 Exercise Logging Trace Events lel 4 29 NightTrace Manual 4 Generating Trace Event Logs with ntraceud Overview A user daemon is required in order to capture trace events logged by user applications There are two methods for controlling user daemons Use the graphical user interface provided in the ntrace session manager dialog as described in the preceeding chapter Use the command line tool ntraceud The interactive session manager is often more convenient and easier to use and addition ally offers concurrent viewing of trace events while the application continues to log trace data this optional feature is called streaming Alternatively the ntraceud command line tool is useful in scripts where automation is required This chapter describes the nt raceud command line tool broken down into the following topics e The nt raceud daemon The default user daemon configuration ntraceud modes ntraceud options Invoking ntraceud The ntraceud Daemon When you start up nt raceud it creates a daemon background process and returns your prompt The daemon creates a shared memory buffer in global memory Your application writes trace events into this buffer and the daemon copies these trace events to a trace event file You supply the name of the trace event file on your ntraceud invocation and in the trace begin library call in your application If this file does not exist ntraceud creates it
141. TR SYSCALL ENTRY pid 1s tid 1250 0 time 94473 S syscall read device file 3393 cpu 01 TR_EXCEPTION_ENT pid 1s tid 1250 0 time 94528 S vector data access 3394 cpu 01 TR PAGEFLT ADDR pid ls tid 1250 0 time 94534 S kernel data page fault addr 0xele18000 PC 0x000931cc 3395 cpu 01 TR EXCEPTION EXI pid ls tid 1250 0 time 94590 S vector data access 3396 cpu 01 TR SYSCALL EXIT pid ls tid 1250 0 time 94659 S syscall read device file 3397 cpu 01 TR SYSCALL ENTRY pid 1s tid 1250 0 time 94715 S syscall close device file 94342 Y d filestats Screen 5 3 Example of ntrace listing Output with data page fault Displays simple statistics about all trace event file s arguments to standard output similar to the display on the Global Window and exits See ntrace Global Window on page 5 25 The statistics are grouped by trace event file with cumulative statistics for all trace event files The statistics include the number of trace event files their names the num ber of trace events logged and the number of trace events lost di Screen 5 4 shows an example with log The user trace event file map The event map file continuation events The NT CONTINUE trace events that ntraceud logs for multi argument trace events amber2 ntrace filestats nl cap vectors cap p 1 trace event log file read Kernel trace event log file nl cap 10916 trace events plus 9863 continuation events 10
142. The new interval ends at ordinal trace event number offset Event End and has a length of the specified Time Length unless the Time Length forces Event End to change 6 15 NightTrace Manual Field Editing You make changes to fields by following these steps 1 2 3 Editing Text Fields Do one of the following e Click with a mouse button on the field you want to edit Clicking with mouse button 1 leaves a blinking vertical bar called the text cursor where you clicked in the field Clicking with the other mouse buttons leaves the text cursor at the end of the field Drag with mouse button 1 on the field you want to edit e If there already is a text cursor in a field you can press Tab to move to the next field or lt Shift gt lt Tab gt to move to the previous field Use the built in field editor to change values Editing procedures follow Either press lt Enter gt or click on the Apply push button This is called applying your changes You can make the following types of editing changes in a text field Insert text Delete text Replace text Undo a text change Table 6 4 Making Editing Changes Goal Steps to Attain Goal Insert 1 Position the text cursor where you want to insert character s character s 2 Type in the additional character s Delete one character 1 Position the text cursor to the left of the character to be deleted to the right 2 Simultaneously press lt Ctrl gt lt d
143. Thus if the same minimum or maximum was seen more than once the offset corresponds to the first one seen TIP Including min offset max offset min ormax in your summary text tells you the inclusive range of matches that you summarized and the summary matches function tells the number of matches that you summarized TIP Sometimes there are anomalies in the trace information logged by an application such as an unusually long state duration during program start up this can throw off the duration statistics when analyzing typical program performance You can use the Start Expression for state summaries and the If Expression for event summaries to limit the range of trace events summarized and remove extraneous trace events from the statistics produced See Chapter 8 for information about configuration parameters See Chapter 9 for information on nt race expressions For more information on the Summarize push but ton see Summarize Form Push Buttons on page 10 8 Summarize Form Push Buttons 10 8 Following is a summary of the effects of clicking on the push buttons in the Summarize Form Apply default Validates any field change s on the Summarize Form Clicking on Apply is equivalent to pressing Enter Reset Restores changed field s on the Summarize Form to the value s they had after the last Apply or Enter This works only if you Using the Built In Tools have not already pressed lt Enter gt or cl
144. Trace Manual Option to Get Version Information version The ntraceud version option displays the current ntraceud version stamp on standard output SYNTAX ntraceud version DESCRIPTION The ntraceud version option displays version stamp information for this ntraceud daemon 4 8 Generating Trace Event Logs with ntraceud Option to Disable the IPL Register ipldisable The ntraceud ipldisable option disables the default use of the system s interrupt priority level IPL register by ntraceud and by the NightTrace library routines in your application SYNTAX ntraceud ipldisable trace_file DESCRIPTION On PowerMAX OS by default NightTrace modifies a shared memory region bound to the system s interrupt priority level IPL register to control preemption On RedHawk Linux by default NightTrace uses rescheduling variables to prevent process preemption this does not prevent preemption by machine interrupts but this is not of concern on RedHawk Linux since user applications cannot run at inter rupt level On PowerMAX OS if your application lacks read and write privilege to dev sp1 the NightTrace daemon and library initialization routine exit with errors On RedHawk Linux if your application lacks privileges to be able to use rescheding variables the NightTrace daemon and library initialization routines will exit with errors If you still want to trace events you must invoke the nt raceud daemon with th
145. Version 4 2 Copyright lt C 1996 Concurrent Computer Corporation 1 trace event log file read Kernel trace event log file tt hydrogen home rwb tmp n1 jedi 36111 trace events plus 31240 continuation events 36111 events saved in memory 0 trace events lost 9 5582967s time span from 0 0000000s to 9 5582967s T 2 3 4 5 6 7 8 3 Time Base Register was used to time stamp events 36111 total events read from disk plus 31240 continuation events 36111 total events saved in memory 1 events internal to ntrace 0 total trace events lost 9 5582967s total time span saved in memory Figure 5 2 Global Window for a Single Trace Event File Message Display Area The message display area of the Global Window presents the following information Version information 5 25 NightTrace Manual Trace session statistics grouped by trace event file Cumulative statistics for all trace event files Event timestamp source Error messages about corrupted or syntactically incorrect ntrace file arguments You can also obtain this statistical information by invoking ntrace with the filestats option If you invoke ntrace without any trace event file arguments nt race displays most of the statistics as zero values By default ntrace is sized to hold twenty lines in the message display area You can alter this number by changing the size of the Global Window To change the Global Window size resize your window by using feat
146. What is NightTrace The NightTrace toolset is part of the NightStar family consisting of an interactive debugging and performance analysis tool trace data collection daemons and an Applica tion Programming Interface API allowing user applications to log data values Night Trace allows you to graphically display information about important events in your appli cation and the kernel for example event occurrences timings and data values NightTrace consists of the following parts NightTrace API ntrace ntraceud Libraries and include files for use in user applications that log trace events to shared memory A graphical tool that controls daemon sessions and displays user and kernel trace events in trace event file s A daemon program that copies user applications trace events from shared memory to trace event file s NightTrace is flexible As a user you control Selection of user tracing of your application or kernel tracing Selection of timestamp source Trace point placement within your application The source language of the trace application The number of processes and CPUs you gather data on The amounts and types of information you display race event searches and summaries 1 1 NightTrace Manual User and Kernel Tracing If interactions are important you can simultaneously capture event information from your application and from the kernel Alternatively you can capture just user
147. _format on page 9 79 A newly created DataBox contains a 0 See DataBox on page 8 13 for more information DataBoxes can appear anywhere on the grid except within a Column You can put several DataBoxes on a grid Some examples of data that you can configure a DataBox to show are The tag of the last trace event before the current time See Table 8 3 The NightTrace thread name of the last trace event before the current time See Table 8 3 A particular argument logged with the last trace event before the current time See arg on page 9 20 The total amount of time the application was in a particular state before the current time See state_dur on page 9 57 and sum on page 9 72 The number of times a particular trace event has occurred before the current time See event_matches on page 9 33 A string of characters generated by a format expression See format on page 9 80 Figure 7 9 Column Example Clicking on Create 1 Column lets you draw or create a Column display object on the grid When a Column is first created it is an empty rectangle that does not display data of its own A Column holds StateGraphs EventGraphs DataGraphs and Rulers It provides a convenient way of associating these graphical display objects Figure 7 9 shows a Column after a Ruler has been added 7 13 NightTrace Manual StateGraph 7 14 Columns ensure that all graphical display
148. _name 9 68 DESCRIPTION The offset_tid_table_name function returns the name of the inter nally assigned NightTrace thread identifier table TID table for the ordinal trace event offset SYNTAX offset tid table name offset expr PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE string SEE ALSO tid table name on page 9 29 start tid table name on page 9 44 and end tid table name on page 9 55 DESCRIPTION The o fset node name function returns the name of the system from which the ordinal trace event offset was logged SYNTAX offset node name offset expr PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE string Using Expressions SEE ALSO node_name on page 9 30 start_node_name on page 9 45 and end_node_name on page 9 56 offset process name DESCRIPTION The offset process name function returns the name of the process PID from which the ordinal trace event offset was logged SYNTAX offset process name offset expr PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE string SEE ALSO process name on page 9 30 offset task name DESCRIPTION The offset task name function returns the name o
149. _table what_pid item 1 Trace event 1 logged by pid d d raw_pid lwpid default item Unaccounted for event ID d id y Assume that you make the following call in the Then Expression of a DataBox get_format what_pid id 9 79 NightTrace Manual In this example the what pid format table associates one dynamically generated string with trace event ID 1 id 1 and another string with all other trace events default_item When NightTrace processes a trace event for the display object with the above get_format it 1 Evaluates the NightTrace id function Assume it evaluates to 1 2 Calls get_format 3 Uses this value 1 as an index into the what_pid format table 4 Retrieves the associated string Trace event 1 logged by pid d d from the what_pid format table 5 Evaluates the NightTrace raw_pid and lwpid functions Assume they evaluate to 213 and 1 respectively 6 Replaces the d conversion specifiers with the raw pid and lwpid values 7 Displays Trace event 1 logged by pid 213 1 RETURN TYPES On successful completion get_format returns a format table string Other wise it returns an empty string For more information on format tables see Format Tables on page 5 17 and the usr lib NightTrace tables file For more examples of function calls with pre defined format tables see Table 8 3 format The format routine display
150. a display page Operations on Display Objects 7 4 This section describes some operations you can perform on display objects The four operations discussed are Creating new display objects and placing them on the grid Selecting display objects Moving display objects around the grid Resizing display objects Each of these operations involves using the mouse buttons and the grid Figure 7 4 Table 7 1 and the NightTrace Pocket Reference card show which mouse buttons correspond to which operations These operations are referred to as grid operations You can perform other operations on display objects using the Edit and Configure menus Edit operations are discussed later in this chapter See Chapter 8 for more information on configure operations Create or Select Creating Display Objects Resize Figure 7 4 Button Functions on a Mouse Table 7 1 Edit Mode Mouse Button Operations Button Use Within the Grid Mouse button 1 lt Ctrl gt mouse button 1 lt Shift gt mouse button 1 Mouse button 2 lt Ctrl gt mouse button 2 Mouse button 3 lt Ctrl gt mouse button 3 Create new objects single select by clicking or multiple select by dragging Select the Column display object Multiple select or toggle selection Move display objects Move the Column display object Resize display objects Resize the Column display object Creating Display Objects Before you can do any of the other op
151. a mechanism for referencing trace event configurations within some functions for example they cannot appear alone in a DataBox configuration You may use a qualified event in trace event functions For more information see Trace Event Functions on page 9 19 To create a qualified event definition select the Qualified Events menu item from the Expressions menu see Expressions Menu on page 9 1 to open the Qualified Events Dialog Box see Expression Dialog Boxes on page 9 2 for details on this type of dialog Click the Add button on the Qualified Events Dialog Box select the qualified event from the list and click on the Configure button to pop up a Qualified Event Config uration Form like the one shown in Figure 9 7 9 81 NightTrace Manual 9 82 AAA A gt JH evento QualifiedE vent event 001 Node List AL Event List TRINTERRUPT_ENTRY If Expression RE CPU List a PID List fal TID List fac tes Reset Restore _ciose Figure 9 7 Qualified Event Configuration Form The following parameter is specific to the Qualified Event Configuration Form QualifiedEvent The name by which you refer to this qualified event in expres sions TIP Consider giving your trace events upper case names in event map files and giving any corresponding qualified event the same name in lower case NOTE The Node List field appears in this dialog only when NightTrace
152. a will be displayed as New and the Save Session dialog see Save Session on page 3 6 will be presented when this menu item is selected to allow the user to navigate to the desired directory in which to save the session configuration file 3 3 NightTrace Manual Open Session 3 4 Save Session As Displays the Save Session dialog see Save Session on page 3 6 allowing the user to navigate to the desired directory in which to save the session configuration file The name of this file will then appear in the Session Configuration File Name Area see Session Configuration File Name Area on page 3 17 Close Session Closes the current session but leaves the NightTrace Session Manager open If changes have been made to the current configuration but have not yet been saved the Unsaved Changes dialog is presented to the user see Unsaved Changes on page 3 8 Close Window Exit Closes the NightTrace Session Manager but leaves NightTrace open If changes have been made to the current configuration but have not yet been saved the Unsaved Changes dialog is presented to the user see Unsaved Changes on page 3 8 Closes the NightTrace Session Manager and exits NightTrace completely If changes have been made to the current configuration but have not yet been saved the Unsaved Changes dialog is presented to the user see Unsaved Changes on page 3 8 The Open Session dialog is a standard fil
153. about trace event IDs Sometimes it is useful to log the current value of a variable or expression arg along with your trace event The trace event logging routines provide this capability They differ by how many and what types of numeric arguments they accept The trace_event rou tine takes no args The trace_event_arg routine takes a type long arg The trace_event_flt and trace event two flt routines take floating point type of float args The trace event dbl and trace event two dbl routines take floating point type double args The trace event four arg routine takes four type long args If you want the ntrace display utility to display these trace event arguments in anything but decimal integer format you can enter the trace event in an event map file See Understand ing Event Map Files on page 5 9 for more information on event map files and formats Alternatively you could call the for mat function See format on page 9 80 for details Every call to trace event four arg causes NightTrace to log an NT CONTINUE overhead trace event A trace point is a place in your application s source code where you call a trace event logging routine Usually this location marks a line that is important to debug ging or performance analysis Ideally trace events provide enough information to be useful but not so much information that it is overwhelming Meeting these goals requires careful trace point plann
154. ace Color Ruler markColor Ntrace Color Ruler lostEventColor TIP On a monochrome display make sure that you can differentiate among display objects within a Column The easiest way to do this is to leave at least one grid dot between dis play objects in a Column and to make the background color of the Column black Grid object settings apply if you have not set the corresponding setting for a specific display object The general grid object resource strings are Nt race Color GridObject background Ntrace Color GridObject foreground Ntrace Color GridObject borderColor For information about setting X resources for kernel displays see Color Information on page 11 16 Examples GUI Customization Setting X resources to values is most consistent if the values of the X resources do not contain spaces For example even if your rgb txt color file contains a color called navy blue for simplicity type it as one word without any quotation marks In the following examples you are making navy blue navyblue the foreground color foreground of all grid objects GridObject on a color monitor Color for ntrace Nt race This example shows how this line may appear in your Xdefaults file Ntrace color GridObject foreground navyblue The following example shows how you can use this setting on the ntrace invocation line Note there must not be any spaces between the colon and the value ntrace xrm Ntrace color Gri
155. ace Event File keyboard Glossary A time period in the trace session delimited by the Time Start and Time End fields of the interval control area The region of the display page that holds nine numeric fields that define and manipulate the interval and the display objects on the grid The system timer on the NightHawk 6000 Series and TurboHawk systems that NightTrace uses to timestamp trace events A trace event file is generated by a kernel trace daemon This file contains raw ker nel data and is automatically transformed into a filtered file with a new filename using the nt suffix by ntrace Either a raw kernel trace event file or a fil tered file may be specified to ntrace The filtering process also creates a vectors file which is formed by appending a vec suffix to the original trace event file name A traditional input device for entering text into fields In this manual this is a standard 101 key North American keyboard last completed instance of a state last exception box last interrupt box last syscall box The most recent instance of a state that has already completed Thus the current time line would be positioned either on or after the end event for a state A DataBox on a kernel display page It displays the last exception prior to the current time line that executed and may still be executing on the associated CPU A DataBox on a kernel display page It displays the name of the last inter
156. ace_begin call an enhanced replacement for trace start allows the user to define the following settings in a user appli cation those values listed under the Spin Lock Protection and Page Critical Locking categories on this page the Buffer Size also found on this page the setting for the Timing Source which appears on the General page of the Daemon Definition dialog see General on page 3 28 See trace begin on page 2 5 for more information on this API Inherit settings from running user application When this is checked trace settings defined by a running user application are silently preferred if those definitions differ from those made in the Session Manager If not checked trace settings defined by user applications must match those in the current daemon definition See above for details on which trace settings may be inherited The daemon and the user applications communicate with each other through shared mem ory The shared memory segment identifier is calculated from the Key File setting see Key File on page 3 31 The shared memory segment is automatically destroyed after the last user application and or the daemon exits normally if the daemon or user applications are aborted the shared memory segment will remain it will be reinitialized on subsequent use shmid Permissions If the daemon is initiated before any user applications then the shared memory seg ment will be created with the
157. aceud help version ipldisable lockdisable filewrap bytes bufferwrap memsi ze count timeout seconds cutoff percent clock source reset quit quit stats join disable ID ID enable D ID trace file You can abbreviate all ntraceud options to their shortest unambiguous length but most of the examples in this manual use the long option name These options are case insensitive The following examples are all equivalent ntraceud help ntraceud hel ntraceud he ntraceud h ntraceud H ntraceud HE ntraceud Hel ntraceud HELP You can invoke ntraceud more than once with different options during a single trace session each invocation passes additional options and values to the running ntraceud daemon Usually you do this to dynamically enable or disable trace events or to obtain current statistical information Options that are available only at ntraceud start up are described that way The following sections discuss the nt raceud options Generating Trace Event Logs with ntraceud Option to Get Help help The ntraceud help option displays the ntraceud invocation syntax on standard output SYNTAX ntraceud help DESCRIPTION The ntraceud help option displays a brief help message showing the complete invocation syntax for ntraceud Screen 4 1 shows an example of help option output GA e ntraceud help version ipldisable lockdisable gt
158. age 9 55 DESCRIPTION The node name function returns the name of the system from which a trace event was logged SYNTAX node name QE PARAMETERS QE RETURN TYPE string SEE ALSO A user defined qualified event If supplied the function returns the name of system from which the last instance of the trace event which satisfies the conditions for the specified qualified event was logged If omitted the function returns the name of the system from which the current trace event was logged For more information see Qualified Events on page 9 81 start node name on page 9 45 offset node name on page 9 68 and end node name on page 9 56 DESCRIPTION The process name function returns the name of the process PID associated with a trace event Using Expressions SYNTAX process name QE PARAMETERS QE A user defined qualified event If supplied the function returns the name associated with the PID of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the name asso ciated with the PID of the current trace event For more infor mation see Qualified Events on page 9 81 RETURN TYPE string SEE ALSO offset_process_name on page 9 69 task_name DESCRIPTION The task name function returns the name of the task associated with a trace event NOTE This function is only meaningfu
159. ame of the current session configuration file Session configuration file New l TR m T Figure 3 14 Session Configuration File Name Area Session configuration file The name of the current session configuration file If the current session configura tion has not yet been saved to a file New will be displayed in this area To save the current session configuration to a file select either the Save Session see Save Session on page 3 3 or Save Session As see Save Session As on page 3 4 menu item from the NightTrace menu see NightTrace on page 3 2 Daemon Details Area The area located directly beneath the Session Configuration File Name Area displays information about the daemons defined in the current session NightTrace Manual Type Daemon Name Streaming Logged Figure 3 15 Daemon Details Area Type Indicates what type of trace events the daemon is logging U indicates that the associated daemon is logging user trace events K indicates that the associated daemon is logging kernel trace events The type of trace event that the daemon is logging is configured by selecting either the Kernel or the User Application radiobutton in the Trace section on the General page of the Daemon Definition dialog see General on page 3 28 Daemon Name The name of the daemon as configured in the Name field on the General page of the Daemon Definition dialog see Name on page 3 29
160. an run ntrace with multiple event map files however if a trace event ID is multiply defined nt race writes an error message in the message display area of the Global Window For more information on the Global Window see ntrace Global Window on page 5 25 TIP If you used symbolic constants to represent numeric trace event IDs you may be able to simply reformat this information in the event map file To load an existing event map file either Start an ntrace session with the file name as an argument e Click on File Read Event Map File on the Global Window You must create an event map file with a text editor before you invoke ntrace The file contains lines of ASCII text separated by newlines There is one trace event tag mapping per line White space separates each field except the conversion specifications commas separate the conversion specifications ntrace ignores blank lines and treats text following a as comments The syntax for the trace event mappings in the event map file follows event ID event tag nargs conv spec Fields in this file are event The keyword that begins all trace event name mappings ID A valid integer in the range reserved for user trace events 0 4095 inclusive Each time you call a NightTrace trace event logging routine you must supply a trace event ID event tag A character string to be associated with event ID Trace event tags must begin with a letter and consist
161. arch from the current time through the end or beginning of the trace session This is the default setting Click on Interval Search to search only between this interval s Time Start and Time End The Interval Manipulation radio buttons let you choose the action nt race takes if a trace event meets all your criteria This decision can affect both the grid and the interval control area e Click on Scroll Current Time to Event if you want nt race to set the current time to the time when the trace event occurred and move the interval This is the default setting e Click on Zoom to Include Event to zoom out the interval end time for forward searches or the interval start time for backward searches to include the found trace event Clicking this radio button also updates the current time e Click on Do Not Move Current Time if you want nt race to just write a message to the message display area of the display page without repositioning you on the grid or in the interval control area a side effect of this setting is that repeatedly clicking on the Search push button does not find trace events after the first one found This is because the current time has not changed Search Form Push Buttons Following is a summary of the effects of clicking on the push buttons in the Search Form Apply Reset Prev Next default Validates any field change s on the Search Form Clicking on Apply is equivalent to pressing lt Ente
162. args on page 9 21 end num args on page 9 48 and offset num args on page 9 61 DESCRIPTION The start pid function returns the global process identifier PID associated with the start event of the most recent instance of a state NOTE A global process identifier does not have the same meaning as the typical operating system definition of pid A PID within Night Trace is a 32 bit integer value that contains the operating system process identifier raw PID in the upper 16 bits and the light weight process identifier LWPID in the lower 16 bits Consult the lwp global self 2 man page for more information SYNTAX start pid QS NightTrace Manual PARAMETERS QS RETURN TYPE integer SEE ALSO A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 pid on page 9 22 raw pid on page 9 23 Iwpid on page 9 23 end pid on page 9 48 and offset pid on page 9 62 start raw pid DESCRIPTION The start raw pid function returns the process identifier raw PID associ ated with the start event of the most recent instance of a state NOTE A NightTrace raw PID has the same meaning as the typical oper ating system definition of pid See the getpid 2 man page for more infor
163. ark zoom Region Zoom In Zoom Out Refresh Figure 11 13 TR_SWITCHIN vs TR_PAGEFLT_ADDR and TR_PROTFLT_ADDR Events Syscall Information Last Syscall Name Current Time Line E ys Bc bl pde llli Context Switch Lines Associated Device if any Syscalls Figure 11 14 Last Syscall Box and Syscall Graph Figure 11 14 shows a last syscall box and a syscall graph The syscall graph displays a state that is drawn whenever a system call syscall is executing on the associated CPU Unlike interrupts syscalls cannot nest so they are always graphed with the same height Context switch lines are superimposed on syscall graphs It is common to see a context switch line at what looks like the very end or beginning of a syscall Usually this does not indicate that the syscall has ended only that it has been suspended because the process that originated the syscall has switched out The syscall resumes when the process is switched back in again An example of a syscall being suspended and resumed can be seen at the right end of the syscall graph in Figure 11 14 The last syscall box displays the last syscall prior to the current time line that executed and may still be executing on the associated CPU If the syscall is associated with a device the name of the device is shown after the name of the syscall 11 15 NightTrace Manual Color Information The last syscall box can be used with the syscall graph t
164. art time end time 0 000030 0 000090 1 631870 1 631840 The current time line was positioned inside a current instance of the state instance C The duration of time in seconds between event d and event e The function operated on the most recent instance of the state instance C and the immediately preceding instance instance B The duration of time in seconds between event c and event d The function operated on the last com pleted instance of the state instance B Assuming no other instances of the state preceded those shown in the fig ure The function operated on all com pleted instances of the state which included instances A and B The time associated with event e The function operated on the most recent instance of the state instance C The time associated with event d The function operated on the last com pleted instance of the state instance B Assuming the current time line was positioned at time line z in Figure 9 6 the various instances would be defined as current instance last completed instance most recent instance No current instance is defined since the cur rent time line is positioned on the end event of an instance of a state Instance C Instance C 9 17 NightTrace Manual The table below indicates the information returned by various NightTrace functions assuming the current time line was positioned at time line z in Figure 9 6 sta
165. at has an trace event file in a trace file analysis The name of a system from which a trace event file can come A pre defined dynamically generated string table It is internal to NightTrace and associates node ID numbers with node names A pre defined dynamically generated string table It is internal to NightTrace and associates process identifiers PIDs with process names for a particular node The name of each node s table is pid_nodename where nodename is the node s name If kernel tracing this table is stored in the vectors file A pre defined dynamically generated string table It is internal to NightTrace If user tracing it associates NightTrace thread ID numbers with thread names for a particular node If kernel tracing this table is not used The name of each node s table is t id_nodename where nodename is the node s name An overhead trace event that ntraceud logs at the beginning and end of each process NT_ASSOC_TID NT_CONTINUE ntrace display utility ntraceud object offset offset function OK Open ordinal trace event number panel Glossary An overhead trace event that ntraceud logs at the beginning and end of each thread and Ada task An overhead trace event that ntraceud logs for multi argument trace events The part of NightTrace that graphically displays trace events trace event data and states for debugging and performance analysis The NightTrace daemon process that allows yo
166. ately A valid change keeps Zoom Factor greater than or equal to 1 If you set Zoom Factor to the word default or a space ntrace resets Zoom Factor to the default value 2 6 13 NightTrace Manual Increment A valid change keeps percentages greater than 0 and less than or equal to 100 and absolute numbers greater than 0 microseconds and less than or equal to the end time of the trace session If you set Increment to the word default or a space ntrace resets Increment to the default value 25 If Increment is less than 100 when you click on an interval scroll bar arrowhead you see part of the previous interval in this interval However if Increment is equal to 100 you see a completely new interval For more information on the interval scroll bar see The Interval Scroll Bar on page 6 7 Increment Increment multiplied multiplied by 2 by 2 Increment Increment Figure 6 7 Amount of Scrolling Due to Increment Value Editing Multiple Fields Sometimes it makes sense to change multiple fields for a single effect for example you may wish to change both the Time Start and Time End fields or you may wish to change both the Time Start and Event Count fields In these cases apply your changes only once after you have edited each field of interest Changing some combinations of fields is not meaningful for example you may try to change both Time Length and Event Count When ntrace detects a meaningless co
167. ation Figure 2 3 entry_exit c After Instrumentation This exercise continues in Exercise Logging Trace Events on page 4 29 2 29 NightTrace Manual 2 30 3 NightTrace Session Manager Using the NightTrace Session Manager 0 00 cece eee eee eee ee 3 1 NightTrace Session Manager Menu Bar 00 0000 02sec eee 3 2 NightIrace viv te oP DERI ii 3 2 Open Session tacos ee ee se ee ede ee yeh eee ee EROR RUE 3 4 Saye SESSION Loo P ebo ee es I eb re eh Nae e nik I ku 3 6 Unsaved Changes cairo hera ene p E ra 3 8 Daemons AER EMEN 3 9 LEOBIY mS ERE ARE ERN ee Pee EREMO Rey Sa 3 10 Enter Password cse oo EIVRR REESE E 3 11 Attach Daermons cone poi esp p eM nr eR PE ett ie 3 12 OPONE rec eed e 3 14 Refresh Intervals coe oe aoa muss UE RIDERS Na 3 14 TOONS EIS eee ek Eas eek See ak eb pA iss 3 14 A ANN 3 16 Session Configuration File Name Area oooooooccococooc eee 3 17 Daemon Details ATea miroir ek RAEE aa OM CAS a 3 17 Daemon Control Ar a o oo oococccocoorc en eens 3 20 Enable Disable Trace Events 3 22 Daemon Definition Dialog eleseeeee eee 3 25 Import Daemon Definition 0 0 0 cee eee 3 27 Generali rd 3 28 LU a A de 3 29 Trace Events Output s escoria e eee 3 31 User Trace yikes eats A 3 33 Locking Politica ea 3 34 Shared Memory hu erue ees Se SORES BALES dos 3 36 Timestamp Heartbeat 0 0 eee eee 3 36 User Event Bunter A eee RATER ee eae 3 37 EY
168. ation see Qualified Events on page 9 81 offset time Using Expressions SEE ALSO start cpu on page 9 41 end_cpu on page 9 52 and offset cpu on page 9 66 DESCRIPTION The offset function returns the ordinal number offset of a trace event SYNTAX offset QE PARAMETERS QE A user defined qualified event If supplied the function returns the ordinal number offset of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the ordinal number off set of the current trace event For more information see Qualified Events on page 9 81 RETURN TYPE integer SEE ALSO start offset on page 9 42 end offset on page 9 53 min offset on page 9 73 and max_offset on page 9 73 DESCRIPTION The t ime function returns the time in seconds associated with a trace event Times are relative to the earliest trace event from all trace data files currently in use SYNTAX time QE PARAMETERS QE A user defined qualified event If supplied the function returns the time in seconds of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the time in seconds of the cur 9 27 NightTrace Manual RETURN TYPE rent trace event For more information see Qualified Events on page 9 81 double
169. ation about event map files cp usr lib NightTrace examples entry exit map map Invoke ntrace with the trace event file you created in the last exercise and the configuration file you just created 5 35 NightTrace Manual 5 36 ntrace log page NightTrace presents a display page Concentrate on the dotted grid area in the middle and the row of push buttons at the bottom Keep clicking on the Zoom Out push button until the display quits changing Click on the Ruler around 2 seconds The display object with digital waves is a StateGraph It graphically displays trace events and states The two floating DataBoxes contain textual information about the current trace event and its first argument respectively Notice that the current trace event is identified by its cryptic trace event ID number See StateGraph on page 7 14 and DataBox on page 7 12 for more information about StateGraphs and DataBoxes The next few steps get the same display page to show symbolic tags instead of numeric IDs for trace events Close the display page by clicking on File I Close See Close on page 7 19 for more information about this menu item Read in the event map file named map by clicking on File I Read Event Map File See Read Event Map File on page 5 31 for more information about this menu item Re open the configuration file named page by clicking on File 1 Open Config File See Open Config File
170. atters for the data stored in the trace event file Every time a display object is updated it filters through the data in the trace event file The display object accepts input in the form of a trace event record processes and reformats the information and displays it The following information is in a trace event record numeric trace event ID global process identifier PID NightTrace thread identifier TID time and optional arguments NightTrace also keeps track of the ordinal number offset of a trace event You can use nt race functions to express any of these values For more information about functions see Functions on page 9 9 Although the trace event file contains trace events it also implicitly contains states The concepts of trace events and states are key to understanding display objects trace event Corresponds to the point in the execution of your application when a trace_event call was executed All the data logged at that time trace event ID arguments etc is considered a trace event state A state is bounded by two trace events a start event and an end event An instance of a state is the period of time between the start event and end event including the start and end events themselves Addi tional conditions may be specified in a state definition to further con strain the state Instances of states do not nest that is once a state becomes active events that might normally satisfy the conditions for th
171. ay automatically place a window frame around your windows It may also provide you with a means of performing some standard operations such as minimizing also known as iconifying or maximizing the window size If your window manager provides a window frame then nt race puts a window title in the title area of this frame If you minimize a window ntrace provides all or part of the window title for the icon that represents that window ntrace windows may contain different mixtures of components In ntrace the window components include 5 22 Invoking the ntrace Display Utility Menu bars and pull down menus Push buttons Radio buttons A menu bar appears at the top of the window It consists of labeled pull down menu s When you select click on a pull down menu entries appear in a vertical list Selecting a menu entry with ellipses on it causes a dialog box or form to appear Entries are grouped by function with separators dividing groups Any destructive items appear last A push button is a graphic image of a labeled button on a panel Push buttons are evenly spaced in a horizontal panel at the bottom of the window The default push button is highlighted by having a border around it Pressing lt Enter gt makes the default push button take effect Push buttons are organized by frequency of use and grouped by functionality Push button names are active verbs The most com monly used push buttons in ntrace are Apply Reset and
172. black bg white flat gray75 fn fixed Your X terminal vendor supplies you with vendor specific directories and files that pertain to colors and fonts The file that contains available colors is called rgb txt The directory for fonts is usr lib X11 fonts For more information on X options see xterm 1 orX 1 ntrace Arguments 5 8 You can invoke ntrace with arguments that provide information about trace events their tags other labels and desired display object layout ntrace identifies the purpose of a file argument by its contents therefore the order and number of these arguments is not significant Invoking the ntrace Display Utility SYNTAX ntrace option trace files event map files config files session files ARGUMENTS trace files Trace event files contain sequences of trace events logged by your application or the operating system kernel and capture by a user or kernel trace daemon event map files Event map files map short mnemonic trace event tags to numeric trace event IDs and associates data types with trace event arguments This is a hand edited ASCII file config files Configuration files define macros qualified events qualified states string tables format tables display objects and display pages These ASCII files are usually created with ntrace session files Session manager configuration files define a list of deamon sessions and their individual configuration for quick invoca tion These files ar
173. bseqently Enable gt or Disable gt it The user may use the event name associated with the event type e g TR_SYSCALL_RESUME or the numerical value of the trace event type e g 4131 The user may also enter a range of values either using the event names or their numerical values e g TR INTERRUPT ENTRY TR EXCEPTION EXIT or 4112 4117 Use Event Names Allows the user to view the event names of the trace event types in the Disabled Events and Enabled Events lists instead of their numerical values For kernel events these mappings are provided in the file usr lib NightTrace eventmap which is automatically loaded by the NightTrace Session Manager For user trace events the user may load user defined event map files using the Load Event Names dialog see Load Event Names on page 3 39 Use the Load Event Names button to access this dialog Load Event Names Allows the user to load various user defined event map files by presenting the Load Event Names dialog see Load Event Names on page 3 39 The user may then view the event names of the user trace event types in the Dis abled Events and Enabled Events lists instead of their numerical values by checking the Use Event Names checkbox on this dialog Load Event Names This dialog allows the user to load various user defined event map files thereby enabling the user to view the event names of the user trace event types instead of their numerical values in t
174. buffer to the trace event file If you use the bufferwrap option you must start up ntraceud with it NOTE You control automatic buffer flushing by setting the ntraceud sleep interval and buffer full cutoff percentage in expansive default mode and in file wraparound mode In contrast there is no automatic buffer flushing in buffer wraparound mode these values have no effect in this mode Invoking ntraceud with the bufferwrap option makes ntraceud ignore any time out and cutoff options In buffer wraparound mode you can estimate the maximum number of trace events to be written to your trace event file by using the following formula max events max events in buffer flush count where max events The maximum number of trace events 4 13 NightTrace Manual 4 14 max_events_in_buffer The number of trace events the shared memory buffer can hold You can set this value when you invoke ntra ceud with the memsize option flush_count The number of trace_flush calls your application executes For example if you set your shared memory buffer size to 1000 trace events then max events in buffer is 1000 If you expect your three trace_flush calls to execute two times each then flush_count is six 3 2 Calculating max_events gives you about 6000 1000 6 trace events in your trace event file Buffer wraparound mode Can help you with debugging Can reduce trace events to a manageable number May
175. bugging and monitoring tool specifically designed for real time applications NightView can monitor debug and patch multiple real time processes running on multiple processors with minimal intrusion See also e NightView User s Guide 0890395 The Help menu appears on the NightTrace Session Manager menu bar see NightTrace Session Manager Menu Bar on page 3 2 Figure 3 13 Help Menu On Window Displays the help topic for the current window On Item Gives context sensitive help on the various menu options dialogs or other parts of the user interface Help for a particular item is obtained by first choosing the On Item menu option then clicking the mouse pointer on the object for which help is desired the mouse pointer will become a floating question mark when the On Item menu item is selected In addition context sensitive help may be obtained for the currently highlighted option by pressing the F1 key The HyperHelp viewer will display the appropriate topic NightTrace User s Guide Opens the online NightTrace User s Guide NightTrace Session Manager Bookshelf Opens a HyperHelp window that lists all of the Concurrent online publications cur rently available on the local system About NightTrace Displays version and copyright information for the NightTrace product Session Configuration File Name Area The area located directly beneath the NightTrace Session Manager Menu Bar displays the n
176. can use pre defined format tables anywhere that format tables are appropriate Use the get_format function to look up values in format tables For information about the get_format function see get_format on page 9 79 For examples of func tion calls with format tables see Table 8 3 Configuring Display Pages 5 20 The configuration file usually contains display page s You use nt race to put these display page s in your configuration file ntrace lets you customize the layout of your ntrace display pages You do this by coloring selecting sizing positioning grouping and otherwise configuring display objects on a particular display page Once you have created a useful display page you may save it for future ntrace sessions Saving a display page is the same as creating or modifying a configuration file See Chapter 7 and Chapter 8 for more information Invoking the ntrace Display Utility Rather than creating your own display page you can let nt race create a default display page for you For more information on the default display page see Default Page on page 5 28 and Kernel String Tables on page 11 16 ntrace User Interface ntrace displays textual and graphical information and it provides you with ways to manipulate this information These displays and mechanisms make up the nt race user interface The next sections describe the following nt race user interface issues Using the mouse Under
177. cation Links in the NightTrace library Creates an executable named a out if there were no major errors For more information on compiling and linking h 77 programs see the hf77 Fortran Reference Manual Ada Example For a complete example on accessing the NightTrace library routines from an Ada appli cation see the section titled NightTrace Binding in the MAXAda Reference Manual Exercise Instrumenting Code Putting library calls in your application is called instrumenting your code The following application is in usr lib NightTrace examples entry_exit c 2 25 NightTrace Manual ude lt sys types h gt include lt time h gt include lt stdio h gt void take_a_nap sleep_str struct timespec sleep_str printf Sleeping for 3f seconds n float sleep_str tv_nsec 1e 09 nanosleep amp sleep str NULL make the spacing between states obvious Sleep str tv nsec 30000000 nanosleep amp sleep str NULL main int 1 struct timespec sleep str for i20 i 10 sleep_str tv_nsec 1000 1000000 take_a_nap sleep_str Figure 2 2 entry_exit c Before Instrumentation Make a copy of this file in your directory and call it entry_exit c Make the following changes by inserting trace event library calls at appropriate places in the application Start the trace session and log trace events to a file named log Open a thread nam
178. cations modify zero or more args An optional expression to be formatted and displayed NOTE NightTrace does not perform semantic error checking of func tions For example if you ask for information about the second argument but no second argument was logged NightTrace does not tell you Similarly NightTrace does not flag the use of unde fined macros qualified events and qualified states it temporarily puts their names in the appropriate Dialog Box in case you want to configure these constructs For more information about these Dialog Boxes see Expression Dialog Boxes on page 9 2 Function Terminology In order to use the NightTrace functions effectively it may be useful to understand some of the concepts associated with them Remember that an event or trace event is either a user defined point of interest in an application s source code or a predefined point of interest in the kernel In addition a state is defined to be a region of source code bounded by two events 9 13 NightTrace Manual The descriptions of the functions further speak in terms of instances of states These are best defined as current instance The instance of a state which has begun but has not yet completed Thus the current time line would be positioned within the region from the start event up to but not including the end event last completed instance The most recent instance of a state that has already completed Thus the
179. ce The NightTrace menu contains session related items such as initiating a new session saving the current session to a configuration file and opening a previously saved configu ration file 3 2 NightTrace Session Manager The NightTrace menu appears on the NightTrace Session Manager menu bar see NightTrace Session Manager Menu Bar on page 3 2 Figure 3 2 NightTrace Menu New Session Creates a new session If an existing session is open it is first closed by this operation If changes have been made to the current configuration but have not yet been saved the Unsaved Changes dialog is presented to the user see Unsaved Changes on page 3 8 Open Session Opens a previously saved session configuration file Selecting this option brings up the Open Session dialog see Open Session on page 3 4 If an attempt is made to open a previously saved session configuration file when changes have been made to the current configuration but have not yet been saved the Unsaved Changes dialog is presented to the user see Unsaved Changes on page 3 8 Save Session Saves the current configuration to the current session configuration file The name of the current session configuration file appears in the Session Configuration File Name Area see Session Configuration File Name Area on page 3 17 If the current configuration has not yet been saved the name in the Session Config uration File Name Are
180. ce filestats Output esses 2 6 Table 4 1 NightTrace Configuration Defaults 0 00008 2 4 Table 4 2 Mode Selection Guidelines 0 00 00 cece eee eee 2 5 Table 4 3 NightTrace Operating Modes 0 0 0 0 eese 2 5 Table 4 4 ntraceud Disable Sequence 1 0 00 00 c eee eee ee 2 25 Table 4 5 ntraceud Disable Sequence 2 lesse eee 2 25 Table 4 6 ntraceud Enable Sequence 1 00 0 0 eee eee 2 27 Contents Table 4 7 ntraceud Enable Sequence 2 eee eee ee eee 2 27 Table 5 1 ntrace Pointer Shapes and Their Usage 00000 2 22 Table 6 1 View Mode Mouse Button Operations 004 2 3 Table 6 2 Manipulating the Interval Scroll Bar o ooooooooooooo oo 2 8 Table 6 3 Valid Multiple Field Changes 0 0 00 0000002 2 15 Table 6 4 Making Editing Changes 0 0 0 0 cece eee eee ee 2 16 Table 6 5 Positioning Within a Text Field eese 2 18 Table 7 1 Edit Mode Mouse Button Operations 00008 2 5 Table 8 1 Common Configuration Parameters 0000000008 2 2 Table 8 2 Examples of If Expressions l l 2 5 Table 8 3 Examples of Then Expressions eee esee 2 5 Table 9 1 Time Units and Constant Suffixes 00 0 0 00 0 0 00 00008 2 6 Table 9 2 A Comparison of Functions and Macros 0 00005 2 6 Table 9 3 NightTrace Functions
181. ce event 9 69 thread identifer ending trace event 9 55 thread identifer offset 9 68 thread identifer starting trace event 9 44 thread identifier table name 9 29 thread name 9 32 thread name ordinal trace event 9 70 TID table name 9 29 timestamp 1 2 5 4 9 27 9 42 9 54 9 66 timing distortion 2 21 4 17 TR_EXCEPTION_ENTRY 11 3 TR_EXCEPTION_EXIT 11 3 TR_EXCEPTION_RESUME 11 3 TR_EXCEPTION_SUSPEND 11 3 TR_INTERRUPT_ENTRY 11 2 TR_INTERRUPT_EXIT 11 3 TR_PAGEFLT_ADDR 11 5 11 13 TR_PROTFLT_ADDR 11 5 11 13 TR_SWITCHIN 11 2 TR_SYSCALL_ENTRY 11 4 TR_SYSCALL_EXIT 11 4 TR_SYSCALL_RESUME 11 4 TR_SYSCALL_SUSPEND 11 4 Trace event see Event Trace file see Trace event file Trace point 1 2 1 4 2 13 5 9 trace_close_thread 2 22 trace_disable 2 16 4 24 trace_disable_all 2 16 2 24 trace_disable_range 2 16 4 24 trace_enable 2 16 4 26 Index 15 NightTrace Manual race enable all 2 16 race enable range 2 16 4 26 race end 2 7 2 20 2 23 4 4 4 17 4 21 race event 2 11 7 8 race event arg 2 11 race event dbl 2 11 race event flt 2 11 race event four arg 2 11 race event two flt 2 11 race flush 2 20 4 4 4 5 4 13 4 14 4 17 4 28 CE at E coc ACR cb ct SG ct A 2 A 3 trace open thread 2 9 2 15 2 18 2 22 6 4 7 17 8 3 trace start 2 5 2 10 2 15 2 18 2 23 4 1 4 28 C 1 trace trigger 2 20 4 4 4 17 A 2 A 3 A 4 Trace Event File radio button 10 6 Tracing disabli
182. ce event 9 69 task_id function 9 25 task_name function 9 31 Text configuration parameter 8 12 Text field 8 12 Background Color 8 3 CPU List 8 2 9 81 9 83 Glossary 14 Current Time 6 7 6 11 6 13 cursor 6 16 editing operations 6 16 End Events 8 15 9 7 9 85 End Expression 8 15 9 4 9 7 9 83 10 7 Glossary 14 Event Color 8 15 Event Count 6 12 6 13 Event End 6 7 6 12 6 13 Event List 8 2 9 83 9 85 10 5 10 7 Event Start 6 7 6 11 6 13 Expression 9 7 File Name 5 33 5 34 Index 14 Filter 5 35 Filter Expression 9 4 10 7 Font 8 3 Foreground Color 8 3 If Expression 8 2 9 4 9 7 9 81 9 83 9 85 10 7 10 8 Glossary 14 Increment 6 7 6 12 6 14 Lost Event Color 8 19 MacroDefinition 9 7 Mark Color 8 19 Maximum 8 18 Minimum 8 18 Name 8 2 No Event List 10 5 Node List 8 3 9 81 9 83 Glossary 14 PID List 8 3 9 81 9 83 Glossary 14 QualifiedEvent 9 82 QualifiedState 9 84 Start Events 8 15 9 7 9 85 Start Expression 8 15 9 4 9 7 9 83 10 8 Summary Expression 9 4 9 81 10 7 10 8 10 11 10 12 Text 8 12 Then Expression 8 2 TID List 9 81 9 83 Glossary 14 Time End 6 7 6 11 6 12 10 3 10 6 Time Length 6 11 6 13 Time Start 6 7 6 11 6 12 10 3 10 6 Zoom Factor 6 12 6 13 Text Gravity configuration parameter 8 4 8 10 Text Gravity radio button 8 4 Text Justify configuration parameter 8 3 8 9 Text Justify radio button 8 3 Then Expression configuration parameter 8 2 8 5 9 4 9 76 9 78 9 79 9 80 Then E
183. ce event file s Normally it then creates and displays a default display page with one StateGraph per process in the trace event file s The StateGraph is configured to show all user trace events If there are so many processes that their StateGraphs do not fit on the screen then nt race does not display any StateGraphs When analyzing trace events timestamped by an RCIM synchronized tick clock if a thread name is not unique in the trace events nt race prints a node name along with the process ID number and thread ID number in the associated GridLabel to identify that thread Figure 5 5 shows a default display page A brief description of the display objects on this display page follows See Chapter 7 for a more detailed description of display objects and the display page Invoking the ntrace Display Utility NOTE The dynamic information that display objects present relates to the interval a window into your trace session Time Start 0 0000000s Time Length 2 2504284s Time End 2 2504284s Event Start o Event Count 4 Event End s Zoom Factor 2 0 Increment 25 007 Current Time 1 1252140s Apply Reset Center Mark Zoom Region Zoom In Zoom Out Refresh Figure 5 5 A Default Display Page GridLabel A textual display object that contains a static user specified text string This object labels other objects for clarity DataBox A display object that displays textual or numeric information such as the trac
184. ception 11 13 last interrupt 11 12 last syscall 11 15 Node 11 11 running process 11 10 Buffer wraparound mode 2 21 4 4 4 13 4 17 4 18 A 1 A 4 Button dimmed see Button disabled disabled 5 27 7 4 grayed out see Button disabled C Clanguage 1 3 compiling and linking 2 25 source considerations 2 1 C thread identifier 8 8 Cancel push button 5 30 5 31 Center push button 6 9 Clear push button 10 9 10 10 clock_settime 3C routine 2 1 C 1 clock_synchronize 1M command 2 8 Close menu item 7 19 10 10 Close push button 8 11 9 2 9 3 10 4 10 5 Color display 11 16 B 1 B 4 Column 5 29 6 2 6 3 6 4 7 5 7 9 7 13 8 1 9 7 B 4 moving 7 5 resizing 7 5 selecting 7 5 Comments configuration file 5 12 event map file 5 10 Common configuration parameters 8 1 Configuration file 5 9 5 11 Configuration form 8 4 8 10 8 12 8 13 8 14 8 16 8 17 8 19 9 4 DataBox 8 13 DataGraph 8 17 Index 1 NightTrace Manual EventGraph 8 16 Expression 9 2 GridLabel 8 12 Macro 9 6 9 7 Qualified Event 9 81 Qualified State 9 83 9 84 Ruler 8 19 StateGraph 8 14 Configuration form radio button Text Gravity 8 4 Text Justify 8 3 Configuration parameters Background Color 8 3 8 9 common 8 1 CPU List 8 2 8 6 End Events 8 15 End Expression 8 15 Event Color 8 15 Event List 8 2 8 4 Expression 9 4 9 7 Fill Style 8 17 Font 8 3 Fonts 8 9 Foreground Color 8 3 8 9 If Expression 8 2 8 4 Lost Event Color 8
185. ception box displays the name of the last exception that executed it is possible for there to be no exceptions visible on the exception graph even though the last exception box contains a valid exception name This just signifies that the last exception on the CPU ended prior to the beginning of the current interval The last exception box is a DataBox display object and the last exception graph is a State Graph display object See Chapter 7 and Chapter 8 for more information on creating and configuring DataBoxes and StateGraphs Lines indicating TR_PAGEFLT_ADDR and TR_PROTFLT_ADDR events are also superim posed on exception graphs Exception graphs display these trace points to allow you to obtain a formatted dump of them in the message display area by clicking on the events with mouse button 2 An example of a TR_PAGEFLT_ADDR and a TR PROTFLT ADDR event as well as their associated data in the message display area can be seen in Figure 11 12 11 13 NightTrace Manual NightTrace ntracBAAa001cB offset 1451 id TR_PROTFLT_ADDR pid inetd tid 3040 cpu 1 time 9 459738s argl 0xb0172b04 arg2 0xb00fecbB arg3 0x0 offset 1461 id TR_PAGEFLT_ADDR pid inetd tid 304 0 cpu 1 time 9 460050s argl Oxelf00000 arg2 0x96e6c arg3 0x1 resched strsched a data access ee ie inetd open file open file A E 3 453 9 4558 34593 3 46 3 46 LE sert Bai Sea saa pene 3 4602023s 1452 3 45938387s Figure 11 12 TR PAG
186. cified the maximum value will be the greatest value found in the trace event run up to that point in time Note that the maximum can change as time increases and new maximums are encountered The Minimum parameter determines what data value corresponds to the bottom of the DataGraph The possible values are integers or CALC If an integer is specified as the minimum any data that is equal to or less than that value will result in no line or bar on the DataGraph If CALC is specified the minimum value will be the smallest value found in the trace event run up to that point in time Note that the minimum can change as time increases and new minimums are encountered Figure 8 11 shows the same set of data drawn in three DataGraphs each configured differently The data range in value from 1 to 6 and are shown at the bottom of the figure The top DataGraph is configured with a minimum of 2 and a maximum of 4 Notice that several bars reach the top of the DataGraph even though they represent different data values also note that there is no bar where data has a value less than the minimum The middle DataGraph is configured with a minimum of 0 and a maximum of 10 Notice that the bars do not reach the top of the DataGraph and that the differences between values are harder to discern The bottom DataGraph is configured with a minimum of 0 and a maximum set to CALC Notice that the two occurrences of the maximum value of six cause bars to reach
187. cking on Summarize without making any field or radio button changes makes ntrace summarize all trace events in your trace event file s If you want to restrict the summary by trace event ID trace event tag CPU number node process name or global process identifier thread name or NightTrace thread identifier or expression you can do that by editing one or more of the fields on the Summarize Form When you have finished editing the Summarize Form fields press Enter or click on Apply This causes ntrace to validate the data in each field you modified For general information on field editing and how ntrace handles editing errors see Field Editing on page 6 16 When you are ready for ntrace to summarize data click on the Summarize push button nt race logical ORs comma separated lists of values within a field and logical ANDs fields values This means that a summary object must match at least one Using the Built In Tools entry in each list and all criteria from the fields Every time you click on Summarize ntrace writes lines of statistics in the summary display area For more information on the Summarize push button see Summarize Form Push Buttons on page 10 8 The text fields on the Summarize Form differ depending on the selected summary type See Chapter 8 for a definition of each field except those described below and all its possible values There is only one difference between the ntrace behavior described there an
188. cking on the Start button in the NightTrace Session Manager window Once the state displayed in the session manager for the daemon changes to Paused click the Resume button Allow the daemon to capture data for a few seconds then click on the Flush button followed by the Stop button To display the kernel data click on the Display button This will cause a default kernel page to pop up Repeatedly click on the Zoom Out button on that page until you see data in the display pane Note if any display page is already open clicking the Display button will not automatically 1 5 NightTrace Manual create a kernel display page In such a case open a default kernel page from the main NightTrace dialog The Trace Event Analysis Procedure Display Page Hm garcha 205970 Ihread thMead1 cap B15 0 l E naro 0 8 1 s 2 s as User Events Time Start 0 0000000s Time Length 4 8261836s Time End 4 8261896s Event Start o Event Count 3 Event End 8 Zoom Factor 2 0 Increment 25 002 Current Time 1 2065472s Apply Reset Center Mark Zoom Region Zoom in Zoom out Refresh L Figure 1 2 Example of a User Display Page with Display Objects 5 Iteratively locate and analyze significant data Search for trace events of interest You do this by controlling the window that displays a portion of the trace event file This window is called the interval
189. conserve disk space Buffer wraparound mode can be useful in debugging Assume that you are debugging a fault in a large application Follow the steps below to accomplish your task 1 Inserta trace flush call in your code where you believe the fault occurs 2 Compile and link your application 3 Invoke ntraceud with the bufferwrap option 4 Run your application When your application executes the trace 1lush call ntraceud copies all trace events still in the shared memory buffer to the trace event file You can then use the ntrace display utility to graphically analyze only the trace events immediately preceding the fault Buffer wraparound mode can also be useful in reducing trace events to a manageable number In this mode when the shared memory buffer is full the newest trace events overwrite the oldest ones This means that if the shared memory buffer becomes full before your application executes the trace 1ush call ntraceud copies only the current contents of the buffer to the trace event file This way you can exclude the oldest trace events from your ntrace displays In buffer wraparound mode ntraceud usually flushes fewer trace events to the trace event file than in expansive mode Thus this mode can conserve disk space If you want to determine how much disk space is available run the d 1 command with the k option and look at the avail column Use the following command to see the system setting
190. criptions of the common config uration parameters that DataBoxes use To determine the format of the data displayed in the DataBox give the Then Expression parameter an expression value See Then Expression on page 8 5 for examples exception TR_EXCEPTION_ENTRY TR EXCEPTION RESUME TR EXCEPTION EXIT TR EXCEPTION SUSPEND TR SHITCHIN LL ITR EXCEPTION ENTRY TR EXCEPTION RESUME ITR EXCEPTION EXIT TR EXCEPTION SUSPEND TRUE TRUE EA RA oo ooo TRUE A D E rc Figure 8 7 StateGraph Configuration Form The configuration form for the StateGraph is shown in Figure 8 7 NOTE The Node List field appears in this dialog only when NightTrace is configured to use an RCIM to timestamp events Configuring Display Objects A state is bounded by two user specified trace events a start event and an end event An instance of a state is the period of time between the start event and end event including the start and end events themselves Instances of the same state do not nest thus once a state becomes active events that might normally satisfy the conditions for the start event are ignored until the end event is encountered The Start Events parameter determines the trace events that can begin a state This parameter along with the End Events parameter defines part of what will be consid ered a state for this display object These parameters work exactly like the Event List parameters discussed earlier in C
191. ction may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 node id on page 9 28 offset node id on page 9 67 and end node id on page 9 54 9 43 NightTrace Manual start_pid_table_name start_tid_table_name 9 44 DESCRIPTION The start pid table name function returns the name of the inter nally assigned NightTrace process identifier table PID table associated with the start event of the most recent instance of a state SYNTAX start pid table name 0S PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE string SEE ALSO pid table name on page 9 29 offset pid table name on page 9 67 and end pid table name on page 9 55 DESCRIPTION The start tid table name function returns the name of the inter nally assigned NightTrace thread identifier table TID table associated with the start event of the most recent instance of a state SYNTAX start tid table name QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that stat
192. current time line in a Column Divides Time Length by the value of Zoom Factor this provides a microscopic view of a smaller interval Makes corresponding changes to Time Start Time End Event Start Event Count and Event End Zoom Out Centers the interval around the current time line in a Column Multiplies Time Length by the value of Zoom Factor this provides a macroscopic view of a larger interval Makes corresponding changes to Time Start Time End Event Start Event Count and Event End Refresh 6 10 Updates the grid to reflect the result of changes in configuration Is implicit with any action that updates the grid Should be used when you Viewing Trace Event Logs with ntrace Open a display page Switch to View mode from Edit mode Change a configuration parameter from View mode Resize the grid e Differs from the X window manager s Refresh which redraws the windows without notifying ntrace The Interval Control Area Reading Fields The interval control area is a region of the display page that contains nine fields of statistics If you have not already positioned your interval somewhere else ntrace displays in the interval control area the earliest 5 percent of your trace session Usually this information is uninteresting and you want to see other parts of your trace session You can do two things with the statistics in the interval control area Read the fields to obtain information about t
193. d here and specific features appear in later sections 9 2 Using Expressions The push buttons on a Configuration Form appear in Figure 9 3 Apply Reset Restore Close Figure 9 3 Configuration Form Push Buttons A description of these push buttons follows Apply default Validate the changes you made to the configuration parameters and apply the changes to the selected expr This is equivalent to pressing lt Enter gt Reset Discard all changes made since the last Apply or lt Enter gt This is equivalent to pressing lt Esc gt Restore Discard all changes made since the window was opened Close Discard any changes made since the last change was applied and close the window When you have finished editing the fields on the Configuration Form press lt Enter gt or click on Apply This causes NightTrace to validate the data in each field you modified For general information on field editing and how NightTrace handles editing errors see Field Editing on page 6 16 exprs are saved in a configuration file but are global to all display pages That is if an expr is created in one display page it may be used by any other display page This means how ever that if an expr is saved in one configuration file but altered in another you will have to reopen the file with the original copy of the expr and save the new value NightTrace prevents you from creating more than one definition for a specific expr
194. d the behavior of the Summarize Form on the Summarize Form ntrace textually summarizes all data rather than displaying individual values that meet the criteria The following text describes fields specific to the Summarize Form their possible and default values and how the Summarize push button behaves when you modify that field Filter Expression This text field has all the characteristics of If Expression except it is evaluated only if the If Expression for trace event summaries or End Expression for state summa ries are true Values may be a boolean nt race expres sion the word TRUE or the word FALSE The default is TRUE When you click on Summarize ntrace evaluates the expression for every trace event it summarizes A FALSE in this field essentially disables the summary Summary Expression This text field is evaluated every time the If Expression or End Expression and Filter Expression configuration criteria for your summary are met It lets you specify the format of the summary text Values may be a call to the format or get_format function The default is a get format event summary call for trace event summaries For state summaries the default isaget format state summary call For more information about these format tables see Pre Defined Format Tables on page 5 20 For example if you wanted to limit your summary to trace events with a first argument value between 5 and 100 your If Expres
195. d to trigger a context switch to run the highest priority process that is ready to run An interrupt used to process system callout queue entries An interrupt that usually indicates an unreported or already removed interrupt This interrupt appears only in kernel traces An interrupt indicating that a fatal hardware condition has been detected A user level interrupt See iconnect 3C for a description of enabling user level interrupts An inter processor interrupt used for cache flushing delivering exceptions to another processor performance monitoring and halting processors For more information about interrupts see intstat 1M and uistat 1M 11 19 NightTrace Manual Device Related Interrupts Exceptions 11 20 The names printed for device interrupts correspond to the device names in the system configuration files See System Administration Volume 2 for information on adding devices to a system Table 11 3 provides an alphabetical list of the most common device specific interrupts For more information on a device specific interrupt refer to the documentation associated with the particular device Table 11 3 Device Related Interrupt Reference Interrupt Description consintr A console terminal interrupt eg An Eagle ethernet controller interrupt eti_intr An edge triggered interrupt ex An Excellan ethernet controller interrupt gpib An IEEE 488 GPIB controller interrupt hardclock A 60 Hertz clock
196. dObject foreground navyblue Exercise Customizing Display Colors Edit your Xdefaults file so it defines background colors for the following display objects Suggested colors are provided Table B 2 Suggested Colors for X Resources Display Object Suggested Color Column CornflowerBlue DataGraph PowderBlue StateGraph LightSteelBlue Ruler PaleGreen DataBox Aquamarine GridObject SkyBlue B 5 NightTrace Manual A possible solution follows Nt race Color Column background CornflowerBlue Ntrace Color DataGraph background PowderBlue Ntrace Color StateGraph background LightSteelBlue Ntrace Color Ruler background PaleGreen Ntrace Color DataBox background Aquamarine Ntrace Color GridObject background SkyBlue To test your entries at an X terminal invoke nt race with the log trace event file and bring up the default display page B 6 0 0 7O 2 0 an C Answers to Common Questions What can I do if trace events are not logging at all Verify that the trace event file name on the trace begin call matches the one on the user daemon invocation Furthermore check that the file exists and that you have permission to read and write it Additionally be sure your thread name contains no embedded spaces or punctuation including periods See trace begin on page 2 5 and trace open thread on page 2 9 for more information When should I log a different trace event ID number Each
197. depending upon system activity protection faults may occur reasonably frequently This trace event has three arguments arg The data address which caused the protection fault If the protection fault occurred on an instruction then this will be set to zero arg2 The program counter value at the time of the protection fault arg3 The flag indicating whether the fault occurred on a kernel address or on a user address A value of zero indicates that the fault occurred on 11 5 NightTrace Manual a user address A value of one indicates that the fault occurred on a kernel address Viewing Kernel Trace Event Files All of the kernel trace event tags are defined in the usr lib NightTrace eventmap file This file is automatically read by ntrace at start up time You may design your own display pages to view kernel trace event files see Chapter 7 and Chapter 8 for more information Alternatively you may use pre defined kernel display pages These pages are discussed in the following sections Kernel Display Pages Figure 11 1 shows the File menu of the ntrace Global Window This menu contains a Default Kernel Page menu item which is used to open a dynamically built kernel dis play page The Default Kernel Page menu item is enabled only if a converted Kernel Trace trace event file has been supplied to ntrace on the command line Figure 11 1 Global Window File Menu Figure 11 2 shows a sample kernel display page in View mode c
198. diately preceeding occurrence SYNTAX event gap QE 9 32 Using Expressions PARAMETERS QE A user defined qualified event If supplied the function calclu ates the gap between the two most recent occurrences of events which satisfy the conditions of the specified qualilfied event If omitted the function calculates the gap between the current trace event and the event immediately preceeding it For more information see Qualified Events on page 9 81 RETURN TYPE double precision floating point SEE ALSO time on page 9 27 state_gap on page 9 57 and state dur on page 9 57 event matches DESCRIPTION The event matches function returns the number of occurrences of a trace event on or before the current time line SYNTAX event matches QE PARAMETERS QE A user defined qualified event If supplied the function calcu lates the number of occurrences of events which satisfy the conditions of the specified qualified event on or before the cur rent time line If omitted the function calculates the number of occurrences of all events on or before the current time line For more information see Qualified Events on page 9 81 RETURN TYPE integer SEE ALSO summary matches on page 9 74 9 33 NightTrace Manual State Functions Start Functions 9 34 In its simplest form a state is a region of source code bounded by two trace events A state definition requ
199. down Ctrl and clicking on mouse button 2 causes ntrace to write a statistic in the message display area that tells how far your pointer is from the mark A positive number means your 6 9 NightTrace Manual pointer is to the right of the mark A negative number means your pointer is to the left of the mark Simultaneously holding down lt Ctrl gt and clicking on mouse button 3 causes ntrace to write a statistic in the message display area that tells how far your pointer is from the current time line A positive number means your pointer is to the right of the current time line A negative number means your pointer is to the left of the current time line Simultaneously holding down lt Ctrl gt holding down mouse button 1 and dragging your pointer horizontally in a Column makes nt race move the mark to the beginning point of the drag region and move the current time line to the ending point of the drag region The region is highlighted as you drag the pointer Zoom Region Sets the interval to be the time between the mark and the current time line inclusive Sets Time Start to either the mark or the current time line whichever is leftmost Sets Time End to either the mark or the current time line whichever is rightmost Centers the current time line in a Column Displays an error message in the message display area if the mark and the current time line are at the same place Zoom In Centers the interval around the
200. e Note that when viewing a user trace event file a kernel trace event file is required in order to resolve which CPU each pro cess was logging trace events from See Chapter 11 for more information The information associated with the node field appears in this listing only when NightTrace is configured to use an RCIM to timestamp events NOTE 3536 DoS S vecto 5538 S argl 55397 S vecto 5540 S vecto 5541 S vecto 55424 cpu 01 TR_PAGEFLT_ADDR pid cpu 01 TR_EXCEPTION_SUS pid r inst access cpu 01 TR_SWITCHIN pid 0 cpu 00 TR_INTERRUPT_ENT pid r hardclock level 1 cpu 00 TR INTERRUPT EXI pid r hardclock level 0 cpu 01 TR_INTERRUPT_ENT pid r softclock level 1 cpu 01 TR_INTERRUPT_EXI pid r softclock level 0 scheme S user instr page fault PC 0x1000fd54 scheme idle idle idle idle idle tid Eirg tag tid tid tad tid 1241 0 1241 0 0 0 0 0 0 0 0 0 0 0 time time time time time time time 305441 SISSI 313408 SS So 313425 305205 305441 P Screen 5 2 Example of ntrace listing Output with instr page fault 5 5 NightTrace Manual 13290 cpu 01 TR_PAGEFLT_ADDR pid 1s tid 1250 0 time S user data page fault addr 0x300ad1c0 PC 0xb0121fbc 3391 cpu 01 TR EXCEPTION EXI pid 1s tid 1250 0 time 94460 S vector data access 3392 cpu 01
201. e ipldisable option If you use the ipldisable option you must start up ntraceud with it You must not use the ipldisable option if your user level interrupt routine logs trace events to the shared memory buffer CAUTION The ipldisable option should be used with great care to avoid deadlock This may occur if more than one LWP each biased to run on the same CPU is logging trace events to a trace file created by an ntraceud invoked with the ipldisable option Consider the following scenario an LWP preparing to log a trace event locks the spin lock to protect the shared memory buffer It 1s preempted by a second LWP which also attempts to log a trace event However due to priority inversion the first LWP cannot release the spin lock causing the second LWP to loop infinitely 4 9 NightTrace Manual waiting for the spin lock to be released This deadlock could be avoided if ntraceud were invoked with out the ipldisable option This would allow the first LWP to release the spin lock before being preempted SEE ALSO For more information on the IPL register see the Power MAX OS Programming Guide 4 10 Generating Trace Event Logs with ntraceud Option to Prevent Page Locking lockdisable The ntraceud lockdisable option disables default page locking by ntraceud and by the NightTrace library routines in your application SYNTAX ntraceud lockdisable trace_file DESCRIPTION You can identify a running nt
202. e 9 37 end arg on page 9 47 and offset arg on page 9 60 start arg dbl DESCRIPTION The start arg db1 function returns the value of a particular trace event argument associated with the start event of the most recent instance of a state SYNTAX start arg N ab1 QS PARAMETERS N QS RETURN TYPE Specifies the Mh argument logged with the start event Defaults to 1 A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 double precision floating point SEE ALSO arg dbl on page 9 21 start arg on page 9 35 start num args on page 9 37 end arg dbl on page 9 47 and offset arg dbl on page 9 61 9 36 start_num_args start_pid Using Expressions DESCRIPTION The start_num_args function returns the number of arguments associated with the start event of the most recent instance of a state SYNTAX start num args QOS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO start arg on page 9 35 num
203. e For more information see Qualified States on page 9 83 start_node_name End Functions Using Expressions RETURN TYPE string SEE ALSO tid_table_name on page 9 29 offset_tid_table_name on page 9 68 and end_tid_table_name on page 9 55 DESCRIPTION The start node name function returns the name of the system from which the start event of the most recent instance of a state was logged SYNTAX start_node_name QS PARAMETERS OS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE string SEE ALSO node_name on page 9 30 offset_node_name on page 9 68 and end_node_name on page 9 56 The end functions provide information about the end event of the last completed instance of a state The state to which the end function applies is either the qualified state specified to the function or the state being currently defined Thus if a qualfied state is not speci fied end functions are only meaningful when used in expressions associated within a state definition NOTE End functions provide information about the last completed instance of a state whereas start functions see Start Functions on page 9 34 provide information about the mos
204. e 8 1 Figure 8 1 Configure Command Menu Sections on configuring display objects discuss the following topics Configuration parameters that are common to many display objects Operations you can perform on the configuration data Configuration parameters that are specific to each type of display object NOTE Columns are the only display objects that are not configurable Common Configuration Parameters Different types of configuration parameters exist Some parameters are concerned with how the information appears in the display object These parameters are Foreground Color Background Color Font Text Justify Text Gravity Fill Style Event Color Lost Event Color Mark Color Maximum and Minimum For each config uration parameter that pertains to color there is an equivalent X resource See Appendix B for more information Other parameters are concerned with determining the content of the information in the display objects The parameter that does this is Then Expression 8 1 NightTrace Manual The last type of parameter is concerned with constraining the information that appears in the display object These parameters act as filters allowing only data that meets certain criteria to be displayed These parameters are Event List If Expression CPU List PID List TID List Start Events End Events Start Expression and End Expression The configuration parameters are changed with the same editing methods used in the
205. e 9 32 event match s 20 eee ee A ats ees oe 9 33 State Functions iocis as 9 34 Start P nctionS io il Mates pad eet a ees 9 34 start id oa 9 35 start ars sce RS ERA EUER eR aay ae RU ENSE ue E RUNS 9 35 Start ar amp dbl sob DRE Ae EU 9 36 Start nunm args ceni e Ao ee alee UR De RR pede well UE 9 37 Start pid icr sre est e d dee E e 9 37 Statt taw pid ous sense Ghote eo EORR o ODER RU NR 9 38 start IwpidQ ss oom ek ook ee ERU decd RR EE 9 39 start thread ad iis ata oh a BA ES 9 39 start task 6 oues eet eese Rees 9 40 NightTrace Manual start tid ones eee oe ee Lee ee eee ee ee 9 41 Start Cp jag tis in dad ERG 9 41 Start offset ici a da MER e 9 42 start tmel 2 A AA toe eke 9 42 Start node 1dQ ii We ed TV ER ER ES 9 43 start pid table name 0 0c eee eee ee eee 9 44 start_tid_table_name 0 0 ccc eee eee eee 9 44 start node name llle 9 45 End Functions o R a le ea YI e be ea be 9 45 end drid UA CLP IER REN 9 46 end arg c Suo E 9 47 end ars dhl 4 uv ec dr RR e SUE RUE 9 47 end num args eee cece e 9 48 end pid ive hr be phe DvU EPIO Opes mes 9 48 end_raw_pid 1 eee ee e 9 49 end Iwpid s cr ERN ee RH ORDRES 9 50 end thread lO ii RR EDU 9 51 end task Td erro LER RR REC Eu Ris 9 51 end d dece REIR OI e e 9 52 end Cp iunctus ns RR eps 9 52 end offset cocoa lu RR Pri ee ee 9 53 end ME suddetto e pH ots wa cl
206. e a 9 54 end node 1d eA ee AR ER PIE RE ES 9 54 end_pid_table_name 0 eee eee eee ees 9 55 end_tid_table_name 0 0 ccc eee eee 9 55 end node name 0 cc cee cee eee An ENEE 9 56 Multi State Functions 0 0 ce eee eens 9 56 State Dap sse sk ecer ere EN EE ORI RR A 9 57 state dur oso i dba P ew 9 57 state Matches elos publ a bald dae paid uer 9 58 state status cient ete tret AE ee 9 58 Offset Bunctlons xor ree A EU aue SUR To eos RR 9 59 offset idees oee i ee E ibi bee Sb 9 60 Offset atg e aie E E ET AE AR EAE T nep Ede 9 60 offset arg dbl ica nc id ete 9 61 Offset MUM CAES o AA A a ee 9 61 Of SEPIA cio EFI Ee 9 62 Offset raw pid 5 3 see wba RISO DEAR E DRE a EPIIqgs 9 63 offset IWpid oe erg CE oet Ste OE gutter e 9 63 Offset thread di C eC E oe Be Se RA RO 9 64 offset task Tdi Do RA XR RR 9 64 offset tid a sm bd eps ELA PINE UD ED 4 9 65 Offset cpu eter tcd or ot cer e OR Rd eld 9 66 Offset UMEO pours RA tA ee oe ee Ae Ne 9 66 offset node 1d icio haw a 9 67 offset_pid_table_name 0 0 0 cece eee eee eee 9 67 offset tid table name 000 9 68 offset node name o 9 68 offset process name 0 0 cece ee 9 69 offset task name llle a 9 69 offset thread name oo 9 70 Summary Functions 2 0 0 0 eee eee eee 9 70 inth coe DEM eth ead a A eh ERR Sa 9 71 Sum Sieg pe De PP QU b See Shae ven lea DE oes 9 72 m
207. e appears here after being selected from the Files list The filename may be manually entered into this field by the user as well Runtime The Runtime page of the Daemon Definition dialog see Daemon Definition Dia log on page 3 25 allows the user to specify the scheduling policy CPU bias and mem ory binding policies for the daemon Figure 3 24 Daemon Definition dialog Runtime 3 4 NightTrace Manual Scheduling 3 42 Scheduling Policy POSIX defines three types of policies that control the way a process is scheduled by the operating system They are SCHED_FIFO FIFO SCHED_RR Round Robin and SCHED_OTHER Time Sharing Each of these scheduling policies is associated with one of the System V scheduler classes See either the PowerMAX OS Programming Guide 0890423 or the RedHawk Linux User s Guide 0898004 for more detailed information regarding these policies and their associated classes FIFO The FIFO first in first out policy SCHED_FIFO is associated with the fixed priority class in which critical processes and LWPs can run in predeter mined sequence Fixed priorities never change except when a user requests a change This policy is almost identical to the Round Robin SCHED_RR policy The only difference is that a process scheduled under the FIFO policy does not have an associated time quantum As a result as long as a process sched uled under the FIFO policy is the highest pr
208. e calls unless you are using the NightView debugger This is called instrumenting your code Before you begin this task you should read the appropriate language section below NightTrace applications written in C include the NightTrace header file usr include ntrace h with the following line include lt ntrace h gt The ntrace h file contains the following Function prototypes for all NightTrace library routines Return values for all NightTrace library routines e C macros described in Disabling Tracing on page 2 24 The library routine return values identify the type of error if any the NightTrace routine encountered If you think you may want to disable the NightTrace library routines in the future without having to remove them from your source code then you must include this file in your application 2 1 NightTrace Manual Fortran Ada C programs that are multi thread can also be traced with the NightTrace library routines For multi thread programs a C thread identifier is stored in each trace event uniquely identifying which C thread was running at the time the trace event was logged For more information on C see C A Reference Manual and the Concurrent C Reference Manual The Fortran version of the NightTrace library routines follow h 77 function naming and argument passing conventions For more information on h 77 see the hf77 Fortran Reference Manual All NightTrace library routines r
209. e created and saved by the session manager dialog See the NightTrace Pocket Reference card for a syntax summary of formats for event map files string tables and format tables Understanding Trace Event Files Trace event files are created by user and kernel trace daemons They consist of header information and individual trace events and their arguments as logged by user applications or the operating system ntrace detects trace event files as specified on the command line and does the required initialization processing so that the trace events contained in the files are available for display Understanding Event Map Files ntrace does not require you to use event map files However if you use these file s you can improve the readability of your nt race displays A trace point is a location in the application s source code where you call a NightTrace trace event logging routine Each trace point has a corresponding trace event ID number An event map file allows you to associate meaningful tags or labels with the more cryptic trace event ID numbers It also allows you to associate additional information with a trace event including the number of arguments and the argument conversion specifications or display formats Although ntrace does not require you to use event map files labels and correct display formats can make graphical nt race displays and textual summary information much more readable 5 9 NightTrace Manual 5 10 You c
210. e daemon s shared memory is owned by the creator the user attaching to the user trace daemon could be relevant in terms of permissions Refresh Queries the target system for active trace daemons Cancel Closes the Attach Daemons dialog without attaching to any of the listed dae mons Help Provides online help for this dialog NightTrace Manual Options Refresh Interval Tools 3 14 The Options menu appears on the NightTrace Session Manager menu bar see Night Trace Session Manager Menu Bar on page 3 2 Figure 3 10 Options Menu Refresh Rate Displays the Refresh Interval dialog see Refresh Interval on page 3 14 allowing the user to specify how often the statistics displayed in the Daemon Details Area are requested and updated for running daemons This dialog sets the display rate for those daemons currently selected in the Daemon Details Area see Daemon Details Area on page 3 17 This dialog allows the user to specify how often the statistics displayed in the Daemon Details Area are requested and updated for running daemons This dialog sets the display rate for those daemons currently selected in the Daemon Details Area see Daemon Details Area on page 3 17 Refresh Interval Refresh Interval secs I Figure 3 11 Refresh Interval dialog Refresh Interval The number of seconds between queries The Tools menu appears on the NightTrace Session Manager menu bar
211. e defined string table defined in the usr lib NightTrace tables file It associates 0 with false and all other values with true The mode that causes the ntraceud daemon to treat the shared memory buffer as a circular queue and to overwrite the oldest trace events with the newest ones this means that ntraceud intentionally discards the oldest trace events to make room for the newest ones Invoke ntraceud with the bufferwrap option to obtain this behavior The two other ntraceud modes are expansive mode and file wrap around mode See mouse button push button and radio button To press and release a mouse button without moving the pointer Usually you do this in NightTrace to select menu items push buttons or radio buttons Glossary 1 NightTrace Manual Close color display Column configuration configuration file Configuration Form Configure context switch context switch line control Glossary 2 A push button that closes a dialog box This can also be a menu item that makes a window close An X server display that contains greater color variety than black gray and white See also monochrome display A display object that constrains the width of StateGraphs EventGraphs Data Graphs and Rulers The definition of a display object macro qualified event or qualified state An NightTrace generated ASCII file that holds display pages macro qualified event and qualified state definitions This can
212. e event ID or tag and the time the trace event occurred The information it displays is related to the current time A DataBox s main use is to display data that is variable in nature and does not lend itself to graphical representation Column A scrollable display object that does not display data itself but holds graphical display objects StateGraphs EventGraphs DataGraphs and Rulers Its purpose is to define the width of and group together graphical display objects StateGraph A scrollable display object that graphically displays states as bars and trace events as vertical lines in a Column The StateGraph shows relative chronological positions of trace events and states since the trace started 5 29 NightTrace Manual Open Config File 5 30 EventGraph DataGraph Ruler A scrollable display object that graphically displays trace events as vertical lines in a Column It shows relative chronological positions of trace events since the trace started A scrollable display object that graphically displays a trace event expression as a vertical line or bar in a Column A DataGraph shows the relative chronological positions of trace event arguments since the trace started The height of the line or bar is proportional to the value of the expression A scrollable display object resembling a ruler that graphically displays the time A Ruler appears within a Column and shows what time a trace event occurred in a StateGraph Even
213. e event two flt e trace event dbl e trace event two dbl e trace event four arg and it can control which trace events are logged and when they are written to disk using e trace enable e trace enable range e trace enable all e trace disable e trace disable range e trace disable all e trace flush e trace trigger Figure 2 1 Inter Process Communication and Library Routines 2 4 trace_begin Adding Library Calls to Your Application The trace_begin routine initializes the trace mechanism and acquires resources for your process SYNTAX E Fortran Ada PARAMETERS key_file config int trace begin char key file ntconfig t config integer function trace begin trace file character trace file integer config 7 function trace begin trace file 3 string buffer_size integer 1024 16 use_spl boolean true use_resched boolean false lock pages boolean true clock ntclock_t NT_USE_ARCHITECTURE_CLOCK shmid_perm integer 8 666 inherit boolean true return ntrace error the user daemon logs trace events to an output file key file When you invoke the user daemon you must specify this file s name For the user daemon to log your process trace events to this file the trace event file parameter in your trace begin call must corre spond to the key file value on the daemon invocation The
214. e information SYNTAX start thread id QS 9 39 NightTrace Manual start_task_id 9 40 PARAMETERS QS RETURN TYPE integer SEE ALSO A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 thread id on page 9 24 end thread id on page 9 51 and offset thread id on page 9 64 DESCRIPTION The start task id function returns the Ada task identifier associated with the start event of the most recent instance of a state NOTE This function is only meaningful for trace events logged by Ada tasking programs SYNTAX start task id OS PARAMETERS QS RETURN TYPE integer SEE ALSO A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 task id on page 9 25 end task id on page 9 51 and offset task id on page 9 64 start_tid start_cpu Using Expressions DESCRIPTION The start_tid function returns the internally assigned NightTrace thread identifier TID associated with the start event of the most recent instance of a state SYNTAX start tid QSp
215. e information on these conversion specifications see printf 3S A value associated with the first conversion specification in format_string The value may be a constant string literal expression or an ntrace expression A string literal expres sion must begin and end with a and must be enclosed in double quotes for example WWW string expression An expression may bea get_string call a description of the get_string function appears in get_string on page 9 75 For more information on expressions see Chapter 9 format string may contain any number of conversion specifications There is a one to one correspondence between conversion specifications and quoted values A particular con version specification quoted value pair must match in both Invoking the ntrace Display Utility data type and position For example if format_string contains a s and a d the first quoted value must be of type string and the second one must be of type decimal integer If the number or data type of the quoted value s do not match format_string the results are not defined The optional default item line associates all other integer values with a single format item nt race flags it as an error if an expression evaluates to a value that is not on an item line and you omit the default item line The fields of the default item line are default_item The keyword that begins all default item lines format_string See format_st
216. e last subobject may be the name of your display object For more information about display object names see Display Object Name on page 8 4 attribute Is a characteristic of the last subobject this always begins on a lower case letter value Is a setting for the attribute It is possible to omit levels from the widget hierarchy If you specify all levels of the widget hierarchy and then a value the value applies to that specific widget If you leave B 1 NightTrace Manual out levels of the widget hierarchy the attribute applies more generally possibly to a class of widgets For more information on X resources see Recommended Reading on page 1 7 and the X Window System User s Guide Default X Resource Settings for ntrace B 2 ntrace s default X resource settings follow They are primarily grouped by window and display object There are some subobjects and attributes that appear in many settings Table B 1 lists several common subobjects and attributes along with their meanings Table B 1 Meanings of Common Subobjects and Attributes Subobject Attribute Meaning TextScrollbox The message or summary display area Dialog The dialog box name The window title Any window that has a name attribute also has a geometry attribute geomet ry The location and dimensions of the window See Recom mended Reading on page 1 7 for more information open A push button name in a File Selection Dialog Box
217. e pre defined event table either the trace event ID num ber or its corresponding trace event tag is displayed See get string on page 9 75 Pre Defined String Tables on page 5 14 and id on page 9 19 Depending on whether the global process identifier returned by pid is in the pre defined pid table either the global process identifier PID or its corresponding process name is displayed See get_string on page 9 75 Pre Defined String Tables on page 5 14 and pid on page 9 22 8 5 NightTrace Manual Table 8 3 Examples of Then Expressions Cont Then Expression Resulting Value or Meaning get_string tid tid get_string boolean arg get_string syscall arg get_string vector arg get_format next_state id get_format state_summary get_format event_summary get_format event_arg_summary 3 get_format event_arg_dbl_summary 1 Depending on whether the NightTrace thread identifier returned by tid is in the pre defined t id table either the NightTrace thread identifier TID or its corresponding thread name is displayed See get_string on page 9 75 Pre Defined String Tables on page 5 14 and tid on page 9 25 If arg has the value 0 false is displayed Otherwise t rue is displayed See get string on page 9 75 and Pre Defined String Tables on page 5 14 arg s value is looked up in the pre defined sysca1
218. e selection dialog that allows the user to navi gate to the desired directory and select a previously saved session configuration file to open NOTE Filenames are relative to the host system the system where the NightTrace GUI is running NightTrace Session Manager Open Session sample Figure 3 3 Open Session dialog Directory The directory in which to find the desired session configuration file The user may type the path into this field directly Filter Of all the files contained in the current Directory display only those Files that match the specified filter Directories Contains a list of all the subdirectories within the current directory Selecting any of these will change the current Directory to that subdirectory Double clicking on any of these directory names will change to that directory and update the Files list accordingly 3 5 NightTrace Manual Save Session 3 6 Files Within the current Directory this is a list of the files that match the specified Fil ter Any of these filenames can be selected When selected the filename appears in the Selection field Selection The name of the desired session configuration file appears here after being selected from the Files list The filename may be manually entered into this field by the user as well The Save Session dialog is a standard file selection dialog that allows the user to navi gate to the desired directory and specify the na
219. e start event are ignored until the end event is encountered Different types of display objects display information in different ways Depending on the type of information you want to display you choose the display object or objects you wish Creating Display Objects to create You can then configure those display objects to filter out unwanted data and process the information that you want displayed This allows you to watch only the data of interest Of course all of this is dependent on the application having the necessary trace_event calls enabled and inserted in the appropriate places All display objects are rectangular but you specify the dimensions of the rectangle Other properties of display objects you should be aware of are Display objects can be dynamic or static Dynamic means the contents vary depending on values in the trace event file and may change depending on what point in the execution of the application you are looking at Static means the contents do not change All display objects except the GridLabel and the Ruler are dynamic Display objects can be configurable or non configurable Configurable means you define the parameters that determine the content of the display object Non configurable means the display object has no parameters All display objects except the Column are configurable Display objects can be textual or graphical Textual means the contents consist of words or numbers Graphical means
220. e thread ID numbers with thread names In kernel tracing this table is not used When analyzing trace event files timestamped by the RCIM synchronized tick clock thread identifiers are not guaranteed to be unique across nodes Therefore accessing the t id table may result in an incorrect thread name being returned for a par ticular thread ID To get the correct thread name for a thread ID the tid table for the node on which the process identifier occurs should be used instead The t id table is maintained for backwards compatibility This table is indexed by a thread identifier or a thread name Examples of using this table are get string tid tid get item tid cleanup thread See TID List on page 8 8 for more information 5 15 NightTrace Manual boolean name_pid name_tid node_name pid_nodename tid_nodename 5 16 A string table defined in the usr lib Night Trace tables file It associates 0 with false and all other values with true A dynamically generated string table internal to ntrace It maps all known node ID numbers which are internally assigned by ntrace to the name of the node s process ID table This table is indexed by a node identifier or a node name Examples of using this table are get string name pid node id get item name pid system123 A dynamically generated string table internal to ntrace It maps all known node ID numbers which are internally assigned by nt
221. eans 12 288 bytes If you use the filewrap option you must start up ntraceud with it Your application logs enabled trace events into a shared memory buffer ntraceud copies these trace events to the trace event file In expansive mode this file can grow indefinitely The ntraceud filewrap option makes NightTrace operate in file wraparound mode rather than in expansive mode In file wraparound mode the trace event file can become full of trace events When this happens ntraceud overwrites the oldest trace events in the beginning of the file with the newest ones intentionally discarding the oldest trace events to make room for the newest ones In expansive default and file wraparound modes you control automatic buffer flushing by setting the ntraceud sleep interval shared memory size and buffer full cutoff percentage In contrast there is no automatic buffer flushing in buffer wraparound mode these values have no effect in this mode File wraparound mode can be beneficial if you are short of disk space With this mode you specify the maximum size of the trace event file instead of allowing it to grow indefinitely Consider using this option if you are interested only in the most recent of many trace events logged by an application over a long period of time If you want to determine how much disk space is available run the df 1 command with the k option and look at the avail column SEE ALSO For a comparison of expansi
222. earch on the display page allows you to locate areas of interest in your trace event file s When you click on Tools I Search the Search Form appears This form lets you provide search specifications and define conditions you wish to find in your trace event file s The Search Form consists of Radio buttons Push buttons Text fields Figure 10 2 illustrates the Search Form 10 1 NightTrace Manual rocess name ktrace L RLL RLL Poy mese Prev wen Search cise Figure 10 2 The Search Form NOTE The Node List field appears in this dialog only when NightTrace is configured to use an RCIM to timestamp events Search Form Radio Buttons Through the Search Form s radio buttons you can choose The direction of a search The interval to search The effect of a search on the grid and interval control area of a display page The Search Direction radio buttons let you search forward or backward in your trace session relative to the current time Click on the Forward radio button to search through newer trace events This is the default setting Click on the Backward radio button to search through older trace events Note This is a much less efficient search than a forward search The Search Constraints radio buttons let you limit your search to the entire trace session or to the current interval 10 2 Using the Built In Tools e Click on the Global Search radio button to se
223. ed 2 Start application 3 Call trace_disable 52 Trace event 52 disabled 4 Call trace_event 52 Trace event 52 not logged 5 Invoke ntraceud e52 Trace event 52 enabled Table 4 7 ntraceud Enable Sequence 2 From the Shell From the Application Comments 1 Invoke ntraceud All trace events enabled 2 Start application 3 Call trace_event 52 Trace event 52 logged 4 Invoke ntraceud e52 No effect 5 Call trace_disable 52 Trace event 52 disabled 4 27 NightTrace Manual SEE ALSO For information on disabling trace events see Option to Disable Logging dis able on page 4 24 and trace_enable trace_disable and Their Variants on page 2 16 Invoking ntraceud 4 28 This section shows a few common nt raceud invocation examples In each example the trace_file argument corresponds to the trace event file name you supply on your call to the trace_begin library routine Normally your first ntraceud invocation looks something like the following sample ntraceud trace_file The next sample invocation assumes that you lack both page lock privilege lockdisable and read and write access to dev spl needed to modify the interrupt priority level register C3pldisable or lack sufficient privileges required for rescheduling variables ntraceud lockdisable ipldisable trace_file The following invocation might be used when tuning your NightTrace configuration because you lost trace events last time
224. ed format table defined in usr lib NightTrace tables It contains formats for statistical displays of trace event matches and trace event time gaps It determines the default event summary output format A pre defined dynamically generated string table It is internal to NightTrace and maps all known numeric trace event IDs with symbolic trace event tags See trace event tag An event internal to the currently executing process that stops the current execution stream Exceptions can be suspended and resumed A StateGraph on a kernel display page It displays states representing exceptions executing on the associated CPU The default mode that causes the ntraceud daemon to copy all trace events that ever reach the shared memory buffer to the indefinitely sized trace event file Invoke nt raceud without the filewrap and bufferwrap options to obtain this behavior The two other ntraceud modes are buffer wraparound mode and file wraparound mode A combination of operators and operands that evaluate to a value Operands include constants macro calls function calls qualified events and qualified states A menu item that terminates an NightTrace session file wraparound mode flushing the buffer font format function format table function gap global process identifier Global Window graphical user interface Glossary The mode that causes the nt raceud daemon to overwrite the oldest trace events in the begi
225. ed qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE string SEE ALSO tid_table_name on page 9 29 start_tid_table_name on page 9 44 and offset_tid_table_name on page 9 68 end_node_name DESCRIPTION The end_node_name function returns the name of the system from which the end event of the last completed instance of a state was logged SYNTAX end node name QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE string SEE ALSO node name on page 9 30 start node name on page 9 45 and offset node name on page 9 68 Multi State Functions Multi state functions return information about one or more instances of a state state gap state dur 9 56 state_gap state_dur Using Expressions state matches state status For restrictions on usage see StateGraph on page 8 14 DESCRIPTION The state gap function returns the time in seconds between the start event of the most recent instance of the state
226. ed timings Log trace event NAP START with trace event ID 10 and the type Long number of nanoseconds to sleep sleep_str tv_nsec before the first nanosleep call in take_a_nap 2 26 Adding Library Calls to Your Application e Log trace event NAP END with trace event ID 20 after the first nanosleep call in take_a_nap NAP_START and NAP_END form the boundaries of a State e Close the thread End the trace session An example solution follows 2 27 NightTrace Manual For brevity no return values are checked include lt ntrace h gt include lt sys types h gt include lt time h gt include lt stdio h gt define NAP START 10 define NAP_END 20 void take_a_nap sleep_str struct timespec sleep_str NAP_START amp NAP_END are the boundaries of a state trace_event_arg NAP_START sleep_str tv_nsec printf Sleeping for 3f seconds n float sleep_str tv_nsec 1e 09 nanosleep amp sleep str NULL trace event NAP END make the spacing between states obvious Sleep str tv nsec 30000000 nanosleep amp sleep str NULL main int i struct timespec sleep_str trace_begin log trace_open_thread timings for 1 0 i 10 i sleep_str tv_nsec 1000 1000000 take_a_nap sleep_str trace close thread trace end exit 0 2 28 Adding Library Calls to Your Applic
227. ee Daemon Details Area on page 3 17 Deselect All Deselects all daemon definitions listed in the Daemon Details Area see Daemon Details Area on page 3 17 This dialog is presented when attaching to a daemon on a remote system see Attach Daemons on page 3 12 or when importing daemon attributes based on a user application running on a remote system see Import Daemon Definition on page 3 27 NightTrace Session Manager Target System A Figure 3 7 Login dialog After filling in the required fields in the Login dialog the Enter Password dialog see Enter Password on page 3 11 is displayed allowing the user to enter the password for the specified User on the specified Target System NOTE Passwords are not included in the configuration files written by NightTrace They are retained only during the current invocation of NightTrace Target System The name of the target system to which the user wishes to connect User The login name of the user on the specified Target System Enter Password The Enter Password dialog is displayed during user authentication on a target system NOTE The Enter Password dialog is not displayed if a valid pass word has already been entered for the specified user on the speci fied target system during the current invocation of NightTrace NightTrace Manual Enter Password Figure 3 8 Enter Password dialog Enter the password for the specified use
228. ee ee eee ee 2 14 Figure 11 1 Global Window File Menu 0 0 0 0 eee ee eee ee 2 6 Figure 11 2 Sample Kernel Display Page 0 0 00 00000088 2 7 Figure 11 3 Node Selection Dialog eese 2 8 Figure 11 4 Node Selection Warning Dialog lesse ese 2 9 Figure 11 5 Per CPU Information llsleeeeeeee eee 2 9 Figure 11 6 CPU Box iUt E durer eIeb ue 2 9 Figure 11 7 Running Process Boxes 0 0 0 eee eee eA 2 10 Figure 11 8 Node Box 0 0 eect eens 2 11 Figure 11 9 Context Switch Lines 0 0 0 0 2 11 Figure 11 10 Last Interrupt Box and Interrupt Graph 0 2 12 Figure 11 11 Last Exception Box and Exception Graph 2 13 Figure 11 12 TR_PAGEFLT_ADDR and TR_PROTFLT_ADDR Events 2 14 Figure 11 13 TR_SWITCHIN vs TR_PAGEFLT_ADDR and TR_PROTFLT_ADDR ENS O NS SLE NS Tae Sai 2 15 Figure 11 14 Last Syscall Box and Syscall Graph o oooooooomoomo o 2 15 Figure 11 15 Color Key cesca o ei or 2 16 Screen 4 1 Sample Output from the ntraceud help Option 2 7 Screen 4 2 Sample Output from ntraceud stats Option 2 23 Screen 5 1 Sample Output from the ntrace help Option 2 4 Screen 5 2 Example of ntrace listing Output with instr page fault 2 5 Screen 5 3 Example of ntrace listing Output with data page fault 2 6 Screen 5 4 Example of ntra
229. efault but two of them deserve special mention These two events allow you to determine areas in your applica tion code where address faults are occurring to minimize such faults and thus improve the application s performance The following sections discuss the page fault and protec tion fault kernel trace points Page Fault Event There is one page fault trace event TR_PAGEFLT_ADDR argl arg2 arg3 This trace event is logged whenever a kernel or user page fault occurs The page fault can be either on a data address or on an instruction address This trace event is not enabled by default because depending upon system activity page faults may occur reasonably fre quently This trace event has three arguments arg The data address which caused the page fault If the page fault occurred on an instruction this will be set to zero arg2 The program counter value at the time of the page fault arg3 The flag indicating whether the fault occurred on a kernel address or on a user address A value of zero indicates that the fault occurred on a user address A value of one indicates that the fault occurred on a kernel address Protection Fault Event There is one protection fault trace event TR_PROTFLT_ADDR argl arg2 arg3 This trace event is logged whenever a kernel or user protection fault occurs The protec tion fault can be either on a data address or on an instruction address This trace event is not enabled by default because
230. efinition of pid See the getpid 2 man page for more information SYNTAX raw pid QE PARAMETERS QE A user defined qualified event If supplied the function returns the process identifier of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the process identifier of the cur rent trace event For more information see Qualified Events on page 9 81 RETURN TYPE integer SEE ALSO PID List on page 8 7 pid on page 9 22 Iwpid on page 9 23 start_raw_pid on page 9 38 end_raw_pid on page 9 49 and offset_raw_pid on page 9 63 Iwpid DESCRIPTION The 1wpid function returns the lightweight process identifier LWPID associ ated with a trace event NOTE See the _1wp_self 2 man page for more information 9 23 NightTrace Manual SYNTAX iwpid QED PARAMETERS QE A user defined qualified event If supplied the function returns the lightweight process identifier of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the lightweight process identifier of the current trace event For more infor mation see Qualified Events on page 9 81 RETURN TYPE integer SEE ALSO PID List on page 8 7 pid on page 9 22 raw pid on page 9 23 start Iwpid on page 9 39 end Iwpid
231. en Save Ntrace SummarizeForm SaveTextAsPopup caption to Ntrace Summarize Form SaveTextAs Enter file name to save text Popup geometry B 3 NightTrace Manual B 4 TIP If you sometimes work at a monochrome monitor you may want to have two sets of the following X resource settings one for color and one for monochrome The color settings follow The resource names for monochrome settings are identical except they say Mono instead of Color TIP Experiment with colors and shadings until you find a set you like To avoid visual fatigue use highly contrasting colors and values sparingly The resource strings for the specific display objects are Nt race Color GridLabel background Nt race Color GridLabel foreground Nt race Color GridLabel font Nt race Color GridLabel textJustify Nt race Color GridLabel textGravity Ntrace Color DataBox background Ntrace Color DataBox foreground Ntrace Color DataBox font Nt race Color DataBox textJustify Nt race Color DataBox textGravity Ntrace Color Column background Nt race Color Column foreground Ntrace Color StateGraph background Ntrace Color StateGraph foreground Ntrace Color StateGraph eventColor Ntrace Color EventGraph background Ntrace Color EventGraph foreground Ntrace Color DataGraph background Ntrace Color DataGraph foreground Ntrace Color Ruler background Ntrace Color Ruler foreground Ntrace Color Ruler font Ntr
232. end of a displayed state to obtain information about the state start and state end trace events Figure 7 11 EventGraph Example Clicking on Create I EventGraph lets you draw or create an EventGraph display object on the grid An EventGraph represents trace events as a thin vertical line Event Graphs must be placed in a Column Figure 7 11 shows an EventGraph with a Ruler below it Some examples of information that an EventGraph can be used to display are The times your application starts executing a particular subroutine The sequence of execution of various modules in your application The timing of the birth and death of child processes See EventGraph on page 8 16 for more information In View mode to find out more information about a particular trace event position the cursor on the line and click once with mouse button 2 Information about that trace event is displayed in the message display area 7 15 NightTrace Manual DataGraph 7 16 18 4722s 18 4723s 18 4724 18 4725s 18 4726s 18 47278 I Figure 7 12 DataGraph Examples Clicking on Create I DataGraph lets you draw or create a DataGraph display object on the grid DataGraphs must be placed in a Column They represent data as either vertical lines or bars of varying height In Figure 7 12 the same set of data is used to draw the two basic types of DataGraph The top DataGraph is a line DataGraph which shows the data as vertical lines of varyi
233. end tid function returns the internally assigned NightTrace thread identi fier TID associated with the end event of the last completed instance of a state SYNTAX end tid Q5 PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO tid on page 9 25 start tid on page 9 41 and offset tid on page 9 65 end cpu DESCRIPTION The end cpu function returns the logical CPU number associated with the end event of the last completed instance of a state CPUs are logically numbered start ing at 0 and monotonically increase thereafter 9 52 Using Expressions NOTE This function is only valid when applied to events from Night Trace kernel trace event files SYNTAX end cpu QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO cpu on page 9 26 start cpu on page 9 41 and offset cpu on page 9 66 end offset DESCRIPTION The end offset function returns the ordinal number offset of the end ev
234. endpoint of a state should have a different trace event ID number Usually each trace event logging routine logs a different trace event ID number This lets you easily identify which source line logged the trace event how often that source line executed and what order source lines executed in However it is sometimes useful to log the same trace event ID in multiple places This makes it possible to group trace events from related but not identical activities For more information see trace event and Its Variants on page 2 10 How can I prevent user trace events from being discarded or lost Use expansive mode avoid use of buffer or file wrapping options Flush the shared memory buffer more often by tuning The shared memory buffer size The shared memory buffer s flush percentage ncrease the priority of the user trace daemon Bind the user trace daemon to a CPU with minimal activity See Preventing Trace Events Loss on page A 1 and Chapter 4 for more information What can I do if trace events are not appearing in an ntrace display Press Refresh fill out the Search Form fill in values in the interval control area use the interval scroll bar keep pressing the Zoom Out push button until you see trace events examine a display object configuration so you know what it is listening for add or reconfigure display objects on the grid See Chapter 6 for more information My trace event timings occasionally have huge gap
235. ent of the last completed instance of a state SYNTAX end offset QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer 9 53 NightTrace Manual end_time end node id 9 54 SEE ALSO offset on page 9 27 and start offset on page 9 42 DESCRIPTION The end time function returns the time in seconds associated with the end event of the last completed instance of a state Times are relative to the earliest trace event from all trace data files currently in use SYNTAX end time OS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE double precision floating point SEE ALSO time on page 9 27 start time on page 9 42 state gap on page 9 57 state dur on page 9 57 and offset time on page 9 66 DESCRIPTION The end node id function returns the internally assigned node identifier asso ciated with the end event of the last completed instance of a state SYNTAX end node id QS PARAMETERS QS A
236. ent tags A similar association appears in the usr lib NightTrace eventmap file this is an pid tid Invoking the ntrace Display Utility event map file that associates trace event IDs with kernel trace event tags This table is indexed by an event code or an event code name Examples of using this table are get_string event 4112 get_item event TR_INTERRUPT_EXIT A dynamically generated string table internal to ntrace In user tracing it associates global process ID numbers with pro cess names of the processes being traced In kernel tracing it associates process ID numbers with all active process names and resides in the dynamically generated vectors file When analyzing trace event files timestamped by the RCIM synchronized tick clock process identifiers are not guaranteed to be unique across nodes Therefore accessing the pid table may result in an incorrect process name being returned for a particular process ID To get the correct process name for a process ID the pid table for the node on which the process identifier occurs should be used instead The pid table is maintained for backwards compatibility This table is indexed by a process identifier or a process name Examples of using this table are get_string pid pid get item pid ntraceud See PID List on page 8 7 for more information A dynamically generated string table internal to nt race In user tracing it associates NightTrac
237. entifier associated with a trace event tid QE start_tid QS end tid QS offset tid offset expr 9 10 The integer NightTrace thread identifier TID associated with a trace event Table 9 3 NightTrace Functions Using Expressions Syntax Return Type cpu QE start cpu QS end cpu QS offset cpu offset expr The integer logical CPU number associated with a trace event This function is only valid when applied to events from Night Trace kernel trace event files time QE start time QS end time QS offset time offset expr The double precision floating point time expressed in units of seconds between a trace event and the earliest trace event from all trace event files currently in use node id QED start node id QOS end node id QS offset node id offset expr The internally assigned integer node identi fier associated with a trace event pid table name QE start pid table name QS end pid table name QS offset pid table name offset expr The string describing the name of the pro cess identifier table PID table associated with a trace event tid table name QE start tid table name QS end tid table name QS offset tid table name offset expr The string describing the name of the inter nally assigned thread identifier table TID
238. er Product Name 0 5 from top of spine Helvetica 36 pt Bold exp O D Volume Number if any Helvetica 24 pt Bold Volume Name if any Helvetica 18 pt Bold Manual Title s Helvetica 10 pt Bold Manual centered vertically within space above bar double space between each title 0890398 Bar 1 x 1 8 beginning 1 4 in from either side Part Number Helvetica 6 pt centered 1 8 up
239. er performs this conversion For example suppose that two processes named a out exist on a particular system and that one has a PID of 1234 and the other has a PID of 5678 Further suppose that you wish to create a StateGraph to display events only for PID 1234 Prior to Version 4 1 if you entered 1234 in the PID List parameter ntrace would have converted that to a out As the events were being analyzed any event that had a PID of 5678 would also have been displayed by the StateGraph since a process named a out also existed with a PID of 5678 If the trace event file has multiple processes with the same name for example a out specifying any one of the PIDs for that process selects all the PIDs of that process To avoid this it is recommended that all processes be given unique names If that is not possi ble you can isolate individual processes by including a PID restriction in the If Expression parameter For example if a out includes PIDs 100 1 200 1 and 300 1 and you want information only on PID 100 1 set the PID List parameter to a out and the If Expression to pid 100 1 For more information about the pid function see pid on page 9 22 8 7 NightTrace Manual TID List 8 8 A NightTrace thread identifier TID is a 32 bit integer It includes a 16 bit integer raw PID and a 16 bit integer C thread or Ada task identifier If neither C threads nor Ada tasks are in use then the 16 bit integer will a
240. erations you must first create a display object When you create a display object you choose its place on the grid and its size 7 5 NightTrace Manual Creating display objects involves three steps selecting loading the type of display object to be drawn selecting the place on the grid where the display object will go and selecting the size of the display object Some display objects go only inside of other display objects StateGraphs EventGraphs DataGraphs and Rulers go only inside a Column To create a display object and place it on the grid do the following 1 Place the pointer on the Create entry on the menu bar and click mouse button 1 2 Select the type of display object you want to create Note that the pointer is now a crosshair The display object is now loaded 3 Move the pointer until it is on the grid where you want to place a corner of the display object As mentioned previously some display objects go only inside of Columns If the cursor is on the border of a Column or outside of one you will not be able to draw these display objects Note that the left and right sides of these display objects are determined by the Column and you only have to place the pointer somewhere on the intended top or bottom edge of the display object 4 Click and drag mouse button 1 until the display object is the size you want it to be While you are sizing a display object its boundaries are shown as dashed lines Note
241. erface to trace event analysis If you do not have any ntrace related files but you still want to try out this tool just type ntrace You can override some default functionality by invoking ntrace with options and arguments The full nt race invocation syntax is ntrace help version listing mgr filestats nohardclock process all name PID start offset time slu percent end offset time slu percent flat color Xoption file Depending on your ntrace options and arguments when you invoke ntrace it Loads all trace event information into memory Checks the syntax of specifications in each file argument Processes each file argument Loads any display pages and their objects into memory Presents any display pages See Chapter 6 Displays the Global Window See ntrace Global Window on page 5 25 The following sections discuss the nt race options and arguments ntrace Options You can abbreviate all ntrace options to their shortest unambiguous length but most of the examples in this manual use the long option name These options are case insensitive The following examples are all equivalent ntrace help ntrace hel ntrace he ntrace h ntrace H ntrace HE ntrace Hel ntrace HELP ntrace options include 5 3 NightTrace Manual help Displays the ntrace invocation syntax on standard output and exits Screen 5 1 shows an exam
242. erminology is defined later in this chapter Table 6 1 View Mode Mouse Button Operations Button Use Within a Column Mouse button 1 Hold down lt Ctrl gt and click mouse button 1 Hold down lt Ctrl gt hold down mouse but ton 1 and drag horizon tally Mouse button 2 Hold down lt Ctrl gt and click mouse button 2 Mouse button 3 Hold down lt Ctrl gt and click mouse button 3 Move the current time line to the place where the pointer rests or put the text cursor where you clicked in the text field Move the mark and the current time line to the place where the pointer rests Move the mark to the beginning point of the drag region and move the current time line to the ending point of the drag region The drag region is highlighted as you drag the pointer Write a statistic in the message display area that tells about the trace event where the pointer rests in a StateGraph or EventGraph Write a statistic in the message display area that tells how far the pointer is from the mark A positive number means the pointer is to the right of the mark A negative number means the pointer is to the left of the mark Write a statistic in the message display area that tells about the data item where the pointer rests in a DataGraph Write a statistic in the message display area that tells how far the pointer is from the cur rent time line A positive number means the pointer is to the right of the current time li
243. es Corp Motif OSF and OSF Motif X Window System and X are trademarks of The Open Group Printed in U S A Revision History Level Effective With Original Release August 1992 000 NightTrace 1 0 Current Release June 2002 080 NightTrace 5 1 Scope of Manual Preface This manual is a reference document and users guide for NightTrace a graphical inter active debugging and performance analysis tool and KernelTrace a tool that collects and textually analyzes system performance Structure of Manual A brief description of the chapters and appendixes in this manual follows Chapter contains introductory material on NightTrace Chapter 2 gives the syntax and examples of NightTrace library calls Chapter 3 introduces the NightTrace Session Manager and its usage Chapter 4 describes the nt raceudcommand line user daemon Chapter 5 describes how to invoke the nt race display utility Chapter 6 shows how to view trace event logs with ntrace Chapter 7 illustrates nt race display objects and their creation Chapter 8 shows how to configure ntrace display objects Chapter 9 defines NightTrace expressions Chapter 10 tells about NightTrace s built in tools Chapter 11 describes kernel tracing This manual also contains three appendixes a glossary and an index Appendix A describes performance tuning Appendix B describes graphical user interface GUI customization Appendix C provides answers to common question
244. es click on Cancel as a result ntrace removes the dialog box and you can save your changes If you do not want to save your changes click on OK this causes ntrace to exit Tools Menu Item The tools menu allows you to invoke the session manager an integral part of the Night Trace graphical interface in addition to other tools in the NightStar family including NightProbe NightBench NightSim and NightView 5 32 Invoking the ntrace Display Utility Help Menu Item When you click on the Help menu item on the Global Window the pull down menu shown in Figure 5 8 appears ance Copyright lt C 1996 Co Figure 5 8 Global Window Help Menu The Online Manual item opens the online version of the NightTrace Manual using the HyperHelp viewer that is shipped as part of the X Window System x11 product The online NightTrace Manual can also be accessed using the nhelp utility shipped with the X Window System The manual name is ntrace For example from the command line nhelp ntrace opens the most recently installed version of the NightTrace Manual in the HyperHelp viewer The File Selection Dialog Box The File Selection Dialog Box gives you three ways to find a file Type in the exact file name Scroll through existing file names until you see the one you want e Type in a filter file name pattern for nt race to locate Typing in the Exact File Name If you know the exact file name use the following steps t
245. es e g TR_INTERRUPT_ENTRY TR_EXCEPTION_EXIT or 4112 4117 Use Event Names Allows the user to view the event names of the trace event types in the Disabled Events and Enabled Events lists instead of their numerical values For kernel events these mappings are provided in the file usr lib NightTrace eventmap which is automatically loaded by the NightTrace Session Manager For user trace events the user may load user defined event map files using the Load Event Names dialog see Load Event Names on page 3 39 Use the Load Event Names button to access this dialog NightTrace Manual Load Event Names Allows the user to load various user defined event map files by presenting the Load Event Names dialog see Load Event Names on page 3 39 The user may then view the event names of the user trace event types in the Dis abled Events and Enabled Events lists instead of their numerical values by checking the Use Event Names checkbox on this dialog 3 24 NightTrace Session Manager Daemon Definition Dialog The Daemon Definition dialog allows the user to create and modify the various aspects of a daemon configuration Figure 3 18 Daemon Definition dialog The Daemon Definition dialog is divided into a number of subpages that contain spe cific information about the current configuration These subpages are General This page contains information such as the name of the daemon configuration the target
246. es provide a mechanism for ref erencing state configurations within some functions You may use a qualified state in the following predefined functions start functions end functions and multi state functions For more information see Start Functions on page 9 34 End Functions on page 9 45 and Multi State Functions on page 9 56 To create a qualified state definition select the Qualified States menu item from the Expressions menu see Expressions Menu on page 9 1 to open the Qualified States Dialog Box see Expression Dialog Boxes on page 9 2 for details on this type of dialog Click the Add button on the Qualified States Dialog Box select the qualified state from the list and click on the Configure button to pop up a Qualified State Config uration Form like the one shown in Figure 9 8 9 83 NightTrace Manual state_001 R_EXCEPTION_ENTRY R_EXCEPTION_EXIT m ITR EXCEPTION ENTRY TREXCEPTION EXIT ooo TRUE TRUE cap o AL AL r Figure 9 8 Qualified State Configuration Form The following parameter is specific to the Qualified State Configuration Form QualifiedState The name by which you refer to this qualified state in expres sions NOTE The Node List field appears in this dialog only when NightTrace is configured to use an RCIM to timestamp events See Node List on page 8 9 for more information about this field For information about other confi
247. ession possibly containing functions or macros See Chapter 9 for more information Then Numeric expression or string Expression is any valid C like expression Expression possibly containing functions or macros See Chapter 9 for more information ALL All CPUs are listened to GPU Tist NONE No CPUs are listened to Lf 25 34 Listed CPUs are listened to 8 2 Table 8 1 Common Configuration Parameters Cont Configuring Display Objects Parameter Name Possible Values Meaning Any meaningful combination of the fol lowing PID List ALL All PIDs are listened to NONE No PIDs are listened to e 123 1 4561 78971 Listed PIDs are listened to A comma separated list of alphanu The name of a process meric strings beginning with letters Underscores are also allowed Spaces are not allowed Any meaningful combination of the fol lowing TID List ALL e All TIDs are listened to NONE No TIDs are listened to 1231 43661 1891 Listed TIDs are listened to A comma separated list of alphanu The name of a thread as specified in meric strings beginning with letters the trace open thread call Underscores are also allowed Spaces See trace open thread on page are not allowed 2 9 for more information Any meaningful combination of the fol lowing Node List ALL All nodes are listened to NONE No nodes are listened to 0 1 4
248. est ing by increasing the height of the state bar Although interrupts can nest all interrupts must complete before the process they interrupt can be switched out Therefore you will never see a context switch occur in the middle of an interrupt The last interrupt box displays the name of the last interrupt prior to the current time line that executed and may still be executing on the associated CPU It can be used with the interrupt graph to identify any interrupts that are currently visible on the graph Simply move the current time line onto a graphed interrupt and the last interrupt box will update to display the name of the interrupt Because the last interrupt box displays the name of the last interrupt that executed it is possible for there to be no interrupts visible on the interrupt graph even though the last interrupt box contains a valid interrupt name This just signifies that the last interrupt on the CPU ended prior to the beginning of the current interval An interrupt that is seen very often is the hardclock interrupt which usually accounts for 15 of the total number of trace events logged by the kernel If you are not interested in hardclock interrupts they can be ignored by ntrace improving performance and readability See ntrace Options on page 5 3 for more information The last interrupt box is a DataBox display object and the last interrupt graph is a Data Graph display object See Chapter 7 and Chapter 8 for
249. et to the characters that are to appear in the GridLabel For example if you want a box on the grid containing the phrase Flight Simulation Trace Screen you would enter the following text in the Text field Flight Simulation Trace Screen See GridLabel on page 7 12 for more information See Common Configuration Param eters on page 8 1 for descriptions of the common configuration parameters that GridLabels use DataBox Configuring Display Objects R_SWITCHIN TRUE ormat pid s get_string pid_cap argl D LL E E e A A Figure 8 6 DataBox Configuration Form The configuration form for the DataBox is shown in Figure 8 6 NOTE The Node List field appears in this dialog only when NightTrace is configured to use an RCIM to timestamp events A DataBox can be used as a counter A counter is simply a DataBox that counts the occurrences of a particular trace event or other condition up to the current time For example if you wanted to display the number of trace events occurring before the current time set the Event List parameter to ALL and put the following expression in the Then Expression field event_matches 8 13 NightTrace Manual StateGraph 8 14 This expression counts the number of times the criteria were met See Chapter 9 for more information on expressions See DataBox on page 7 12 for more information See Common Configuration Parameters on page 8 1 for des
250. eturn INTEGERS but because they begin with a t Fortran implicitly types them as REAL You must explicitly type them as INTEGER so that they work correctly For example to explicitly type the trace_begin routine use the following declaration integer trace_begin Ada applications can access the NightTrace library routines via the Ada package night_trace_bindings which is included with the MAXAda product currently only available on PowerMAX OS systems The bindings can be found in the bind ings general environment in the source file night_trace a The night_trace_bindings package contains the following An enumeration type consisting of the return values for all NightTrace library routines The bindings that permit Ada applications to call the C routines in the NightTrace library and to link in the NightTrace library Many of the NightTrace functions have been overloaded as procedures These procedures act as the corresponding functions except they discard any error return values Ada programs that use tasking can also be traced with the NightTrace library routines For multitasking programs an Ada task identifier is stored in each trace event uniquely identifying which Ada task was running at the time the trace event was logged For more information on Ada see the section titled NightTrace Binding in the MAXAda Reference Manual Adding Library Calls to Your Application Inter Process Communication
251. exit c After Instrumentation 0 0008 2 29 Figure 3 1 NightTrace Session Manager esee eee eee 2 2 Figure 3 2 NightTrace Menu ca peier nae ea eaei e e 2 3 Figure 3 3 Open Session dialog esee 2 5 Figure 3 4 Save Session dialog 1 0 0 0 eee eee 2 7 Figure 3 5 Unsaved Changes dialog 00 0 cece eee eee 2 8 Figure 3 6 Daemons Menu 0 0 eee ee 2 9 Figure 3 7 Login dialog 1 0 0 2 eee cece eee eee 2 11 Figure 3 8 Enter Password dialog 2 00 0 cece ee eee 2 12 14 Figure 3 9 Figure 3 10 Figure 3 11 Figure 3 12 Figure 3 13 Figure 3 14 Figure 3 15 Figure 3 16 Figure 3 17 Figure 3 18 Figure 3 19 Figure 3 20 Figure 3 21 Figure 3 22 Figure 3 23 Figure 3 24 Figure 3 25 Figure 5 1 Figure 5 2 Figure 5 3 Figure 5 4 Figure 5 5 Figure 5 6 Figure 5 7 Figure 5 8 Figure 6 1 Figure 6 2 Figure 6 3 Figure 6 4 Figure 6 5 Figure 6 6 Figure 6 7 Figure 7 1 Figure 7 2 Figure 7 3 Figure 7 4 Figure 7 5 Figure 7 6 Figure 7 7 Figure 7 8 Figure 7 9 Figure 7 10 Figure 7 11 Figure 7 12 Figure 7 13 Figure 7 14 Figure 7 15 Figure 8 1 Figure 8 2 Figure 8 3 Figure 8 4 Figure 8 5 Figure 8 6 Figure 8 7 Contents Attach Daemons dialog 0 eee eee ee eens 2 12 Options Menu 0565 seeks rial Te mE SM EE 2 14 Refresh Interval dialog 0 0 0 0 eect eee 2 14 Tools Men s v
252. f omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO num args on page 9 21 start num args on page 9 37 end arg on page 9 47 and offset num args on page 9 61 DESCRIPTION The end pid function returns the global process identifier PID associated with the end event of the last completed instance of a state Using Expressions NOTE A global process identifier does not have the same meaning as the typical operating system definition of pid A PID within Night Trace is a 32 bit integer value that contains the operating system process identifier raw PID in the upper 16 bits and the light weight process identifier LWPID in the lower 16 bits Consult the lwp global self 2 man page for more information SYNTAX end pid OST PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO pid on page 9 22 raw pid on page 9 23 Iwpid on page 9 23 start pid on page 9 37 and offset pid on page 9 62 end raw pid DESCRIPTION The end raw pid function returns the process identifier raw PID a
253. f the state Otherwise it is the last completed instance of a state In this manual a three button pointing device for point and click interfaces mouse button multi event function multi state function name_pid table name_tid table New Page NightTrace NightTrace thread NightTrace thread identifier Glossary A part of the mouse that you can press to alter aspects of the application Each mouse button has a different purpose Button 1 is usually for selecting or dragging Button 2 is usually for moving display objects Button 3 is usually for resizing display objects You can make multiple selections by simultaneously pressing lt Shift gt and clicking mouse button 1 You may click drag press and release mouse buttons Multi event functions return information about ocurrences of events or relation ships between occurrences of events before the current time line Multi state functions return information about instances of states or relationships between instances of states before the current time line A pre defined dynamically generated string table It is internal to NightTrace and associates node ID numbers with the the name of each node s process ID table A pre defined dynamically generated string table It is internal to NightTrace and associates node ID numbers with the the name of each node s thread ID table A menu item that creates an empty display page The interactive debugging and performance
254. f the task from which the ordinal trace event offset was logged NOTE This function is only meaningful for trace events which were logged from Ada tasking programs SYNTAX offset task name offset expr PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event 9 69 NightTrace Manual offset_thread_name RETURN TYPE string SEE ALSO task_name on page 9 31 DESCRIPTION The o fset thread name function returns the thread name from which the ordinal trace event offset was logged SYNTAX offset_thread_name offset_expr PARAMETERS offset_expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE string SEE ALSO thread_name on page 9 32 Summary Functions 9 70 You usually use summary functions on the Summarize Form Except for summary_matches all of these functions take another expression as a parameter They include the following min max avg sum min_offset max offset summary matches min max DESCRIPTION Using Expressions The min function returns the minimum value of all occurrences of expr within a time range When used ina Summarize Form the time range is defined by that form When used elsewhere the time range is defined as the region starting with the first trace event and ending with the current
255. fset raw pid offset lwpid offset thread id offset task id offset tid offset cpu offset time offset node id offset p offset t id tab id tab e name e name offset node name offset process name offset task name offset thread name Usually these functions take one of the following functions as a parameter offset 9 59 NightTrace Manual offset_id offset_arg 9 60 start offset e end offset min offset max offset For information about these functions see offset on page 9 27 start offset on page 9 42 end offset on page 9 53 min offset on page 9 73 and max offset on page 9 73 DESCRIPTION The o fset id function returns the trace event ID of the ordinal trace event offset SYNTAX offset id offset expr PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE integer SEE ALSO idQ on page 9 19 start id on page 9 35 and end id on page 9 46 DESCRIPTION The o fset arg function returns the value of a particular trace event argu ment for the ordinal trace event offset SYNTAX offset arg N offset expr PARAMETERS N Specifies the Nth argument logged with the trace event Defaults to 1 offset_arg_dbl offset_num_args Using Expre
256. ge 7 16 for more information about these display objects RETURN VALUES The trace event trace event arg trace event dbl and trace event four arg routines return a zero value NTNOERROR on successful completion Otherwise they return a non zero value to identify the error condition A list of error codes for these routines follows NTINVALID NTINIT NTLOSTDATA SEE ALSO Adding Library Calls to Your Application An invalid trace event ID has been supplied Solution Use trace event IDs only in the range 0 4095 inclusive The NightTrace library routines were not initialized or they were initialized but no user daemon has yet been initiated Ensure a trace begin and trace open thread call precede the trace event logging routine call Once a user daemon is started subsequent attempts at logging events will succeed The trace event was lost because the shared memory buffer was full This can occur if the user daemon cannot empty the shared memory buffer quickly enough Increase the priority of the user daemon and or schedule it on a CPU with less activity Additionally the size of the shared memory buffer can be increased Related routines include trace flush trace trigger trace enable trace enable range trace enable all trace disable trace disable range trace disable all 2 15 NightTrace Manual trace_enable trace_disable and Their Variants 2 16 B
257. ges macro definitions qualified events qualified states and tables Maps event ID numbers with event tag names Holds events logged by your application and ntraceud P Screen 5 1 Sample Output from the ntrace help Option version mgr listing Displays the current ntrace version stamp on standard output and exits Starts the NightTrace graphical interface and pops up the ses sion manager dialog for configuration and control of user and kernel daemons Chronologically lists all trace events in the trace event file s to standard output and exits The output includes the following information about a trace event relative timestamp trace event ID any trace event argument s the global process identifier PID or thread name and the CPU The timestamp for the first trace event is zero seconds 0s All other timestamps are rela tive to the first one If you supply an event map file on the invocation line ntrace displays symbolic trace event tags instead of numeric trace event IDs and ntrace displays trace event arguments in Invoking the ntrace Display Utility the format you specify in the file rather than the hexadecimal default format For more information on event map files see Understanding Event Map Files on page 5 9 In kernel tracing the vectors file provides names for system processes interrupts and exceptions Screen 5 2 and Screen 5 3 show examples from a kernel trace event fil
258. gned decimal integer default Sounsigned octal integer xunsigned hexadecimal integer 1fsigned double precision decimal floating point For more information on these conversion specifications see printf 3S The following line is an example of an entry in an event map file event 5 Error 2 x Slf Trace event ID 5 is an error condition when appropriate ntrace displays trace event 5 and labels the trace event Error Trace event 5 also has two 2 arguments ntrace displays the first argument in unsigned hexadecimal integer x format and the second argument in signed double precision decimal floating point 1 format You may over ride these conversion specifications when you configure display objects For more information on event map files see Pre Defined String Tables on page 5 14 Read Event Map File on page 5 31 and the ntrace 4 man page For information about trace event tags for kernel trace events see the usr lib Night Trace eventmap file Understanding Page Configuration Files nt race does not require you to use configuration files However using these file s Allows you to associate macros qualified events and qualified states with particular display page s mproves the readability of your displays e Saves you time laying out your display pages A configuration file is an ASCII file that contains definitions These definitions look like initialized C structures A
259. gs 9 48 end_offset 9 53 end_pid 9 48 end_pid_table_name 9 55 end_raw_pid 9 49 end_task_id 9 51 end_thread_id 9 51 end_tid 9 52 end_tid_table_name 9 55 end_time 9 54 event_gap 9 32 event_matches 8 13 9 33 format 9 75 format 8 5 9 4 9 7 9 80 10 7 10 8 get_format 5 20 8 6 9 4 9 7 9 79 10 7 10 8 get_item 9 4 9 77 get_string 5 17 5 18 5 19 8 5 9 4 9 75 8 5 9 19 9 79 9 81 lwpid 9 23 max 9 71 10 7 10 8 o max_offset 9 73 10 8 min 9 71 10 7 10 8 min_offset 9 73 10 7 10 8 multi event 9 32 multi state 9 56 node_id 9 28 node_name 9 30 num_args 9 21 offset 9 59 offset 5 19 9 27 offset_arg 9 60 offset_arg_dbl 9 61 offset_cpu 9 66 offset_id 9 60 9 73 9 74 offset_lwpid 9 63 offset_node_id 9 67 offset_node_name 9 68 offset_num_args 9 61 offset_pid 9 62 offset_pid_table_name 9 67 offset_process_name 9 69 offset_raw_pid 9 63 offset_task_id 9 64 offset_task_name 9 69 offset_thread_id 9 64 offset_thread_name 9 70 offset_tid 9 65 offset_tid_table_name 9 68 offset_time 9 66 pid 8 5 8 7 9 22 9 79 pid_table_name 9 29 process_name 9 7 9 30 raw_pid 9 23 start 9 34 Index 6 start_arg 9 35 start arg dbl 9 36 start cpu 9 41 start id 9 9 9 35 start lwpid 9 39 start node id 9 43 start node name 9 45 start num args 9 37 start offset 9 42 start pid 9 37 start pid table name 9 44 start raw pid 9 3
260. gt Delete one character 1 Position the text cursor to the right of the character to be deleted to the left 2 Either press lt Backspace gt lt Delete gt or simultaneously press 6 16 Ctrl lt h gt Viewing Trace Event Logs with ntrace Table 6 4 Making Editing Changes Goal Delete adjacent character s Replace adjacent character s Replace all character s Restore the default value Undo editing change s since the last lt Enter gt or Apply e SNe Re Steps to Attain Goal Point to the first character to be deleted Drag the pointer across any other characters to be deleted release the mouse button and keep the pointer in the field This highlights the characters you dragged the pointer across Either press lt Backspace gt lt Delete gt or simultaneously press lt Ctrl gt lt h gt Point to the first character to be replaced Drag the pointer across any other characters to be replaced release the mouse button and keep the pointer in the field This highlights the characters you dragged the pointer across Type in the new character s Position the text cursor anywhere in the field you want to modify Simultaneously press lt Ctrl gt lt u gt This highlights all characters in the field Type in the new character s Replace all character s in the field with either a single space character or the word default Note Some fields do not have default values Press lt En
261. guration parameters see Chapter 8 especially Com mon Configuration Parameters on page 8 1 and StateGraph on page 8 14 Configuring qualified states is similar to configuring StateGraph display objects The configuration parameters for a qualified state are identical to those that are used to configure a StateGraph display object See StateGraph on page 8 14 for information on how to configure a StateGraph EXAMPLE Qualified states can be useful when you are interested in a trace event that occurs while a certain state is active The following qualified state 9 84 Using Expressions QualifiedState foo_state Start Events PROG A BEGIN End Events PROG A EXIT defines a state that is active whenever program A is running Assume that another process is logging FOO trace events asynchronously If you are interested only in the FOO trace events that are logged while program A is running you can define an EventGraph as follows Event List FOO If Expression state status foo state true This graphs only FOO trace events that occur while the qualified state oo state is active The true is not necessary Thus you see only FOO trace events logged while program A is running 9 85 NightTrace Manual 9 86 10 Using the Built In Tools HA as hea We e a et E S eA PIE a A eee 10 1 Searching for Points of Interest 2 0 0 ee eee 10 1 Search Form Radio Buttons
262. h Buffer Wraparound Mode bufferwrap 4 13 Option to Define Shared Memory Buffer Size memsize 4 16 Option to Set Timeout Interval timeout 0 0 0 ee ooo 4 17 Option to Set the Buffer Full Cutoff Percentage cutoff 4 18 Option to Select Timestamp Source clock 0 00 0 00000088 4 19 Option to Reset the ntraceud Daemon reset 0 0 0 eee eee eee 4 20 Option to Quit Running ntraceud quit 02 0 0 0 0 eee eee ee eee 4 21 Contents Option to Present Statistical Information stats 0 0 0 0 ess 4 22 Option to Disable Logging disable 0 00 00 eee ee eee 4 24 Option to Enable Logging enable 0 00 e eee eee eee 4 26 Invoking ntraceud zs erret A a AR RS OV Leek Pee hee 4 28 Exercise Logging Trace Events 0 0 rererere eee eee 4 29 Chapter 5 Invoking the ntrace Display Utility OVeEVIe WC ii A a A E NI 5 1 X and NightTrace Vocabulary 0 0 eee cece eee 5 1 System Environment 0 0 0 cee cece e 5 2 Invoking ntrace reos e oe ee Sete ce ae EE Oe eed ee 5 3 ntrace Optians le totes Ue es 5 3 ntrace Arguments cese her p ELIS ra Eel Ta RIT E 5 8 Understanding Trace Event Files 0 0 0 0 eee 5 9 Understanding Event Map Files 00 0 eee 5 9 Understanding Page Configuration Files eese esee 5 11 ntrace Tables Uat a Eden 5 12 String Tables v
263. haracter with the choice An ellipsis follows an item that can be repeated An arrow separates a menu bar item from a pull down menu entry The window images in this manual come from a Motif environment If you are using another environment your windows may differ slightly from those presented here Referenced Publications The following publications are referenced in this document 0830048 0830046 0890240 0890300 0890378 0890380 0890395 0890423 0890429 HN6200 Architecture Reference Manual HN6800 Architecture Manual hf77 Fortran Reference Manual X Window System User s Guide C A Reference Manual OSF Motif Documentation Set 3 volumes NightView User s Guide PowerMAX OS Programming Guide System Administration Volume 1 1 NightView is a trademark of Concurrent Computer Corporation 2 PowerMAX OS is a trademark of Concurrent Computer Corporation 0890430 0890460 0890466 0890474 0890516 0891019 0891055 0891082 Preface System Administration Volume 2 Compilation Systems Volume 2 Concepts PowerMAX OS Real Time Guide NightTrace Pocket Reference MAXAda Reference Manual Concurrent C Reference Manual Elan License Manager Release Notes Real Time Clock and Interrupt Module User s Guide PowerPCTM 604 RISC Microprocessor User s Manual not avail able from Concurrent Computer Corporation 3 Elan License Manager is a trademark of Elan Computer Group Inc 4 PowerP
264. he Enable Disable Trace Events dialog see Enable Disable Trace Events on page 3 22 or on the Events page of the Daemon Definition dialog see Events on page 3 37 The user can access this dialog by pressing the Load Event Names button on either of the aforementioned dialogs The Use Event Names checkbox on either of these dia logs allows the user to toggle between viewing the event names and their numerical val ues 3 39 NightTrace Manual NOTE Event map files are relative to the host system the system where the NightTrace GUI is running Read Event Map File sample Figure 3 23 Load Event Names dialog Directory The directory in which to find the desired event map file The user may type the path into this field directly Filter Of all the files contained in the current Directory display only those Files that match the specified filter Directories Contains a list of all the subdirectories within the current directory Selecting any of these will change the current Directory to that subdirectory Double clicking on 3 40 NightTrace Session Manager any of these directory names will change to that directory and update the Files list accordingly Files Within the current Directory this is a list of the files that match the specified Fil ter Any of these filenames can be selected When selected the filename appears in the Selection field Selection The name of the desired event map fil
265. he interval Edit the fields to change the interval Time Start 4 3124876s Time Length 12 0000000s Time End 16 3124876s Event Start 6 Event Count 15 Event End 20 Zoom Factor 2 00 Increment 25 002 Current Time 10 9124876s Figure 6 6 The Interval Control Area All field values in the interval control area are non negative numbers Some fields have default values Time fields all display the time in seconds with the s suffix A description of each field follows In the following text interval is the time from Time Start through Time End Time Start Is the beginning time of the interval in seconds Time End Is the ending time of the interval in seconds Time Length Is the amount of time within this interval in seconds It is the difference between Time End and Time Start Current Time Is the present time within the interval in seconds Event Start Is the ordinal number offset not the trace event ID of the first trace event in this interval 6 11 NightTrace Manual Event End Is the ordinal number offset not the trace event ID of the last trace event in this interval Event Count Is the quantity of trace events present in this interval It is the difference between Event End and Event Start plus one Zoom Factor Is the number of times to magnify or reduce the interval each time you click on Zoom Out or Zoom In The default is 2 Increment Controls how much the current interval scrolls and the slider moves
266. he mpadvise 3C library routine to help you determine which CPUs exist on this system You can the trace daemon and your application to particular CPUs with the run 1 command run bbias command Conserving Disk Space To determine how much disk space is available on your system run the d 1 command with the k option and look at the avail column You can conserve disk space if you permit NightTrace to discard some trace events To do this invoke ntraceud with either the filewrap option or the bufferwrap option The ntraceud filewrap option makes NightTrace operate in file wraparound mode rather than in expansive mode In file wraparound mode the trace event file can become full of trace events When this happens ntraceud overwrites the oldest trace events at the beginning of the file with the newest ones The overwriting is called discarding trace events For more information on file wraparound mode see Option to Establish File Wraparound Mode filewrap on page 4 12 The ntraceud bufferwrap option makes NightTrace operate in buffer wraparound mode rather than in expansive mode When the buffer is full in buffer wraparound mode the application treats the shared memory buffer as a circular queue and overwrites the oldest trace events with the newest ones This overwriting continues until your application explicitly calls trace lush or trace trigger Only then does ntraceud copy the remaining trace events from t
267. he shared memory buffer to the trace event file The overwriting is called discarding trace events For more information on buffer wraparound mode see Option to Establish Buffer Wraparound Mode bufferwrap on page 4 13 By default ntraceud operates in expansive mode not file wraparound or buffer wrap around mode In expansive mode NightTrace uses the most disk space because it does not discard any trace events You can also conserve disk space by invoking nt raceud with the disable option so it logs fewer trace events For details see trace enable trace disable and Their Vari ants on page 2 16 Conserving Memory and Accelerating ntrace A 4 ntrace can be a memory intensive tool By default when ntrace starts up it loads all trace event information into memory therefore the more trace events in your trace event file s the more memory ntrace uses When you move the scroll bar on the Display Page to change the displayed interval nt race processes all trace events between the last interval and this one if there are many trace events the display update or search may seem slow To conserve memory and accelerate ntrace Log only trace events you are really interested in Performance Tuning Invoke ntrace only with the trace event files that are essential to your analysis Invoke ntrace with options nohardclock process start and end that restrict which trace events get loaded For more informa tion
268. he window images in this manual come from a Motif environment If you are using another environment your windows may differ slightly from those presented here ntrace displays appear on your terminal only if you set your DISPLAY environment variable Determine if this variable is set by issuing the following command echo SDISPLAY If this variable is not set you must set it manually to a value based on the name of your X server For example in Bourne shell set the DISPLAY environment variable for a terminal named eagle this way DISPLAY eagle 0 0 export DISPLAY In the Korn shell this is export DISPLAY eagle 0 0 In the C shell this is setenv DISPLAY eagle 0 0 The Xdefaults or Xresources file in your login directory establishes default environmental settings for your X sessions You may use special ntrace settings in this file to customize your ntrace displays ntrace runs on both monochrome and color monitors See Appendix B for information about setting color and other X resources that pertain to ntrace TIP Experiment with colors and shadings until you find a set you like To avoid visual fatigue use highly contrasting colors and values sparingly For more information on window system concepts or Motif see Recommended Reading on page 1 7 Invoking the ntrace Display Utility Invoking ntrace The ntrace display utility resides on your system under usr bin ntrace It is the graphical user int
269. icator in command line option summaries and syntax listings that tells you more than one occurrence of the previous syntactic component is allowed A state function that provides information about the ending trace event of the last completed instance of a state The state to which the end function applies is either the qualified state specified to the function or the state being currently defined Thus if a qualfied state is not specified end functions are only meaningful when used in expressions associated within a state definition See trace event event_arg_dbl_summary table event arg summary table EventGraph A pre defined format table defined in usr 1ib NightTrace tables It contains formats for statistical displays of trace event matches and type double arguments A pre defined format table defined in usr lib NightTrace tables It contains formats for statistical displays of trace event matches and type long arguments A scrollable display object that graphically displays trace events as vertical lines in a Column Glossary 5 NightTrace Manual event ID event map file event_summary table event table event tag exception exception graph expansive mode expression Exit Glossary 6 See trace event ID User generated ASCII file that lets you associate or map short mnemonic fags or labels with numeric trace event IDs The kernel s event map file is usr lib NightTrace eventmap A pre defin
270. icked on the Apply push button Clicking on Reset is equivalent to pressing lt Esc gt Restore Restores changed field s on the Summarize Form to the original value s they had when you brought up the form Clear Erases all text in the summary display area Summarize Saves your field changes and summarizes the requested data If you have made a field change clicking on this push button makes ntrace temporarily save your field settings Clicking on this push button causes ntrace to sum marize summary data from your trace event file s based on the criteria from the Summarize Form fields and the radio button settings Because all fields and radio buttons on the Summarize Form have default settings you can click on the Summarize push button without modifying anything in this window The default summarize behavior is ntrace writes statistical messages in the summary display area that tell about trace event data through the entire trace session for any trace event from any process on any CPU If you have configured the Summarize Form by specifying additional criteria the summarize behavior is Ifa trace event or state meets all these criteria ntrace writes statistical messages in the summary display area that tell the trace events or states involved and minimum maximum average and total for intervals and or trace event arguments If no trace event or state meets all these criteria ntrace writes a message in the su
271. if you previously called trace open thread SEE ALSO Related routines include t race open thread trace end 2 22 Adding Library Calls to Your Application trace_end The trace_end routine frees resources and terminates trace event tracing in your pro cess SYNTAX C int trace end Fortran integer function trace end Ada function trace end return ntrace error DESCRIPTION Generally call trace end only once per logging process However for processes using C threads or Ada tasks trace end must also be called by any individual threads or tasks that have previously called trace begin trace end performs the following operations Terminates trace event tracing in this process or thread Flushes trace events from the shared memory buffer to the trace event file Detaches the shared memory buffer timestamp source and interrupt priority level IPL register Notifies the user daemon that the current process has finished log ging trace events RETURN VALUES The trace end routine returns a zero value NTNOERROR on successful completion Otherwise it returns a non zero value to identify the error condition A list of trace end error codes follows NTFLUSH A failure occurred while attempting to flush the shared mem ory buffer Solution Verify the status of the user daemon if necessary restart it and rerun the trace NINODAEMON There is no user daemon with a trace even
272. ightView 1 4 2 1 Default Kernel Page menu item 5 27 11 6 Default Page menu item 5 27 5 28 Default push button 5 23 Delete menu item 7 18 Delete push button 9 2 Deselect All menu item 7 18 device table 5 17 11 4 11 17 device_nodename table 5 17 11 18 df 1M command 4 12 4 14 A 2 A 3 Dialog box File Selection 5 33 Macro 9 4 9 6 B 3 Open Config File 5 30 Qualified Event 9 81 B 3 Qualified State 9 83 B 3 Read Event Map File 5 31 Warning 5 32 5 34 7 19 B 3 Dialog box see Window Dimmed button see Disabled button push button see Disabled push button Dimmed field label 5 27 Dimmed label 5 27 Dimmed menu item see Disabled menu item Directory dev A 3 etc conf mtune d A 2 usr lib NightTrace 5 11 5 14 5 16 5 20 9 77 9 78 9 80 10 8 11 6 usr lib NightTrace examples 2 25 5 35 usr lib X11 fonts 5 8 8 3 B 1 Disabled button 5 27 7 4 Disabled menu item 5 27 7 4 7 18 10 10 Disabled push button 5 27 7 4 Disabling IPL usage 4 4 4 9 4 28 A 3 library routines 2 1 2 16 2 24 page locking 4 4 4 11 4 28 A 3 trace events 2 17 4 4 4 24 tracing 2 16 2 24 Discarding trace events 2 21 A 1 A 4 C 1 Display color 5 2 11 16 B 1 B 4 monochrome 5 2 7 14 B 1 B 4 DISPLAY environment variable 5 2 Display object 1 3 5 1 7 1 7 8 Column 5 29 6 2 6 3 6 4 7 5 7 9 7 13 8 1 9 7 B 4 configuring 8 1 creating 7 1 7 5 DataBox 5 29 7 12 8 13 9 79 9 82 10 11 11 10 11 12 11 13 11 16 B
273. iii tes did a 5 13 Pre Defined String Tables oooococooocoooooocmomo mo 5 14 Format Tables ect sd PEE 5 17 Pre Defined Format Tables 0 0 0 rnaen eee eee 5 20 Configuring Display Pages 0 0 0 ees 5 20 ntrace User Interface is oc Gs A Ce ie Se E 5 21 Using th Mouse cui ses me eua RII E Up 5 21 Understanding Pointer Shapes 0 00 0c ee eee 5 22 Anticipating Window Layout 0 0 eee eee Ih 5 22 Resizing WindOWS ix Pee eee Ee aR ee yes ne Pai ENDE ERR 5 24 ntrace Notation Conventions 0 0 0 ccc eee eee 5 24 ntrace Global Window 0 0c ccc cette eens 5 25 Message Display Area 2 0 eee cece eee 5 25 Menu Bar vis c it RA EN A AAA ERA 5 26 NightTrace Menu Item 0 0 0 cece eee eee 5 26 New Paperi heisse e lia 5 27 Default Page sc e ahs re RD ds Sa eit aad See d 5 28 Open Config Ell amp cere RA ie hae Ee XM 5 30 Read Event Map File 0 0 eee eee eee eee 5 31 Elba PEPA Bae ee 5 32 Tools Menu Item s hosna III 5 32 Help Men iS See A ON See cher te ee 5 33 The File Selection Dialog Box oooooooooccococono eee 5 33 Typing in the Exact File Name 0 00 0 e eee eee eee 5 33 Scrolling Through Existing File Names eee eese 5 34 Typing in a Filter File Name Pattern eee 5 35 Exercise Displaying Trace Events lees 5 35 Chapter 6 Viewing Trace Event Logs with ntrace OVOLVIeW ati teet e tn eR E
274. ileChooser width 180 Nt race OpenPopup name Nt race OpenPopup open Open Nt race OpenPopup caption Nt race OpenPopup geomet ry Nt race ReadPopup name Nt race ReadPopup open Read Nt race ReadPopup caption Ntrace ReadPopup geometry Ntrace SaveAsPopup name Ntrace SaveAsPopup open Save Ntrace SaveAsPopup caption Ntrace SaveAsPopup geometry Open Dialog Enter configuration file name Read Dialog Enter event map file name Save As Dialog Enter configuration file name to save The resource strings for the other dialog box titles and descriptive text are Ntrace WarningDialog name Ntrace QuestionDialog name Ntrace WorkingDialog name Ntrace MacroDialog name Ntrace MacroDialog caption Ntrace QualifiedEventDialog name Ntrace QualifiedEventDialog caption Ntrace QualifiedStateDialog name Ntrace QualifiedStateDialog caption Warning Dialog Question Dialog Working Dialog Macros List of Macros Qualified Events List of Qualified Events Qualified States List of Qualified States The resource strings for the window title and descriptive text for all Forms are Ntrace SearchForm name Search Ntrace Summarize Ntrace Summarize Ntrace Summarize Form name Form TextScroll Form TextScrollbox defaultLines Summarize 84 14 box defaultChars Ntrace SummarizeForm SaveTextAsPopup name Save Summary Text As Dialog Ntrace SummarizeForm SaveTextAsPopup op
275. ilter 5 35 Index Filter field 5 35 Filter Expression field 9 4 10 7 Finding see Searching Flushing shared memory buffer 2 20 4 5 4 13 4 18 4 21 4 28 A 1 A 2 Font configuration parameter 8 3 8 9 Font field 8 3 Fonts 5 8 8 3 B 1 Foreground Color configuration parameter 8 3 8 9 Foreground Color field 8 3 fork 2 service 2 7 Form Configuration 8 4 8 10 8 12 8 13 8 14 8 16 8 17 8 19 9 2 9 4 DataBox configuration 8 13 DataGraph configuration 8 17 EventGraph configuration 8 16 GridLabel configuration 8 12 Macro configuration 9 6 9 7 Qualified Event configuration 9 81 Qualified State configuration 9 83 9 84 Ruler configuration 8 19 Search 9 4 10 2 B 3 C 1 StateGraph configuration 8 14 Summarize 9 4 10 12 10 14 B 3 Format functions 9 75 format function 8 5 9 4 9 7 9 80 10 7 10 8 Format table 5 9 5 17 9 79 event_arg_dbl_summary 5 20 8 6 event arg summary 5 20 8 6 10 8 event summary 5 20 8 6 10 7 get format function 5 20 8 6 9 4 9 7 9 79 10 7 10 8 state summary 5 20 8 6 10 7 Fortran language 1 3 compiling and linking 2 25 source considerations 2 2 Forward radio button 10 2 Functions 9 9 arg 9 9 9 20 arg dbl 9 21 argl 5 19 8 5 9 9 9 81 10 7 10 8 arg2 8 5 9 12 avg 9 72 cpu 8 5 9 26 end 9 45 nd arg 9 47 nd arg dbl 9 47 nd cpu 9 52 nd id 9 46 nd lwpid 9 50 nd node id 9 54 nd node name 9 56 0000000 Index 5 NightTrace Manual end_num_ar
276. ime as the timeout interval ignoring the value specified 4 17 NightTrace Manual Option to Set the Buffer Full Cutoff Percentage cutoff 4 18 By default when the shared memory buffer becomes 20 percent full of unwritten trace events the ntraceud daemon wakes up and copies the trace events to the trace event file The ntraceud cutoff option lets you alter this percentage SYNTAX ntraceud cutoff percent trace file DESCRIPTION The ntraceud cutoff option lets you set the buffer full cutoff percentage percent for the shared memory buffer percent is an integer percentage in the range 0 99 inclusive The process of copying trace events from the shared memory buffer to the trace event file trace file on disk is called flushing the buffer When a logged trace event causes the buffer to reach the buffer full cutoff percentage ntraceud wakes up and flushes the buffer Waking the ntraceud daemon incurs overhead that can distort trace event timings decreasing the shared memory buffer full cutoff percentage makes it more likely that the daemon will be wakened by the application You can also decrease the per cent parameter to prevent trace event loss the effect is an increase in the buffer flush frequency If you use the cutoff option you must start up ntraceud with it If you invoke ntraceud with both the cutoff and bufferwrap options ntraceud ignores the cutoff option Generating Trace Event Logs with ntraceud
277. in offset uc eell toi 9 73 max Offsets A E ea Se EM Pe er XUL SP NR 9 73 summary matches sese n 9 74 Format and Table Functions 0 0 0 00 e cece een eee 9 75 Bet SWINE O oso tete aet ER edo EE ge 9 75 SEU AILEM Ole sis A ees ae ele ait Te ee a 9 77 set format oe esheets e ems 9 79 TOMA aia a a dat 9 80 Qualified Events coi e edu ed eer ar es 9 81 Qualified States ci eee ee SCUTIS E E RCM EMEN SEN 9 83 NightTrace Manual Overview 9 Using Expressions NightTrace allows you to define macros qualified events and qualified states to aid in the analysis of trace data Macros are named expressions provided for flexibility and convenience Qualified events provide a mechanism for referencing trace event configu rations within certain functions Qualified states provide a mechanism for referencing state configurations within certain functions as well The Expressions menu contains menu items for creating these entities See Expres sions Menu on page 9 1 Macros on page 9 6 Qualified Events on page 9 81 and Qualified States on page 9 83 for further information Macros qualified events and qualified states are configured using expressions in much the same way as display objects See Expressions on page 9 4 for a complete explana tion of expressions In addition Chapter 8 Configuring Display Objects may provide some helpful information as well Expressions Menu Figure 9
278. ing TIP To save time re editing recompiling and relinking your application consider beginning with a few too many trace points in the source code You can dynamically enable or disable specific trace events You can also save time by using ntrace options to restrict which trace events are loaded for analysis See ntrace Options on page 5 3 for details Some typical trace points include the following Suspected bug locations Process subprogram or loop entry and exit points 2 13 NightTrace Manual 2 14 Timing points especially for clocking I O processing Synchronization points multi process interaction Endpoints of atomic operations Endpoints of shared memory access code Call one trace event logging routine at each of the trace points you have selected When you call this routine it writes the trace event information including timings and any arguments to a shared memory buffer By default if this write fills the shared memory buffer or causes the buffer full cutoff percentage to be reached the user daemon wakes up and copies the trace event to the trace event file on disk Usually each trace event logging routine logs a different trace event ID number This lets you easily identify which source line logged the trace event how often that source line executed and what order source lines executed in However it is some times useful to log the same trace event ID in multiple places This makes it po
279. ing Tables on page 11 16 syscall See Kernel String Tables on page 11 16 device See Kernel String Tables on page 11 16 vector_nodename See Kernel String Tables on page 11 16 syscall nodenameSee Kernel String Tables on page 11 16 device_nodename See Kernel String Tables on page 11 16 You can use pre defined string tables anywhere that string tables are appropriate Use the get_string function to look up values in string tables For information about the get_string function see get_string on page 9 75 For examples of function calls with these tables see Table 8 3 Like string tables format tables let you associate an integer value with a character string however in contrast to a string table string a format table string may be dynamically formatted and generated Labeling numeric values with text can make the values easier to interpret The syntax for a format table is format_table table_name item int const format string valuel default item format string valuel y You do not need to separate the parts tokens of the format table with white space Format table tokens are indivisible although these tokens need not break on the lines shown they must appear in the order shown Include all special characters from the syntax except the ellipses and square brackets The fields in a format table are format_table The key
280. ink entry_exit c with the ntrace library Give the executable the name entry_exit Start the ntraceud daemon Look at the trace_start call to determine the trace event file name You may need some additional options if you cannot lock pages in memory or cannot read and write to the IPL register Execute the entry exit program Get the ntraceud daemon to give you statistics When the program completes stop the ntraceud daemon An example solution follows cc Xa o entry exit entry exit c lntrace lud ntraceud log entry exit ntraceud stats log ntraceud quit log Xr Xr our 4 Ur This exercise continues in Exercise Displaying Trace Events on page 5 35 4 29 NightTrace Manual 4 30 5 Invoking the ntrace Display Utility OVERVIEW Le To 5 1 X and NightTrace Vocabulary 0 0 0c eee es 5 1 System Environment cc 5 2 Invoking Ntrace ovs IA os Oe eee Pek ee ety 5 3 Dtrace ODtOnS 2o ELE Veh as eth id ee ee dU KR ER 5 3 NACE Arguments sor sat hes MESS RIS whe Mee ee BRI IRE RES 5 8 Understanding Trace Event Files 0 00 0 eese 5 9 Understanding Event Map Files 00 0 eee 5 9 Understanding Page Configuration FileS ooooooocoooooooooo ooo 5 11 ntrace Tables ij risseenkssrsi ets bea Ate ee BALTES EE 5 12 String Tables umi ti ae 5 13 Pre Defined String Tables 0 0 0 eee eee eee 5 14 Format Tables vie tan Wane i ees Wee SR Be 5 17 Pre Defined Format Tab
281. int get_item table_name str const PARAMETERS table name str const table name is an unquoted character string that represents the name of a string table To avoid possible forward reference problems try to make your get item calls refer to previ ously defined string tables The following string table names are pre defined in NightTrace event pid tid bool ean name pid name tid node name pid nodename tid nodename vector syscall device event summary event arg summary event arg dbl summary state summary For more information on these tables see Pre Defined String Tables on page 5 14 and Kernel String Tables on page 11 16 str const is a string constant literal that acts as an index into the specified string table str const must either exactly match a string value in the table name string table or the table name string table must have a default item line otherwise the results 9 77 NightTrace Manual are undefined A table_name may contain several item lines with the same str_const value DESCRIPTION Usually you would put a get_item call in a Then Expression of a display object configuration to locate an index number in a string table For each get item call NightTrace follows these steps 1 Uses str const value as an index into table name 2 Retrieves the first associated index number from fable name 3 Returns the index number Assume that the following string table definition is
282. interrupt hd An HDC disk controller interrupt hps An HPS serial line controller interrupt hrm A reflective memory interrupt hsa An HSA disk controller interrupt hsd An HSD controller interrupt ie An integral ethernet interrupt is An integral SCSI controller interrupt mpcc An MPCC controller interrupt pgintr An FDDI controller interrupt rtcintr A real time clock interrupt xy A Xylogics tape controller interrupt There are many different types of exceptions that can be logged by the kernel The possible types are listed in the system dependent vector string table in the vectors file Syscalls Tracing the Kernel Table 11 4 is an alphabetical list of the most common exceptions See the PowerPC 604 RISC Microprocessor User s Manual for more information Table 11 4 Exception Reference Exception Description data access decrementer float unavail inst access inst brkpt kstack overflow machine check misaligned program trace An exception indicating that a page fault for a data page occurred An exception that occurs when the decrementer register counts down to zero An exception that occurs the first time a process attempts to use the floating point unit A page fault exception that occurs during an instruction fetch An exception indicating that a breakpoint instruction was executed A fatal exception generated due to kernel errors A fatal exception generated for various rea
283. iority process scheduled on a par ticular CPU it will continue to execute until it voluntarily blocks Round Robin The Round Robin policy SCHED_RR like the FIFO policy is associated with the fixed priority class in which critical processes and LWPs can run in predetermined sequence Fixed priorities never change except when a user requests a change A process that is scheduled under this policy as opposed to the FIFO policy has an associated time quantum Time Sharing The Time Sharing policy SCHED_OTHER is associated with the time sharing class changing priorities dynamically and assigning time slices of different lengths to processes in order to provide good response time to interactive processes and LWPs and good throughput to CPU bound processes and LWPs Priority The Priority is relative to the selected Scheduling Policy see Scheduling Pol icy on page 3 42 and the range of allowable values is dependent on the operating system CPU Bias NUMA NightTrace Session Manager For example on PowerMAX OS systems the priority values for the FIFO class include 0 59 where 59 is the most urgent user priority available on the system On RedHawk systems the priority values for the FIFO class include 1 99 where 99 is the most urgent user priority available on the system It is recommended that a reasonable urgent priority is specified when using the FIFO scheduling policy to prevent event loss
284. ires the specification of two trace events a start event and an end event respectively Additional conditions may be specified in a state definition to further constrain the state The state functions include the following Start functions e End functions e Multi state functions The start functions provide information about the start event of the most recent instance of a state The state to which the start function applies is either the qualified state specified to the function or the state being currently defined Thus if a qualfied state is not specified start functions are only meaningful when used in expressions associated within a state def inition In addition start functions should not be used in a recursive manner in a Start Expression a start function should not be specified in a Start Expression that applies to the state definition containing that Start Expression Conversely an End Expression may include start functions that apply to the state definition containing that End Expression NOTE Start functions provide information about the most recent instance of a state whereas end functions see End Functions on page 9 45 provide information about the last completed instance of a state Start functions include the following start 1d start arg e start arg dbl start num args start pid start raw pid start thread id start task id start tid start lwpid
285. ise Displaying Trace Events on page 5 35 Invoke the NightTrace display utility with the log trace event file the map event map file and the page configuration file ntrace log map page 10 14 Using the Built In Tools After the display page appears press the Refresh push button at the bottom right of the page The current time line should now be positioned inside the first visible state What is the tag of the current trace event NAP START Now bring up the Search tool by clicking on Tools I Search Use the default settings to search globally forward for all trace events and make the interval scroll the current time to the trace event After one search what is the tag of the current trace event NAP_END Keep searching forward until you reach the end of the trace You should continue to see alternating NAP_START and NAP_END trace event tags Close the Search Form by clicking on the Close push button Exercise Using the Summarize Tool The following exercise has you summarize trace events you logged in Exercise Logging Trace Events on page 4 29 While still in ntrace bring up the Summarize tool by clicking on Tools Summarize Press the Summarize button for the default event summary How many matches were summarized 22 At which offset does the largest gap occur 2 How large is this gap about 1 8 seconds Close the Summarize Form by clicking on File Close 10 15 NightTrace Manual 10
286. it A 16 bit integer that makes up the first half of a PID Real Time Clocks and Interrupts Module It provides a synchronized clock edge triggered interrupts real time clocks and programmable interrupts Some set of interrupts can be distributed and sent to all connected RCIMs The RCIM hard ware is available via a standard PCI mezzanine card PMC RCIM synchronized tick clock Read record region release Reset Restore The primary clock on an RCIM It is a 64 bit non interrupting counter that counts each tick of the clock 400 nanoseconds When connected to other RCIMs the synchronized tick clock provides a time base that is consistent for all connected sin gle board computers A menu item and push button that read an existing file See trace event The period of time between the mark and the current time To let go of the currently pressed mouse button A push button that cancels undoes all unapplied changes A push button that cancels all changes since the dialog box was displayed Glossary 15 NightTrace Manual Ruler running process box Save Save As Save Text Save Text As SBC scroll bar Search Form selection Glossary 16 A scrollable display object that appears as a hash marked timeline within a Column The Ruler may also contain reverse video L s indicating lost trace events and user defined marks A DataBox that shows the process that is executing at the curren
287. it is useful to disable the extraneous trace events You can disable trace events temporarily where you disable and later re enable them You can also disable trace events permanently where you disable them before the application runs or during its execution and never re enable them In the first format the ntraceud disable option dynamically disables a specific trace event ID JD from logging to the shared memory buffer In the second format the ntraceud disable option dynamically disables a range of trace event IDs D low through ID high from logging to the shared memory buffer In either case trace event IDs are integers in the range 0 4095 inclusive At defined times ntraceud copies trace events from the shared memory buffer to the trace event file trace file NOTE The disable option disables trace events in all processes that rely on the same nt raceud daemon to log to the same trace event file This first format provides the same functionality as the trace disable Night Trace library routine The second format provides the same functionality as the trace disable range NightTrace library routine One advantage of using the disable option rather than the library routine is that you do not have to re edit recompile and relink your application For more information on disable library routines see trace enable trace disable and Their Variants on page 2 16 Note In the following text the names of the trace e
288. iting Operations vips pi e 7 17 Select Alen tants A E E qe eis 7 18 Deselect Allin 4 ssi A ee ees 7 18 Delete ys Cs ese ea ere ee ST e Sl Ra eer ete 7 18 Fil Operations s sau se nai A da 7 18 Bayeo e Pte eh ique eA URL arce ee ds 7 18 DIVA dcs csv RR eU NN BAAR eRe NEVER OU eU Leta NS 7 19 Close ie cose Eom Muse UR Ce a ek eect DUUM NT Sii UE MI si 7 19 Chapter 8 Configuring Display Objects 10 OVERVIEW iiss Soi his ONO 8 1 Common Configuration Parameters 0 0 0 cece ee eee 8 1 Display Object NaMe rrot ene Dei I 8 4 Event East enir enee A ea E E A Se a 8 4 TEEXPLOS STO e E A A AR 8 4 Then ExpressiOn sei ianea a ace E 8 5 CPU List acidos oie did she ii e e te iae 8 6 PID Last ce er ARA BAER eee Se DU Ee Iud prs 8 7 TID Tasten See t ecu eue e RR RR ecd 8 8 Node List ete Mee a ERESUCRESEURETI E S 8 9 Foregro nd Color ugs er eee e eee da 8 9 Background Color errenneren eaea Re DI e E S e es 8 9 A caret Ae RU C RI ATE AN A A fiche e DR nose b ERR Rin 8 9 Contents Text IUS TL Y os vecti UIT REIS I hao BEENDEN sk pes 8 9 Text Gravity pi ORE REMO RemePLH dos 8 10 Configuration Form Push Buttons lees eee 8 10 Specific Configuration Parameters 0 0 00 cece eee eee eee 8 11 GridIabel a ie eros Des ALL LA Do e o Bd oh oh oh fos 8 12 DataBOX cta dd 8 13 StateGraph s r e se Sed bee pap 8 14 EventGraphi sis econ AE EE PAAR AA AEN 8 16 b taGraphi O Sd ie eh ard aches Aneel icone oe es 8 17 Ruletzoossa
289. ize of 4096 12 bytes on RedHawk systems a kernel trace buffer is 500000 bytes 3 32 User Trace NightTrace Session Manager Buffer Wrap When this is checked the daemon will overwrite the least recently recorded events in the trace buffer when it reaches its maximum size For user trace events the size of the buffer is specified in the Buffer Size field on the User Trace page of the Daemon Definition dialog see User Trace on page 3 33 For kernel trace events the size of the buffer is defined by the operating system On a PowerMAX OS system a kernel trace buffer has a fixed size of 4096 12 bytes which holds 4095 kernel events The total number of trace buffers for kernel events is specified by the kernel tunable TR BUFFER COUNT the default value of which is 5 NOTE The value of TR_BUFFER_COUNT may be changed on a Power MAX OS system via the config command However if this tunable is changed the kernel must be rebuilt and the system restarted for the change to take effect On RedHawk systems there are two kernel trace buffers each of which is 500000 bytes in size The User Trace page of the Daemon Definition dialog see Daemon Definition Dia log on page 3 25 contains settings for locking policies associated with the daemon and the corresponding user applications using the NightTrace API shared memory permis sions and the duration of the timestamp heartbeat as well as specifications of the si
290. l conditions may be specified in a state definition to further constrain the state Instances of states do not nest that is once a state becomes active events that might normally satisfy the conditions for the start event are ignored until the end event is encountered See also qualified state The class of NightTrace functions which provide information about states includ ing start functions end functions and multi state functions Glossary 17 NightTrace Manual StateGraph state_summary table streaming string table Summarize Form summary display area summary function summary syscall syscall Glossary 18 A scrollable display object that graphically displays states as bars and trace events as vertical lines in a Column A pre defined format table defined in usr lib NightTrace tables It contains formats for statistical displays of state matches state durations and state time gaps It determines the default state summary output format The method used by the NightTrace Session Manager of sending trace data from daemons directly to the NightTrace display The pre defined or user defined structure that allows you to group related integer values together and associate each one with a corresponding static character string You hand edit string tables into configuration files The related structure is a format table The NightTrace form that allows you to obtain trace event and state statistics such as minim
291. l for trace events which were logged from Ada tasking programs SYNTAX task_name QE PARAMETERS QE A user defined qualified event If supplied the function returns the name of the task associated with the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the name of the task associated with the current trace event For more infor mation see Qualified Events on page 9 81 RETURN TYPE string 9 31 NightTrace Manual SEE ALSO offset_task_name on page 9 69 thread_name DESCRIPTION The thread_name function returns the thread name associated with a trace event SYNTAX thread name QE PARAMETERS QE A user defined qualified event If supplied the function returns the thread name associated with the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the thread name associ ated with the current trace event For more information see Qualified Events on page 9 81 RETURN TYPE string SEE ALSO offset thread name on page 9 70 Multi Event Functions Multi event functions return information about one or more instances of an event event gap event matches event gap DESCRIPTION The event gap function returns the time in seconds between the most recent occurrence of a specific event and its imme
292. l function calls except summary format and table functions is as follows Optional parts of function calls are in brackets function name parameter The prefix of the function name determines its class as follows offset start end state event Functions with this prefix provide information about the trace event at the specified offset or ordinal trace event number See Offset Functions on page 9 59 Functions with this prefix provide information about the start event of the most recent instance of a state See Start Functions on page 9 34 Functions with this prefix provide information about the end event of the last completed instance of a state See End Functions on page 9 45 Functions with this prefix provide information about instances of states See Multi State Functions on page 9 56 Functions with this prefix provide information about instances of events See Multi Event Functions on page 9 32 Some functions can be optionally suffixed by a number N which specifies the Nth argu ment logged with the trace event N defaults to 1 and can have the values 1 through the maximum argument logged For example arg Returns the first argument argl Returns the first argument arg3 Returns the third argument start id Returns a trace event ID state gap Returns the time between instances of a state 9 9 NightTrace Manual Table 9 3 contains a complete list of functions
293. l until a daemon is started at which point log ging of events will succeed NTALREADY The application has already initialized the trace without an intervening trace end Tracing can continue in spite of this error Solution Remove redundant trace begin calls NTBADVERSION The calling application is linked with the static NightTrace library and the static library is not compatible with the Night Trace library being used by the user daemon Solution Relink 2 7 NightTrace Manual 2 8 NTMAPCLOCK NTPERMISSION NTMAPSPLREG NTPGLOCK NTNOSHMID SEE ALSO the application with the static library version which matches the library version being used by the daemon The selected event timestamp source could not be attached Solution If read access is lacking see your system administra tor This can also occur if the RCIM synchronized tick clock is selected as the event timestamp source but the tick clock is not counting Solution Start the synchronized tick clock by using the clock_synchronize 1M command and restart the application The calling application lacks permission to attach the shared memory buffer Solution Make sure that the same user who started the user daemon is the current user logging trace events in the application The system s IPL register could not be attached Solution If read or write access is lacking see your system administrator or set ntc_use_spl to FALSE
294. les 0 0 0 0 c eee eee eee 5 20 Configuring Display Pages 0 0 0 ees 5 20 ntrace User Interface A eee EAM ee 5 21 Using th Mouse boats ia 5 21 Understanding Pointer Shapes 0 00 0 c eee eee eee eee 5 22 Anticipating Window Layout llle 5 22 Resizing WindOWS i reise ceeds Meek outs Oe a ace oe 5 24 ntrace Notation Conventions 0 0 0c cece eee eee eee 5 24 ntrace Global Window 1 0 0 0 entes ccc ec eee nee ene 5 25 Message Display Area 0 2 eee cece eee 5 25 Menu Bar CIL 5 26 NightTrace Menu Item 0 0 0 eee eee eee 5 26 NeW Paris pl 5 27 Detaul Page iii id tin 5 28 Open Config Elle cres eek hues oe ee ER 5 30 Read Event Map File 0 0 00 eee eee eee 5 31 Elba PEA Bae 5 32 Tools Menu Item s noson a III 5 32 Help Ment Heni a oo okies A cae AS ADM 5 33 The File Selection Dialog Box 00 0 0 e eee cee 5 33 Typing in the Exact File Name 5 33 Scrolling Through Existing File Names eese oo 5 34 Typing in a Filter File Name Pattern eee eene 5 35 Exercise Displaying Trace Events leere 5 35 NightTrace Manual 5 Invoking the ntrace Display Utility Overviewc The trace event display utility ntrace is an interactive graphical debugging and performance analysis tool ntrace textually presents trace run statistics As a tool built on the X Window System it can graphically displays user trace events and
295. lets you get statistical informa tion about trace events and states When you click on Tools I Summarize the Summarize Form appears This form lets you constrain the information to be summarized The Summarize Form consists of Radio buttons Text fields e Summary display area 10 5 NightTrace Manual Push buttons Menu items Figure 10 4 and Figure 10 5 show two Summarize Forms with different configurations Summarize Form Radio Buttons Through the Summarize Form s radio buttons you can choose Whether to summarize trace events or states The interval to summarize The Summary Type radio buttons let you specify the type of information you want summarized Click on the Event radio button to summarize trace event information This is the default setting Click on the State radio button to summarize state information The Summary Range radio buttons let you limit the summary to the current interval to the time between a mark and the current time or to the entire trace session Click on the Trace Event File radio button to summarize data through out the trace session This is the default setting Click on Region to summarize data only between the mark and the current time Click on Interval to summarize data only between the current interval s Time Start and Time End Summarize Form Fields 10 6 All fields of the Summarize Form have default values Because of these defaults cli
296. lock depending on either the system architecture or user specified options to ntraceud The system timer on the Power Hawk PowerStack systems that NightTrace uses to timestamp frace events The fixed period of time for which the kernel allocates the CPU to a process All trace points are associated with a log file established via trace start anda thread name established via trace open threaog If two processes or tasks or threads are associated with the same log file and thread name then they are said to have the same trace context If they differ in log file thread name or both then they have different trace contexts A user defined point of interest in an application s source code that NightTrace represents with an integer trace event ID Alternatively this may be a predefined point of interest in the kernel Along with the trace event ID NightTrace records the timestamp when the trace event occurred any arguments logged with the trace event and the logging process identifier PID A user defined numeric value logged by an application via a trace event trace event file trace event function trace event ID trace event tag trace point trough vector table View mode widget window Glossary An ntraceud created binary file that contains sequences of trace events and data that your application and the ntraceud daemon logged The class of NightTrace functions that provide information about trace events
297. luding the end event The time in the interval up to which all display objects on a display page have been updated The dashed vertical bar that represents the current time in a Column The last trace event on or before the current time line See text cursor The configuration of a particular trace daemon which includes daemon collection modes and settings daemon priorities and CPU bindings and data output formats as well as which trace event types are handled by that daemon A display object that displays possibly variable textual or numeric information A scrollable display object that graphically displays a bar chart of an expression s value as it changes over the interval A menu item that automatically creates a display page to depict context switches interrupts exceptions and system calls with display objects for each CPU on the system Glossary 3 NightTrace Manual Default Page Delete device table dialog box dimmed disabled discarded trace event display object display page dotted area Glossary 4 A menu item that automatically creates a display page with a StateGraph for each trace event logging process in your trace event file s Remove the selected macro qualified event or qualified state A pre defined dynamically generated string table in the vectors file created by nt race when consuming raw kernel trace data files string table contains the names of the devices that are cu
298. lways be zero The syntax for specifying a TID is raw PID task id Or raw PID thread id The TID List parameter is the list of NightTrace thread identifiers TIDs or thread names that the display object will accept trace events from If the trace event did not occur in a thread listed in this parameter the trace event is ignored If a number or name is specified that is not a valid TID a warning message is displayed Multiple numbers and names must be separated by commas NOTE Prior to Version 4 1 ntrace converted thread identifiers into thread names during TID List input verification for a display object For each thread identifier in the TID List ntrace would try to find its associated thread name and display that name in the TID List However because multiple threads having the same name may exist on a system changing a thread identifier into a thread name introduces the possibility that the display object will accept trace events from undesirable threads Therefore ntrace no longer performs this conversion For example suppose that two threads named daemon exist on a particular system and that one has a TID of 1234 1 and the other has a TID of 5678 3 Further suppose that you wish to create a StateGraph to display events only for TID 1234 1 Prior to Ver sion 4 1 if you entered 1234 1 in the TID List parameter ntrace would have converted that to daemon As the events were being analyzed any event that had a TID
299. ly be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO argO on page 9 20 start_arg on page 9 35 end arg dbl on page 9 47 end num args on page 9 48 and offset arg on page 9 60 DESCRIPTION The end arg dbl function returns the value of a particular trace event argu ment associated with the end event of the last completed instance of a state SYNTAX end arg N db1 QS PARAMETERS N Specifies the Nth argument logged with the trace event Defaults to 1 9 47 NightTrace Manual end_num_args end_pid 9 48 OS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE double precision floating point SEE ALSO arg dbl on page 9 21 start_arg_dbl on page 9 36 end_arg on page 9 47 end num args on page 9 48 and offset arg dbl on page 9 61 DESCRIPTION The end num args function returns the number of arguments associated with the end event of the last completed instance of a state SYNTAX end num args QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies I
300. ly called a trace event In user traces each trace event has a user defined trace event ID number and two different trace event IDs delimit the boundaries of a user defined state Some typical user trace point locations include Suspected bug locations Process subprogram or loop entry and exit points Timing points especially for clocking I O processing Synchronization points multi process interaction Endpoints of atomic operations Endpoints of shared memory access code Careful trace point placement allows you to easily identify patterns and anomalies in your application s behavior Introduction Kernel trace points and trace events are pre defined and embedded in the kernel source Languages Supported The NightTrace library is callable from C Fortran and Ada This means that your application can be written in any combination of these languages and still log trace events Processes and CPUs A user daemon is responsible for actually recording the trace events logged by an applica tion to disk It can interact with single process and multi process applications the pro cesses may even run on different CPUs When you log a trace event NightTrace identi fies both the originating process and optionally the CPU User daemons are initiated and managed via the session manager in the ntrace graphical tool or via the ntraceud command line tool Information Displayed The ntrace display utility lets you examine some
301. ly only the default item line A get_string call with this table name as the first parameter needs no second parameter nt race returns a string of the item number in decimal if There is no default item line and the specified item is not found The string table is not found The first time ntrace cannot find a particular string table nt race flags it as an error The following lines provide an example of a string table in a configuration file string_table curr_state item 3 Processing Data item 1 Initializing item 99 Terminating default_item Other y In this example your application logs a trace event with a numeric argument that identi fies the current state curr_state This argument has three significant values 3 1 and 99 When curr_state has the value 3 the ntrace display shows the string Processing Data When it has the value 1 the display shows Initializing When it has the value 99 the display shows Terminat ing For all other numeric values the display shows Other For more information on string tables and the get_string function see set_string on page 9 75 and the usr lib NightTrace tables file Pre Defined String Tables 5 14 The following string tables are pre defined in ntrace event A dynamically generated string table internal to ntrace It maps all known numeric trace event IDs with symbolic trace ev
302. m the Application Comments 1 Invoke ntraceud All trace events enabled 2 Invoke ntraceud d52 Trace event 52 disabled 3 Start application 4 Call trace_event 52 Trace event 52 not logged 5 Call trace_enable 52 Trace event 52 enabled Table 4 5 ntraceud Disable Sequence 2 From the Shell From the Application Comments 1 Invoke ntraceud All trace events enabled 2 Start application Trace event 52 enabled 3 Call trace_event 52 Trace event 52 logged 4 Call trace_enable 52 No effect 5 Invoke ntraceud d52 Trace event 52 disabled SEE ALSO For information on enabling trace events see Option to Enable Logging enable on page 4 26 and trace_enable trace_disable and Their Variants on page 2 16 4 25 NightTrace Manual Option to Enable Logging enable 4 26 By default all trace events are enabled for logging to the shared memory buffer The nt raceud enable option makes the application notice previously disabled requests to log a specific trace event or a range of trace events SYNTAX ntraceud enable JD trace file ntraceud enable D low ID high trace file DESCRIPTION In the first format the ntraceud enable option dynamically re enables a specific disabled trace event ID ID for logging to the shared memory buffer In the second format the ntraceud enable option dynamically re enables a range of disabled trace event IDs D low through ID high for logging to the shared
303. mation SYNTAX start raw pid OS PARAMETERS QS RETURN TYPE integer SEE ALSO A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 pid on page 9 22 raw pid on page 9 23 Iwpid on page 9 23 end pid on page 9 48 and offset pid on page 9 62 9 38 Using Expressions start_lwpid DESCRIPTION The start lwpid function returns the lightweight process identifier LWPID associated with the start event of the most recent instance of a state NOTE See the _1wp_self 2 man page for more information SYNTAX start lwpid QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO pid on page 9 22 raw pid on page 9 23 Iwpid on page 9 23 end pid on page 9 48 and offset pid on page 9 62 start thread id DESCRIPTION The start thread id function returns the thread identifier associated with the start event of the most recent instance of a state NOTE See the thr self 3thread man page for mor
304. mbination of changes it displays an error message in the message display area and restores the affected fields to their previous values When ntrace detects an invalid value it restores the affected field to its previous value Some general rules apply to multiple field editing You must not simultaneously apply changes to more than two trace event fields You must not simultaneously apply changes to more than two time fields for these purposes Current Time is not considered to be a time field Youcan change Current Time with any other valid field changes as long as Current Time falls within the new interval You can change Zoom Factor with any other valid field changes 6 14 Viewing Trace Event Logs with ntrace You can change Increment with any other valid field changes Simultaneously modifying one time field and clearing another time field makes ntrace use the static and modified fields to determine the values of the cleared time field and the other fields e Simultaneously modifying one trace event field and clearing another trace event field makes nt race use the static and modified fields to determine the values of the cleared trace event field and the other fields The following table shows all the valid multiple field changes except those that involve Current Time Zoom Factor or Increment For information on editing specific fields of the interval control area see The Interval Control Area on page 6 1
305. me of the file to which the session configu ration will be saved NOTE Filenames are relative to the host system the system where the NightTrace GUI is running NightTrace Session Manager Save Session sample Figure 3 4 Save Session dialog Directory The directory in which to save the session configuration file The user may type the path into this field directly Filter Of all the files contained in the current Directory display only those Files that match the specified filter Directories Contains a list of all the subdirectories within the current directory Selecting any of these will change the current Directory to that subdirectory Double clicking on any of these directory names will change to that directory and update the Files list accordingly 3 7 NightTrace Manual Unsaved Changes Files Within the current Directory this is a list of the files that match the specified Fil ter Any of these filenames can be selected When selected the filename appears in the Selection field Selection The name of the file to save the session configuration file to appears here after being selected from the Files list The filename may be manually entered into this field by the user as well This dialog is presented whenever the user attempts to terminate a session which has changes that have not yet been saved to the session configuration file Warning There are unsaved changes to the new session
306. mmary display area that says that there are no trace event or state matches to summarize Menu Bar The menu bar of the Summarize Form consists of the following menu item File File Operations When you click on the File menu item on the Summarize Form the pull down menu shown in Figure 10 3 appears 10 9 NightTrace Manual Save Text Save Text As Close Figure 10 3 Summarize Form File Menu When you click on File I Save Text on the Summarize Form ntrace saves your summary text to the file you saved to last time Any changes you have made since the last Save Text or Save Text As operation will be saved You can continue running summaries after this operation The Save Text operation is disabled dimmed if you have not both done a Save Text As and changed the summary display Instead use Save Text As When you click on File Save Text As on the Summarize Form ntrace saves your summary text to the specified file You can continue running summaries after this operation Save Text As usesa File Selection Dialog Box to prompt you for a file name See The File Selection Dialog Box on page 5 33 for more information When you click on File 1 Close on the Summarize Form ntrace ends the current summary session resets all field and radio button settings and clears the summary display area It does not prompt you to save your summary text since the last time you did a Save Text or Save Text As
307. more information on creating and configuring DataBoxes and DataGraphs Tracing the Kernel Exception Information Last Exception Name Current Time Line Exceptions data access Context Switch Lines Figure 11 11 Last Exception Box and Exception Graph Figure 11 11 shows a last exception box and an exception graph The exception graph displays a state that is drawn whenever an exception is executing on the associated CPU Unlike interrupts exceptions cannot nest so they are always graphed with the same height Context switch lines are superimposed on exception graphs It is common to see a context switch line at what looks like the very end or beginning of an exception Usually this does not indicate that the exception has ended only that it has been suspended because the process that originated the exception has switched out The exception resumes when the process is switched back in again An example of an exception being suspended and resumed can be seen at the left end of the exception graph in Figure 11 11 The last exception box displays the last exception prior to the current time line that executed and may still be executing on the associated CPU It can be used with the exception graph to identify any exceptions that are currently visible on the graph Simply move the current time line onto a graphed exception and the last exception box will update to display the name of the exception Because the last ex
308. ms Tables oir a Seo es 11 16 Kernel Reference ia CaS eo e ES ook eas Wa ig 11 18 Intertuptsz a auc se bey to VR One bh o pedro ee toS PEN igen desk aes 11 19 Non Device Related Interrupts 0 0 0 0 cece eee eee 11 19 Device Related Interrupts 0 00 ruere enner 11 20 EXCeDLODS d nere ie a arci soto NO 11 20 NN rv RRRORPRARG RrskR N Gu nE E RUN Ue nE wala gle SI ER NR 11 21 Appendix A Performance Tuning OVERVIEW A A PORE SOM SRE Rea Ree oes A 1 Preventing Trace Events Loss o oooooococorcrcrc eer nee A 1 Ensuring Accurate Timings 0 0 0 eee eee cee eee ee A 3 Optimizing File System and CPU Usage 0 0 0 0 cece eee eee A 3 Conserving Disk Space ns eee wage perl MOR eee E EP E glee eer A 4 Conserving Memory and Accelerating ntrace 0 00 eee eee A 4 Appendix B GUI Customization OVerview vios E br de ee nde ee E dee a B 1 Default X Resource Settings for ntrace lees B 2 Exampl s ee gx Nee up eee e CR RR B 5 Exercise Customizing Display Colors leeren B 5 Appendix C Answers to Common Questions Illustrations Figure 1 1 Example of Instrumented C Code 0 00005 1 4 Figure 1 2 Example of a User Display Page with Display Objects 1 6 Figure 2 1 Inter Process Communication and Library Routines 2 4 Figure 2 2 entry exit c Before Instrumentation 000 2 26 Figure 2 3 entry
309. n It is possible for several TR_EXCEPTION_SUSPEND TR_EXCEPTION_RESUME trace event pairs to occur if the process is switched in and out several times before the exception completes The TR EXCEPTION RESUME trace event has one argument that is identical to the argument logged with the TR_EXCEPTION_ENTRY trace event TR_EXCEPTION_EXIT argl This trace event is logged whenever an exception is completed It has one argument that is identical to the argument that is logged with the TR_EXCEPTION_ENTRY trace event 11 3 NightTrace Manual Syscall Trace Events 11 4 There are four trace events associated with syscalls TR SYSCALL ENTRY argl arg2 arg3 This trace event is logged whenever a syscall is entered It has three arguments arg This argument is always zero for historical reasons arg2 The syscall number that identifies the syscall This is an index into the pre defined syscal1 string table arg3 The device number that indicates the type of device that is associated with the syscall if any This is an index into the pre defined device string table For more information about the pre defined syscall and device string tables see Kernel String Tables on page 11 16 TR_SYSCALL_SUSPEND argl arg2 arg3 This trace event is logged whenever a syscall is suspended by a context switch I
310. n empties the file and prepares the running daemon for another trace run Use the reset option when you are no longer inter ested in the contents of an active trace event file You can invoke ntraceud multi ple times with the reset option SEE ALSO For information on quitting an nt raceud session without clearing the trace event file see Option to Quit Running ntraceud quit on page 4 21 4 20 Generating Trace Event Logs with ntraceud Option to Quit Running ntraceud quit The ntraceud quit and quit options terminate a running ntraceud process SYNTAX ntraceud quit trace_file ntraceud quit trace_file DESCRIPTION Running ntraceud daemons are located using the shared memory identifier keyed by the trace event file name trace file A process completes its NightTrace session by calling trace end or exiting normally The quit and quit option instruct ntraceud to terminate tracing When quit is used ntraceud will wait for all user processes associated with this daemon that are currently tracing to terminate whereas use of quit skips this check The following actions are then taken Remaining trace events are flushed to the trace event file The output file is closed The shared memory buffer is removed unless user applications still exist The running ntraceud daemon terminates TIP You cannot get statistical information after you quit running nt raceud Consider getting statistical info
311. n page 2 16 If your application calls any of the enable routines this method is not entirely effective By disabling tracing this way you have to rebuild your application and there is no saving in compilation time or execution time e Start a user daemon with all events disabled By disabling tracing this way you do not have to rebuild your application but there is no saving in compilation time or execution time Do not start a user daemon By disabling tracing this way you do not have to rebuild your application but there is no saving in compilation or execution time Compiling and Linking You must link in the NightTrace library so that your application can initialize its trace mechanism and log trace events The name of this library depends on your source language C and Fortran applications must link in the usr lib libntrace a library 2 24 Adding Library Calls to Your Application C Example cc fl sim c lntrace lud for PowerMAX OS cc fl sim c lntrace lccur rt for RedHawk Linux This step Compiles the 1 sim capplication Links in the NightTrace library Creates an executable named a out if there were no major errors For more information on compiling and linking C programs see the Concurrent C Reference Manual Fortran Example h 77 turn matrix f lntrace lud for PowerMAX OS S h 77 turn matrix f lntrace lccur rt for RedHawk Linux This step e Compiles the turn matrix f appli
312. n page 9 20 start num args on page 9 37 end num args on page 9 48 and offset num args on page 9 61 pid DESCRIPTION The pid function returns the global process identifier PID associated with a trace event NOTE A global process identifier does not have the same meaning as the typical operating system definition of pid A PID within Night Trace is a 32 bit integer value that contains the operating system process identifier raw PID in the upper 16 bits and the light weight process identifier LWPID in the lower 16 bits Consult the _lwp_global_self 2 man page for more information SYNTAX pid QE PARAMETERS QE A user defined qualified event If supplied the function returns the global process identifier of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the global process iden tifier of the current trace event For more information see Qualified Events on page 9 81 RETURN TYPE integer SEE ALSO PID List on page 8 7 raw_pid on page 9 23 lwpid on page 9 23 start pid on page 9 37 end pid on page 9 48 and offset pid on page 9 62 9 22 Using Expressions raw_pid DESCRIPTION The raw_pid function returns the process identifier raw PID associated with a trace event NOTE A NightTrace raw PID has the same meaning as the typical oper ating system d
313. nds into the trace session ntrace start 5s end 15s log 5 7 NightTrace Manual For example the following invocation skips the first 10 of trace events loads the next 15 of trace events and skips the remaining 75 of trace events ntrace start 10 end 25 ulog If you invoke ntrace with several start options ntrace pays attention only to the last one The same is true if you invoke ntrace with several end options If you invoke ntrace with both a start and a end option and the end condition precedes the start condition ntrace does not load any real trace events it loads two dummy trace events You can establish a default windowing environment for all your ntrace sessions in your Xdefaults or Xresources file You can invoke ntrace with X options to e Customize an individual nt race session e Override any corresponding settings in the Xdefaults file Possibly improve the readability of your ntrace display You can invoke ntrace with the following options flat color Color to use for the window edges scroll bars push buttons and menu bars Xoption This option includes all of the standard X Tool kit command line options see X 1 TIP Consider experimenting with these options and then saving their counterpart values in your Xdefaults or Xresources file Invoking ntrace on a color X server with no nt race options and no ntrace settings in Xdefaults is nearly equivalent to ntrace fg
314. ne A negative number means the pointer is to the left of the current time line 6 3 NightTrace Manual The Grid 6 4 offset 3 User Events argl 0 id 1662 Thread threadi Figure 6 2 The Grid The grid is a region of the display page that is filled with parallel rows and columns of dots These dots serve as reference points for display object alignment You can alter the grid dimensions by changing the size of the display page To change the display page size resize your window by using features of your window manager The trace open thread routine and the ntraceud daemon write overhead trace events into your trace event file The tags for these trace events are NT ASSOC PID and NT ASSOC TID In View mode you may see these trace events in display objects on the grid ntrace assigns each trace event in the trace session a unique ordinal number or off set beginning with ordinal number 0 These ordinal numbers appear in the interval control area and in the message display area For more information on ordinal trace events see The Interval Control Area on page 6 11 Some display objects on the grid contain vertical lines Each vertical line in a StateGraph or EventGraph represents a user trace event kernel trace event or NightTrace overhead trace event If you click on a trace event with mouse button 2 nt race writes information about that trace event in the message
315. ng trace events 2 17 4 4 4 26 End functions 9 45 end_arg function 9 47 Index 3 NightTrace Manual nd_arg_db1 function 9 47 nd cpu function 9 52 nd idfunction 9 46 nd lwpid function 9 50 nd node id function 9 54 nd node name function 9 56 nd num args function 9 48 nd offset function 9 53 nd pid function 9 48 nd pid table name function 9 55 nd raw pid function 9 49 nd task id function 9 51 nd thread id function 9 51 nd tidfunction 9 52 nd tid table name function 9 55 nd time function 9 54 End Events configuration parameter 8 15 End Events field 8 15 9 7 9 85 End Expression configuration parameter 8 15 End Expression field 8 15 9 4 9 7 9 83 10 7 Glossary 14 Environment variable DISPLAY 5 2 Event gap 9 32 matches 8 6 8 13 9 33 qualified 9 81 summary type 10 12 tag see Trace event tag Event Color configuration parameter 8 15 Event Color field 8 15 Event Count field 6 12 6 13 Event End field 6 7 6 12 6 13 Event ID see Trace event ID Event List configuration parameter 8 2 8 4 Event List field 8 2 9 83 9 85 10 5 10 7 Event radio button 10 6 Event Start field 6 7 6 11 6 13 Event summary 10 11 event table 5 14 8 5 Event see Trace event event arg dbl summary table 5 20 8 6 event arg summary table 5 20 8 6 10 8 event gap function 9 32 event matches function 8 13 9 33 event summary table 5 20 8 6 10 7 EventGraph 5 30 6 4 7 15 8 16 9 7 B 4 configuration form 8 16 Event map file 2 13 5 4 5 9
316. ng 2 16 2 24 kernel 1 1 5 15 8 2 8 6 11 1 user 1 1 U umask 1 command 4 1 User tracing 1 1 User level interrupts 4 9 4 11 V Variable DISPLAY 5 2 vector table 5 17 8 6 11 2 11 3 11 17 11 19 vector nodename table 5 17 11 18 vectors file 5 5 5 15 11 2 11 17 11 18 Version ntrace 5 4 ntraceud 4 8 View mode 5 28 6 1 6 6 7 4 9 7 10 1 11 6 Viewing strategy ntrace 6 5 w Window Configuration 8 4 8 10 8 12 8 13 8 14 8 16 8 17 8 19 9 2 9 4 Display Page 5 27 6 2 11 6 Index 16 File Selection Dialog Box 5 31 5 32 5 33 7 19 10 10 B 3 Global 5 3 5 6 5 10 5 25 11 6 A 1 B 2 iconified 5 22 Macro configuration 9 6 9 7 manager 5 22 5 26 6 11 Qualified Event configuration 9 81 Qualified State configuration 9 83 9 84 resizing 5 24 5 26 Search 9 4 10 2 B 3 C 1 Summarize 9 4 10 12 10 14 B 3 Window component menu bar 5 23 5 24 pull down menu 5 23 5 24 push button 5 23 6 8 radio button 5 23 scroll bar 5 23 scroll region 5 23 text field 5 23 Window see Dialog box X X resources display page B 2 X Window System desk accessories 1 7 options 5 8 resources 1 7 11 16 X 1 utility 5 8 xmodmap 1 utility 5 21 xrdb 1 command B 1 xterm 1 utility 1 7 5 8 Zoom Factor field 6 12 6 13 Zoom In push button 6 10 6 12 Zoom Out push button 6 10 6 12 C 1 Zoom Region push button 6 10 Zoom to Include Event radio button 10 3 Spine for 1 5 Bind
317. ng height The bottom DataGraph is a bar DataGraph which consists of bars of varying height The height of the line or bar is proportional to data from the trace event file This display object is usually used to display values of arguments in the trace event record Some examples of ways that a DataGraph can be used are Track the value of an expression over time Identify when an application variable takes on an abnormally high or low value When choosing a size for your DataGraphs make sure that they are high enough for you to distinguish differences in data values See DataGraph on page 8 17 for more information TIP The higher you make the DataGraph the easier it is to differentiate similar data points In View mode to find out about the trace event that caused the data value expression to be evaluated at a particular point position the cursor on the line or bar and click once with mouse button 2 Information about the trace event is displayed in the message display area In View mode to find out the value of a particular data item position the cursor on the line or bar and click once with mouse button 3 The value of that data item is displayed in the message display area Creating Display Objects Ruler 18 47228 18 47238 18 4724 18 47258 18 4726s 18 47278 tra bs ala rra tr el Figure 7 13 Ruler Example The interval control area which is described in The Interval Control Area on page 6
318. ng time in the trace session The new interval starts at the specified time Time Length remains unchanged but other fields including Time End change appropriately If you set Time Start to the word start ntrace resets Time Start to the start time 0 microseconds of the trace session Time End A valid change keeps Time End greater than the beginning time in the trace session and greater than or equal to Time Length The new interval ends at the specified time Time Length remains unchanged but other fields including Time Start change appropri ately Time Length Current Time Event Start Event End Event Count Zoom Factor Viewing Trace Event Logs with ntrace If you change Time End so it is smaller than Time Length ntrace sets Time End to Time Length If you set Time End to the word end or an arbitrarily large number nt race resets Time End to the last time recorded in the trace event file s and changes other fields appropriately A valid change keeps Time Length greater than 0 and less than or equal to the last recorded time in the trace session The new interval length is the specified length Time End and other fields change appropriately If you set Time Length to the word all or an arbitrarily large number nt race resets Time Length to the last time recorded in the trace event file s and changes other fields appropriately The current time is the specified time If the new current time is inside the cur
319. ng trace events The shared memory buffer had 0 events lost There have been 0 unrequested buffer flushes The total number of trace events logged to shared memory is 5 Enabled Events 0 4095 Y Screen 4 2 Sample Output from ntraceud stats Option Defaults for some of these values exist in the header file usr include ntrace h You can override the default values with ntraceud options See Table 4 1 for more information on the default values and the corresponding options used to override them SEE ALSO For information on trace event loss prevention see Option to Establish File Wrap around Mode filewrap on page 4 12 Option to Set Timeout Interval timeout on page 4 17 and Option to Set the Buffer Full Cutoff Percentage cutoff on page 4 18 4 23 NightTrace Manual Option to Disable Logging disable 4 24 By default all trace events are enabled for logging to the shared memory buffer The ntraceud disable option makes the application ignore requests to log a specific trace event or range of trace events SYNTAX ntraceud disable ID trace file ntraceud disable D low ID high trace file DESCRIPTION Sometimes ntraceud logs so many trace events that it is hard to understand the ntrace display Occasionally you know that a particular trace event or trace event range is not interesting at certain times but is interesting at others When either of these conditions exist
320. nic tags or labels with numeric trace event IDs When you click on File I Read Event Map File on the Global Win dow a dialog box appears This window prompts you for an existing event map file To avoid any ntrace errors you must have read permission for this file Once ntrace reads an event map file it is able to display short trace event tags rather than less meaningful trace event IDs By naming your trace events you can make your displays much easier to understand The Read Event Map File Dialog Box is made up of A File Name text field A Filter text field A file name scroll region and its scroll bar The default Read push button The Cancel push button If you decide not to read an event map file click on Cancel This action causes the Read Event Map File Dialog Box to go away If you still want to read an existing event map file see The File Selection Dialog Box on page 5 33 For more information on event map files see Understanding Event Map Files on page 5 9 5 31 NightTrace Manual Figure 5 7 The Read Event Map File Dialog Box Exit When you click on File I Exit on the Global Window nt race Prompts you to save any unsaved changes if appropriate Discards unsaved changes if appropriate Deallocates memory it used to store trace events Exits ntrace puts up a Warning Dialog Box if you try to exit from ntrace without saving your display page changes If you want to save your chang
321. nisesseiswRLRELIDA EON ea elee REE 8 19 Chapter 9 Using Expressions OVERVIEW coupe oar ed et ae A E eae 9 1 Expressions Menu 0 0 eect cence nes 9 1 Expression Dialog Boxes 0 0 00 0 0 eee eee es 9 2 Expression Configuration Forms 0 00 0 serrer errre 9 2 EXPIESSIONS ia Suelo uals sy wl MOA OU ne wad hale ata peta E 9 4 Operators coe tetuer tedbheteiere ete m RU EE is 9 4 Operands ocu eur ec DS EE NU RENE PIDE 9 5 Constants a ERA AE 9 5 Macros WC a e pe is 9 6 Functions a at Ree dd bee is 9 9 Function Parameters uel A oes RARE DRE gt 9 12 Function Terminology 0 0 0 cece eee eee eee 9 13 Trace Event Functions 0 0 cece eee ene rra 9 19 1d i eed e Cbar ot uet Seen 9 19 atg A AAA UU 9 20 aig dbl sk io ee 9 2 num Oria BER 9 2 pid oio op eee ERA oa ee Rad een 9 22 PAW pid iss EE SO oe ag DU RU AUR MT IO UT MAT 9 23 Iwpid ze tosh peas ee EP DM es Soh ce une ks 9 23 thiread 1ad s sor I IDOL EDI E DR MENS er 9 24 task id T 9 25 td cx 9 25 PU ac Pee ee Ogee Ac e CENE 9 26 A O 9 27 A eases Se Saba el ea el tents D 9 27 NODE AAO is go A Y OM 9 28 pid table name Lio nien Ru n ae ee hh co 9 29 tid table name siii RU asin tae meee 9 29 node nam e sse Ice eed ed deer res 9 30 process nare J o hee Sess DIS Ie FR i PEN 9 30 task name iis Seb ee Sk olei 4k e Ar E 9 31 thread inanie i cse pere ead aa 9 32 Multi Event Functions
322. nning of the trace event file with the newest ones this means that ntraceud intentionally discards the oldest trace events to make room for the newest ones Invoke ntraceud with the filewrap option to obtain this behavior The two other ntraceud modes are expansive mode and buffer wrap around mode The process of the ntraceud daemon copying trace events from the shared memory buffer to a trace event file A style of text characters A function that allows you to display a string The pre defined or user defined structure that allows you to group related integer values together and associate each one with a corresponding dynamically formatted and generated character string You hand edit format tables into configuration files The related structure is a string table A pre defined NightTrace entity that may be used in an expression NightTrace pro vides several classes of functions trace event multi event start end multi state offset summary format and table functions The period of time between two trace events possibly the end of one state and the beginning of another See PID The NightTrace window that displays summary statistics pertaining to your trace event files and allows you to open NightTrace related files The mechanism NightTrace uses to receive input and provide displays It is based on the X Window System and Motif Glossary 7 NightTrace Manual grid GridLabel GUI Help host system icon
323. node a warning message is displayed Multiple numbers and names must be separated by commas Foreground Color The Foreground Color parameter determines the color of items in the foreground of the display object which usually corresponds to the data being displayed by the display object Background Color Font Text Justify The Background Color parameter determines the color of the background of the display object Although this is not the color used to display the data of interest in the display object it should be a color that contrasts well with the Foreground Color This will make the data easier to read The Font parameter determines the font that characters in the display object are displayed in Use of a small font size is recommended due to the fact that there is generally a lot of data being displayed and a small font size will help conserve screen space All examples in this manual use the default fixed font that is supplied with all X servers The Text Justify parameter determines the justification of the text in the display object Figure 8 2 shows what each type of text justification looks like 8 9 NightTrace Manual m This is center jus This is Heft justi fied text de LE This is C right just Ps ified text tified text Figure 8 2 Left Center and Right Justified Text Text Gravity The Text Gravity parameter determines whether text in the object will float to
324. not change its appearance or contents depending on the trace event data In addition to specifying the text inside of the GridLabel you also specify the color of the text and background the font of the text and where in the box the text will appear for example top vs bottom See Chapter 8 for more information hardclock Figure 7 8 DataBox Examples Clicking on Create 1 DataBox lets you draw or create a DataBox display object on the grid A DataBox is a rectangle that textually displays data from the trace event file Column Creating Display Objects Although the data is usually related to the last trace event received it can also be a cumu lative total or other manipulations of data in the trace event file DataBoxes are useful when you want to display data that does not lend itself to graphical representation as shown in Figure 7 8 This figure shows three databoxes the top DataBox contains the interrupt name the middle contains the exception name and the bot tom contains the syscall name If the value is too large to fit into the DataBox e g a long trace event tag the lower right corner of the box is filled in If this occurs you should resize the DataBox This is described in Resizing Display Objects on page 7 7 By default numeric data is displayed in decimal integer For information about overriding this default see Understanding Event Map Files on page 5 9 format on page 9 80 and get
325. ns get_format The get_format routine dynamically looks up a string in a format table SYNTAX get_format table_namel int_expr PARAMETERS table_name table_name is an unquoted character string that represents the name of a format table To avoid possible forward reference problems try to make your get_format calls refer to pre viously defined format tables int_expr int_expr is an integer expression that acts as an index into the specified format table int_expr must either match an identify ing integer value in the table_name format table or the table_name format table must have a default item line other wise the results are undefined Often int_expr is based on a NightTrace function If your table consists of only a default item line omit this parameter DESCRIPTION A call to get_format must be the first function call in an expression You must not nest calls to get_format The Then Expression parameter of a DataBox configuration and the Summarize Expression ona Summary Form can call get_format to dynamically locate a string in a format table For each get_format call Night Trace follows these steps 1 Evaluates int_expr 2 Uses this value as an index into table_name 3 Retrieves the associated string from table_name 4 Replaces any conversion specifications in the associated string 5 Returns a string Assume that the following format table definition is in your configuration file format
326. nt tracing in this process PowerMAX Only For more information on shared memory and the system s inter rupt priority level IPL register see the PowerMAX OS Real Time Guide For infor mation about page locking privilege P PLOCK see intro 2 A process that results from the exec 2 system service does not inherit a trace mechanism Therefore if that process is to log trace events it must initialize the trace with trace begin Processes that result from a fork in a process that has already initialized the trace need not call trace begin The trace begin routine must be called only once per parent process unless atrace end call has been made For processes using C threads and PowerMAX OS Ada tasks all threads and tasks will inherit the trace context of the trace begin call that is made by any thread or task of the process RETURN VALUES Upon successful operation the trace begin routine returns NTNOERROR or NTLISTEN the latter in the case where no daemon has yet been started A list of trace begin return codes follows NTNOERROR A daemon has already been started that matches the filename passed as key file The application can begin to log trace events after calling trace open threaoa NTLISTEN All operations where successful but no user daemon matching the filename passed as key file could be found The application can continue to make NightTrace API calls but attempts to log events will fai
327. ntage 4 4 4 5 4 18 A 3 C 1 disable disable logging 4 4 4 24 enable enable logging 4 4 4 26 filewrap file wraparound mode 4 4 4 5 4 12 A 1 A 4 help help 4 7 ipldisable do not set IPL 4 4 4 9 4 28 A 3 lockdisable do not lock pages 4 4 4 11 4 28 A 3 memsize memory size 4 4 4 5 4 16 A 3 C 1 quit quit running 4 21 4 29 reset reset ntraceud 4 20 stats statistical information 4 22 A 1 timeout timeout interval 4 4 4 5 4 17 A 3 version version information 4 8 num_args function 9 21 O Object see Display object Offset 5 7 6 4 6 11 6 13 6 15 7 8 9 9 9 12 9 13 0000000 000000000000 O0 f set_ ffset 9 59 9 60 9 61 9 62 9 63 9 64 9 65 9 66 9 67 9 68 9 69 9 70 10 4 10 11 10 13 ffset function 5 19 9 27 ffset functions 9 59 ffset_arg function 9 60 ffset arg dbl function 9 61 Ffset cpu function 9 66 id function 9 60 9 73 9 74 lwpid function 9 63 Ffset node id function 9 67 ffset node name function 9 68 ffset num args function 9 61 ffset_pid function 9 62 ffset_pid_table_name function 9 67 ffset_process_name function 9 69 ffset raw pid function 9 63 Ffset task id function 9 64 Ffset task name function 9 69 fset thread idfunction 9 64 fset thread name function 9 70 fset tidfunction 9 65 fset tid table name function 9 68 fset time function 9 66 Open Config File menu item 5 12 5 27 5 30 5 36 Open push butt
328. nteresting yet Do one of the following Use interval scroll bar to slowly scroll through total trace run Click on Zoom Out Return to Edit mode alter the display page and return to View mode Click on Tools I Search and set the search criteria Change settings in the interval control area Could the No display use improvement Analyze trace event information Yes Do one of the following Click on Zoom In e Click on Center Click on Mark align the interval and click on e Zoom Region i Figure 6 3 Deciding What to Do Next in View Mode Viewing Trace Event Logs with ntrace The Interval Scroll Bar Although by its position it may look as if it scrolls the grid the interval scroll bar scrolls the interval Moving the slider of the interval scroll bar allows you to examine different intervals in your trace session By moving the slider you change the displays in display objects on the grid and in the interval control area Changes in the display objects are most obvious when you have a Column that contains both a StateGraph and a Ruler For more information on the interval control area see The Interval Control Area on page 6 11 See Chapter 7 for more information on display objects The interval scroll bar is horizontal and extends the entire width of the grid The left arrowhead represents the beginning of the entire trace session not just the part displayed
329. ny field changes makes ntrace search for the next or previous trace event in your trace event file s If you want to restrict this operation by trace event ID trace event tag CPU number node process name or global process identi fier PID thread name or NightTrace thread identifier or expression you can do that by editing one or more of the fields on the Search Form You can restore a field to its default value by entering a single space character or the word default into the field and clicking Apply or pressing Enter 10 4 Using the Built In Tools When you have finished editing the Search Form fields press lt Enter gt or click on Apply This causes nt race to validate the data in each field you modified For general information on field editing and how nt race handles editing errors see Field Editing on page 6 16 When you are ready for nt race to do a search click on the Search push button ntrace logical ORs comma separated lists of values within a field and logical ANDs fields values This means that a trace event must match at least one entry in each list and all criteria from the fields If ntrace locates a trace event that meets every field criterion 1t writes an informative message in the message display area on the display page Depend ing on your preferences it may also reposition the interval and current time line If ntrace does not locate a trace event that meets every field criterion it writes an error
330. o identify any syscalls that are currently visible on the graph Simply move the current time line onto a graphed syscall and the last syscall box will update to display the name of the syscall Because the last syscall box displays the name of the last syscall that executed it is possible for there to be no syscalls visible on the syscall graph even though the last syscall box contains a valid syscall name This just signifies that the last syscall on the CPU ended prior to the beginning of the current interval It is possible for the first syscall logged by a process since kernel tracing began to be unknown This can occur if the process is switched in and immediately resumes a syscall that was previously suspended If this occurs the last syscall box will display can t determine for the name of the syscall The last syscall box is a DataBox display object and the last syscall graph is a StateGraph display object See Chapter 7 and Chapter 8 for more information on creating and configuring DataBoxes and StateGraphs Exception Suscall Figure 11 15 Color Key Figure 11 15 shows the color key that is located on the bottom left of the grid on the pre defined kernel display pages The color key is useful only on X terminals that support more colors than just black and white The text in the color key is color coded By default the word Interrupt is red and all display objects on the kernel display page that display
331. o open the file 1 Type a file name possibly with leading directory name s into the File Name text field 2 If you mistype the file name see Field Editing on page 6 16 and correct the problem 3 Press Enter This causes ntrace to remove the File Selection Dialog Box 5 33 NightTrace Manual If you have read permission to this file and it is of the right type ntrace opens the file If it is a configuration file ntrace puts up any display page s from this file If it is an event map file ntrace adds those trace event tags and trace event argument formats to its internal list If opening the file was not successful ntrace puts up a Warning Dialog Box The warning message in the dialog box differs depending on the problem When you have read the warning click on OK As a result ntrace removes the dialog box If you cause nt race to bring up this File Selection Dialog Box again all fields contain the same values as when you left this dialog box except the File Name text field never comes up with more than a directory name Scrolling Through Existing File Names 5 34 If you would recognize the file name if you saw it use the following steps to find and open it 1 Use the scroll bar to examine the alphabetical list of file and directory names displayed in the scroll region 2 Try to find the file name you are seeking 3 If you find the file a Click on the file name to select and highlight it b Click on
332. objects within them have the same physical starting point and ending point and the same time scale Columns are not configured so the only variations between Columns are in their height and width Without a Column you cannot put any StateGraphs EventGraphs DataGraphs or Rulers on your grid so you must create a Column before you can create any of these display objects You can place a Column anywhere on the grid You can put more than one Column on a grid This allows you to group related graphical objects together All of the Columns however show the same interval and current time in View mode To hold a Ruler and any other graphical display object Columns must be at least five grid dots high Wider Columns are recommended because they determine the resolution to which trace events can be displayed TIP On a monochrome display make sure that you can differentiate among display objects within a Column The easiest way to do this is to leave at least one grid dot between display objects in a Column and to make the background color of the Column black For more information on setting a Column s background color see Default X Resource Set tings for ntrace on page B 2 Figure 7 10 StateGraph Example A state is bounded by two user specified trace events a start event and an end event An instance of a state is the period of time between the start event and end event including the start and end events themselves
333. ode id pid table name tid table name node name process name task name thread name Multi event functions DESCRIPTION The id function returns the trace event ID of the last instance of a trace event SYNTAX id QE 9 19 NightTrace Manual PARAMETERS QE A user defined qualified event If supplied the function returns the trace event ID of the last instance of the trace event which satisfies the conditions of the specified qualified event If omitted the function returns the trace event ID of the current trace event For more information see Qualified Events on page 9 81 RETURN TYPE integer SEE ALSO start id on page 9 35 end id on page 9 46 and offset_id on page 9 60 arg DESCRIPTION The arg function returns the value of a particular trace event argument SYNTAX arg N K OED PARAMETERS N Specifies the Mth argument logged with the trace event Defaults to 1 QE A user defined qualified event If supplied the function returns the specified argument for the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the specified argument for the current trace event For more information see Qualified Events on page 9 81 RETURN TYPE integer SEE ALSO arg dbl on page 9 21 num_args on page 9 21 start_arg on page 9 35 end arg on
334. of the Stream checkbox on the General page of the Daemon Definition dialog see Stream on page 3 31 Advanced settings with respect to streaming can be found on the Streaming page of the Daemon Definition dialog see Streaming on page 3 45 If streaming is not in effect data will be written to the output file see Output File on page 3 32 as specified on the General page of the Daemon Definition dia log Logged The number of trace events that have been written to the stream or written to the file by the associated daemon See Streaming above Buffer The number of trace events currently held in the buffer 3 19 NightTrace Manual These events will be flushed from the buffer either when the Flush Threshold see Flush Threshold on page 3 37 is reached or when the user flushes them man ually see Flush on page 3 21 Lost Lost events occur when the daemon cannot keep up with the rate at which events are being added to the buffer To combat this adjust the Runtime attributes of the daemon by raising its Priority and or by changing its CPU Bias to bind it to a specific CPU See Runtime on page 3 41 for a description of these settings NOTE Events that are discarded when a daemon is Paused see State on page 3 19 are not included in the Lost count Also events that are discarded when the daemon is in Buffer Wrap mode see Buffer Wrap on page 3 33 i e older events
335. olumn hold down the lt Control gt key and click mouse button 2 Display objects must not overlap except certain display objects must be placed inside a Column If you try to move a display object on top of another display object ntrace displays an error message in the message display area and aborts the move Resizing Display Objects To resize a display object on the grid do the following 1 Select the display object See Selecting Display Objects on page 7 6 for more information 2 Using mouse button 3 click on a handle and drag until the desired size is reached 3 Release the right button When display objects are inside a Column it is sometimes difficult to resize the Column To resize a selected Column hold down the lt Control gt key and click mouse button 3 Note that a Column cannot be vertically resized smaller than the minimum space required to hold all the StateGraphs EventGraphs DataGraphs and Rulers that it contains 7 7 NightTrace Manual Display objects must not overlap with the exception that certain display objects need to be placed inside a Column If you try to resize a display object on top of another display object ntrace displays an error message in the message display area and aborts the resize Display Objects 7 8 Figure 7 5 Create Display Objects Menu Display objects which are created via the Create menu shown in Figure 7 5 can be thought of as combination filters and form
336. ommon Configuration Parameters on page 8 1 Like the Event List they each have corresponding If Expressions called Start Expres sion and End Expression respectively The Start Expression parameter determines the criteria in addition to the start trace event s and other criteria which must be true before a state is considered to be started active The End Expression parameter determines the criteria in addition to the end trace event s and other criteria which must be true before a state is considered to be ended inactive The following semantic rules apply to these expressions In these rules defining state means a state with trace events in the Start Events and End Events lists e Start Expression must not refer to its defining states For example it must not call state dur state gap start or end functions for these states See Multi State Functions on page 9 56 Start Functions on page 9 34 and End Functions on page 9 45 for details Calling these functions for these states would be an attempt to define a state based on its own definition Note that Start Expression may call all of these functions for qualified states End Expression must not refer to its defining states For example it must not call state dur state gap or end functions for these states Calling these functions for these states would be an attempt to define a state based on its own definition Note that End Expression may call sta
337. on 5 30 Operands constants 9 5 functions 9 9 macros 5 9 5 11 5 27 5 30 7 18 8 2 qualified events 5 9 5 10 5 11 5 27 5 30 7 18 8 4 9 81 qualified states 5 9 5 11 5 27 5 30 7 18 8 4 9 13 9 35 9 36 9 37 9 38 9 39 9 40 9 41 9 42 9 43 9 44 9 45 9 46 9 47 9 48 9 49 9 50 9 51 9 52 9 53 9 54 9 55 9 56 9 57 9 58 9 59 9 83 Operands in expressions 9 5 Operators in expressions 9 4 Options see ntrace option Options see ntraceud option Options see System configuration option P_PLOCK A 3 Page configuration file 5 9 5 11 default 5 27 5 28 default kernel 5 27 11 6 display 7 2 11 6 B 2 lock disable 4 11 lock privilege 4 28 A 3 new 5 27 Parameters see Configuration parameters Performance considerations ntrace 5 7 10 2 A 5 ntraceud 4 1 4 17 4 18 A 1 PID 7 8 8 5 8 7 9 10 9 22 9 37 9 48 9 62 10 4 10 6 11 2 pid function 8 5 8 7 9 22 9 79 PID List configuration parameter 8 3 8 7 PID List field 8 3 9 81 9 83 Glossary 14 pid table 5 15 8 5 11 18 PID table name 9 29 pid_nodename table 5 16 11 17 pid_table_name function 9 29 Pointer shapes 5 22 Pop up window Open B 3 Read B 3 SaveAs 7 19 B 3 SaveTextAs 10 10 B 3 Pop up window see Dialog box Pre defined tables 5 14 5 20 11 4 11 16 Prev push button 10 3 10 5 printf 3S routine 5 11 5 18 9 80 Privilege page lock 4 28 A 3 Index Process ntrace 5 7 Process box 11 10 Process identifier ending
338. on Definition dialog General Target Name The name for this daemon definition This field is automatically populated with the name daemon x where x is a num ber starting at 0 which increments with each new daemon definition The Name is merely a label to aid the user in identifying specific daemons with a session It has no external meaning and is unrelated to the NightTrace API The user may change this to a name of their choosing Target System The system on which this trace daemon will run NightTrace Manual User The name of the user on the specified Target System responsible for running this daemon Trace Indicates what type of trace events this daemon will be logging Kernel Indicates that the daemon is logging kernel trace events Kernel events are automatically generated by the operating system kernel when a kernel daemon is initiated if the operating system kernel was built with tracing support See the PowerMAX OS Real Time Guide 0890466 for information on con figuring the kernel for kernel tracing on a PowerMAX OS system For systems running RedHawk Linux see the Concurrent Real Time Linux RT User Guide 0898004 for more detailed information User Application Indicates that the daemon is logging user trace events User trace events are generated by user applications that use the NightTrace API the NightProbe tool see the description of the To NightTrace menu item in the chapter titled
339. on Form 0 00 00 e eee eee 2 14 15 NightTrace Manual Screens Tables 16 Figure 8 8 EventGraph Configuration Form 0 000000 2 eee 2 16 Figure 8 9 DataGraph Configuration Form 0 00 00 eee eee 2 17 Figure 8 10 Solid vs No Fill 0 0 2 eee eee ee 2 18 Figure 8 11 Maximum vs Minimum Values 0 00200 eese 2 18 Figure 8 12 Ruler Configuration Form 0 0 00 eee eee ee eee 2 19 Figure 8 13 Mark and Lost Event Markers 0 0 00 00s ee mo m 2 19 Figure 9 1 Expressions Menu oooocccococooo urene rrr eneee 2 1 Figure 9 2 Macro Dialog Box 0 0 eee e 2 2 Figure 9 3 Configuration Form Push Buttons 00 00 0048 2 3 Figure 9 4 Macro Configuration Form 0 0 0 c eee eee eee 2 7 Figure 9 5 Function Terminology Illustrated ooo ooooococooomoooo o 2 14 Figure 9 6 States and Events 0 0 0 ccc re 2 15 Figure 9 7 Qualified Event Configuration Form 004 2 82 Figure 9 8 Qualified State Configuration Form 0 000000 2 84 Figure 10 1 Tools Menu 0 0 0c eee cee eee 2 1 Figure 10 2 The Search Form 0 0 0 cece cee eee 2 2 Figure 10 3 Summarize Form File Menu 0 00000 eee eee 2 10 Figure 10 4 The Event Summarize Form 00 0 e eee eee 2 12 Figure 10 5 The State Summarize Form 0 00 00 e e
340. on page 5 33 for more information Close Close ends the current editing viewing session resets all field and radio button settings and clears the message display area If you have unsaved changes and you do a Close a Warning Dialog Box appears reminding you that you may want to save you changes 7 19 NightTrace Manual 7 20 8 Configuring Display Objects OVERVIEW cues pen Le s CA eu esee Le p ede wel PE dun Acard Common Configuration Parameters 0 0 00 eee eee eee Display Object NaMe s inon ee Ih Event dic es are dees aie pers uev i Er d b Ou PES It Expression i EDI eben BAS bid ThenzEXpressitOn s iv eoiteei uc HOLS AUG hee ARE REARS ee pese RES CPU iste eS Eb vp eb S phe e ders PIDE a Su EE Un m erattu Decet TID List iib A GW A We E Ek Node Liste treege sa dau exe Ps teat Epiit ME Dd mS RES Foreground Color eoo et b RADAR AERE LUE RE Backeround Color vos ease eee RA KENIA Text Justify rrpp pU bh bte ede bts Text Gravity a te te ER E pe Configuration Form Push Buttons 0 0 00 eee ees Specific Configuration Parameters 00 0 0 0 cece eee eee Gridlsabel zu nePLe ker ppt returned er bI RE 8 1 8 1 8 4 NightTrace Manual 8 Configuring Display Objects Overview Customizing a display object so that it displays only the information you want it to in the way that you want it to is called configuring Configuring is done with the Configure I Content menu item shown in Figur
341. on parameters for an event summary are identical to those that are used to configure a DataBox display object See DataBox on page 8 13 for information on how to configure a DataBox By default the Summary Expression for an event summary type displays one line for each of the following in the summary display area The range of ordinal trace event numbers offsets summarized The number of matches summarized The minimum time gap between matches and the ordinal trace event number offset where it began The maximum time gap between matches and the ordinal trace event number offset where it began The average time gap between matches Figure 10 4 shows an event summary 10 11 NightTrace Manual Summarizing from offset 0 thru offset 10915 3877 event matches summarized gap min of 0 000011s B offset 2070 max of 0 offset 10525 avg of 0 013520s pp TR_INTERRUPT_ENTRY E TRUE TUE TRUE Figure 10 4 The Event Summarize Form NOTE The Node List field appears in this dialog only when NightTrace is configured to use an RCIM to timestamp events State Summaries 10 12 Configuring state summaries is similar to configuring StateGraph display objects The configuration parameters for a state summary are identical to those that are used to configure a StateGraph display object See StateGraph on page 8 14 for information on how to configure a StateGraph The main difference between a state
342. one seen SYNTAX max offset expr PARAMETERS expr A numeric expression RETURN TYPE integer NOTE There is no function that returns the trace event ID where the maximum value of the first argument occurred for all matches in the time range You could obtain this value by nesting the functions as follows offset id max offset argl SEE ALSO Summary Functions on page 9 70 and Summarizing Statistical Information on page 10 5 DESCRIPTION The summary matches function returns the number of times the summary cri teria and Filter Expression were matched in the time range NOTE This function should only used in the Summarize Form Its behavior elsewhere is undefined See Summarizing Statistical Information on page 10 5 for more information SYNTAX summary matches RETURN TYPE integer SEE ALSO Using Expressions event_matches on page 9 33 and state_matches on page 9 58 For informa tion about Filter Expression see Summarize Form Fields on page 10 6 Format and Table Functions The format function allows you to display a string The table functions allow you to extract information from user defined and pre defined string and format tables These functions include the following get_string get item get format format For more information about tables see ntrace Tables on page 5 12 and Kernel String Tables on page 11
343. ons Apply 10 3 Close 10 4 10 5 Next 10 3 10 5 Prev 10 3 10 5 Reset 10 3 Search 10 4 Search form radio buttons Backward 10 2 Do Not Move Current Time 10 3 Forward 10 2 Global Search 10 3 Interval Manipulation 10 3 Interval Search 10 3 Scroll Current Time to Event 10 3 Search Constraints 10 2 Search Direction 10 2 Zoom to Include Event 10 3 Search push button 10 4 Searching trace event 1 3 1 6 6 1 6 5 10 1 10 1 Select All menu item 7 18 Select mouse operation 7 6 Shared memory buffer 1 5 4 13 4 16 failure to attach 2 8 flushing 2 20 4 5 4 13 4 18 A 1 A 2 SHMMAX 4 14 4 16 A 2 Start functions 9 34 start arg function 9 35 Start arg dbl function 9 36 start_cpu function 9 41 start_id function 9 9 9 35 start_lwpid function 9 39 start_node_id function 9 43 start_node_name function 9 45 start_num_args function 9 37 start_offset function 9 42 start_pid function 9 37 start_pid_table_name function 9 44 start_raw_pid function 9 38 start_task_id function 9 40 start_thread_id function 9 39 start_tidfunction 9 41 start_tid_table_name function 9 44 start_time function 9 42 Start Events configuration parameter 8 15 Start Events field 8 15 9 7 9 85 Start Expression configuration parameter 8 15 Start Expression field 8 15 9 4 9 7 9 83 10 8 State 1 2 2 14 7 8 7 14 8 15 11 12 11 13 duration 9 57 gap 9 57 matches 8 6 9 58 qualified 9 83 summary type 10 14 State radio button 10 6 State
344. onstructed from trace files on two different nodes NOTE The node information is displayed only when NightTrace is con figured to use an RCIM to timestamp events Tracing the Kernel NightTracet ntracAAAa00BpT buzzard CPU o buzzard CPU 1 hardclock CPU o HE pl TH pid idle oll nip L 0 I LILIMM d gi papio EPU 0 PU 1 22 Interrupt Exception Susall n f i al 0 0000000s me Le 6 2907552s BE 5 2907552s CC Si enEnd zo BO te 3 145376 Figure 11 2 Sample Kernel Display Page nt race determines the number of CPUs on the system from information in the converted KernelTrace trace event file RCIM Default Kernel Display Page When viewing KernelTrace trace event files that have been timestamped by the RCIM tick clock ntrace determines the number of distinct nodes hosts which have trace files and constructs a default display page accordingly When you create a default kernel display page from trace event files that have been timestamped by the RCIM tick clock ntrace pops up a dialog box that allows you to select the nodes you wish to display on that kernel page 1 7 NightTrace Manual 11 8 Select Nodes To Display Figure 11 3 Node Selection Dialog The Available Nodes list shows all nodes that NightTrace has found in the trace files The Selected Nodes list contains all nodes you want shown on the kernel display page you are building NOTE An
345. or all trace events Data appear as numerical statistics and as graphical images You can create and configure the graphical components called display objects or use the defaults By creating your own display objects you can make the graphical displays more meaningful to you You can customize display objects to reflect your preferences in content labeling position size color and font Searches and Summaries With the ntrace display utility you can perform searches and summaries Searches let you filter out unwanted data and zero in on trouble spots and specific data Summaries let you define characteristics of the trace event data to be summarized in several different ways Logging and Analysis NightTrace supports two activities trace event logging and trace event analysis 1 3 NightTrace Manual The User Trace Event Logging Procedure The following text describes user trace event logging Follow these steps in the order shown 1 Establish a suitable environment for running your application and captur ing trace data Make sure you meet all the system requirements discussed in the NightTrace Release Notes for the version you are running 2 Select trace points in your source code A trace point marks a point in your application that is important to debugging or performance analysis 3 Insert a call to a NightTrace trace event logging routine at each trace point in your source code so you can later see the trace event info
346. ore any display objects have been created Interval control area Contains information on the current interval being displayed and the controls to manipulate the display Display Page Modes Edit v View Figure 7 3 Edit and View Mode Buttons Display pages can be operated in one of two modes Edit mode or View mode Edit mode lets you make changes to the display objects View mode lets you view the execution of your application via the trace event file The buttons for Edit and View mode are in the upper left hand corner of the display page If the display is in Edit mode the button beside the word Edit is depressed Otherwise the View button is depressed and the display will be in View mode To change modes click with any mouse button on the button beside the desired mode 7 3 NightTrace Manual Edit Mode View Mode When the display page is in Edit mode you can perform any of the operations on the menu bar except Tools which is disabled dimmed The interval scroll bar push buttons and fields in the interval control area are disabled too Once you have created a set of display objects and configured them you can view the trace event information in the trace event file To view the data in the trace event file the display page must be in View mode However if the display page is in View mode you will not be able to create edit or configure display objects See Chapter 6 for information on running viewing
347. ound Trace event importance General All trace events are important Disk space and mem ory are plentiful Newest trace events are important Disk space is limited Events just before a trace flush are important Program will run a long time Table 4 3 shows how each NightTrace operating mode reacts to a particular condition The process of copying trace events from the shared memory buffer to the trace event file on disk is called flushing the buffer Table 4 3 NightTrace Operating Modes MODE Condition Expansive File Wraparound Buffer Wraparound ntraceud sleep Flush the buffer Flush the buffer No reaction interval exceeded timeout Buffer full cutoff Flush the buffer Flush the buffer No reaction percentage exceeded cutoff Shared memory Flush the buffer Flush the buffer Overwrite the buffer is full buffer s oldest memsi ze trace events with the newest ones Trace event file is N A Overwrite the N A full filewrap file s oldest trace events with the newest ones ntraceud Options ntraceud always copies trace events from the shared memory buffer to the trace event file trace_file You can override some other NightTrace defaults by invoking ntraceud 4 5 NightTrace Manual 4 6 with option s You can also use options to quit running or reset ntraceud and to obtain version statistical or invocation syntax information The full ntraceud invocation syntax is ntr
348. owhead Moves the slider to the right Scrolls forward Increment seconds or Increment percent of the current display interval Click 1 Between an Moves the slider to the side you clicked on arrowhead and Scrolls the current interval by twice the number of seconds in the slider Increment or by twice the percentage in Increment Click or 2 Between an Moves the slider where you clicked and or dragged Drag arrowhead and Scrolls the current interval accordingly the slider If your current time line was not centered centers it Drag lor2 Slider Same as preceding entry Press and Any Left or right Causes animated scrolling of data in the direction the arrow points Hold arrowhead The Interval Push Buttons 6 8 Apply Reset Center Mark Zoom Region Zoom in Zoom Out Refresh Figure 6 5 The Interval Push Buttons The interval push buttons let you examine different intervals in your trace session The eight push buttons appear just below the grid on the display page In the following push button descriptions Click on a push button by first pointing to it and then clicking with mouse button 1 Current Time Time Start Time End Time Length Event Start Event End and Event Count refer to fields in the interval control area Viewing Trace Event Logs with ntrace Except for the Reset push button each push button has an effect on The fields in the interval control area The display objects on the g
349. ows your current edit position within the field A sequence of instructions and associated data that is scheduled and executed as an independent entity Every UNIX process linked with the Threads Library contains at least one and possibly many threads Threads within a process share the address space of the process A 16 bit integer chosen by the threads library that uniquely identifies a thread within a given process Glossary 19 NightTrace Manual TID TID table timestamp Time Base Register time quantum trace context trace event trace event argument Glossary 20 A 32 bit integer that represents a unique context to which trace events can be associated The following syntax numerically specifies a TID raw_PID task_id raw PID thread id or raw PID 0 if neither Ada tasks nor threads are in use The operating system process ID raw PID is contained in the upper 16 bits and either a thread ID task ID or zero is contained in the lower 16 bits A pre defined dynamically generated string table It is internal to NightTrace and associates NightTrace thread identifiers TIDs with thread names This table is not used in kernel tracing The time at which a specific trace event was logged This provides the means by which the chronology of the trace events logged by multiple processes can be assembled The timestamp is obtained from the system interval timer the Time Base Register or the RCIM synchronized tick c
350. page 9 2 for details on this type of dialog Click the Add button on the Macro Dialog Box select the macro from the list and click on the Configure button to pop up a Macro Configuration Form like the one shown in Figure 9 4 9 6 Using Expressions MacroDefinition stddew Expression maxtargl mintargl gt gt 6 Apply Reset Restore Close Figure 9 4 Macro Configuration Form The following parameters can be configured for a macro MacroDefinition The name by which you refer to this macro in expressions Only references to this macro have a dollar sign prefix Expression Any valid expression You must not call macros recursively if you try it NightTrace issues an error and you get undefined results Macros must not call the format and get_format functions For more information about these functions see format on page 9 80 and get_format on page 9 79 EXAMPLES A StateGraph configuration is a good candidate for a macro because it has two expressions that are often related For example the following configuration Start Events FOO Start Expression argl 0x1234 amp amp arg2 0 arg3 gt 700 End Events BAR End Expression argl 0x1234 amp amp arg2 0 arg3 gt 700 graphs states of trace event FOO through trace event BAR where the arguments of the trace events must meet an identical criteria to be considered
351. page 9 47 and offset arg on page 9 60 9 20 arg_dbl num_args DESCRIPTION Using Expressions The arg_db1 function returns the value of a particular trace event argument SYNTAX arg N _db1 QET PARAMETERS N OE RETURN TYPE Specifies the Nth argument logged with the trace event Defaults to 1 A user defined qualified event If supplied the function returns the specified argument for the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the specified argument for the current trace event For more information see Qualified Events on page 9 81 double precision floating point SEE ALSO arg on page 9 20 num_args on page 9 21 start_arg_dbl on page 9 36 end_arg_dbl on page 9 47 and offset_arg_dbl on page 9 61 DESCRIPTION The num_args function returns the number of arguments logged with a trace event SYNTAX num args QE PARAMETERS QE A user defined qualified event If supplied the function returns the number of arguments of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the number of arguments of the current trace event For more information see Qualified Events on page 9 81 9 21 NightTrace Manual RETURN TYPE integer SEE ALSO argO o
352. pens up the real possibility that the user process or the daemon could preempt each other if they are allowed to operate on the same CPU However appropriate use of CPU bindings and or priority at which the dae mon operates can prevent such occurrences Thus if the user also takes simi lar care to schedule their user applications then selecting this is sufficient Page locking is required to prevent preemption while holding a spin lock Without this choice it is possible that a page fault could occur while a spin lock is held allowing for a user application or daemon to spin forever When ntraceud is idle it sleeps The process of copying trace events from the shared memory buffer to a trace event file is called flushing the buffer ntraceud wakes up and flushes the buffer when any of these conditions exist ntraceud s sleep interval elapses The buffer full cutoff percentage is exceeded The shared memory buffer is full of unwritten trace events 4 3 NightTrace Manual Your application calls trace end A summary of NightTrace configuration defaults follows trace flush trace trigger or Table 4 1 NightTrace Configuration Defaults Characteristic Default Modifying Option ntraceud sleep interval 5 seconds timeout seconds Buffer full cutoff percentage 20 full cutoff percent Shared memory buffer size Flush mechanism 16K 16 384 trace events See above memsi ze count bufferwrap
353. pid on page 9 37 and end_pid on page 9 48 offset Iwpid DESCRIPTION The of fset_lwpid function returns the lightweight process identifier LWPID from which the ordinal trace event offset was logged NOTE See the _1wp_self 2 man page for more information SYNTAX offset lwpid offset expr 9 63 NightTrace Manual offset thread id offset task id 9 64 PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE integer SEE ALSO pid on page 9 22 raw pid on page 9 23 Iwpid on page 9 23 start Iwpid on page 9 39 and end Iwpid on page 9 50 DESCRIPTION The offset thread id function returns the thread identifier from which the ordinal trace event offset was logged NOTE See the thr self 3thread man page for more information SYNTAX offset thread id offset expr PARAMETERS offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE integer SEE ALSO thread id on page 9 24 start thread id on page 9 39 and end thread 1d on page 9 51 DESCRIPTION The offset task id function returns the Ada task identifier from which the ordinal trace event offset was logged offset_tid Using Expressions NOTE This function is only meaningful for trace events logged by Ada tasking programs
354. ple EN help usage ntrace h Options that write help version Slasitang filestats Options to select e nohardclock process all process name process PID start offset start time s u start percent end offset end time s u end percent Options for graphic fa laisse Or Ors Xoption Files config file event map file trace file x elp version Salers err gi filestats nohardclock process all name PID start offset time s u percent end offset time s u percent Stat oto XO nae AMAR to standard output Write this message to standard output Write current ntrace version stamp to standard output Chronologically list all events to standard output Write simple trace_file statistics to standard output vents Do not load kernel hardclock interrupt events Load kernel events for all user traced processes Load kernel events associated with process_name Load kernel events associated with PID Load events after the specified event of set Load events after the specified relative time Load events after the specified percent of total events Load events before the specified event offset Load events before the specified relative time Load events before the specified percent of total events al displays Color to use for all flat areas and frames Any standard X Toolkit command line options see X 1 Holds configuration information display pa
355. r 8 2 8 4 If Expression field 8 2 9 4 9 7 9 81 9 83 9 85 10 7 10 8 Glossary 14 Increment field 6 7 6 12 6 14 Inter process communication 2 4 Interrupt 5 5 11 1 11 2 11 12 11 17 11 18 11 19 11 20 device related 11 20 graph 11 12 hardclock 5 7 11 12 11 20 non device related 11 19 user level 4 9 4 11 Interval 1 6 6 1 control area 6 11 push buttons 6 8 scroll bar 6 7 C 1 Interval Manipulation radio buttons 10 3 Interval radio button 10 6 Interval Search radio button 10 3 IPL register disabling 4 9 Index failure to attach 2 8 use 4 9 K Kernel buffer allotment A 2 display page 5 27 11 6 NightTrace trace event file 5 5 5 7 5 27 Kernel tracing 1 1 5 15 8 2 8 6 11 1 Label dimmed 5 27 7 4 Language Ada 1 3 2 25 C 1 3 2 1 2 25 Fortran 1 3 2 2 2 25 Last exception box 11 13 Last interrupt box 11 12 Last syscall box 11 15 libntrace a 2 24 Library routines 2 1 disabling 2 1 overloading in Ada 2 2 return values 2 1 race_close_thread 2 22 race_disable 2 16 4 24 race disable all 2 16 2 24 race disable range 2 16 4 24 race enable 2 16 4 26 race enable all 2 16 race enable range 2 16 4 26 race end 2 7 2 20 2 23 4 4 4 17 4 21 DT EYES GG DE EHOW DE G0 UCES GUY CDOX DT UST race event 2 11 7 8 race event arg 2 11 race event dbl 2 11 race event flt 2 11 race event four arg 2 11 race event two dbl 2 11 race event two flt 2 11 race flush 2
356. r gt Restores changed field s on the Search Form to the value s they had after the last Apply or lt Enter gt This works only if you have not already pressed lt Enter gt or clicked on the Apply push button Clicking on Reset is equivalent to pressing lt Esc gt Goes backward one group of field settings in the search history and displays those settings in the fields You may click on this push but ton multiple times to go backward several groups of settings Click ing on this push button from the earliest group of settings has no effect This push button is useful only after you have clicked on the Search push button Goes forward one group of field settings in the search history and displays those settings in the fields You may click on this push but ton multiple times to go forward several groups of settings Clicking on this push button from the most recent group of settings has no effect This push button is useful only after you have clicked on the Search and Prev push buttons 10 3 NightTrace Manual Close Search Closes the Search Form window and erases all but the last group of field settings from the search history That is if you click on Close and reopen this window during the same nt race session ntrace displays your most recent field settings until you save more field settings clicking on Prev and Next have no effect Performs a search starting at the current time and saves your field changes but not yo
357. r on the specified target system NOTE Passwords are not included in the configuration files written by NightTrace They are retained only during the current invocation of NightTrace Attach Daemons The Attach Daemons dialog is displayed when the user attempts to attach to a daemon running on a remote target system This dialog is presented following user authentication see Login on page 3 10 and Enter Password on page 3 11 on that system Attach Daemons Figure 3 9 Attach Daemons dialog 3 12 NightTrace Session Manager Program ID The process ID PID of the user trace daemon on the remote system Creator The login name of the user who owns the user trace daemon on the remote system Attach as User The login name of the user attaching to the user trace daemon This value defaults to the user specified in the Login dialog see Login on page 3 10 presented prior to this dialog Key File The filename which is used to calculate the shared memory segment identifier asso ciated with the logging of user trace events See Key File on page 3 31 for more information The following buttons appear at the bottom of the Attach Daemons dialog and have the specified meaning OK Attaches to the daemon selected in the list and closes the Attach Daemons dia log Attach as User Brings up a dialog allowing the user to specify the login name used to attach to the selected daemon s Since th
358. race to the name of the node s thread ID table This table is indexed by a node identifier or a node name Examples of using this table are get string name tid 1 get item name tid charon A dynamically generated string table internal to ntrace lt associates node ID numbers which are internally assigned by ntrace with node names This table is indexed by a node identifier or a node name Examples of using this table are get string node name node id get item node name gandalf A dynamically generated string table internal to ntrace In kernel tracing it associates process ID numbers with all active process names for a particular node and resides in that node s vectors file In user tracing it associates global process ID numbers with process names of the processes being traced for a particular node This table is indexed by a process identifier or a process name Examples of using this table are get string pid sbcl pid get item pid engsim nfsd A dynamically generated string table internal to ntrace In kernel tracing this table is not used In user tracing it associ ates NightTrace thread ID numbers with thread names for a particular node Format Tables Invoking the ntrace Display Utility This table is indexed by a thread identifier or a thread name Examples of using this table are get_string tid_harpo 1234567 get_item tid_shark reaper_thread vector See Kernel Str
359. race events and states from your trace event file s nt race displays this information Graphically in configured display object s on the grid Statistically in fields of the interval control area Uniformly on all display page s This means that changes on one page are reflected on all pages ntrace uses the same display page s in both Edit and View modes However toggling between modes changes the interval scroll bar fields in the interval control area and the push buttons In View mode the message display area shows some statistics as well as errors and warnings The default mode for an existing display is View mode View mode lets you locate interesting parts of your trace session by Shifting with the interval scroll bar Clicking on some of the interval push buttons Editing some field s in the interval control area Using the built in Search tool See Chapter 10 for more information See Chapter 7 for more information on Edit mode the components of the display page and display objects This chapter assumes that you have already created or loaded a display page with configured display objects This manual uses the following term conventions lt Enter gt The key on your keyboard that issues a carriage return and line feed lt Backspace gt The key on your keyboard that issues a Ctrl lt h gt In nt race this is also lt Delete gt interval A time period in the trace session that has a specific sta
360. raparound mode NT_M_BUFFERWRAP meaning buffer wraparound mode Shared memory buffer size Buffer full cutoff percentage ntraceud timeout interval Number of threads or processes logging in your application Number of times trace events were lost This statistic refers to a situation that infrequently arises during a NightTrace session ntraceud may lose some trace events if the trace events enter the shared memory buffer faster than ntraceud can copy them to the trace event file For more information on this topic see Preventing Trace Events Loss on page A 1 Number of automatic buffer flushes For more information on buffer flushes see trace flush and trace trigger on page 2 20 Number of trace events logged to shared memory ntraceud and some NightTrace library routines occasionally log predefined trace events into the shared memory buffer Therefore the statistic for number of trace events logged to shared memory may exceed the number of times your application logs a trace event Trace event IDs enabled Screen 4 2 shows a sample of stats option output 4 22 Generating Trace Event Logs with ntraceud E ntraceud stats log OS NTRACEUD STATISTICS The ntraceud daemon is running in NT M DEFAULT mode There is a maximum of 16384 trace events in the shared memory buffer The buffer full threshold is 20 or 3276 trace events The daemon timeout period is 5 seconds There are 1 thread s loggi
361. ration form instead it is labeled with the name of a display object type for example DataBox This parameter allows You to name a particular display object configuration By default newly created display objects bear the name unnamed object You to later define X resources to apply to the named display object See Appendix B for more information ntrace to reference the display object by name in error messages Event List The Event List parameter restricts the trace events on which the display object can display information The display object ignores any trace event IDs or trace event tags that are not on the trace event list If an explicit list of trace event tags and trace event IDs is specified the tags and IDs on the list must be separated by commas Only listed trace events are examined Qualified events and qualified states must not appear in the list If Expression The If Expression parameter determines whether the Then Expression parameter is evaluated If Expressions are boolean i e they should evaluate to false 0 or true non zero If the If Expression is true the Then Expression is evaluated and displayed in the display object assuming all other criteria are met If an If Expression evaluates to false the Then Expression retains its last value See Chapter 9 for more 8 4 Configuring Display Objects information on expressions Some examples of valid If Expressions and their effect on the Then Expre
362. rding trace events is undesirable run ntraceud in expansive mode To do this invoke ntraceud without the filewrap and buffer wrap options See Conserving Disk Space on page A 4 for more information When NightTrace discards trace events it is intentional When NightTrace loses trace events it is not NightTrace does not report discarded trace events it does however report lost trace events Most trace event loss is preventable by flushing the shared memory buffer often NightTrace shows trace event loss in the following ways e Asa non zero events lost statistic from ntraceud stats trace file from ntrace filestats oron the ntrace Global Window Asareverse video L on the ntrace display page Ruler at the location where the trace event was lost If trace event loss seems excessive you can do the following A 1 NightTrace Manual A 2 Action Reason Decrease cutoff the shared memory buffer full Increase the chance that the cutoff percentage for ntraceud ntraceud daemon will have enough time to copy the trace events in the shared memory buffer to disk before the shared memory buffer fills up Decrease timeout the ntraceud timeout Same interval Call trace flush ortrace trigger Same often from within your application especially when your application is at a non time critical point Increase memsize the shared memory buffer Same size for ntraceud Use the following command to
363. re vent event loss The Events page of the Daemon Definition dialog see Daemon Definition Dialog on page 3 25 allows the user to specify which trace event types will be handled by the daemon The user may also change this list dynamically while the daemon is executing by pressing the Trace Events button in the Daemon Control Area of the NightTrace Session Manager see Daemon Control Area on page 3 20 to bring up the Enable Disable Trace Events dialog see Enable Disable Trace Events on page 3 22 NightTrace Manual 0 4099 4356 4911 Figure 3 22 Daemon Definition dialog Events Disabled Events This is a list of user trace or kernel trace event types that are disabled Disabled events are not logged to daemon buffers and therefore are not included in event trace outputs Enabled Events This is a list of user trace or kernel trace event types that are enabled Enabled events are allowed to be placed into daemon buffers and are subsequently transferred to the output device see Trace Events Output on page 3 31 Enable gt Moves the selected items from the Disabled Events list or Trace Event field to the Enabled Events list 3 38 NightTrace Session Manager lt Disable Moves the selected items from the Enabled Events list or Trace Event field to the Disabled Events list Trace Event Allows the user to enter a particular trace event type or range of trace event types and su
364. re trace event ID event type arg integer procedure trace event ID event type arg float 2 11 NightTrace Manual 2 12 functions procedure trace_event ID event_type arg float arg2 float procedure trace event ID procedure trace event ID event type event type arg long float arg long float arg2 long float i procedure trace_event ID event_type arg integer arg2 integer arg3 integer arg4 integer function trace event ID return ntrace error function trace event ID return ntrace error function trace event ID return ntrace error function trace event ID event type arg float arg2 float return ntrace error function trace event ID return ntrace error function trace event ID event type event type event type arg integer event type arg float event type arg long float arg long float arg2 long float return ntrace error PARAMETERS ID argN DESCRIPTION Adding Library Calls to Your Application function trace_event ID event_type arg integer arg2 integer arg3 integer arg4 integer return ntrace_error Each trace event has a user defined trace event ID ID This ID is a valid integer in the range reserved for user trace events 0 4095 inclusive See Pre Defined String Tables on page 5 14 for more information
365. rectly to the NightTrace display buffer Otherwise the output will be written to the Output File see below Key File Specifies a filename which is used to calculate the shared memory segment identi fier associated with the logging of user trace events The daemon and the Night Trace API use the ftok 2 service to map the specified filename to a shared mem ory identifier as used by shmat 2 NOTE When the output method is NOT streaming see Stream above the Key File defines the name of the Output File where trace events are written see Output File on page 3 32 The Key File is relative to the target system It does not necessarily need to be accessible from the host system the system where the NightTrace GUI is running however that can be convienient for subsequent analysis via NightTrace Furthermore the Key File does not have to pre exist If a user application has not already created it via a NightTrace API call the daemon will create the file if it does not exist Browse Brings up a standard file selection dialog so that the user may navigate to the desired location of the Key File 3 31 NightTrace Manual In order to browse the Target System see Target System on page 3 29 must be operational The file selection dialog invoked by that button shows files relative to the Target System Output File The name of the file to which trace events are written The Output File is relative to the
366. rent interval the current time line moves appropriately in any Columns and the current interval remains unchanged If the new current time is outside the current interval the interval shifts so the current time is centered in the interval the current time line is centered in any Columns and the interval length remains unchanged A valid change keeps Event Start less than the number of trace events logged in the trace session The new interval starts at the specified ordinal trace event number offset Time Length remains unchanged but other fields change appropriately If you set Event Start to the word start ntrace resets Event Start to 0 and Time Start to 0 microseconds A valid change keeps Event End non negative The new interval ends at the specified ordinal trace event number offset Time Length remains unchanged but other fields change appropriately If you set Event End to the word end or an arbitrarily large num ber ntrace resets Event End to the total number of trace events in your trace event file s A valid change keeps Event Count less than or equal to the ordinal position offset of the last trace event recorded in the trace session The new trace event count is the specified count Fields change appropriately If you set Event Count to the word all or an arbitrarily large number nt race resets Event Count to the total number of trace events in your trace event file s and changes other fields appropri
367. rid The current time line on the grid The effect of clicking on a particular push button appears next Apply the default Validates any field change s in the interval control area and takes appropriate action Makes corresponding changes to other field s Possibly updates display objects on the grid Possibly moves the current time line in a Column e Is equivalent to pressing Enter Reset Restores changed field s in the interval control area to the value s they had immediately after the last Apply or Enter This works only if you have not already pressed Enter or clicked on the Apply push button sequivalent to pressing Esc Center Centers the interval around the current time line in a Column Makes corresponding changes to Time Start Time End Event Start and Event End Mark Sets a mark that points to a particular time A mark is represented by a solid triangle on the Ruler ntrace currently supports only one mark By default this mark is at time 0 Puts a mark at the current time line of all Rulers e s useful before clicking on Zoom Region Can provide a statistic about the distance between your pointer and the mark Some control sequences pertain to the mark the current time line and your pointer e Simultaneously pressing Ctrl and clicking on mouse button 1 moves the mark and the current time line to the place where your pointer rests Simultaneously holding
368. ring above valuel See valuel above TIP If your table needs only one entry you may omit the item line and supply only the default item line A get_format call with this table name as the first parameter needs no second parameter The following lines provide an example of a string table and format table in a configuration file string_table curr_state item 3 Processing Data item 1 Initializing item 99 Terminating default_item Other y format_table event_info item 186 Search for the next time we process data item 25 The current state is s Wget_string curr_state argl item 999 Current state is s current trace event is d Wget_string curr_state argl offset default_item Other y In this example the first numeric argument associated with a trace event represents the current state curr_state and the event_info format table represents information associated with the trace event IDs When trace event 186 occurs a get_format event_info 186 makes ntrace display Search for the next time we process data When trace event 25 occurs ntrace replaces the conversion specification s with the result of the get_string call If arg1 has the value 1 then nt race displays The current state is Initializing When trace event 999 occurs ntrace replaces the first conversion specification s with
369. ritical settings in the con figuration defined by the user application differ from those specified by ntraceud then ntraceud will fail to initialize with an appropriate diagnostic In the default configuration all trace events are enabled for logging Your application logs trace events to the shared memory buffer By default an architecture specific timing source is utilized For Intel based machines the Intel Time Stamp Counter TSC regis ter for Night Hawk 6000 series machine the interval timer for PowerHawk and Power Stack series machines the Time Base Register TBR However the Real Time Clock and Interrupt Module RCIM can be used as a timestamp source by using the clock option to ntraceud see Option to Select Timestamp Source clock on page 4 19 The ntraceud daemon operates in expansive mode In expansive mode ntraceud copies all trace events from the shared memory buffer to the trace event file This behavior differs from file wraparound mode and buffer wraparound mode If the trace event file does not exist when ntraceud starts up ntraceud creates it otherwise ntraceud overwrites it ntraceud and the NightTrace library routines use page locking to prevent page faults during trace event logging nt raceud uses high performance low intrusion spin locks to protect critical sections involved in logging and consuming trace events These spin locks require preemption control so that processes on the same CPU don t
370. rmation before you quit running ntraceud For statistical information on your trace session see Option to Present Statistical Information stats on page 4 22 Assume that you have invoked nt raceud with the quit option and you want to reinvoke ntraceud with the same trace event file Your next ntraceud invocation will automatically overwrite the trace event file SEE ALSO For information on resetting ntraceud and the trace event file for another session see Option to Reset the ntraceud Daemon reset on page 4 20 4 21 NightTrace Manual Option to Present Statistical Information stats The ntraceud stats option presents a display of statistical information for a running nt raceud daemon on standard output SYNTAX ntraceud stats trace_file DESCRIPTION Running ntraceud daemons are located using the shared memory identifier keyed by the trace event file name trace_file The stats option provides statistical information that tells you about your current NightTrace configuration and resource use This information can help you deter mine if you have adequate resources for your application If you are interested in watching changes in the statistics invoke ntraceud multiple times with the stats option Specifically the stats option provides information on ntraceud mode ntraceud may run in the following modes NT_M_DEFAULT meaning expansive default mode NT_M_FILEWRAP meaning file w
371. rmation graph ically in ntrace You can manually insert these calls into your source code or insert them into the final executable with the NightView debugger See the NightView User s Guide for more information 4 Insert calls at appropriate places in your application to initialize the Night Trace trace event logging library and terminate logging This is necessary for resource allocation and deallocation file creation and flushing trace events to disk Steps 3 and 4 are called instrumenting your code Figure 1 1 shows instrumented C code include lt ntrace h gt define START 10 define END 20 main race_begin log 0 race_open_thread main_thread race event START CEE AI process trace_event END trace close thread trace end exit 0 Figure 1 1 Example of Instrumented C Code 5 Compile and link your application with the NightTrace trace event logging library For example cc main c process c lntrace lud for PowerMAX cc main c process c lntrace lccur rt for RedHawk 6 10 Introduction Start NightTrace and user the NightTrace Session Manager to define a dea mon session used to capture user and or kernel data For example ntrace mgr amp In the NightTrace Session Manager window select the Daemon gt New menu item which brings up aDaemons Definition dialog Click on the User Application radio button to define this as a
372. rrently configured in the kernel A transient secondary window that accepts input or conveys a message for example information errors warnings and questions This construct is occasionally called a pop up window See disabled To flag a component such as a menu item or push button as temporarily unavail able by graying out the label A trace event that ntraceud intentionally did not log in buffer wraparound or file wraparound mode A user configured graphical component of a display page that shows trace events States trace event arguments other numeric and text data Display objects include the following GridLabels DataBoxes Columns StateGraphs EventGraphs Data Graphs and Rulers The NightTrace window that allows you to layout display objects and see trace event and state information in them You can store display pages in configuration files See grid drag duration Edit mode ellipses end function event Glossary To press and hold down a mouse button while moving the mouse Usually you do this in NightTrace to position a display object The period of time between the start and end trace events of some state The display page mode that allows you to create edit and configure display objects macros qualified events and qualified states The other display page mode is View mode An indicator at the end of a menu item that tells you this selection makes a dialog box appear Also an ind
373. rt functions for these states because at this point in the state definition the state has started Note also that End Expression may call all of these functions for qualified states The Event Color parameter specifies the color of the vertical lines that represent the events in the Event List The possible values are the colors your X server supports as specified in the rgb txt file See Appendix B for more information StateGraphs indicate when a state is active by drawing a rectangle in the Foreground Color that spans the time when the start state and end state criteria are met In addition to drawing this state rectangle StateGraphs can behave exactly like EventGraphs by using the Event List and If Expression fields Trace event lines are superimposed on the state rectangle which is useful for diagnosing problems where the criteria for starting the state are met multiple times before the criteria for ending the state are met See StateGraph on page 7 14 for more information See Common Configuration Parameters on page 8 1 for descriptions of the common configuration parameters that StateGraphs use 8 15 NightTrace Manual EventGraph unnamed_object Figure 8 8 EventGraph Configuration Form The configuration form for the EventGraph is shown in Figure 8 8 All of the parameters for the EventGraph are discussed in Common Configuration Parameters on page 8 1 See EventGraph on page 7 15 for more information
374. rting and end ing time It is the window into the trace session that appears on the display page 6 1 NightTrace Manual Menu Bar Mode Buttons Message Display Area The Grid E Thread threadl E User Events Interval Scroll Bar Time Start 0 0000000s Time Length 4 5008572s Time End 4 5008572s Event Start o Event Count 5 Event End 4 Interval Zoom Factor 2 0 Increment 25 00z Current Time 1 1252140s Control Apply Reset Center Mark Zoom Region Zoom In Zoom Out Refresh Area Interval Push Buttons Figure 6 1 A Display Page in View Mode current time The instance in time currently being displayed It occurs within the interval Searches begin at the current time current time line The dashed vertical bar that represents the current time in a Column This line moves to the location of your pointer when you click with mouse button 1 in a Column The position of the current time line determines the values that appear on display pages This chapter covers the following topics Mouse button operations in View mode Understanding the grid Deciding what to do next in View mode Using the interval scroll bar Using the interval push buttons Understanding the interval control area 6 2 Viewing Trace Event Logs with ntrace Field editing Mouse Button Operations Mouse button operations in View mode appear in Table 6 1 and in the NightTrace Pocket Reference card Unfamiliar t
375. rupt prior to the current time line that executed and may still be executing on the associated CPU A DataBox on a kernel display page It displays the last syscall prior to the current time line that executed and may still be executing on the associated CPU lightweight process identifier See LWPID Glossary 9 NightTrace Manual lost trace event LWPID macro mark match menu menu bar message display area monochrome display A trace event ntraceud was unable to log Several ntraceud options exist to prevent this trace event loss An integer that represents an operating system lightweight process identifier It makes up the second half of a PID A user defined named expression stored in a configuration file When you call a macro precede the macro name with a dollar sign The solid triangle on a Ruler that points to a particular time A trace event or state that meets user defined qualifying configuration criteria A list of user selectable choices The horizontal band near the top of a window that contains a list of labeled pull down menus The scrolling region of the Global Window or the display page that holds textual statistics as well as error and warning messages A black gray and white X server display See also color display most recent instance of a state mouse Glossary 10 If the current time line is positioned within a current instance of a state then it is that instance o
376. s The glossary contains an alphabetical list of NightTrace X and Motif words and phrases used in this manual and their definitions The index contains an alphabetical list of topics names etc found in the manual Man page descriptions of programs system calls subroutines and file formats appear in the system manual pages NightTrace Manual Syntax Notation The following notation is used throughout this guide italic list bold list emphasis window i Books reference cards and items that the user must specify appear in italic type Special terms and comments in code may also appear in italic User input appears in list bold type and must be entered exactly as shown Names of directories files commands options and man page references also appear in list bold type Operating system and program output such as prompts and mes sages and listings of files and programs appears in list type Keywords also appear in list type Words or phrases that require extra emphasis use emphasis type Keyboard sequences and window features such as button field and menu labels and window titles appear in window type Brackets enclose command options and arguments that are optional You do not type the brackets if you choose to specify such option or arguments Braces enclose mutually exclusive choices separated by the pipe I character where one choice must be selected You do not type the braces or the pipe c
377. s a string SYNTAX format format_string arg PARAMETERS format_string format_string controls how the optional args are displayed format_string is based on the format parameter used in the printf 3S routine in C It is a character string enclosed in double quotes that contains literal characters and conversion specifications The literals are copied as is to the display object Conversion specifications modify zero or more args arg arg is an optional expression to be formatted and displayed DESCRIPTION Call the format function to display a string You can do this only from the Then Expression parameter of a display object configuration or the 9 80 Using Expressions Summary Expression of the Summarize Form A call to format must be the first function call in an expression You must not nest calls to format The following lines provide examples of format statements and what they dis play Assume all variables have a value of 10 decimal format Error Error format Event d id Event 10 format Argument is X argl Argument is A RETURN TYPES On successful completion format returns a string Otherwise it returns an empty string Qualified Events A qualified event is a user defined named event configuration that consists of a set of one or more trace events possibly restricted by an If Expression CPU List TID List PID List and Node List Qualified events provide
378. s for the current default minimum and maximum shared memory segment size etc conf bin idtune g SHMMAX See the idtune 1M man page for more information Generating Trace Event Logs with ntraceud SEE ALSO For more information on trace flush see trace flush and trace trigger on page 2 20 For a comparison of expansive file wraparound and buffer wrap around modes see ntraceud Modes on page 4 4 For information on limiting the number of logged trace events see Option to Define Shared Memory Buffer Size memsize on page 4 16 4 15 NightTrace Manual Option to Define Shared Memory Buffer Size memsize By default the shared memory buffer can hold 16 384 trace events When the buffer is full of unwritten trace events the ntraceud daemon wakes up and copies the trace events to the trace event file The ntraceud memsi ze option lets you alter the size of the shared memory buffer SYNTAX ntraceud memsize count trace file DESCRIPTION The ntraceud memsize option lets you set the shared memory buffer size Specify the count parameter as a maximum number of trace events or as a number with a K or M suffix to show that the count parameter is in kilobyte or megabyte units respectively For example 12K means 12 288 trace events ntraceud rounds that number up to a full page boundary A trace event with zero or one argument takes up 16 bytes a trace event with more arguments takes up 32 bytes 16 by
379. s of time between them What is the cause You are probably running your application on a Series 6000 system and are calling clock settime This system call can corrupt the system interval timer which NightTrace uses for trace event timings NightTrace Manual Q How can I prevent kernel trace events from being lost A e Verify that the raw kernel trace output file is on a local file system and not an NFS file sys tem e Ask your system administrator to increase the size of TR BUFFER COUNT kernel tunable parameter PowerMAX Only ncrease the priority of the kernel trace daemon Bind the kernel trace daemon to a CPU with minimal activity Glossary This glossary defines terms used in the documentation Terms in italics are defined here Ada task Add Apply argument boolean table buffer wraparound mode button click An Ada task is a construct of statements which logically execute in parallel with other tasks within an Ada program process Tasks communicate asynchronously via variables whose visibility is defined by normal Ada scoping rules Tasks communicate synchronously via rendezvous between a calling and accepting task A push button that creates a new macro qualified event or qualified state on the current display page A push button that validates and saves all changes The same functionality is available by pressing lt Enter gt in a modified field See trace event argument A pr
380. s see TID List on page 8 8 Naming your threads can make the displays much more readable trace open thread lets you associate a meaningful character string name with the current threads more cryptic numeric TID If you provide a character string as the thread name the ntrace dis play utility uses it as a label in its displays Because nt race may be unable to display long strings in the limited screen space available keep thread names short Long thread names cause NightTrace to log an NT_CONTINUE overhead trace event The following words are reserved in NightTrace and should not be used in upper case or lower case as thread names NONE ALL ALLUSER ALLKERNEL TRUE FALSE CALC See Pre Defined String Tables on page 5 14 for more information about thread names NOTE Thread names must begin with an alphabetic character and consist solely of alphanumeric characters and the underscore Spaces and punctuation are not valid characters 2 9 NightTrace Manual DESCRIPTION A NightTrace thread can be a process C thread or Ada task For ntrace displays trace open thread associates a thread name with the process thread or task logging trace events Each process including child processes that logs trace events must have its own trace open thread call In addition C threads and Ada tasks may call trace open thread individually to associate unique thread names with their trace
381. see NightTrace Session Manager Menu Bar on page 3 2 NightTrace Session Manager Figure 3 12 Tools Menu NightProbe Opens the NightProbe Data Monitoring application NightProbe is a real time graphical tool for monitoring recording and altering program data within one or more executing programs without significant intrusion NightProbe can be used in a development environment as a tool for debugging or in a production environment to create a control panel for program input and output See also e NightProbe User s Guide 0890480 NightBench Builder Opens the NightBench Program Development Environment NightBench is a set of graphical user interface GUI tools for developing software with the Concurrent C C and MAXAda compiler toolsets NOTE NightBench is currently not available on RedHawk systems See also NightBench User s Guide 0890480 NightSim Scheduler Opens the NightSim Application Scheduler NightSim is a tool for scheduling and monitoring real time applications which require predictable repetitive process exe cution With NightSim application builders can control and dynamically adjust the periodic execution of multiple coordinated processes their priorities and their CPU assignments See also NightSim User s Guide 0890480 3 15 NightTrace Manual Help 3 16 NightView Debugger Opens the NightView Source Level Debugger NightView is a graphical source level de
382. see the system settings for the current default minimum and maximum shared memory segment size etc conf bin idtune g SHMMAX See the idtune 1M man page for more information A few other factors can affect trace event loss Processes in your application may write trace events into the shared memory buffer at the same time that nt raceud is flushing trace events from the shared memory buffer to the trace event file if the trace event incoming rate exceeds the flush rate trace events may not be recorded Furthermore when NightTrace must choose between operating unobtrusively and logging all trace events it favors being unobtrusive See Chapter 4 for more information on ntraceud options and modes For more information on trace flush or trace trigger see trace flush and trace trigger on page 2 20 If events are being lost during kernel tracing Verify that the output KernelTrace trace event file is on a local file system and not an NFS file system If you run the following command and there is a colon in the Filesystem column the file is on an NFS file system df kernel trace file e Ask your system administrator to increase the size of TR BUFFER COUNT in etc conf mtune d trace by running the idtune 1M com mand rebuild and reboot the system Usually a TR BUFFER COUNT of 5 is sufficient The kernel allocates buffers of 3 pages each 12 288 bytes for kernel tracing This is part of the kernel s
383. sion would look something like argl gt 5 amp amp argl 100 If you wanted to determine the largest of these argument values your Summary Expression would look something like max argl In another example the following configuration Event List 100 If Expression TRUE Summary Expression min argl prints out the minimum value of the first argument of every trace event logged with a trace event ID of 100 To find the offset where this minimum occurred set Summary Expression min offset argl 10 7 NightTrace Manual If you want both statistics use the following Summary Expression format min d at Sa min argl min_offset argl TIP If you are interested in many statistics or if you are going to reuse this summary format at a later date consider defining and using a format table For example Summary Expression get_format my_table The lack of a second parameter indicates that the only entry in format table my_t able is the default item line The pre defined event_arg_summary format table has four formats defined in it Format 1 produces summary data on arg1 format 2 does the same for arg2 etc For more information about format tables see Format Tables on page 5 17 and the end of usr lib NightTrace tables TIP The min offset and max offset functions return the offset of the first trace event where the expression minimum or maximum was seen
384. sons including parity errors hardware failures and kernel errors An exception indicating that a load store or exchange instruction was attempted with a destination memory address not consistent with the size of the access An exception indicating one of several possible conditions including divide by zero invalid instruction and privilege violation An exception generated during single stepping of the CPU The list system calls can be found in the architecture dependent syscal1 string table that is dynamically generated into the vectors file 11 21 NightTrace Manual 11 22 Overview A Performance Tuning Although NightTrace s defaults are designed for maximum efficiency your NightTrace environment and application may have special requirements that warrant some performance tuning You may want to investigate the following issues Preventing trace event loss Ensuring accurate timings Optimizing file system and CPU usage Conserving disk space Conserving memory and accelerating ntrace Preventing Trace Events Loss By default NightTrace copies all user trace events from the shared memory buffer to the trace event file This means that normally NightTrace neither discards nor loses trace events To conserve disk space you may invoke ntraceud with the filewrap or buffer wrap option However by doing so you are telling NightTrace to intentionally discard older or less vital trace events If disca
385. ss Syscall Graph Color Key Current Time Line Figure 11 5 Per CPU Information There are several pieces of information being displayed for each CPU The position of the current time line determines the values that appear on the kernel display pages Moving the current time line within the current interval does not change the graphical displays However the textual displays always reflect the last values prior to the current time line The following sections discuss all of the different pieces of information in detail CPU Information CPU o Figure 11 6 CPU Box 11 9 NightTrace Manual Figure 11 6 shows a CPU box The CPU box simply identifies which logical CPU the displayed data corresponds to Logical CPU numbers are related to but not necessarily identical to physical CPU numbers Each CPU in a system has a four bit physical CPU number The physical CPU number is dependent on which card slot the CPU card containing the CPU is in and which location on the card the CPU is in The low two bits of the number specify the location on the card that the CPU is in These bits are either 00 for the first CPU location or 01 for the second The high two bits of the physical CPU number contain the CPU card slot number These bits can be 00 01 10 or 11 or in decimal 0 1 2 or 3 For simplicity most kernel utilities translate the physical CPU numbers into logical CPU numbers The mapping is accomplished by lis
386. ss name l l 9 69 offset task name llle 9 69 offset_thread_name 0 0 0 0 ccc eee ees 9 70 Summary Functions 0 0 0 ec en 9 70 TING Lt else nena A A A a 9 71 IDax oe EIE EE RCM ene ee a 9 7 AVEO RC PIT EP 9 72 Sum Yrs bere ene cnet DSL GER IE Se ied d 9 72 MIOS a RE ERDEMUU Nx IG IEEE 9 73 max _ OffsetQ 2 4 ec eol dd bee e m es 9 73 summary matches sese s 9 74 Format and Table Functions 0 0000 e cece eens 9 75 PSL STINE tri toe dete wed E a 9 75 O A E E E d gre ee E de 9 77 PS ose bw ee eae Ras aoe Pe Ae IMS 9 79 format s A eae 9 80 Qualified EVO comio dara a E CECI 9 81 Qualified States de oie Se es 9 83 Chapter 10 Using the Built In Tools DN AL An ces US E 10 1 Searching for Points of Interest llle 10 1 Search Form Radio Buttons 0 0 ce eee eee nes 10 2 Search Form Push Buttons 0 0 0 0 eee en eee nes 10 3 Search Form FieldS ooooooooo ooo 10 4 Summarizing Statistical Information 00 0 0 cece eee eee 10 5 Summarize Form Radio Buttons 0 0 0 een nee 10 6 Summarize Form Fields 0 0 00 00 ccc teen o 10 6 Summarize Form Push Buttons 0 0 0 0 eens 10 8 Menu Bar 24 E AA AA BG A UA AU US ADS 10 9 Pile peratlons o sete A ed E x 10 9 Dave Texts nu T ee RU oes EE DINI UV SEED INA EZ 10 10 Save TEXAS s sneha Che tte eile dy ose at De a eal ERA 10 10 GIOSE2 NN eat 10 10
387. ssible to group trace events from related but not identical activities In this case a change of trace event ID usually separates or subdivides groups Probably the most common use of trace events is to identify states Two different trace event IDs delimit the boundaries of a state Most applications log recurring states with different time gaps from the end of one instance of a state to the start of another and different state durations from the start of one instance of a state to its end TIP Consider putting related trace event IDs within a range Library routines and user daemon options let you manipulate trace events by using trace event ID ranges By default all trace events are enabled for logging The NightTrace library contains routines that allow you to selectively or globally enable or disable trace events The user daemon has options that provide similar control Attempting to log a disabled trace event has no effect See trace_enable trace_disable and Their Variants on page 2 16 for more information TIP Consider using symbolic constants instead of numeric trace event IDs This would make your calls to NightTrace routines more readable Once your application logs all of its trace events you can look at them and their arguments graphically with StateGraphs EventGraphs and DataGraphs in the ntrace display utility See StateGraph on page 7 14 EventGraph on page 7 15 and DataGraph on pa
388. ssion are shown in Table 8 2 Table 8 2 Examples of If Expressions If Expression Effect on Then Expression true Always evaluated false Never evaluated id 200 Evaluated if current trace event ID is equal to 200 id lt 200 Evaluated if current trace event ID is less than 200 pid 237 1 Evaluated if current global process ID is equal to 237 1 tid 895 3 Evaluated if current NightTrace thread ID is equal to 895 3 cpu cpu Evaluated if current trace event occurred on CPU 2 or 4 Then Expression The Then Expression parameter determines what the output of the display object is when the If Expression is true If the If Expression is false the Then Expression retains its last value The possible values are a numeric expression or string See Chapter 9 for more information on expressions Some examples of valid Then Expressions and their resulting values are shown in Table 8 3 Table 8 3 Examples of Then Expressions Then Expression Resulting Value or Meaning id arg2 format V abc d argl get_string curr_state id get_string event id get_string pid pid The current trace event ID The second argument of the current trace event The string abc 10 if argl is 10 See format on page 9 80 The string from the curr_state string table pointed to by id if any Depending on whether trace event ID returned by id isin th
389. ssions offset_expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE integer SEE ALSO argO on page 9 20 start_arg on page 9 35 end_arg on page 9 47 offset arg dbl on page 9 61 and offset_num_args on page 9 61 DESCRIPTION The o fset arg dbl function returns the value of a particular trace event argument for the ordinal trace event offset SYNTAX offset arg N dbl offset expr PARAMETERS N Specifies the Nth argument logged with the trace event Defaults to 1 offset expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE double precision floating point SEE ALSO arg dbl on page 9 21 start arg dbl on page 9 36 end arg dbl on page 9 47 offset arg on page 9 60 and offset num args on page 9 61 DESCRIPTION The offset num args function returns the number of arguments logged with the ordinal trace event offset SYNTAX offset num args offset expr 9 61 NightTrace Manual offset_pid 9 62 PARAMETERS offset_expr An expression that evaluates to the offset or ordinal trace event number of a trace event RETURN TYPE integer SEE ALSO num_args on page 9 21 start_num_args on page 9 37 end_num_args on page 9 48 offset_arg on page 9 60 and offset arg dbl on page 9 61 DESCRIPTION
390. ssociated with the end event of the last completed instance of a state NOTE A NightTrace raw PID has the same meaning as the typical oper ating system definition of pid See the getpid 2 man page for more information SYNTAX end raw pid OS 9 49 NightTrace Manual end_Iwpid 9 50 PARAMETERS QS RETURN TYPE integer SEE ALSO A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 pid on page 9 22 raw pid on page 9 23 Iwpid on page 9 23 start pid on page 9 37 and offset pid on page 9 62 DESCRIPTION The end 1wpid function returns the lightweight process identifier LWPID associated with the end event of the last completed instance of a state NOTE See the _1wp_self 2 man page for more information SYNTAX end lwpid QS PARAMETERS QS RETURN TYPE integer SEE ALSO A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 pid on page 9 22 raw pid on page 9 23 Iwpid on page 9 23 start pid on page 9 37 and offset pid
391. st appear in the order shown Include all special characters from the syntax except the ellipsis and square brackets The fields in a string table definition are string table The keyword that starts the definition of all string tables table name The unique user defined name of this table This name describes the relationship of the numeric values in this string table An item line associates an integer value with a character string This line extends from the keyword item through the semicolon You may define any number of item lines in a sin gle string table The fields in an item line are item The keyword that begins all item lines 5 13 NightTrace Manual int_const An integer constant that is unique within table_name It may be decimal octal or hexadecimal Decimal values have no special prefix Octal values begin with a zero 0 Hexadecimal values begin with Ox str_const A character string to be associated with int_const Keep this string short otherwise ntrace may be unable to display it in the limited window space available Use a n for a newline not a carriage return in the middle of the string The optional default item line associates all other integer values with a single string The fields of the default item line are default item The keyword that begins all default item lines str_const See str_const above TIP If your table needs only one entry you may omit the item line and supp
392. standing pointer shapes Antcipating window layout Resizing windows Using the Mouse It is assumed that your X server has a three button mouse By default mouse button 1 is the leftmost button button 2 the middle button and button 3 the rightmost button You can reassign the functions associated with mouse buttons by using xmodmap 1 If you do not have a three button mouse or a standard 101 key North American keyboard see your system administrator or read sections on input and navigation in the OSF Motif Style Guide You use the mouse with point and click interfaces in nt race Each mouse button has a different purpose The only mouse button operation you need to know for now is that clicking mouse button 1 usually does a single selection 5 21 NightTrace Manual Understanding Pointer Shapes When you move the mouse the mouse pointer moves on the screen You use this pointer to point to particular parts of the screen The shape of the pointer shows the current usage The following table describes when nt race uses each pointer shape Table 5 1 ntrace Pointer Shapes and Their Usage Shape When Used k By default A When a menu action is pending d While moving a display object E T Al While resizing a display object Ks 2 During display object placement During time consuming operations for example e while scrolling through a large trace event file Anticipating Window Layout Your window manager m
393. start offset QS trace event end offset 0S min offset expr The integer ordinal number offset of a max offset expr trace event associated with a minimum or maximum occurrence of expr min expr The minimum maximum average or sum max expr of expr values before the current time The avg expr return type is that of expr sum expr get string fable name int expr The character string associated with item int expr in string table table name get item table name str const The first integer item number associated with string str const in string table table name get format fable name int expr The character string associated with item int expr in format table table name format format string arg A character string to format and display Function Parameters If the function has a parameter the parentheses are required Otherwise they are optional For example arg2 No parentheses are required arg2 No parentheses are required arg2 GAK Parentheses are required In many functions the parameter is optional because it can be inferred from context For trace event functions the current trace event is used if the parameter is omitted For state functions the state being defined is used if the parameter is omitted Thus state func tions without parameters can only be used inside state definitions For example argl Operates on the current trace event argl
394. summary 10 12 state_dur function 9 57 state_gap function 9 9 9 57 state_matches function 9 58 state_status function 9 58 9 85 state_summary table 5 20 8 6 10 7 StateGraph 5 28 5 29 6 4 7 14 8 14 9 7 9 84 10 12 11 13 11 16 B 4 configuration form 8 14 End Events configuration parameter 8 15 End Expression configuration parameter 8 15 Start Events configuration parameter 8 15 Start Expression configuration parameter 8 15 Statistics 10 1 multi event 9 32 multi state 9 56 ntrace 5 6 6 9 A 1 ntraceud 4 22 A 1 summary 9 4 9 70 String table 5 9 5 13 9 75 9 77 boolean 5 16 8 6 device 5 17 11 4 11 17 device_nodename 5 17 11 18 event 5 14 8 5 get_item function 9 4 9 77 get_string function 5 17 5 18 5 19 8 5 9 4 9 75 name_pid 5 16 11 17 Index name_tid 5 16 node_name 5 16 11 17 pid 5 15 8 5 11 18 pid_nodename 5 16 11 17 syscall 5 17 8 6 11 4 11 17 11 21 syscall_nodename 5 17 11 18 tid 5 15 8 6 tid_nodename 5 16 vector 5 17 8 6 11 2 11 3 11 17 11 19 vector_nodename 5 17 11 18 sum function 9 72 Summarize menu item 10 5 Summarize form 9 4 10 12 10 14 B 3 Event summary type 10 12 State summary type 10 14 summary display area 10 7 10 9 10 10 10 11 10 12 Summarize form fields 10 6 Filter Expression 9 4 10 7 Summary Expression 9 4 9 81 10 7 10 8 10 11 10 12 Summarize form push buttons Apply 10 8 Clear 10 9 10 10 Reset 10 8 Restore 10 9 Summarize 10
395. system A 2 A 3 C 2 NightStar tool kit 1 1 NightTrace environment defaults 4 2 product 1 1 NightTrace thread identifier 7 8 8 5 8 6 8 8 9 10 9 25 9 41 9 52 9 65 10 4 10 6 NightView debugger 1 4 2 1 No Event List field 10 5 Node box 11 11 Node identifer 9 28 Node identifier ending trace event 9 54 offset 9 67 starting trace event 9 43 Node List configuration parameter 8 3 8 9 Node List field 8 3 9 81 9 83 Glossary 14 Node name 9 30 ending trace event 9 56 ordinal trace event 9 68 starting trace event 9 45 node_id function 9 28 node_name function 9 30 node_name table 5 16 11 17 NT_ASSOC_PID 6 4 NT_ASSOC_TID 6 4 NT_CONTINUE 2 9 2 13 4 16 5 6 NT M BUFFERWRAP see Buffer wraparound mode NT M DEFAULT see Expansive mode NT_M_FILEWRAP see File wraparound mode ntrace 1 3 5 1 format tables 5 9 5 17 functions 9 9 invoking 5 3 notation conventions 5 24 operands 9 5 operators 9 4 performance considerations 5 7 10 2 A 5 string tables 5 9 5 13 viewing strategy 6 5 ntrace field Background Color 8 3 CPU List 8 2 9 81 9 83 Glossary 14 Current Time 6 7 6 11 6 13 End Events 8 15 9 7 9 85 Index End Expression 8 15 9 4 9 7 9 83 10 7 Glossary 14 Event Color 8 15 Event Count 6 12 6 13 Event End 6 7 6 12 6 13 Event List 8 2 9 83 9 85 10 5 10 7 Event Start 6 7 6 11 6 13 Expression 9 7 File Name 5 33 5 34 Filter 5 35 Font 8 3 Foreground Color 8 3 If Expression 8 2 9 4 9 7
396. system on which the daemon will run the user s login on that system and set tings specifying whether kernel or user application tracing will be performed Items related to trace events output such as the names of output and key files and settings such as whether or not streaming will be performed by this daemon are found on this page as well See General on page 3 28 for more detailed information NightTrace Manual 3 26 User Trace This page contains settings for user trace daemons such as locking policies associ ated with the daemon shared memory permissions and the duration of the times tamp heartbeat as well as specifications for the size and flush threshold of the user event buffer See User Trace on page 3 33 for more detailed information Events This page allows the user to specify which events may be logged while tracing See Events on page 3 37 for more detailed information Runtime This page allows the user to specify the scheduling policy CPU bias and memory binding policies for the daemon See Runtime on page 3 41 for more detailed information Streaming This page allows the user to specify advanced settings with respect to the transfer of trace data from the daemon to the NightTrace display buffer See Streaming on page 3 45 for more detailed information The following buttons appear at the bottom of the Daemon Definition dialog and have the specified meaning OK
397. system trace events as well as manages the configuration and operation of user and kernel daemons to collect trace data ntrace is flexible you choose the look of your graphical display pages ntrace provides many different built in graphical components called display objects You can color select size position and group these objects and direct particular trace events to specific objects this is called configuring display objects There are also ways to label trace events trace event arguments and other numeric values This chapter covers the following topics e X and NightTrace vocabulary System environment ntrace invocation ntrace options ntrace arguments ntrace user interface ntrace notation conventions ntrace Global Window X and NightTrace Vocabulary The Massachusetts Institute of Technology developed a windowing system called the X Window System or X for short If you are unfamiliar with standard X terminology you may find the glossary near the end of this manual useful It contains definitions of words and phrases about e X applications in general The ntrace display utility 5 1 NightTrace Manual Window components Common push buttons and menu item labels Mouse operations System Environment 5 2 To run the ntrace display utility you need an installed X server ntrace uses an X server to support windowing in trace event displays Motif is a user environment based on X T
398. t has three arguments that are identical to the arguments logged with the TR SYSCALL ENTRY trace event TR SYSCALL RESUME argl arg2 arg3 This trace event is logged whenever a syscall is resumed i e the process that caused the syscall to occur which was switched out before the syscall could be completed is switched back in A TR SYSCALL RESUME trace event will always follow a TR SYSCALL SUSPEND trace event unless the process is being switched in for the first time since kernel tracing began Itis possible for several TR SYSCALL SUSPEND TR SYSCALL RESUME trace event pairs to occur if the process is switched in and out several times before the syscall completes The TR SYSCALL RESUME trace event has three arguments that are identical to the arguments logged with the TR SYSCALL ENTRY trace event However if a TR SYSCALL RESUME trace event does not follow a TR SYSCALL SUSPEND trace event i e it is the first syscall trace event logged by the process since kernel tracing began arg2 identifies the syscall as can t determine TR SYSCALL EXIT argl arg2 arg3 This trace event is logged whenever a syscall is completed It has three arguments that are identical to the arguments logged with the TR SYSCALL ENTRY trace event Tracing the Kernel Kernel Trace Points Not Enabled By Default There are several kernel trace points which are not enabled by d
399. t string call NightTrace follows these steps 1 Evaluates int expr 2 Uses this value as an index into table name 3 Retrieves the associated string from table name 4 Returns a string The following lines provide a brief example of a call to get string 9 76 Using Expressions string_table conditions item item item 1 normal 50 YELLOW ALERT 99 RED ALERT default_item N A y In this example the numeric argument associated with a trace event represents the current conditions conditions If the argument has the value 99 NightTrace 1 Uses the value 99 as in index into conditions 2 Retrieves the associated string RED ALERT from conditions 3 Returns R RETURN TYPES ED ALERT On successful completion get_string returns a string from a string table NightTrace returns a string of the item number int_expr in decimal if table_name is not found or if int_expris not found and there is no default item line The first time table_name is not found NightTrace issues an error message Because get_string returns a string you can use it anywhere a string expression is appropriate For more information on string tables see String Tables on page 5 13 Table 8 3 and the usr lib NightTrace tables file get_item The get item routine looks up an item number in a string table SYNTAX
400. t file name that matches the one on the trace begin call attached to the shared memory region This condition is not always detect able Solution Use the ntrace display utility to analyze your logged trace events SEE ALSO Related routines include t race_begin trace_close_thread 2 23 NightTrace Manual Disabling Tracing There are four ways to disable tracing in your application For C applications puta include lt ntrace h gt in your source code You must either recompile your application with the DNNTRACE preprocessor option or insert the following preprocessor control statement before the include lt ntrace h gt define NNTRACE The NightTrace header file ntrace h contains macro counterparts for each NightTrace library routine When you define NNTRACE the compiler treats your NightTrace routine calls as if they were macro calls that always return a success zero status For more information on preprocessor options see cpp 1 Use a command similar to the following one to turn off tracing in your application fl sim c cc DNNTRACE fl sim c lud By disabling tracing this way you have to rebuild your application but you save compilation and execution time Call the trace disable all routine near the top of the source recompile and relink your application with the NightTrace library For more information about this routine see trace_enable trace disable and Their Variants o
401. t percent end offset end time slu end percent From the NightTrace kernel trace event file load only excep tions and system calls associated with process es in the user trace event file s This implies that you invoke ntrace with both a kernel trace event file and user trace event file s From the NightTrace kernel trace event file load only excep tions and system calls associated with this global process iden tifier PID Note that a global PID is different than a raw PID For more information on global process identifiers see PID List on page 8 7 From the NightTrace kernel trace event file load only excep tions and system calls associated with this process name name This implies that you invoke ntrace with both a kernel trace event file and user trace event file s Load trace events after the specified trace event offset See The Grid on page 6 4 for information about trace event off sets Load trace events after the specified relative time in seconds s or microseconds u Load trace events after the specified percent of total trace events The is required Load trace events before the specified trace event offset Load trace events before the specified relative time in seconds s or microseconds u Load trace events before the specified percent of total trace events The is required For example the following invocation loads trace events logged between 5 and 15 sec o
402. t recent instance of a state 9 45 NightTrace Manual End functions include end_id end arg end arg dbl end num args e 0 nd_pid e 0 nd raw pid end lwpid end thread id end task id end tid e end cpu end offset end time end node id end pid table name end tid table name end node name end id DESCRIPTION The end id function returns the trace event ID associated with the end event of the ast completed instance of a state SYNTAX end_id QS PARAMETERS OS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer 9 46 end_arg end_arg_dbl Using Expressions SEE ALSO idQ on page 9 19 start_idQ on page 9 35 and offset_idQ on page 9 60 DESCRIPTION The end arg function returns the value of a particular trace event argument associated with the end event of the last completed instance of a State SYNTAX end_arglN 108 PARAMETERS N Specifies the Nth argument logged with the trace event Defaults to 1 OS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may on
403. t time line on the associated CPU If the RCIM module is used to timestamp events this DataBox will show the process that is executing at the current time line on both the associated CPU and node A menu item and push button that overwrite an existing configuration file with the current display page A menu item that saves the current display page in a new configuration file A menu item that overwrites an existing summary text file with text from the summary display area A menu item that saves the current summary text from the summary display area into anew summary text file Single board computer The narrow rectangular graphic device used to change a display that would not otherwise fit in the window It consists of a trough a slider and arrowhead buttons If the slider does not fill the trough there is a gap on one or both sides The NightTrace form that allows you to define criteria to be used to locate a trace event in a trace event file by its configured characteristics and its location in the file The display object that you clicked on Alternatively a selection may be the region of a text field you dragged the mouse over For menu items push buttons and radio buttons NightTrace indicates selection by highlighting your choice For display separator session shared memory buffer slider spin lock start function state state function Glossary objects NightTrace places handles on the display object
404. t will be considered by this display object If the trace event sent to the display object is not on the list of CPUS then the trace event is ignored A CPU number can be specified only if a NightTrace kernel trace event file is specified Multiple CPU numbers must be separated by commas PID List Configuring Display Objects A global process identifier PID is a 32 bit integer It includes a 16 bit integer raw PID and a 16 bit integer lightweight process identifier LWPID The syntax for specifying a PID is raw PID LWPID The PID List parameter is the list of global process identifiers PIDs or process names that the display object will accept trace events from If the trace event did not occur in a process listed in this parameter the trace event is ignored If a number or name is specified that is not a valid PID a warning message is displayed Multiple numbers and names must be separated by commas NOTE Prior to Version 4 1 ntrace converted process identifiers into process names during PID List input verification for a display object For each process identifier in the PID List ntrace would try to find its associated process name and display that name in the PID List However because multiple processes hav ing the same name may exist on a system changing a process identifier into a process name introduces the possibility that the display object will accept trace events from undesirable processes Therefore nt race no long
405. tGraph or Data Graph A configuration file contains user created display page s macros qualified events qualified states and or table definitions When you click on File I Open Config File on the Global Window a dialog box appears This window prompts you for an existing configuration file name To avoid any ntrace errors you must have read permission for the file When you open a configuration file all regular features of a dis play page are immediately available When you open an existing configuration file that contains display page s ntrace displays them This can save time configuring display objects and let you immediately begin trace event analysis The Open Config File Dialog Box shown in Figure 5 6 is made up of e A File Name text field e A Filter text field A file name scroll region and its scroll bar The default Open push button The Cancel push button If you decide not to open a configuration file click on Cancel This action causes the Open Config File Dialog Box to go away If you still want to open an existing configuration file see The File Selection Dialog Box on page 5 33 For more information on configuration files see Understanding Page Configuration Files on page 5 11 Invoking the ntrace Display Utility Enter configuration file name Read Event Map File Figure 5 6 The Open Config File Dialog Box An event map file associates user created mnemo
406. tTrace display buffer for immediate analysis even while additional trace data is being collected Using the NightTrace Session Manager The NightTrace Session Manager provides users with the ability to define a session that can be saved for future use A session is simply a collection of one or more daemon definitions Individual daemons within a session may or may not be related to each other in any mean ingful way One might use a session simply to hold several daemon definitions that are commonly used but not necessarily all at the same time When creating and saving sessions only the daemon definitions are being created or saved not the trace data that may be captured using the daemons 3 1 NightTrace Manual NightTrace Session Manager Type Daemon Name Target State Attached Streaming Logged Buffer Lost Figure 3 1 NightTrace Session Manager The NightTrace Session Manager consists of the following components NightTrace Session Manager Menu Bar Session Configuration File Name Area Daemon Details Area Daemon Control Area NightTrace Session Manager Menu Bar The NightTrace Session Manager menu bar is a part of the NightTrace Session Man ager window see Using the NightTrace Session Manager on page 3 1 The NightTrace Session Manager menu bar provides access to the following menus NightTrace Daemons Options Tools Help Each menu is described in the sections that follow NightTra
407. te_status false The current time line was not posi tioned inside a current instance of the state Even though the current time line is positioned on an end event of the state event f the corresponding instance is said to have already com pleted state gap 0 000030 The duration of time in seconds between event d and event e The function operated on the most recent instance of the state instance C and the immediately preceding instance instance B state dur 0 000040 The duration of time in seconds between event e and event f The func tion operated on the last completed instance of the state instance C state_matches 3 Assuming no other instances of the state preceded those shown in the fig ure The function operated on all com pleted instances of the state which included instances A B and C start_time 1 631870 The time associated with event e The function operated on the most recent instance of the state instance C end_time 1 631910 The time associated with event f The function operated on the last com pleted instance of the state instance C 9 18 Trace Event Functions id Using Expressions The trace event functions operate on either the qualified event specified to that function or the current trace event They include the following id arg arg dbl num args pid raw_pid lwpid cpu thread id task id tid offset time n
408. ter gt or click on Apply Position in the window you want to modify Press lt Esc gt or click on Reset if this is available Sometimes it is desirable to change multiple fields before applying the changes In these cases apply your changes only once after you have edited each field of interest When you press lt Enter gt or click on Apply at the end of your editing ntrace validates the data in each field you modified ntrace rarely issues error messages about editing errors it detects Usually it takes a default action Some of the default actions include If you enter an invalid value for example alphabetic characters in a numeric field ntrace ignores the changes and restores the previous val ues Usually if you enter a number that exceeds the maximum value ntrace replaces it with the maximum value If a range s starting value exceeds its ending value ntrace swaps them NightTrace Manual Positioning Within Text Fields 6 18 You can either position the text cursor to a particular place within a field by either clicking or typing in key sequences The following key sequences move the text cursor only if you are already positioned in a text field Table 6 5 Positioning Within a Text Field Goal Steps to Attain Goal Move text cursor left one character Move text cursor right one character Move text cursor to next field Move text cursor to pre vious field Press lt LeftArrow gt or sim
409. terrupted the second interrupt etc arg3 The interrupt vector number of the previous interrupt that this inter rupt entry is interrupting if any TR_INTERRUPT_EXIT argl arg2 arg3 This trace event is logged whenever an interrupt is exited Its arguments are identi cal to those of the TR_INTERRUPT_ENTRY trace event Exception Trace Events There are four trace events associated with exceptions TR_EXCEPTION_ENTRY argl This trace event is logged whenever an exception is entered It has one argument arg The exception vector number that indicates the type of exception This is an index into the vector string table that is contained within the vectors file For more information about the vector string table see Kernel String Tables on page 11 16 TR_EXCEPTION_SUSPEND argl This trace event is logged whenever an exception is suspended by a context switch It has one argument that is identical to the argument logged with the R_EXCEPTION_ENTRY trace event E R_EXCEPTION_RESUME argl This trace event is logged whenever an exception is resumed i e the process that caused the exception to occur which was switched out before the exception could be completed is switched back in A TR_EXCEPTION_RESUME trace event will always follow a TR_EXCEPTION_SUSPEND event unless the process is being switched in for the first time since kernel tracing bega
410. tes for the basic trace event and one argument and 16 bytes for the NT CONTINUE overhead trace event and the remaining arguments On PowerMAX OS use the following command to see the system settings for the current default minimum and maximum shared memory segment size etc conf bin idtune g SHMMAX See the idtune 1M man page for more information By default if the shared memory buffer becomes full ntraceud wakes up and copies trace events from the shared memory buffer to the trace event file trace file You can increase the count parameter to prevent trace event loss If you use the memsi ze option you must start up ntraceud with it By changing the shared memory buffer size you can Alter the buffer flush frequency Control the number of trace events copied to the trace event file in buffer wraparound mode SEE ALSO For information on limiting the number of logged trace events see Option to Estab lish Buffer Wraparound Mode bufferwrap on page 4 13 4 16 Generating Trace Event Logs with ntraceud Option to Set Timeout Interval timeout By default ntraceud sleeps 5 seconds after writing trace events to disk The ntraceud timeout option lets you set this timeout interval SYNTAX ntraceud timeout seconds trace_file DESCRIPTION You can identify a running ntraceud daemon by its trace event file name trace_file When ntraceud is idle the daemon sleeps By default the sleep interval is a
411. that if you press the lt Esc gt key before releasing mouse button 1 the operation aborts The display object is still loaded as signified by the crosshair at the pointer location so you can immediately try to recreate the display object Also note that display objects must not overlap except for graphical display objects which must overlap a Column 5 Release mouse button 1 The display object should appear on your grid with solid line boundaries unless there was an error e g you placed a DataBox on top of an existing GridLabel Notice that the display object is also selected corners have handles This is in case you want to move configure or resize it at this time Selecting Display Objects Often you must select a display object before performing grid and edit operations For example before you can resize a display object you must first select the display object To select a single display object simply click on the display object with mouse button 1 The display object now has handles at the corners indicating that the display object is selected When display objects are inside a Column it is sometimes difficult to select the Column To select an unselected Column hold down the lt Control gt key and click mouse button 1 If you perform the same action in a selected Column the Column is deselected You can select multiple display objects three different ways The first way to select multiple display objects is as follo
412. the top or sink to the bottom of the display object Figure 8 3 shows what each type of text gravity looks like Figure 8 3 Top vs Bottom Gravity Configuration Form Push Buttons Apply Reset Restore Close Figure 8 4 Configuration Form Push Buttons Figure 8 4 shows the push buttons that all display object configuration forms have After you have changed the configuration parameters of a display object these buttons allow you to perform the following operations Apply default Validate the changes you made to the configuration parameters and apply the changes to the display object This is equivalent to pressing Enter 8 10 Reset Restore Close Configuring Display Objects Discard all changes made since the last Apply or lt Enter gt This is equivalent to pressing lt Esc gt Discard all changes made since the window was opened Discard any changes made since the last change was applied and close the window Specific Configuration Parameters The following sections discuss the configuration parameters specific to the following display objects GridLabel DataBox StateGraph EventGraph DataGraph Ruler 8 11 NightTrace Manual GridLabel 8 12 unnamed_object Figure 8 5 GridLabel Configuration Form The configuration form for the GridLabel is shown in Figure 8 5 The Text parameter is the only parameter that is unique to GridLabels This parameter is s
413. the trace started Use this display object if you want to know when particular trace events occur DataGraph Dynamic scrollable graphical display object that displays a data as a vertical line or bar and indicates its relative position in time since the trace started The height of the line or bar can be made proportional to the value of a trace event argument or other data Use this display 7 9 NightTrace Manual 7 10 object to display relative values of arguments in the trace event record Ruler Static scrollable graphical display object resembling a Ruler that displays the time Rulers are used with StateGraphs EventGraphs and DataGraphs to show what time a trace event occurred Each display page can hold multiple instances of these display objects usually with each display object uniquely configured All display objects on all display pages reflect the same interval display object type size configuration and position have no bearing Display objects just created in Edit mode contain little useful information The illustrations of display objects in this chapter show the display objects in View mode Figure 7 6 contains a flowchart to help you decide what display objects suit your needs To use the flowchart decide what type of information you want to display Then start at the upper left hand corner of the chart in the box labeled Start Creating Display Objects Do you
414. the user has insufficient privi leges to lock pages see the system administrator or specify that page locking is not requested on the trace begin call and or with the user daemon invocation SEE ALSO Related routines include trace begin trace close thread trace event and Its Variants 2 10 The following routines log an enabled trace event and possibly some arguments to the shared memory buffer Adding Library Calls to Your Application SYNTAX C int trace_event int ID int trace_event_arg int ID long arg int trace_event_flt int ID float arg int trace_event_two_flt int ID float argl float arg2 int trace event dbl int ID double arg int trace event two dbl int JD double argl double arg2 int trace event four arg int ID long argl long arg2 long arg3 long arg4 i Fortran integer function trace event JD integer ID integer function trace event arg ID arg integer ID arg integer function trace event flt D arg integer ID real arg integer function trace event two flt 1D argl arg2 integer ID real argl arg2 integer function trace event dbl ID arg integer ID double precision arg integer function trace event two dbl ID argl arg2 integer ID double precision argl arg2 integer function trace_event_four_arg JD argl arg2 arg3 arg4 integer ID argl arg2 arg3 arg4 Ada type event type is range 0 4095 procedures procedure trace event ID event type procedu
415. the vectors file To move the mouse so the mouse pointer is positioned at the place of interest A graphical symbol that represents the mouse pointer s current location in the window The shape of the pointer shows the current usage Usually a pointer is shaped like an arrow pointing to the upper left See dialog box To hold down a mouse button without releasing it or to depress a keyboard key A list of related choices called menu items pulled down from the menu bar Click on a menu item to select it A graphic image of a labeled button Click on a push button to select it A user defined named event configuration that consists of a set of one or more trace events possibly restricted by an If Expression CPU List TID List PID List and Node List Qualified events provide a mechanism for referencing trace events configurations within certain functions These definitions are stored in configuration files A user defined named state configuration that consists of a set of one or more states possibly restricted by a Start Expression End Expression CPU List TID List PID List and Node List Qualified state provides a mechanism radio button raw PID RCIM Glossary for referencing state configurations within certain functions These definitions are stored in configuration files A graphic labeled diamond shape that represents a mutually exclusive selection from related radio buttons Click on a radio button to select
416. ting the physical CPU numbers of all configured CPUs in ascending order and then numbering them sequentially starting with zero For example a four CPU system having two CPUs on a card in slot 1 and two CPUs on a card in slot 3 will have physical CPU numbers 4 0100 5 0101 12 1100 and 13 1101 Table 11 1 shows the logical CPU mapping of this example system Table 11 1 Example Logical CPU Mapping Physical CPU Number Logical CPU Number 4 0100 0 5 0101 1 12 1100 2 13 1101 3 The CPU box is a GridLabel display object See Chapter 7 and Chapter 8 for more infor mation on creating and configuring GridLabels Running Process Information 11 10 rer Figure 11 7 Running Process Boxes Figure 11 7 shows two examples of running process boxes The running process box shows the process that is executing at the current time on the associated CPU The process is listed by name or by its raw PID and LWPID if no name is available See PID List on page 8 7 for more information about PIDs raw PIDs and LWPIDs You can supply NightTrace trace event files to ntrace along with converted KernelTrace trace event files ntrace uses the process names of all processes that logged trace events when displaying the running process The running process box is a DataBox display object See Chapter 7 and Chapter 8 for more information on creating and configuring DataBoxes Tracing the Kernel Node Information Figure 11 8
417. trace event SYNTAX pid_table_name QE PARAMETERS QE A user defined qualified event If supplied the function returns the name of the process identifier table PID table of the last instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the name of the process identifier table PID table of the current trace event For more information see Qualified Events on page 9 81 RETURN TYPE string SEE ALSO start_pid_table_name on page 9 44 offset_pid_table_name on page 9 67 and end_pid_table_name on page 9 55 DESCRIPTION The tid_table_name function returns the name of the internally assigned NightTrace thread identifier table TID table associated with a trace event SYNTAX tid table name QE PARAMETERS QE A user defined qualified event If supplied the function returns the name of the thread identifier table TID table of the last 9 29 NightTrace Manual node_name process_name 9 30 RETURN TYPE string SEE ALSO instance of the trace event which satisfies the conditions for the specified qualified event If omitted the function returns the name of the thread identifier table TID table of the current trace event For more information see Qualified Events on page 9 81 start tid table name on page 9 44 offset tid table name on page 9 68 and end tid table name on p
418. trace event 9 55 offset 9 67 starting trace event 9 44 Process identifier table name 9 29 Process name 9 30 ordinal trace event 9 69 process name function 9 7 9 30 Pull down menu see Window component pull down menu Push button Add 9 2 Apply 6 9 6 12 6 17 8 10 9 3 10 3 10 8 Cancel 5 30 5 31 Center 6 9 Clear 10 9 10 10 Close 8 11 9 2 9 3 10 4 10 5 Configure 9 2 default 5 23 Delete 9 2 dimmed see Push button disabled disabled 5 27 7 4 grayed out see Push button disabled Mark 6 9 Next 10 3 10 5 on configuration form 8 10 Open 5 30 Prev 10 3 10 5 Read 5 31 Refresh 6 10 Reset 6 9 8 11 9 3 10 3 10 8 Restore 8 11 9 3 10 9 Search 10 4 Summarize 10 6 10 9 10 10 Zoom In 6 10 6 12 Zoom Out 6 10 6 12 C 1 Zoom Region 6 10 Q Qualified events 5 9 5 10 5 11 5 27 5 30 7 18 8 4 9 81 9 81 Configuration form 9 81 dialog box 9 81 QualifiedEvent configuration parameter 9 82 9 83 Qualified states 5 9 5 11 5 27 5 30 7 18 8 4 9 13 Index 11 NightTrace Manual 9 35 9 36 9 37 9 38 9 39 9 40 9 41 9 42 9 43 9 44 9 45 9 46 9 47 9 48 9 49 9 50 9 51 9 52 9 53 9 54 9 55 9 56 9 57 9 58 9 59 9 83 9 83 Configuration form 9 83 9 84 dialog box 9 83 QualifiedState configuration parameter 9 84 9 85 QualifiedEvent configuration parameter 9 82 9 83 QualifiedEvent field 9 82 QualifiedState configuration parameter 9 84 9 85 QualifiedState field 9 84 R Radio button Backward 1
419. tries in the syscal11 table see Syscalls on page 11 21 This table is indexed by a system call number or a system call name Examples of using this table are get string syscall 44 get string syscall arg2 get item syscall fork This string table contains the names the devices that are currently configured in the kernel It is contained in the vectors file This table is indexed by a device number or a device name Exam ples of using this table are get string device arg3 get string device 720900 get item device gd This string table contains the name of each node s process ID table It is dynamically built as the trace event files are processed upon ini tialization This string table contains the names of all nodes that have a trace event file associated with them It is dynamically built as the trace event files are processed upon initialization This string table contains the names associated with all process iden tifiers found in trace event files for node name nodename It is dynamically built as the trace event files are processed upon initial ization It is contained in the vectors file Because process identifiers are not guaranteed to be unique across nodes using the predefined string table pid to get the process name for a process ID may result in an incorrect name being returned from the table Using the node process ID tables ensures that the correct process name is
420. u to log user defined trace events and data from user applications written in C Fortran or Ada These applications may be composed of one or more processes running on one or more CPUs See display object The number that identifies the position of a trace event in the chronologi cally ordered sequence of trace events regardless of the trace event ID Counting starts from zero For example if a trace event with trace event ID 71 is the third trace event in the trace session then its offset is 2 A function that takes an expression that evaluates to an offset as a parameter A push button that acknowledges the warning in a dialog box A menu item and push button that opens an existing file See offset A window component that groups related buttons for example push buttons Glossary 13 NightTrace Manual PID PID table point pointer pop up window press pull down menu push button qualified event qualified state Glossary 14 A 32 bit integer that represents an operating system process The following syntax numerically specifies a PID raw_PID LWPID The operating system process iden tifier raw PID is contained in the upper 16 bits and the lightweight process identi fier LWPID is contained in the lower 16 bits A pre defined dynamically generated string table It is internal to NightTrace and associates process identifiers PIDs with process names If kernel tracing the pid string table in
421. ultaneously press lt Ctrl gt lt b gt This action may cause scrolling Press lt RightArrow gt or simultaneously press lt Ctrl gt lt f gt This action may cause scrolling Press lt Tab gt Press lt Shift gt lt Tab gt 7 Creating Display Objects OVERVIEW S os A 7 1 The Display Paro E N RS PE AS NEP 7 2 Display Page Modes sssslseeeee e 7 3 Edit Mode oss a A SAY kn eS P Oe geo ES 7 4 Mew ModE eue a Gab he heath Hd ets 7 4 Operations on Display Objects o oo ooooocooocccocor eee 7 4 Creating Display Objects 2 0 2 ec eh 7 5 Selecting Display Objects 00 0 0 eee eee eee 7 6 Moving Display Objects 0 0 eee eee eee 7 7 Resizing Display Objects 7 7 Display Objects coe e ede RAUS 7 8 GridLabel sii iii ADS TA 7 12 DiaB A oie Sheba a dd ueris aree 7 12 A a O 7 13 StateGraph moi is ER a a 7 14 EventGraph ii AR A A 7 15 Dy aba Gra pn secs et etu a o tl ted ME 7 16 Rulo ds doi ev Rowe o SNR 7 17 Editing Operations 1 0 runur runerne rrera 7 17 Select Alli eee ete E ate PA eee a E UR Bate sumet s 7 18 Deselect Allo it rd e sey NSE ds bres 7 18 D el t ai ari e 7 18 File Operations o eR e tai 7 18 SAV sic REIN RSA ENA TIENES UG HIN RORIS S IP Aen 7 18 O o acts Seats rt hebt obe tthe ar Eh ae b e icu Pere 7 19 NightTrace Manual 7 Creating Display Objects Overview Display Page Thrbad thread garcia 205970 Ihread thMead1 cap B15 0
422. um maximum average and total values of gaps durations and trace event arguments The scrolling region of the Summarize Form that holds textual summary Statistics A function that takes another expression as a parameter except for summary matches A system call that is a special type of exception A syscall is made when a user program forces a trap into the operating system via a special machine instruction A syscall is used to request a given service from the kernel Many library routines supplied as part of the operating system make syscalls to accomplish their functions Syscalls can be suspended and resumed System call syscall graph syscall table table table function tag task task ID text cursor thread thread ID Glossary A StateGraph on a kernel display page It displays states representing system calls syscalls executing on the associated CPU A pre defined dynamically generated string table in the vectors file This string table contains the names of all the possible system calls syscalls that can occur on the system See format table and string table A function that allows you to extract information from user defined and pre defined string tables and format tables See trace event tag See Ada task A 16 bit integer chosen by the Ada run time executive that uniquely identifies an Ada task within an Ada program The blinking vertical bar in an editable text field that sh
423. umber of completed instances of a state on or before the current time line SYNTAX state matches QS PARAMETERS QS A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 RETURN TYPE integer SEE ALSO Start Functions on page 9 34 and summary matches on page 9 74 DESCRIPTION The state status function indicates whether the current time line resides within a current instance of a state Thus if the current time line is positioned in the region from the start event up to but not including the end event of an instance of the state the return value is TRUE Otherwise it is FALSE SYNTAX state status QS Offset Functions PARAMETERS QS RETURN TYPE boolean Using Expressions A user defined qualified state If supplied it specifies the state to which the function applies If omitted the function may only be used within a state definition and then applies to that state For more information see Qualified States on page 9 83 All offset functions take an expression that evaluates to an ordinal trace event offset as a parameter Offsets begin at zero These functions include the following offset id offset arg offset arg dbl offset num args e offset pid of
424. ur radio button settings Clicking on this push button causes ntrace to search through your trace event file s based on the criteria from the Search Form fields and the radio button settings If you have made a field change clicking on this push button makes ntrace temporarily save your field settings in the search history in memory By saving your field settings in the search history ntrace gives you an easy way to retrieve groups of field settings for use in future searches Because all fields and radio buttons on the Search Form have default settings you can click on the Search push button without modifying anything in this window The default search behavior is Search forward through the entire trace session for any trace event from any process on any CPU e Ifa trace event meets all these criteria nt race Writes an informative message in the message display area of the display page that tells which ordinal trace event offset it found Sets the current time to the time when the trace event occurred Updates the grid and fields in the interval control area of the display page e If no trace event meets all these criteria ntrace writes an error message in the message display area of the display page that tells from which ordinal trace event offset it began the search Search Form Fields All fields of the Search Form have default values Because of these defaults clicking on Search without making a
425. ure 5 1 Window Components Resizing Windows You can resize all windows in ntrace However many windows require a minimum size to display information You can resize some windows only horizontally When you make a window wider ntrace horizontally stretches any push buttons in that window to take up the new width ntrace Notation Conventions This manual uses the following notation convention to reference menu entries Menu bars consist of one or more menu items Clicking on a menu item causes a pull down menu to appear Pull down menus have selectable entries This manual lists menu levels from menu item to menu entry in the order in which they appear An arrow 5 24 Invoking the ntrace Display Utility 1 separates each menu level from the next To show the progression from menu item to menu entry this manual uses the following notation menu item I menu entry For example if you click on the File menu bar item you may then select the Exit menu entry This manual shows this procedure as Click on File I Exit If the menu item consists of more than one word the procedure is shown as Click on File 1 Open Config File ntrace Global Window Unless you invoke nt race with an option that writes to standard output ntrace starts up by displaying the Global Window The ntrace Global Window consists of A message display area and its associated scroll bar A menu bar NightTrace NightTrace performance analyzer
426. ures of your window manager It is not necessary to resize a window to see messages 21 and higher you can scroll through all messages by using the scroll bar Menu Bar The menu bar of the Global Window consists of the following menu items e NightTrace Tools Help NightTrace Menu Item When you click on the File menu item on the Global Window the pull down menu shown in Figure 5 3 appears analuzer Span save Figure 5 3 Global Window File Menu 5 26 New Page Invoking the ntrace Display Utility New Page Default Page Default Kernel Page and Open Config File can all cause nt race to bring up a display page The difference between these menu items is the origin and content of the display page The Default Kernel Page is disabled dimmed unless you provide a NightTrace kernel trace file It is discussed in Kernel Display Pages on page 11 6 The rest of these menu entries are discussed later in this chapter The regular features of a display page include Create and configure display objects so they graphically depict your trace session See Chapter 7 and Chapter 8 Examine trace events states trace event arguments and timings using dif ferent display objects See Chapter 6 Create configure and modify macros qualified events and qualified states See Chapter 9 Search for trace events See Chapter 10 Summarize data into statistics for trace events and states See Chapter
427. ush Flushes trace events from the buffers associated with the daemon s currently selected in the Daemon Details Area to either the NightTrace display buffer see Stream on page 3 31 or to the output file see Output File on page 3 32 Display When data from the selected daemon s is being streamed to the NightTrace display buffer as specified by the setting of the Stream checkbox on the General page of the Daemon Definition dialog see General on page 3 28 pressing this but ton causes a flush of the data currently in the trace buffer to the NightTrace display buffer If no display pages currently exist a default display page will be created when this button is pressed NOTE The user must scroll the NightTrace display in order to see the most up to date data 3 21 NightTrace Manual When data from the selected daemon s is written to output files pressing this but ton causes the data in the output file to be displayed in the NightTrace display Trace Events Presents the Enable Disable Trace Events dialog see Enable Disable Trace Events on page 3 22 allowing the user to dynamically enable or disable selected trace event types while a particular daemon is running A currently executing dae mon must be selected from the Daemon Details Area Enable Disable Trace Events The Enable Disable Trace Events dialog allows the user to dynamically enable or disable selected trace event types while a
428. utility consists of several windows Glossary 21 NightTrace Manual window manager The program that controls window placement size and operations wraparound mode The mode that causes the ntraceud daemon to intentionally discard old events There are two forms of wraparound mode buffer wraparound and file wrap around The other ntraceud mode is expansive mode Glossary 22 Symbols Xdefaults file 5 2 5 8 B 1 B 5 Xresources file 5 2 5 8 dev A 3 dev spl A 3 etc conf mtune d trace A 2 usr bin ntrace 5 3 usr bin ntraceud 4 2 usr include ntrace h 2 1 2 24 4 23 usr lib libntrace a 2 24 usr lib NightTrace eventmap 5 11 5 14 11 6 usr lib NightTrace examples 2 25 5 35 usr lib NightTrace examples entry_exit c 2 25 usr lib NightTrace examples entry exit map 5 35 usr lib NightTrace examples entry exit page 5 35 usr lib NightTrace tables 5 14 5 16 5 20 5 20 9 77 9 78 9 80 10 8 usr lib X11 fonts 5 8 8 3 B 1 A Adalanguage 1 3 compiling and linking 2 25 Ada task identifier 8 8 9 10 9 25 9 40 9 51 9 64 Add push button 9 2 Apply push button 6 9 6 12 6 17 8 10 9 3 10 3 10 8 arg function 9 9 9 20 arg dbl function 9 21 argl function 5 19 8 5 9 9 9 81 10 7 10 8 arg2 function 8 5 9 12 avg function 9 72 Background Color configuration parameter 8 3 8 9 Background Color field 8 3 Backward radio button 10 2 boolean table 5 16 8 6 Box Index CPU 11 10 last ex
429. vd br ales 4b Herren Oe dine PLA b bes 9 52 end CPUs c s eee Bad tan eese ed ace e erc eed dt 9 52 end offset urere ru pO RI UR EPA RS 9 53 end time ill dede oe Meek eb e web 3g D ES 9 54 end node 1d 4 ies Lb bh D ARISU PULS pr R 9 54 end pid table name 0 eee eee eee 9 55 end tid table name llle 9 55 end node name ecceri iese er ccc cee ees 9 56 Multi State Functions 2s 9 56 St te gap cs etre ai quiate n ae dedi 9 57 state dur 5 sea ae eee be dv US EA RE UE 9 57 state matches isis oe i lui uc Pra bugs 9 58 State istatus esie ehe ID TP PRESA EORER Lalas Ne dnd 9 58 Offset FUNCTIONS cueut el e ee ER GA Al oe ane 9 59 Offset 1d cus DO Sa ets ee A ee ee aes 9 60 Offset arg xs wees wie A ne GAS Woh Sa apes 9 60 offset arg dbl e sentuess o Hod Meee La tas 9 61 offset num args rh 9 61 Offset xpld uero E SUPPORT de e ie 9 62 offset raw pid ci e RI ER UR 9 63 offset IwpiQ ir ine te Bose sae o p ie deese 9 63 offset thread 1d severe eek ede ed 9 64 offset task 1d ccc Shae Bk See Ree Ok Bas EU EAE 9 64 offset tid sni Rie BRL RR RE eR bI shew 9 65 Offset cpu s La eoonpOR o EA EA OPE 9 66 offset time ce alligu ew dd 9 66 ottset node 1d ec CENE UBER OCA AS 9 67 offset_pid_table_name 0 0 cece cece ee eee 9 67 offset tid table name 0 ccc eee eens 9 68 Contents offset_node_name 0 ccc ee een eens 9 68 offset proce
430. ve file wraparound and buffer wraparound modes see ntraceud Modes on page 4 4 Generating Trace Event Logs with ntraceud Option to Establish Buffer Wraparound Mode bufferwrap The process of copying trace events from the shared memory buffer to the trace event file on disk is called flushing the buffer With the ntraceud bufferwrap option you can make NightTrace operate in buffer wraparound mode rather than expansive mode In buffer wraparound mode the ntraceud daemon flushes only the most recent trace events rather than all trace events Your application asynchronously triggers every buffer flush SYNTAX ntraceud bufferwrap trace_file DESCRIPTION The ntraceud daemon always logs enabled trace events into a shared memory buffer In expansive mode when the buffer is full or when some other conditions exist ntraceud automatically flushes the buffer to the trace event file trace file The ntraceud bufferwrap option makes NightTrace operate in buffer wrap around mode rather than in expansive mode When the buffer is full in buffer wrap around mode the application treats the shared memory buffer as a circular queue and overwrites the oldest trace events with the newest ones intentionally discarding the oldest trace events to make room for the newest ones This overwriting continues until your application explicitly calls trace 1ush Only then does ntra ceud copy the remaining trace events from the shared memory
431. vent files are varied for interest You can start up ntraceud with the disable d option You can also re invoke ntraceud with this option while ntraceud is running Furthermore using the disable option to disable an already disabled trace event has no effect For example assume that you invoke ntraceud three times sequentially before your application terminates and that ntraceud has not logged to the ntoutput file before Generating Trace Event Logs with ntraceud ntraceud d4 ntoutput trace event 4 is disabled ntraceud d7 ntoutput trace events 4 amp 7 are now disabled ntraceud d4 ntoutput no effect trace events 4 amp 7 disabled There may be any number of disable options on an ntraceud invocation line The following example illustrates this fact ntraceud d10 d15 mytrace trace events 10 amp 15 are disabled You may specify a hyphenated trace event range on the ntraceud invocation line The following example depicts this case ntraceud d23 25 traceoutput events 23 24 and 25 disabled The following two sequences show how important timing can be when you use the disable option The same steps appear in both sequences but their order differs When the first sequence ends nothing has been logged and all trace events are enabled In contrast when the second sequence ends trace event 52 has been logged once and is now disabled Table 4 4 ntraceud Disable Sequence 1 From the Shell Fro
432. witch trace event has been switched out and can no longer run This trace event has one argument argl Interrupt Trace Events 11 2 The numeric 32 bit global process identifier PID of the process being switched in This information is redundant since it is identical to the PID that is already associated with the trace event A PID of O indicates that the CPU is idle The 32 bit global process identifier uniquely identifies the running process on the system This identifier is identical to the return value of the lwp global self system call for PowerMAX OS and the getpid system call under RedHawk Linux See pid on page 9 22 There are two trace events associated with interrupts IR INTI ERRUPT ENTRY argl arg2 arg3 This trace event is logged whenever an interrupt is entered It has three arguments arg arg2 The interrupt vector number that indicates the type of interrupt This is an index into the vector string table that is contained within the vectors file generated by NightTrace when consuming kernel data For more information about the vector string table see Kernel String Tables on page 11 16 The interrupt nesting level used by the pre defined kernel pages to graph the different heights associated with the nesting level This Tracing the Kernel argument will be 1 for the first interrupt 2 for a second interrupt that interrupted the first interrupt 3 for a third interrupt that in
433. word that begins the definition of all format tables table_name The unique user defined name of this table This name describes the relationship of the numeric values in this format table An item line associates a single integer value with a character string This line extends from the keyword item through the semicolon You may have any number of item lines in a single format table The fields in an item line are item The keyword that begins all item lines 5 17 NightTrace Manual 5 18 int_const format_string value An integer constant that is unique within table_name This value may be decimal octal or hexadecimal Decimal values have no special prefix Octal values begin with a zero 0 Hexadecimal values begin with 0x A character string to be associated with int const Keep this string short otherwise ntrace may be unable to display it in the limited window space available Use a n for a newline not a carriage return in the middle of the string The string contains zero or more conversion specifications or display formats Valid conversion specifications for displays include the following iSigned integer uUnsigned decimal integer dSigned decimal integer oUnsigned octal integer xUnsigned hexadecimal integer 1fSigned double precision decimal floating point eSigned decimal floating point exponential notation cSingle character sCharacter string Percent sign nNewline For mor
434. ws 1 Position the cursor outside the display objects you want to select 7 6 Creating Display Objects 2 Click mouse button and drag the mouse until the rectangle that is formed completely surrounds only the display objects you want to select If a display object is not completely surrounded by the rectangle it will not be selected 3 Release mouse button 1 The display objects that were within the rectangle will now have handles at each corner The second way to select multiple display objects is by using the lt Shift gt key Holding down the lt Shift gt key and clicking mouse button 1 while the cursor is in an unselected display object selects that display object without deselecting any other display objects This allows you to select any set of display objects that you want If you perform the same action in a display object that is already selected the display object is deselected The third way to select multiple display objects is described in Select All on page 7 18 Moving Display Objects To move a display object to somewhere else on the grid do the following 1 Select the display object s Refer to Selecting Display Objects on page 7 6 2 Using the mouse button 2 click anywhere on or within the selected display object s and drag to the desired location 3 Release the middle button When display objects are inside a Column it is sometimes difficult to move the Column To move a selected C
435. x OA eo ee eee 2 15 Help Menu eis pd a data Dd IE Lo da oh Bed os 2 16 Session Configuration File Name Area 000 2 17 Daemon Details Area 1 0 eee 2 18 Daemon Control Area 1 0 0 ec ee 2 20 Enable Disable Trace Events dialog ooooooooooo 2 22 Daemon Definition dialog 0 0 eee eee eee 2 25 Import Daemon Definition dialog llle 2 27 Daemon Definition dialog General oo oo o ooomomomooo 2 29 Daemon Definition dialog User Trace ooo ooomomocmomo 2 34 Daemon Definition dialog Events 0000000005 2 38 Load Event Names dialog 0 0 0 0 0 eee eee eee ee 2 40 Daemon Definition dialog Runtime 0 0000 2 41 Daemon Definition dialog Streaming 2 45 Window Components 0 0 00 c eee eh 2 24 Global Window for a Single Trace Event File 2 25 Global Window File Menu oo ooooocoocccoccrc eee eee 2 26 New Display Page 0 0 0 eee eee ee 2 28 A Default Display Page 0 00 eee eee eee 2 29 The Open Config File Dialog Box 00 00 0005 2 31 The Read Event Map File Dialog Box 0004 2 32 Global Window Help Menu 0 0 0 0 eee 2 33 A Display Page in View Mode 0 ccc ee eee ee ene eee 2 2 The Gnd wit Cat EI EOM Re eR bet Gade Seine 2 4 Deciding What to Do Next in View Mode
436. xpression field 8 2 Thread event ordinal 9 68 Thread identifier ending trace event 9 55 offset 9 68 starting trace event 9 44 Thread identifier table name 9 29 Thread name 9 32 ordinal trace event 9 70 Thread names 5 4 5 15 8 3 8 8 10 4 10 6 thread_id function 9 24 thread_name function 9 32 TID 7 8 8 5 8 6 8 8 9 10 9 25 9 41 9 52 9 65 10 4 10 6 tid function 8 6 8 8 9 25 TID List configuration parameter 8 3 8 8 TID List field 9 81 9 83 Glossary 14 tid table 5 15 8 6 TID table name 9 29 tid_nodename table 5 16 E id_table_name function 9 29 Time End field 6 7 6 11 6 12 10 3 10 6 t ime function 9 27 9 83 Time Length field 6 11 6 13 Time Start field 6 7 6 11 6 12 10 3 10 6 Timeout interval 4 4 4 5 4 17 A 3 Times constant 9 6 Timestamp 1 2 5 4 6 7 9 27 9 42 9 54 9 66 Tools menu 10 1 10 5 TR_BUFFER_COUNT tunable parameter A 2 C 2 TR_EXCEPTION_ENTRY trace event 11 3 TR_EXCEPTION_EXIT trace event 11 3 TR_EXCEPTION_RESUME trace event 11 3 TR_EXCEPTION_SUSPEND trace event 11 3 TR_INTERRUPT_ENTRY trace event 11 2 TR_INTERRUPT_EXIT trace event 11 3 TR_PAGEFLT_ADDR trace event 11 5 11 13 TR PROTFLT ADDR trace event 11 5 11 13 TR SWITCHIN trace event 11 2 TR SYSCALL ENTRY trace event 11 4 TR SYSCALL EXIT trace event 11 4 TR SYSCALL RESUME trace event 11 4 TR SYSCALL SUSPEND trace event 11 4 Trace event 1 2 7 8 analysis 5 1 arguments 2 13 5 4 5 10 5 12 5 3
437. y the nt and vec files may be specified together Default Kernel Trace Points The following kernel trace points are enabled by default TR SWITCHIN R INTERRUPT ENTRY and TR INTERRUPT EXI e R_EXCEPTION_ENTRY and TR_EXCEPTION_EXI e 3 i R SYSCALL ENTRY TR IO VNODE TR ALT INT DISPATCH TR PROCESS NAME These default kernel trace points are required to get meaningful kernel performance data in a KernelTrace trace event file However these trace points are not the only trace points 1 1 NightTrace Manual Context Switch Trace Event that you will see with nt race when viewing kernel data Specifically the following trace points are introduced during raw kernel trace data processing by nt race e TR SYSCALL EXIT TR SYSCALL SUSPEND and TR SYSCALL RESUME TR EXCEPTION SUSPEND and TR EXCEPTION RESUME The following sections discuss the trace events that you will see in ntrace as a result of logging the default kernel trace points There is only one context switch trace event TR SWITCHIN argl This trace event is logged whenever a process has been switched in and is ready to be run on a specific CPU Because only one process can run on a given CPU at a time this trace event also signifies that the process that was running on the CPU immediately prior to the context s
438. y default all trace events are enabled for logging to the shared memory buffer The trace disable trace disable range andtrace disable all routines respectively make your application ignore requests to log one or more trace events The trace enable trace enable range and trace enable all routines respectively make your application notice previously disabled requests to log one or more trace events SYNTAX C int trace enable int ID int trace enable range int ID low int ID high int trace enable all int trace disable int ID int trace disable range int ID low int ID high int trace disable all 0 Fortran integer function trace enable ID integer ID integer function trace enable range ID low ID high integer ID low ID high integer function trace enable all integer function trace disable ID integer ID integer function trace disable range ID low ID high integer ID low ID high integer function trace disable all Ada type event type is range 0 4095 procedures procedure trace enable ID event type procedure trace enable ID low event type ID high event type i procedure trace_enable_all procedure trace_disable ID event_type functions PARAMETERS ID ID_low ID_high DESCRIPTION Adding Library Calls to Your Application procedure trace_disable D_low event_type ID_high event_type procedure trace disable all
439. ze and flush threshold of the user event buffer NightTrace Manual Figure 3 21 Daemon Definition dialog User Trace Locking Policies Spin Lock Protection The NightTrace API and associated daemons use high performance low intrusion spin locks to protect critical sections involved in logging and consuming trace events These spin locks require preemption control so that processes on the same CPU don t preempt a daemon or user process in the middle of a critical section and then spin forever waiting for it to be unlocked The spin locks are held for extremely short periods of time Failure to properly select a protection level may result in a process spinning forever on a CPU in the event of unfortunate preemption 3 34 NightTrace Session Manager In more severe cases the system may hang this is only a problem if a user level interrupt preempts another user process or daemon at an unfortunate time and attempts to log trace events to the same trace daemon session Data corruption of the trace data will not occur in any case System priority level SPL This is the safest form of preemption control as it prevents even machine interrupts from preempting the locking process This is required when a user application will be logging events in a user level interrupt handler i e at sys tem interrupt level This is the default locking protection mechanism for PowerMAX OS systems NOTE This mechanism is not available

NightTrace Manual

Contents

Download Pdf Manuals

Related Search

Related Contents