Dashboard - HP NonStop real-time performance monitoring
The Dashboard facility retrieves performance and capacity statistics from a variety of NonStop subsystems (including CPU, Disk, X25, TCP/IP, Process) and then processes them in a number of ways ;
- The statistics are shown in a graphical form in a dedicated Dashboard display.
- The statistics are analysed and compared with pre-configured thresholds. If the thresholds are exceeded, then an EMS event is generated. If a value has returned below its threshold then a different EMS event is generated. This pair of “up” and “vulnerable” events can be used to animate an OverDrive Status Monitor icon and force the icon to turn blue (vulnerable) and grey (up).
- The statistics can be sent to the Insider data warehouse product Sentra, where they can used as part of a trending exercise or for Management Information reporting.
As with the Status Monitor facility, the Dashboard is installed on each Reflex 80:20 node and it retrieves performance statistics for the local node only. Low level, low cost GUARDIAN procedure calls are used to retrieve the latest set of statistics.
The NonStop Measure product is only accessed on user request to retrieve attributes such as file reads, writes and locks that cannot be obtained by procedure calls. The latest ZMS Measure definitions are used where available so that new counters such as IPU can be obtained. Dashboard allows users to drill down for more detailed process metrics. Users can click on a busy process to then start the process status agent to monitor what that process is doing via a call to the stack and then optionally kick off a measurement. Reflex Dashboard does not use Measure as part of it's day-to-day performance monitoring.
You can choose to switch this facility off if there is concern about processing overheads and the poll period for each different type of collection can be set by the Reflex 80:20 administrator to help manage the impact on your system. Newly collected sets of statistics are despatched to a threshold engine where the data is compared against values set by the Reflex 80:20 administrator.
When thresholds are exceeded, exception alerts are created in the local EMS event log, the events can then be transferred to the top-level Network Monitor as described (see Reflex White Paper). When the network node icon flashes, the Reflex 80:20 User has the option of selecting the OverDrive icon based status display or the Dashboard graph based performance display for that node.
Here is a summary of some of the core performance metrics that can be monitored by the Reflex Dashboard module. Those with an asterisk (*) can also be thresholded using a configurable parameter value. This will result in an EMS event alert being raised which can be fed back into the Reflex Reaction engine for graphical monitoring or remote alert escalation, e.g. IBM Tivoli, Email, Mobile SMS, SNMP.
CPU
- Busy *
- Queue Length *
- Page Fault *
- Memory Pressure
- Memory Queue Length *
- PCB Low Pin in Use *
- PCB Low Pin Free *
- PCB High Pin in Use *
- PCB High Pin Free
- TLE Current
- TLE Configured
- PTLE Current
- PTLE Configured
- Disk I/O
Disk
- Size
- Full *
- Number of Fragments *
- Largest Fragment
- CACHE Reads *
- Cache Hits *
- Cache Calls
- Audit Forces *
- I/O's *
Process
- Object Code
- Process Type
- Name
- Busy *
- Priority
- Intial Priority
- User ID
- Looping
- Inspect *
- Debug*
X.25
- Line State
- Line Quality *
- SU's Configured
- SU's in Use *
- SU's Not Started
- Link State
- Modem Errors *
Reflex does not use the HP NonStop Tandem MEASURE subsystem as part of day-to-day, core monitoring of the NonStop platform(s). However, Measure can be run up on an authorised user's request from the Reflex Dashboard display, or via a monitored Process Object component in the Status Monitor view.
Measure requests can be issued for a selected process and / or the files and processes opened by that selected process. Each request can be run for a user defined period, e.g. 30 seconds (this value can be amended via the Admin Parameter: MEASUREMENT-DURATION) and will provide a measurement counters snapshot, where if required, the counter results can be saved as a CSV file for later analysis.
The Measure counters / attributes returned for each type of request are as follows:
'OPEN' Measure Counter list:
1. Date
2. Time
3. Duration
4. System
5. OSVersion
6. FormatVersion
7. DataVersion
8. CPU
9. PIN
10. Fnum
11. Filename
12. FileSysName
13. FileType
14. DeviceType
15. FileBusyTime
16. Reads
17. Writes
18. UpdatesOrReplies
19. DeletesOrWriteReads
20. InfoCalls
21. RecordsUsed
22. RecordsAccessed
23. DiscReads
24. Messages
25. LockWaits
26. TimeoutsOrCancels
27. Escalations
28. OpenerProcessname
29. OpenerProgramname
30. DeviceName
31. OpenerDevicename
32. BytesF
33. DBIOReads
34. DBIOWrites
35. DBIORecordBytes
36. DBIOWriteBytes
37. FileCRVSN
38. OpenerOSSPID
39. OpenerCRVSN
40. FileOpenType
41. IPAddress
42. ReadBytes
43. WriteBytes
44. OSSCacheReads
45. OSSCacheWrites
46. OSSCacheReadBytes
47. OSSCacheWriteBytes
48. OSSBlockReads
49. OSSBlockReadBytes
50. OSSFlowControls
51. MiscCalls
'PROCESS' Measure Counter list:
1. Date
2. Time
3. Duration
4. SystemName
5. OSVersion
6. FormatVersion
7. DataVersion
8. CPU
9. PIN
10. ProcessName
11. ProgramFileName
12. Priority
13. UserID
14. CreatorID
15. AncestorCPU
16. AncestorPIN
17. AncestorSystemName
18. AncestorProcessName
19. DeviceName
20. HometermSystemName
21. Hometerm
22. CPUBusyTime
23. ReadyTime
24. MemQTime
25. Dispatches
26. PageFaults
27. USEMS
28. PagesQTime
29. PagesStart
30. PagesEnd
31. ExtSegsQTime
32. AbsSegsQTime
33. AbsSegsStart
34. AbsSegsEnd
35. RecvQTime
36. MessagesSent
37. MessagesRecvd
38. MsgsSentQTime
39. SentBytesF
40. RecvdBytesF
41. ReturnedBytesF
42. ReplyBytesF
43. SentCBytes
44. ReceivedCBytes
45. ReturnedCBytes
46. ReplyCBytes
47. MQCAllocations
48. MQCAllocFailures
49. MQInuseQTime
50. Checkpoints
51. AllocSegCalls
52. FileOpenCalls
53. InfoCalls
54. ProgramAccelerated
55. NativeProcess
56. SystemProcess
57. AccelBusyTime
58. TNSBusyTime
59. NativeBusyTime
60. CompTraps
61. BeginTrans
62. AbortTrans
63. OSSPid
64. CRVSN
65. OSSTTYReads
66. OSSTTYWrites
67. OSSTTYReadBytes
68. OSSTTYWritesBytes
69. OSSTTYWaitTime
70. OSSDevNullOps
71. OSSNSDDCalls
72. OSSNSRequests
73. OSSNSMessagesBytes
74. OSSNSWaitTime
75. OSSNSRedirects
76. Launches
77. LaunchWaitTime
78. OpenCloseWaitTime
79. IPUSwitches
80. GMOMNode
81. GMOMCPU
82. GMOMPin
83. GMOMJobId
84. GMOMSysName
85. GMOMProcessName
86. IPUs
Hardware and software pre-requisites for the Reflex product are detailed below:
• HP NonStop (ServerNet or Integrity platforms)
• Guardian D38 Operating System (or above)
• TMF
• SQL runtime system (SQL/MP runtime), or alternatively, NonStop SQL product
• Pathway (TS/MP)
• TCP/IP
• Non-RDF/DRNET Audited Volume *
* It is recommended that Reflex is installed on a Non-RDF/DRNET Audited Volume. If this is a cause for concern, please contact Insider Technologies Ltd to discuss further.
Reflex is installed using a menu driven TACL macro for easy deployment. The macro will perform checks and output the results of each stage of the Reflex product install so that if problems are encountered they can be resolved easily.
Windows Platform for GUI Install:
• Modern Pentium PC Specification
• 32Mb on-board RAM
• 20Mb free hard disk space
• Minimum 17” monitor recommended
• Graphics Resolution: 1027 x 768 x 16 minimum
• Windows XP, Vista, 7
* For escalation of HP NonStop issues to enterprise management solutions (e.g. BMC Performance Manager using the PatrolCLI utility), email or mobile SMS, a dedicated Windows box is required.
Reflex 80:20 Version 4.7M – Available Q1, 2010
Individual Disk Monitoring (I/O Times):
Individual Disk I/O monitoring is now available, where this facility can obtain the count of timings that I/O took on each Disk (Primary, Backup, Mirror and Mirror Backup Paths), polling each Disk at regular intervals.
If a specified monitoring value is exceeded then an EMS event is issued to alert HP NonStop personnel of the problem. When the Disk Paths return to specified I/O levels, then a subsequent EMS event is raised to clear the alert. All events can be forwarded to matching Disk Objects in Reflex Status Monitor, or escalated as with other types of Reflex monitoring and alerting.
Disk I/O monitoring continues to take place for all Disk Paths, even if half of a mirrored volume is down and there is no need to run Measure Counters to obtain this information; you just need to specify the I/O values and polling period and let Reflex alert you to any issues with Disk I/O.
Sentra is able to collect HP NonStop CPU, Disk, Process and X25 metrics for rules analysis, escalation and reporting from all installed instances of the Reflex product across the NonStop network.
Performance Counters Collected - Disks
HP NSK Disk Counters
Disc Name
Audit Forces 512
Audit Forces 1024
Audit Forces 2048
Audit Forces 4096
Cache Calls 512
Cache Calls 1024
Cache Calls 2048
Cache Calls 4096
Cache Faults 512
Cache Faults 1024
Cache Faults 2048
Cache Faults 4096
Cache Read Hits 512%
Cache Read Hits 1024%
Cache Read Hits 2048%
Cache Read Hits 4096%
Cache Reads 512%
Cache Reads 1024%
Cache Reads 2048%
Cache Reads 4096%
Number of Fragments
Free Space (MB)
Largest Fragments
Space Used (%)
Total Size (MB)
Performance Counters Collected
a) CPUs
b) Top Ten busy processes per CPU
c) Top Five suspect processes per CPU, e.g. debug or looping
HP NSK Processor Counters
CPU number
CPU type
O/S version
%Busy
Breakpoints
Disk cache hits per second
Disk I/Os per second
Free pages
Locked pages
Memory page size
Memory pressure
Memory queue length
Memory size
Number of suspect processes
Page faults
PCBs free in high PINs
PCBs free in low PINs
PCBs in high PINs
PCBs in low PINs
PTLE configured
PTLE current
Queue length
Send busy
Swappable pages
TLE configured
TLE current
Busy time
Idle time
Interrupt time
HP NSK Process Counters
CPU ID
Event Time
Is Bad
PIN
Busy Percent
Priority
Initial Priority
Process Type
Process ID
Process Name
Object Name
Process State
Wait State
Process Security
Julian Creation Time
Read User ID
Effective User ID
Performance Counters Collected - X25
HP NSK X25 Counters
Line Name
Disconnected Mode (DM) frames received
Clear confirmation packets sent
Clear request packets received
Disconnect (DISC) frames received
Disconnect (DISC) frames sent
Disconnected Mode (DM) frames sent
Frame Reject Response (FRMR) frames received
Frame Reject Response (FRMR) frames sent
Interpret packets sent
Interpret request packets received
Line Quality (%)
Modem errors
Receive Not Ready (RNR) frames received
Receive Not Ready (RNR) frames sent
Receive Not Ready (RNR) packets received
Receive Not Ready (RNR) packets sent
Receive Ready (RR) frames received
Receive Ready (RR) frames sent
Receive Ready (RR) packets received
Receive Ready (RR) packets sent
Reject (REJ) frames received
Reject (REJ) frames sent
Reset packets sent
Reset request packets received
Restart packets sent
Restart request packets received
Selective Reject (SREJ) frames received
Selective Reject (SREJ) frames sent
Sub-units in use
Sub-units started
For more information on what Sentra can monitor on the HP NonStop platform, click here.