In its most basic form, Reflex for HP NonStop Batch Schedules monitoring can be split into two discrete functions; Reactive Monitoring and Proactive Monitoring.
Central to both of these threads is the HP NonStop Event Management Service (EMS). Reflex for HP NonStop Batch Reaction module monitors by connecting to this logging environment to retrieve and process batch events (MultiBatch, NetBatch or in-house macros) that you have nominated. In this way batch schedule failures reported to EMS can be extracted by Reflex, alerted and escalated.
There are 4 options within Reflex for HP NonStop Batch monitoring for reacting to nominated job EMS events;
- Graphically flash the problem in the Reflex Status Monitor screen against the appropriate batch component or view the EMS exception alert in the Console module (provided as part of the standard Reflex deliverable)
- Relay the event to an appropriate Enterprise Management solution such as hp Operations Center, TIVOLI or BMC Performance Manager using a certified Reflex API or SNMP trap
- Escalate the alert using one or a combination of; mobile SMS text, SNMP, e-mail address or radio pager
- Finally, you can relay the event detail to the ITL product SENTRA, where it can be used as part of a trending exercise or for Management Information reporting
Reflex for HP NonStop Batch proactive monitoring will operate quietly in the background, tracking system and application processes. If a process disappears, the Reflex Process Monitor will raise an event to alert operators to the problem.
Reflex for HP NonStop Batch Schedule monitoring provides a simple to use MultiBatch Schedule detection utility. With this, a graphical ‘drill-down’ object tree is automatically generated and stored in the Reflex Object database for each MultiBatch Job Schedule entered. The appropriate MultiBatch PREPARE name is used as the record key for each schedule detected.
The resulting graphical view is arranged hierarchically for each MultiBatch schedule. The Batch Monitor process (BMON) provides the highest level view of the schedule. For each hp NonStop node accessible from the Reflex for MultiBatch Network view, up to 500 BMON processes can be displayed in a scrollable window.
This window (often referred to as the Status Monitor top-level view) provides a summary view of all MultiBatch schedules and their current status.
Drilling down on any BMON process icon reveals the subordinate Jobs, Segments and Units that make up the batch schedule. More detailed information can be requested by clicking on the job component.
Reflex for monitoring HP NonStop NetBatch Scheduler :
RFIMEX (Reflex EMS Event IMPORT and EXPORT utility)
Allows for the Import and Export of Reflex Action Groups for escalation, including use of pre-delivered EMS events ranges via Reflex transit files and also the import of NetBatch events and creation of associated NetBatch Status Monitor graphical objects.
TRANNBAT transit file contains NetBatch events which will utilise NetBatch jobs names (~nbjobname~) for forwarding to an equivalent named icon graphic in Status Monitor.
TRANNBAT TANDEM.9 Tandem NetBatch events ~nbjobname~
The range of NetBatch EMS events contained within the 'TRANNBAT' transit file are:
- TANDEM.9.102
- TANDEM.9.202
- TANDEM.9.203
- TANDEM.9.204
- TANDEM.9.301
When these are imported into the Reflex database via RFIMEX, the following Action Group entries will automatically be configured:
- NETBATCH-JOB-START
- NETBATCH-JOB-NORMAL-STOP
- NETBATCH-JOB-ABNORM-STOP
- NETBATCH-JOB-OVER-LIMIT
- NETBATCH-JOB-START-ERROR
NetBatch Type / Subtype
Via the Reflex 80:20 GUI, drop-down 'Monitor' menu, select the "Type/Subtype Setup" window to provide the following display:
OD Setup Object Configuration
Using the OD Setup Module, a user can add a user-defined NETBATCH JOB object via the "Status Monitor Setup - Objects List" window.
In the "Objects Detail Dialog", the following details can be set :
The Object Manager Node and Process can be specified, e.g. \ITLTECH $ZBAT (Where $ZBAT is the NetBatch Manager Process)
The list of Types and Subtypes is selected for NETBATCH JOB - Initial Status is set to UP
This object can then be added to the Object database and then dragged and dropped into an appropriate Status Monitor tree structure.
Agent-netbatch $RXNB NBATCHAG
This server allows for passing back of complete NetBatch component details to the info request of the Overdrive - Status monitor screen. It will also accept a status request.
Reflex for HP NonStop Batch Job Schedule monitoring provides a graphical summary view of all batch schedules across the entire HP NonStop network. In the Network view of Reflex, any system node that is encountering a batch problem will immediately flash Red (Critical) or Blue (Vulnerable). Vulnerable is used to indicate a potential problem with the batch schedule that may become a critical if left unattended, e.g. a batch job running beyond its configured time window.
Clicking on the alerting NSK system node will provide access to the Reflex Status Monitor top-level view for batch troubleshooting (Overdrive). The Overdrive display is made up of columns known as Classes, and within Classes, icons that can represent a group of batch objects or a single object.
From the Status Monitor view, all batch processes for the system node will be displayed graphically. If a batch schedule is currently encountering a problem, the batch process icon for that batch schedule will be highlighted. Reflex will move it directly to the top left of the Status Monitor view. From here, it will either flash Critical or Vulnerable depending on the nature of the batch problem. This feature counteracts a weakness in traditional text based Consoles where important messages can scroll off the top of your screen.
In Reflex for HP NonStop Batch, Critical and Vulnerable alerts will continue to flash until the problem has been resolved or an authorised user has acknowledged the alert. In the latter case, the batch process icon will stop flashing and be re-positioned to the right of any current batch issues. Acknowledged problems in Reflex for HP NonStop Batch are displayed with a yellow icon.
The Reflex for HP NonStop Batch GUI provides concurrent access to authorised users to view the batch processing status for each NSK node. All batch schedule issues and acknowledged batch problems are highlighted across all active Reflex sessions.
Reflex for HP NonStop Batch incorporates a graphical Network Hierarchy view of all of the HP NonStop systems in the network. Reflex for HP NonStop Batch is installed on each NSK node and is responsible for monitoring the status of the batch schedule locally. The NonStop SQL status database is also held locally and is updated in real-time.
The overall status of the batch schedule (Non-Critical, Critical or Vulnerable) is maintained and at a configurable rate, relayed to each other NSK system in the network. This is a small message consisting of the node name and the appropriate status data as follows;
- no batch events received or
- at least one Vulnerable batch event or
- at least one critical batch event
- a fourth condition also exists, each node expects to receive a message from each other node in the batch network and it does not arrive.
Using this approach, attaching a GUI session to any of the hp NonStop nodes in the network allows a complete networked picture of the batch schedule health to be presented to the Operations Bridge. This can also be provided for new NonStop server platforms such as the hp NonStop Itanium.
The node icons will be shown as grey (no batch messages processed), red (at least one Job down), blue (at least one Job vulnerable) or orange (no networked node message received).
Clicking on any of the hp NonStop systems allows the corresponding batch processes to be displayed for troubleshooting purposes. Multiple Reflex sessions are not required since status data is relayed across EXPAND by operator request. Status messages requested are small by design and contain only information relevant to the appropriate batch schedule component.
Since each licensed installation of Reflex for HP NonStop Batch is aware of all of the other nodes in the network, authorised operators can log directly across to any selected NSK node from a single Reflex GUI session. This is not required for the purposes of pure batch monitoring and troubleshooting but only for configuration changes that maybe required remotely on another Reflex system.
The advantage of being able to log across the network is that only one Reflex for HP NonStop Batch session is required to enable all batch monitoring configuration updates to occur.
Once Reflex for HP NonStop Batch Schedule monitoring has been implemented, a history of all EMS exception alerts for each batch schedule component is held in the Reflex database.
The object history buffer can be listed to the GUI screen so that first level trending can be completed. The number of EMS messages retained for historical look-up is configurable. The EMS history is held locally on the appropriate hp NonStop node where the batch schedule resides.
The current history of batch EMS events can be requested at any level in the schedule. Requesting EMS history at a group level such as the batch process will retrieve all EMS events for the entire batch schedule up to the current time. These events can then be ordered by any of the column headings, e.g. State, Event Number.
Requesting event history at the Unit level of the batch schedule will only display status events for that requested batch component. All batch status events can be archived using supplied HP NonStop Batch utilities.
The automation engine of the Reflex 80:20 product is known as the TaskMaster.
The Reflex 80:20 Event Monitor can intercept batch EMS events that describe error conditions that can be corrected programmatically and without manual intervention and the details of this event can be forwarded to the Reflex 80:20 TaskMaster which will oversee the execution of the appropriate automated corrective action.
At some Reflex 80:20 installations, the TaskMaster is used to start up and close down applications based on a specific event being received from an external gateway, so this functionality can deal with more than just corrective action.
The TaskMaster interface to other subsystems is not based on using programmatic interfaces but on launching command line processes using low-level procedure calls.
The command line instructions are defined in the TaskMaster “program” screen. Users have the option of specifying standard run options such as object code, IN, OUT, home terminal, CPU, memory, priority, start-up parameters, file assignments and PARAMs.
Users can also select the User Id that they want the program to run as. Information from an event, such as the subject value, can be extracted and fed back into the command line.
For example, the command Batch START ~subject~, could be used to restart a batch process automatically once an EMS event had been processed stating that the process was down. The subject value within the event, for example $HKP1, would be extracted from the batch event and used in place of ~subject~. Once programs have been defined, they can be linked together to form a Task.
Once built, the Task can then be assigned one or more owners by the Reflex 80:20 administrator. If you are not an owner, then you cannot execute the Task. Executing a Task manually updates the audit log with a record of who started which Task and when it was started.
Tasks can be executed a number of ways. They can be linked to a batch event in the Reflex 80:20 event database. If you select a Reflex 80:20 monitored batch event and then choose the Task option as your reaction, a further screen is displayed and you can select a Task from a pull down list. When this event next occurs, your appointed Task will execute.
It is possible to execute Tasks from an icon button. Tasks can be mapped to batch object types, so your Reflex 80:20 administrator can create a database that states that my start task for NetBatch processes is “STARTNTB”. Stop Tasks and “other” Tasks can also be configured. When you click on the OverDrive icon you can select the start, stop or “other” option and the allocated Task for this type of object will execute.
Tasks can be executed manually from the Task configuration screen. This type of use can provide Operations staff with access to tested, secure and audited obey file scripts that they can execute as Users other than themselves.
The Reflex for HP NonStop Batch product incorporates Batch Process monitoring to monitor all named processes. The NonStop Reflex for HP NonStop Batch checks for registered processes at a configurable rate. If the process is not running, an EMS event is generated. Users can also set the frequency at which the down event is generated.
Each process is configured with a user assigned ‘UP’ and ‘DOWN’ EMS event. The event can be given a unique Owner and Value. When the Reflex Process Monitor raises the event for up or down, the event tokens are populated with the user assigned event values. The event subject token is set to the process name although users can specify their own Subject (and Subject Manager).
These EMS events can be registered with the Reaction module of Reflex for HP NonStop Batch and mapped directly to graphical objects in the Status Monitor window.
Enterprise Management Relay to hp Operations Center
The ITL Smart Plug-In for Reflex is hp certified and provides a powerful inclusion within hp Operations Center for managing hp NonStop servers. The SPI for Reflex is an out-of-the-box solution that links seamlessly with the hp Management Console using hp approved techniques for integration.
The hp Operations Center Smart Plug-In technology demands a greater level of sophistication for central enterprise control and goes beyond basic event alerting. The SPI for Reflex offers a comprehensive view of your entire hp NonStop network from the hp Management Console.
All events indicating a potential issue in completing an overnight or daily schedule can be quickly isolated and remedied using SPI for Reflex commands that interact with the batch interface process.
Enterprise Management Relay to the TIVOLI Enterprise Console
The solution allows for 2 approaches for connecting the NSK batch alerts to the TIVOLI Enterprise Management Console (TEC).
Reflex also contains a unique module to aid in the configuration of the TIVOLI TEC. With this, a complete set of event files (BAROC and mapping files) for TEC nominated Batch EMS events, are automatically generated on request to be imported into the TEC by TIVOLI administrators. This approach significantly speeds up the set-up of TIVOLI to recognise NSK delivered events in what is typically a time consuming configuration exercise.
More information relating to the certified TIVOLI integration can be supplied if required. This includes the Product Requirements Definition (PRD). This is a detailed breakdown of the Tivoli Ready Reflex solution with accompanying data flow diagrams. This document is a mandatory requirement prior to certification by Team TIVOLI.
BMC Performance Manager
Reflex for HP NonStop Batch offers a relay to forward batch alerts to BMC patrol using a Windows BMC Module.
Using the Reaction list of Reflex, only nominated Batch events will be sent to BMC PM based on dynamic filtering. The BMC PM Reaction module within Reflex allows individual batch events to be tagged with Catalog, Class, Type and Severity as well as the standard EMS event tokens.
The Reflex Knowledge Module constructs a Batch alert message with the appropriate batch event arguments. This message will then be sent to a nominated BMC PEM Agent running either locally or across the network.
Escalation of Batch Alerts Using SNMP, E-Mail, Mobile SMS or Pagers
One of the React choices that was highlighted earlier in this document for nominated batch events was the “inform somebody” option. For support staff who are off site or for unattended overnight batch processing, Reflex for HP NonStop Batch provides a number of React options;
- Email
- SMS Mobile
- Pager
- SNMP
MultiBatch can be provided with an optional Mangement Information Reports module (MCP). This is installed on a nominated MS Windows Server platform with SQL Server 2005. All job related data is stored in the SQL database detailing all batch schedule metrics, e.g. job run times, key jobs.
Users can also script their own reports using the MS Reporting Services tool with SQL Server 2005. This provides a very extensible solution for MI reports for all HP NonStop platforms including Integrity and Blade technologies.
Examples:
HP NonStop Batch Scheduling - MultiBatch Job Run Times
This MultiBatch report provides a bar chart showing the job run times of the specified job id. Additional details are provided in the associated table.
The report allows a user to specify a Start & End Date/Time range; HP NonStop Node name; MultiBatch BMON Process Id; Job Id.
A user can also report on all jobs and/or all BMON processes by specifying 'All' (without quote marks) in the parameter fields.
The line graph and associated table contains the run time (in seconds) of how long the job has taken to 'run'.
Note: There are various reasons for the job 'completion', e.g. the job may have completed successfully, abended, aborted by operator or reached maxtime and aborted. The report is based on the various types of 'completion' events and not just on the MultiBatch 'job complete' event.
The 'Event Text' column contains the MultiBatch EMS event text which will show the reason for the job 'completion'.
HP NonStop Batch Scheduling - MultiBatch Report Key Jobs
This MultiBatch report provides a line graph to show the start & end points of a specified Unit and Job, respectively. These specified values are for key jobs which a user expects to have Started and Completed by a certain time.
The report allows a user to specify a Start & End Date/Time range; HP NonStop Node name; MultiBatch BMON Process Id and the id of the 'start' Job.Segment.Unit and the id of the 'end' Job.
Two additional parameters are required for completion: Batch Hour & Batch Min. A user must specify in these two fields, an expected (or usual) Start Time for the Job.Segment.Unit. For example, if the specified Unit of 'JOBA.AA.01' usually starts ar 15:30, then specify 15 in the Batch Hour field and 30 in the Batch Min field.
The line graph will display two lines, one for the 'start' job.segment.unit and one for the 'end' job. If the 'start' time of the specified Unit differs from the value entered, e.g. 15:30 has been specified but on a particular day the specified Unit actually started at 15:15, then the line graph and associated table will show the differences in start time.
The associated table will show JOB Id1 and its actual start time, with a column showing any + or - differences from the specified 'Usual Start Time'. JOB Id2 is displayed in the table, followd by the 'duration' in minutes, between the start of JOB Id1 and the end of JOB Id2.
Minimum Windows Hardware Requirements for
MultiBatch MS Management Information Reports
For optimum performance, it is recommended that the minimum specification of your hardware and software is as follows :
- Windows Server 2000 / 2003 / 2008 with the latest service packs
- Microsoft SQL Server 2005 / 2008 with the latest service packs
- Either; Tomcat, IBM WebSphere, JBoss (formerly BEA WebLogic) web application servers (ask ITL about MS Internet Information Services- IIS) - also requires Java SE version 6+ (1.6)
- Browsers; Internet Explorer (IE) ver. 7+, Google Chrome, Safari, Firefox
- Intel Mid to High-End, Multi Core processor
- 16+ GB RAM recommended
- SCSI interface (SCSI2 Ultra-Wide recommended)
- 10 GB Single Drive for operating system and SQL Server software
- *40 GB Single Drive for the SQL server database (RAID 0+1 recommended)
- 20 GB Single Drive for the SQL server log file (RAID 0+1 recommended)
- Graphics resolution 1024 x 768 recommended
- 17" or larger colour monitor is also recommended
The above specification is for guidance only. The specification of your Windows server will be dependent on your individual needs. Please contact the Insider Technologies Helpdesk for assistance in establishing the specification of your server.
SQL Server Versions Supported by Sentra (also install Microsoft SQL Reporting Services)
The Sentra database is compatible with the following variants of SQL Server :
- 2005 / 2008 Standard Edition
- 2005 / 2008 Enterprise Edition
- 2005 / 2008 Developer Edition*
- 2005 / 2008 Express – the default installation on the CD uses SQL Express with Advanced services, so that SQL Reporting Services is available
* Some SQL editions include a concurrent workload governor. Performance degrades when more than five queries are executed concurrently. Sentra will work with these versions of SQL Server but performance may be unacceptably slow and its installation is not recommended for high volume usage.
SQL Express editions support databases with a limited maximum size (4Gb for SQL Express 2005). Users who anticipate large database storage requirements should consider installing the Enterprise edition of Microsoft SQL Server, or contact Insider Technologies for advice.
Hardware and software pre-requisites for the Reflex product are detailed below:
• HP NonStop
• Guardian D38 Operating System (or above)
• TMF
• SQL runtime system (SQL/MP runtime), or alternatively, NonStop SQL product
• Pathway (TS/MP)
• TCP/IP
• Non-RDF/DRNET Audited Volume *
* It is recommended that Reflex is installed on a Non-RDF/DRNET Audited Volume. If this is a cause for concern, please contact Insider Technologies Ltd to discuss further.
Reflex is installed using a menu driven TACL macro for easy deployment. The macro will perform checks and output the results of each stage of the Reflex product install so that if problems are encountered they can be resolved easily.
Windows Platform for GUI Install:
• Modern Pentium PC Specification
• 32Mb on-board RAM
• 20Mb free hard disk space
• Minimum 17” monitor recommended
• Graphics Resolution: 1027 x 768 x 16 minimum
• Windows XP, Vista, 7
* For escalation of HP NonStop issues to enterprise management solutions, email or mobile SMS, a dedicated Windows box is required.