Publication Date:  17 February 1996
System Version:  GCCS 2.1/Update 4
Web Page Created:  20 March 1996

Setup.  To accomplish this lesson, you need GCCS connectivity and access to the JOPES database and applications (Session Manager and System Services).  USERID requires NFM functional permissions.  The USERID (user account) must be a member of the UNIX group, JADMIN with execute permissions for the TP/TDS Management subsystems and with execute permissions for Local OPLAN Database Cleanup (DBA function).

Overview.  This lesson provides an understanding of the actions necessary to accomplish a save of the JOPES database in the event of a site database failure.  It continues with a review of the Database Administrator's (or Functional Manager's) capability to manage the activity of the Transaction Processor and the Transaction Distribution Services software.

The lesson also includes the Network Functional Manager's (or DBA's) ability to add/delete/modify database servers from the network.

By now, many of you are probably thinking that much of what you are about to cover is somebody else's job.  It would be nice to know, but that is about it.  It is always good to know how someone else's actions will affect what you do.  That knowledge allows you to better plan your actions. 

OBJECTIVE.  Without references, explain the JOPES database save/archive process.

Purpose for Site Backups.  Site backups should be done as frequently as necessary to ensure the integrity and recovery of the database in case of a catastrophic failure.  Good management of the database requires the Functional Manager to stay in touch with events so that database saves are timely and relevant.  A couple of examples are in order.
Operational (Historical) Reasons.  During Desert Shield/Desert Storm, an individual at an Air Force installation was tracking some validated Air Force records in a force module.  He was finished tracking those records for the moment and decided the force module identifier was no longer necessary.  He used the JDS SF code that deleted not only the identifier, but the associated requirements as well.  The delete action had progressed too far to attempt to stop it.  Luckily, an FM shop had been doing JDS H3 saves of the data, but due to the frequency of the saves, the data used for the recovery was somewhat stale.  After the recovery, a large amount of data had to be reaccomplished.

Potential Site Failures.  In addition to human errors, the potential for a hardware, software, and facility failure is probably higher than you would think.  Failure can occur if the site is invaded by some foreign entity, like water in the computer room from cooling pipes, broken air conditioners that allow the build up of excessive heat, dust that may cause a disk failure, power interruptions, or lightning/static electricity that might cause disk damage.  Any of these events could cause a database site to fail.

The logical answer to recovery after a database failure is to reload from backup data.

Actions Required.  There is no real mystery to a site database backup.
Determine Backup Responsibility.  Probably the hardest decision is agreement on who is responsible for maintaining the backup data (i.e., each user, the FM, the crisis action team, each Service component, or some other entity).

Determine Backup Media.  Once that is established, the next thing is to decide what media will be used to store the information.  The selection of the media will depend on the size of the database, the volume of records to be saved, the frequency of saves, the purposes, and the type of transactions to be saved (TPFDD/OPLAN).

Determine Backup Schedule.  Once those questions are answered, a schedule can be established to accomplish the saves as required.

Determine Networked Database Source.  Each database site should identify another site with similar requirements for networked plans.

No one site maintains a complete backup copy of the entire networked database because they do not need all the plans and the rate of change is just too great.  At least one of the OPLANs in every database is in the process of being changed.  While individual networked OPLANs may be static during certain periods of time, the entire networked database is not.

If a local site's JOPES database needs to be replaced, a "current" copy of the networked information can be obtained from a selected site that is not currently experiencing any problems.

Determine Locally Distributed Database Information Source.  Locally distributed JOPES database information is a slightly different story.  The major difference is that the locally distributed JOPES information is exactly that.  It is locally resident only and is not anywhere else.

If a site failure occurs, a backup copy cannot be obtained from some other site.  For that reason, someone should maintain a complete backup copy of the entire locally distributed database generated at a frequency that is satisfactory to those responsible for the data.  If a copy is required for recovery of the database, then one is readily available.

There are two options available for you to create backup copies of the locally distributed JOPES database.  You are already knowledgeable concerning both of them because you used them in Lesson 4.

The first option is the Create TPFDD File option.  It allows you to save a TPFDD (no S&M data) to a disk file or to a tape.  You can only save the data for one plan at a time.

The second option is the Offload Plan option.  It allows you to save OPLAN data to a disk file or to a tape.  You may save every plan that is locally distributed at one time by entering each PID to the Offload Plans list.

Note:  You will receive screen advisories if the save must be split between several tapes.

Now that you know you must obtain a "current" copy of the networked JOPES database from some other site and how to create a backup copy of the locally distributed portion of the JOPES database, you are ready to review the process for recovering a site after it has failed.

OBJECTIVE.  Without references, explain the process used to recover the JOPES database at a failed site.

Site Recovery Activities.  Site JOPES database recovery activities can address two separate situations:  a complete site recovery where the entire JOPES database must be replaced, and a partial site recovery where portions of the JOPES database need to be repaired.
Complete Site Recovery.  A complete site recovery is necessary when the JOPES database is completely discombobulated and it is easier to replace it instead of trying to repair the damage.  You will recall that the networked OPLANs are recovered differently than the locally distributed OPLANs.
Locate a Suitable Database for Recovery.  A copy of the networked data can be obtained from another database site.  Close coordination among Site/Network FMs and site Database Administrators is necessary to locate a site that most nearly resembles the one that must be replaced.

Remove Local OPLANs from Database.  The FM and DBA at the providing site coordinate to ensure that any local OPLANs are removed from the data being sent to the receiving site.

Transfer the Tailored Database.  The providing site then copies its networked database to a tape or file and the networked data is transferred to the receiving site.

It can be accomplished by a courier carrying a tape or by an electronic file transfer via SIPRNET.

Upon arrival at the receiving site, the networked OPLAN data is loaded to the Oracle database.

During the recovery operation, each site must also consider the impact of stopping the flow of update transactions into their database.

From the time the networked data is saved at the providing site, transactions destined for the receiving site must be stored to update the database when loading is complete.

Controlling the flow of transactions through the TP, the Receive Queue, the TDS, and the Send Queue will be discussed later in this lesson.

Cleanup Recovered Database.  The Local OPLAN Recovery Cleanup option on the Plan Maintenance tear-off is used to ensure that the sending site's locally distributed OPLANs were indeed "cleaned out."

Access to the Local OPLAN Recovery Cleanup window is controlled by UNIX system level file execute permissions.

Only USERIDs designated as members of the UNIX permissions group assigned to the Oracle Database Administrators will be able to execute this option.

Note:  The user account group name for the Database Administrators is site specific.

The Database Administrator depresses either or both database area toggles for recovery and cleanup activities.  Upon depressing the Transmit button, four distinct operations are invoked at the recovering site.
All data for all local plans not created on the receiving site's server are deleted.  Recognition of PID initialization is maintained.

All data for all networked plans not distributed to the receiving site's server are deleted.  Recognition of PID initialization is maintained.

All local plans originated on the local server, but not containing any data, are displayed in a pop-up window for review and further action.  The missing data must be uploaded from previously accomplished local plan saves.

All networked plans routed to the receiving site's server, but not containing any data, are displayed in a pop-up window for review and further action.  The missing data will require upload from previously accomplished saves or the partial site recovery procedures can be used.

Add Local Data.  The upload of locally distributed OPLANs will probably be accomplished by using the Reload Plan option.

If the FM (or site designated individual) periodically copies the locally distributed database using the Offload Plan option, then the reload of this data will be as current as the last time the offload was done.

It is quite possible that the individuals personally responsible for each of those OPLANs may have more current data available.

That is, very briefly, the method by which a complete JOPES OPLAN database can be restored.  But what can be done if a site restores from a backup site and is not perfect.  Some data is still not correct and this major overhaul needs a minor tuneup as well.  You can now go back to the previous menu and the lesson will describe the partial site recovery procedures.

Partial (Selective) Site Recovery.  If the database is missing some or several vital pieces of OPLAN information, then you are basically in a non-synchronized condition.  To repair this condition, the Site Data Recovery option provides the Network FM a way to recover from a partial site failure condition.  As with a complete site recovery, the Site FMs, the Network FM, and the site DBAs should work closely together to accomplish the repair.
The Selective Site Recovery options in the System Services application allows Network FMs to recover portions or all of a networked OPLAN by building transactions from a providing site's database and routing them to a file or to a receiving site for update processing.

The condition that you have now (non-synchronous OPLANs) is identical to the one taught in Lesson 4, so the procedures to follow and the windows to use are the same.

Now that you are familiar with the site JOPES database recovery capabilities of the new software, you will proceed to the functions used to control the activities of the Transaction Processor and the Transaction Distribution Services.

OBJECTIVE.  Given access to a GCCS environment, access the JOPES Transaction Processor and JOPES Transaction Distribution Services windows and explain their function.

Transaction Management.  As with any system, management of updates to not only the locally resident database, but management of the updates being networked to other site databases, are required.  Access to the windows that control the activities of the TP and the TDS are limited to those individuals whose user accounts are members of the UNIX group (site specific) with execute permissions for the TP/TDS Management subsystems.
Transaction Processor Management.  The Transaction Processor is responsible for applying transactions to the locally resident JOPES database.  Configuration (queue size, number of records in the queue, etc.) of the TP is controllable by the System Administrator from the system level prompt (UNIX commands).  There is one window that is available to control the activity of the TP.

The Transaction Processor Management window displays and controls the activity of the TP and its associated Error Log.
Transaction Processor.  This window displays the current status of the TP and allows the operator to activate or terminate (deactivate) the TP.
To activate the TP, select the Activate button and then <POINT AND CLICK (left)> on Transmit.  When the TP is active, it accesses the transactions in the Receive Queue and applies them to the locally resident JOPES database.

To terminate (deactivate) the TP, select the Terminate button and then <POINT AND CLICK (left)> on Transmit.  A "Termination Queued" message will display.  A delayed response to the termination request can be expected if there are any transactions in the TP's internal buffer.  All transactions in the TP's internal buffer must be processed before it will check for a termination request.  When the TP has been terminated (is inactive), it does not communicate with the Receive Queue.  Therefore, transactions in the Receive Queue will not be processed for update to the locally resident JOPES database.

Note:  The full title for the Error Log referred to on the window is General Error Log.  The General Error Log should not be confused with the Error Log.  The Error Log stores transaction based errors and is accessible through the Audit Subsystem.

(General) Error Log.  This window indicates if a General Error Log exists and allows the operator to print and/or reset it.
The General Error Log is a file that resides on disk and contains all non-transaction based errors, such as configuration parameters and Oracle database access problems, that the TP encounters.  Each entry in the General Error Log contains the date/time of the error, an error message, module name, and associated data.  This error log is of interest to the System Administrator.

The General Error Log contents are printed on the default printer as configured by the System Administrator.  To erase (empty) the file, depress the Reset button and then <POINT AND CLICK (left)> on Transmit.

Now that you know how to turn the TP on or turn it off, you will turn your attention to the TDS.

Transaction Distribution Management.  Transaction Distribution Management is a component in the migration of JOPES applications from a mainframe centered environment to a client/server environment.  During the transition process, both components, mainframe database and server database, must remain a part of the architecture.

The TDS routes transactions from server to server, from mainframe to server, and from server to mainframe.  Configuration of the TDS is also controllable by the System Administrator from the system level prompt (UNIX commands).

The Transaction Distribution Management window displays and controls the activity of the TDS and its components.  The Current Status fields display the status of the TDS and its components.

Transaction Distribution.  The TDS must be enabled for transactions to flow through the network, keeping the JOPES databases synchronized.  When the TDS is disabled, all transaction distribution processes controlled by this specific TDS are automatically terminated and all transaction flow is stopped.

To enable TDS, depress the Enable button; to disable TDS, depress the Disable button and then <POINT AND CLICK (left)> on Transmit.  The system will update the Current Status field when the option is executed.  If the selected option fails, the system will display an error message in the Message field.

Note:  The Transaction Distribution Enable and Disable buttons are very similar to master switches.

Local Incoming Transactions.  Local incoming transactions are those transactions generated by other Transaction Distribution Services that are directly connected to this database server.  The distribution of these transactions is not via the mainframe network.

When enabled, Local Incoming Transactions are inserted into the Receive Queue for database update.  When disabled, Local Incoming Transactions are accumulated, but are not placed in the Receive Queue.  Transaction flow throughout the rest of the network is not affected.

Local Outgoing Transactions.  Local outgoing transactions are those transactions in the Send Queue that require distribution to the other TDS systems, which are directly connected to this TDS for update to their database server.  Updates to databases do not flow via the mainframe network.

When enabled, Local Outgoing Transactions are taken from the Send Queue and distributed to the appropriate TDSs.  When disabled, Local Outgoing Transactions remain in the Send Queue.

Host Incoming (Domain Only) Transactions.  Host incoming (Domain Only) transactions are those transactions that must be submitted from this TDS to its directly connected mainframe host.

When enabled, the TDS submits transactions to its associated mainframe.  When disabled, the TDS does not submit any transactions to its associated mainframe.

Host Outgoing (Domain Only) Transactions.  Host outgoing (Domain Only) transactions are those transactions that must be pulled from the directly connected mainframe to this TDS for update to its associated database and forwarding via the Local Outgoing Send Queue as required.

When enabled, the TDS pulls transactions from its associated mainframe.  When disabled, the TDS does not pull any transactions from its associated mainframe.

You may or may not be specifically responsible for the management of the TP and the TDS.  If you are, you should have a good foundation from which to increase your knowledge concerning the operation of the TP and the TDS.  But what happens if a new database server is added to the network or if one is deleted?

OBJECTIVE.  Given a GCCS environment, explain the purpose of the JOPES TDS Network Management window.

Transaction Distribution Services Network Management.  Access to the Transaction Distribution Services Network Management window is restricted to those individuals having NFM permissions.
Purpose.  The TDS Network Management window provides you with the ability to modify or view an existing database server's information.  You can also delete a server from the TDS network.  In addition, you can add a database server to the TDS network.

Access.  Access to the capability is via the System Services application.

View.  To view an existing database server's information, you enter either the server name or the server address in the appropriate field.  After tabbing out of that field, the system will immediately add the name or the address as well as the description, the primary machine, and the user name.
The Server Name is the common name of the server.  An alias can also be used to refer to a server.  If you enter a recognized alias, the program will find the common name of the server and display that name in the Server Name field.

The Server Address is the IP address for that server.

Note:  The Network Type field will display the type of network to which this computer is connected.

The Description is a free text field that describes the server being addressed.

The Primary field displays the name of the mainframe from which this server receives transactions and to which this server sends transactions.  If this field is blank, then this server is not directly connected to a mainframe.

The User Name field is this server's account to which other computers forward transactions for JOPES database updates.

When another computer attempts to forward a JOPES OPLAN transaction, it must also know the password in order to access the user name account.

Modify.  To modify an existing database server's information (not to include the name or address), the operator would first view the information and then tab to the appropriate field and make the modification.  Clicking on Transmit will change the appropriate information.

If the database server name or server address requires modification, the operator should select the Modify Server Name/Address toggle.  This causes all fields except the Server Name and Address to become inactive.  The modification is affected by changing the name and/or address and clicking on Transmit.

Delete.  To delete an existing database server from the TDS network, the operator would first view the information.  By selecting the Delete Server button and clicking on Transmit, the server is deleted from TDS access.

Note:  This database server is only deleted from the TDS network in the sense that other database servers no longer recognize this one as existing for the purposes of distributing JOPES OPLAN transactions.  Physically, it is still connected to the communications network.

Add.  To add a database server to the TDS network, the operator enters the server name or the server address in the appropriate field.  After tabbing out of that field, the system will immediately add the name or the address.  The operator would then complete the remaining fields, as appropriate, and <POINT AND CLICK (left)> on Transmit.

That completes your activities regarding the recovery of a database and the System Services mechanisms in place to control the activities of the Transaction Processor and the Transaction Distribution Services.  You can now quit System Services.

Summary.  During this lesson, you discussed the actions necessary to accomplish a save of the JOPES database in the event of a site database failure and the actions necessary to restore that site.  You also reviewed the Database Administrator's (or Functional Manager's) capability to manage the activities of the Transaction Processor and the Transaction Distribution Services software.  The lesson also included the Network Functional Manager's (or DBA's) ability to add/delete/modify a database servers' information in the network, plus the ordinary user's ability to update TPFDDs using a transaction file generated by software external to the core database.