Tymeac Shut Down

Tymeac may quiesce and terminate gracefully.

Tymeac supports user exits for shut down.

Tymeac shuts down in two (and possibly three) stages. Tymeac is a thread manager. When there are no Queue Threads active, then stage two immediately follows stage one without further user action and the Server terminates quickly.

Quiesce means to quiet or calm. Shut down quiets by rejecting new requests and letting those already in progress complete. When there is no more activity, the Server stops gracefully.

There are three exceptions to in progress completion:

One is scheduling. Since shut down means stop, those requests in the middle of being scheduled cannot complete. The logic here is that the Server may be under stress and the shut down request is meant to quickly terminate the Server. Waiting for each Queue to schedule may hold up the termination indefinitely.

The second is recursive requests. Since the Server cannot know from where the request originates, a recursive request is simply a new request and it rejects.

The third is forcing termination. We get into this when discussing stage 2.

There are two implementation methods for shut down:

public String shutRequest(); -- This is the same as the second with a boolean false.

public String shutRequest(boolean kill); -- This version supports compelling termination in the case when there are unresponsive Queue Threads.

For all normal shut down requests, you should set the boolean to false or simply use version 1. If normal shutdown cannot progress and you need the Server down quickly, then set the boolean to true.

Killing the server still executes the shut down exits, statistics and shut down thread (when chosen.) Killing only ignores the presents of active Queue Threads. If not using the shut down thread and a user exit Class is hanging, then killing has no effect.

The first stage: [the first shutRequest() received by the server]

Blocks the acceptance of further requests into the system. Tymeac rejects any further requests into the system and returns a request brokering error or a scheduling error to the caller.

Multiple Queue requests currently in scheduling reject.

For example, a request with three Queues enters the system.

Queue 1 schedules.
The shut down request enters.
Queue 2 fails scheduling because of shut down.
No further scheduling takes place, the request rejects.

Nested processing requests fail.

For example, a request uses a Queue with a Processing Application Class that recursively calls the Tymeac Server for a new request. (The example section, portfolio revaluation, describes this.) When that Processing Application Class calls the Tymeac Server for the new request, the new request rejects.

Instantiates the stage-1 shut down user exit Classes, if present.

For non-Output Agent Queues:

When there are Synchronous requests working, shut down informs the live Queue Threads to terminate upon completion of all work in the Wait Lists.

When no Synchronous requests are working, shut down terminates the Queue Threads.

For Output Agent Queues:

When there are Asynchronous requests working, the Output Agent Queue Threads remain waiting for new work since they may schedule at any time. (Asynchronous requests may schedule an Output Agent Queue. Therefore, the Output Agent Queue Threads must remain active until all work finishes.)

When no Asynchronous requests are working, shut down terminates the Output Agent Queue Threads.

The second stage: [second shutRequest() received by the server]

Stage 2 (and Stage 3 naturally) cannot begin until Stage 1 is complete.

Stage 1 may use shut down Stage 1 user exits. If one thread is executing a Stage 1 user exit Class and a second thread starts a Stage 2 user exit Class the result could be disastrous. Since the shut down user exit Classes run as part of the main Tymeac structure (not as a separate thread), a stall in one of these Classes stalls the entire shut down sequence.

When there are Synchronous requests working, forces termination of all live, non-Output Agent Queue Threads. The Queue Threads terminate without finishing requests in the Wait Lists. This means that Synchronous request in progress time-out while Asynchronous requests in progress stall.

For Output Agent Queues:

When there were asynchronous requests working at the time of prior shut down, then this stage informs the live Output Agent Queue Threads to terminate upon completion of all work in the Wait Lists. (Just like the stage 1 shutdown for non-Output Agent Queue Threads.) In order to force termination, one must request shut down again. (See the example, below.)

The third or Force stage [subsequent shutRequest() received by the server] (when requests were working in stage two and threads were active.)]

This stage forces termination of all live Output Agent Queue Threads. The Output Agent Queue Threads terminate without finishing requests in the Wait Lists. This means that Asynchronous requests in progress stall. (Just like the stage 2 shutdown for non-Output Agent Queue Threads.)

When all threads complete or in kill mode:
Single threads subsequent processing (i.e. only one thread may continue, other threads return with a "shut down in progress" message.) (There is no way we could multi-thread the following steps.)

Instantiates the stage-2 shut down user exit Classes, if present.

Writes the system statistics.

Instantiates the Embedded Shut Down Class, if present.

Starts the shut down thread (see below) if this option was chosen in the Configuration Data Class TyCfg or the TyUserVariables Class.

There is no way to terminate a Java thread that is not responding. Three options are available for unresponsive Queue Threads:

Use the Tymeac Queue Thread Class (TyQueThd) to logically disable the thread. This does not kill the thread but marks the entry as "disabled". This permits the completion of shut down.

Use a Java/operating system tool to kill the thread. Then use step one, above, to logically disable the entry.

Use the shut down "kill" option. This is a shut request with a boolean true. E.G.
shutRequest(true);

If the Server hangs due to a faulty exit class, then your only alternative is to manually kill the Server.

Two methods for shutting down Tymeac Server are available. A Frame Class and a command line Class.

Frame: TyShutdown

Start the Class as a new process

Press the SHUT Button.
    Observer the message and if necessary
Press the SHUT Button.
     Observer the message.

If the Server does not shut down because of a stall and you wish to force the shut down, then Press the KILL Button.

Command Line: com.tymeac.demo.DemoClienShut

See /Scripts/shut

Do this twice, if necessary.

If the Server does not shut down because of a stall and you wish to force the shut down, then
    See /Scripts/shutforce

Usually, threads are still active and shut down cannot complete fully. Press the SHUT button again or run the command line Class again.

For example, assume there are working asynchronous requests:

Shut down requested, (stage one),

The Tymeac message is that 35 threads are still active.

Shut down requested, (stage two),

The Tymeac message is that 8 threads are still active. [These are the Output Agent Queue Threads]

Shut down requested, (stage Force (three)),

The Tymeac message is that the shut down is complete.

Shut Down Thread

RMI Servers start a thread that never ends. This is so that the Server is persistent until manually shut down. The Internal Server also has a few RMI threads since the TymeacInterface extends the Remote class.

All Tymeac Servers issue a never ending wait() in the start up Class so that necessary variables do not get garbage collected.

Tymeac includes an optional shut down thread. This option is in the Configuration File Maintenance Class, TyCfg, or, for Stand-Alone mode, the TyUserVariables Class.

The thread, after a two (2) second delay, calls System.exit(0) to end the Java Virtual Machine. The delay is so that messages from the Tymeac Server to the initiating Tymeac Client complete the journey.

Q. So, how does one know when there are requests working or when all activity ends? This is the purpose of the Overall Status Display, TyOverall, or the non-frame alternative, TyOverallClient.

After requesting shut down the "Top List" shows pending requests, both synchronous and asynchronous as well as any stalled asynchronous requests.

Tymeac times synchronous requests (this is a parameter for the request to Tymeac). Therefore, hanging requests should time out and purge from the system.

The Tymeac Monitor times asynchronous requests. Therefore, hanging requests stall. One may purge these stalled requests with the frame Class, TyStalled.

When there are no further requests waiting to process (or it is irrelevant that some are hanging) request shut down again. Now the Output Agent Queue Threads terminate when all work finishes in the Wait Lists. The "Bottom List" shows only those Queues with attendant [able to process] threads.

For those Queue Threads that cannot end because of problems, use the Queue Thread Display/Alter frame Class to disable those threads.

When the "Bottom List" shows that there are no Queues with attendant threads, then the system is halted. Requesting shut down again ends the Tymeac Server.

Q. So, what if things get completely out of hand? You just want to kill the Tymeac Server process quickly, but, you also want the statistics that come from shut down?

Request shut down several times to flush out all that can end smoothly.

Request an On-Request statistics. This writes the current statistics, including stalled requests, to the Statistics repository (see alternatives).

Then use an operating system tool to kill the Tymeac Server process.

The final message is that shut down is complete.

"Tymeac Server, Shut down complete for MilliTime= "

MilliTime is the System time in milliseconds since 1970. MilliTime also appears on the successful start up message and within all return messages for Asynchronous Requests. It is the way to identify this session of Tymeac.

N.B. When using the Activation Framework:

Inactivation (after the inactivation time elapses (DBMS mode, Stand-A-Lone mode) is performed by the Monitor. Part of the inactivation procedure is to shut down the Server before inactivating the process from the Activation Run Time.

As long as no requests for shut down are outstanding, then the Monitor runs through stage 1 and 2 as above, then inactivates:
java.rmi.activation.Activatable.inactive()

and issues a System.exit().

If a prior request for shut down is outstanding, then the Monitor waits respectfully for the shut down to complete. After a wait period, the Monitor completes the shut down when possible and un-registers the Server from the Activation System.
java.rmi.activation.Activatable.unregister()

and issues a System.exit().

Therefore, you must re-register the Tymeac Server to use it again.

As noted above, any procedure that stalls, will hang the shut down. When the Monitor believes a stalled shut down is preventing the process from ending, the Monitor logs and prints messages. The Notification System is no long alive since shut down prevents new requests from starting. A shutting down but stalled Server is of no use to anyone and it remains an active process in the computer system.