APP06 Recovery (APP06)

               

APP06 Purpose:

APP06 Servers receive MESSAGING, MESSAGING and MESSAGING from Firms and send responses to these to Firms.

APP06 Servers receive order responses from APP01, APP02, TRFs, as well as “regulatory” drop copies.

APP06 Servers send MESSAGING to APP01, APP02 and TRFs, and drop copies to APP06 Drop Copy processes.

APP06 Servers also send trade reports directly to TRS services when they correct trades.

 

Use the following hyperlinks to jump to the desired section of APP06 documentation:

APP06_Recovery_Considerations

APP06_NTM_Control_Commands

APP06_Troubleshooting_Table

APP06_Monitoring_Considerations

 

 

 

APP06 Recovery Considerations:

                                Stopping/Restart Processes:

-          Use NTM Control Utility - Service Control - Process Controller to stop/restart processes.

-          Use APP06 nodes only when moving between nodes.  (Java code, FireDaemon and Host Specific references in JNLPs required.)

 

-          When stopping/restarting APP06 processes:

1)      Notify the IBs and work in cooperation with them, as appropriate to situations.

2)      Stop/Restart the APP06 process.

 

 

APP06 NTM Control Commands :

Refresh Threshold Data:

-          Use NTM Control Utility – Service Control - APP06 – Refresh Threshold Data.

 

Refresh Trade Ack Data:

-          Use NTM Control Utility – Service Control - APP06 – Refresh Trade Ack Data.

 

Refresh Brokers Clerks Data:

-          Use NTM Control Utility – Service Control - APP06 – Refresh Brokers Clerk Data.

 

Refresh Sub Accounts Data:

-          Use NTM Control Utility – Service Control - APP06 – Refresh Sub Accounts Data.

 

Turn On Clearing Control:

-          Use NTM Control Utility – Service Control - APP06 – Turn on Clearing Control.

 

Turn Off Clearing Control:

-          Use NTM Control Utility – Service Control - APP06 – Turn off Clearing Control.

 

End Of Day:

-          Use NTM Control Utility – Service Control - APP06 – End Of Day to start End Of Day processing.

 

Client Shutdown:

-          Use NTM Control Utility – Service Control – APP06 – Client Shutdown to remotely shutdown all clients for the instance.

 

 

APP06 Troubleshooting Table:

APP06 Symptom

Impacts

Response

Node Crashes

 

Evidenced by:

-          In Solarwinds (and outlook), node and processes will be reported down.

 

IBs will not be able to receive or process MESSAGING via APP06.

 

Refer to:

Server Recovery - APP06.docx

 Server Specific Recoveries.

 

1)      Refer to:

Server Recovery - APP06.docx

 Server Specific Recoveries.

2)      Notify Management.

 

                           

APP06 Monitoring Considerations:

Stats Monitors:
APP06 IPC Instance Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from APP06 servers to Order Sending Firms, Drop Copy Vendors, APP02, APP01 and Trade Reporting Systems.

Monitor shows IPC channel processing statistics.

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Service,
- Hostname,
- Msgs In,
- Msgs Out

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Not all APP06 processes are displayed as expected.

APP06 Service has not been started.
1) Check status of service.

Msgs In and/or Msgs Out are zero when messages are processed.

No messages have been sent/received since that monitor has been started.
1) Check APP06 log files.
2) Call Production Support if necessary.

 

 

 

Stats Monitors:
APP06 to BDC IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from APP06 servers to Order Sending Firms, Drop Copy Vendors, APP02, APP01 and Trade Reporting Systems.

Monitor shows IPC channel connectivity status to BDC processes.

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Not all APP06 or BDC processes are displayed as expected.

APP06 or BDC Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP06 or BDC log files.

Status is not CONNECTED.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

Queue size is non-zero value and not decreasing as expected.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.

 

 

Stats Monitors:
APP06 to DTRF IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from APP06 servers to Order Sending Firms, Drop Copy Vendors, APP02, APP01 and Trade Reporting Systems.

Monitor shows IPC channel connectivity status to DTRF processes.

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Not all APP06 or DTRF processes are displayed as expected.

APP06 or DTRF Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP06 or DTRF log files.

Status is not CONNECTED.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

Queue size is non-zero value and not decreasing as expected.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.

 

 

Stats Monitors:
APP06 to ME IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from APP06 servers to Order Sending Firms, Drop Copy Vendors, APP02, APP01 and Trade Reporting Systems.

Monitor shows IPC channel connectivity status to OMS Bridge / Matching Engine processes.  The only messages going through these channels are Market Data Query messages.

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Not all APP06 or Matching Engine processes are displayed as expected.

APP06 or Matching Engine Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP06 or Matching Engine log files.

Status is not CONNECTED.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

Queue size is non-zero value and not decreasing as expected.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.

 

 

Stats Monitors:
APP06 to OMS IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from APP06 servers to Order Sending Firms, Drop Copy Vendors, APP02, APP01 and Trade Reporting Systems.

Monitor shows IPC channel connectivity status to OMS processes. 

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Not all APP06 or OMS processes are displayed as expected.

APP06 or OMS Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP06 or OMS log files.

Status is not CONNECTED.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

Queue size is non-zero value and not decreasing as expected.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.

 

 

Stats Monitors:
APP06 to TRS IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from APP06 servers to Order Sending Firms, Drop Copy Vendors, APP02, APP01 and Trade Reporting Systems.

Monitor shows IPC channel connectivity status to TRS processes. 

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Not all APP06 or TRS processes are displayed as expected.

APP06 or TRS Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP06 or TRS log files.

Status is not CONNECTED.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

Queue size is non-zero value and not decreasing as expected.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.