APP30 Recovery (APP USE Service)

               

APP30 Purpose:

APP30 reads order related messages from APP01, APP02, and APP03 and forwards them to destinations defined in USE agreements or on messages.

APP30 sends order related messages to APP02, APP01 and/or APP11s depending on applied USE instructions.

 

 

Use the following hyperlinks to jump to the desired section of APP30 documentation:

APP30_Recovery_Considerations

APP30_NTM_Control_Commands

APP30_Troubleshooting_Table

APP30_Monitoring_Considerations

 

 

APP30 Recovery Considerations:

                                Stopping/Restart Processes:

-          Use NTM Control Utility - Service Control - Process Controller to stop/restart processes.

-          Use APP30 nodes only when moving between nodes.  (Java code and FireDaemon references are required.)

 

-          When stopping/restarting APP30 processes:

1)      Notify  Operations and work in cooperation with them, as appropriate to situations.

2)      Stop/Restart the APP30 process.

 

APP30 NTM Control Commands:

Reload Rules:

-          Use NTM Control Utility – Service Control - APP30 – Reload Rules.

 

End Of Day:

-          Use NTM Control Utility – Service Control - APP30 – End of Day.

 

 

 

 

APP30 Troubleshooting Table:

APP30 Symptom

Impacts

Response

Node Crashes

 

Evidenced by:

-          In Solarwinds (and outlook), node and processes will be reported down.

 

 Operations will not be able to query or administratively manage orders, trades or clearing reports via CSI.

 

Refer to:

Server Recovery - Opcon APP30 CSI DCSR.docx

 Server Specific Recoveries.

1)      Refer to:

Server Recovery - Opcon APP30 CSI DCSR.docx

 Server Specific Recoveries.

2)      Notify Management.

 

 

APP30 Monitoring Considerations:

Stats Monitors:
APP30 IPC Instance Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from Order Sending Firms that use APP03 services for special USE (to APP10, APP12 or APP11s).

Monitor shows IPC channel processing statistics.

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Service,
- Hostname,
- Msgs In,
- Msgs Out

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

 

 

 

Not all APP30 processes are displayed as expected.

APP30 Service has not been started.
1) Check status of service.

 

 

 

Msgs In and/or Msgs Out are zero when messages are processed.

No messages have been sent/received since that monitor has been started.
1) Check APP30 log files.
2) Call Production Support if necessary.

 

 

Stats Monitors:
APP30 to APP01 IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from Order Sending Firms that use APP03 services for special USE (to APP10, APP12 or APP11s).

Monitor shows IPC channel connectivity status to APP01 processes. 

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

 

 

 

Not all APP30 or APP01 processes are displayed as expected.

APP30 or APP01 Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP30 or APP01 log files.

 

 

 

Status is not CONNECTED.

 

Messages cannot be sent from source to destination unless IPC channel is connected.

1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

 

 

 

Queue size is non-zero value and not decreasing as expected.

 

Messages cannot be sent unless IPC channel is connected.

1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.

 

Stats Monitors:
APP30 to APP02 IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from Order Sending Firms that use APP03 services for special USE (to APP10, APP12 or APP11s).

Monitor shows IPC channel connectivity status to APP02 processes. 

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

 

 

 

Not all APP30 or APP02 processes are displayed as expected.

APP30 or APP02 Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP30 or APP02 log files.

 

 

 

Status is not CONNECTED.

 

Messages cannot be sent from source to destination unless IPC channel is connected.

1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

 

 

 

Queue size is non-zero value and not decreasing as expected.

 

Messages cannot be sent unless IPC channel is connected.

1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.

 

Stats Monitors:
APP30 to APP03 IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from Order Sending Firms that use APP03 services for special USE (to APP10, APP12 or APP11s).

Monitor shows IPC channel connectivity status to APP03 processes. 

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

 

 

 

Not all APP30 or APP03 processes are displayed as expected.

APP30 or APP03 Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP30 or APP03 log files.

 

 

 

Status is not CONNECTED.

 

Messages cannot be sent from source to destination unless IPC channel is connected.

1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

 

 

 

Queue size is non-zero value and not decreasing as expected.

 

Messages cannot be sent unless IPC channel is connected.

1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.

 

Stats Monitors:
APP30 to ME IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from Order Sending Firms that use APP03 services for special USE (to APP10, Away Destinations, APP11s and Trade Reporting Systems.

Monitor shows IPC channel connectivity status to APP30 Bridge / APP11 processes. 

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

 

 

Not all APP30, APP30Bridge or APP11 processes are displayed as expected.

APP30, APP30Bridge or APP11 Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP30, APP30Bridge or APP11 log files.

 

 

 

Status is not CONNECTED.

 

Messages cannot be sent from source to destination unless IPC channel is connected.

1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

 

 

 

Queue size is non-zero value and not decreasing as expected.

NOTE: APP30Bridge values will always be zero.  Only the "bridge-to-ME" channels will show non-zero values.

1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.

 

 

Stats Monitors:
APP30 to TRS IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from Order Sending Firms that use APP03 services for special USE (to APP10, APP12 or APP11s).

Monitor shows IPC channel connectivity status to TRS processes. 

PROD MENU:
Trading Applications Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

 

 

Not all APP30 or TRS processes are displayed as expected.

APP30 or TRS Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP30 or TRS log files.

 

 

 

Status is not CONNECTED.

 

Messages cannot be sent from source to destination unless IPC channel is connected.

1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

 

 

 

Queue size is non-zero value and not decreasing as expected.

NOTE: Queue size will be one until more than one message is generated.  TRS messaging from APP30 is mostly defunct as of the last update of this documentation.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.