APP01 Recovery (MESSAGING Processes)

 

                APP01 Purpose:

APP01 processes receive MESSAGING from IBs and send them to External Vendors or Order Management Services.

They then receive related responses from these services and send these back to IBs.

 

Use the following hyperlinks to jump to the desired section of APP01 documentation:

APP01_Recovery_Considerations

APP01_NTM_Control_Commands

APP01_Troubleshooting_Table

APP01_Monitoring_Considerations

 

APP01 Recovery Considerations:

                                Stopping/Restarting/Moving Processes:

-          Use NTM Control Utility - Service Control - Process Controller to stop/restart processes.

 

-          When moving between nodes:

o   ALTERNATE NODES are not defined for APP01 services.

o   DR NODES must be allocated un-natted nodes. 

No node can support more than one natted address at the same time.

 

-          When stopping/restarting APP01 processes:

1)      Notify the associated vendor or order management service firm and work in cooperation with them, as appropriate to situations.

2)      Notify Tech Services if moving APP01 processes to DR nodes and NAT addresses need to change to accommodate move.

3)      Stop the APP01 process.

4)      If NOT moving APP01 to new node, skip to step 5.

a)       If moving the APP01 to a new node, copy the day’s APP01 FIX engine files to alternate node:

a)       Copy: \\data\APP01File\*.log, *.body, *.header, *.seqnums, *.session file created for the day to the alternate node.

b)      Copy: \\data\APP01File\Global.* file created for the day to the alternate node.

c)       If the target folder does not exist on the new node, you must first either create the folder, or copy the entire folder.

d)      If these files are not moved before the process restarts on the new node, there will be a chance of sequence number miscommunications between the order sending firm involved and .

5)      Start the APP01 process.

6)      Open the channel for the process affected and confirm order sending firm connects as expected.

 

 

APP01 NTM Control Commands:

Open OSF Channel:

-          Use NTM Control Utility – Service Control - APP01 – Open OSF Channel to make OSF connection possible.

 

Close OSF Channel:

-          Use NTM Control Utility – Service Control - APP01 – Close OSF Channel to make OSF connection impossible.

 

Set Inbound Sequence Number:

-          Use NTM Control Utility – Service Control - APP01 – Set Inbound Sequence Number to set  Inbound Sequence Number.

 

Set Outbound Sequence Number:

-          Use NTM Control Utility – Service Control - APP01 – Set Outbound Sequence Number to set  Outbound Sequence Number.

 

Enable THIRD PARTY Stats:

-          Use NTM Control Utility – Service Control - APP01 – Enable THIRD PARTY Stats to start collection and display of LBM related stats.

 

Disable THIRD PARTY Stats:

-          Use NTM Control Utility – Service Control - APP01 – Disable THIRD PARTY Stats to stop collection and display of LBM related stats.

 

APP01 Troubleshooting Table:

APP01 Symptom

Impacts

Response

Firm disconnects or Logs out of session

 

Evidenced by:

-          EMT message saying {firm} is disconnected and/or {firm} is logged out.

 

-          Stats monitor shows disconnected in status column.

 

{Firm} will be further identified in EMT message by including “LocalFixId” and “RemoteFixID” as configured in APP01Services.xml file within the disconnect message.

 

IB is no longer able to send or receive order or order related messages with Vendor or Order Management Service.

1)      Contact firm.

2)      Work with firm and/or Technical Services as necessary to isolate cause of issues and resolve them.

3)      Stop/Restarts of affected application service may help resolve the issue.

 

 

APP01 Monitoring Considerations:

Stats Monitors:
APP01 App Connect Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:


Processes facilitate communications from  to MESSAGINGs.

Monitor shows connection status between  fix engine and firm, or of fix engine and  application.

PROD MENU:
Firmswitch Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Status,
- Write Queue

Data is RED.

Process is either down or multicast data is not being received by monitor.

1) Check status of process

2) If process is up, call Technical Services.

Status is Disconnected or Open

Firm is not connected.
1) Use NTM Control Utility APP01 Service Controls to Open Channels.
2) Call Production Control if needed.

Write Queue is non-zero values and not reducing as expected.

Firm may not be processing as expected.
1) Call firm and work with Technical Services if necessary.

 

 

Stats Monitors:
APP01 App Processing Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from  to MESSAGINGs.

Monitor shows processing statistics.

PROD MENU:
Firmswitch Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- InCount,
- OutCount,
- LastInTime,
- LastOutTime

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

No data is displayed.

No data has been generated to any MESSAGING. 
1) Check APP01 log files. If all sizes are zero, no traffic has been generated.
2) Test sending order to MESSAGING.

InCount does not match OutCount.

Firm may not be processing as expected.
1) Check APP01 FIX message files to confirm inbound messages match outbound messages.
2) Call firm and work with Technical Services if necessary.

 

Stats Monitors:
APP01 IPC Instance Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from  to MESSAGINGs.

Monitor shows IPC channel processing statistics.

PROD MENU:
Firmswitch Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Service,
- Hostname,
- Msgs In,
- Msgs Out

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Not all APP01 processes are displayed as expected.

APP01 Service has not been started.
1) Check status of service.

Msgs In and/or Msgs Out are zero.

No messages have been sent/received since that monitor has been started.
1) Check APP01 log files.
2) Call firm and work with Technical Services if necessary.

 

 

Stats Monitors:
APP01 to OMS IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from  to MESSAGINGs.

Monitor shows IPC channel connectivity status to OMS processes.

PROD MENU:
Firmswitch Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Not all APP01 or OMS processes are displayed as expected.

APP01 or OMS Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP01 or OMS log files.

Status is not CONNECTED.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

Queue size is non-zero value and not decreasing as expected.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.

 

 

Stats Monitors:
APP01 to FMR IPC Connect Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from  to MESSAGINGs.

Monitor shows IPC channel connectivity status to FMR processes.

PROD MENU:
Firmswitch Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Source,
- Dest,
- Status,
- Queue Size,
- Msgs Out/Sec

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Not all APP01 or FMR processes are displayed as expected.

APP01 or FMR Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check APP01 or FMR log files.

Status is not CONNECTED.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

Queue size is non-zero value and not decreasing as expected.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.