APP16 Recovery (APP16 Fix Engine)

APP16 Purpose:

APP16 Fix Engine receives clearing messages from APP01 and sends them to APP16 for actual clearing.

They then receive related responses from these services for error handling.

Use the following hyperlinks to jump to the desired section of APP16 documentation:

APP16_Recovery_Considerations

APP16_NTM_Control_Commands

APP16_Troubleshooting_Table

APP16_Monitoring_Considerations

APP16 Recovery Considerations:

Stopping/Restart Processes:

- Use NTM Control Utility - Service Control - Process Controller to stop/restart processes.

- Use OMD nodes only when moving between nodes. (OMD NAT addresses must be configured/used by APP16.)

- When stopping/restarting APP16 processes:

1) Notify APP16 and work in cooperation with them, as appropriate to situations.

2) Stop the APP16 process.

3) If NOT moving APP16 to new node, skip to step 5.

a) If moving the APP16 to a new node, copy the day’s APP16 FIX engine files to alternate node:

a) Copy: \\data\APP16\*.log, *.body, *.header, *.seqnums, *.session file created for the day to the alternate node.

b) Copy: \\data\APP16\Global.* file created for the day to the alternate node.

c) If the target folder does not exist on the new node, you must first either create the folder, or copy the entire folder.

d) If these files are not moved before the process restarts on the new node, there will be a chance of sequence number miscommunications between the order sending firm involved and .

4) Start the APP16 process.

5) Open the channel for the process affected and confirm order sending firm connects as expected.

APP16 NTM Control Commands:

Open OSF Channel:

- Use NTM Control Utility – Service Control - APP16 – Open OSF Channel to make OSF connection possible.

Close OSF Channel:

- Use NTM Control Utility – Service Control - APP16 – Close OSF Channel to make OSF connection impossible.

Set Inbound Sequence Number:

- Use NTM Control Utility – Service Control - APP16 – Set Inbound Sequence Number to set Inbound Sequence Number.

Set Outbound Sequence Number:

- Use NTM Control Utility – Service Control - APP16 – Set Outbound Sequence Number to set Outbound Sequence Number.

Enable 29West Stats:

- Use NTM Control Utility – Service Control - APP16 – Enable 29West Stats to start collection and display of LBM related stats.

Disable 29West Stats:

- Use NTM Control Utility – Service Control - APP16 – Disable 29West Stats to stop collection and display of LBM related stats.

APP16 Troubleshooting Table:

APP16 Symptom	Impacts	Response
Firm disconnects or Logs out of session Evidenced by: - EMT message saying {firm} is disconnected and/or {firm} is logged out. - Stats monitor shows disconnected in status column. {Firm} will be further identified in EMT message by including “LocalFixId” and “RemoteFixID” as configured in APP16Services.xml file within the disconnect message.	APP16 is no longer able to receive clearing messages.	1) Contact firm. 2) Work with firm and/or Technical Services as necessary to isolate cause of issues and resolve them. 3) Stop/Restarts of affected application service may help resolve the issue.

APP16 Monitoring Considerations:

Stats Monitors: APP16 App Connect Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate clearing trade delivery from processes to APP16. Monitor shows connection status between fix engines and APP16 Destination as well as processing statistics.	PROD MENU: Market Data Monitoring Menu To Exit: Close Window	- Color of data in columns - Status, - OutMsgs, - InMsgs, - OutTrdCapRpts	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Status is Disconnected or Open	Firm is not connected. 1) Use NTM Control Utility APP16 Service Controls to Open Channels. 2) Call Production Control if needed.
			OutTrdCapRpts value does not match OutCnt value in RTC01 to APP16 App Queues monitor.	We may not be processing as expected. 1) Check RTC log files and APP16 FIX message files to confirm inbound messages match outbound messages. 2) Work with Production Support if necessary.