APP24 Recovery (Matching Engine)

APP24 Purpose:

APP24 Processes read order messages from APP01s and the APP02.

APP24 Processes read order and trade modification messages from CSI and APP03 (also via APP02).

APP24 Processes read MESSAGING from APPGROUP01 processes and MESSAGING Inquiry messages from APP03.

APP24 Processes send order responses to APP01s and APP02, and drop copies to APP04 processes.

APP24 Processes send quotes, trades and bookfeed messages to APP05 processes.

APP24 Processes also send database loader messages to TXR for database loading.

Use the following hyperlinks to jump to the desired section of APP24 documentation:

APP24_Recovery_Considerations

APP24_NTM_Control_Commands

APP24_Troubleshooting_Table

APP24_Monitoring_Considerations

APP24 Recovery Considerations:

Stopping/Restart Processes:

- Use NTM Control Utility Service Control Process Controller to stop/restart processes.

- Use APP24 nodes only when moving between nodes. Bookfeed interfaces must be configured.

NOTE: Existing bug exists when moving TXR processes from one node to another in an orderly fashion whereas APP24 messages may get lost as a result. If this occurs, records will need to be extracted from APP24_LBM logs and provided to PTT developers to try and recreate data in the database. This is a very cumbersoAPP24 operation and should be avoided if at all possible.

- When stopping/restarting APP24 instances:

1) Stop APP15 testing for the APP24 instance involved before stopping the ME

2) Stop the APP24 instance

3) Determine if associated TXR and DLAPP24 instances used by the affected APP24 instance also require recovery.

a) TXR and DLMEs are configured to run on different nodes from the APP24 (unless they are for test instances of ME).

b) If the APP24 is moving to a host within the saAPP24 data center, it is possible that the TXR/DLMEs do not need to move.

c) If the APP24 is moving to a host in a different data center, then TXR and DLMEs should move as well.

4) Confirm that the associated Database loader is up to date IN THE DATA CENTER that the APP24 will be restarted in.

a) If the APP24 is restarted without the database up to date with all most recent transactions, soAPP24 order processing may not be handled as expected.

b) If there are database loader rejects outstanding, they must be replayed before APP24 restarts. Use database loader reject procedures to replay this data.

c) It may be necessary to move the Database Loader to another node to complete loading this data. See step 5.

5) If the associated TXR and DLAPP24 instances are NOT moving to an alternate node, skip to step 6.

a) If moving TXR and DLAPP24 process:

i) Stop TXR process on current node.

ii) Stop DLAPP24 process on current node.

iii) If NOT moving the Database Loader files to a new node, skip to step 5-iv.

(a) Database Loader files need only be moved if “pre-move” database loading can’t be completed without doing so.

(b) If moving Database Loader files:

Copy :\\data\DL*.log, DL*.pos, DL*rejects.log files (created on day) for each Database Loader process moving

to the alternate node.

iv) Start TXR process on new node.

v) Start DLAPP24 process on new node.

6) Start the APP24 instance and confirm it starts without errors.

7) Start APP15 testing for the APP24 instance involved and confirm testing is conducted without errors.

8) Confirm APP15 orders and trades before and after restart can be queried in CSI to insure that TXR communications are as expected.

(continue procedure on next page)

9) ResuAPP24 trading in all stocks for affected ME.

a) Trading will be halted by default if APP24 startup occurs after any stock’s primary session begins.

b) Do not OPEN stocks. RESUME.

c) Check with Control Room staff to see if any issues need to be re-halted via a halt.

10) Determine if associated DLMP processes need to recovered.

a) DLMP (Performance Loaders) are coded such that they must run on the saAPP24 nodes as the ME.

b) If the APP24 has moved to another node, the DLMP must move as well.

11) If DLMP processes will move:

a) Stop associated DLMP processes

b) Start associated DLMP processes.

APP24 NTM Control Commands:

Halting Issues by ME/Stock:

- Use NTM Control Utility – Service Control – APP24 - options to Halt Issues.

- Select (and highlight) desired matching engines and right click to see and select desired options.

1) The maintenance client instrument queries can inform a user which MEs are assigned to which stocks, if needed.

2) The maintenance client instrument queries can also be used to easily store this information in an excel spreadsheet if needed.

3) Halting issues results in Halts, regardless of what the listing market centers are doing.

- From the APP24 options, the following can be used to halt issues:

1) Halt All Issues

2) Halt APP15 Issues

3) Halt Issue (if wanting to halt an individual stock versus a group of stocks)

4) Halt Issue by Market Center (need to know Market Center code)

5) Halt Exclusive Issue

Resuming/Opening Issues by ME/Stock:

- Use NTM Control Utility – Service Control – APP24 options to ResuAPP24 or Open Issues.

- Select (and highlight) desired matching engines and right click to see and select desired options.

1) The maintenance client instrument queries can inform a user which MEs are assigned to which stocks, if needed.

2) The maintenance client instrument queries can also be used to easily store this information in an excel spreadsheet if needed.

3) Resuming issues results in setting the issue’s trading status to match the last received status by APPGROUP01 processes.

4) Opening issues results in Openings, regardless of what the listing market centers are doing or what is last known by APPGROUP01.

- From the APP24 options, the following can be used to resuAPP24 issues:

1) ResuAPP24 All Issues

2) ResuAPP24 APP15 Issues

3) ResuAPP24 Issue (if wanting to resuAPP24 an individual stock versus a group of stocks)

4) ResuAPP24 Exclusive Issue

- From the APP24 options, the following can be used to open issues:

1) Open All Issues

2) Open Issue (if wanting to open an individual stock versus a group of stocks)

3) Open Issue by Market Center (need to know Market Center code)

LULD Pausing/Resuming Issues by ME:

- Use NTM Control Utility – Service Control – APP24 - options to LULD Trading Pause/Resume.

- Select (and highlight) desired matching engines and right click to see and select desired options.

- Under LULD options, user will be required to enter Issue Symbol to Pause/Resume.

Getting Issues Open/Issues Not Open by ME:

- Use NTM Control Utility – Service Control – APP24 - options to Get Issues Open/Not Open.

- Select (and highlight) desired matching engines and right click to see and select desired options.

Resend quotes by ME:

- Use NTM Control Utility – Service Control – APP24 – Resend APP24 Quotes.

- Select (and highlight) desired matching engines and right click to see and select desired options.

Set Quote Conditions or Zero Quotes by ME:

- Use NTM Control Utility – Service Control – APP05 (Options by ME) – to:

1) Zero Quote by ME, Set Quote Condition Auto, Set Quote Condition Manual.

- Select (and highlight) desired matching engines and right click to see and select desired options.

Enable/Disable MKT IOC by ME:

- Use NTM Control Utility – Service Control – APP24 – Enable/Disable Market IOC.

- Select (and highlight) desired matching engines and right click to see and select desired options.

Forcibly cancel orders (by either ME, order sending firm, or stock) by ME:

- Use NTM Control Utility – Service Control – APP24 - options to Forcibly Cancel All Orders, Order for Firm or Orders for Issue.

- Select (and highlight) desired matching engines and right click to see and select desired options.

Enable/Disable 29West LBM Stats by ME:

- Use NTM Control Utility – Service Control – APP24 - options to Enable/Disable 29West Stats.

- Select (and highlight) desired matching engines and right click to see and select desired options.

APP24 Troubleshooting Table:

APP24 Symptom	Impacts	Response
Node Crashes Evidenced by: - In EMT, applications report lost communications to MEs - In Solarwinds (and outlook), node and processes will be reported down.	is not trading issues as expected. Order sending firms will not be receiving responses/updates and industry will not be getting quotes or lastsales. Refer to: Server Recovery - Matching Engine.docx Server Specific Recoveries.	1) Refer to: Server Recovery - Matching Engine.docx Server Specific Recoveries. 2) Confirm APP05 zeroed quotes for all affected MEs 3) Notify Management. 4) Work with Tech Services to confirm server status. 5) If node is not to be used and processes need to be moved, determine if database loading has been completed, and complete per recovery procedures. 6) Restart affected MEs. See APP24_Recovery_Considerations 7) Trading will be halted by default upon restart. 8) Notify industry after trading is resumed.
APP24 hangs (suspends processing) Evidenced by: - In APP24 Thread stats, non-zero values are seen in queues and not reducing.	is not trading issues as expected. Order sending firms will not be receiving responses/updates and industry will not be getting quotes or lastsales. Refer to: Generalized Event - Experiencing Issues in Order Handling.docx Generalized Recovery Scenarios for more.	1) Stop/Restart affected MEs. See APP24_Recovery_Considerations 2) Notify management. 3) Trading will be halted by default upon restart. 4) Notify industry after trading is resumed.

APP24 Symptom

Impacts

Response

Node Crashes

Evidenced by:

- In EMT, applications report lost communications to MEs

- In Solarwinds (and outlook), node and processes will be reported down.

is not trading issues as expected.

Order sending firms will not be receiving responses/updates and industry will not be getting quotes or lastsales.

Refer to:

Server Recovery - Matching Engine.docx

Server Specific Recoveries.

1) Refer to:

Server Recovery - Matching Engine.docx

Server Specific Recoveries.

2) Confirm APP05 zeroed quotes for all affected MEs

3) Notify Management.

4) Work with Tech Services to confirm server status.

5) If node is not to be used and processes need to be moved, determine if database loading has been completed, and complete per recovery procedures.

6) Restart affected MEs.

See APP24_Recovery_Considerations

7) Trading will be halted by default upon restart.

8) Notify industry after trading is resumed.

APP24 hangs (suspends processing)

Evidenced by:

- In APP24 Thread stats, non-zero values are seen in queues and not reducing.

is not trading issues as expected. Order sending firms will not be receiving responses/updates and industry will not be getting quotes or lastsales.

Refer to:

Generalized Event - Experiencing Issues in Order Handling.docx

Generalized Recovery Scenarios for more.

1) Stop/Restart affected MEs.

See APP24_Recovery_Considerations

2) Notify management.

3) Trading will be halted by default upon restart.

4) Notify industry after trading is resumed.

APP24 Monitoring Considerations:

Stats Monitors: MT APP24 Statistics	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate order and trade processing. Monitor shows matching engine status of order routing and processing statistics.	PROD MENU: Trading Applications Monitoring Menu To Exit: Close Window	- Color of data in columns - in rate, - routing enable, - ors directed connected	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			In_rate and/or msg_in value is zero.	Matching Engine may not be processing as expected. 1) Check APP24 status. 2) Call Production Support if needed.
			Routing_enable and/or Ors_dreicted_connected flags are N.	Matching Engine is not enabled for Outbound Routing. 1) Check status of ORS process. Routing will be disabled if process is not connected. 2) Use NTM Control Utility Matching Engine Service Controls to Enable Reg NMS Routing. 3) Call Production Support if needed.

Stats Monitors: MT APP24 Thread Statistics	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate order and trade processing. Monitor shows matching engine THREAD status and processing statistics.	PROD MENU: Trading Applications Monitoring Menu To Exit: Close Window	- Color of data in columns - data queue, - ctrl que	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
		- Color of data in columns - data queue, - ctrl que	Data_queue and/or Ctrl_que column is non-zero value and not decreasing as expected.	Matching Engine may not be processing as expected. 1) Check APP24 status. 2) APP24 may need to be stopped/restarted. Call Production Support if needed.

Stats Monitors: APP24 IPC Instance Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate order and trade processing. Monitor shows IPC channel processing statistics.	PROD MENU: Trading Applications Monitoring Menu To Exit: Close Window	- Color of data in columns - Service, - Hostname, - Msgs In, - Msgs Out	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Not all APP24 processes are displayed as expected.	APP24 Service has not been started. 1) Check status of service.
			Msgs In and/or Msgs Out are zero when messages are processed.	No messages have been sent/received since that monitor has been started. 1) Check CSI log files. 2) Call Production Support if necessary.

Stats Monitors: APP24 to OMS IPC Connect Queues	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate order and trade processing. Monitor shows IPC channel connectivity status between OMS Bridge and OMS processes.	PROD MENU: Trading Applications Monitoring Menu To Exit: Close Window	- Color of data in columns - Source, - Dest, - Status, - Queue Size, - Msgs Out/Sec	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Not all APP02 or OMS processes are displayed as expected.	OMS Bridge or OMS Service has not been started or hasn't processed any messages since monitor has been started. 1) Check status of service. 2) Check OMS Bridge or OMS log files.
			Status is not CONNECTED.	Messages cannot be sent from source to destination unless IPC channel is connected. 1) Stop/Restart destination process if other processes connecting to the saAPP24 are showing similar issues; Otherwise, stop/restart source process. 2) Notify Production Support if issues.
			Queue size is non-zero value and not decreasing as expected.	Messages cannot be sent from source to destination unless IPC channel is connected. 1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process. 2) Notify Production Support if issues.

Stats Monitors: 29West APP24 LBM Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate order and trade processing. Monitor shows 29 West statistics (by topic) for Matching Engine processes, receiving from APP01, APP02, ORS and MESSAGING Processor processes. Sending to APP01, APPGROUP01 (download messages only), OMS Bridge, ORS, APP05, TRS, and TXR (for database loader and DCSF) processes.	PROD MENU: 29 West LBM Monitor Menu To Exit: Close Window	- Color of data in columns - ContextName, - Service, - Topic_name, - Type, - Rate, - Persistence, - MsgCount	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Statistics are not shown for APP24 process as expected.	Process may not be up, or 29 West Stats have not yet been enabled for process. 1) Check status of process 2) Via NTM Control APP24 Service Controls, enable 29 West stats for process.
			Rate and/or MsgCount values are not incrementing as expected.	Order and/or trade related processing may not be working as expected. 1) Check LBM APP24 log files to confirm 29 West related processing. 2) Call Production Support if necessary.

Stats Monitors: 29West APP24 LBM RCVRM Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate order and trade processing. Monitor shows 29 West "receiving" statistics (by sender, and topic) for Matching Engine processes, receiving from APP01, APP02, ORS and MESSAGING Processor processes.	PROD MENU: 29 West LBM Monitor Menu To Exit: Close Window	- Color of data in columns - ContextName, - Service, - Topic_name, - Lost-Recovered, - Lost-Unrecovered-Txm, - Lost-unrecovered-tmo, -Msgs_rcved	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Statistics are not shown for APP24 process as expected.	Process may not be up, or 29 West Stats have not yet been enabled for process. 1) Check status of process 2) Via NTM Control APP24 Service Controls, enable 29 West stats for process.
			Msgs_rcved values are not incrementing as expected.	Order and/or trade related processing may not be working as expected. 1) Check LBM APP24 log files to confirm 29 West related processing. 2) Call Production Support if necessary.
			Lost-unrecovered values are non-zero. NOTE: Lost-recovered values may also indicate problems, but that 29west auto-recovered lost messages.	Order and/or trade related processing may not be working as expected. 1) Check LBM APP24 log files to confirm 29 West related processing. 2) Call Production Support if necessary.