APP23 Recovery (APP01/APP03/APP02/APP04)

 

                APP23 Purpose:

The APP23 process group is comprised of four separate application service types: APP01, APP02, APP03, APP04.

Together, these service types read inbound market data distributed by SITE 1 and SITE 2 and forward it to APP01s and readers for database loaders.

 

APP01 Processes read SITE 1 quote data over multicast and send it to APP01s and MESSAGING readers.

APP03 Processes read SITE 2 quote data over multicast and send it to APP01s and MESSAGING readers.

APP01 and APP03 processes also calculate Best Bid Offer Duration values to be sent to MESSAGINGreaders.

 

APP02 Processes read SITE 1 lastsale data over multicast and send it to APP01s and MESSAGING readers.

APP04 Processes read SITE 2 lastsale data over multicast and send it to APP01s and MESSAGING readers.

 

Use the following hyperlinks to jump to the desired section of APP23 documentation:

APP23_Recovery_Considerations

APP23_NTM_Control_Commands

APP23_Troubleshooting_Table

APP23_Monitoring_Considerations

 

 

APP23 Recovery Considerations:

                                Stopping/Restart Processes:

-          Use NTM Control Utility - Service Control - Process Controller to stop/restart processes.

-          Use APP23 nodes only when moving between nodes.  (APP23 interfaces must be configured/enabled.)

-          APP23 processes do not move between data centers.  (Only “alternate” nodes are defined in service tables.)

 

-          When stopping/restarting APP23 instances:

1)      Stop/Start “A” series and “B” series separately so as to avoid causing suspending trading in APP01s.

 

 

 

 

APP23 NTM Control Commands:

Enabling/Disabling Channel Readers:

-          Use NTM Control Utility – Service Control - APP23 - Control Multicast Readers.

-          User may have to use REFRESH button multiple times to see all APP23 processes/channels.

-          Select (and highlight) desired channels and right click to see and select desired options, including:

1)      Start Primary Reader

2)      Start Alternate Reader

3)      Start Both Readers

4)      Stop Primary Reader

5)      Stop Alternate Reader

6)      Stop Both Readers

 

Flush Message Queue:

-          Use NTM Control Utility – Service Control - APP23 – Flush Message Queue.

-          User may have to use REFRESH button multiple times to see all APP23 processes/channels.

-          Select (and highlight) desired channels and right click to see and select Flush Internal Msg Queue.

 

Reset Multicast Sequence Number:

-          Use NTM Control Utility – Service Control - APP23 – Reset Multicast Sequence Number.

-          User may have to use REFRESH button multiple times to see all APP23 processes/channels.

-         Select (and highlight) desired channels and right click to see and select Reset Multicast Sequence Number.

 

 

APP23 Troubleshooting Table:

APP23 Symptom

ImpaAPP02

Response

SITE 2 moves to DR site (CRITICAL)

                       

Evidenced by:

-          Stats monitor shows zero SITE 2 primary data received in both data centers and zero SITE 2 alternate data received in DC02 – but SITE 2 alternate data processed in DC01.

 

For APP03:

-APP01 trading without quote related market data for SITE 2 issues in DC02 only.

-Post trade processing will not include all quote related market data.

 

For APP03:

-APP01 trading without lastsale related market data for SITE 2 issues in DC02 only.

-Post trade processing will not include all lastsale related market data.

 

See Generalized Recovery Scenarios for more:

Trading must be halted in affected issues if problem goes on too long.

 

1)      Refer to:

 Generalized Event -  Cannot Process Quotes From SIP.docx

And

 Generalized Event -  Cannot Process Trades From SIP.docx

 Generalized Recovery Scenarios.

 

For APP03:

1)      Production Support must replace APPCFG APP23 APP03_Services file with SITE 2 DR version.

2)      APP03 Services in DC02 must be stopped/restarted.

-          See  APP23_Recovery_Considerations

 

For APP04:

1)      Production Support must replace APPCFG APP23 APP04_Services file with SITE 2 DR version.

2)      APP04 Services in DC02 must be stopped/restarted.

-          See  APP23_Recovery_Considerations

 

For all:

1)      Notify management.

2)      If trading was halted, resume trading when stable and notify industry.

 

No Multicast Data Received (CRITICAL) :

 

-          BOTH Primary and Alternate Channels.

-          BOTH “A” and “B” series of Processes.

 

Evidenced by:

-          In APP23 Stats, zero values seen in rates columns.

For APP01/APP03:

-APP01 WILL be trading without quote related market data.

-Post trade processing WILL not include all quote related market data.

 

For APP02/APP04:

-APP01 WILL be trading without lastsale related market data.

-Post trade processing WILL not include all lastsale related market data.

 

See Generalized Recovery Scenarios for more:

Trading must be halted in affected issues if problem goes on too long.

 

1)      Refer to:

 Generalized Event -  Cannot Process Quotes From SIP.docx

And

 Generalized Event -  Cannot Process Trades From SIP.docx

 Generalized Recovery Scenarios.

 

2)      Confirm scope of impaAPP02.

In Stats Monitor:

-          Are problems specific to:

-          SITE 1 and/or SITE 2

-          DC01 and/or DC02

-          Servers? Processes?  Channels?

3)      Notify management.

4)      Determine corrective actions.

5)      If trading was halted, resume trading when stable and notify industry.

 

No Multicast Data Received (NON-CRITICAL):

 

-           EITHER Primary or Alternate Channel.

-          ONLY “A” or “B” series of Processes.

 

Evidenced by:

-          In APP23 Stats, zero values seen in rates columns.

 

For APP01/APP03:

-APP01 MAY be trading without quote related market data.

-Post trade processing MAY not include all quote related market data.

 

For APP02/APP04:

-APP01 MAY be trading without lastsale related market data.

-Post trade processing MAY not include all lastsale related market data.

 

1)      Confirm scope of impaAPP02.

In Stats Monitor:

-          Are problems specific to:

-          SITE 1 and/or SITE 2

-          DC01 and/or DC02

-          Servers? Processes?  Channels?

2)      Notify management.

3)      Determine corrective actions.

 

Sequence gaps reported (NON-CRITICAL)

 

Evidenced by:

-          In EMT, sequence gaps reported

 

Same as No Multicast Data Received (NON-CRITICAL) symptom.

1)      Same as No Multicast Data Received (NON-CRTICAL) symptom.

2)      Notify management only in critical situations.

Dupes reported (NON-CRITICAL)

 

Evidenced by:

-          In EMT, dupes reported

 

Same as No Multicast Data Received (NON-CRITICAL) symptom.

 

1)      Same as No Multicast Data Received (NON-CRTICAL) symptom.

2)      Notify management only in critical situations.

Market Wide Circuit Breaker (NON-CRITICAL)

 

Evidenced by:

-          In EMT, Process reports MWCB messages

 

Listing Exchanges will Halt Trading in their issues for 15 minutes, and Resume when appropriate.

 

NOTE:  currently has no exclusively listed issues. 

1)      Halt trading in all  exclusive issues.

2)      Confirm MEs halt trading accordingly and resume trading accordingly in all stocks.

-          EMT/ER may be useful.

-          NTM Control Utility ME Service Controls “Get Issues Open” and “Get Issues Not Open” may also help.

3)      Notify management.

 

                           


 

APP23 Monitoring Considerations:

Stats Monitors:
APP23 from SIP Queues (DC01 and DC02)

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from SITE 2 and SITE 1 SIPs to APP01s and Market Data Database Loaders via  Market Data Processors.

Monitor shows connection status between APP23 process and SIP primary/alternate channels as well as processing statistics of each process.

PROD MENU:
Market Data Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Primary Connection,
- Alt Connection,
- Primary Rate,
- Alt Rate,
- Prim Total Msgs,
- Alt total msgs

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Primary Channel and/or Alt Channel are not ENABLED.

Processes are not reading multicast SIP data.
1) Use NTM Control Utility APP23 Service Controls to control multicast readers.
2) Call Production Control if needed.

Primary Rate and/or Alt Rate show sustained rate of zero during trading hours.

Multicast feed is not being processed as expected.
1) Work with SIP and Technical Services if necessary.

Primary Total Msgs and Alt Total Msgs are not showing (relatively) the same numbers of messages processed per process. 

Multicast feed is not being processed as expected.
1) Work with SIP and Technical Services if necessary.

 

 

Stats Monitors:
APP23 to  App Queues

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from SITE 2 and SITE 1 SIPs to APP01s and Market Data Database Loaders via  Market Data Processors.

Monitor shows 29 west to APP01 processing statistics for each process, including queues.

PROD MENU:
Market Data Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- nMsgQueSize,
- InMsgRate

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

nMsgQueSize is non-zero value and not reducing as expected, or InMsgRate is not increasing as expected.

Receiving process may not be up and/or there are 29west delivery issues.
1) Check status of receiving processes
2) Involve Production Support if necessary.

 

 

Stats Monitors:
APP23 to ACTR IPC Connect Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from SITE 2 and SITE 1 SIPs to APP01s and Market Data Database Loaders via  Market Data Processors.

Monitor shows connection status between  (APP23) applications and Instrument System Activity Reader.

PROD MENU:
Market Data Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Status,
- Write Queue

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Not all APP23 or ACTR processes are displayed as expected.

APP23 or ACTR Service has not been started or hasn't processed any messages since monitor has been started.
1) Check status of service.
2) Check ACTR xml data files.

Status is not CONNECTED.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process.
2) Notify Production Support if issues.

Queue size is non-zero value and not decreasing as expected.

Messages cannot be sent from source to destination unless IPC channel is connected.
1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process.
2) Notify Production Support if issues.

 

 

Stats Monitors:
29West APP23 QUOTES LBM Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate inbound quote processing and database loading.

Monitor shows 29 West statistics (by topic) for APP23s (recieving download messages from APP01s and sending to Market Data Loaders, and MESSAGINGand MESSAGING database loading processes, receiving from APP23 processes. 

PROD MENU:
29 West LBM Monitor Menu

To Exit:
Close Window

- Color of data in columns
- ContextName,
- Service,
- Topic_name,
- Type,
- Rate,
- Persistence,
- MsgCount

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

 

 

 

Rate and/or MsgCount values are not incrementing as expected.

Market Data may not be being processed or loaded into database as expected.
1) Check APP01/APP03 log files and BBOD/QMTR Binary data files to confirm inbound messages received and loaded into database.
2) Call Production Support if necessary.

 

 

 

By matching topic_name, Rcv MsgCount in this monitor does not match total Src MsgCount values.

Market Data may not be being processed or loaded into database as expected.
1) Check APP01/APP03 log files and BBOD/QMTR Binary data files to confirm inbound messages received and loaded into database.
2) Call Production Support if necessary.

 

 

Stats Monitors:
29West APP23 QUOTES LBM RCVRM Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate inbound quote processing and database loading.

Monitor shows 29 West "receiving" statistics (by sender, and topic) for APP23s (receiving download messages from APP01s) and MESSAGINGand MESSAGING Database Loading  processes, receiving from APP23 processes.

PROD MENU:
29 West LBM Monitor Menu

To Exit:
Close Window

- Color of data in columns
- ContextName,
- Service,
- Topic_name,
- Lost-Recovered,
- Lost-Unrecovered-Txm,
- Lost-unrecovered-tmo,
-Msgs_rcved

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

 

 

 

Msgs_rcved values are not incrementing as expected.

Market Data may not be being processed or loaded into database as expected.
1) Check APP01/APP03 log files and BBOD/QMTR Binary data files to confirm inbound messages received and loaded into database.
2) Call Production Support if necessary.

 

 

 

Lost-unrecovered values are non-zero.

NOTE: Lost-recovered values may also indicate problems, but that 29west auto-recovered lost messages.

Market Data may not be being processed or loaded into database as expected.
1) Check APP01/APP03 log files and BBOD/QMTR Binary data files to confirm inbound messages received and loaded into database.
2) Call Production Support if necessary.

 

 

Stats Monitors:
29West APP23 LASTSALES LBM Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate inbound lastsale processing and database loading.

Monitor shows 29 West statistics (by topic) for Market Data Processors (recieving download messages from APP01s and sending to Market Data Loaders, and Last Sale Montage database loading processes, receiving from Market Data Processor processes. 

PROD MENU:
29 West LBM Monitor Menu

To Exit:
Close Window

- Color of data in columns
- ContextName,
- Service,
- Topic_name,
- Type,
- Rate,
- Persistence,
- MsgCount

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

 

 

 

Rate and/or MsgCount values are not incrementing as expected.

Market Data may not be being processed or loaded into database as expected.
1) Check APP02/APP04 log files and LSMR Binary data files to confirm inbound messages received and loaded into database.
2) Call Production Support if necessary.

 

 

 

By matching topic_name, Rcv MsgCount in this monitor does not match total Src MsgCount values.

Market Data may not be being processed or loaded into database as expected.
1) Check APP02/APP04 log files and LSMR Binary data files to confirm inbound messages received and loaded into database.
2) Call Production Support if necessary.

 

Stats Monitors:
29West APP23 LASTSALES LBM RCVRM Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate inbound lastsale processing and database loading.

Monitor shows 29 West "receiving" statistics (by sender, and topic) for Market Data Processors (receiving download messages from APP01s) and Last Sale Montage Database Loading  processes, receiving from Market Data Processor processes.

PROD MENU:
29 West LBM Monitor Menu

To Exit:
Close Window

- Color of data in columns
- ContextName,
- Service,
- Topic_name,
- Lost-Recovered,
- Lost-Unrecovered-Txm,
- Lost-unrecovered-tmo,
-Msgs_rcved

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

 

 

 

Msgs_rcved values are not incrementing as expected.

Market Data may not be being processed or loaded into database as expected.
1) Check APP02/APP04 log files and LSMR Binary data files to confirm inbound messages received and loaded into database.
2) Call Production Support if necessary.

 

 

 

Lost-unrecovered values are non-zero.

NOTE: Lost-recovered values may also indicate problems, but that 29west auto-recovered lost messages.

Market Data may not be being processed or loaded into database as expected.
1) Check APP02/APP04 log files and LSMR Binary data files to confirm inbound messages received and loaded into database.
2) Call Production Support if necessary.