Order Management

How it works

Client strategies must manage the order life-time using a single order_id. This is a unique integer used to identify or reference the same original order.

In particular, these update events will contain the same order_id

Also, the following order actions will use the same order_id

The gateway will communicate the highest previously seen order_id during the download phase

The client is then simply expected to use max_order_id + 1 for the next order_id.

Pros and Cons

Client

Pros

  • Makes it easy (and efficient) to reference local order state using a simple integer lookup

  • Does not need to know about exchange-specific conventions for managing order ID’s

  • Strategy implementation is easily transferrable and can be used with any gateway

Cons

  • Can not pipeline order actions (must wait for an ack before a subsequent action can be requested)

Gateway

Pros

  • Can freely use the exchange’s client order ID to encode additional information about e.g. originating strategy (user_id) and its local order reference (order_id)

  • Can decode the value from exchange’s client order ID and route ack, order updates, and fills to the originating strategy without using a database persist such information

  • The client order ID is not the only pass-through field which provides the gateway an opportunity to avoid database, sometimes other order creation fields can provide this functionality

  • Can “multiplex” requests and updates from/to any number of strategies using a single account

Cons

  • This only works when the exchange allows free-form client order ID’s (some exchanges only allow very specific ID’s such as UUID)

  • There is a small risk of re-using a client’s order_id’s if the exchange only supports the download of working orders and the gateway is restarted. (There is a requirement for the client to use max(local_max_order_id, DownloadEnd.max_order_id) which would prevent re-use as long as the strategy is not restarted. However, the gateway can not enforce this.)

Note

The Roq gateways have more functionality than traditional order routing gateways. In particular (for the context being discussed here), they are designed to support

  • object caching so clients can reconnect at any time and quickly catch up to current state with both market and account data, then simply join the live stream of updates

  • multiplexing orders to a single account originated from any number of strategies

Batching

Some exchanges support a batch of multiple order requests.

Benefits may include

  • More relaxed rate-limiting

  • Atomic processing of order requests

The client dispatcher (roq::client::Dispatcher) accepts an is_last boolean flag.

The default is true as an instruction for the gateway to immediately process the request.

However, setting this flag to false will allow the gateway to optionally withhold the request and await another later request having the flag set to true.

When the gateway has received all requests, it may elect to send all (or some of) the requests using the most optimal exchange API.

Warning

There is currently no generic implementation to support gateway batching. This is partially due to exchange API’s often being very custom and partially due to the very complex error handling.

In short

  • Do not assume batching is implemented for the gateway you’re using

  • Do not assume requests are withheld by the gateway if you choose is_last=true

  • Always read the reference documentation carefully to see if there is support

Routing

When creating a new order (using roq::CreateOrder), a routing_id string may be supplied. This field is pass-through and meant to capture an original order ID when routing orders from another system. If present, the gateway may encode the routing ID in the client order ID (or another free-form field). All update events (see list above) will have this field present making it possible to communicate the original order ID back to the other system.

What it doesn’t solve

  • The introduction of further client order ID’s generated by the originating system when modifying order attributes

  • Pipelining, i.e. when the originating system sends multiple order modifications without waiting for ack

ClOrdId

  • When free-form text:

    • Roq will use 20 chars (base64 encoded) to carry important information about user_id, order_id and a nanosecond precision timestamp. This information is used to enable automatic download and recovery

    • The routing_id field will be appended. This may be an issue if (20 + len(routing_id)) exceeds any limit the exchange may have

  • Otherwise following exchange conventions with no option to have automatic download and recovery. For this case it is recommended to auto-cancel orders on exchange following a broken connection

routing_id

  • We do not currently allow ModifyOrder to supply a new routing_id. There are multiple reasons

    • There is an inherent race between Ack and ExecutionReport, especially when the underlying communication happens on multiple connections, but also if fills are reported while the modification request is in-flight

    • Managing a chain of Ack’s (and the life-time of each routing_id) is very dependent on exchange sending back all acks, in the right order

order_template

  • The order_template is not encoded in any field sent to the exchange OrderUpdate can therefore only inform of this field until the gateway is restarted or download is initiated

streams

  • Orders are cached per stream – but not yet cleared

Download (by gateway)

When a gateway stream, supporting orders, experiences a disconnect, it will immediately try to reconnect. When connected, the download phase normally begins. This is what happens

  • All orders (known to the gateway) are internally marked as stale

  • Working orders are downloaded from the exchange

  • Orders that can be downloaded will be communicated with update_type=SNAPSHOT

  • When download completes, and before the stream becomes ready, remaining orders will be communicated with update_type=STALE

  • Stale will only be communicated once, snapshot may be communicated following every reconnect.

Note

During a disconnect, orders could be potentially be (partially or completely) filled or auto-canceled by the exchange. The procedure mentioned above helps us identify

  • current status for working orders, and

  • those orders which are in an unknown state

Since we can’t know if an order was completed or canceled during disconnect, it is necessary to use other means to synchronize e.g. positions. For that there’s potentially PositionUdpate and FundsUpdate. A final solution might be to require human intervention when seeing stale order updates.