Release 0.7.7

    Posted by
  • Hans Erik Thrane on 2021-09-20

This is a minor release including bug fixes and new features

  • FTX order book validation has pushed for moving more low-level order book features to the API for the benefit of all use-cases. All gateways will now automatically resubscribe those symbols where order books have been detected to be in a "bad state".
  • Multiple strategies can now leverage pub/sub to communicate between themselves.
  • Market making strategies can now monitor when the gateway's rate-limiter has been triggered and algorithms can therefore more efficiently back off by not sending order requests which would anyway be rejected.

Market by Price

There are various scenarios which could lead to an internally cached order book ending up in a "bad state"

  • Dropped messages (exchange or CDN, since all currently supported exchanges use TCP)
  • Configuration mistakes (gateway)
  • Bugs (exchange or gateway)

"Bad state" can be sometimes be detected by price inversion (if unexpected by the gateway) or by other means, e.g. detecting dropped sequence numbers or by validating the current snapshot of an order book.

What is new is that the gateways will auto-resubscribe the relevant symbol(s) whenever "bad state" has been detected.

On the client side, there is the GatewaySettings event which includes the required information to correctly mirror and maintain order books.

Note

The DepthBuilder interface has been updated to accept the GatewaySettings event. Applying this event is required for the client to maintain an order book that will exactly mirror the order book maintained by the gateway.

Important

Inversion and rounding errors may result from not applying GatewaySettings!

This change was prompted by FTX potentially dropping messages during fast market conditions.

Note

FTX requires the client side (the gateway) to use a very expensive checksum validation (expectation is 10-20 microseconds per update).

Validation is only enabled by setting the --ws_order_book_checksum_every command-line flag to a positive value. This is used to validate on every N'th update (of the symbol).

Custom Metrics

Clients can now broadcast custom calculations for other clients to consume or simply to be collected in the event-logs for offline analysis.

Caching and broadcasting is currently managed by the gateways. However, this is not ideal because a single client can now potentially spam a gateway thereby causing unnecessary latency, or worse. A dedicated pub/sub service will be introduced at a later point.

Rate Limit Triggers

Trading strategies should ideally back off when the gateway's rate-limiter has been triggered.

The immediate response is for the gateway to reject via OrderAck to the client who triggered the rate-limiter.

However, there are other concerns

  • The OrderAck message contains no information about when the the ban expires.
  • Several clients can potentially be affected by a ban, e.g. when there is a globally configured rate-limiter. This can obviously not be captured by the OrderAck event since it's only sent to a single client.

The solution is to broadcast the RateLimitTrigger event to all connected clients. Each client must then decide if it is impacted by the ban.

CHANGELOG

Milestone on GitHub.

Changes

  • Allow price inversion (Gemini) (#67).
  • New enum value: "delisted" (JSON) (Coinbase PRO) (#66).
  • Subscribe to reference data changes (FIX) (Deribit) (#65).
  • Resubscribe MarketByPrice when updates have been lost (#31).
  • HTTP request tracking (#55).
  • Added new JSON fields liquidationFee and marketTakeBound (Binance Futures) (#54).
  • string_buffer now has O(1) length() (#53).
  • Move OMS interfaces to API (#51).
  • Publish RateLimitTrigger (#34).

Features

  • Support client publication of custom metrics (#8).
  • New error code (Rate limit exceeded, retry in 1 seconds.) (BitMEX) (#64).
  • Add currency (15) to ExecutionReport (8) (FIX Bridge) (#63).
  • Config file can be used to specify what "zero" means (FIX Bridge) (#62).
  • Option to terminate on order-ack timeout (FIX Bridge) (#60).

Fixed

  • Reject (3) message with "rate_limit_exceeded" (Deribit) (#73).
  • API exceptions should compile with C++14 (#69).
  • Only subscribe products with status "online" (Coinbase PRO) (#68).
  • OrderAck was filtered for missing routing_id (FIX Bridge) (#61).
  • RequestStatus was not managed correctly for timeout scenarios (#59).
  • Order ack handlers were not protected by try-catch (Kraken Futures) (#58).
  • Validate MarketByPrice (FTX) (#56).