Release 0.8.6#

2022-07-18

Gateways#

KuCoin Spot:

  • New time fields were announced

KuCoin Futures:

  • New flags: --api (v1 (default) or v2)

Note

This is currently work in progress awaiting end-point availability for the new test environment.

API#

The MBPUpdate struct now supports update action (new/change/delete).

Note

The field number_of_orders has been changed from uint32_t to uint16_t such that it is possible to naturally align this structure with the cache-line size. There are no adverse effects of doing this because the field was not really used until now. The Flatbuffers schema has been updated such that event-logs remain backwards compatible.

The client::EventLogReader now exposes metadata.

Several methods on cache::MarketByPrice has been changed to return std::span<MBPUpdate>.

MarketByPrice#

Gateways will now populate the update_action and price_level fields of the MBPUpdate struct. This allows clients to better integrate with third-party software solutions which manage book updates by position.

Example

market_by_price_update={
  stream_id=4,
  exchange="deribit",
  symbol="ETH-PERPETUAL",
  bids=[],
  asks=[
    {
      price=1794.5,
      quantity=19050,
      implied_quantity=nan,
      number_of_orders=0,
      update_action=CHANGE,
      price_level=15
    },{
      price=1794.75,
      quantity=21974,
      implied_quantity=nan,
      number_of_orders=0,
      update_action=NEW,
      price_level=20
    }
  ],
  update_type=INCREMENTAL,
  exchange_time_utc=1654841111363000000ns,
  exchange_sequence=0,
  price_decimals=_3,
  quantity_decimals=_0,
  max_depth=0,
  checksum=0
}

Event Logs#

Metadata is now exposed via the API interface provided by client::EventLogReader. This makes it easier to export descriptive information to third-party database solutions.

Furthermore, a new adapter library has been made available allowing easier integration with third-party database solutions. (See section below discussing open sourced projects.)

Tools#

The roq-dump tool has been changed

  • Prefix all events with the origin_crate_time_utc timestamp.

  • Use color highlighting (when terminal supports)

  • Drop support for --summary.

Example

1655215881111196396ns gateway_settings={supports=REFERENCE_DATA|MARKET_STATUS|TOP_OF_BOOK|MARKET_BY_PRICE|TRADE_SUMMARY|STATISTICS|CREATE_ORDER|MODIFY_ORDER|CANCEL_ORDER|ORDER_ACK|ORDER|TRADE|POSITION|FUNDS, mbp_max_depth=0, mbp_tick_size_multiplier=0.1, mbp_min_trade_vol_multiplier=nan, mbp_allow_remove_non_existing=true, mbp_allow_price_inversion=true, oms_download_has_state=false, oms_download_has_routing_id=false, oms_request_id_type=BASE64}
1655215881306805542ns stream_status={stream_id=4, account="", supports=REFERENCE_DATA|MARKET_BY_PRICE|TRADE_SUMMARY, transport=TCP, protocol=FIX, encoding=FIX, priority=PRIMARY, connection_status=LOGIN_SENT}
1655215881306805542ns gateway_status={account="", supported=REFERENCE_DATA|MARKET_BY_PRICE|TRADE_SUMMARY, available=, unavailable=REFERENCE_DATA|MARKET_BY_PRICE|TRADE_SUMMARY}
1655215881306944445ns gateway_status={account="A1", supported=CREATE_ORDER|MODIFY_ORDER|CANCEL_ORDER|ORDER_ACK|ORDER|TRADE|POSITION, available=, unavailable=CREATE_ORDER|MODIFY_ORDER|CANCEL_ORDER|ORDER_ACK|ORDER|TRADE|POSITION}
1655215881344797226ns stream_status={stream_id=4, account="", supports=REFERENCE_DATA|MARKET_BY_PRICE|TRADE_SUMMARY, transport=TCP, protocol=FIX, encoding=FIX, priority=PRIMARY, connection_status=DOWNLOADING}
1655215881356409974ns stream_status={stream_id=3, account="", supports=MARKET_STATUS|TOP_OF_BOOK, transport=TCP, protocol=WS, encoding=JSON, priority=PRIMARY, connection_status=DOWNLOADING}
[...]

A new tool roq-summary has been introduced.

Example

metadata:

┌──────────────────┬────────────────────────────────────┐
│version           │                               0.8.5│
│application       │                         roq-deribit│
│name              │                             deribit│
├──────────────────┼────────────────────────────────────┤
│hostname          │                           VQ-WRK-01│
│username          │                             thraneh│
│session id        │0d3163eb-8f6c-4faa-afb0-8df8f030b1c2│
├──────────────────┼────────────────────────────────────┤
│encoding          │                         FLATBUFFERS│
│compression method│                              BROTLI│
│compression level │                                   4│
└──────────────────┴────────────────────────────────────┘

header:

┌────────────┬──────────────────────────────┐
│begin       │2022-06-12T14:01:43.669509626Z│
│end         │2022-06-12T14:02:59.498907935Z│
│duration    │                  0d 0h 1m 15s│
├────────────┼──────────────────────────────┤
│uncompressed│                     0.023 GiB│
│compressed  │                     0.003 GiB│
│ratio       │                           14%│
├────────────┼──────────────────────────────┤
│max(seqno)85,535│
└────────────┴──────────────────────────────┘

users:

  []

accounts:

  ["A1"]

exchanges:

  ["deribit"]

symbols:

  ["BTC-17JUN22","BTC-24JUN22","BTC-29JUL22","BTC-30DEC22","BTC-30SEP22","BTC-31MAR23","BTC-DERIBIT-INDEX","BTC-PERPETUAL","BTC-USDC-DERIBIT-INDEX","BTC_USDC-PERPETUAL","ETH-17JUN22","ETH-24JUN22","ETH-29JUL22","ETH-30DEC22","ETH-30SEP22","ETH-31MAR23","ETH-DERIBIT-INDEX","ETH-PERPETUAL","ETH-USDC-DERIBIT-INDEX","ETH_USDC-PERPETUAL"]

data:

┌────────────┬──────────────────────────────┐
│begin       │2022-06-12T14:01:43.669509626Z│
│end         │2022-06-12T14:02:59.498907935Z│
│duration    │                  0d 0h 1m 15s│
├────────────┼──────────────────────────────┤
│uncompressed│                     0.023 GiB│
│compressed  │                     0.003 GiB│
│ratio       │                           14%│
└────────────┴──────────────────────────────┘

messages:

┌──────────────────────┬──────┬──────────┬─────────┐
│type                  │ count│     bytes│avg. size│
├──────────────────────┼──────┼──────────┼─────────┤
│gateway_settings      │     100.0│
│stream_status         │     81,704│    213.0│
│external_latency      │    297,896│    272.3│
│rate_limit_trigger    │      │          │         │
│gateway_status        │     81,368│    171.0│
│reference_data        │    207,512│    375.6│
│market_status         │    162,864│    179.0│
│top_of_book           │ 7,105│ 1,971,592│    277.5│
│market_by_price_update│74,953│22,248,304│    296.8│
│market_by_order_update│      │          │         │
│trade_summary         │   16547,424│    287.4│
│statistics_update     │ 2,431│   730,216│    300.4│
│create_order          │      │          │         │
│modify_order          │      │          │         │
│cancel_order          │      │          │         │
│cancel_all_orders     │      │          │         │
│order_ack             │      │          │         │
│order_update          │      │          │         │
│trade_update          │      │          │         │
│position_update       │      │          │         │
│funds_update          │      │          │         │
│custom_metrics_update │      │          │         │
│custom_message        │      │          │         │
├──────────────────────┼──────┼──────────┼─────────┤
│total                 │84,736│25,018,880│    295.3│
└──────────────────────┴──────┴──────────┴─────────┘

Open Source#

New projects have been added to GitHub

  • The ClickHouse adapter will monitor one or more event-log directories and export events to a ClickHouse database. The project can be found here.

  • The ZeroMQ adapter will connect to one or more gateways and publish events to a ZeroMQ network. The project can be found here.

  • The Redis adapter will connect to one or more gateways and publish events to a Redis memory cache. The project can be found here.

They reflect common use-cases and it is easy to copy the project and customize to personal preferences.

It is the hope that these projects will see contributions from the open-source community.

Other Work#

Socket and web protocol code have been refactored to make it possible to experiment with various I/O solutions, e.g.

  • Linux’ io_uring

  • Xlinx’s kernel-bypass solutions

Other long-term benefits

  • An I/O context can be made available to client libraries thereby making it possible to implement custom network code with the same benefits just mentioned.

  • These changes are done to to ultimately open source certain binaries, e.g. the JSON bridge.

CHANGELOG#

Milestone on GitHub.

Added#

Changed#

  • New and changed fields (KuCoin) (#248).

  • Externalise an event-loop interface (Core) (#242).

  • Prevent copy/move of Event<> and Trace (API) (#247).

  • Flag to control terminal color output (Logging) (#245).

  • API update (KuCoin Futures) (#228).

  • return std::span<MBPUpdate> from cache::MarketByPrice (API) (#241).

  • Filter the removal of unknown price-levels from MbP updates (Core) (#238).

  • New roq-summary + review roq-dump (Tools) (#237).

  • Support limited-depth MbP (FIX Bridge) (#236).

Fixed#

  • Auto-reconnect after exchange maintenance (Deribit) (#246).

  • ReferenceData required update following exchange removing API fields (Coinbase-PRO) (#243).

  • Using –ws_books_depth=5 was not working (OKX) (#239).

  • MbP snapshot did not contain exchange_time_utc (Gate.io Futures) (#235).

  • MbP snapshot did not contain exchange_time_utc (Deribit) (#234).

Other#