Release 0.8.6#

2022-07-18

Gateways#

KuCoin Spot:

New time fields were announced

KuCoin Futures:

New flags: --api (v1 (default) or v2)

Note

This is currently work in progress awaiting end-point availability for the new test environment.

API#

The MBPUpdate struct now supports update action (new/change/delete).

Note

The field number_of_orders has been changed from uint32_t to uint16_t such that it is possible to naturally align this structure with the cache-line size. There are no adverse effects of doing this because the field was not really used until now. The Flatbuffers schema has been updated such that event-logs remain backwards compatible.

The client::EventLogReader now exposes metadata.

Several methods on cache::MarketByPrice has been changed to return std::span<MBPUpdate>.

MarketByPrice#

Gateways will now populate the update_action and price_level fields of the MBPUpdate struct. This allows clients to better integrate with third-party software solutions which manage book updates by position.

Example

market_by_price_update={
  stream_id=4,
  exchange="deribit",
  symbol="ETH-PERPETUAL",
  bids=[],
  asks=[
    {
      price=1794.5,
      quantity=19050,
      implied_quantity=nan,
      number_of_orders=0,
      update_action=CHANGE,
      price_level=15
    },{
      price=1794.75,
      quantity=21974,
      implied_quantity=nan,
      number_of_orders=0,
      update_action=NEW,
      price_level=20
    }
  ],
  update_type=INCREMENTAL,
  exchange_time_utc=1654841111363000000ns,
  exchange_sequence=0,
  price_decimals=_3,
  quantity_decimals=_0,
  max_depth=0,
  checksum=0
}

Event Logs#

Metadata is now exposed via the API interface provided by client::EventLogReader. This makes it easier to export descriptive information to third-party database solutions.

Furthermore, a new adapter library has been made available allowing easier integration with third-party database solutions. (See section below discussing open sourced projects.)

Tools#

The roq-dump tool has been changed

Prefix all events with the origin_crate_time_utc timestamp.
Use color highlighting (when terminal supports)
Drop support for --summary.

Example

1655215881111196396ns gateway_settings={supports=REFERENCE_DATA|MARKET_STATUS|TOP_OF_BOOK|MARKET_BY_PRICE|TRADE_SUMMARY|STATISTICS|CREATE_ORDER|MODIFY_ORDER|CANCEL_ORDER|ORDER_ACK|ORDER|TRADE|POSITION|FUNDS, mbp_max_depth=0, mbp_tick_size_multiplier=0.1, mbp_min_trade_vol_multiplier=nan, mbp_allow_remove_non_existing=true, mbp_allow_price_inversion=true, oms_download_has_state=false, oms_download_has_routing_id=false, oms_request_id_type=BASE64}
1655215881306805542ns stream_status={stream_id=4, account="", supports=REFERENCE_DATA|MARKET_BY_PRICE|TRADE_SUMMARY, transport=TCP, protocol=FIX, encoding=FIX, priority=PRIMARY, connection_status=LOGIN_SENT}
1655215881306805542ns gateway_status={account="", supported=REFERENCE_DATA|MARKET_BY_PRICE|TRADE_SUMMARY, available=, unavailable=REFERENCE_DATA|MARKET_BY_PRICE|TRADE_SUMMARY}
1655215881306944445ns gateway_status={account="A1", supported=CREATE_ORDER|MODIFY_ORDER|CANCEL_ORDER|ORDER_ACK|ORDER|TRADE|POSITION, available=, unavailable=CREATE_ORDER|MODIFY_ORDER|CANCEL_ORDER|ORDER_ACK|ORDER|TRADE|POSITION}
1655215881344797226ns stream_status={stream_id=4, account="", supports=REFERENCE_DATA|MARKET_BY_PRICE|TRADE_SUMMARY, transport=TCP, protocol=FIX, encoding=FIX, priority=PRIMARY, connection_status=DOWNLOADING}
1655215881356409974ns stream_status={stream_id=3, account="", supports=MARKET_STATUS|TOP_OF_BOOK, transport=TCP, protocol=WS, encoding=JSON, priority=PRIMARY, connection_status=DOWNLOADING}
[...]

A new tool roq-summary has been introduced.

Example

metadata:

┌──────────────────┬────────────────────────────────────┐
│version           │                               0.8.5│
│application       │                         roq-deribit│
│name              │                             deribit│
├──────────────────┼────────────────────────────────────┤
│hostname          │                           VQ-WRK-01│
│username          │                             thraneh│
│session id        │0d3163eb-8f6c-4faa-afb0-8df8f030b1c2│
├──────────────────┼────────────────────────────────────┤
│encoding          │                         FLATBUFFERS│
│compression method│                              BROTLI│
│compression level │                                   4│
└──────────────────┴────────────────────────────────────┘

header:

┌────────────┬──────────────────────────────┐
│begin       │2022-06-12T14:01:43.669509626Z│
│end         │2022-06-12T14:02:59.498907935Z│
│duration    │                  0d 0h 1m 15s│
├────────────┼──────────────────────────────┤
│uncompressed│                     0.023 GiB│
│compressed  │                     0.003 GiB│
│ratio       │                           14%│
├────────────┼──────────────────────────────┤
│max(seqno)  │                        85,535│
└────────────┴──────────────────────────────┘

users:

  []

accounts:

  ["A1"]

exchanges:

  ["deribit"]

symbols:

  ["BTC-17JUN22","BTC-24JUN22","BTC-29JUL22","BTC-30DEC22","BTC-30SEP22","BTC-31MAR23","BTC-DERIBIT-INDEX","BTC-PERPETUAL","BTC-USDC-DERIBIT-INDEX","BTC_USDC-PERPETUAL","ETH-17JUN22","ETH-24JUN22","ETH-29JUL22","ETH-30DEC22","ETH-30SEP22","ETH-31MAR23","ETH-DERIBIT-INDEX","ETH-PERPETUAL","ETH-USDC-DERIBIT-INDEX","ETH_USDC-PERPETUAL"]

data:

┌────────────┬──────────────────────────────┐
│begin       │2022-06-12T14:01:43.669509626Z│
│end         │2022-06-12T14:02:59.498907935Z│
│duration    │                  0d 0h 1m 15s│
├────────────┼──────────────────────────────┤
│uncompressed│                     0.023 GiB│
│compressed  │                     0.003 GiB│
│ratio       │                           14%│
└────────────┴──────────────────────────────┘

messages:

┌──────────────────────┬──────┬──────────┬─────────┐
│type                  │ count│     bytes│avg. size│
├──────────────────────┼──────┼──────────┼─────────┤
│gateway_settings      │     1│         0│      0.0│
│stream_status         │     8│     1,704│    213.0│
│external_latency      │    29│     7,896│    272.3│
│rate_limit_trigger    │      │          │         │
│gateway_status        │     8│     1,368│    171.0│
│reference_data        │    20│     7,512│    375.6│
│market_status         │    16│     2,864│    179.0│
│top_of_book           │ 7,105│ 1,971,592│    277.5│
│market_by_price_update│74,953│22,248,304│    296.8│
│market_by_order_update│      │          │         │
│trade_summary         │   165│    47,424│    287.4│
│statistics_update     │ 2,431│   730,216│    300.4│
│create_order          │      │          │         │
│modify_order          │      │          │         │
│cancel_order          │      │          │         │
│cancel_all_orders     │      │          │         │
│order_ack             │      │          │         │
│order_update          │      │          │         │
│trade_update          │      │          │         │
│position_update       │      │          │         │
│funds_update          │      │          │         │
│custom_metrics_update │      │          │         │
│custom_message        │      │          │         │
├──────────────────────┼──────┼──────────┼─────────┤
│total                 │84,736│25,018,880│    295.3│
└──────────────────────┴──────┴──────────┴─────────┘

Open Source#

New projects have been added to GitHub

The ClickHouse adapter will monitor one or more event-log directories and export events to a ClickHouse database. The project can be found here.
The ZeroMQ adapter will connect to one or more gateways and publish events to a ZeroMQ network. The project can be found here.
The Redis adapter will connect to one or more gateways and publish events to a Redis memory cache. The project can be found here.

They reflect common use-cases and it is easy to copy the project and customize to personal preferences.

It is the hope that these projects will see contributions from the open-source community.

Other Work#

Socket and web protocol code have been refactored to make it possible to experiment with various I/O solutions, e.g.

Linux’ io_uring
Xlinx’s kernel-bypass solutions

Other long-term benefits

An I/O context can be made available to client libraries thereby making it possible to implement custom network code with the same benefits just mentioned.
These changes are done to to ultimately open source certain binaries, e.g. the JSON bridge.

CHANGELOG#

Milestone on GitHub.

Added#

Changed#

New and changed fields (KuCoin) (#248).
Externalise an event-loop interface (Core) (#242).
Prevent copy/move of Event<> and Trace (API) (#247).
Flag to control terminal color output (Logging) (#245).
API update (KuCoin Futures) (#228).
return std::span<MBPUpdate> from cache::MarketByPrice (API) (#241).
Filter the removal of unknown price-levels from MbP updates (Core) (#238).
New roq-summary + review roq-dump (Tools) (#237).
Support limited-depth MbP (FIX Bridge) (#236).

Fixed#

Auto-reconnect after exchange maintenance (Deribit) (#246).
ReferenceData required update following exchange removing API fields (Coinbase-PRO) (#243).
Using –ws_books_depth=5 was not working (OKX) (#239).
MbP snapshot did not contain exchange_time_utc (Gate.io Futures) (#235).
MbP snapshot did not contain exchange_time_utc (Deribit) (#234).

Release 0.8.6#

Gateways#

API#

MarketByPrice#

Event Logs#

Tools#

Open Source#

Other Work#

CHANGELOG#

Added#

Changed#

Fixed#

Other#