We will use the Deribit gateway for the following examples.


All gateways use a core set of flags and similar template for the config file. Differences normally relate to authentication and how to define the end-points needed to communicate with the exchange.


1conda install -y --channel \
2    roq-deribit


You can start by making a copy of the template config file

1cp $CONDA_PREFIX/share/roq-deribit/config.toml ~/deribit.toml

You should edit the copy: look for these lines and replace as appropriate


The login and secret must be created using the Deribit web interface


1roq-deribit \
2    --name "deribit" \
3    --config_file ~/deribit.toml \
4    --client_listen_address ~/deribit.sock

This will launch the gateway using the config file you just created.

The gateway will be listening for client connections using the UNIX domain socket openend on ~/deribit.sock.

The name flag is required and e.g. used to identify the gateway when a client connects to it.

Event Logs

All events are automatically persisted to your filesystem using the relative path: <ISO week>/<name>/<timestamp>.roq, for example 2020-W31/deribit/1596198613995.roq. The timestamp is the process startup time in milliseconds since epoch.

This convention avoids duplicates, allows you to easily sort based on time and is convenient for persisting files so they can easily be backed up.


Event logs are not updated in real time. Default is to flush when a 1MB memory buffer is full or if 5 minutes have passed (whichever condition is met first). The primary reason for throttling output is that data is being compressed: the compressor can simply do a better job when it can processes more data in memory. These are just the defaults and the behavior be overriden by setting appropriate flags.

You can use event logs when simulating trading strategies or to feed third-party database solutions.


Adding support for metrics

1roq-deribit \
2    --name "deribit" \
3    --config_file ~/deribit.toml \
4    --metrics_listen_address 1234 \
5    --client_listen_address ~/deribit.sock

From another terminal

1curl http://localhost:1234/metrics

Will give you very detailed information about gateway performance using Prometheus’ Exposition Format.

For example

 2# TYPE roq_events histogram
 3roq_events_bucket{source="deribit", le="500"} 689
 4roq_events_bucket{source="deribit", le="1000"} 7920
 5roq_events_bucket{source="deribit", le="2000"} 16035
 6roq_events_bucket{source="deribit", le="5000"} 18918
 7roq_events_bucket{source="deribit", le="10000"} 19207
 8roq_events_bucket{source="deribit", le="20000"} 20093
 9roq_events_bucket{source="deribit", le="+Inf"} 20711
10roq_events_sum{source="deribit"} 5.69656e+07
11roq_events_count{source="deribit"} 20711

This gives us some information about the distribution of processing times

  • processed 20711 events (_count)

  • 689 of those was done in less than 500ns (_bucket{…, le=”500”})

  • 7920 of those was done in less than 1000ns (_bucket{…, le=”1000”}) — this includes the 689 also counted for less than 500ns

  • The total time spent processing events is 5.69656e+07ns (_sum).

From this we have very useful information about the distribution of processing times.

The gateways will automatically collect such metrics and varying levels of granularity, for example

 1# TYPE roq_profile histogram
 2roq_profile_bucket{source="deribit", connection="enqueue", function="market_by_price_update", le="500"} 0
 3roq_profile_bucket{source="deribit", connection="enqueue", function="market_by_price_update", le="1000"} 0
 4roq_profile_bucket{source="deribit", connection="enqueue", function="market_by_price_update", le="2000"} 71
 5roq_profile_bucket{source="deribit", connection="enqueue", function="market_by_price_update", le="5000"} 1181
 6roq_profile_bucket{source="deribit", connection="enqueue", function="market_by_price_update", le="10000"} 4428
 7roq_profile_bucket{source="deribit", connection="enqueue", function="market_by_price_update", le="20000"} 5752
 8roq_profile_bucket{source="deribit", connection="enqueue", function="market_by_price_update", le="+Inf"} 6034
 9roq_profile_sum{source="deribit", connection="enqueue", function="market_by_price_update"} 5.31952e+07
10roq_profile_count{source="deribit", connection="enqueue", function="market_by_price_update"} 6034

Prometheus can be configured to automatically scrape this information.

Grafana is a dashboard solution allowing you to chart the data collected by Prometheus.