C++ HFT on Crypto Exchanges with μs Latency!

2019-11-20

A preview meant to demonstrate the main features of our product. By publishing this document, we hope to get in contact with early adopters from within the professional trading community. We obviously appreciate feedback from anyone.

This document is kept as brief as possible whilst still explaining most of the issues faced by an algo trader managing his own trading platform.

What is it?

Trading Infrastructure

A collection of solutions allowing you to implement your own bespoke low latency trading infrastructure.

The overriding idea is that the solutions should allow you to focus on your primary goal: design and run your own trading strategies.

The design is that of micro-services: you should be able to reconfigure and restart individual components without having to bring down your entire trading platform.

Internal latency (between receiving and sending network packets) is in the order of low single-digit microseconds. Other factors are likely more significant: location of host, network configuration and latency to exchange.

Note

Predictable low latency is perhaps not your highest concern. However, when investigating worse profitability than expected, it is simply one less factor to worry about.

Live Trading

Your strategy must interact asynchronously with one or more gateways through a normalized C++ interface. (You can find examples on GitHub).

The gateways are primarily responsible for managing connectivity and translating messages. However, the gateways will also do the following

  • Cache updates (reference data, maintain full image of order book, etc.) such that a client can connect at any time and automatically receive and continue from current state. This is required to support a micro-service design.

  • Correctly and safely route order updates between exchange and the originating client. This is required to allow multiple strategies trade the same accounts without risking cross-pollination.

  • Manage permissioning on a per-client basis: configuration includes the accounts and symbols allowed for trading. This is required for proper segregation between risk management and trading.

  • Manage a circuit breaker by monitoring per-client request frequency and activate “halt” when pre-configured parameters have been violated. (A recurring scenario, due to asynchronicity, is a strategy sending the same order request over and over). This is required to have proper segregation between risk management and trading.

  • Capture all messages into an event log. This is required to allow real-time operational support. This is also extremely useful for accurate historical simulation.

Historical Simulation

Our client library includes a simulator framework with the following capabilities

  • Exact replay of gateway captured event logs

  • Simulation of external latencies by queuing both incoming and outgoing messages

  • Order matching (external to the strategy implementation)

  • Result collection (external to the strategy implementation)

The main idea is that the strategy implementation should be exactly the same for live trading as for historical simulation. (A fundamental premise for trusting back-test results).

The client library includes a simple FIFO order matcher which conservatively takes into account market liquidity.

The simulator will access order matching through an interface: you can therefore implement you own order matching assumptions.

The simulator will emit all events to a collector interface: you can therefore implement your own metrics to assess strategy performance.

System Performance

All trading components (not only gateways) will automatically capture metrics for latencies and profiling.

Prometheus’ Exposition Format is supported. Using Prometheus enables AlertManager (notifications) and Grafana (dashboards).

Prometheus and Grafana are both open source.

Time-Series Database

Event logs are streamed to a storage device and can be exported in real-time to a time-series database.

Our InfluxDB exporter is available for free.

InfluxDB is open source.

Provisioning / Deployment

We use Conda packaging for distributing binaries. This solution will automatically manage dependencies and it doesn’t require root access. You can create as many Conda environments as you want.

Although Docker images are available, Conda is our preferred delivery mechanism.

Note

We do not recommend using Docker for any trading component.

We publish an Ansible script on GitHub. The script makes it possible to provision a server from fresh Linux install.

An Ansible Playbook, also on GitHub, helps you get started with host and gateway configurations.

Every deployment environment is different. Our hope is that we have provided enough examples to support most use-cases.

Who is it for?

Short answer: “professional traders”.

Hedge Funds

Back-testing and strategy optimization is very important. A fully autonomous trading strategy may be desired. Multiple trading strategies must be supported, possibly developed by completely independent traders. We support these use-cases.

Arbitrageurs

Simultaneous access to multiple exchanges is a requirement. Low latency is very important. We support these use-cases.

Market makers

Back-testing (randomized) order flow and risk management is very important. Low latency is very important. We support these use-cases.

Pricing

Everything is designed to be cost-effective

  • Extensive use of open source (Linux, Prometheus, Grafana, InfluxDB, etc).

  • Gateway licenses will be affordable to start-up hedge-funds and private individuals.

  • Optional support agreement.

Gateways

Pricing will be announced when we launch early 2020.

Note

The software will remain “beta” until launch. You should not expect the software to be stable. We will also not require the use of a license key until the launch date.

Everything else

Free! You can either find source code on GitHub or download binaries from our Conda repository.

Support

Contact us!

Constraints

Every design has constraints: these are the main ones

C++

We are currently using C++17.

Linux

Our aim is to support the most recent stable release from RHEL, CentOS, Ubuntu and Debian.

Linux-only is motivated by third-party kernel-bypass solutions such as OpenOnload, f-Stack/DPDK, etc. We use epoll which should be supported by most such solutions.

Single host

Shared memory is being used to communicate between trading components.

This is motivated by the overriding goal of making the infrastructure predictable such that the focus can be on strategy performance rather than having to investigate random system-wide issues.

This is also not as bad as it sounds: most high-end CPUs nowadays come with dozens of cores.

Busy polling

Each trading component will use a CPU core 100% to busy poll on shared memory.

This is again motivated by predictability and low latency.

Physical machine

Access to BIOS (disable hyperthreading) and kernel boot parameters (isolcpus) are both strongly recommended steps.

Design

Here we describe the most relevant components of a trading infrastructure.

Color coding

  • Red is 100% owned by you

  • Blue requires a license (developed by Roq Trading Solutions)

  • Green is free (developed by Roq Trading Solutions)

  • Yellow is open source (developed by third-party)

../../_images/overview.png
  1. Currently supported exchanges

    • Coinbase Pro

    • Deribit

    More exchanges will follow in the near future.

    Note

    The infrastructure can support any kind of asset class. Feel free to contact us if you have specific requirements.

  2. Gateways have these responsibilities

    • Maintain connectivity

    • Normalize messages

    • Monitor client (strategy) activity

    • Event logging

    Each operation (event) is expected complete within a budget of a few microseconds. (This is only a goal, not a hard requirement and it obviously depends on available exchange protocols: JSON is not an efficient format, FIX is better, etc).

  3. Clients (strategies) communicate over shared memory with one or more gateways.

    This is an example of 1-way heartbeat latencies between gateways and clients

    ../../_images/heartbeat.png

    Note

    Communication using shared memory is highly dependent on the efficiency of cache-line sharing as well as deployment vs the NUMA configuration. The example was running on a relatively low-powered AMD EPYC 3251 SoC: it is obvious to see that there are 2 distinct latencies (around 0.4-0.5 microseconds and around 0.8-1.0 microseconds).

  4. Trading strategies must implement the Roq API.

    This is the inbound interface

    ../../_images/api_in.png

    And this is the outbound interface

    ../../_images/api_out.png

    The interfaces are defined here: $CONDA_PREFIX/include/roq/client.h and the underlying structs are defined here: $CONDA_PREFIX/include/roq/api.h.

    Note

    It is the exactly same interface is used for live trading as for historical simulation: there is no difference as seen from the trading strategy.

  5. Gateways will automatically capture metrics.

    This is an example showing key profiling information , including conditional distributions (e.g. “what is the % of events where time spent was more than 5 microseconds?”).

    ../../_images/profile.png

    Note

    First, profiling can be done multiple times within a single call stack (think: flame graphs).

    This is evident when looking at market_by_price and ws_l2update.

    Second, we collect histograms such that it’s possible to demonstrate the percentage of observations larger than a threshold.

    Here it’s clear to see that a small percentage takes more than 5 microseconds and almost no events takes longer than 10 microseconds.

  6. Prometheus can be configured to scrape metrics from the gateways.

    This is an example selecting profiling metrics using the Prometheus query interface

    ../../_images/prometheus.png
  7. AlertManager can be configured to issue alerts to popular communication channels (email, chat, etc).

    Typically one would monitor for gateway restarts, too many (client) exceptions over a time-window, rejects from the exchange, etc.

  8. Grafana is a popular tool to visualize metrics captured by e.g. Prometheus.

    This is an example of high-level gateway monitoring

    ../../_images/gateways.png
  9. Event logs are automatically generated by the gateways (at zero extra latency) and streamed to a storage device in near real-time.

    Note

    The event log is not synchronous, i.e. we do not instruct the kernel to sync the file after each message has been appended to the file. This service is only best-effort.

  10. Historical simulation will use event logs to generate update events to the trading strategy. We will demonstrate this in the following tutorial.

  11. The InfluxDB exporter (roq-influxdb) will monitor event logs and automatically insert updates into InfluxDB.

  12. InfluxDB can be used to investigate “what happened?”.

    This is an example of selecting the raw market-by-price updates using CLI

    ../../_images/influxdb.png

Tutorial

Our objective here is to simply demonstrate the minimum number of steps required to get started with simulation and trading.

Conda

Download the Miniconda installer

wget -N https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

Install Miniconda

bash Miniconda3-latest-Linux-x86_64.sh -b -u -p ~/miniconda3

Add roq-trading as a channel

cat > ~/miniconda3/.condarc << EOF
channels:
  - https://roq-trading.com/conda/stable
  - defaults
  - conda-forge
EOF

Activate Miniconda

source ~/miniconda3/bin/activate

Install dev tools

conda install -y git cmake gxx_linux-64 gdb_linux-64

Important

Use the Conda provided compiler toolchain to avoid compatibility issues!

Build

Clone roq-samples from GitHub

git clone https://github.com/roq-trading/roq-samples

Change to the newly created directory

cd roq-samples

Update git submodules

git submodule update --init --recursive

Install the Roq client library

conda install -y roq-client

Run CMake

cmake \
    -DCMAKE_AR="$AR" \
    -DCMAKE_RANLIB="$RANLIB" \
    -DCMAKE_NM="$NM" \
    -DCMAKE_BUILD_TYPE=Debug

Note

Remove the CMakeCache.txt file, if you must repeat any of the previous steps.

Compile the project

make -j4

Simulation

Download Roq sample data

conda install -y roq-data

Note

Many thanks to Deribit for giving their permission to distributing this small dataset.

First change directory

cd ~/roq-samples/src/roq/samples/example-3

Run the simulation

./roq-samples-example-3 \
    --name "example-3" \
    --simulation \
    "$CONDA_PREFIX/share/roq/data/deribit.roq"

Note

The last argument is the path to an event log file.

You should now see output like this

I1114 13:42:49.374023 26158 application.cpp:41] ===== START =====
I1114 13:42:49.377464 26158 main.cpp:182] [deribit:BTC-27DEC19] connection_status=CONNECTED
I1114 13:42:49.377756 26158 main.cpp:229] [deribit:BTC-27DEC19] market_data_status=CONNECTING
I1114 13:42:49.377769 26158 main.cpp:243] [deribit:BTC-27DEC19] order_manager_status=CONNECTING
I1114 13:42:49.377786 26158 main.cpp:229] [deribit:BTC-27DEC19] market_data_status=LOGIN_SENT
I1114 13:42:49.377795 26158 main.cpp:243] [deribit:BTC-27DEC19] order_manager_status=LOGIN_SENT
I1114 13:42:49.377801 26158 main.cpp:229] [deribit:BTC-27DEC19] market_data_status=DOWNLOADING
I1114 13:42:49.377809 26158 main.cpp:243] [deribit:BTC-27DEC19] order_manager_status=DOWNLOADING
I1114 13:42:49.377837 26158 main.cpp:260] [deribit:BTC-27DEC19] tick_size=0.5
I1114 13:42:49.377844 26158 main.cpp:267] [deribit:BTC-27DEC19] min_trade_vol=1.0
I1114 13:42:49.377850 26158 main.cpp:274] [deribit:BTC-27DEC19] multiplier=10.0
I1114 13:42:49.377858 26158 main.cpp:289] [deribit:BTC-27DEC19] trading_status=OPEN
I1114 13:42:49.377880 26158 main.cpp:229] [deribit:BTC-27DEC19] market_data_status=READY
I1114 13:42:49.377887 26158 main.cpp:243] [deribit:BTC-27DEC19] order_manager_status=READY
I1114 13:42:49.377892 26158 main.cpp:385] [deribit:BTC-27DEC19] ready=true
I1114 13:42:49.419499 26158 main.cpp:523] DIRECTION: SELLING
I1114 13:42:49.420500 26158 main.cpp:517] SIGNAL: BUY @ 8788.0
W1114 13:42:49.420508 26158 main.cpp:722] Trading *NOT* enabled
[...]
I1114 13:42:49.547995 26158 main.cpp:182] [deribit:BTC-27DEC19] connection_status=DISCONNECTED
I1114 13:42:49.548006 26158 main.cpp:385] [deribit:BTC-27DEC19] ready=false
I1114 13:42:49.550151 26158 application.cpp:43] ===== STOP =====

Important to note is connectivity to a gateway is being simulated: the strategy receives connection status and reference data before it reaches the “ready” state.

However, you will notice that orders are being blocked: the Trading *NOT* enabled message.

You must explicitly instruct the strategy to enable trading

./roq-samples-example-3 \
    --name "example-3" \
    --simulation \
    "$CONDA_PREFIX/share/roq/data/deribit.roq" \
    --enable-trading

The output should now include the order related messages

[...]
I1114 13:47:07.353455 26889 main.cpp:523] DIRECTION: SELLING
I1114 13:47:07.354487 26889 main.cpp:517] SIGNAL: BUY @ 8788.0
I1114 13:47:07.354517 26889 main.cpp:668] OrderAck={account="A1", order_id=1, type=CREATE_ORDER, origin=EXCHANGE, status=ACCEPTED, error=NONE, text="", gateway_order_id=1, external_order_id="", request_id=""}
I1114 13:47:07.354551 26889 main.cpp:675] OrderUpdate={account="A1", order_id=1, exchange="deribit", symbol="BTC-27DEC19", status=WORKING, side=BUY, price=8787.5, remaining_quantity=1.0, traded_quantity=0.0, position_effect=UNDEFINED, order_template="", create_time_utc=0ns, update_time_utc=0ns, commissions=0.0, gateway_order_id=1, external_order_id=""}
I1114 13:47:07.354561 26889 main.cpp:340] [deribit:BTC-27DEC19] position=0.0
I1114 13:47:07.355589 26889 main.cpp:684] TradeUpdate={account="A1", trade_id=1, order_id=1, exchange="deribit", symbol="BTC-27DEC19", side=BUY, quantity=1.0, price=8787.5, position_effect=UNDEFINED, order_template="", create_time_utc=0ns, update_time_utc=0ns, gateway_order_id=1, gateway_trade_id=0, external_order_id="", external_trade_id=""}
I1114 13:47:07.355614 26889 main.cpp:675] OrderUpdate={account="A1", order_id=1, exchange="deribit", symbol="BTC-27DEC19", status=COMPLETED, side=BUY, price=8787.5, remaining_quantity=0.0, traded_quantity=1.0, position_effect=UNDEFINED, order_template="", create_time_utc=0ns, update_time_utc=0ns, commissions=0.0, gateway_order_id=1, external_order_id=""}
I1114 13:47:07.355619 26889 main.cpp:340] [deribit:BTC-27DEC19] position=1.0
[...]

Deribit

Important

You should start a new terminal, so you can run the gateway and the example code side-by-side.

Activate your Conda environment

source ~/miniconda3/bin/activate

Install the deribit gateway

conda install -y roq-deribit

The gateway comes with a config file example. Make a copy of it

cp $CONDA_PREFIX/share/roq/deribit/config.toml ./deribit.toml

You should now edit this file and update with your Deribit API credentials.

Important

Make sure you’re using Deribit’s test platform: You can find the Deribit API settings here.

You should look for these lines and replace

login = "YOUR_DERIBIT_LOGIN_GOES_HERE"
secret = "YOUR_DERIBIT_SECRET_GOES_HERE"

The gateway is started like this

roq-deribit \
    --name "deribit" \
    --config-file deribit.toml \
    --client-listen-address ~/deribit.sock

Note

Default gateway command-line flags will always point you to an exchange’s test platform.

You can easily see the defaults like this: roq-deribit --help.

Live Trading

Note

Now return to the terminal where you built your examples.

If you need to start a new terminal, remember to activate your Conda environment first: source ~/miniconda3/bin/activate.

Running the strategy against the gateway is almost identical to the simulation.

./roq-samples-example-3 \
    --name "example-3" \
    ~/deribit.sock

The output should now be familiar.

If you check the terminal, where the gateway runs, you’ll see something like this

[...]
I1114 14:19:42.726331 1545 controller.cpp:150] Got connection
I1114 14:19:42.726402 1545 session.cpp:43] Adding user id=1, name="example-3"
I1114 14:19:42.726542 1546 pollster.cpp:230] SubscribeEvent={message_info={source=1, source_name="example-3", source_session_id="TODO", source_seqno=1, receive_time_utc=1573741182726521821ns, receive_time=950644177300600ns, source_send_time=950644177283535ns, source_receive_time=950644177283535ns, origin_create_time=950644177283535ns, is_last=true, opaque=0}, subscribe={accounts={"A1"}, symbols_by_exchange={""={"BTC|USD"}, "deribit"={"BTC-27DEC19"}}}}
I1114 14:19:42.726566 1546 state.cpp:93] SubscribeEvent={message_info={source=1, source_name="example-3", source_session_id="TODO", source_seqno=1, receive_time_utc=1573741182726521821ns, receive_time=950644177300600ns, source_send_time=950644177283535ns, source_receive_time=950644177283535ns, origin_create_time=950644177283535ns, is_last=true, opaque=0}, subscribe={accounts={"A1"}, symbols_by_exchange={""={"BTC|USD"}, "deribit"={"BTC-27DEC19"}}}}
[...]

Important

Probably you will now want to add the --enable-trading flag.

Please check once again that you are connected to Deribit’s test platform: you’re almost guaranteed to lose money if you run the example strategy :-)

Roadmap

2019-11
  • Fully functional gateways: Deribit and Coinbase Pro

  • Examples

  • Documentation

  • ShowHN

2019-12
  • Make gateways production-ready: extensive testing

  • Better simulation: support more order types, extensive testing

  • User feedback: depending on interests, possibly update the roadmap

  • Documentation and web-site

  • Finalize license and support models

2020-01
  • Add further gateways. Coverage should include top exchanges for crypto cash, futures and options and also cover geographical locations.

  • Work with early-adopter clients to implement bespoke solutions

  • Marketing

2020-02
  • Launch

  • License model enforced on gateways