Flags#

Here we describe common options for all gateway solutions. You can see all options by using --help from the command-line.

Note

Most options have sensible defaults for a balanced configuration.

You should only have to override the defaults, if you want to tune the configuration for ultra low latency.

Basic#

--name

A name used to identify the service.

--exchange

The exchange name (or a list of exchange names). This will often be used to populate the exchange field of event messages.

Note

The implementation of the flag is owned by each gateway. (It is not a generic flag.)

For some gateways it will be a single name. Most often you can choose the name you want to identify the exchange by.

For other gateways it may be a comma-separated list of exchange names. This is almost always the case if you connect to a broker. However, it may also be the case if you connect to an exchange gateway that acts as a front for a number of distinct exchanges. For example, CME Globex is a front for several exchanges such as BrokerTec and Nymex. For these cases you must provide exchange names matching those used by the broker or exchange.

--cache_dir

Directory used to cache data between runs, e.g. config history or authentication tokens.

--drop_copy

Enable drop-copy mode? This mode requires support from the gateway: the user must be configured accordingly.

Config#

--config_file

The config file path. See here for further details.

--secrets_file

An optional secrets file path. See here for further details.

--cache_config_retain_users_for

User configuration is retained for this period.

Note

The user_id is an integer which is encoded into the ClOrdID when order requests are being sent to an exchange. If a gateway is restarted, it must somehow remember the mapping from user_id back to the name found in the configuration file. The cached config history is used to track this mapping.

Since user_id has limited range (254 possible values), we need to recycle id’s. However, due to the exchange remembering orders for a certain period, it’s not safe to recycle immediately after a name has been removed (or renamed) from the config file. We therefore retain the “last seen” timestamp and will only recycle the id after the period configured here.

Warning

There are no guarantees that an old user_id mapping can’t collide with a newly created mapping.

You should ensure orders are canceled when a user name is removed (or renamed). And you should set this value to a longer period than the one used by the exchange to publish closed orders.

Clients#

--client_listen_address

A filesystem path used by clients to establish communication with the service.

Note

Unix domain sockets are used to exchange shared memory file descriptors between clients and gateway. You can therefore only connect clients running on the same host.

After an initial handshake, a Unix domain socket remains connected with the only purpose of checking the liveness of a connected client. All following communication is over shared memory.

Service#

All services (gateways, clients, bridges, …) support a HTTP interface for accessing and/or updating internal state.

--service_listen_address

The path or port used to expose service metrics.

--url_prefix

The HTTP URL prefix path used to query for service metrics.

--web_dir

Directory containing static files, e.g. $CONDA_PREFIX/share/roq/web.

Loop#

Gateway services are primarily implemented around a core event loop.

Note

The defaults will allow the gateway to sleep (and therefore preserve CPU). This is for low latency (response time: double-digit microseconds).

For ultra low latency (response time: single-digit microseconds) make sure to pin the dispatch thread to a CPU using --loop_cpu_affinity and choose the following options for busy-polling

  • --loop_sleep=0ns

  • --loop_timer_freq=250ns

--loop_cpu_affinity

Used to pin the main dispatch thread to a specific CPU.

--loop_sleep

Use to relinquish control to the kernel.

--loop_timer_freq

Gateway timer frequency.

Note

The core event loop looks a bit like this (pseudo code)

std::chrono::nanoseconds next_timer_update = {};
while (true) {
  auto now = get_monotonic_clock();
  if (now > next_timer_update) {
    next_timer_update = now + FLAGS_loop_timer_freq;
    drain_epoll_queues();  // socket processing (*no* wait!)
    gateway_timer();  // gateway specific, perhaps send ping to exchange
  }
  drain_shared_memory_queues();  // connected clients
  if (FLAGS_loop_sleep) {
    std::this_thread::sleep_for(FLAGS_loop_sleep);
  }
}

  • The core event loop is primarily designed for low latency, i.e. busy-polling.

  • --loop_sleep=0ns will never relinquish the CPU.

  • --loop_timer_freq=250ns will regularly drain socket queues. This is about achieving the best responsiveness, when communicating with the exchange, whilst reducing the time spent transitioning between user-space and kernel. It is here important to understand that syscall’s are expensive and will block the gateway from responding to client requests communicated over shared memory. You are encouraged to experiment and optimize for your own server configuration.

  • --loop_timer_freq=0ns will always drain socket queues. This may be a good choice, if you use a kernel bypass solution, e.g. something based on SolarFlare or DPDK. You are encouraged to experiment and optimize for your own server configuration.

Net#

Gateways may opt-in to use either of these flags

--net_connection_timeout

Override the connection timeout. Default is 5 seconds.

--net_disconnect_on_idle_timeout

Disconnect if nothing “important” has been received before the timeout. This is only implemented where relevant and, when it is, it depends on the communication protocol. Default is to not disconnect.

--net_tls_validate_certificate

Validate TLS certificates.

Cache#

The gateway will cache certain views (based on rolled up events) for the following purposes

  • Allow download of current state to newly established client connections.

  • Early detection of e.g. bad order book state. These situations may occur as a result of lost messages and/or programming mistakes. It is important that the gateways can act as a first order level of protection such that clients (trading strategies) don’t act on incorrect information.

--cache_fills_max_depth

The size of a pre-allocated vector required to hold a list of order fills.

--cache_mbp_allow_price_inversion

For various reasons, an exchange may allow choice or inverted order books to be disseminated. However, choice or inverted prices will most often indicate a problem somewhere in the stack (lost message, programming mistakes, …)

--cache_mbp_compute_checksum

Compute checksum? This can be useful when verifying the client order book is exactly identical to the gateway’s.

--cache_mbp_max_depth

The size of a pre-allocated vector required to cache a rolled-up view of MarketByPrice .

--cache_trades_max_depth

The size of a pre-allocated vector required to hold a list of exchange trades.

--cache_all_reference_data

This flag allows you (opt-in) to cache all reference data including symbols that would normally be discarded due to regex matching.

Event-Log#

The gateway can automatically persist a log of all events. This is useful for simulation, back-testing and investigations.

--event_log_dir

Directory used to write an event-log. Default is roq-data under the current directory. Disable by using an empty directory name, i.e. --event_log_dir="".

Note

The event log is saved to this relative path

<category>/<iso-week>/<name>/<name>-<category>-<timestamp>.roq

  • Category is either md or om

  • The ISO week is formatted like this 2020-W37.

  • The name is what you have specified as --name.

  • The filename is constructed from startup time in milliseconds since the epoch.

--event_log_iso_week

Sub-directory as YYYY-WWww? (Alternative is YYYY-MM-DD.)

--event_log_sync_freq

Maximum time between buffering (encoding and compression) and write-to-disk.

--event_log_utimes_on_sync

Update last modified/access times on sync?

Note

Exceptions may be logged if the event-log file has been moved to another path. This will not interfere with other event-logging operations.

--event_log_output_buffer_size

Size of the output buffer in bytes.

--event_log_encoding

Encoding method.

Note! Currently only supporting Flatbuffers.

--event_log_compression_level

Compression level. Depends on the compression method.

Note! Currently only supporting Brotli.

Note

The event log is not persisted to disk in real time. This is primarily to avoid wearing out your storage device and allow for better compression.

--event_log_symlink

Create symlink to latest event-logs at the top-level directory?

License Manager#

To unlock all features you must acquire an access token from the license manager.

--auth_keys_file

A file containing a public/private key pair generated by roq-keygen for further details. (from the roq-tools package).

--auth_license_manager_uri

URI’s used to contact the license manager (comma-separated list).

--auth_is_uat

Is the service deployed to UAT?

--auth_hide_name

The name is useful when investigating what services have currently locked all licenses. However, you may decide to opt-out from revealing the service name.

--auth_cache_dir

Directory where access tokens can be cached.

--auth_refresh_freq

Time between refresh.

--auth_lock_period

Desired lock period. (The license manager will ultimately control if this is accepted or overriden.)

Warning

You should NOT expect the gateway to be fully operational without a valid access token.

External Orders and Trades#

--include_external_trades

Whether to capture external trades into the event-log

Rate-Limiter#

Most exchanges implement a request rate-limiter.

Sometimes it’s possible to receive information from the exchange’s rate-limiter making it possible for us to at least try to mirror what is the current state.

Warning

The implementation is experimental and is likely to be removed again. You can effectively disable this check by settings --rest_rate_limit_max_requests to some high number.

--rest_rate_limit_interval

The time interval being monitored. We use a fixed window for efficiency reasons (as opposed to sliding window which requires us to track the full history of requests).

--rest_rate_limit_max_requests

The maximum limit for requests multiplied by their weights. The weights are hard-coded matching what can be found from exchange documentation. A weight of 1 is being used when such information can not be found. Disable by setting this to a high number.

Risk (EXPERIMENTAL)#

--enable_risk

Allows a risk manager (drop-copy client) to publish risk limits. Order requests will then be validated against current positions, current risk exposure and risk limits. See here for further details.

UDP (EXPERIMENTAL)#

Gateways may optionally broadcast messages to UDP.

Note

The gateway will only publish to localhost. It is possible to use the tee module of iptables to clone packets for distribution to other hosts (serverfault).

--udp_snapshot_port

Port(s) used to broadcast snapshot messages.

--udp_snapshot_address

Optional IPv4 address(es), e.g. 127.0.0.1 for localhost (also the default).

--udp_incremental_port

Port(s) used to broadcast incremental update messages.

--udp_incremental_address

Optional IPv4 address(s), e.g. 127.0.0.1 for localhost (also the default).

--udp_snapshot_delay_freq

Delay between snapshot updates, e.g. 10ms.

--udp_snapshot_repeat_freq

Minimum time until repeat, e.g. 1min.

--udp_incremental_heartbeat_freq

Heartbeat frequency, e.g. 3s.

--udp_encoding

Message encoding, e.g. flatbuffers.