Flags#
Here we describe common options for all gateway solutions.
You can see all options by using --help
from the command-line.
Note
Most options have sensible defaults for a balanced configuration.
You should only have to override the defaults, if you want to tune the configuration for ultra low latency.
Basic#
--name
A name used to identify the service.
--exchange
The exchange name (or a list of exchange names). This will often be used to populate the
exchange
field of event messages.Note
The implementation of the flag is owned by each gateway. (It is not a generic flag.)
For some gateways it will be a single name. Most often you can choose the name you want to identify the exchange by.
For other gateways it may be a comma-separated list of exchange names. This is almost always the case if you connect to a broker. However, it may also be the case if you connect to an exchange gateway that acts as a front for a number of distinct exchanges. For example, CME Globex is a front for several exchanges such as BrokerTec and Nymex. For these cases you must provide exchange names matching those used by the broker or exchange.
--cache_dir
Directory used to cache data between runs, e.g. config history or authentication tokens.
--drop_copy
Enable drop-copy mode? This mode requires support from the gateway: the user must be configured accordingly.
Config#
--config_file
The config file path. See here for further details.
--secrets_file
An optional secrets file path. See here for further details.
--cache_config_retain_users_for
User configuration is retained for this period.
Note
The
user_id
is an integer which is encoded into the ClOrdID when order requests are being sent to an exchange. If a gateway is restarted, it must somehow remember the mapping fromuser_id
back to the name found in the configuration file. The cached config history is used to track this mapping.Since
user_id
has limited range (254 possible values), we need to recycle id’s. However, due to the exchange remembering orders for a certain period, it’s not safe to recycle immediately after a name has been removed (or renamed) from the config file. We therefore retain the “last seen” timestamp and will only recycle the id after the period configured here.Warning
There are no guarantees that an old
user_id
mapping can’t collide with a newly created mapping.You should ensure orders are canceled when a user name is removed (or renamed). And you should set this value to a longer period than the one used by the exchange to publish closed orders.
Clients#
--client_listen_address
A filesystem path used by clients to establish communication with the service.
Note
Unix domain sockets are used to exchange shared memory file descriptors between clients and gateway. You can therefore only connect clients running on the same host.
After an initial handshake, a Unix domain socket remains connected with the only purpose of checking the liveness of a connected client. All following communication is over shared memory.
Service#
All services (gateways, clients, bridges, …) support a HTTP interface for accessing and/or updating internal state.
--service_listen_address
The path or port used to expose service metrics.
--url_prefix
The HTTP URL prefix path used to query for service metrics.
--web_dir
Directory containing static files, e.g.
$CONDA_PREFIX/share/roq/web
.
Loop#
Gateway services are primarily implemented around a core event loop.
Note
The defaults will allow the gateway to sleep (and therefore preserve CPU). This is for low latency (response time: double-digit microseconds).
For ultra low latency (response time: single-digit
microseconds) make sure to pin the dispatch thread
to a CPU using --loop_cpu_affinity
and choose the
following options for busy-polling
--loop_sleep=0ns
--loop_timer_freq=250ns
--loop_cpu_affinity
Used to pin the main dispatch thread to a specific CPU.
--loop_sleep
Use to relinquish control to the kernel.
--loop_timer_freq
Gateway timer frequency.
Note
The core event loop looks a bit like this (pseudo code)
std::chrono::nanoseconds next_timer_update = {};
while (true) {
auto now = get_monotonic_clock();
if (now > next_timer_update) {
next_timer_update = now + FLAGS_loop_timer_freq;
drain_epoll_queues(); // socket processing (*no* wait!)
gateway_timer(); // gateway specific, perhaps send ping to exchange
}
drain_shared_memory_queues(); // connected clients
if (FLAGS_loop_sleep) {
std::this_thread::sleep_for(FLAGS_loop_sleep);
}
}
The core event loop is primarily designed for low latency, i.e. busy-polling.
--loop_sleep=0ns
will never relinquish the CPU.--loop_timer_freq=250ns
will regularly drain socket queues. This is about achieving the best responsiveness, when communicating with the exchange, whilst reducing the time spent transitioning between user-space and kernel. It is here important to understand thatsyscall
’s are expensive and will block the gateway from responding to client requests communicated over shared memory. You are encouraged to experiment and optimize for your own server configuration.--loop_timer_freq=0ns
will always drain socket queues. This may be a good choice, if you use a kernel bypass solution, e.g. something based on SolarFlare or DPDK. You are encouraged to experiment and optimize for your own server configuration.
Net#
Gateways may opt-in to use either of these flags
--net_connection_timeout
Override the connection timeout. Default is 5 seconds.
--net_disconnect_on_idle_timeout
Disconnect if nothing “important” has been received before the timeout. This is only implemented where relevant and, when it is, it depends on the communication protocol. Default is to not disconnect.
--net_tls_validate_certificate
Validate TLS certificates.
Cache#
The gateway will cache certain views (based on rolled up events) for the following purposes
Allow download of current state to newly established client connections.
Early detection of e.g. bad order book state. These situations may occur as a result of lost messages and/or programming mistakes. It is important that the gateways can act as a first order level of protection such that clients (trading strategies) don’t act on incorrect information.
--cache_fills_max_depth
The size of a pre-allocated vector required to hold a list of order fills.
--cache_mbp_allow_price_inversion
For various reasons, an exchange may allow choice or inverted order books to be disseminated. However, choice or inverted prices will most often indicate a problem somewhere in the stack (lost message, programming mistakes, …)
--cache_mbp_compute_checksum
Compute checksum? This can be useful when verifying the client order book is exactly identical to the gateway’s.
--cache_mbp_max_depth
The size of a pre-allocated vector required to cache a rolled-up view of MarketByPrice .
--cache_trades_max_depth
The size of a pre-allocated vector required to hold a list of exchange trades.
--cache_all_reference_data
This flag allows you (opt-in) to cache all reference data including symbols that would normally be discarded due to regex matching.
Event-Log#
The gateway can automatically persist a log of all events. This is useful for simulation, back-testing and investigations.
--event_log_dir
Directory used to write an event-log. Default is
roq-data
under the current directory. Disable by using an empty directory name, i.e.--event_log_dir=""
.Note
The event log is saved to this relative path
<category>/<iso-week>/<name>/<name>-<category>-<timestamp>.roq
Category is either
md
orom
The ISO week is formatted like this
2020-W37
.The name is what you have specified as
--name
.The filename is constructed from startup time in milliseconds since the epoch.
--event_log_iso_week
Sub-directory as
YYYY-WWww
? (Alternative isYYYY-MM-DD
.)--event_log_sync_freq
Maximum time between buffering (encoding and compression) and write-to-disk.
--event_log_utimes_on_sync
Update last modified/access times on sync?
Note
Exceptions may be logged if the event-log file has been moved to another path. This will not interfere with other event-logging operations.
--event_log_output_buffer_size
Size of the output buffer in bytes.
--event_log_encoding
Encoding method.
Note! Currently only supporting Flatbuffers.
--event_log_compression_level
Compression level. Depends on the compression method.
Note! Currently only supporting Brotli.
Note
The event log is not persisted to disk in real time. This is primarily to avoid wearing out your storage device and allow for better compression.
--event_log_symlink
Create symlink to latest event-logs at the top-level directory?
License Manager#
To unlock all features you must acquire an access token from the license manager.
--auth_keys_file
A file containing a public/private key pair generated by roq-keygen for further details. (from the
roq-tools
package).--auth_license_manager_uri
URI’s used to contact the license manager (comma-separated list).
--auth_is_uat
Is the service deployed to UAT?
--auth_hide_name
The name is useful when investigating what services have currently locked all licenses. However, you may decide to opt-out from revealing the service name.
--auth_cache_dir
Directory where access tokens can be cached.
--auth_refresh_freq
Time between refresh.
--auth_lock_period
Desired lock period. (The license manager will ultimately control if this is accepted or overriden.)
Warning
You should NOT expect the gateway to be fully operational without a valid access token.
External Orders and Trades#
--include_external_trades
Whether to capture external trades into the event-log
Rate-Limiter#
Most exchanges implement a request rate-limiter.
Sometimes it’s possible to receive information from the exchange’s rate-limiter making it possible for us to at least try to mirror what is the current state.
Warning
The implementation is experimental and is likely to be removed again.
You can effectively disable this check by settings --rest_rate_limit_max_requests
to some high number.
--rest_rate_limit_interval
The time interval being monitored. We use a fixed window for efficiency reasons (as opposed to sliding window which requires us to track the full history of requests).
--rest_rate_limit_max_requests
The maximum limit for requests multiplied by their weights. The weights are hard-coded matching what can be found from exchange documentation. A weight of 1 is being used when such information can not be found. Disable by setting this to a high number.
Risk (EXPERIMENTAL)#
--enable_risk
Allows a risk manager (drop-copy client) to publish risk limits. Order requests will then be validated against current positions, current risk exposure and risk limits. See here for further details.
UDP (EXPERIMENTAL)#
Gateways may optionally broadcast messages to UDP.
Note
The gateway will only publish to localhost. It is possible to use the tee module of iptables to clone packets for distribution to other hosts (serverfault).
--udp_snapshot_port
Port(s) used to broadcast snapshot messages.
--udp_snapshot_address
Optional IPv4 address(es), e.g. 127.0.0.1 for localhost (also the default).
--udp_incremental_port
Port(s) used to broadcast incremental update messages.
--udp_incremental_address
Optional IPv4 address(s), e.g. 127.0.0.1 for localhost (also the default).
--udp_snapshot_delay_freq
Delay between snapshot updates, e.g. 10ms.
--udp_snapshot_repeat_freq
Minimum time until repeat, e.g. 1min.
--udp_incremental_heartbeat_freq
Heartbeat frequency, e.g. 3s.
--udp_encoding
Message encoding, e.g. flatbuffers.