roq-journal#

$ mamba install \
  --channel https://roq-trading.com/conda/stable \
  roq-journal

$ mamba install \
  --channel https://roq-trading.com/conda/unstable \
  roq-journal

roq-journal#

$ roq-journal [FLAGS]

Description#

roq-journal supports distributed persistence of order requests and updates.

  • Listens for multicast OrderAck and OrderUpdate messages.

  • Appends messages to a journal on the local filesystem.

  • Inserts messages into relevant database tables.

  • Acknowledges receipt by sending a UDP message to the original sender of the received message.

  • Replays the local journal after restart.

Any number of roq-journal instances can be deployed to lower the round-trip latency and to reduce the risk of data loss.

The local journal can be synchronized before or after acknowledging receipt, or not at all.

Note

roq-journal was designed to support low-latency persistence of client order-id mappings (routing_id). This is specifically required to support the OrderCancelRequest <F> and OrderCancelReplaceRequest <G> FIX messages.

Flags#

$ roq-journal --help
--name

Service name

--cache_dir (roq-cache)

Cache directory (path)

--cache_retain_for (48h)

Retain cache files for this period

--cache_datasync (false)

Enable filesystem sync?

--cache_datasync_before_respond (false)

Issue filesystem sync before responding? (depends on –cache_datasync)

--multicast_port

Multicast port

--multicast_address

Multicast address

--local_interface

Local interface

--buffer_size (2048)

Buffer size

--database_uri (http://localhost:8123)

Database URI

--database_username

Database username

--database_password

Database password

--database_name

Database name

--database_create_if_not_exists (true)

Create database?

--database_request_timeout (5s)

Database request timeout

--database_ping_freq (30s)

Database ping frequency

--database_ping_path (/ping)

Database ping path

--database_encode_buffer_size (16777216)

Encode buffer size (bytes)

--database_decode_buffer_size (1048576)

Decode buffer size (bytes)

--database_retain_for (720h)

Retain orders for this period

--database_purge_freq

Purge orders frequency

--database_retry_freq (2s)

Request retry frequency

--loop_cpu_affinity (-1)

CPU affinity for the main dispatch thread

--loop_sleep (500ns)

Sleep period for the main dispatch thread

--loop_timer_freq (2500ns)

Timer frequency

--service_listen_address

Listen address

--service_url_prefix

URL prefix

--service_session_timeout (5s)

Session time-out period

roq-journal-admin#

$ roq-journal-admin [FLAGS] COMMAND

Description#

roq-journal-admin is a database maintenance tool

  • Create database and tables.

  • Purge unused request and order history.

  • Repair the database following a failed purge request.

Flags#

$ roq-journal-admin --help
--database_uri (http://localhost:8123)

Database URI

--database_username

Database username

--database_password

Database password

--database_name

Database name

Commands#

CREATE

  • Create the database (if not exists).

  • Create the tables (if not exists).

PURGE

  • Remove final orders from the order_ack and order_update tables.

  • Retain latest request version from the order_ack table.

  • Retain latest order version from the order_update table.

Warning

Although attempts are made to deal with errors and roll-backs, this is an error-prone operation due to ClickHouse not supporting transactions. ALWAYS make sure you create a BACKUP before running this command!

REPAIR

  • Attempt to repair the database following a failed PURGE request.

Warning

There are edge-cases where it is not possible to repair the database.

References#