roq-clickhouse#

Purpose#

  • Monitor event-logs streamed to a filesystem

  • Upload history to a Clickhouse timeseries database.

Conda#

$ mamba install \
  --channel https://roq-trading.com/conda/stable \
  roq-clickhouse
$ roq-clickhouse \
      --name "clickhouse" \
      --service_listen_address "$TCP_LISTEN_PORT" \
      --event_log_dir "$EVENT_LOG_DIR" \
      --flagfile "$FLAG_FILE"

Flags#

$ roq-clickhouse --help
--print_ddl (false)

Print ddl? (and exit)

--event_log_poll_freq (60s)

Event-log poll frequency

--event_log_max_batch_events (1000)

Max events to process before rotating to next event-log

--event_log_dir

Event-log directory

--event_log_include

Event-log include patterns (regex)

--event_log_exclude

Event-log exclude patterns (regex)

--request_buffer_flush_after (30s)

Flush request buffer after idle

--request_encode_buffer_size (4194304)

Encode buffer size (bytes) for requests

--request_max_rows (32768)

Max rows per request

--request_timeout (120s)

Request timeout

--request_rate_limit_interval (5s)

Rate limit interval

--request_rate_limit_max_requests (100)

Rate limit maximum requests per interval

--server_ping_freq (30s)

Ping frequency

--server_ping_path (/ping)

Ping path

--request_retry_wait (10s)

Retry wait period

--request_retry_max (3)

Max retry attempts

--market_by_price_cache_max_depth (32768)

Market-by-price book depth

--market_by_price_view_depth (3)

Market-by-price view depth

--market_by_order_cache_max_depth (32768)

Market-by-order book depth

--market_by_order_view_depth (3)

Market-by-order view depth

--snapshot_lookback_days (200)

How many days to include when finding max(seqno) per session_id

--retention_policy_time_to_live_days (0)

Time-to-live (days)

--uri (http://localhost:8123)

Server URI

--proxy

Proxy URI

--username

Username

--password

Password

--database_name (roq)

Database name

--create_database (false)

Create database?

--temp_drop_mbp (false)

Do not populate MbP? (temporary work-around)

Environments#

# flagfile

--uri=http://localhost:8123
--database_name=roq
--create_database=True
--retention_policy_time_to_live_days=2