SBE#

Overview#

Format#

  • A header occupies the first 16 bytes.

  • The header contains fields to manage and detect

    • Reset (session id)

    • Re-assembly (sequence number, current fragment number, max fragment number)

Offset

Width

Type

Description

0

1

uint8

Control (bit fields)

  • 0-3: Encoding (enum) (1 Roq, 2 FlatBuffers, 3 SBE, …)

  • 4: update type (bool) (0 snapshot, 1 incremental)

  • 5-7: unused

1

1

uint8

object type (distinct for session, opaque value)

2

2

uint16

Session ID (random number, e.g. seconds since epoch % (2^16-1))

4

4

uint32

Sequence number (will wrap around to zero after reaching (2^32-1))

8

1

uint8

Current fragment number (0-based)

9

1

uint8

Max fragment number (0-based)

10

2

uint16

Object ID (distinct for session, opaque value)

12

4

uint32

Last published sequence number of the encoded object (identified by object type + object id)

16

[0;1400]

Encoded payload (may be empty, e.g. heartbeat)

Features#

Object Type and Object ID#

The object type can be used to filter updates without having to decode the payload.

The object id may be a useful key when caching data in memory.

  • Subscribers may use object type and id for caching purposes for the duration of a publisher session.

Warning

Object id’s are transient and only within a session. You should clear any lookup tables upon detecting a new session id or sequence number reset.

Session ID#

At start-up, the publisher will generate a random number to identify the session.

  • Subscribers must monitor the session id and reset internal caches when this value changes.

Sequence Number#

Packets are associated with a sequence number (an incrementing value that wraps around when it reaches 2^32).

Each channel will have its own sequence number.

  • Subscribers must use the sequence number to possibly re-order packets and to detect packet loss.

Last Sequence Number#

For an object type and id, this is the sequence number from last published message to the incremental channel.

The purpose is two-fold

  • Detect if packet loss could be associated with an object type and id

  • Correlate updates between the snapshot and incremental channels (needed when initializing certain objects).

Fragments#

There are two fields to manage fragments: current and maximum fragment number.

  • There can be a maximum of 256 fragments.

  • The total length of the message is known when current fragment number equals maximum fragment number: the total length is then current fragment number * max(payload) + len(payload).

Encoding#

The payload is encoded using the SBE protocol. The schema is made available from the roq-sbe-codec package.

TODO#

  • Describe logic to re-assemble message from fragments

  • How to correlate incremental and snapshot? (Like UDP? Or gateway’s sequence number?)

    • Snapshot not yet implemented

    • Do we need different snapshot channels for reference data vs MbP ?

  • Do we need the option to mark stale and only reset if a sequence number was truly lost?

  • Are we bandwidth constrained? The Roq API uses fairly wide fixed-length strings…

    • CME uses an integer to represent exchange/symbol, but they also “own” the universe… we have a dynamic universe

  • Only publish top N price levels for MbP?

    • We can maximum transmit ~35k

    • Each price level use 32 bytes

    • That is ~11k price levels for the sum of bids and asks