SBE¶
Overview¶
Format¶
A header occupies the first 16 bytes.
The header contains fields to manage and detect
Reset (session id)
Re-assembly (sequence number, current fragment number, max fragment number)
Offset |
Width |
Type |
Description |
|---|---|---|---|
0 |
1 |
|
Control (bit fields)
|
1 |
1 |
|
object type (distinct for session, opaque value) |
2 |
2 |
|
Session ID (random number, e.g. seconds since epoch % (2^16-1)) |
4 |
4 |
|
Sequence number (will wrap around to zero after reaching (2^32-1)) |
8 |
1 |
|
Current fragment number (0-based) |
9 |
1 |
|
Max fragment number (0-based) |
10 |
2 |
|
Object ID (distinct for session, opaque value) |
12 |
4 |
|
Last published sequence number of the encoded object (identified by object type + object id) |
16 |
[0;1400] |
Encoded payload (may be empty, e.g. heartbeat) |
Features¶
Object Type and Object ID¶
The object type can be used to filter updates without having to decode the payload.
The object id may be a useful key when caching data in memory.
Subscribers may use object type and id for caching purposes for the duration of a publisher session.
Warning
Object id’s are transient and only within a session. You should clear any lookup tables upon detecting a new session id or sequence number reset.
Session ID¶
At start-up, the publisher will generate a random number to identify the session.
Subscribers must monitor the session id and reset internal caches when this value changes.
Sequence Number¶
Packets are associated with a sequence number (an incrementing value that wraps around when it reaches 2^32).
Each channel will have its own sequence number.
Subscribers must use the sequence number to possibly re-order packets and to detect packet loss.
Last Sequence Number¶
For an object type and id, this is the sequence number from last published message to the incremental channel.
The purpose is two-fold
Detect if packet loss could be associated with an object type and id
Correlate updates between the snapshot and incremental channels (needed when initializing certain objects).
Fragments¶
There are two fields to manage fragments: current and maximum fragment number.
There can be a maximum of 256 fragments.
The total length of the message is known when current fragment number equals maximum fragment number: the total length is then current fragment number * max(payload) + len(payload).
Encoding¶
The payload is encoded using the SBE protocol.
The schema is made available from the roq-sbe-codec package.
TODO¶
Describe logic to re-assemble message from fragments
How to correlate incremental and snapshot? (Like UDP? Or gateway’s sequence number?)
Snapshot not yet implemented
Do we need different snapshot channels for reference data vs MbP ?
Do we need the option to mark stale and only reset if a sequence number was truly lost?
Are we bandwidth constrained? The Roq API uses fairly wide fixed-length strings…
CME uses an integer to represent exchange/symbol, but they also “own” the universe… we have a dynamic universe
Only publish top N price levels for MbP?
We can maximum transmit ~35k
Each price level use 32 bytes
That is ~11k price levels for the sum of bids and asks