Status Update

This blog is an attempt to briefly describe the past 7 years of building software solutions for Roq GmbH.

I think it’s fair to say that a lot has been achieved. The product has much to offer to those savy users who appreciate the value of “library” and “modular” solutions.

One thing I always have to explain is that it is all about building the product. Not about putting the company on a growth trajectory. This goes against all wisdom. Few really understand what I have been doing and, maybe more importantly, why.

But I just love the programming challenges coming from the real problems faced by real clients. Give me such challenges and I will find the best possible solutions. (Given my experience and what I’m able to invent, of course.) I can’t explain it any more simple than that.

I also get the feeling that all Roq’s clients agreed with my approach. There were never any attempts by me to convince anyone to use the Roq solutions. I always explained all the pros and cons, even if I ended up promoting other vendor solutions.

The clients always came to me after having followed the work on GitHub for a while. Sometimes years, even… This is how I like it: It’s all about quality of the software.

Every client has also been able to work very closely with me. I have listened to all feedback. Issues were always resolved quickly. Together we looked for generic solutions. (This last point is very important: A cornerstone was always that every client should feel that their IP is treated with respect.)

Without these dedicated clients, Roq would not be able to offer the variety of solutions which exist today.

Finally, this year was one of the biggest personal challenges of my life. I was forced to change and I felt my future looked very uncertain for a while. At times, it seemed hard to even consider a future. This was difficult because I had never before questioned the support from my closest ones.

I don’t think any of the existing clients noticed what I went through: I was still responsive to any and every client request. But there were periods were I questioned everything, including the continuation of Roq.

This summer was a lot of introspection and it led me to remember how Roq started. I made a conscious choice to go back to the roots and mainly work on projects that align with the original vision. I also understood that the recent years had taken me in a different direction which, although exciting, were not really aligned with those early ideas.

Now I know that Roq will continue. The focus is more clear. It is what I love to do. And I will not rest until the product is perfect.

The following will explain the journey in more detail, with a focus on the bigger milestones.

Hans

FEMAS API

In the beginning there was an idea to build an easy-to-use C++ library for market access using the FEMAS API, a library used to directly access Chinese financial exchanges leveraging a broker’s existing infrastructure. Roq’s first client wanted to use this new software to self-host his own trading strategies.

The practical reason behind this request was that it was common to ship a shared library to a broker and have them execute the trading strategies. Although not necessarily a reality, it included the risk of the broker front-running or reverse-engineering the trading strategies for own gain.

After a few software iterations, it became clear that it was desirable to implement a client-server model so multiple trading strategies could access the same market (and account) through the same connection. A bit like multiplexing.

This design also allowed the client to implement an interface (Roq’s C++ API) which could either connect to live gateways or be used by a simulator to replay historical events.

From this design, the idea of the gateway event-log was born: Allowing 100% accurate replay of events.

For efficiency reasons, the client-server communication protocol was based on an initial snapshot followed by incremental updates. The implication was that a recovery protocol had to be implemented by each client to initialize state after a reconnect. This recovery protocol was quite similar to the multicast protocols typically used by financial exchanges.

Roq’s C++ API has always leaned heavily on the protocols used by FIX and CME. This was motivated by a longer term plan to support many exchanges (and even brokers). To this day, we remain focused on aligning new interface designs with existing exchange protocols.

The earliest version of the software used Unix domain sockets for communication between connected components. Although reasonable speed could be achieved with a simple setup, the design was not ideal, had scaling issues and would never be able to achieve low-latency. It turned out to be almost impossible to remove the jitter caused by the the kernel.

Unfortunately, this early project only lasted a year after which the client decided to terminate the contract. I lost market access and were forced to rethink the scope.

Pivot to Crypto (R&D)

Many lessons had been learned. This was a good time to revisit the software design.

The most important lesson was that low-latency could never be achieved when starting from the top (the interface) and then working down the software stack. It had to be the other way around: Latency had to be considered at every step when designing all parts of the trading system. From the ground up.

The software were then completely rewritten, starting with a shared-memory based client/server communication protocol. This from the realization that a low-latency solution should be able to avoid the kernel entirely (in the hot path, at least).

The C++ interfaces also saw a radical upgrade now relying on the most modern C++ features to avoid allocations and to enable better software design.

The software rewrite was done during the Crypto bubble in the years prior to the collapse of FTX. Every exchange started to offer easy-to-use JSON REST/WS based API’s. Everybody wanted “bots”.

It was a good time to pivot Roq to Crypto!

The initial gateways focused on supporting FIX (Deribit, Bitstamp, …) and JSON (FTX, Binance, BitMEX, Coinbase PRO, …). It was good to get exposure with different protocols: It helped shaping the tools now used for Roq’s auto-generated protocol parsers.

Roq’s C++ API was publicly available on GitHub during the entire rewrite. It turned out that some people were closely following the progress…

FIX Protocol

Roq’s first big client had existing trading strategies which they wanted to apply to Crypto markets. The challenge they faced were that their vendor solution (Tbricks/Itiviti) only supported financial exchanges. However, this vendor solution had very strong FIX capabilitites.

Roq was therefore asked to implement a FIX bridge between Roq’s C++ API and the vendor solution.

We started by supporting market data and quickly expanded to support many exchanges. The challenges were mostly about following standard FIX conventions and work around FIX for some special cases (e.g. to support funding rates).

However, order management turned out to be a tough challenge for many reasons.

The biggest challenge was that Crypto exchanges typically adopted a combined REST and WebSocket based protocol effectively exposing order updates to a “race” between the two connections. This is a particular difficult problem to solve when considering the FIX protocol which requires very strict update sequencing based on ClOrdID / OrigClOrdID rewrite (modify or cancel an order).

A few attempts were made before Roq’s C++ API settled on a version based request management solution. This allowed good communication between the gateways and the FIX bridge: Almost all FIX state changes could be supported.

There was reasonable support for the simple use-cases and we had just started testing order modifications when FTX collapsed… Although it took some time after, this event effectively terminated the client’s project.

CME

The “Tbricks/Itivity” client had access to CME’s multicast feeds and we did some preliminary work to support CME market data.

This prompted another client to contact Roq asking about further developing the gateway to support Treasury futures with a particular focus on suporting Level 3 (aka. Market by Order).

Roq’s CME gateway supports MDP3 full-depth multicast feed and has preliminary iLink order management support (tested and certified, but so far not used in production).

Later developments include support for importing Databento’s .pcap files to support historical simulations.

Ideas have been floated around also supporting CBOE, Eurex and maybe ICE. Even though this would be very good for Roq, these projects have unfortunately not yet materialized.

Order Routing

Roq’s biggest client yet is a broker who invested in a greenfield project to offer their clients low latency order routing.

We spent considerable time planning, designing and prototyping the required solutions.

In particular, the main ideas were centered around offering a standard FIX based protocol for order management and a SBE/multicast based protocol for market data. The broker could then offer order routing with latency close to DMA.

For Roq, this meant “dynamic order routing” which deviated a bit from existing software design.

The initial work involved building a FIX proxy solution to support horizontal scaling in front of the FIX bridge. The dynamic order routing could then be implemented between the proxy and the bridge.

Other challenges involved low-latency persistence of the client’s ClOrdID. This is a non-trivial problem and the idea of a distributed “journal” was born.

Technically, the gateways could wait for K (< N) journals to respond before releasing an order action to the exchange. A bit like combining RAID ideas with latency hiding because not all journals would experience the same latency.

Position management also became a priority because many (real) clients would trade on the same accounts. New ideas were developed with an external position manager solution communicating with gateways to support the gateways in “approximating” position updates with zero latency when observing exchange fills. This was achieved using sequence numbers thereby abstracting out the latency of the position database.

We had all the component built and deployed for client-testing when the project was terminated. Although the solutions were a bit rough at that time, it was all very close to being ready…

The positive spin is that Roq now has solutions to support horizontal scaling.

Algo Framework

Although the industrial-scale projects were exciting, recent reflection has convinced me that I somehow lost track of those original ideas from where it all started. In particular, support for algo development and historical simulation were de-prioritized during recent years.

With no current big project, my focus has been to find back to those roots from where it all started.

At first I started experimenting with SLURM for distributed simulations which has been asked by clients from time to time.

It quickly became clear that testing would benefit from some almost finished trading strategies.

This gave the idea to a new algo framework which will include reference implementations for some common and well-known trading strategies.

Important

The reference implementation are NOT meant to be used for real trading!

You should ONLY use the reference implementations as the basis for your own development!

After a while it also became clear that, although very powerful, SLURM can never be a good solution: It is just too complex for the average user to install, configure and maintain.

Then came the idea of using Python to orchestrate parallel simulations using the multi-processing features. With modern CPUs everyone now has access to many cores on their workstation, sometimes even 100s of cores.

Current work is around better supporting Python (and Jupyter notebook) to make it easier to implement, debug and optimize trading strategies.

A template project is currently being implemented to support all aspects from R&D to deployment.

Where to next?

Roq will continue to focus on the smaller funds and startups where there is a desire to outsource the software needed to standardize market access. This is a cost-effective option compared to writing (and, more importantly, maintain) this software themselves.

The tooling around the market access solutions should to a high degree focus on improving the strategy development experience. The recently developed Algo framework is an emphasis on that priority.

Bigger projects may be undertaken from time to time. But the focus should never override the longer term plan to build easy-to-use tools for the individual quant developer.

Although market access solutions are of the highest production quality, at the core, Roq builds libraries and is primarily R&D focused company.

Joint Venture opportunities exist for those who can see (and want) the opportunity to monetize Roq’s solutions. This was always an option, but the right partner has not been found yet.