We are always grateful for community contributions back to Timeplus Proton. This week, we'd like to highlight a significant contribution from Marvin Hansen, Director of Emet-Labs, a fintech research company specializing in applying computational causality to financial markets. Marvin built a Rust client for Timeplus Proton, and it is now available at: https://crates.io/crates/proton_client.
In the post below, Marvin explains the issues he encountered building real-time streaming analytics in Rust, and what brought him to Timeplus Proton. Thanks again for the contribution, Marvin!
Introduction
I ran into multiple database issues while working on a demo project that explores real-time streaming analytics in Rust. Still, the most unexpected source of problems came from the prevalent separation between data-at-rest in tables and data-in-motion via streams. The longer I thought about the dichotomy, the less it made sense. Granted, the project was all about analyzing streams of historical trade data, but when you think about it, is there a discernible difference between a data stream from historical data and a live data stream from an exchange? Probably not. Also, the next item on the project agenda is the addition of real-time data streams. In response, I started looking for a new data solution to help me converge data at rest and data in motion in one system. Timeplus Proton came to the rescue.
Â
What is Timeplus Proton?
Â
Timeplus Proton is a new converged data solution that unifies streaming and historical data processing. It is fast, scary fast. It already has blueprint solutions for real-time market data streaming or post-trade analytics and empowers real-time machine learning. All of these are on my to-do list, so what’s the catch?
There was no Rust client; therefore, I built one.
Â
Luckily, Proton was built on the shoulders of giants, this time atop ClickHouse, and that means I may lean on an existing ClickHouse client to get started. I got lucky, indeed. By wrapping up the existing ClickHouse Rust client, I put together the first Proton Client for Rust. Because the new Rust Proton Client wraps the underlying ClickHouse Client, you can still use it instead if you need more fine-grained access to specific operations not yet covered in the Proton Client.
Â
However, this is the first version of the Rust Proton Client. It is fully async and includes tests, code examples, and documentation, but it also comes with some caveats:
Caveats
The client uses HTTP instead of the more efficient native TCP protocol.
There are two different ports for historical (table) and stream data queries:
8123 is the port to run DDL and queries in historical mode.
3218 is the default port to run DDL and queries in streaming mode.
You can still wrap the stream with the table function to query them in historical mode.
Rust types like Time, Decimal, and Enums require a touch or two to make them work. I will add more code examples over time.
Async is currently implemented using the Tokio runtime. If you need Async for another runtime, please fill out an issue.
Core Features
Despite its early stage, the Rust Proton Client already covers all the core features:
Insert – For a single insert
Inserter – For bulk inserts
Query
Fetch
FetchAll
FetchOnce
FetchOptional
Â
Rust with Streaming SQL
Streaming SQL works well, although you must use port 3218 for that. Please refer to the official documentation for all ports available in Proton.
When you connect to Proton on port 3218, you can query tables and continuous streams. The only difference is that you must add the table keyword to the query; otherwise, Proton assumes you are querying a live stream. The concept guide is worth reading for more details on non-streaming and streaming queries on live-stream data. Speaking of live-stream data, you can create an internal stream as a continuous SQL query or connect directly to an external stream like Kafka. If you need a connector that is not yet available, you can either contribute one if you're an avid C++ programmer or write a simple web service that connects to your live data feed and inserts into a proton stream. Internally, you place a continuous SQL query to output your data in the shape you want.
Â
Available Now
The Early Access version of Rust Proton Client is available on GitHub and crates.io. Feel free to test drive the client, report issues, or join the Timeplus Community Slack for help.
Â
For my project mentioned at the beginning, I am already migrating away from my previous database to Timeplus Proton. I will share my experience adopting Proton as a unified solution for historical and real-time data in Rust in a later blog post.
About the Author
Marvin Hansen is the director of Emet-Labs, a FinTech research company specializing in applying computational causality to financial markets. He is the author and maintainer of the DeepCausality project. You can connect with the author on Maven, the world’s first serendipity network.