A case study of Timeplus + Alpha Stream
By Jove Zhong and Hamilton Araujo
The financial services industry is no stranger to real-time analytics; no other industry has been so thoroughly transformed by the real-time data revolution. But even the most sophisticated enterprises can be slowed down by complex and cumbersome implementation. In this blog, we’ll discuss how leading fintech player Alpha Stream deployed Timeplus to quickly upgrade its real-time analytics capabilities.
But first, a bit about us:
Alpha Stream is a leading real-time event processing technology company that provides corporate solutions to financial institutions in Brazil. Together with leading banks, brokers, and the Brazilian Stock Exchange, Alpha Stream is a pioneer in providing real-time trading capabilities to leading regional financial institutions.
Timeplus is is a purpose-built real-time analytics platform which solves enterprises’ need for a powerful turnkey solution that turns streaming data into business value. Timeplus provides dynamic schemas for real-time analytics, bringing sub-second low latency and high throughput to streaming data processing. This empowers enterprises to extract substantial value from data before it goes obsolete.
Core Challenge
Developing an event processing application and still adding analytics capabilities can be difficult and complex to implement and sometimes hard to maintain and extend.
The Alpha Stream team saw Timeplus had the potential to help them scale their real-time analytics solutions quickly. Working with Timeplus, Alpha Stream is now able to efficiently push real-time market data to Timeplus via REST API and Kafka sources, on a continuous basis. They then are building real-time pricing models with streaming SQL, and sending the output to a downstream Kafka topic for further real-time financial automations.
This diagram helps illustrate how the “magic” of real-time financial data processing is made using Timeplus:
The market data is pushed directly to Timeplus via REST API to minimize latency (Timeplus is able to produce single digit millisecond end-to-end streaming processing). To integrate with existing services, some data are also pulled from Apache Kafka topics.
Once the market data arrives in Timeplus, it is ready within a few milliseconds to be filtered, aggregated and routed.
Alpha Stream users can explore real-time data with a few clicks to understand data patterns and write streaming SQL to define their real-time pricing model.
The processed data can be visualized as a real-time dashboard, or sent to a new Kafka topic for further processing or automations.
The benefits of this solution are:
Ultra-low latency: For the first time, Alpha Stream can process data within a few milliseconds. This empowers Alpha Stream to provide best-in-class services such as dynamic asset pricing, sentiment analysis, and correlation calculation.
Highly customizable real-time pricing model with very low maintenance effort: The pricing model can be expressed with SQL, eliminating the need for microservices, with their high development and maintenance costs.
Unparalleled high throughput: Timeplus can process 10 million real-time events per second, with much lower hardware requirements than traditional Java-based data processing technologies.
Intuitive user interface: From data onboarding to stream exploration, from ad-hoc streaming queries to high frame-per-second live dashboards, in every aspect of Timeplus UI and API, Alpha Stream users can interact with real-time data and fine tune the real-time pricing model with little additional work.
Real-time Pricing Model
In this simplified example, we summarize the model used by Alpha Stream. One goal is to get the latest price for symbol-A and the latest price for symbol-B, then calculate a new price to try to balance them (you can imagine one as the bid from buyers and one as the bid from sellers. In this case, the system needs to automatically set a price which can make the transaction happen.)
Using Timeplus, Alpha Stream can run streaming queries with auto-complete and charting support. Users don’t need to explicitly build a data pipeline or submit a processing job. They can simply just run a query, get the results in the next second, visualize them, and refine the query to improve results.
The streaming query is live and long-running and won’t stop unless the user cancels it. For example, running this query will filter all market results for particular symbol and volume:
SELECT * FROM MarketData WHERE symbol='A' AND size >100
In the above scenario, we can create real-time pricing with SQL:
SELECT
now() AS ts,
map_cast(group_array(symbol),group_array(p)) AS lookup,
lookup['A']/2+lookup['B']/2 AS price
FROM (SELECT symbol, latest(price) AS p
FROM MarketData WHERE symbol in ('A','B') GROUP BY symbol)
HAVING price!=lag(price)
Note:
In the subquery we list all live market data for symbol A and B, and run a special aggregation function to get the latest price per symbol
Then we use group_array and map_cast functions to convert the data to a live lookup
Finally use the `a/2+b/2` as the simplified model to get the new price
The output of the streaming query will be sent to the downstream Kafka topic. Within the same second, the system is able to generate the best price to maximize the possibility of transactions.
Additional advanced streaming analytics functions enabled by Timeplus include:
Hopping window to show the moving average price for certain ticker
Create indicators like the Relative Strength Index (RSI) for a sentiment analysis on which stocks are overbought or oversold
Session window to identify active buyers and sellers and flow
Stream to stream join to run correlation calculation
Built-in streaming predictive analytics functions
User-defined functions (UDF) to enable customized functions and AI/ML models
Hamilton Araujo, Alpha Stream Managing Director, commented:
Timeplus is such an amazing system. We are able to simply plug the sources into Timeplus and start writing hot queries over the streaming data to get the results. No need to compile and deploy a code. Also we can direct the results to sinks to consume in dashboards or even combine into another analysis. This makes prototyping to deploy applications really fast.
Got Real-time?
Looking to build out your own real-time analytics capabilities? Check out timeplus.com to learn more and sign up for our private beta.