CHALLENGES:
Storage and Compute Costs
Storing voluminous and complex JSON documents in Kafka and subsequently in OLAP data warehouses lead to substantial storage and computational costs.
Complex Data Structures
The GitHub schema often results in verbose JSON documents with complex, nested structures, potentially spanning hundreds of lines per message. Parsing and filtering these documents efficiently and accurately presents a significant challenge.
Continuous and Incremental Analytics Demand
Batch ETL and traditional BI typically cannot support continuous and incremental processing.
SolutioN
Timeplus seamlessly connects with Kafka API and read incoming and existing messages. You can run SQL query directly via the Kafka External Stream without saving any data in Timeplus.
To avoid the high cost of storing billions of events in Kafka and also to improve the query performance, it's recommended to create Materialized Views in Timeplus to save Kafka message locally and apply optimized columar storage or index. Materialized Views also enable incremental updates to turn BI reports from minutes to sub-seconds.
Step 1
Call GitHub API to get real-time events and push to Kafka
Step 2
Create External Stream in Timeplus to read data from Kafka
Step 3
Create Materialized Views to process and analyze the dataCreate Materialized Views to process and analyze the data
Step 4
Create real-time alerts or visualization in Timeplus or Grafana
SEE IT IN ACTION:
data lineage:
Imagine you have millions of GitHub events, such as commits, pull requests, issues, and releases, streaming in real-time. You want to monitor these events to gain insights into rising new projects, popular projects, and developer contributions. Querying them directly from Apache Kafka can be challenging due to the complexity of the data and the need for efficient processing.
Timeplus provides a solution to this problem by allowing you to create materialized views that can process and analyze these events in real-time, enabling you to gain valuable insights quickly.