Timeplus' Elastic Autoscaling: 100 GB/s Real-Time Compute with Materialized Views
- Timeplus Team
- 14 minutes ago
- 6 min read
Real-time analytics has moved beyond simple stream ingestion. Modern data teams are not just collecting events and storing them for later. They are continuously filtering, joining, enriching, aggregating, alerting, and serving results while the data is still moving.
A single real-time application may include hundreds of materialized views, multiple derived streams, and downstream dashboards or alerts that all need fresh results with low latency. This changes the scaling problem.
In a traditional architecture, storage and computation often scale together. If the workload needs more processing capacity, teams add more nodes, even if the bottleneck is not storage. If the workload is spiky, they provision for peak load and pay for idle capacity later. If every continuous query requires hot standby replicas, high availability becomes expensive at large scale.
Timeplus takes a different approach: separate durable streaming state from elastic real-time compute.
Here is a Timeplus cluster running hundreds of materialized views sustained 102.1 GB/s total real-time processing throughput, with 69.9 GB/s read throughput, 32.2 GB/s write throughput, and 1.5 billion events per second. In this blog, we walk through the architecture that makes these metrics possible.
The Cluster Model: Metadata, Data, and Compute
A Timeplus cluster is made up of nodes running the same core engine, timeplusd. A node’s role is defined by configuration, and each node can serve one or more roles:
Metadata nodes manage the cluster topology and critical system metadata, including streams, materialized views, tasks, alerts, UDFs, dictionaries, and users. They also coordinate internal routines such as scheduling and load balancing.
Data nodes manage production data. They handle persistence, replication, and read/write operations. In many deployments, data nodes can also execute compute workloads.
Compute nodes are specialized for running materialized views, tasks, alerts, and other compute-heavy workloads. They do not store production data. That makes them elastic by design: they can be added when workload increases and removed when demand drops.
This separation is important. Real-time workloads often scale unevenly. A team may ingest a stable amount of data but suddenly add many new materialized views. Another team may see compute demand spike during business hours, then drop at night. By separating compute from durable data ownership, Timeplus can scale the part of the system that actually needs capacity.
NativeLog: The Distributed Streaming Foundation
Under the metadata and data layers, Timeplus uses NativeLog as its distributed journal and write-ahead log.
NativeLog provides the durability, replication, and consistency foundation for streaming data. In a clustered deployment, streams can be divided into multiple shards, and each shard is managed independently through its own Raft group.
When data is ingested into a sharded stream, Timeplus splits incoming batches based on the stream’s sharding expression. Each shard-level batch is written and replicated independently. Once the relevant shard batches are successfully committed, the ingest request can complete.
This gives Timeplus a distributed streaming foundation rather than a single-node execution model with clustering added later. Ingest, storage, and query execution can all operate against a sharded and replicated streaming substrate.
Real-Time Compute Through Materialized Views
Materialized views are the always-on compute layer in Timeplus.
A materialized view continuously consumes input streams, processes events, and writes results into an output stream. This makes it a natural unit of real-time computation: each view can represent a transformation, enrichment, aggregation, anomaly detector, or serving pipeline.
In the demo, generated device streams feed many parallel materialized views. Those materialized views process the incoming data and write aggregated results into per-network one-minute output streams. As the workload ramps, the number of running materialized views increases from dozens to hundreds.
At one point, the cluster is running 264 materialized views across 25 total nodes, with 19 nodes online and 3 data nodes online. The cluster reports 176 total CPU cores and continues processing at around 102.1 GB/s total throughput.
That is the key point: the benchmark is not just about writing data into a stream. It is about running continuous compute over streaming data at very high throughput.
A simplified version of the demo workload looks like the sample SQL below. One generated telemetry stream feeds a materialized view, and the view continuously writes one-minute aggregates into an output stream. The benchmark scales this same pattern across many networks and hundreds of materialized views.
High Availability Without Over-Provisioning Every Workload
For real-time systems, scaling is only useful if the system remains reliable. Timeplus supports two high availability models for materialized views, each optimized for different deployment needs.
The default model is Raft-based HA. In this mode, Timeplus places three replicas of a materialized view on different nodes. One replica acts as the leader and executes the query. The other replicas act as followers and receive replicated checkpoint state through Raft.
If the leader fails, a follower can be elected as the new leader, rebuild the processing pipeline, recover from the replicated checkpoint, and resume execution. This model is a good fit when fast failover is the priority and the workload is relatively steady.
The second model is scheduler-based HA. In this mode, a centralized scheduler monitors materialized views and reschedules them when failures occur. The materialized view checkpoints state to shared storage, and the scheduler can move the workload to a healthy node when needed.
This model is especially important for elastic compute. Because the materialized view state is externalized, compute nodes do not need to permanently own that state. They can run workloads while needed, then be replaced, drained, or removed. If a node fails, the scheduler can reschedule its materialized views elsewhere and recover from shared checkpoint storage.
For large-scale or spiky workloads with many materialized views, this avoids the cost of maintaining multiple hot standby replicas for every continuous query.
Zero Replication NativeLog and Shared Storage
Timeplus also supports a shared-storage mode for NativeLog, sometimes described as zero replication at the storage layer.
In the traditional model, streaming data is stored on local disks and replicated across nodes. This is a strong fit for latency-sensitive workloads and on-prem deployments where shared cloud storage may not be available.
In shared-storage mode, streaming data is written to S3-compatible storage. Instead of replicating the full data payload across local disks, Timeplus replicates the metadata needed to locate committed data. The actual data files live in shared storage, where they can be fetched by the nodes that need them.
This has several advantages for cloud-native scale:
it reduces cross-AZ replication cost
it lowers local disk I/O and bandwidth pressure
it allows multiple nodes to ingest through shared storage
it makes large-scale streaming query processing easier to pair with elastic compute nodes
Because this is configured per stream, a Timeplus cluster can support both models at the same time. Latency-sensitive streams can use local replicated storage, while massive-scale elastic workloads can use shared storage.
Why This Matters for 100 GB/s Real-Time Compute
The 100 GB/s result comes from combining these architectural pieces.
NativeLog provides the distributed streaming substrate. Sharding allows ingest and processing to scale across the cluster. Shared storage reduces the need to replicate all streaming data across local disks. Compute nodes provide elastic execution capacity. Scheduler-based materialized view HA lets continuous workloads move across the compute tier without tying state permanently to individual nodes.
The result is a real-time compute architecture that behaves more like cloud-native infrastructure: add compute when continuous processing demand rises, remove it when demand falls, and keep durable state separate from ephemeral execution.
This is especially useful for workloads such as:
high-cardinality observability analytics
real-time security detection
IoT telemetry aggregation
financial market data processing
clickstream and user behavior analytics
large fleets of customer-specific materialized views
In each case, the challenge is not only ingesting data. The real challenge is continuously computing over that data at scale.
Choosing the Right Scaling Model
Timeplus gives operators flexibility rather than forcing one architecture for every workload.
Use local replicated NativeLog when the workload is latency-sensitive, steady, or running in an environment where shared storage is not available.
Use shared-storage NativeLog when the workload is large, cloud-native, and benefits from lower replication overhead and elastic processing.
Use Raft-based materialized view HA when fast failover is the priority and the additional standby replica cost is acceptable.
Use scheduler-based materialized view HA when the workload includes many materialized views, dynamic compute demand, or autoscaling through Kubernetes HPA or cloud auto scaling groups.
That flexibility matters because real-time systems are rarely uniform. One cluster may need both low-latency pipelines and massive elastic processing jobs. Timeplus is designed to support both.
Conclusion
Elastic real-time compute requires more than adding nodes to a cluster. It requires a clean separation between durable streaming state and compute execution.
Timeplus achieves this through a distributed NativeLog foundation, separate metadata/data/compute roles, shared-storage support, and materialized view high availability models designed for different workload patterns.
In the demo, that architecture reaches more than 100 GB/s of real-time processing throughput while running hundreds of continuous materialized views. More importantly, it shows a practical model for scaling real-time analytics: keep the data durable, make compute elastic, and let continuous workloads move with demand.
Try Timeplus free for 30 days: See how Timeplus handles real-time ingestion, continuous materialized views, and elastic compute in your own environment. Start a free 30-day trial and build your first streaming analytics pipeline today.
Further reading in our docs: