Timeplus Enterprise 3.3: Converged MPP and Cloud-Native at Production Scale, Eliminating Trade-offs Between Low-Latency, Scalability and Visibility

Timeplus Team
4 days ago
10 min read

When we announced Timeplus 3.0 last October, we introduced an industry-first converged MPP and cloud-native architecture. It was elastic, featured per-stream aggregated and disaggregated capabilities, and was designed to scale from the edge with shared-nothing efficiency to 50–100GB/s clusters using zero-replication storage. Our promise was clear: a real-time, incremental compute pipeline that eliminates the trade-off between low latency and massive cloud scale.

Per-Stream Shared-Nothing & Shared-Storage, the architecture from Timeplus 3.0

Most streaming platforms bolt real-time processing onto a storage layer built for something else — a data warehouse, a lakehouse, or a cache in front of object storage. Timeplus’ cluster was designed the other way around: node roles, write paths, and the query engine are all built specifically for continuous, elastic stream processing.

Three disaggregated roles, one binary. A cluster runs a single binary, timeplusd, where roles are pure configuration — metadata, data, or compute, and a node can hold more than one.

Metadata nodes own topology, streams, materialized views, tasks, alerts, and credentials, plus load balancing and scheduling.
Data nodes persist and replicate production data and can run compute directly against it
Compute nodes hold no data at all. They exist purely to run materialized views, tasks, and alerts, which makes them disposable and elastic by design, built to sit behind AWS Auto Scaling or Kubernetes HPA for spiky workloads.

Streaming queries don't shuffle across the cluster. Historical queries work like a conventional distributed database — an initiator fans a plan out to replicas and merges what comes back. Streaming queries take a different path on purpose: shuffling, joins, and aggregations happen entirely on the initiator node, which lets Timeplus skip cross-node watermark coordination, one of the more expensive problems in distributed stream processing. Hybrid joins, hybrid aggregations, checkpointing query state in cloud storage, and scheduling that spreads queries across compute nodes for balance are what make that design scale instead of just being fast in a demo.

Zero-Replication NativeLog: the same log, two storage modes, chosen per stream. Every write lands first in NativeLog, the Multi-Raft journal underneath the cluster. Local Disk mode is classic MPP, replicated node to node. Cloud Storage mode makes NativeLog diskless: writes batch in a ZeroReplicationClient, upload to S3 once they hit a size or timeout threshold, and only the file's path — not the data — replicates across the Raft group.

Architecture deep-dive: elastic ingest, NativeLog, and the local disk tier, from Timeplus 3.0

In many architectures, the storage trade-off is significant. EBS offers sub-2ms latency but is often 7-10x more expensive than S3 when accounting for replication. Conversely, S3 is cost-effective, but network round-trips can push p99 read latencies beyond 100ms. Most platforms simply force users to pick a single point on this cost-performance curve.

ClickHouse Cloud optimizes disaggregated storage (SharedMergeTree) primarily for query indexing, lacking an equivalent focus on continuous, low-latency transformation pipelines.
Databricks layers dedicated serving engines (like Lakehouse/RT) alongside existing lakehouse components.
Materialize and RisingWave push streaming state directly to object storage.

Each is a reasonable answer to the same design question: “Where does real-time data actually live?”

Timeplus uses a per-stream policy instead of a platform-wide pick. Hot streams stay on local storage for data locality, purposely built for latency-sensitive use cases. Historical data is tiered to S3 based on a TTL. The same cluster executes the same SQL across both tiers without secondary bridge systems.

Timeplus Enterprise 3.3 hardens this foundation for heavy production workloads. It removes key performance and scaling bottlenecks while improving operational visibility. Today, leading global enterprises run mission-critical pipelines on Timeplus:

Network & Security Detection Use Case: One deployment tracks hundreds of thousands of IPs across 10 million events per second with sub-second latency.
Gaming Operations Telemetry Use Case: Processing and transforming over billions events, 30TB of daily data, peaking at 700MB/s. These insights previously took hours to reach operational teams. Replacing a complex stack of Databricks, Flink, and Kafka with a single timeplusd binary delivered second latency while significantly reducing TCO through S3 storage tiering.

Read on to learn more about the features we’ve shipped:

Python UDFs now execute genuinely in parallel across query threads.
LEFT ANTI JOIN: emit each left-side event whose join key has no match in the current snapshot of the right side.
Preserving partitioning across query boundaries
Shard pruning for two common predicate shapes
Cluster balancing by cost, not by headcount
Faster recovery, more resilient startup, and offline stream backup
Other operational improvements

Python UDFs Go Truly Parallel

One lesson from large production deployments is that more and more business logic now lives inside the streaming SQL pipeline, AI inference, fraud scoring, entity resolution, geospatial enrichment, policy evaluation, and custom business logic. Python UDF should never become the bottleneck.

Until 3.3, that escape hatch had a hard ceiling. Every Python UDF call in the entire server was serialized on one global interpreter lock. Core count didn't matter. Query parallelism didn't matter. A Python-heavy pipeline ran at roughly one core, and the only ways around it were unappealing: rewrite the logic in SQL and lose expressiveness, or split the workload across more servers and pay for capacity you weren't using.

3.3 upgrades the embedded runtime from CPython 3.10 to CPython 3.14, and Linux x86_64 builds ship the free-threaded (no-GIL) interpreter. Python UDFs now execute genuinely in parallel across query threads.

Check out a sample SQL here:

The practical effect is that Python stops being the thing you design around. Pipelines you'd previously have kept deliberately thin — because you knew the UDF would become the bottleneck — can now carry real work.

Two supporting changes turn Python UDFs from "works" into "production-grade."

Initialization hooks. A UDF can now declare a one-time init function: the right place to load a model, open a client connection, or read credentials, instead of relying on import-time side effects or a lazy-init check on every single call. Parameters can be passed inline, or pulled from a named collection so that API keys never appear in the UDF body or in SHOW CREATE FUNCTION output.

Loading a model once per module instead of once per call is often a bigger practical speedup than the parallelism itself.

Declarative package management. Point the server at a requirements.txt hosted in S3, and every node installs what's missing at startup and re-checks periodically. This is aimed squarely at Kubernetes deployments with ephemeral compute nodes, where imperative SYSTEM INSTALL PYTHON PACKAGE calls were thrown away on every pod reschedule and keeping node environments consistent was a losing battle. It's fail-open (an unreachable file never blocks startup), idempotent, and install-only by design — pin your versions and every node converges identically.

And a flush hook for Python sinks. Python external streams used as a sink can declare a flush_function_name, called on every checkpoint and once on graceful close. If your sink batches rows internally to amortize an API call, you now have a reliable signal to push the buffer — buffered rows are no longer lost on shutdown.

One thing to plan for: free-threading means multiple query threads can enter the same Python function concurrently, so module-level mutable state in your UDFs now needs to be thread-safe. You'll also need to reinstall packages after upgrading and confirm your dependencies ship Python 3.14 free-threaded (cp314t) wheels.

Streaming SQL: Express More, Scan Less

LEFT ANTI JOIN: The no match pattern, finally direct

A surprising share of real-world streaming logic is about absence rather than presence. Orders with no corresponding shipment. Device IDs not in the registry. Transactions with no reference record. Logins from a machine that isn't on the allowlist.

Expressing that in a streaming context used to require workarounds — outer joins with null filters, staged materialized views, or logic pushed out of SQL entirely into an application layer. 3.3 supports LEFT ANTI JOIN directly: emit each left-side event whose join key has no match in the current snapshot of the right side.

It works across all combinations of append streams, versioned_kv streams, and mutable streams on either side, in both streaming and historical modes. The semantics are one-directional enrichment: the right side is a snapshot of the left stream probes, and a right-side row arriving later doesn't retract an already-emitted output — which is exactly the behavior you want for detection and alerting.

For anyone building anomaly detection, compliance checks, or reference-data validation, this collapses a multi-stage pipeline into a single join.

Preserving Partitioning Across Query Boundaries

SHUFFLE BY and PARTITION BY declared inside a CTE, subquery, or view are now visible to the outer query. Previously the partitioning was forgotten at every boundary and the data was re-sharded all over again — pure wasted network and CPU on exactly the kind of layered query that well-organized SQL produces.

There's no new syntax and nothing to enable. Existing queries simply stop doing redundant work — including views over views and materialized view pipelines. The optimization is strictly correctness-preserving: it applies only when the inner keys cover the outer query's needs and nothing in between could break the partitioning. In every other case, the query re-shards exactly as it did before.

This is the best kind of change: your SQL doesn't change, and it gets faster.

Shard Pruning for Two Common Predicate Shapes

Queries on multi-shard streams now skip shards for multi-column (tuple) IN predicates, which works out of the box:

And, opt-in, for IN with a subquery — the subquery is evaluated first and its result drives the pruning:

Both shapes previously scanned every shard, every time. On a wide cluster with a well-chosen sharding key, that's the difference between touching one node and touching all of them. Pruning stays conservative — NOT IN, NULLs, oversized result sets, and streaming subqueries all fall back safely to a full scan.

Cluster Balancing by Cost, Not by Headcount

Running dozens of nodes continuously exposed another challenge: clusters gradually become imbalanced after workload changes or failovers, leaving expensive hardware underutilized. You size a cluster for its total capacity, but you only get that capacity if the work is actually spread across it. Two things were getting in the way.

The scheduler couldn't tell heavy work from light work. It counted materialized views. A node running three large stateful aggregations and a node running three trivial pass-through views looked identical to it — so heavy views piled onto the same machine while trivial ones spread out politely.

3.3 makes placement class-aware. Every scheduled materialized view is classified (explicitly by tag, by declared resource weights, or automatically as aggregation / join / plain) and each class is spread evenly across eligible nodes.

A background pass every 30 seconds gently corrects residual imbalance, and it only ever makes strictly-improving moves, so it cannot thrash a cluster that's already fine. Placement is fully observable through system.execute_stream_assignments, which now exposes class, tag, weights, and node preference.

Work concentrated after failovers and never came back. When a node goes down, Raft leadership for replicated materialized views and stream shards moves to the survivors — and it used to stay there permanently, even after the recovered node rejoined and sat idle. You'd end up running a three-node cluster at the throughput of one busy node, with no way to fix it short of a rolling restart.

The reliable rebalancer now handles both. It got separate memory and CPU overload thresholds (memory pressure becomes dangerous well before CPU pressure does), it will never push the receiving node into overload to relieve the sender, and it can optionally rebalance on pure leader-count skew — which is precisely the post-failover situation. It can now also balance stream shard leadership, conceptually like Kafka partition-leader rebalancing, weighted by each stream's recent throughput.

It deliberately makes one move per cycle and re-measures before the next, which absorbs the new leader's warm-up and prevents oscillation.

And when you'd rather drive it yourself, three new commands give you direct control — useful for draining a node before maintenance, cleaning up after dropping a batch of views, or pinning a specific view where you want it:

The net effect is simple: you get the capacity you already bought, and you keep it after an incident.

Faster Recovery, More Resilient Startup, and Offline Stream Backup

Timeplus 3.3 includes several engine-level improvements that make startup and recovery more reliable and predictable for production deployments with large persisted state, tiered storage, constrained memory, interrupted workloads, or partial failures.

Faster, more reliable startup: Primary key indexes now load lazily rather than for every part at startup, bounding startup memory on large deployments. A single table that fails to start is now contained and logged instead of aborting the whole server, improving isolation between independent storage objects.
Smarter recovery under memory pressure: Materialized views also behave more defensively during reboot recovery. Before rebuilding, they check for memory headroom and defer with backoff when the node is already under pressure. This helps avoid recovery storms where too many materialized views compete for memory at once, slowing or destabilizing the node during restart.
New offline stream backup and restore: Timeplus 3.3 also introduces a new timeplusd stream CLI for offline backup, restore, and repair of stream historical data. It supports sequence-aware recovery, automatic rollback stashing, and refuses to run against a live server, turning stream recovery from an improvised repair task into a procedure operators can document, test, and repeat.

Together, these changes reduce startup memory pressure, contain localized failures, and give operators a clearer recovery path when something goes wrong.

More Enhancements in 3.3

Timeplus 3.3 also includes several operational improvements that make continuous streaming workloads easier to run.

Defaults now better match how streaming ingestion actually behaves. Streaming writes small blocks continuously, so part counts legitimately run higher than in batch-loading systems. The “too many parts” thresholds have been re-tuned: the delay threshold moves from 150 to 1000 parts, and the rejection threshold moves from 300 to 3000. This reduces artificial insert delays and rejections in healthy pipelines.
Merge behavior is also more configurable. Tiered storage policies gain prefer_not_to_merge, so cold S3-backed volumes can be excluded from background merges. Vertical merge is restored for wide tables to reduce peak merge memory, and merge concurrency is now tunable through config.yaml.
Kafka connectivity is quieter and steadier. Broker outages no longer flood server logs with thousands of repeated warnings per second, and consumer stall detection avoids cascading redundant consumer recreations on multi-partition topics.

Additional enhancements include:

New SYSTEM STOP MOVES and SYSTEM START MOVES controls for pausing and resuming background data movement during maintenance or recovery.
New offsets-only checkpoint mode for pipelines whose state is cheap to rebuild from source offsets.
NATS NKey authentication support, plus support for setting JWT and seed content through settings.
Historical-store size reporting now uses a TTL cache, avoiding repeated expensive S3 listing work in object-storage deployments.
Additional metrics and memory-accounting cleanup, including fixes for phantom memory tracker amounts and high-frequency metric-log overhead.

Before You Upgrade

A few changes are worth planning for before moving to 3.3:

Rebalancer configuration moved from scheduler.rebalancer.* to a top-level rebalancer: section.
The Python runtime jump to 3.14 requires reinstalling UDF packages, verifying 3.14 wheel availability, and reviewing UDF thread safety.
NATS NKey authentication now requires the new nats_nkey setting alongside nats_nkey_seed.
Mutable streams reject secondary indexes that duplicate a leading prefix of the primary key — such DDL will fail until the redundant index is removed.

Full details are in the 3.3 release notes, and our team is glad to walk through an upgrade plan with you.

The 3.3 Takeaway

Timeplus 3.3 is about removing ceilings. Python UDFs run in parallel instead of on one core. Anti-joins, cross-boundary partitioning, and smarter shard pruning let you express more in SQL and scan less to answer it. And the cluster balances by what work actually costs, then reclaims that capacity after a failover instead of stranding it.

For teams building real-time analytics, operational monitoring, alerting, fraud detection, IoT pipelines, security analytics, or event-driven applications, Timeplus 3.3 provides a more resilient foundation for production streaming workloads.

Timeplus Enterprise 3.3 is built for systems that need to stay online, recover cleanly, rebalance intelligently, and keep processing data as conditions change. Less designing around the platform, more building on it.

See our full release notes here: https://docs.timeplus.com/enterprise-v3.3

Ready to try Timeplus Enterprise? Get started with a 30-day free trial:

https://timeplus.com/download