Sigma is a Launch Partner for Databricks Lakehouse // RT
Building real-time analytics on a modern lakehouse has always carried a hidden cost. The infrastructure is powerful, the governance is solid, and the data is all in one place. Yet the moment a team needs sub-second query response at high concurrency, the standard playbook has been to reach for something else: stand up a specialized serving system, duplicate the data, manage a second platform, accept the governance gap, and pay twice.
That compromise is what Databricks Lakehouse//RT (real-time) is built to eliminate. Sigma is proud to have been selected as one of Databricks' launch partners.
Why “Fast Enough” isn’t enough for real-time workloads
For most analytical workloads, a few seconds of query latency is tolerable. Batch reports, quarterly business reviews, historical trend analysis: those aren’t real-time problems. A growing class of workloads demands something different.
These workloads include operational apps that refresh live for hundreds of users, product analytics that track user behavior as it happens, and customer-facing applications where stale data means a broken experience. They also include AI agent workflows that query a data store as part of an automated decision loop. These workloads don’t have the luxury of waiting. Running them on a lakehouse meant accepting a fundamental performance ceiling or engineering around it with external systems that created complexity faster than they solved it.
The result has been sprawl: data pipelines forking into purpose-built serving tiers, Unity Catalog permissions becoming incomplete because some data lives outside the governed system, and engineering teams maintaining infrastructure that exists purely to compensate for latency the core platform couldn’t address.
Lakehouse//RT is built to close that gap at the architecture level.
What is Databricks Lakehouse//RT?
Lakehouse//RT is Databricks’ new real-time data warehouse, built to deliver millisecond query performance directly on your lakehouse without data movement or separate systems. This is not an incremental improvement to existing infrastructure. The engineering team rebuilt the engine from scratch, purpose-built for operational analytics, BI and app serving, and observability workloads.
The core architectural shift is a native, non-blocking asynchronous execution engine. Where the prior architecture passed every query through a multi-step compilation and translation process that introduced overhead before execution even began, Lakehouse//RT eliminates that layer entirely. The result is an engine that holds low latency under load, at scale, and on complex analytical queries: the three conditions where purpose-built serving systems have historically broken down.
Two additional design choices set it apart from traditional warehouses. First, Lakehouse//RT auto-sizes compute automatically, so teams no longer pick a cluster size and hope it holds. Second, it scales incrementally by adding and removing individual nodes as concurrency changes, rather than doubling capacity in large increments. That means you pay for exactly what the workload requires, not the next tier up.
Lakehouse//RT also integrates with ZeroBus Ingest, Databricks’ low-latency ingestion layer, so end-to-end latency from ingest to query is measured in seconds rather than minutes.
How fast is Lakehouse//RT at high concurrency?
With Databricks Lakehouse//RT, query patterns that previously returned in seconds now return in milliseconds. High-cardinality aggregations that once created bottlenecks at scale now execute in a fraction of prior response times. Under extreme concurrency, with hundreds of dashboards refreshing simultaneously while data continues ingesting in real time, serving latency holds steady. There is no cliff where performance degrades as load increases, which is the failure mode that has historically made high-concurrency BI on a lakehouse impractical.
This is the class of performance that previously required standing up a dedicated serving system outside the warehouse. Lakehouse//RT delivers it natively on existing Delta tables, with full Unity Catalog governance, and no data movement required.
Why Sigma Is Built to Take Full Advantage of Databricks Lakehouse//RT
Sigma’s architecture was designed around a single core principle: push compute to the data warehouse, not the other way around. Every query Sigma executes runs directly on the warehouse. There is no extract layer, no in-memory cache the product depends on to feel fast, no shadow copy of data living inside Sigma’s infrastructure.
That design choice has always been the right one. Lakehouse//RT is the moment it becomes the best one.
When Lakehouse//RT delivers sub-second responses from Databricks, Sigma workbooks inherit that speed directly. Live workbooks, high-concurrency embedded apps, and real-time operational views all run through the same warehouse-native connection, with Unity Catalog permissions enforced end to end, exactly as they are on every other query Sigma runs.
There is no rearchitecting required. Sigma was already built to pass queries through to the warehouse. Lakehouse//RT makes the warehouse fast enough that those queries return before users notice they were sent.
ETL Cache and Lakehouse//RT: A Tiered Architecture for Any Workload
Sigma’s forthcoming ETL Cache feature extends this further, and will soon be available in private beta. ETL Cache allows teams to pre-materialize the results of complex or expensive computations, so those results are served instantly without re-running heavy queries on every load.
Paired with Lakehouse//RT, the two capabilities create a complete query architecture inside a single governed workspace. Lakehouse//RT handles the live, high-concurrency serving layer: the workloads where freshness is non-negotiable and response times must be sub-second. ETL Cache handles the computationally complex pre-materialized layer: the workloads where the calculation is expensive but the result can be refreshed on a schedule.
Teams no longer have to choose between freshness and performance, or architect separate systems to handle each. Both live inside Sigma, both run on Databricks, and both stay under Unity Catalog governance.
Note: ETL Cache is currently in development and will be available in private beta soon.
The Workloads that Lakehouse//RT Is Built For
Lakehouse//RT is purpose-built for a specific set of workload patterns. Being clear about that scope is part of what makes the architecture trustworthy.
Lakehouse//RT excels at observability and monitoring at scale, tracking metrics, logs, and traces across large, fast-moving systems. It is designed for user-facing analytics where response time directly affects the quality of the experience, for product analytics and clickstream data where volume and velocity are both high, for customer-facing applications and embedded dashboards that cannot tolerate latency, and for AI agent-serving workflows where an agentic application is querying the lakehouse as part of a live decision loop.
Sigma as a Lakehouse//RT Launch Partner
Sigma has been selected as one of Databricks’ few launch partners for Lakehouse//RT, representing the BI and analytics category. That selection reflects the depth of the existing partnership, and the architectural alignment between how Sigma is built and what Lakehouse//RT enables.
Over the coming months, Sigma will be developing and validating its Lakehouse//RT connector in close collaboration with the Databricks engineering team. As capabilities come online post-launch, Sigma will incorporate them as they become available.
The goal is a fully production-validated Sigma connector for Lakehouse//RT, built on the same warehouse-native architecture that has defined Sigma from the start.
Stay Close to What’s Coming
The Sigma-Lakehouse//RT connector is in active development. As milestones are reached and availability expands, the Sigma team will share updates across our website, blog, LinkedIn, and X account.
If your team is building on Databricks and thinking about what real-time analytics looks like when latency stops being a constraint, this is the right moment to stay close. Schedule a demo with a Sigma expert to learn more.
