The Evolution of Data Warehouse Analytics
Director Content Marketing, Sigma
As with biological evolution, the evolution of data warehouses and data warehouse analytics has simultaneously created greater complexity and capability. Technology has built upon itself, adapted, morphed into something altogether new — yet clearly recognizable as a modern relative of more primitive systems.
These developments have dramatically increased the amount of data that it’s practical to store and have boosted the computing power needed to work with the data. According to IDC, the “Global Datasphere” reached 18 zettabytes in 2018, and IDC is predicting that number will grow to 175 zettabytes by 2025.
But although data warehouses and data analytics have evolved dramatically, most companies aren’t taking advantage of these new (and valuable) capabilities. In this post, we take a look at how this evolution unfolded and the potential it holds for companies seeking ways to pull ahead.
Slow-moving dashboards and reports for on-prem data warehouses
The first data warehouse was created in the 1980s as companies began the pursuit of competitive advantage in earnest. Globalization made it a necessity. Teams within just about every department were looking to data to help them make more strategic decisions. Companies that could afford to accumulate server hardware and built data teams who could clean, structure, and work with the data to deliver reports.
But these early seekers of insights were held back by several technological challenges. For one thing, the first on-premises data warehouses were limited in the amount of data they could store. Very few companies could invest in vast numbers of servers, so historical data had to be dumped regularly. Without historical insights, decision-makers were missing a significant part of the picture.
Second, processing power was limited, so IT professionals had to wait and run reports during non-peak hours. When decision-makers faced acute challenges or time-sensitive opportunities, they were stuck waiting on important reports because there was no way to quickly scale to meet the increased reporting need. Real-time analysis was, for most companies, impossible.
Over time, improvements in on-premises warehouses sped up processing and made storage more affordable. Indeed, many companies today are still using on-premises warehouses, for a variety of reasons. But on-premises warehouses require dedicated investment — servers have to be purchased, so it still isn’t easy to quickly scale up and down. As a result, the amount of data and type of data that’s practical to store remains limited. And the mechanism for getting multiple data sources into an on-prem warehouse is still cumbersome and time-consuming.
Sigma’s dashboards are fast, always updated, and easy to share or embed. Schedule a demo to learn more.
Faster-moving dashboards and reports for cloud-based data warehouses
Because the demand for data-driven insights was so strong, bright minds quickly developed solutions in the form of cloud-based data warehouses and cloud analytics tools. This enabled companies to use exactly the resources they needed (no more, no less), so computing power and storage space were now almost infinitely scalable. And companies could scale up or down quickly, allowing them to be more agile in response to threats and opportunities.
As a result, companies could afford to keep historical data, making them more effective at spotting trends and identifying problems before they became more challenging. Additionally, this affordable access to massive computing power allowed reports to be run at any time. No more waiting for low-load nights and weekends.
But bottlenecks remained. Reports still required technical users — people who knew how the warehouses were structured and who had the necessary SQL skills to work with the data. Ad hoc analysis was elusive.
Ad hoc analysis uses all the capabilities of the managed cloud data warehouse
Big Data was the next change that triggered new developments in the data warehouse and data analytics. With mobile apps, SaaS products, wearables, and other data-producing technology generating enormous amounts of data that could be mined for insights, companies sought greater flexibility.
The modern data warehouse can store structured, semi-structured, and unstructured data, thanks to new advances in ELT (extract, load, transform) technology that has replaced the old ETL sequence. Data now only requires a simple cleaning before going into the warehouse, since the transformation step can happen when the data is queried using modern tools. And data vault modeling speeds things up even further by bypassing the judgment of what’s valuable and what isn’t. At the same time, it allows the integration of data from different systems and the tracing of all data origins. The line between the data warehouse and the data lake has blurred, empowering organizations to use more data than ever before.
While most data analytics tools have simply adapted over time to accommodate the newest advances in data warehouse technology, a few (like Sigma) are truly cloud-native, built with the modern data warehouse in mind.
In the case of Sigma, ad hoc reporting bottlenecks are obliterated, because even non-technical users who have been lightly trained in data skills can use spreadsheet-like tools to query structured and semi-structured data on their own. Without any knowledge of code, users can ask the follow-up questions they need to for deeper insights. Technical teams are freed to focus on other work, including vetting data, training, and helping non-technical users with more complex queries.
SEE IT IN ACTION
See the power and speed of Sigma in action. Watch a demo and learn more here.
The data warehouse analytics evolution birthed data democratization
The evolution of data warehouses and data analytics tools has democratized data. Even small companies can benefit from data-driven insights in ways that large corporations could only dream of in the past. Teams who invest in today’s data warehouse analytics tools like Sigma can take advantage of everything modern data warehousing has to offer.
Sigma has proved to be priceless in helping us make more informed decisions and empowering all employees to think like analysts.
Product Manager, Growth at Blue Bottle Coffee
To learn more about what’s possible with modern cloud BI and analytics for data warehouses, see our Definitive Guide to Cloud BI & Analytics for Data Warehouses.