The Definitive Guide to Cloud BI & Analytics for Data Warehouses
Content Marketing Manager, Sigma
In the past, a plodding approach to data processing and analysis worked just fine. Everyone was dealing with limited on-premises servers and had to rely on IT to run reports in slow-moving batches overnight and on weekends. People might have wished for faster access to insights based on their data, but a more efficient data warehouse looked like a castle in the sky, and everyone faced the same challenges.
Today, that’s changed. The cloud brought these sky-castles down to earth, completely transforming what’s possible with the data warehouse. And modern analytics tools have turbocharged organizations’ ability to quickly make use of their data. Now, with powerful managed cloud storage solutions like Google BigQuery, Amazon Redshift, and Snowflake, data doesn’t even need to be fully summarized to work with it effectively. Companies that aren’t taking advantage of what’s possible are falling behind those that are.
In this guide, we explore how data warehouses and BI have evolved in the cloud, the benefits of using modern cloud BI and analytics, and additional resources for continuing the journey toward gaining actionable insights from your data more quickly.
Data warehouses: a brief history
The data warehouse first appeared in the 1980s when relational databases using SQL replaced outmoded methods of storage, making databases more valuable. The warehouse served as a single (on-premises) storage location for data from disparate sources, where BI teams could find cleaned and organized data.
On-premises data warehouses
With the advent of the internet and resulting globalization, competition between companies worldwide expanded, and organizations were motivated to mine their data for better decision-making. Data warehouses became essential. Companies began investing in server hardware and data teams who could clean, structure, and work with the data to provide decision-makers with necessary reports.
But on-prem data warehouses had significant limitations. Processing power was restricted to the server system a company could support, so reports had to be run in batches during non-peak hours. And only those with knowledge of server structure and SQL could run reports, so ad hoc analysis wasn’t possible for smaller teams.
Cloud-based data warehouses 1.0
Demand for more and better data insights drove the development of cloud-based data warehouses. No longer were companies limited by storage restrictions. The cloud was nearly infinitely scalable, and this technology was available at much lower costs than were involved in obtaining and managing on-prem resources. Now organizations could afford to keep historical data for broader insights. And the cloud delivered as much computing power as needed, so reports could be run at any time.
But a company’s ability to produce actionable reports still relied on data teams with SQL skills. Although new dashboard tools gave non-technical business users the ability to see pre-configured reports, these users couldn’t dive deeper without involving data experts.
Additionally, cloud-based databases still required data to be fully structured and summarized before it could be used. So bottlenecks created by lack of IT staff continued to hold back insights.
Cloud-based data warehouses 2.0
When the amount of data skyrocketed due to the introduction of mobile apps, SaaS products, wearables, and other data-producing technology, the drive to make use of Big Data resulted in the next iteration of cloud-based data warehouses.
Modern cloud data warehouses are significantly more flexible than the first iterations were. Reports can still be run at any time due to even higher computing power and scalability, and companies can now store vast amounts of historical data at an extremely low cost.
The biggest difference in modern cloud data warehouses is their ability to store both structured and semi-structured data. New advances in technology have allowed ELT (extract, load, transform) processes to replace the old ETL sequence, meaning that data now only needs to go through a simple cleaning before going into the warehouse. The bulk of the transformation step happens on an ad hoc basis, when the data is queried using modern tools. And data vault modeling allows even more flexibility for those who need it, since it bypasses the judgment of what’s valuable and what isn’t while integrating data from different systems and tracing of the origin of all data coming in.
Kent Graziano, industry veteran who is now serving as Chief Technical Evangelist at Snowflake, explains, “One of the things I’m seeing is the evolution of the data lake and data warehouse. It’s becoming more about a data platform and a place for doing your analytics and getting all the data consolidated. The cloud presents a massive opportunity here because of the flexibility and nearly unlimited scale that it provides. It has removed the on-prem constraints from the conversation.”
How cloud data analytics has evolved
Cloud data analytics has matured alongside developments in data warehouses. Let’s look at this process of evolution.
Dashboards and reports are time-consuming to create
SAP, Oracle, and other companies have provided data visualization and facilitated reporting since data warehouses first came on the scene. These data warehouse analytics tools were built for on-prem servers and adapted over time to accommodate cloud-based data warehouses. But iterations to the software have been based on the original premise that users must be highly-technical data experts in order to go beyond drilling down into pre-configured dashboard views. And these tools don’t take advantage of the data velocity capabilities that modern cloud warehouses and data modelling provide.
When managed cloud warehouse solutions became mainstream, newer tools like Tableau and Qlik made reporting faster and easier than was possible with historical tools. These systems aimed to be self-service, but still require significant training and technical skills to create additional dashboards or answer deeper questions like why trends are happening. So non-technical business users must wait for IT to build out dashboards and/or run reports for these insights. Large corporations can afford to fully build out data teams, but midsize and small companies continue to struggle with bottlenecks created by limited IT staff. These limitations have resulted in a meager 35% adoption rate for cloud-based analytics — they’re still too cumbersome.
Cloud analytics: new tools expand data exploration and ad-hoc analysis
Cloud-native analytics tools, such as Sigma, are built with the modern data warehouse in mind, fully taking advantage of all its capabilities. Non-technical business users lightly trained in data skills can use spreadsheet-like tools (that are far more powerful than a standard spreadsheet) to query vast repositories of structured and semi-structured data on their own. They can ask important follow-up questions and get the insights they need, when they need them — no code required. Technical staff can focus on vetting data, training business users on analytics best practices, and helping users with especially-complex queries.
Companies investing in data and using today’s managed cloud data warehouses to store and process it can now take advantage of everything modern technology has to offer. And with the ability to scale up or down as needed, even small companies can benefit.
SEE FOR YOURSELF
See what’s possible with a modern cloud analytics solution like Sigma. Watch the demo.
Benefits of modern cloud BI and analytics
We’ve referenced a few advantages of fast ad-hoc reporting, but the benefits of modern cloud BI and analytics go well beyond these. Let’s take a look at exactly how companies can be more competitive with today’s tools.
Let’s start with the most obvious benefit: the ability for non-technical business users to get to essential insights quickly, without relying on the data team. Self-service data analytics has been a goal since analytics tools were first developed. Ad hoc reports are a necessity for companies that must move quickly to respond to competitive threats and other challenges (essentially every company in existence today). But when business users must wait for technical team members to generate these reports, valuable time slips by, and companies lose ground. Faster insights translate into a more agile, more competitive company which, in turn, drives profit.
Break down silos
How faster insights are made possible is a function of modern tools’ ability to break down silos. These tools can easily access multiple databases of varying types, both structured and semi-structured data. For example, Fivetran has overcome traditional ETL processes to quickly connect to entire data sets, without the need to get IT involved. Now you can even connect to unstructured data lakes if desired, bringing in additional data. Using Sigma, for example, business users can connect to a company’s SQL databases to work with available data and then bring in data from JSON databases and extract specific columns to expand the query. Sigma automatically generates SQL code based on the user’s manipulations of the spreadsheet-like interface. And users can build a wide variety of visualizations right from the same window. Data residing in different locations no longer poses a problem.
The breaking down of silos simultaneously makes increased collaboration possible. New data modeling methodologies help guide data exploration while adhering to data governance frameworks. And now, every department within an organization can benefit from data generated by others, as interactive dashboards and shareable analyses make collaboration simple. Business users can work together in a variety of ways, such as building predictive models based on a full picture of data across departments. For example, marketing data can combine with sales data and customer service data to reveal developing trends or triggers for churn later down the road. Additionally, internal teams can build on one another’s work, making the entire process more efficient.
Better security and data governance
A central access point also enables better data governance. Granular controls can be put in place to limit access to parts of the data warehouse or specific types of data. The modern data warehouse makes it easy to see who accessed what data when, allowing companies to quickly identify potential weaknesses.
Ability to scale
All of the modern cloud tools are easily scalable, allowing companies to pay for the precise subscriptions needed at any given moment. This capability dramatically reduces risk. If a market downshift impacts the business, the company’s data resources can immediately adjust to the need. On the flip side, if the need for analytics suddenly increases, the company can scale up quickly.
Teams who can make informed decisions quickly are better able to respond to threats and opportunities. Better decisions lead to a more sustainable company. Additionally, scalability combined with better security dramatically reduces cost. Maintaining a full data team with expensive expertise and paying for enough technical staff to manage and address security vulnerabilities is beyond the reach of many companies. The relatively low costs associated with modern cloud data warehouses and cloud BI and analytics tools increase profit margins and bring previously-unreachable capabilities to smaller companies.
Improved data literacy across the organization
Working with data leads to increased data literacy. As organizations give data access to decision-makers, these users get curious about data and everything they can accomplish with it. When organizations offer training on the fundamentals of data analytics, business users gain power to use data in smart ways, asking questions out of their deep subject matter knowledge. Everyone in the organization benefits.
READY TO BUY?
Read our free buyer’s guide before you build a modern cloud analytics stack.
How Clover improved time to data insights by 90% with Sigma
Let’s take these advantages from the hypothetical to real life by looking at how one of Sigma’s customers is putting them into action.
Clover offers a global open-architecture point of sale solution aimed at small and medium-sized business owners. The company relies on data insights to drive product development, create a top-notch customer experience, and inform business strategy.
Clover’s Data Reporting and Analytics (DRA) team regularly provides domain experts across the organization with dashboards, reports, and data extracts for further analysis.
Challenged by an out-of-control ad hoc report queue
Clover’s Snowflake data warehouse stores information from dozens of sources, including MySQL, Salesforce, Google Analytics, Heap, and Greenhouse. Having a central source of data was great, but business users without the SQL background required to explore and analyze it had to submit ad hoc requests and wait for the DRA team to deliver. As you can imagine, the highly data-driven company was producing massive numbers of requests, burying the DRA team and preventing them from working on other priorities.
Sigma’s solution makes it simple for business users to mine data for insights
The DRA team started looking for a tool that would allow Clover’s business users to quickly explore, analyze, and visualize data using an interface they were used to. Sigma’s software, built to function like Excel, was ideal. Quick conversions from worksheet to visualizations and dashboards allowed Clover’s business users to easily find answers and see the insights they needed.
Clover quickly achieves a 90% decrease in time to data insight
Onboarding was simple thanks to Sigma’s familiar Excel-like interface. 75 employees across seven teams were able to quickly begin producing insights. Since the DRA team now needs to be only minimally involved in ad hoc reporting, users have seen a 90% decrease in time to data insight. The DRA team is now focused on data training to ensure everyone at Clover has a solid background in data awareness and literacy, giving them even more power.
Where to go from here
If you’re intrigued by the possibilities of modern cloud BI and analytics, you’re probably wondering where you can find additional resources. Here are our recommendations for further education:
Ready, Set, ROI: Accelerating Analytics in the Cloud with Snowflake and Sigma — This on-demand webinar brings together experts from Snowflake, E.W. Scripps, and Sigma to share actionable strategies that speed up data query times by up to 100x, reduce time to data insight by 90%, and analyze 2x as much data at no extra cost.
Choosing the Right Cloud Data Warehouse — Selecting the right cloud data warehouse isn’t an easy decision. But your data experience hinges on the CDW vendor you choose. This guide shares the key factors to consider and a list of popular providers.
Building a Data Lake or Data Warehouse in the Cloud: What You Need to Know — If you’re unsure of the difference between a data lake and a data warehouse, or if you’re having trouble figuring out if you need both, this guide will help. It breaks down the differences between the data lake and the data warehouse and explores the things to keep in mind when choosing a solution for your business.
Level the Playing Field with Cloud BI & Analytics — In this resource designed for small teams, you’ll learn how cloud BI helps you overcome common analytics hurdles, the unique features of the cloud analytics stack that make it ideal for data-driven startups, and how cloud analytics fits into overall organizational goals and strategy.
What to Look for in a Cloud BI & Analytics Platform — What does your company need in an analytics platform? Every organization’s needs are different, and each tool functions differently. This guide shares what to consider when evaluating the match between tools and your needs.
Buyer’s Guide: Building a Cloud Analytics Stack — This buyer’s guide walks you through what you need to know about how to establish analytics goals for your company, how to collect, store, and analyze organizational data, how to evaluate ETL/ELT providers, data warehouses, and analytics tools, and how to empower business experts and drive BI adoption.
Cloud BI and data warehouses are the future
There’s power in the cloud, and companies of all sizes are taking advantage of what it has to offer for storage, processing, and analytics. Using cloud-native tools allows you to access insights quickly, capturing the competitive advantage you need to reach your company’s goals.