October 25, 2019

Building a Data Lake or Data Warehouse in the Cloud: What you need to know

TABLE OF CONTENTS

We’re all swimming in data. Whether it’s transaction data from a purchase, customer interactions inside an app, or information from website visitors, I’m willing to bet your company is flush with data. It’s everywhere.

Of course, all this data must be stored inside of a database or data lake as it’s collected. But the type of data store you choose largely depends on the kinds of data you collect and how you plan to use it.

If you’re rethinking your data infrastructure or migrating it to the cloud for the first time, you might be wondering whether to go with a cloud data lake or cloud data warehouse. But what’s the difference? And do you need to decide on one versus the other? Maybe you need both? In this post, I break down the differences between the data lake and the data warehouse—and explore the things to keep in mind when choosing a solution for your business.

What is a cloud data warehouse?

Capturing data is just the beginning. To understand that data, it must be stored in a relational data warehouse using schemas that provide a way to query the data. Think of the modern cloud data warehouse as your data hub at the center of your analytics stack. Cloud data warehouses give teams the power to centralize and explore data to generate insights with analytics tools.

Think of the data warehouse as your data hub at the center of your analytics stack.

The data warehouse isn’t a new concept. But many data warehouses in use today were built to service the on-premises data centers of the past. These solutions are a dying breed as they get replaced with the next generation of cloud warehouses designed to provide greater flexibility and manage real-time data demands.

Why is the on-premises data warehouse going the way of the dodo? They require a significant upfront investment in hardware, license fees, and ongoing maintenance costs to manage. On-prem warehouses cannot elastically scale up or down to meet real-time data demands—meaning companies have to pay to provision a warehouse for peak use despite varying workloads that change over time as analytics needs arise. Together, this leaves your company overpaying for data management and wasting IT resources, which you could spend on higher-value projects.

Why build a cloud data warehouse?

Modern cloud data warehouses eliminate upfront infrastructure costs and don’t require the ongoing investment to partition, optimize, or vacuum data. They can also collect data from many sources and scale elastically to support nearly infinite users and analytic workloads for faster insights. This includes structured and unstructured data—such as JSON—reducing the need for open-source projects like Hadoop to manage these data types. These benefits are good news for your data team as they cut the upkeep and maintenance they may face daily.

Cloud data warehouses allow enterprises to add any number of users, implement familiar, easy-to-use analytics tools, and benefit from lower costs—all without sacrificing security, governance, or data compliance.

Learn more about the benefits of cloud data warehouses here:

What is a cloud data lake?

Data lakes work as a central cloud repository for all your structured and unstructured data. This means you can store information in its natural state without having to structure data first. Data lakes are a flexible option to store data outside of the rigid schemas required in the data warehouse.

Data lakes let you store information in its natural state without having to structure data first.

Why choose a cloud data lake?

Analytics stacks built entirely on a data warehouse make it harder to analyze data outside the schema without constant efforts to curate and clean the data regularly. You can also scale to data of any size and save time because you won’t need to define structures, schemas, or transformations. In cases where you have massive amounts of data collected in real time and stored outside of your schemas, this approach makes a lot of sense.

Data lakes also make it possible to store non-relational data from mobile apps, IoT devices, and other non-traditional data sources. Data captured outside of your pre-defined data schema is better stored in a data lake because you may not know what types of questions you want to ask of this data upfront. And because it lives in the data lake, you can always decide down the line on how to use it.

Learn more about data lakes here: “What is a data lake?” from Amazon

So, does your company need both?

Good question. It’s not uncommon for companies today to have both a cloud data warehouse and data lake. Each approach serves different uses and provides unique benefits. Deployed together, you can solve a variety of business use cases.

Cloud data warehouses can provide business analysts and data teams with a central “source of data truth” to run analytical queries against, helping them report business performance and make informed decisions. They are the foundation of a solid analytics strategy.

Data lakes and data warehouses are different tools for different purposes. If you already have an established data warehouse, you might choose to implement a data lake alongside it to solve for some of the constraints you experience with a data warehouse.

Bernard Marr

Forbes

Cloud data lakes are an excellent tool for data scientists to crunch unstructured data, run AI/ML applications, and discover useful trends that business teams can put to use downstream. They are slightly more flexible and can feed your data warehouse once you’ve structured that data or stored it in a schema that’s queryable by your warehouse.

Ultimately, it’s up to your data team to decide what infrastructure will work best for your business needs. Then you’ll need to choose an analytics tool that integrates with your chosen data warehouse so that business teams can make the most of the data.

Thirsty for more? Check out these additional resources about building a successful cloud data and analytics infrastructure here: