Into the Unknown: 8 BI & Data Trends That Are Shaping 2021
The economic uncertainty caused by the COVID-19 pandemic proved that analytics is more than a competitive advantage: It is the most critical component of a competitive, highly successful, and future-proof business.
Analytically mature organizations adapted to 2020’s rapidly changing conditions by using data to effectively adjust their strategies and quickly pivot to take advantage of new opportunities as they presented themselves.
For example, the marketing team at Migo, a cloud-based platform that enables B2C companies in emerging markets to offer credit to their customers, was in full growth mode at the start of 2020. But the challenges brought on by the COVID-19 pandemic required the marketing team to quickly shift gears and focus on customer retention and loan recovery.
“Because we had immediate access to all of our data, could run any analysis without the assistance of the BI team, and knew the data was accurate, we were able to pivot and relaunch our entire marketing strategy in just 30 days,” shares Migo’s Head of Marketing, Alex Harvey. The result? A 47% increase in campaign response rates.
As we prepare to enter a new year of great unknowns, data — and the underlying tactics, tools, and technologies that enable companies to harness its full value — remains the answer. A new generation of data analytics is upon us, and it’s real-time, scalable, and accessible to every level of technical expertise.
The Death of
Imagine you’re a marketing executive, just for a moment:
You spent weeks preparing for the quarterly board meeting, and you arrive armed with a flashy new dashboard your BI team built for you. But during the meeting, one of the board members asks a question that isn’t answered on your dashboard. You have no choice but to tell everyone they’ll have to wait at least a week for an answer so your BI team can update the dashboard with this new analysis. The energy in the room quickly evaporates, and you leave the meeting feeling frustrated, powerless, and a little embarrassed.
Given this scenario, it’s easy to see why the novelty of visualizations and dashboards from the last two decades has worn off for line of business teams.
Point-and-click dashboards were great when being “data-driven” meant reporting out on a limited, predefined set of business metrics. At best, these traditional dashboards offer surface-level reporting on what is already known, and worse, leave their consumers without the ability to ask follow up questions without going back to the BI team for help.
But needs and expectations have evolved. Today’s business teams require the ability to dig directly into data — often at the most granular level — to make critical decisions on-demand. As a result, BI solutions and their users are moving away from canned, surface-level dashboards toward dynamic, in-context analyses that surface real-time, highly relevant insights.
In short, the death of the dashboard as we know it is giving rise to BI and cloud data exploration tools that empower non-technical business experts to go beyond the dashboard to independently find answers to their most pressing questions –– without having to wait on data and BI teams.
Sigma is like I’m working in a spreadsheet, but all the data is live. I can explore all of our data at the lowest level of detail and do ad hoc analyses in real-time without any limitations. I love being able to join my own datasets based on unique customer identifiers and build on dashboards without having to go back to Joseph [BI] and re-work things.”
Marketing Lead at Migo
The Rise of Cloud Data Exploration
Research shows 81% of companies agree that data should be at the heart of all decision making — that’s why the BI market is a nearly $30 billion market. But while modern data integration tools paired with a cloud data platform solve the first part of this goal by preparing data for analysis and transforming it into a usable asset, 63% of decision-makers still report that they’re unable to get the answers they need in the required timeframe.
of companies agree that data should be at the heart of all decision making
Business Intelligence Market value (projected to reach $54.76 Billion by 2026)
of decision-makers still report that they’re unable to get the answers they need in the required timeframe
Today’s business teams need real-time access to the massive amounts of data pouring in across channels to make the daily, on-demand micro-decisions that add up and impact the organization’s macro, long-term success. The bottom line: Static reports and surface-level dashboards simply won’t cut it for current data-driven decision makers looking to beat the competition.
Companies have fallen short of enabling organization-wide data-driven decision-making for three primary reasons:
- InfrastructureMost of today’s analytic systems and tools were designed for on-premise warehouses and have been retrofitted as SaaS tools. They often require data to be extracted for preparation and heavily modeled by the BI team before it can be used by domain experts. This not only prevents business teams from getting a complete, up-to-the-minute picture of their data, but it also makes it impossible for them to analyze it at its lowest level of detail. What’s more, these “cloud” solutions are still known to choke on large datasets.
- AccessMost analytic solutions require the use of SQL or proprietary code to drill into data, which prevents non-technical business users from getting their hands on the data they need to make timely decisions. These individuals can’t afford to wait at the back of the BI team’s request queue and are forced to access the data the only way they know how: by extracting it to spreadsheets. This creates its own set of issues including stale data, data silos, scale limitations, and worst of all, governance and security risks.
- DashboardsDomain experts are often limited to view-only metrics in surface-level, static dashboards, which prevent them from performing more in-depth analyses. If they have follow-up questions about the data, they must go back to their data or BI team — a cycle that can take days, if not weeks, to finally obtain useful insights. (See Trend #1).
What is the modern cloud data analytics stack?
The modern cloud data analytics stack consists of three layered technologies and cloud-based services that collect, store, and analyze data. Together, these tools allow organizations to unlock the full value of their data and fuel smarter decision making for all.
THE DATA pipeline
Data must be collected and integrated across applications, databases, files, and more so it can be easily accessed, modeled, and holistically analyzed. The modern data pipeline (e.g., Fivetran) automatically connects and normalizes data from across sources in real-time, preparing it for storage and querying using analysis-ready schemas. Plus, it’s true self-serve, as, with only a few clicks and a 14-day free trial, anyone can start pulling data into their data platform.
THE Data platform
Most companies wrestle with disparate data: some is structured, some semi- or unstructured, and there is no single source of truth from which they can reliably consolidate data and correlate analytics. Cloud data platforms serve as a centralized repository for all of the data In an organization. The Snowflake Data Cloud provides elastic infrastructure, unlimited scale, cost-effective risk mitigation, security management, and other cloud-specific benefits traditional on-prem warehouses do not.
Cloud data exploration
To maximize the value of the data inside the warehouse and enable data-driven decision-making, companies must empower employees of all technical abilities to independently interact with data at cloud speed and scale. Cloud-native BI solutions (like Sigma) give everyone the ability to directly query live data from the Data Cloud, down to row-level detail — no manual SQL or proprietary coding required — while maintaining strict data governance. Teams can create visualizations, join data sources on the fly, unravel JSON, do rapid what-if analysis, and more via user interfaces that resemble tools they already know and love, like spreadsheets. In turn, BI experts can escape report factory hell and focus on the innovative and strategic projects they love.
Companies That Have Adopted a Modern Cloud Data Analytics Stack See Results Like:
Accelerated time to insight by building a key market report in 2 hours with Sigma vs. 2 months using previous tool.
Took back 50% of time spent filling ad hoc data requests to focus on more strategic and impactful data projects.
Cut customer acquisition cost by 50% by understanding the buyer journey and making data-driven marketing decision.
Still looking to complete your cloud data analytics stack?
Check out this BI Buyer’s Kit filled with free resources including an analytics maturity assessment, RFP template, and more!
Data Governance, Goldilocks Style
Rock-solid governance is a non-negotiable piece of every good data strategy. But it shouldn’t be a stumbling block for business teams. Especially in today’s volatile market, decision-makers need direct access to complete, real-time data for “on the fly” analyses and on-demand decision making.
Yet traditional compliance regulations and security protocols have left BI teams with two equally unfavorable options:
The Wild West
Open data access means business teams use desktop tools like spreadsheets to extract data and conduct independent analyses. This level of ungoverned access causes a number of problems for data and BI teams, including data silos, conflicting insights, inaccurate analyses, security risks, and noncompliance.
The Ivory Tower
Data experts attempt to keep data safe and centralized by locking it down behind proprietary languages or SQL. Non-coding business teams can’t directly access or explore data without asking BI teams for assistance, causing mass frustration that ultimately leads back to risky data extracts.
A modern approach to data governance offers a third, more inclusive option, striking a balance between data access and control in your organization. To achieve it, organizations must follow
5 clear steps:
- Automate your data pipeline. Manually extracting data from different sources is a time-consuming and brittle process. This leads engineers to closely guard and monitor their pipelines to ensure no one breaks them, or that only data that’s easily maintained is brought in for analysis. Either way, this approach leads to outdated and incomplete data. By bringing in an automated data pipeline tool, companies can ensure that all necessary data is brought into their cloud data platform, insights are always fresh, and engineers are able to focus on more meaningful work.
- Keep your data in your cloud data platform. The latest generation of cloud analytics solutions gives teams direct, governed access to live data inside your data platform. For example, with the Snowflake Data Cloud, your team does not have to deal with excess complexity or costs — no extracts, copies, aggregates, or caching required. This approach not only enables teams to take full advantage of the speed, scale, and compute power of the cloud, but it also means your data is always safe, current, and complete.
- Choose a BI solution your team knows how to use. True self-service BI solutions empower anyone to independently analyze and explore data, regardless of technical expertise (see Trend #2). Not only does this minimize extracts, but it also eliminates lengthy request queues and frustrating back and forth, freeing BI teams to focus on higher-value projects.
- Set data access roles and permissions. A secure and compliant BI solution should offer the most recent governance-related features and capabilities, including:
Role-based access permissions and sharing controls
OAuth support to inherit data access permissions directly from the cloud data platform.
Audit logs and usage dashboards to record all user actions and queries
Fine-grained row-level security (RLS)
Designated team workspaces
Support for AWS PrivateLink
Single Sign-on (SSO)
GDPR, CCPA, Privacy Shield, HIPAA, and SOC support/compliance
- Embrace flexible and collaborative data modeling. Unlike traditional BI tools that require analysts to predict the line of business’ needs, modern BI solutions take a flexible approach to data modeling that allows advanced users to directly query tables in their cloud data platform without any premodeling required. Because these tools don’t require SQL to analyze data, BI teams can also curate datasets and “link” different sources to give less technical domain experts an endorsed path for exploration. Pulling business teams into the modeling process to add clear, relevant definitions and calculations is also easy, ensuring everyone in the organization is using the right data correctly and consistently.
Want to learn more? Download this free guide to modernizing data governance →
“It Takes a Village:”
The Next Phase of Collaborative Analytics
Many companies have already joined the growing collaborative analytics movement, which enables teams to more easily collaborate on analyses, share reports, and communicate insights internally. But its next phase, community-driven analytics, takes these efforts to the next level.
What Is Community-driven Analytics?
Community-driven analytics is achieved when anyone in an organization — regardless of their technical ability — is able to contribute their domain expertise to the company’s business intelligence initiative and collaborate on analyses that fuel faster, more effective data-driven decisions, lead to better products, and create new revenue streams. It builds on the concept of collaborative analytics by extending relevant data, analyses, and insights across departments, partners, customers, applications, ecosystems, and even entire industries.
Community-driven analytics is made possible by recent advancements in areas including data integration, modeling, and governance frameworks. It requires the ability to seamlessly ingest and normalize massive volumes and varieties of data in real-time into a centralized repository.
Domain experts like marketers and sales leaders must then be able to work alongside BI experts to align the data in this repository to business goals and processes and make it approachable for exploration. This requires self-service BI and data exploration tools, as well as a modern approach to data governance (see Trends #1 and #2).
Best-in-class teams at leading companies like Showpad actually create and hold quarterly workshops to tackle this work together. Bringing marketing, engineering, and BI teams together to workshop their analytic projects allows them to ask more creative questions and find answers more quickly. Ensuring that teams are speaking the same language from the get-go about what business problems they’re trying to answer and what data they’re using empowers everyone to do their job more effectively.
Snowflake is also a key driver behind this trend. The Snowflake Data Cloud eliminates data silos not just within internal businesses, but also across organizations using the Data Cloud. It replaces existing costly and cumbersome data sharing methods with the ability to concurrently connect to a single copy of live data for instant unification, analysis, and monetization. In turn, Snowflake’s Data Marketplace makes it easy to access live, governed data from a company’s ecosystem of partners and customers, as well as participating third-party data providers, and pull it into analyses.
Community-driven analytics starts with accelerating and extending the value of your company’s data for internal stakeholders. It then slowly ripples outward to the external partners, customers, and broader industries your company engages with every day.
A major sporting goods retailer is having an online flash sale to move out last year’s ski equipment and make room for new inventory. The marketing/merchandising team works closely with the BI team to integrate data around the previous year’s most popular SKUs, sale prices, and available inventory. The model also includes third-party data around weather forecasts and trends by zip code. The merchandising team is then able to independently analyze this data to determine which items to promote, how heavily to discount them, and where “flash ski sale” ads will be most relevant this time of year.
The retailer in the above example is trying to secure additional inventory from their preferred supplier to accomplish this. The retailer provides them with a dashboard showcasing total sales of their products over time. The supplier can drill into this information to see exactly which SKUs are selling at which locations for what prices. They can use this data to determine the right quantities and types of products to send, as well as combine it with information from other retailers to see how this particular sporting goods brand stacks up.
To learn more about community-driven analytics and how to get started, check out this eBook by Snowflake and Sigma →
NEW YEAR, NEW OPPORTUNITIES:
Data Monetization Brings New Business
As companies embrace community-driven analytics and get better at processing and analyzing their mountains of data, new opportunities to monetize this information and open up new revenue streams are emerging. One of the most promising ways leading companies have started monetizing their data is through embedded analytics. In fact, according to Allied Market Research, the embedded analytics market is projected to reach $60.28 BN by 2023.
Embedded analytics focuses on integrating analytical content like dashboards and visualizations within business process applications, web pages, customer portals, and existing products or services. It makes it easy to transform proprietary data, reports, and visualizations into a premium product that generates revenue or increases stickiness by providing additional value for data consumers.
Out-of-the-box tools for embedding dashboards typically fall in one of two camps: zero transparency and no control over the static end-user experience, or complete dependence on proprietary, expensive coding knowledge to create and edit custom dashboards. But the next generation of BI tools (Trend #2) coupled with a modern approach data governance (Trend #3) makes it easy for anyone in the organization to build interactive, embeddable dashboards and control access with row level security (RLS).
Embedded Analytics In Action: Payload Unlocks a New Revenue Stream
After spending years focused on perfecting product functionality, Payload decided to shift its strategy and truly differentiate itself from the competition by adding an analytics solution to its application. “We used Fivetran to ingest data across sources and pull it into Snowflake, which was working great,” recalls Iain Letourneau, Payload’s BI Lead.
But report lead times reached an average of 4-6 days as request queues grew, rendering the data stale and insights outdated by the time they reached customers. Building an analytics solution into the Payload application was projected to take 2 full-time employees 6 months with Payload’s current BI solution.
After evaluating multiple BI tools, the team chose Sigma thanks to its spreadsheet-like user interface, which gives everyone the power of SQL without having to manually write code.
Now anyone on the team can spin up a customer dashboard or dig into an existing one to do
BI Lead & DevOps Analyst at Payload
“We embed Sigma dashboards into Payload and authenticate our customers through our own application,” he continues. “Sigma’s modern approach to data governance keeps our data safe and secure, but does it in a way that enables data access, visibility, and insight, rather than forcing our team to act as gatekeepers.
”Using Sigma’s embedded analytics functionality, Payload launched two new data products and more than 30 standard reports in just a couple months — without adding any additional headcount. The end result? A brand new revenue stream at a 600% cost savings!
The Rise of REAL DATA
Over the past 30 years, the enterprise data warehouse (EDW) has been the de facto source of truth for reporting and analytics. In this system, data is created and sent to the EDW, captured, and stored. The raw data is then cleaned, prepared, and modeled by data and BI teams before it’s handed off to business domain experts as a report or dashboard. At this point, domain experts can finally make clear and informed decisions to drive the business forward.
This process was repeated for every new data source, and it worked — until now. The proliferation of new data sources (marketers, for example, have over 8,000 tools at their fingertips), velocity of business, decrease in storage costs, and the rise of cloud computing has changed the game. Today, the cloud data warehouse (CDW) is the preferred way to store data.
But while storing data is easier and less expensive than ever before, the explosion of new semi-structured and unstructured data types (e.g., JSON and logs) pouring in from applications, APIs, websites, smart devices, etc. is overwhelming data and BI teams. No wonder Forrester reports that between 60% and 73% of all company data goes unused.
of all company data goes unused
Business needs and requirements are constantly changing, and domain experts need to be able to tap into the data they need, in any format, across multiple sources, in real-time. As a result, data-driven companies must update their reporting and analytics method to one that empowers domain experts generate their own insights and make decisions on-demand without data and BI teams’ intervention.
The emergence of automated data pipeline tools, data platforms like the Snowflake Data Cloud, and the modern cloud data analytics stack is enabling businesses to move beyond the legacy systems and processes that are holding them back and harness the value of “real” data.
How You Can Get More “Real” Data Into Your Decision Making In 2021?
By embracing all – no longer is there a reason to not bring in a dataset because it’s complicated to do so. Get a modern automated pipeline to take the heavy-lifting off your plate.
Bring it into the cloud – data living across point solutions and spreadsheets isn’t going to be the real data that moves the needle for you or your business. Get all your data in one place to do important and meaningful modeling work.
Get analytics in everyone’s hands – ultimately, data isn’t real if it’s not being used to make decisions, change plans, and illuminate growth opportunities. Giving everyone in your business the ability to access and explore this data is crucial.
Capturing the Most Elusive Data Source:
What’s Inside People’s Heads
Be honest: How often have you heard the phrase “single source of truth” over the course of your career? Probably hundreds if not thousands of times. Thankfully, an automated data pipeline flowing into the modern cloud data platform meets this need by providing a centralized, scalable repository where all data types can be stored, refreshed, combined, and analyzed.
But with nearly 2MB of data being created every second by every person in the world, data and BI teams would be naive to believe that there is valuable data within their organization that doesn’t exist outside the data platform. These missing data sources include spreadsheet extracts on local desktops, third-party applications, and — the most elusive of all — people’s minds.
Returning to the sporting goods retailer before, imagine a merchandiser walking around the camping department. She’s wondering how the pricing and placement of items will impact sales. As she walks down the camping furniture aisle, she asks herself, “What would happen if we moved a particular line of chairs that isn’t selling well to a different, more visible location?” She decides to move the chairs, taking note of this in her notebook. Later in the month, as she analyzes the department’s data, she sees an increase in sales for those chairs but isn’t sure why. Then she sees her notes and is reminded of the experiment she started earlier, completing her analysis and verifying her original hypothesis.
It’s no surprise that business domain experts often want to integrate data from these outlying sources into their analyses. However, BI solutions as we know them today are one-directional. Given the right tools, teams can access and analyze data from the data platform, but sending data to the data platform is a different story.
As a result, these teams typically extract data from the warehouse and munge it together with additional sources in a spreadsheet — a practice that causes data security, governance, silo, and freshness issues (see Trend #3). To minimize risky extracts and democratize valuable data outside the warehouse, business teams must have a way to add this information directly into the “single source of truth” for others to access and use.
Cloud data platforms are now able to take in sources as varied as SaaS platforms and Excel spreadsheets, so teams don’t have to leave any bit of information off the table or out of the decision-making process. 2021 will see modern BI tools continuing to solve for this challenge in a highly governed fashion, leading to more detailed, accurate, and contextual insights for the entire company.
The AI Hangover
Is Setting in
2020 saw the continued surge of alluring analytics offerings powered by artificial intelligence (AI), machine learning, and natural language processing (NLP). These solutions promise to surface unidentified challenges and untapped opportunities while saving companies time and resources by automating insight generation.
But businesses are quickly realizing these tools are still in their infancy. And while they claim to be the future of self-service analytics, they have yet to prove their ability to consistently deliver relevant, actionable insights.
For example, some popular BI tools offer the ability to automatically run thousands of queries in search of statistical anomalies in customers’ data. Although this is sure to surface the occasional insight, randomly identified anomalies don’t point directly to business opportunities, which means these “insights” are usually either obvious or nonsensical. These queries also have the potential to skyrocket compute costs at little to no advantage.
In truth, uncovering truly meaningful business insights requires analyzing data in the context of business processes, market trends, and company goals. Interpreting data through the lens of tacit knowledge and previous experiences is also highly impactful — and something AI simply can’t do.
In 2021, the “shininess” of these AI-powered analytics tools will begin to wear off. Leading companies will turn to BI solutions that empower their most pivotal resources — business domain experts — to incorporate their contextual knowledge and proven experience into complex, iterative analysis. The ability to explore data in search of untapped opportunities, run rapid what-if analyses to uncover potential risk, and schedule automated alerts if specific events occur uncovers truly relevant and impactful insights in seconds.
Imagine you’re a sales director at a tech company. Here are a few examples of the types of insights each of the analytical tactics discussed in this section might reveal:
5% of open sales opportunities have a contact with the first name Michael.
Sales dipped in June because marketing generated 30% fewer leads from our top-converting channel compared to April and May.
If the team can increase ACV by 3% for the remainder of the quarter we will hit our revenue target.
The probability to close for a deal with a projected ACV >$1,000,000 has dropped to 50%.
Navigate 2021 With Data, Your North Star
It wasn’t long ago that the cloud was the future of data analytics and BI. Today, it’s the foundation. The modern cloud data stack is ushering in a new era of business analytics with the power to unlock the true, unbridled value of data.
In 2021, business domain experts will be empowered to go beyond the dashboard to explore and integrate live data at scale — without fear of security or compliance risk. These developments will continue to extend and amplify the value of organizations’ data, paving the way for collaborative environments and new and exciting ways to ethically monetize data.
Alongside the rise of the modern cloud data stack comes a significant shift regarding traditional analytics workflows and the integration, storage, and accessibility of a greater volume, variety, and velocity of data. This includes semi- and unstructured types, spreadsheet extracts on local machines, and even knowledge inside workers’ heads.
Solving today’s complex and highly nuanced business problems requires not just more data, tighter governance, and better accessibility, but also calls for greater context, proven experience, and tacit knowledge. As a result, companies will seek out data and analytics tools that empower their most pivotal resources — their people — rather than relying on premature AI, ML, and NLP offerings.
As we prepare to enter a new year of great unknowns, the organizations in the best position to successfully weather the storms 2021 brings will be those that continue to discover better ways to use technology to transform data into decisions that move businesses forward.
Sigma and Fivetran are here to help leading companies navigate the future of data analytics and business intelligence. To see how we can empower your business in 2021 and beyond, visit www.sigmacomputing.com and www.fivetran.com
Sigma is the only cloud analytics and business intelligence platform empowering business teams to break free from the confines of the dashboard, explore data for themselves, and make better, faster decisions. The award-winning platform was built to capitalize on the performance power of cloud data warehouses to combine data sources and analyze billions of rows of data instantly via an intuitive, spreadsheet-like interface – no coding required. Sigma automates workflows and balances data access with unparalleled data governance to make self-service data exploration safe for the first time.
To learn more and start a free 14-day trial, visit sigmacomputing.com
Fivetran is the leader in automated data integration, delivering ready-to-use connectors, transformations and analytics templates that adapt as schemas and APIs change to ensure reliable data access. Fivetran enables businesses to quickly adopt new SaaS apps, and keep up with continual app changes. Fivetran is ready to go, with prebuilt connectors and customizable templates that accelerate analytics. Data is ready to access with continuous data synchronization from sources to warehouse. Fivetran is ready to adapt by dynamically adjusting to schema and API changes.
Learn more about data integration that keeps up with change: fivetran.com