Big Data and AI Disillusionment

Juluan Alvarado

Sr. Content Marketing Manager, Sigma

Everyone thought that 2020 would be the year artificial intelligence (AI) went mainstream in data analytics and business intelligence. They promised to reveal game-changing insights while saving companies time and resources by automating the process. But the promise of Big Data and AI has remained unfulfilled, and business teams are getting disillusioned.

The reality is that these technologies are still in their infancy. And while they’re excellent for certain applications, like making car safety systems smarter, they can’t match the creativity and curiosity of the human brain — both of which are crucial to business intelligence. In this post, we look at the problems of Big Data and AI today and share how you can empower your business teams to get the insights they need for daily decision-making.

Problems with Big Data Artificial Intelligence

It will be interesting to see how Big Data and artificial intelligence develop over the next several years. Currently, however, using AI-powered BI solutions has two significant problems.

Random Anomalies Don’t Translate into Insight

AI’s real ability lies in identifying anomalies in massive amounts of data, making it ideal for use in production facilities and safety features of various kinds. But the problem Is that  anomalies are only the first step to actionable insights, and not all insights are anomaly-based anyway. Additionally, this process runs thousands of queries, eating up compute power and skyrocketing costs — with little of value to show for it.

Finding truly meaningful business insights requires analyzing data in the context of business processes, market trends, and company goals. Knowing what to look for in the data and interpreting findings through the lens of domain knowledge and previous experiences is also crucial to the process. Big data AI just doesn’t have this capability – yet.

Shallow Depth of Insight

The insights that AI does uncover are typically surface-level, leaving much unknown and problems unresolved. For example, a sales director needs to stay updated on sales performance. AI will be able to tell her that a certain percentage of sales opportunities closed in a given month. But she needs to know more. She needs to know why sales dipped in a given month, which can only happen when she compares sales data with marketing data to learn that the company’s top-converting marketing channels delivered a significant percentage fewer leads that month. Only data exploration can tell her that. She also needs to know if her team could increase ACV by a certain percentage for the remainder of the quarter to make up the shortfall. To learn this, she must conduct a what-if analysis.

The truth is, you don’t need AI to uncover even surface-level insights since your domain experts will know what questions to ask, eliminating the need to run thousands of queries on big data sets. You can more efficiently target the queries that will produce insights, getting stronger insights while using less compute power.

Business Domain Experts > AI

Your business domain experts are infinitely more valuable than AI to business intelligence. They can incorporate their contextual knowledge and proven experience into the analysis process. Giving them the ability to explore data in search of untapped opportunities, run rapid what-if analyses to uncover potential risk or problems, and schedule automated alerts for specific events will result in truly impactful insights.

The (possibly literal) million-dollar question is, “How can we empower our business teams to surface the insights they need?” To generate meaningful insights today, you need a BI solution that supports business teams and data teams alike. Here’s what to consider.

Built for the Cloud Data Platform

A modern cloud data platform like Snowflake will provide you with unlimited scale and speed, an essential foundation for agile analytics. A BI solution built for the cloud will allow you to take full advantage of what your cloud data platform offers. Avoid tools that have simply been retrofitted for use with the cloud since they have limitations. For example, most traditional analytics tools require that that data be extracted for use. The problem with data extracts Is that they quickly become stale and usually require preparation and heavy modeling by the BI team before being analyzed.

Serves Both Business Teams and Data Teams

62% of businesses say self-service analytics is mission-critical — they can’t wait around for their requests to filter through the data team’s overflowing queue. At the same time, data and BI experts play a crucial role in ensuring data quality, curation, governance, integration, and more. The most impactful insights are uncovered when data and domain experts can collaborate.


of businesses say self-service analytics is mission-critical

For this reason, you need a BI solution that meets the needs of both data experts and business users. Analysts and data engineers should be able to use SQL as desired or add and integrate new data sources and types, and domain experts should be able to conduct queries using a familiar interface, like a spreadsheet.

Capable of Answering Next-Level Questions

Even non-technical business users should have the ability to easily conduct complex, iterative analyses on real-time data. Using a solution like Sigma, users can dive into anything from churn analysis to year-over-year calculations — directly accessing and analyzing data using familiar functions in a spreadsheet-like interface.

Encourages Data Exploration

One of the most powerful aspects of modern cloud data platforms is their ability to use lightning-fast ELT processes to prepare data for use. A BI solution built for the cloud will let you use this capability for limitless data exploration on-demand. Additionally, if the BI tool connects directly to the cloud data platform and allows non-technical users to access and work with data, It solves the problem of business teams waiting  for the BI team to model additional data sets.

For example, if either AI or a standard conditional alert reveals that the average cost per lead spiked in a given month, a marketing leader will want to know what drove the increase. Without a BI tool built for the cloud and usable by non-technical business teams, this marketing leader would likely have to go back to the BI team to model this data.

Thoughtful and exploratory what-if analysis is far more efficient at detecting potential issues and missed opportunities than AI spotting statistical anomalies. For this reason, companies are turning away from AI-driven BI solutions built for data teams alone. Instead, they’re seeking out big data BI solutions that allow any user to manipulate parameters, factor in multiple data sources, and set automatic alerts once specific conditions are met.

Big Data and AI May Develop in the Future, but People Are Your Most Pivotal Resources

While AI is likely to evolve and become more capable, the reality is that it will be a while before it can replace human-directed data exploration. Solving today’s complex and highly nuanced business problems requires a perspective with context, proven experience, and domain knowledge — which only humans can deliver.

Ready to try Sigma’s data exploration capabilities for yourself?

Start asking more of your data.