Report Factory Hell

You deliver a requested report to a business colleague, but they aren’t happy. They claim you didn’t effectively fill their request, or they have follow-up questions.

Before you know it, you’ve spent hours generating low-level, ad-hoc reports in attempt to give them the answers they need. This is report factory hell, and it’s a terrible place to be. But don’t lose hope. You can escape. Allow us to show you how.

The Big Bang

Source: datanami.com

Over the last decade, the amount of data that is created, consumed, and stored has exploded. This exponential growth has been fueled by wearables, IoT, social media, and mobile phones. Just take a look at what’s created in just one internet minute!

This is what happens in an internet minute (2019)

All of this data is a potential treasure trove for companies that are able to use it effectively. Combing through structured, unstructured, and semi-structured data, leading companies are hoping to find patterns, identify emerging trends, and extract insights that aren’t obvious from the surface — insights that will drive better decision-making and give them an edge over the competition. They’re employing data engineers, data scientists, and data analysts in record numbers to carry out these initiatives.

But despite the mountains of data at their fingertips, and armies of data experts at their disposal, most organizations are not enjoying data bliss. In fact, most BI teams have descended into the depths of report factory hell.

The Six Circles of Report Factory Hell

Far from basking in a utopia of data-fueled insights, data and BI teams find themselves bombarded with competing requests, working through a never-ending queue of low-level ad-hoc reports. This renders them unable to tackle the high-value data projects that move the needle and provide the job satisfaction they’re seeking.

The descent usually follows this predictable pattern:

Gluttony. As the amount of data grows, the number of requests for reports, dashboards, and answers grows along with it. The needs of the many rely on the expertise of a few, resulting in a broken and bottlenecked process. And because these requests come from a variety of departments, there’s no real way to prioritize them — leaving everyone on the BI team scrambling to complete dozens of “urgent” requests.

Anger. Business leaders tend to underestimate the complexity of their requests and overestimate what data teams actually have access to. Data and BI teams are left operating with little context about the problems they’re being asked to solve, and may find it difficult to present their findings in terms business users understand. When data falls short of being the infallible guide business users expect, or a report takes weeks of back-and-forth and still isn’t quite right, they can lose confidence in the organization’s investment in data science.

Treachery. Ad-hoc requests are a major drain on the BI team’s time and resources. Ad hoc is Latin for “as the occasion requires.” But business is ad hoc by nature. Competing requests pour in, inundating the BI team with tedious, time consuming, and often duplicate work. Some data analysts spend up to 50% of their time fulfilling ad-hoc requests.

When data governance is weak and people engage in shadow IT, data becomes a weapon. Sensitive information may find its way into dozens of downloaded spreadsheets. Ungoverned and insecure data increases the risk of security breaches and can violate data privacy laws, which is a huge liability in our post-GDPR world.

Greed. In today’s on-demand world, companies need to move lightning fast, and course-correct in real time to compete. The confidence that data-driven decision making can provide is appealing to business leaders across the organization. In every team and department, people need to be able to ask questions, get answers, and keep iterating to gain insights.

But this puts incredible strain on the BI team. A sharp divide exists between those who speak the “language of data” and non-technical domain experts. Instead of investing in education, promoting data literacy, and laying the foundation for collaboration between these two camps, companies throw bodies and technology at the problem, setting everyone up for failure.

Heresy. When business users start getting impatient and fail to see the value of the BI team’s insights, they may take matters into their own hands. A dubious patchwork of data extracts and spreadsheet sprawl results in analyses that contradict each other. Confusion reigns when there’s no single source of truth.

Limbo. These reports often lead to more questions, creating an endless loop of creation, presentation, and iteration. Even worse, these low level reports tend to add little value to the org as a whole, leaving BI teams with minimal time and resources to do broader, more strategic work with farther-reaching impact.

This is report factory hell. But how did we get here?
And more importantly, how do we get out?

The Great Divide

The term “business intelligence” was coined by Richard Miller Devens in his 1865 work Cyclopaedia of Commercial and Business Anecdotes. He used it to describe how banker Sir Henry Furnese had put himself ahead of the competition by collecting and analyzing the information at his disposal to make better business decisions. At this point, the business expert and the person collecting and analyzing the data were one and the same. And the tools required to do this were little more than a pen and paper.

In the

1970s,

relational databases were invented by Edgar F. Codd

In the

1970s,

relational databases were invented by Edgar F. Codd

Source: spectrum.ieee.org

But starting in the 1950’s, aided by the growth of computers and the birth of the digital era, business intelligence became a specialized occupation. As computing continued to mature throughout the 60s, the skills necessary to collect and translate data grew too complex for the average office worker to manage.

In the 1970s, relational databases were invented by Edgar F. Codd. SQL became the language of databases and created a gap between those who could access and interpret data and those who couldn’t but needed it to make business decisions.

For over a decade now, self-service analytics have been heralded as the solution that would bridge this gap, freeing BI teams to focus on their work while giving business users the ability to get the reports they need to make better decisions. But in practice, self-service analytics have been a tremendous disappointment.

Want to keep reading?

Learn how Sigma can help you evaluating BI tools.