January 6, 2021

Exploring Community-Driven Analytics and BI

Exploring Community-Driven Analytics and BI

The desire to be data-driven is a uniting force for both data and business domain experts. This is why it’s no surprise that 97% of executives say their organizations are investing in big data initiatives to become nimble and data-driven. While this top-down mandate has become the norm in today’s business environment, most knowledge workers get shut out of the data conversation — and 39% of them don’t even know what it means to be data-driven.

This lack of understanding stems from the historical relationship between knowledge workers and data. Generating data-driven insights has traditionally fallen on the shoulders of the data and BI teams, not domain experts in marketing, sales, or operations. These data-focused teams largely function to support business teams — serving up reports and dashboards, and providing answers to ad hoc questions that arise.

Analytics requests get submitted and filled, which often takes days or weeks depending on complexity and competing priorities. Business teams are dependent upon those that write SQL — or another programming language — for the information they need to make informed decisions.

Nobody is happy about the imbalance this back-and-forth scenario creates. The constant waiting game breeds frustration, suppresses collaboration, and often results in business teams skirting IT compliance and data governance initiatives to use whatever data they can get their hands on.

Taking a new approach to A&BI that amplifies collaboration and the value of data

Analytics and business intelligence can no longer center around ad hoc requests, reports, and dashboards. Strategic A&BI leaders must align the data agenda to business objectives if they want to succeed.

The best way to do that is to bring business teams into the A&BI process, transforming it into a community-driven effort that extends the value of data, analyses, and insights across the larger business community. This includes internal teams, partners, vendors, customers, applications, ecosystems, and even entire industries.

Community-driven collaboration is not only a requirement in today’s business environment — it helps companies thrive.

Today’s agile organizations have changed the traditional nature of “jobs.” The workplace has become more specialized, with a renewed focus on more in-depth expertise in a single field.

Jobs are turning into roles, roles are becoming more highly specialized, and the new currency of value is expertise, not simply experience.

Josh Bersin Business Analyst
Bersin by Deloitte

While collaboration has always been a crucial part of doing business, this new normal has transformed the makeup of business teams and shifted how individuals must work together. Today’s most successful companies like Amazon, Facebook, and Apple have redesigned their organizations to build dynamic, cross-functional teams that communicate faster and require more experts than ever. As marketing guru Seth Godin puts it, this has brought about the “end of the average worker.”

This fundamental shift in the way companies build teams and collaborate on business initiatives requires a corporate about-face as teams combine highly-specialized skill sets to achieve success. While this may not come as good news for the “average worker,” it has been a boon for corporate productivity. According to a joint study into 1100 companies, collaborative working means that businesses are five times more likely to be high-performing.

Technology is paving the way for closer collaboration within business communities, both within and outside organizations.

Corporate and team structures aren’t the only things that have brought about these swift changes. The rise of more remote and globalized workforces — and even stay-at-home pandemic protocols — have contributed to these trends. Meanwhile, technology has emerged to meet the challenges of highly-specialized and distributed workforces head-on.

In the old world, collaboration was all about physical board rooms, team meetings, and whiteboards. But today’s cloud-based technology stack has taken traditional collaboration tools to the digital realm. Today, 83% of professionals rely on technology to connect with their teams, increase productivity, and drive innovation and revenue. The cloud empowers employees within organizations and makes things more productive for partners and customers, allowing for closer collaboration throughout entire business communities, industries, and sectors such as A&BI.

In this eBook, we explore the role of community-driven A&BI in today’s business environment and how data and BI teams can use Sigma and Snowflake to:

  • Fuel more relevant and accurate insights by incorporating each team’s unique expertise into A&BI processes
  • Take a modern approach to data governance that powers deeper collaboration without sacrificing security and compliance
  • Accelerate time to insight and boost productivity by enabling teams to reuse and repurpose analyses
  • Extend the value of data analysis and insight to partners, customers, and industry leaders

Chapter 1:
Benefits of a Community-first Approach
to A&BI

Collaboration is not an entirely new concept to the world of analytics and business intelligence. Today, nearly 4 in 10 organizations use collaboration to support analytics processes. But while collaborative analytics tools have existed for the better part of the last decade, they haven’t been able to connect larger business communities or allow for more meaningful contributions to the A&BI process from knowledge workers.

More work is needed to fill the gap that has emerged between business experts and IT. And as we’ll see, many BI tools that employ these collaborative capabilities have actually created further divides between teams, rather than uniting them.

If organizations are to achieve their data-driven goals, it’s clear that a community-driven approach is required. But what does a community-driven data organization look like in practice? And what benefits can data and business teams expect if they adopt such an approach?

Faster time to data insights

It’s never easy to derive insights when data is siloed or out of reach from decision-makers. Data sources and datasets can easily remain undiscovered or get overlooked without someone pointing them out to the larger business team. When a broad spectrum of people serving a variety of roles is involved in the data analytics process, an organization can identify and share all its valuable data and put it to use.

Companies that adopt a community-driven approach eliminate these barriers and empower teams to discover the most up-to-date information when they need it by bringing it into existing workflows and generating data-driven insights on demand. Whether it’s by embedding analytics into company software tools and portals — like Salesforce — or creating team workspaces where relevant datasets reside and can be used for analysis, these approaches drastically reduce the time it takes to surface insights.

With Sigma and Snowflake, we reduced our time to data insight by 93%.

Jessica Nielsen Data Analytics Manager, The E.W. Scripps Company

More relevant and valuable data for decision makers

Companies sit on a treasure trove of data. But 73% of it goes unused. In a community-driven world of analytics, people can put data to work because it’s not only more accessible, but also properly curated, pre-joined, labeled, and provided in business context. Collaborative teams can better use the data on-hand when business context and domain expertise is incorporated into data models and sets, making all data relevant to a given question clear and approachable.

Data and BI teams can use community-driven analytics processes to reduce friction and enable entire organizations to make the best use of existing data with the right data modeling approach. By curating and endorsing the most relevant and accurate data for business users, they help teams unlock greater value.

Smarter and faster data-driven decisions

By adopting community-driven analytics initiatives, companies can get to the heart of the “why” faster, and ensure the answers are more accurate. Any person that has a question is empowered to dig into the most up-to-date and accurate information, test their hypothesis, and trust the answers they find. This cultural shift nurtures curiosity and helps frontline workers start thinking and acting like data analysts.

Sigma has proved to be priceless in helping us make more informed decisions and empowering all employees to think like analysts.

Lucy Dana Product Manager – Growth, Blue Bottle Coffee

Happier, more fulfilled data and BI teams

27% of data experts report feeling ‘unfulfilled’ or ‘very unfulfilled’. These feelings largely stem from disappointment over what they imagined themselves doing in their roles versus the reality of their day to day work. Data and BI teams feel trapped in report factory hell and never ending ad hoc queues, with little time for higher-value data projects that lead to greater personal fulfillment.

In a community-driven analytics environment most ad hoc workflows get shifted to frontline workers on the business teams, freeing up data engineers and scientists to work on projects that they enjoy and deliver more value to their organization — such as uncovering new data sources, building new data models, and solving more impactful problems.

Greater productivity and fewer redundancies

The rise of hyperspecialization was supposed to help companies become more efficient and productive. Unfortunately, this hasn’t been the case for A&BI. Legacy BI tools build walls around data and require vast resources to deliver value and ROI from an initial data investment.

Companies that invest in community-driven analytics unlock higher value from their data by reducing the resources necessary to deliver reports, dashboards, and ad hoc analyses. By breaking down the technical hurdles that have stunted insights in the past, multiple areas of the business community can begin to extract value through new endeavors or by leveraging the work completed by others.

Instead of reinventing the analytics wheel every time a question arises, internal and external stakeholders can put relevant, curated datasets to work. They can also extend them to other tools in the analytics ecosystem. This dramatically reduces the amount of time, technical requirements, and effort required to deliver insights.

Sigma is an order of magnitude less work than our old tool, which required us to learn a proprietary coding language, and prevented us from being agile.

Chris Lambert CTO, Payload

Chapter 2:
The Barriers to Community Data Collaboration are Real

Although organizations aspire to make A&BI collaborative, the reality is that many organizational, cultural, and technical roadblocks exist. Here’s a look at some of the most common obstacles that hold companies back from achieving their community-driven goals.

The relationship between data and people breeds frustration

At the heart of the community-driven analytics movement lies the relationship between data and people. They know the importance of using data to make decisions. Still, the lack of structured processes, poor data literacy skills, and a common data language between data creators and data users causes more than 60% of data projects to fail.

A recent survey report illustrates the complexity, frustration, and tensions that arise from this relationship. Despite technology improvements and higher spending on A&BI, 86% of business experts report that their data skills have room to improve. Not being versed in the ‘language’ of data has caused many domain experts to feel self-conscious or even intimidated by their relationship with data.

About 35% say they are not very confident or not at all confident in their ability to articulate their data questions or needs to their BI teams and analysts. And 30% report feeling embarrassed by their lack of data knowledge or skills. This lack of skills, fear and embarrassment around data leads to tension between BI and business teams.

Each group points blame at the other for poor collaboration. 20% of domain experts report they rarely or never feel their data needs are adequately met by their data team. Meanwhile, data experts are quick to lay the blame on their less technical colleagues, with 36% having felt that domain experts don’t understand what can and can’t be done with data.

There is frustration and distrust on both sides of the business. Data experts feel that data gets used inaccurately to produce insights. And business experts feel they aren’t getting what they want or need from the data team.

The only way to close this gap and eliminate the frustration is for business and BI teams to work in closer harmony with each other to impact the business. Sadly, that’s not how their relationships have developed — or how traditional A&BI tools have worked to support them.

Data remains siloed from internal and external stakeholders

Data remains elusive for organizations and people. It’s spread across a patchwork of systems, software, and databases. These silos bury data and make accessing data and sharing insightful analyses a challenge for employees, customers, and partners because a central source of data truth often doesn’t exist.

The need for better data collaboration and sharing capabilities leaves companies struggling to meet demand from external stakeholders in particular. Rising demand to collaborate with partners through data has emerged and is reflected in a recent Accenture C-suite survey, where 36% of executives indicate that the number of organizations they partnered with had doubled or more in the last two years. The same survey also revealed that 71% of executives anticipate the volume of data exchanged with ecosystems to increase.

Organizations recognize these challenges. A recent Harvard Business Review Analytics Services Survey suggests that 78% of companies highlighted the ability to easily access and combine data from a variety of external sources as very important for a data-driven enterprise. However, only 23% said they were currently very effective in this area, and just 15% shared data with key vendors and suppliers.

This is, of course, a technology challenge for many companies. Sourcing data and making it available to different departments, suppliers, partners, and customers is difficult with legacy data sharing practices. Outdated methods like data transfer via FTP are complex and error-prone. Copying files to and from cloud buckets is potentially insecure, and working with APIs is engineering-intensive. On top of that, the process of extracting, transforming, and loading data (ETL) is time-consuming for organizations sharing data.

If organizations are to overcome these challenges, they must adopt processes and technology that close these gaps and make it easier for business communities to work more closely together through data.

Outdated data governance frameworks exacerbate the problem

Data governance has traditionally faced a catch-22. An “ivory tower” governance framework that’s too strict stifles a company’s ability to be data-driven. Domain experts can’t access the data they need when they need it, and important decisions get made based on guesswork.

Ivory tower scenarios often lead business experts to download data directly to their PCs, which creates a “wild west” governance strategy that’s too loose and opens up unnecessary risk. This makes it nearly impossible to ensure compliance. Endless data extracts and spreadsheet sprawl cultivates shadow IT scenarios, increasing the chances of security breaches and unreliable or inaccurate use of data.

Depending on who you ask, complete open data access is the most progressive action a company can take, or the most frightening.

Rob Woollen CTO, Sigma

Currently, most companies are erring on the side of too-strict governance. In a 2019 survey by NewVantage Partners, executives reported that 95% of their difficulty in becoming data-driven is a result of cultural challenges around data. But everyone recognizes the importance of being data-driven in the modern world, and business leaders understand that being able to master the data governance balance will deliver a competitive advantage.

If you’re building a true community-driven data culture, you’ll need a new, more modern approach to unlock more value from data investments.

Lack of data access and familiar tools excludes business expertise

Despite rapid advances in cloud data platforms like Snowflake, most A&BI software tools have left the business community behind. These solutions are complex, require SQL or proprietary coding knowledge, and limit exploration and analysis to the most technical.

They trap business users in stale reports and surface-level dashboards that leave them with unanswered questions. Because they cannot drill down or explore underlying data, they must constantly turn to BI teams to get answers — which often takes weeks. In the business world, that can feel like a lifetime. Which is why it’s no surprise that 25% of business experts have given up on getting an answer they needed because the data analysis took too long.

These lengthy, siloed analytics processes are hardly conducive to the close-knit meeting of the minds required to achieve community-driven data processes. Not to mention, excluding business domain experts from the data conversation makes it extremely difficult for data and BI teams to conduct and present analyses in the context of business needs and goals.

Data experts face constant roadblocks and redundant analytics processes

Data and business analysts also feel pain. Analysts spend 80% of their time finding, deciphering, validating, and preparing data from raw sources and sprawling data marts before they can even start an analysis.

They often wait for data engineers to gather data from raw sources and data lakes, and complete complex joins between data warehouses, data marts, and tables. This wastes valuable time, holding people back from generating insights and acting quickly.

To make matters worse, the need to frequently update data models every time a new question gets asked slows data experts down and often results in datasets getting used once. All the prep work isn’t made useful beyond the original analyst or across the ecosystem — meaning efficiency is lost and redundancies exist throughout the A&BI workflow.

The legacy analytics & BI environment simply wasn’t built to empower communities. These outdated paradigms only serve the most technical and cannot keep up with the increased demand for real-time insights. But newer approaches exist that focus on community enablement and driving more value from company data.

Chapter 3:
How to Take a Community-Driven Approach to A&BI

We know that the current paradigm isn’t working. But what does this new approach look like in practice?

Taking a community-driven approach to data analytics empowers teams to share valuable insights, build on each other’s analyses, and collaborate to make better decisions faster. But it takes time, a conscious and deliberate cultural shift, and the right tools to make it a reality in every organization.

Get everyone speaking the same data language

Data literacy is no longer just a nice to have skill for business teams, it’s key to growth. Before an organization can become data-driven, it must work to develop a culture that promotes curiosity and arms everyone with the skills and tools to work with data in meaningful ways.

Without a strong background in the fundamentals of statistics and an understanding of the systems at play, business teams are bound to fail. They need to grasp how to approach data, ask good questions, and explore data to find their answers.

Bridging the gap between data and business teams to improve data literacy across the organization starts at the top. Leaders of both of these groups must band together and lead the charge to break down the data language barrier. A comprehensive data literacy program prepares everyone to participate in the data conversation, surface impactful insights, and work together to drive exponential business growth.

Put data in context for business teams

Data access is not enough to deliver value to decision-makers. To amplify the value of the data, and ensure it’s used properly, BI teams need to present the information to domain experts in a way that makes sense to them, and that allows them to extract insights.

Data modeling plays an essential role in making this happen. Good data models ensure accuracy, context, and trust between teams — all of which are required if your organization wants to embrace a community-driven culture of analytics.

Data modeling is another area that has seen tremendous progress in recent years. By the time data gets to the analysis stage, it’s already been collected, transformed, and modeled at the warehouse layer. But semantic data modeling and final clean up are often required before it’s useful to domain experts. This is especially important if your company uses a cloud data warehouse.

CDWs combine many aspects of traditional enterprise data warehouses and data lakes, meaning that some data will get modeled and ready to analyze — while other data will require some “last-mile” data prep. This last mile is where the focus shifts from technology to people. Here, data is inserted into people’s everyday workflows to help influence decision-making and usually includes extracting semi-structured data, filtering out values, deduplicating data, linking datasets, adding useful descriptions, and more.

Take a modern approach to data governance

Community-driven analytics and BI requires a solution that takes a truly modernized approach to data governance by making data accessible and approachable for everyone while upholding strict compliance and security standards.

To achieve success, companies need enough control over data to maintain security compliance without stunting collaboration by keeping data walled off from those who need it to make decisions. This requires investing in analytics solutions that don’t rely on data extracts, copying, or caching — and leverage the data warehouse to perform real-time analytics without migrating data. By keeping data inside the data warehouse, you can avoid shadow IT scenarios like spreadsheet sprawl that escape IT and data teams’ purview.

Using a CDW-native analytics tool connected to a cloud data warehouse provides a third way. Data teams can prepare and model the data in the warehouse and then create an approved workspace for business users. There they can ask and answer questions themselves without actually touching data in the warehouse. This way, all teams maximize their expertise, and those closest to the data have a seat at the table no matter where their expertise lies.

This approach requires solutions with robust role-based permissions based on how employees, partners, and even customers interact with data, conduct analyses, and share insights in a secure environment.

Maximize everyone’s specialized expertise

In today’s hyper-specialized business environment, everyone has something to bring to the analytics conversation. While data and BI teams possess the technical expertise and background to build systems for easier analytics processes, they usually lack the understanding those closest to the data (marketing, sales, product, finance and operations specialists) can contribute to conversation.

To harness the collective intelligence of everyone in your organization — as well as your partners, customers, and other industry leaders — you’ll need to ensure that you’re making the best use of everyone’s role and experience.

This means freeing data teams to escape endless ad hoc queues to work on higher-value projects, and empowering the business teams to conduct analyses, build dashboards and reports, and share insights with those inside and outside of the organization without help from data experts.

Extend the value of your data externally

Understanding how different levels of the business community interact with data and use insights to drive value is fundamental to community-driven A&BI. Without this understanding, data and BI teams can’t align systems to maximize the inherent value data provides.

You need a modern analytics stack that drives value for internal stakeholders. Still, you’ll also need to find secure ways to share data outside of your organization to lift partners, customers, and the broader industries your company engages with every day.

With Sigma’s Application Embedding capability, we were able to create data-rich and interactive dashboards that show our customers all of the key metrics they need for daily decision-making and embed them directly into our proprietary products without any interruption to the service we provide to our customers.

Chris Lambert CTO at Payload.

This will require investing in solutions that offer tools like embedded analytics functionality to securely embed dashboards in external-facing and custom applications. Secure data exchanges where people can share and use the most relevant datasets are another key capability to seek.

Extract more value throughout the data value chain

To amplify the value of company data, you must provide a platform that sources and surfaces data so that knowledge workers can make the most use to inform future decisions. Data and BI teams can drive higher value from existing data by eliminating redundancies and silos in the analytical process and empowering better use of data throughout the data value chain.

Organizations must adopt technology that increases the value at the tail end of the data value chain. These solutions make it possible not only to gain access to relevant information but also to continually iterate on other’s work, eliminate analytical redundancies, and make more profound impacts throughout the business community.

To unlock more value, you’ll want to look for a data platform and analytics tool that:

  • Isolates analytical workloads to ensure high performance at all times for your entire organization
  • Provides capabilities to create and share iterable datasets throughout the analytics ecosystem
  • Delivers balanced data governance features that promote data access without sacrificing security or compliance
  • Allows anyone to dig into underlying data behind dashboards and worksheets, and build on other people’s analyses
  • Enables embedded dashboards, visualizations, and reports into existing business workflows

Chapter 4:
Become Community-Driven with Snowflake and Sigma

Community-driven A&BI requires a single integrated platform that everyone, regardless of technical ability, can use to explore, analyze, and share data. With the combined power of Snowflake and Sigma, you can centralize your data sources, increase data governance and security, expand data access, and unlock rapid analytical insights for any analyst or domain expert in your business community.

Snowflake

As a data platform, Snowflake brings all your data together and makes it fast and easy to use for everyone. By standardizing access to governed data, the correct data gets used to make business decisions. Snowflake’s unique cloud architecture provides data and business teams with consistently fast queries so that more users analyze more data and collaborate with their peers through secure data sharing.

Data sharing is a unique part of Snowflake’s technology architecture, which allows for secure collaboration on shared data. It enables data sharing without the need to copy or move it to the data consumer’s account by creating a virtual pointer to the original source. This means that when sharing data via Snowflake, the receiving party can immediately access data without having to download or process the data through an ETL pipeline — providing the data in ready-to-query format for immediate use. Since the shared data is a virtual pointer to the original dataset, every time the data is updated those changes get reflected for the data consumer immediately.

The challenges of sourcing and sharing data discussed in Chapter 2 are therefore eliminated with Snowflake, allowing companies to more easily access the data they need without the costs and overhead associated with traditional data sharing methods.

Accelerated data access

Snowflake provides isolated resources for data pipelines so that ELT (and ETL) workloads do not compete with analytics queries. Information can be continuously loaded as you ingest streaming data in real time. This means you never suffer from stale data and can always generate the freshest insights.

Faster queries

Analysts and business users no longer have to wait minutes or even hours for their queries to complete with Snowflake. They can make better, more timely business decisions without frustration.

Rapid, scalable transformation of data

Offered as a service, Snowflake seamlessly scales to 1,000’s of users for collaboration across the business. Resource isolation means everyone has a consistently fast, frustration-free experience. And customers only pay for what they use with per-second billing and rapid resource elasticity.

Centrally managed Data Exchange

Create your own data exchange as an advanced, yet simple way to share data, increase internal collaboration and find paths to monetize your data assets at scale with the Snowflake Data Marketplace. You can easily, securely, and cost-effectively share data within a controlled and managed environment. Invite employees, subsidiaries, partners, and customers to securely access data assets curated by your team or other members of the exchange — without having to move, copy, or transfer data. And leverage public datasets to enrich and supplement internal data and improve your analytics for faster and better-informed business decisions.

Better governance and control

Because you can manage the data exchange from your Snowflake account, you have all the tools necessary to ensure data privacy, governance, and security. You have complete control over what data is made available, and who can access that data. You can also view key metrics to track data access and usage, making it a perfect solution for exchanging data in a controlled and highly-secure environment.

Sigma

Sigma helps companies realize the full benefits of the cloud data warehouse and drive faster analytics adoption across every business team.

Instead of making business teams rely on analysts to provide insights, Sigma puts everyone’s power through a spreadsheet interface that doesn’t require SQL to ask the tough questions of your data. Sigma empowers both SQL fans and spreadsheet lovers to work together in a single platform that gives everyone in the organization direct, real-time, and governed access to Snowflake. This transforms analytics and BI into a community-driven process where teams can share, amplify, and accelerate their data insights.

How Does Sigma Enable Community-driven Analytics?

Sigma’s key community-driven features include:

Visual data modeling

Bring data and business teams together to build centralized models and reusable Datasets that make data approachable and insightful. Pre-model joins between data sources and models, endorse Datasets and worksheets, add context and descriptions to data objects, and more.

Dataset warehouse views

Model data once in Sigma and use it anywhere. Create Datasets in Sigma and write them as SQL views back to your cloud data warehouse. They can easily be queried and reused to power analyses and visualizations across your internal and external applications.

Dynamic dashboards

Create, manage, and share custom dashboards using Sigma’s worksheet visualizations. Explore data directly from the dashboard using filters and parameters, or click into the underlying data to easily explore follow-up questions and update visualizations as business needs change.

Embedded analytics

Extend the value of Sigma dashboards and visualizations by embedding them directly into internal and external applications. Authenticate users through those apps while maintaining data permissions directly from Sigma. You can further extend these analyses to partners and customers for a truly community-driven approach.

A new era for data governance

Community-driven analytics and BI requires a solution that takes a truly modernized approach to data governance by making data accessible and approachable for everyone while upholding strict compliance and security standards.

Sigma supports this modern approach through:

  • Direct, real-time, guided access to the cloud data warehouse
  • No data extracts, copies, or caching
  • Role-based tools for data exploration, visualization, modeling, and enrichment
  • Instant visibility into the data questions teams are asking
  • Team workspaces where relevant data is made readily available
Next Steps

If you’re ready to start building a more community-driven analytics ecosystem and culture at your company, Sigma and Snowflake can help.

About Sigma

Sigma is the first enterprise-ready cloud business intelligence and analytics (A& BI) solution designed to run natively inside cloud data warehouses (CDWs). Providing live, guided access to CDWs, Sigma maximizes their value, eliminates the need to change data models as new questions arise, and transforms A& BI into an iterative process. The Sigma Spreadsheet empowers anyone to analyze data — without code or extracts — and make insight-driven decisions quickly, freeing data experts to focus on more innovative, fulfilling projects. Sigma powers a collaborative, community-driven approach to A& BI and delivers on the self-service promise.

About Snowflake

Snowflake’s cloud data platform shatters the barriers that have prevented organizations of all sizes from unleashing the true value from their data. Thousands of customers deploy Snowflake to advance their businesses beyond what was once possible by deriving all the insights from all their data by all their business users. Snowflake equips organizations with a single, integrated platform that offers the data warehouse built for the cloud; instant, secure and governed access to their entire network of data; and a core architecture to enable many types of data workloads, including a single platform for developing modern data applications. Snowflake: Data without limits.

No items found.
No items found.