Introduction: The Evolution of Modern BI
2018 was an exciting year for self-service analytics and cloud data. We saw a renewed focus on citizen data scientists, and many forward strides in data privacy and security in the wake of GDPR legislation. With 2019 well underway, we’re excited for what’s to come over the next year as businesses continue investing in solutions to democratize data analysis across their organizations, and migrate to the cloud.
Despite forward movement in 2018, Business Intelligence (BI) still leaves much to be desired—which is why widespread adoption across organizations has laregly failed. BI was invented to answer questions at the highest level of an organization. Since then, the needs of the BI community have evolved. The modern data economy demands more as companies dig deeper and ask more of their data.
Today’s answers should not only help you steer decisions at the top, but deliver the insight to fundamentally shift the way your organization operates at every level.
BI needs to shape up and start solving the problems that permeate below the c-suite and affect every level of the modern enterprise.
What trends will drive the industry in 2019? And more importantly, how will they affect the way you make use of data?
Read on to get our top predictions for 2019, and learn how they will impact everyday workflows as you tackle the challenging questions that lie ahead.
Spreadsheet Interface is King
Last year, there were three main approaches to making BI more accessible: simplifying modern cloud interfaces, improving search, and adding Natural Language Processing. In 2019, the familiarity, power, and specificity of a spreadsheet interface will win widespread support over the simple requests possible with Natural Language Processing (NLP) and search.
“Take any spreadsheet you’ve built and imagine the most intelligent system in the world: what would you tell that system to make that spreadsheet? It wouldn’t be a sentence; it would be a book.”
— Rob Woollen, CEO of Sigma
The spreadsheet is unmatched in its power and familiarity. It is a critical tool that business experts already use daily across every industry. From online tools like Smartsheet and Airtable to modern BI tools like Sigma, spreadsheets are everywhere.
85% of people use spreadsheet software in their work, and 76% of people rate their skill level as Excellent or Good. The familiarity with the interface gives people a head start when learning a spreadsheet-like interface.
Spreadsheets often serve as de facto BI tools for companies. When exploring data, 62% of people use Excel most often because of its flexibility and ease of use. Even people with technical skills love spreadsheets. 88% of people who write SQL still use Excel when exploring data.
BI tools that utilize existing knowledge around spreadsheets can open accessibility to BI in ways that code-based tools cannot match.
People want more access to data, and a flexible interface makes that possible. Adding a spreadsheet interface to cloud data combines the benefits of both mediums.
“If you deliver a complicated solution that’s difficult to navigate and presents a lot of technical information that’s not clear to a business user, they’ll be less likely to adopt it,” says Gina Papush, chief data and analytics officer for QBE. “But if the solution is highly visual and, ideally, it’s integrated within the system of record that business people use to perform their jobs, adoption will certainly be more successful.” 
Spreadsheets give users a familiar and powerful tool to explore data that can’t be matched by existing BI solutions— even with the addition of search and advanced NLP.
Sigma is like working in a spreadsheet with the power of Snowflake—giving our business experts access to the compute power of the cloud but with the freedom they’d get from a spreadsheet.
— Aash Anand, Data Analytics at Lime
In 2019, BI will take inspiration from the spreadsheet, and the evolution in self-service will become powerful and widespread.
Natural Language Processing Disappoints
Natural Language Processing (NLP) is heralded as the key to democratizing data. The problem is, BI platforms must bake in multiple definitions before someone can ask a seemingly simple question. NLP is just a new way to ask the same predetermined questions that ultimately leads to more requests from business teams.
The dream is that NLP can take any question and turn it into a query, without someone having to use a specific language. This isn’t a new concept. “The history of SQL is natural language,” says Jason Frantz, CTO at Sigma. “How it’s written is supposed to mimic natural requests. The problem is, natural language isn’t specific. To write a request and be sure that it will be interpreted correctly, you end up back at a programming language.”
Many hope that NLP brings analysis to a broader audience. In reality, the familiarity and power of a spreadsheet-like interface will succeed in bringing BI to business experts.
Traditional BI tools limit business people with predefined questions that a data team has pre-modeled for them. Some interfaces offer flexibility, letting people filter data and click to drill down. But users are still confined to the questions the data team thought they would ask, and they can quickly run into the edge of what the modeled data can show.
The promise is that NLP will help with those data limitations. Rather than dashboards, people can ask any question they want. That utopian dream quickly falls apart in practice.
“What’s the change in revenue in the Northeast over the last three quarters?” Within this simple request are multiple definitions. How is the revenue calculated? Are the regions in the database called “Northeast” or are there multiple names that all map to the region? How does the company define a quarter?
Either definitions are pre-modeled, which strips NLP of its supposed power, or the processor decides on definitions that may—or may not—line up with definitions in other places in the company.
In the latter case, you need to explore how things are defined and potentially change them to ensure metrics line up. For that, you either need an interactive user interface, or an essay written to pick apart the nuance.
The democratization of data needs both power and access to thrive. While NLP may simplify the process of asking basic questions, it doesn’t have the necessary power to change how business teams interact with data.
The Hybrid Cloud is Dead
For companies that invested in the cloud early on, your time has come. In 2019, we expect the cloud-first BI stack to reach maturity. With the awkward teen years in the rearview mirror, data-forward companies embrace cloud-first technology, seeking the benefits of a fully cloud-native stack and the modern data policies that come with it.
Previously, companies explored a hybrid approach to cloud data, but with the maturation of cloud technologies, the hybrid approach delivers the worst of both worlds. “It’s combining the cost of having your own data center with the inflexibility of not using the cloud,” says Jason Frantz, CTO of Sigma. “It’s like putting a carburetor in your horse — it’s a terrible idea.”
Cloud-first applications fully realize the benefits of the cloud because they are no longer beholden to the limits of on-premises. The ability to elastically scale up and down on computing power—paying only for what you use rather than what you may use—and the flexibility to change based on your business needs have become possible through the advent of a cloud-first approach.
Today’s cloud-first applications offer a nimbleness and flexibility that is truly unique. More data storage, more users, and new integrations can all happen with a click. There’s no infrastructure to provision and build. It moves at the speed of your business, giving data access to the whole company.
These trends also drive greater self-service analytics adoption. Usage elasticity makes cloud technology the key to self-service analytics. Having enough power on-prem to handle peak demand means you are over-provisioned 99% of the time. It becomes easier to limit access to the main server and have people work on data extracts. The elastic nature of cloud-first services allocates resources as needed, so you aren’t paying for extra users or the hardware to support them. Rather than always paying for the maximum potential usage, you only pay for exactly what you use. The seamless integration of cloud-first apps builds on those strengths. The trial process for different technologies is instant, and the data passes between applications with ease. Snowflake’s Partner Connect is one example of this type of integration. With Snowflake, you can test an ETL solution like Fivetran or a cloud built analytics solution like Sigma with a few clicks. Partner Connect provides a convenient option for trying additional tools, and then adopting the ones that best meet your business needs.
Flexible pricing enables businesses to easily experiment with their data stack to find the combination of cloud-first services that best fit their needs. This fluidity creates positive changes in how companies interact with their data.
On-prem solutions limit computing power and user seats to keep costs down, cutting most people off from self-service. The cloud stack’s flexible pricing and dynamic provisioning provides everyone with access to data—making it easy to utilize analytics tools when and how you want. The integration of cloud-first technology makes true self-service possible for the first time.
The Modern Workforce Demands Better BI Tools
As businesses continue to realize the value of domain experts performing analysis, the top-down, overly-governed approach taken by traditional BI tools will fail to deliver the insights needed to steer decisions.
“Analytics needs to provide a bottoms-up approach,” Jason Frantz, CTO of Sigma says. “People are smart. They’ll figure out interesting things in their domains.”
Current BI solutions often cause frustration the moment someone wants to answer a unique question.
People across the business are becoming more involved in analytics. A new generation of tools designed with the needs and skills of business people in mind helps fuel the transition to an organizational approach.
“BI has gained a bad rap because it doesn’t give the answers that people are looking for. Users often hit a wall, either blocked by a UI constraint or needing to use SQL to go deeper into a question. “
— Dave McCandless, CIO at Navis.
Traditional BI tools promised easy access to data, but business experts remain limited—forcing them to find other solutions.
Excel is still the dominant tool used to explore company data. That’s true even for people who have access to a BI tool. Only 31% of people said they primarily used their BI tool to explore data, while 50% said they used Excel most often. Spreadsheets offer flexibility and ease of use that isn’t matched by traditional BI platforms.
When blocked by impenetrable BI tools, people create new ways to answer questions. While helpful for many things, Excel doesn’t integrate into a cohesive, integrated analytics flow.
“To perform calculations, analyze data, and create graphs, spreadsheet users will import data from numerous data sources,” says David Stodder, the Senior Director of TDWI Research. “However, well-documented problems can arise due to the manual and haphazard fashion in which the data is prepared, cleansed, and transformed for personal use. Spreadsheet data chaos is a significant pain point for many organizations”.
Companies seeking a balance between control and accessibility now look to BI tools that empower their business experts—and deliver on the promise of data access for all.
“Organizations are struggling to find consistency across their key metrics when relying on spreadsheet-driven analytics, damaging business user trust.”
— Austin Kranz, Assoc. Principal Analyst at Gartner
Smart Cloud Pricing Gains Traction
Companies are tired of paying for what they don’t use, and with the cloud, they don’t have to. Rather than accepting huge upfront costs based on an educated guess, cloud technology provides pricing that scales as your organization grows.
Cloud utility pricing is elastic, meaning companies can quickly dial up and down as solutions gain traction. Upfront provisioning of resources with fixed costs may have been necessary in the days of on-premise hardware, but the cloud has changed this. And it makes sense—cloud pricing should reflect the flexibility it offers.
The Slack model of only paying for active users is coming to BI. Usage-based pricing forces BI to show engagement, rather than being a fixed cost with questionable ROI.
When an entire company or department can be onboarded right away, collaboration isn’t constrained. Anyone can get set up in minutes, with costs accruing only when they engage with the tool. You’re only paying for people who are doing the analysis. Everyone can view insights and user costs only occur when the tool is providing valuable analytic capabilities.
“Set up is a big overhead for us because we’re going to be hiring for the foreseeable future. We don’t want to make people jump through hoops to do their job.”
— Aash Anand, Data Analytics Team at Lime.
With flexible pricing, usage can increase organically as new use cases are found, instead of needing to renegotiate a contract before sending a coworker a report.
Data Access and Understanding Remains a Major Issue
The type of data companies care about continues to change and grow over time. In 2019, the ability to access and analyze that data is going to become a key focus. It’s no longer about more data—it’s about useful data.
Data velocity is only increasing, and data diversity along with it. Semi-structured data like JSON data is now the norm. With every webpage you visit, JSON data is flowing back and forth between systems. BI tools aren’t keeping up: they can only deal with the data once it is flattened. This means most people can only access information once the data team cleans and curates it. It’s another casualty of the traditional approach of data teams playing gatekeeper.
“When people are looking to explore their data, asking someone to pre-model it defeats the point,” says Jason Frantz, CTO of Sigma. “The people who need the data don’t always know what’s contained in the JSON — they can’t tell you what they want yet. Traditional BI can’t handle the ambiguity.”
The gap between access to useful data and the ability to develop practicable insights has doubled since 2012, according to MIT Sloan Management Review.
In a recent study, 52% of people said they have been unable to access the data they need to perform their job and 63% are unable to access the data in the required timeframe—while 38% of people have given up on asking for a piece of data entirely.
Outdated systems (36%), data not accessible (31%) and overworked BI/staff (29%) were the top three factors leading employees to seek workarounds. When needing to work around the lack of access to data, people most often resorted to estimating figures (51%) and using old data or incomplete data (over ⅓ ). 
Data from ResearchScape suggests that outdated systems, data access, and overworked BI staff contribute to employees seeking workarounds—such as Excel.
Even when people have data, 44% of people are working with data that is at least a week old, 21% are working with month old data.
According to ResearchScape, access to data is holding back business users from generating useful insights.
We live in a digital world where every second matters, meaning week old data is obsolete before analysis has even started.
 Computing Curiosity Gap Study, Researchscape 2018
Your Dad’s Dashboard Gets a Facelift
Static dashboards are relics of the old school analytics era that are now being left behind by modern, data-forward companies. In 2019, context and collaboration will be integrated into dashboards. Rather than just showing high level trends, dashboards will become a place to kick off exploration and collaboration.
“People are more demanding of their products. The consumption only mode is no longer acceptable,” Jason Frantz, CTO of Sigma says. “Consumers are no longer excited about exploring the answers to questions that have already been asked for them. “
With traditional BI, dashboards are a stop sign, not a starting point. Dashboards are built and information is pushed out from the data team, with no opportunity to ask deeper questions or launch collaboration. “Our local teams don’t need dashboards,” says Garvit Patel, Data Analyst at Lime. “Dashboards are often a high-level, aggregated visualizations. Our business experts were unable to ask additional questions, pull in more data, or easily create regional slices.”
Modern cloud collaboration starts with the dashboard: it’s no longer a dead end.
Often, the number of people with full access to the company BI tool severely limits who can update or build on dashboards. If someone wants additional context about what’s happening on the dashboard, that means a ticket to the data team rather than an opportunity to explore on their own. That is finally changing.
Now flexible pricing models and performance elasticity is shrinking the divide between dashboard consumers and dashboard builders. Having the entire organization on the platform is finally feasible.
Dashboards will evolve to become an invitation to explore the data, not the intended endpoint. For years, dashboards have been little more than billboards. They may help raise awareness, but they don’t answer questions. Modern BI is taking the dashboard in a new direction, to make it the center point of collaborative analysis. Each person will be able to interact beyond the limited confines of dashboard filters, diving into the data that powers the high-level analysis to find the information most relevant to them.
Bringing together people and data will be key in 2019. “One key to digital business success will be combining technology and talent in new ways to create and sustain new, high-value business and operating models,” says Gartner . How companies bring together people with technology is ripe for a paradigm shift, and opening up data access has the greatest potential to build business value.
“One key to digital business success will be combining technology and talent in new ways to create and sustain new, high-value business and operating models.”
Data Teams and Business Experts Finally Speak the Same Language
It’s time the data team and business teams were on the same page. Better data literacy across the organization —combined with tools that better serve business experts—helps close the communication gap between data teams and the rest of the organization.
“Employees are the biggest asset to any company. They are smart, they know their domain — that’s why you hired them,” says Rob Woollen, CEO of Sigma. “Self-serve, ad-hoc analysis puts data in the hands of the people that know the domain best. It unlocks the full potential of your employees.”
Tools designed with the needs of business experts in mind are starting to create more opportunities for collaboration between the data team and the line of business people they support. It helps put data where it needs to be. Data is the language of business. It’s no longer the domain of a single team—it’s the shared responsibility of every employee.
Increased data knowledge creates a common language between the data team and business experts, helping prevent confusion and back and forth. More importantly, improved tools help business experts access and learn the data in a way that is intuitive to them.
Education and access to data both work together in closing the gap between business experts and the data team. As Gartner says “Organizations no longer can treat technology and people investments as two separate activities.”
Any time you move your data it creates vulnerabilities, whether that’s moving data to an extract, having people download data to their PC, or using emails to share reports.
Bringing together people and data will be key in 2019. Gartner states in The Future of Work and Talent: Culture, Diversity, Technology, “One key to digital business success will be combining technology and talent in new ways to create and sustain new, high-value business and operating models.”
How companies bring together people with technology is ripe for a paradigm shift, and opening up data access has the greatest potential to build business value.
Data Analysis as a Collaborative Platform Fueled by AI & Cloud Capabilities
If you discover insights, and you have no way to share it, does it make an impact? Communication is key. Collaboration is key. But both are difficult with traditional BI tools.
Collaboration tools have undergone a renaissance. Slack and Google Docs changed how people work together, and set expectations for how easy things should be. It’s time for BI to catch up.
Collaborative BI means being able to work seamlessly and effortlessly with internal and external partners, easily finding and building on the most relevant analysis. Fully-cloud systems and improved AI-driven algorithms enable new collaborative platforms.
“Good AI in the context of Business analytics is about recommendations. It’s about how the system can help you find the data you need. It balances the system with human signals.”
— Rob Woollen, CEO of Sigma
These improvements in AI algorithms will help business experts find data and complete analysis without the intervention of the data team. Collaboration between the people closest to the data will become more seamless, without the intermediate filter of having to explain to the data team precisely what the question is and why it is important.
Collaborative systems should allow anyone to view and build on questions raised when looking at a dashboard. They should be able to share their analysis with a coworker, and have their coworker build on their work. It should be easy to share insights with partners outside of the organization, so collaboration can extend to outside teams of experts.
With traditional BI data is either up to date, easily shareable, or in a form that allows you to be flexible in how you ask questions. It is never all three at once.
“Excel offers accessibility. But when doing analysis, you have no idea what other Excel users are doing. There’s no collaboration, no way to see what you might be able to build on. Companies need a platform that offers accessibility, collaboration, and access to live data.”
— Rob Woollen, CEO of Sigma
Often, spreadsheet programs become a solution to add functionality where BI has failed. “Excel offers accessibility,” says Rob Woolen, CEO of Sigma. “But when doing analysis, you have no idea what other Excel users are doing. There’s no collaboration, no way to see what you might be able to build on. Companies need a platform that offers accessibility, collaboration, and access to live data.”
Being able to see the analysis your coworkers are building and the data they are referencing lets you build on the base that someone else has created. It helps build analytic compound interest, where one insight can quickly lead to others. Collaboration helps surface the most useful insights, without locking people into predefined questions.
Organizational BI, AI-powered augmented analytics experiences, and interactive dashboards combine to create a new generation of BI that finally has the technical capabilities to promote seamless collaboration.
Data Security and Privacy Stay Top of Mind
With data breaches in the news every other day, along with new sets of laws aimed at data privacy going into effect, there will be a laser focus data security in 2019. And for good reason, the average data breach cost companies $3.86 million last year—up 6.4% from 2017.
A rash of data breaches in 2017 prompted legislation to help promote data security . That trend will continue in 2019 and beyond. The key takeaway will be the exploration of how to balance security and access.
“The safest place for your data is in your data warehouse,” says Rob Woollen, CEO of Sigma. “Any time you move your data it creates vulnerabilities, whether that’s moving data to an extract, having people download data to their PC, or using emails to share reports. With Sigma, you can analyze your data without moving it.”
The physical storage of data is only the start of the security journey. How the data is accessed and used has many consequences that add up. Data extracts are common in legacy BI tools, but extracts mean that someone has to consistently maintain the security of the data in two separate places—and the second extracts are created they become out of date.
Excel can also be a major source of security concern. Once you extract data to an Excel file, the data team and security team have no idea where that file has gone. Files often get emailed to the wrong person, or emails can be hacked to find information. And unfortunately, you cannot rely on Excel’s password encryption feature for security. Microsoft’s help docs state, “You should not assume that just because you protect a workbook or worksheet with a password that it is secure.” 
The most secure place for data is in the data warehouse. Cloud computing can offer better physical security benefits that on premise. Cloud providers have government oversight to ensure their compliance with security standards, as well as dedicated personnel to keep data at scale secure. When handing over the physical control of your data, you are handing it to a company that specializes in keeping that data online and secure.
The increased concern over privacy and security will have IT departments looking at how to implement a data stack that minimizes the risk of security breaches.
Conclusion: Closing the Gap
There is more data generated than ever before—and more demand for it from every level of organizations. Keeping up with the growing demand for this data is a challenge all companies face in the year ahead. The good news? New tools and analytics platforms are rising to meet these challenges head on. Cloud Computing makes data more accessible, no matter where you sit in the business. And flexible pricing and computing power can help your team scale dynamically by eliminating older payment models and replacing them with more affordable options that allow for leaner operations.
So what does this new approach hold for companies seeking an alternative path? By tackling the issues facing data teams with new cloud-based tools, an organizational BI solution can help your company meet growing demands and solve problems faster than ever before. Business experts want to be involved in asking questions and getting answers from their data. And why shouldn’t they?
It’s time businesses acknowledged this and act to bridge the gap that has emerged with legacy solutions. Open new doors for workers at every level of your organization by providing them with tools that are built with this framework in mind.
Position your company to take advantage of the new BI landscape. Learn how Sigma’s intuitive spreadsheet interface for your cloud data warehouse can change the way your company approaches data by signing up for a free trial today.