I’m Sorry… Text To SQL And Chatting With Your Data Is Not The Answer

For more than 25 years, we’ve strived for data democratization: enabling anyone to leverage their company’s data to gather insights and make decisions. With the recent AI advancements, many products pitch they have a solution: a natural language chatbot where users can simply ask their data questions.

These natural language query chatbots all follow a similar pattern. There is a search box where a user types a question, an LLM “thinks” and then answers with a visualization, and if you dig deeper you can see the SQL or Python it generated for you.

At first glance, this seems like an amazing solution. The demo videos are slick, and it really seems like if you just had ChatGPT for your data, this problem would be solved.

However, if you dig a little deeper, I think you’ll come to the same conclusion as me: text-to-SQL and chatting with your data are just lazy attempts at adding LLMs into a UI and calling it innovation. There’s no deeper thinking about how it makes sense to work with AI in daily workflows. There’s no extra care about how to investigate the AI’s response. There’s just a flashy demo with the latest LLM, purporting to solve all your business needs without really thinking about how your business works with data.

Don’t rely on “magic” answers

What’s the first thing that happens immediately after you announce a new data insight? Usually, someone responds with a doubtful, “How did you get that number?”

Your answer can’t be, “The magic AI system told me.”

Microsoft’s answer is you’ll simply read the Python code: “You can view the code it’s running in real time and check its work.”

That’s quite an ambitious statement, considering that understanding and validating generated code is difficult even for advanced programmers. Does anyone really think the average person can quickly read Python or SQL code in real time and know whether it’s correct?

No. We all know that won’t happen. Realistically, these products are asking people to just blindly trust that the AI gets everything right. Are they accurate enough?

90% is an F in this class

Many systems proudly announce an accuracy metric for their text to SQL or AI models. The best now claim close to 90% these days on their benchmarks.

First of all, this is on a benchmark, not your data warehouse. It’s difficult to know if the model has been overtrained and can actually deliver these results outside of the benchmark data.

But even if we believe the 90% accuracy, it’s misleading. No one guesses the perfect question and gets data insights by running one query. An exploration could easily take 10-20 queries or likely more.

And that’s the problem. Run one query, and there’s a 90% chance the AI gets it right. Ask 10 queries, and you’re down to ~35% odds it gets them all right. By 20 queries, you’re down to a 12% chance every query is correct.

So now your business user has 20 blocks of SQL or Python code and a high likelihood that at least one AI-generated answer is wrong. Is this what data democratization looks like to you?

Semantic models won’t save you

Modeling your data, establishing metrics, relationships, descriptions, etc. is absolutely essential for AI (and humans) to effectively and correctly analyze your data.

Some vendors tout their chatbot is 100% accurate because it leverages a semantic model. Technically, this is pretty easy: on any question, force the user to disambiguate anything and only ask questions that use pre-defined metrics and dimensions.

This works, but it defeats the entire purpose of having a natural language interface. The great thing about natural language is you can ask open-ended, ambiguous questions: “How is our company doing?”

If you want to restrict users to only using pre-defined measures and dimensions and nothing else, then just build them a dashboard. A dashboard makes what “users can ask” clear. It’s much easier picking a measure from a dropdown than trying to guess what measures are available via natural language.

Natural language is great for ambiguous, ad-hoc questions—not reinventing dashboards. If you have to be very specific and exact, you might as well write formulas or SQL.

Enabling data experts is not enough

Text to SQL and Python copilots are great if you’re a SQL/Python programmer. It’s clear these copilots can accelerate authoring SQL and Python.

Just making data experts more productive won’t solve the self-service challenges. Data experts can never be experts in every domain. Asking data teams to thoroughly understand sales, marketing, product, finance etc. has never been realistic nor can data teams anticipate every question/insight their teams will need.

Data democratization is still the right challenge. These interfaces just aren’t the right answer. 

So what is the right answer?

To truly democratize data and unlock self-service analytics, you need an AI system that everyone can trust. AI that encourages deeper learning. AI that gives you full visibility into what it did, in a language anyone can understand—human language. 

Sound better than text-to-SQL? It is, and we built it. We call it Ask Sigma: agentic AI that actually shows its work. 

Ask Sigma is already live for our customers and seeing rapid adoption. But we’re constantly adding more to it. 

Join Sigma’s AI showcase on May 20th to see what’s new in Ask Sigma (hint: admins will love it) and learn how our teams deliver AI innovation—in a way that actually answers the problem of data democratization.