I just learned about Vanna.AI, so I thought I’d write a comparison article since I’m deeply familiar with DataLine’s inner workings.
In this article I’ll go over the feature differences, an example of quality differences on a sample dataset (with pictures of both), and importantly, privacy and security differences that are relevant for users.
TLDR: VannaAI is more tailored towards developers, and single question/answer flows. It supports advanced integrations and has built a lot of features around contextual knowledge. DataLine is also suitable for non-technical personnel (ex. PMs, BAs) and is focused on a simpler and less customizable user experience.
What are these to begin with?
Both Vanna.ai and DataLine were created to help developers and non-technical people query their data with natural language. They’re both open source, can generate SQL, tables, and charts to answer user queries. They both target regular users as well as enterprise customers, with self-hosted deployments and fully open-sourced codebases.
To look at the differences, we have to dig deeper. There differences in how data security is handled, and some feature differences as well, both of which are relevant for enterprise users mainly.
Vanna.ai Features
Vanna supports quite a few features:
- Web interfaces built on top of Python’s Streamlit, allowing re-use of community building blocks but limiting frontend development to Python
- Connecting to multiple SQL databases (Snowflake, Postgres, BigQuery, SQLite, and more)
- Supports adding contextual examples like example queries or documentation to aid with query generation on complex datasets (which they market as “training” to communicate to the masses)
- Has a user interface with chat, that generates SQL, results, and plotly charts that are generated by executing python code sent by the LLM (more on this later)
- Supports a slack connector, allowing access to results from slack threads
- Is also a Python library, allowing others to extend it building their own products and interfaces on top
DataLine Features
DataLine is built a little different:
- Web platform is built from the ground up in React and FastAPI, allowing the broader dev community to contribute to a standard web app and allowing for easier customization of UX/UI
- Connecting to multiple databases + sources is supported (CSV, SQLite, Postgres, MySQL, Snowflake, and more)
- Does not yet support adding contextual examples. Experimented with this at some point but failed to get a significant quality bump so scrapped it, but we’re working on it again now.
- Has a chat interface with SQL and table results
- Major difference in charting: DataLine has ChartJS charts that allow for customization and replayability of results when the SQL is modified. More on chart differences in the security section 👀
Enough talk! How do they differ in quality?
I could dig into the technical details of both implementations, but not all of you are interested in that, so I’ll jump right into an example. I’ll still mention a few things on the tech side at the end!
Let’s start with Vanna. I booted up a container to install it and run the interface in a sandbox environment (more on this in the security section). It was a little frustrating to run locally, the docs all recommend the hosted solutions. Honestly took me around 10 minutes to get things up and running, but was excited to try it out!
I will use GPT-3.5-turbo for both tests, and a Titanic dataset (https://www.kaggle.com/datasets/ibrahimelsayed182/titanic-dataset) to keep things simple.
Testing exploration abilities
It had some issues with one of my sqlite samples, but the titanic dataset worked fine. It was very frustrating that it wasn’t a chat however, I had to ask a new question in a new page every time. Also had to modify the SQLite query to fetch table metadata to make this work.
In DataLine, I went with the command line install and ran brew tap ramiawar/dataline && brew install datalineand got it running with dataline
Also got some examples and it worked fine:
2. Specific data fetching
This was pretty fast compared to DataLine. That’s due to the more complex flow in DataLine internally. Vanna.ai has a straightforward flow on the other hand: generate SQL, generate chart, and we’re done. That can be a positive or a negative (in case query fails or chart generation fails for example, DataLine could self-correct and try again until it succeeds).
Alright. Let’s spice it up. Can I ask multiple questions? Running out of ideas here!
3. Multiple questions:
Okay Vanna got flustered with this one.
DataLine has got result grouping as well, related tables/charts/SQL is grouped visually to help users understand which answers reply to which questions!
Enough yapping, let’s move on to the last section.
Privacy and Security
I dug into Vanna.ai’s code and noticed a few things.
First of all, the only things sent to LLMs are metadata about the table. Schemas and such. That’s similar to DataLine in this sense, and it can be disabled optionally through code. You could go and add the flag `allow_llm_to_see_data=True` to let the LLM see your data.
You can also do this in DataLine with a simple toggle:
Moving on to the spicy parts: How are the results generated?
Well both systems here perform similarly. They both run the query locally, and they fetch the results. That means nothing goes over the internet (unless you’re using the hosted version of Vanna.ai, but this comparison will look at the more secure options).
How does charting work in both cases?
Vanna.ai asks LLMs to generate Python code that draws a Plotly chart to draw the graphs. They mention that this is unsafe in their “hardening” docs section (https://vanna.ai/docs/hardening-guide/).
What does this even mean? Why is it a big deal?
Well if the LLM is generating code that will be run dynamically on a user’s computer or on a company server, then there’s a chance this code is malicious because it can’t be fully trusted. It’s also a vulnerability because if hackers ever impersonated the LLM, they could execute arbitrary code on those machines. I’ve already seen it mentioned by cybersecurity researchers.
To turn it off is not very simple. You’d have to override a function in the code to prevent it from actually generating code dynamically. In that case, Vanna will attempt to auto-deduce what kind of chart to plot. In general, Vanna is very tailored towards developers I feel like.
You can see the Python exec and auto inference here.
DataLine’s approach
DataLine takes a simpler approach: Using an LLM to generate ChartJS json and then rendering the chart.
This also gives the frontend the ability to easily regenerate the chart and customize it, something that isn’t as easy when using plotly and controlling the chart from the backend.
Quick summary of technologies used
Techies, this one is for you.
Digging into Vanna’s code, they seems to use custom constructions from the ground up. No langchain or mirascope or anything for doing the LLM queries. I think this is very lightweight and is what gives them fast response times. But sadly we live in a world where LLMs make mistakes and tools fail, so I think there needs to be more work done to handle such failures gracefully.
DataLine on the other hand uses Langgraph and Mirascope to build a state machine with different nodes. This allows DataLine to be more flexible in it’s responses and more resilient to errors.
Already talked about charting and result generation.
To conclude
There’s a lot more in both tools that I didn’t go over, I just wanted to cover the basics: Will this replace writing SQL? Can non-technical people use this? And I think I answered those questions. I could dig deeper into different areas if you’re interested, so be sure to leave comments!
