Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards
Text-to-SQL is increasingly important in data analytics and database-driven applications. It reduces the effort required for experienced users to write complex SQL queries while enabling non-SQL users to query databases through natural language. Reflecting this growing demand, major data platforms now support text-to-SQL capabilities through systems such as Databricks Genie Agents, Snowflake Cortex, Gemini in BigQuery, and Amazon Quick Sight.
To support the development and evaluation of text-to-SQL systems, researchers have developed a growing number of benchmarks. However, we found pervasive annotation errors in two widely used text-to-SQL benchmarks: BIRD and Spider 2.0-Snow.
Do these errors change how we judge the state of the art in text-to-SQL agents? To answer this question, we re-evaluated all 16 open-source agents from the BIRD leaderboard, uncovering relative performance changes ranging from −7% to +31% and ranking shifts of -9 to +9 positions. The previous state-of-the-art, Contextual-SQL, dropped from 1st to 7th, while GenaSQL (6th) and CHESS (7th) rose to a tie for 1st. These findings show that annotation errors can significantly distort reported performance and rankings, underscoring the urgent need for high-quality text-to-SQL benchmarks. And if you are an agent or model developer, don’t blindly trust benchmark scores or leaderboards! Dive deeper into our findings in our latest paper.
Annotation Errors Are Pervasive in Text-to-SQL Benchmarks
We conducted an in-depth analysis of annotation errors in BIRD and Spider 2.0-Snow, two benchmarks widely used by both academic and industrial teams (e.g., Google, OpenAI, Databricks, Snowflake, Amazon, Alibaba, and ByteDance).
Our analysis consists of three stages: (1) an AI agent producing per-annotation diagnostic reports; (2) human SQL experts adjudicating all cases flagged by the agent and assigning final correctness labels; and (3) to identify errors the agent may have missed, the experts also review unflagged examples for recurring error patterns uncovered in stage two.
Through our in-depth examination, we found that:
BIRD Mini-Dev has an annotation error rate of 52.8%
Spider 2.0-Snow (August 20, 2025 date cutoff) has an annotation error rate of 62.8%
In the rest of this blog post, we first assess the impact of these errors, then break them down into four recurring patterns with illustrative examples, and finally introduce our AI agent we designed to assist finding these errors.
Annotation Errors Can Significantly Change Measured Agent Performance and Leaderboard Rankings
We evaluated all open-source text-to-SQL agents listed on the BIRD leaderboard as of August 20, 2025. To quantify the impact of annotation errors, we randomly sampled 100 examples from the BIRD Dev set and manually corrected all annotation issues we identified. We then re-evaluated all 16 open-source agents on both the original subset and the corrected version.
After re-evaluation, we found
Agent execution accuracy changes ranging from −4% to +19%
Agent ranking changes ranging from −9 to +9 positions
Notably, CHESS and GenaSQL demonstrate large relative improvements (+30.6% and +26.6%, respectively), moving from mid-ranked positions (7th and 6th) to a tie for 1st place at 81% execution accuracy on the corrected subset.
These results demonstrate that the measured performance and rankings of agents are highly sensitive to the annotation errors in the benchmark. When error rates are high, the reliability of the benchmark and leaderboard deteriorates, and it may no longer reflect the true relative performance of different agents.
Takeaway: Our findings highlight the urgent need for high-quality text-to-SQL benchmarks. And if you’re developing or selecting a text-to-SQL agent for a downstream application, don’t blindly rely on existing benchmark scores or leaderboards—verify the data and inspect failure cases.
Where Do These Errors Come From?
A text-to-SQL annotation instance consists of three components: the natural language input (T), the ground-truth SQL query (Q), and the target database (D). Annotation errors can occur within any of these components or arise from inconsistencies among them. We categorize such errors into four recurring patterns:
E1: Mismatches between the semantics of Q and the logic of T.
E2: Mismatches between the semantics of Q and D due to limited understanding of the data and the schema.
E3: Mismatches between the semantics of Q and the domain knowledge relevant to T, or misannotated domain knowledge in T.
E4: Ambiguity in T, such as multiple possible interpretations or an unclear output format.
We categorized examples by error pattern (each example may exhibit multiple patterns). E2 is the most common in both benchmarks: among examples with annotation errors, it occurs in 57.79% of BIRD Mini-Dev examples and 57.89% of Spider 2.0-Snow examples.
Human-in-the-Loop Error Detection
To make error detection scalable and systematic, we introduced SAR-Agent (SQL Annotation Reviewer), to our knowledge, the first AI agent designed to assist SQL experts in detecting text-to-SQL annotation errors.
SAR-Agent: SQL Annotation Reviewer
Given an annotation (the natural-language input and the “ground-truth” SQL) and the target database schema, SAR-Agent interacts with the database to generate a diagnostic report assessing the correctness of the annotation. Human SQL experts then review SAR-Agent-flagged cases, verify the stated reasons, and assign the final correctness label. To identify errors the agent may have missed, the experts also examine unflagged examples for recurring error patterns uncovered during this review.
SAR-Agent Helps Experts Find More Errors
We evaluated SAR-Agent’s ability to detect annotation errors by comparing it against SQL experts from the Arcwise team. After manual verification, we found that SAR-Agent correctly detects 138 of 161 Arcwise-reported error cases (85.7% hit rate). In addition, SAR-Agent identifies 41.6% more error examples than the Arcwise team.
We released Arcwise-Plat-SQL and Arcwise-Plat on our GitHub repository, extending Arcwise’s corrections to produce cleaner, more reliable BIRD Mini-Dev variants. We also made the SAR-Agent code publicly available.
Conclusion
In this work, we find that widely used text-to-SQL benchmarks contain high annotation error rates, which substantially undermine the reliability of benchmark results and leaderboards. We develop a toolkit and offer practical guidance for building high-quality text-to-SQL benchmarks.
To better support both text-to-SQL research and real-world deployment, benchmark developers should prioritize annotation quality—not just quantity. We advocate using our proposed pipeline to create more reliable text-to-SQL benchmarks. Please see our paper for details and our GitHub repository for the released resources.