Your Agent's 80% Accuracy Band Is a Coin Flip. The Only Benchmark About Your Data Is the One You Write.

On August 13, Hex published DataBench: 100 analytics tasks run inside Shorelane Commerce, a company that does not exist. It sells office supplies, books about $129 million a year, and has six years of the kind of data every real company has. Customer IDs were dropped in a 2021 platform migration. An acquisition never fully merged. A channel called “mobile” became “mobile_app” without a backfill. There are five plausible columns named some version of revenue. Stripe and Salesforce tables sit beside a Shopify dataset that is mostly a red herring.

Ten of the tasks are traps, questions with an obvious answer that is wrong. Averaged across the models in Hex’s first sweep, Q&A tasks passed about 75% of the time, open-ended tasks about 66%, and the trap tasks about 54%: a coin flip, on exactly the questions where a wrong answer looks like a right one.

Hex’s author, Izzy Miller, names the pattern: “Frontier models do not like to throw in the towel.” The right answer to a lot of real questions is that the data can’t support one. That is where the models scored worst. One was asked whether to merge customer records from the 2021 acquisition. It noted that no identities had been linked, then recommended merging about 100 anyway.

Then the number moved. On September 17, Hex recalibrated its judges and rubrics, and its changelog says the benchmark got “significantly harder, especially for Anthropic models.” The leaderboard as of October 7 tops out at Opus 5.5 at max effort, 70.5%, at $3.57 and eleven minutes a task. Nothing about the models changed that day. The grader did. An accuracy score belongs to the test as much as to the model, and every score you have seen so far was written by somebody else’s test.

What’s actually shipping

TextQL, October 1: Argo-Bench. 210 tasks on a simulated New York food-delivery platform, 7.49 billion rows across 235 Oracle EBS tables, graded by what the agent’s actions would have done. The best of 14 models, Opus 5.5, averaged 59.5 out of 100. Cost per task ran from $0.06 to $4.89. Gemini 3.8 Flash and Muse Spark 1.3 averaged 197 and 250 model calls a task to Opus 5.5’s 82, and scored less than half as well. The objective did the damage. Asked where to cut $9.6 million from a courier-bonus budget, Opus 5.5 saved $3.09 million of an attainable $3.12 million, while GPT-6 Astra’s plan lost $86,281 where a uniform cut would have saved $0.40 million. Telling the agents what the bonuses were for, and that a holdout existed, lifted the average across every setting from 18.5 to 61.7. Spend and effort did not buy judgment. Picking the objective did.

Snowflake, August 28: Cortex Agents. Snowflake now recommends moving from Cortex Analyst to Cortex Agents, which uses Analyst as its tool for structured queries and adds search over unstructured data, multi-step orchestration, and threads that remember the conversation. The release note promises “higher answer quality.” It publishes no number for how much higher, or on what. The useful line is further down: semantic views carry over, and the verified queries stored in them “continue to apply.” A verified query is a question with a known right answer. That is an eval case someone on your team has already written.

Anthropic, October 8: Claude Dashboards (beta). It connects to Redshift, BigQuery, ClickHouse, Databricks and Snowflake, plus connected apps including Salesforce, builds a dashboard from a plain-language request, and keeps it current. Each chart shows its refresh time, and any number opens to the query behind it. Anthropic pitches it for quick, exploratory questions alongside BI tools, and has published no accuracy results for it. The surface that will carry an agent’s number into the QBR now exists. Its only freshness signal is a timestamp. Nothing on it says how often numbers like it have been right.

Hex, August 4: Evals. The same machinery, pointed at your workspace. A suite is a YAML file of up to 100 cases. Each case is a prompt plus rubrics: a numeric check against a target that can be a SQL query, with a tolerance (1% by default), or an LLM judge that reads the whole thread and can grade whether the agent flagged a known discrepancy or refused when the context was thin. Each case runs one to three attempts, and the suite can run on a schedule. Chime, profiled at launch, gives each of its 50-plus data domains its own eval set and gates the domain before the company gets it. Read the pass rule before you trust the green: with three attempts, a case passes if two of them do. A passing case can be wrong one time in three.

The counter-read is fair. Argo-Bench simulates a food-delivery company, not a CRM, and its authors say plainly that it compares agents rather than predicting how they will do on your warehouse. If anything it flatters them: the authors left out the drift and the tables that disagree, the mess DataBench builds in on purpose. Each model and effort setting got one run per task, so the paper has no run-to-run variance; its confidence intervals reflect the choice of tasks only. Your agent’s bands might cover 60%. They might cover 30%. A benchmark you can dismiss as simulated is a question you can answer with data you own.

What I’d build this week

An accuracy benchmark of your own: 30 questions your business has already answered, run against whichever agent wants to answer the next one.

Business moment. The first time an agent’s answer is about to go into a deck that someone else will present.

Data shape. The Salesforce objects already landing in Snowflake (Opportunity, Account, and whatever your sync names their history tables), plus the verified queries already sitting in your semantic views. Results go in one Snowflake table, agent_eval_runs: run_at, model, effort, case_id, case_type, attempt, answer_value, target_value, passed, stated_low, stated_high, stated_level.

Eval shape. Ten cases of each kind. Ten answerable, each a numeric check whose target is a verified query, with the tolerance written down. Ten traps, where the obvious field is the wrong one; most Salesforce orgs have a pair like Amount and the custom ARR field finance actually reports. Ten unanswerable, where the data does not exist. An unanswerable case passes only when the agent says it can’t answer. Any case where the agent states a range gets its low, high, and stated confidence logged, so coverage builds up week by week.

System shape. The suite lives as YAML in the repo beside the semantic models and goes through the same review. Hex Evals runs it as written (hex eval run); tag each rubric with a measure (answerable, trap, unanswerable) and Hex reports three pass rates instead of one. Snowflake’s own Cortex Agent evaluations, generally available with version targeting since August 21, let you pin the judge’s version, which is the lesson of September 17 written as a config key. The same 30 prompts and targets work in any harness; the cases are the asset, not the runner.

Latency budget. Weekly on a schedule, and once before any change to a semantic view or agent guide merges.

Cost shape. 30 cases at three attempts is 90 agent runs a sweep. At DataBench’s published per-task costs, from $0.25 (GPT-6.1 Sol at medium) to $3.57 (Opus 5.5 at max), that is roughly $23 to $320 a sweep. Hex’s prices on Hex’s tasks, not your invoice, but it tells you the order of magnitude: an hour of an analyst’s time, not a line in the budget.

Instrumentation. Store every attempt, not the case verdict. Report three-for-three separately from two-for-three. Score the three kinds separately and never average them; the unanswerable column is the one no vendor publishes. Count refusals on answerable cases too, so an agent can’t buy the third column by giving up on the first. For every stated range, log whether the target landed inside; coverage is hits over ranges, by model and stated level.

Two weeks in. Success is three numbers per model and effort setting, answerable accuracy, trap accuracy, and the share of the time it said “I can’t answer that,” that you would say out loud before an agent’s answer reaches a deck, plus a first count of how often its 80% bands held. Credible failure is not being able to write ten unanswerable questions, because nobody on the team can say what data you don’t have. That is a finding too, and it is the one the agent will find first.

Hex built a fake company to find out where the agents go wrong. You already own a real one.

---

Sources

• Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows — Tomitsuka, Raayatsanati, Xing, Gand, Ma (TextQL), arXiv 2610.02122, October 1, 2026 (210 tasks, 14 models, 235 tables and 7.49 billion rows; Opus 5.5 at 59.5 mean; nominal 80% intervals containing the realized value 44.8% of the time across 4,553 series from 3,346 runs on 72 tasks; courier-quest cut and the 18.5 → 61.7 hinted lift; 82 / 197 / 250 model calls per task; $0.06–$4.89 per task; greenfield warehouse without drift; one run per setting). https://arxiv.org/abs/2610.02122

• Introducing DataBench — Hex, Izzy Miller, August 13, 2026 (100 tasks, Shorelane Commerce and its defects, ten trap tasks, about 75% Q&A / 66% open-ended / 54% trap, Fable 5 Max acquisition-merge example, Fable 5 at 85/100, GPT-5.6 Sol judge with majority of three at 96% agreement, benchmark kept private; vendor-authored). https://hex.tech/blog/databench-agentic-analytics-benchmark/

• DataBench leaderboard — Hex, last updated October 7, 2026 (Opus 5.5 Max 70.5% at $3.57 and 677 seconds a task; Fable 5 Max 18%; GPT-6.1 Sol Medium at $0.25; v1.1 changelog of September 17 fixing judges and rubrics that rewarded misleading responses). https://hex.tech/databench/

• Introducing Evals — Hex, Andrew Lee, August 4, 2026 (YAML suites in version control, LLM judge over the full thread including refusals and flagged discrepancies, up to three runs per case, scheduled runs, Chime’s per-domain eval sets). https://hex.tech/blog/evals/

• Evals documentation — Hex (numeric_value rubric with a SQL target and 1% default tolerance, 1–3 attempts with two of three required to pass, measures, 100-case suite limit, schedules, Team and Enterprise plans). https://learn.hex.tech/docs/agent-management/evals

• Snowflake recommends transitioning from Cortex Analyst to Cortex Agents — Snowflake release note, August 28, 2026. https://docs.snowflake.com/release-notes/2026/other/2026-08-28-cortex-analyst-transition-cortex-agents

• Version targeting for Cortex Agent and Cortex Analyst evaluations (GA) — Snowflake release note, August 21, 2026 (metric versions pin the judge model, prompt, rubric and thresholds). https://docs.snowflake.com/en/release-notes/2026/other/2026-08-21-cortex-agent-eval-version-targeting-ga

• Build live dashboards and animate explainers with Claude — Anthropic, October 8, 2026. https://www.youtube.com/watch?v=en0GuyhieQk