Golden Data

Issue #13 · August 2, 2026

The vendor wrote the definition of success

Paul BrownWrites Golden Data from Ogden, Utah6 min read

Every Sunday I look for the sentence a vendor writes when it thinks it is describing a feature and is actually describing a liability. Salesforce wrote one on June 25, in the pricing page for Agentforce Help Agent: you are charged when the agent resolves an issue autonomously, and if the customer asks for a human or walks away unhappy, there is no charge.

Read it twice. Success is the absence of a complaint. That is a defensible product metric. This month it went generally available as an invoice line.

Give the buyers their due. Agentforce launched at Dreamforce ’24 at $2 per conversation, billing the same $2 whether the agent solved the problem or the customer gave up and demanded a human. Eighteen months of pushback produced pay-per-resolution. The industry moved pricing off the seat, onto consumption, and now onto outcomes, and at every step the buyer got a better deal on paper.

Then look at who holds the pen. The vendor writes the definition, the vendor’s classifier applies it, and the vendor sends the bill. The buyers won the pricing argument and lost the definition argument in the same release. First the meter. Now the judge.

What’s actually shipping this week

Salesforce. Agentforce Help Agent, announced June 25 and generally available this month, charges $2 per autonomous resolution against a minimum pre-purchase of 1,000. Escalations, negative feedback, and abandoned conversations are not billed. Multiple questions inside a ten-minute call window collapse into a single resolution. Three vendor-chosen parameters — the definition, the bundling window, the prepaid floor — sit between what the agent did and what the invoice says. The credibility claim underneath the product is that Agentforce resolved 70% of 4.3 million inquiries on Salesforce’s own help site. That number is Salesforce’s, computed with Salesforce’s definition of resolved, and it is what sold the pricing model.

HubSpot. Custom agents left private beta on July 23, and every run now draws down HubSpot Credits at roughly $10 per 1,000, with per-agent caps, a spend ceiling, and a simulation mode. On April 14 HubSpot moved the opposite direction on its packaged agents: Breeze Customer Agent went to 50 credits per resolved conversation, where resolved means no escalation within 72 hours, and Breeze Prospecting Agent to 100 credits per qualified lead. Outcomes for the agent HubSpot built. Runs for the agent you build. A vendor only prices the outcome it gets to define.

The rate card, as published this week. Gorgias bills $0.60 to $1.27 by plan tier. Intercom’s Fin bills $0.99 and documents an explicit confirmation plus an LLM verification step, with conversation-level audit. Zendesk bills $1.50 on committed volume, $2.00 pay-as-you-go. Salesforce bills $2.00. One noun, a threefold spread, four incompatible definitions underneath it. HubSpot scores no escalation in 72 hours as a win. Salesforce scores the abandoned conversation as no charge. The same silent customer is revenue at one vendor and free at the other.

The research. No one has published an audit of a vendor’s billing classifier. The closest evidence concerns judges of the same kind, and it is not reassuring. “Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation” (Vaghasiya, Bhat, Mohsin, Aali; arXiv 2607.02577, June 30) audited four tool-calling benchmark families across 496 expert-reviewed tasks and found 92 evaluator-human disagreements — an 18.5% misalignment rate. The failure modes are the ones that would matter on an invoice: rubric drift, hallucinated completion, and scoring the final answer rather than verifying what the tool did.

Then there is reproducibility. Twenty-three runs of LiveMCPBench on an identical setup returned scores anywhere from 57.9% to 76.8%. Benchmarks are not billing systems. But the mechanism is the same — an automated judge deciding whether a task was completed — and the billing version has never been audited by anyone outside the company collecting the money.

What I’d ship in your app this week

Two builds. The first if you are buying these agents. The second, shorter, if you are selling one.

One: shadow-adjudicate the invoice. You are about to buy a metric your vendor computes. Build the second opinion before the renewal, not after it.

  • User moment. Month end. Your ops lead opens a five-figure invoice with a resolution count on it and no basis on which to disagree.

  • Data shape. One row per billed unit: (vendor_unit_id, conversation_id, vendor_label, billed_cents, reopened_within_7d, escalated_after_close, csat, our_label, disagreement_reason). Inputs are the vendor’s billing export and your own ticket table. You already have both.

  • System shape. A nightly batch join plus one structured-output call on the residual. Not an agent. Deterministic rules catch most of it — a reopen inside seven days is a rules hit, not a judgment call. Off the request path entirely, landing before the ops standup.

  • Cost shape. A few hundred tokens per disputed conversation. At 50,000 billed resolutions a month, under $100 against an invoice two orders of magnitude larger.

  • Eval shape. Two hundred human-labeled conversations, held out, relabeled quarterly. Do not take the number to the vendor until your labelers and your classifier agree at least 90% of the time. An unaudited judge is the thing you are complaining about.

  • Instrumentation. Log the disagreement reason, not just the disagreement. Reopened, escalated late, abandoned then churned. Three different arguments.

  • Two weeks in. Success is a sentence you can say out loud at renewal: nine percent of what you billed as resolved reopened inside seven days. The credible failure is that your classifier is as unreliable as theirs — which is why the human-labeled set is the gate, and why you report a range rather than a point.

Two: publish your own outcome definition. If you sell an AI feature, outcome pricing reaches your board deck this quarter. The differentiator will not be the price. It will be whether a buyer can audit the unit. Version the definition the way you version a schema — (outcome_id, definition_version, positive_criteria, disqualifiers, adjudicator, confidence, appeal_state) — run deterministic rules first with a model only on the residual, adjudicate overnight, and publish your disagreement rate against a reconciliation the customer runs on your own export. Intercom’s conversation-level audit trail is the current floor.

The credible failure here is quiet. The disqualifier list grows one clause at a time, your effective price drifts up, and nobody notices, because every clause was individually reasonable.

The vendors moved the risk of the wrong answer onto their own books. Buyers spent eighteen months winning that. They kept the pen that decides which answers were wrong. Ask for the definition in writing before you ask for the discount.

Sources

Send me an email and we will talk. If something here landed close to what you're working on, the door is open. No calendar funnel, no pitch deck — I read every note that comes in.

Doing the work rather than deciding what to build? Crafting is the column for that chair.

Email meRead more Signal

← All Signal issues · Drafted with Claude · Edited by Paul Brown