Golden Data

Issue #6 · June 14, 2026

The agent became a graph.

Paul Brown Writes Golden Data from Ogden, Utah 6 min read

Every Sunday I look at the demo a vendor will run on stage next week. This Sunday the demo runs on Salesforce’s main stage — Multi-Agent Orchestration general availability, Atlas Reasoning Engine 3.0 routing requests across specialist subagents. Last week the press release became a split test. This week the agent stopped being a call. It became a graph.

For eighteen months the AI feature on your roadmap was a single LLM call wrapped in a few tools. This fortnight three install-base vendors and a model lab quietly admitted the unit moved. The orchestrator is the SKU. The subagent is the line item. The graph is the feature.

The procurement spreadsheet about to grow a row for each specialist agent the orchestrator routes to, and a column for the routing decisions themselves. The CTO defending an internal AI feature now has to defend a graph, not a call.

What’s actually shipping this week

1. Salesforce ships Multi-Agent Orchestration GA on June 15. Summer ‘26 release, announced May 22. Production rollout in waves starting June 13, full availability June 15. Atlas Reasoning Engine 3.0 ships as the routing layer — orchestrator agents inspect which subagents are registered, read their descriptions and available actions, and route the work to the specialist with the right tools. The IT Service Domain Pack ships fifty pre-built specialist agents. Tableau MCP brings analytics into the agent path. The CRM install base turned the orchestrator into the GA product the buyer can sign for.

2. ServiceNow named the production volumes on its Autonomous CRM. Knowledge 2026, May 5. CRM AI specialists available now across qualification, quoting, fulfillment, invoice disputes, service, renewals. The monthly numbers on the slide: over 100 million cases resolved, 16 million orders orchestrated, 7 million quotes configured. Amit Zavery, president and chief product officer: “Advisory AI has run its course. Enterprises need AI that senses, decides, and securely acts.” The install base went from describing the agent to disclosing what it did at scale.

3. Anthropic shipped Multi-Agent Orchestration in Claude Managed Agents. Code with Claude, San Francisco, May 6. A lead agent breaks a job into pieces and delegates each to a specialist with its own model, prompt, and tools, working in parallel on a shared filesystem with persistent events. Netflix has deployed it for the platform team. Anthropic also shipped Agents for Financial Services on May 5 — ten ready-to-run templates covering pitchbooks, KYC screening, and general ledger reconciliation. The model lab is shipping vertical multi-agent templates the way Salesforce ships standard objects.

4. The reliability research caught up to the production claim. A May 2026 arxiv paper on multi-agent LLM orchestration for incident response (arxiv 2511.15755) ran 348 controlled trials: 100% actionable-recommendation quality from the orchestrated configuration versus 1.7% from a single-agent baseline, with 80x specificity and 140x correctness improvements. The MAESTRO evaluation suite (arxiv 2601.00481) instrumented per-step telemetry across multi-agent architectures so failures land at the routing step that caused them. The structural finding: orchestration moves the eval problem from “did the model say the right thing” to “did the right specialist see the request.” The eval harness on the next AI feature has to grade routing, not just generation.

What I’d ship in your app this week

The unit changed. Two simple tools your team can ship in two weeks so the AI feature on your roadmap survives the multi-agent move that already happened in your buyer’s other vendors.

Feature one: the per-step eval harness for the agent path you already shipped. Most production AI features are already multi-step — retrieve, reason, tool call, post-process. Treat the path as the unit. One row per (request_id, step_name) with a pass/fail and a cost. Daily roll-up by step. The step where pass-rate drops is the next sprint.

  • Shape. Trace span per step. No new infrastructure — emit OpenTelemetry spans from the existing path. The harness reads the spans.
  • Data shape. One row per (request_id, step_name, tenant) with pass/fail, latency, tokens in, tokens out, dollars, evaluator note. Lives next to the existing analytics table.
  • System shape. One LLM-judge call per step, run nightly on a sampled 5% of yesterday's traffic. Golden examples seed the judge per step type.
  • Latency budget. None on the production path. Nightly batch under 30 minutes per million spans.
  • Cost ceiling. Under $80 per month at five million daily requests. The judge call is the marginal cost; everything else is storage.
  • Eval. 100 hand-graded judge-vs-human comparisons per step type per month. The judge is wrong below 85% agreement — retune the prompt.
  • Instrumentation. Alert when a step's pass-rate drops more than 10 points week over week. That is the routing failure showing up as a quality failure.
  • Two weeks in. Your team can name which step is the weakest link in production. If they can't, the harness isn't reading the right boundaries — re-cut the spans by user intent, not by code module.

Feature two: the per-customer agent activity feed, on the customer’s admin surface. The CSM team needs to point at the work the agent did. Show it. One feed per tenant — every agent run that touched their account, with the outcome, the tools invoked, the dollars spent. Customer-facing. Trust is built from the failures shown, not the successes claimed.

  • Shape. Simple feed in the customer's existing admin panel. Filter by agent, by outcome, by date. Static render — the data is already in the per-step harness above.
  • Data shape. One row per (tenant, agent_run_id) with timestamp, agent name, outcome category, dollars, top-three tools invoked, and a one-line summary written by the orchestrator at run end.
  • System shape. Aggregation query on the trace table. No new LLM call — the summary line was written at run time.
  • Latency budget. Under 2 seconds to render the last 100 runs. 8 seconds for the 90-day filter.
  • Cost ceiling. Storage only — under $30 per month per 5,000-seat tenant.
  • Eval. 40 customer-facing scenarios reviewed monthly by a senior CSM on "would a thoughtful customer renew after reading this."
  • Instrumentation. Track which feed entries the customer's admin clicks into. The most-clicked failure is the next renewal risk — fix it before the QBR.
  • Two weeks in. Three customers cite a specific agent run in their next QBR — by run ID, not by feature name. If they don't, the feed is too aggregated — show the individual run, not the rollup.

Both ship in two weeks with the team you have. Both turn the multi-agent path you already have into something the buyer and the seller can read together. The vendors who shipped the orchestrator as a SKU this fortnight are answering a procurement question that has not been asked yet. The vendors who haven’t will be answering it on the next renewal call with a single LLM call and a hopeful demo.

Sources

Send me an email and we will talk. If something here landed close to what you're working on, the door is open. No calendar funnel, no pitch deck — I read every note that comes in.

Doing the work rather than deciding what to build? Crafting is the column for that chair.

Email me Read more Signal

← All Signal issues · Drafted with Claude · Edited by Paul Brown