You can't ship non-deterministic systems on vibes. Evals — with observability underneath them — are taking the place tests and ops took a decade ago. The live argument is about how much to trust the LLM judge.
Six talks across three tracks, and at least four more that touched it — the most-worked discipline of the conference. One speaker even noticed the convergence from the stage.
¶1Ron Au gave the theme its plainest definition: “the whole reason we have evals instead of unit tests is because we have to evaluate these natural language problems.” Au ▸ ≈16:03 Tanya Dixit showed what the lazy version looks like — most LLM-as-judge prompts in production are one line, “rate this response 1–10 for helpfulness,” and that one line is “the fitness function for your self-evolving agent.” Dixit ▸ slide Her warning follows directly: “every round of self-improvement makes it more confidently wrong if you're iterating on the wrong thing… that's the paradox of self-evolution.” Hence her title: evaluation precedes evolution. Dixit ▸ ≈01:55
¶2The production story came from REA Group. Yicheng Guo's team shipped an AI property summary — not a demo; they'd iterated the prompts carefully — and in front of real users it hallucinated on “about five percent” of listings: stone benchtops that were laminate. Guo ▸ ≈06:35 The fix wasn't a better prompt. It was an eval system: deterministic software checks, then human labels, then an LLM judge aligned to those labels and run in the live pipeline. Hallucinations went from five per cent to zero reported — with the honest caveat attached: “we don't know what we don't know.” Guo ▸ ≈10:50
¶3Across the rooms, the same design rule kept surfacing: deterministic first. Dixit: checksums, schemas and totals are free compute — “the LLM judge only fires for what genuinely can't be checked deterministically.” slide Justin Barias's government team started judge-first and reversed: “treat evals as the loss function… start with deterministic graders — which is a bit of a theme that I've picked up across the two days.” Barias ▸ ≈17:45 (A speaker noticing the conference converging, from inside it.) And at Canva — 260 million users — the operating model is five layers of evaluation on top of tracing, with a takeaway slide that reads like the theme's slogan: “traces became the language we speak… evals are everyone's job now.” slide
¶4Underneath it all, observability. Daniel Nadasi's case from Google scale: instrument once per framework rather than per team, so developers “get it for free” — and accept that “you will never get to 100%, but you can get to 80% very, very easily, and you can get to 98% very, very easily.” Nadasi ▸ ≈12:42 Dave Slutzkin, sitting on terabytes of real coding-agent logs, added the working engineer's version: you don't have to review every line the agent wrote, “but I do think you should review the tests… you end up with very big and ineffective test suites if you let LLMs write every test they want.” Slutzkin ▸ ≈15:15
¶5The argument inside the theme is about the judge. One camp ships it: REA replaced human review with an aligned judge in production; Canva uses narrow judges so product managers can evaluate without engineers. The other camp rations it: “every LLM-as-a-judge call costs money… rather than setting a budget of 1,000 prompts, set a budget of confidence” Au ▸ ≈13:10 — and Jack Rudenko brought the awkward finding that models “judge to the same manufacturer”: ask a model to grade its own family and it's biased. Rudenko ▸ ≈09:24 Both camps agree on one thing — a judge is only as good as its alignment to human judgment, checked and re-checked.
“The whole reason we have evals instead of unit tests is because we have to evaluate these natural language problems in our space.”
“Every round of self-improvement makes it more confidently wrong if you're iterating on the wrong thing. More compute, more iterations, and it gets worse. That's the paradox of self-evolution… That's why evaluation precedes evolution. Before you build the self-evolution loop, build the rubrics.”
“We found out that we had about five percent hallucination rate in front of real users.” From the results slides: hallucination 5% → 0 reported · engagement up 10%.
“We were able to reduce hallucination from five percent to zero percent. Now I know what you're thinking — it can't be true… this is only theoretical, because we don't know what we don't know.”
“Treat evals as the loss function. You won't be able to do machine-learning-style optimisation without a loss function… Start with deterministic graders — which is a bit of a theme that I've picked up across the two days.”
“No agent developer will tell you they hate the idea of better observability — but it's not top of their stack rank. So you tell them: you're going to get it for free.” And later: “You will never get to 100%, but you can get to 80% very, very easily, and to 98% very, very easily.”
“I don't think you, the developer, have to review every line of code. I do think you should review the tests. You end up with very big and ineffective test suites if you let LLMs write every test they want — and unit tests fail very infrequently with AI-generated code.”
“Most of you doing evals are also doing LLMs-as-judges, and that costs money — every LLM-as-a-judge call costs money… Rather than setting a budget of 1,000 prompts to assess against, set a budget of confidence.”
“If you ask [a model] to judge output of its own family, it's going to be biased. If you ask it to judge another family of model, it's not… they judge to the same manufacturer of the models.”
“Never go and judge a model on the base of the benchmarks. Go and put it out there in a real-world scenario… it's really your engineering team or your domain team that should test out the different models for your use case.”
“Multi-armed bandits — the scientific shotgun for evals — is a way to do AI A/B testing in a cleverer way using old data science techniques. You can use it to save money on evals and serve users better agent outputs.”
“We're seeing a massive increase in the amount of CI resources… as we've adopted agentic coding internally it's been pushing problems into places we didn't think were going to be bottlenecks — that happened in six months.”
“Someone suggested you actually need an eval for your judge that's doing the eval. So it kind of ends up being turtles all the way down — which is probably not what we're looking for, especially if we're mindful of budgets.”
The judge argument, fairly stated: REA aligned a judge to human labels and shipped it live; Canva runs narrow judges so non-engineers can evaluate. The sceptics answer that judges cost real money (Au), favour their own model family (Rudenko), and still need humans at the top writing the ground truth (Barias — and Canva's own slide says a judge “will only be as good as its alignment”). One correction of our own belongs here: a line we'd repeated while planning this report — “we fired our LLM judge” — appears nowhere in the record. Nobody said it. The nearest real sentences are on this page, and every claim here is checked against the recordings. Including ours.
| Engineers | Deterministic checks before any judge — checksums, schemas and totals are free compute (Dixit). Version rubrics in git like code, and score per stage: “a scalar score at the end is a thermometer; what you need is a dashboard.” |
|---|---|
| Team leaders | Calibrate any LLM judge against human labels before trusting it, and re-check it (REA, Canva). Shadow-run changes against live traffic before serving them. Make “send me the trace ID” the debugging norm — and re-plan CI capacity for agentic load before it surprises you (Curtis: orders of magnitude in six months). |
| Org leaders | Point evals at business metrics, not proxies — “a proxy is what an agent learns to game; a business metric is what the business actually cares about” (Dixit). Fund observability once per framework so teams get it free (Nadasi). REA's receipt for the investment: 5% hallucination to zero reported, engagement up 10%. |
| Key takeaway | Stop shipping on the one-line judge. “Rate this 1–10 for helpfulness” isn't an eval — it's the fitness function your self-improving agent will learn to game (Dixit). |
Judge-first or deterministic-first — which is your team today, honestly? And what's the first check you'd move out of the judge into plain code?
For now these are placeholders. Once the individual sessions are up on Conffab, each title here — and every quote above — will link straight to the talk: video, transcript and all.