In late June, AI Engineer ran its World's Fair in San Francisco — three days, around nineteen rooms, five hundred and sixty-one program items, some seven thousand attendees. We sent our agents to the livestreams and captured the Main Stage: every keynote and main-track talk, all three days. This is a report on what that stage said — every quote checked, word for word, against the transcripts.
This isn't our conference, and we didn't see the other rooms — the Main Stage is about ten per cent of the program, though it's the ten per cent with Microsoft, OpenAI, Anthropic, Google DeepMind and Amazon on it. A few talks are missing where the capture failed: the Startup Battlefield finals, Garry Tan's closing, and day two's closing block. Everything else is here, and every quote below was machine-verified against the captured transcripts before it was allowed on this page. An LLM did the reading; the framing and editing are ours. Speech-recognition quirks are glossed with [sic].
The conference named its own days: Software Factories, then Autoresearch, then Harness Engineering — and the third of those is a phrase readers of our Melbourne report will recognise. More on that below.
¶1San Francisco didn't argue about whether the scaffolding around the model matters — it named a day after it, and WorkOS's Michael Grinich defined the term from the stage: “harness engineering is this new domain where if the models get better but you don't have a good harness, it's not gonna be very effective.” Grinich ▸ AWS's Mike Chambers gave it the bluntest definition of the week: “if you take an agent, remove the model part from it, and everything that you have left — that's the harness.” Chambers ▸
¶2The twist Melbourne didn't have: the model labs are absorbing it. OpenAI's keynote pitch was that they ship the harness they use — “we're not building one system for OpenAI and a second system that's simplified for developers. At every layer, we actually use the thing that we give to you.” OpenAI ▸ 13:24 Z.ai ships its own harness alongside its open-weight models. Dex Horthy argued the limit case — “no amount of harness engineering or loops-maxing can solve what is fundamentally a model training issue” Horthy ▸ — and Peter Steinberger, freshly absorbed into OpenAI himself, put the direction plainly: “models are advancing faster than the harnesses and organizations around them. Designing those things is the next engineering problem.” Steinberger ▸ 24:32 His unit of work: “the future is not 20 terminals. It's better loops.”
“Harness engineering is this new domain where if the models get better but you don't have a good harness, it's not gonna be very effective… Today, actually, it might be more important to build for agents than to build for people.”
“A harness is a set of straps and fastenings used to control an animal… If you take an agent, remove the model part from it, and everything that you have left — that's the harness.”
“We're not building one system for OpenAI and a second system that's simplified for developers. At every layer, we actually use the thing that we give to you.”
“No amount of harness engineering or loops-maxing can solve what is fundamentally a model training issue… models can't maintain and improve codebase quality over time, not without a decent amount of human steering.”
“I thought I was orchestrating. Really, I was polling [sic]. I was the scheduler, the router and the memory… I was primarily constrained by tokens. Now, I fixed it by joining OpenAI — I know, I know, this strategy does not scale. Now I'm primarily constrained by attention [sic]. And unlike tokens or compute, I can't simply add more of it.”
¶3Ask this crowd its biggest problem and you get one answer. Barr Yaron surveys the field every year: “every single year that I ask this, the number one answer is evals.” Yaron ▸ Sonar's CEO brought the cost of skipping it — teams see “a three to five x boost in velocity. That dissipates in three months” — and the fix: verification inside the agent loop, not after it, for “a 92% reduction in issues.” Shaukat ▸ Amazon's AGI lab drew the boundary that matters: code is verifiable; most work isn't — “verification really hits the wall right where most of our work lives.” Barth ▸
¶4And the deepest version came from Anthropic's Mike Krieger, on why review is the bottleneck even inside a frontier lab — not reading lines, but comprehension itself: “bottlenecked on human ability to even, like, fully conceptualize what we're doing.” Krieger ▸ Weights & Biases' Tim Sweeney compressed the whole theme to a sentence: “tasks and evals are the new world of CI.” Sweeney ▸
“Every single year that I ask this, the number one answer is evals… 96% of the people in the survey have a problem with the stack. You just can't agree on which one.” From the same survey: 95% now use agents; 89% of those agents can write data; the control layer is “the same toolkit you'd use to manage an intern.”
“What you see is a three to five x boost in velocity. That dissipates in three months… they can get a 92% reduction in issues if you take this guide, verify, solve approach inside of those agentic loops.”
“If your agent, one in four times, deletes the database, you will never touch that agent again… code is verifiable — you can run it, you can test it. There is no unit test that can answer those questions. Verification really hits the wall right where most of our work lives.”
“We are definitely still bottlenecked on review… bottlenecked on human ability to even, like, fully conceptualize what we're doing. — I don't know, man. This is, like, 2,000 lines of code. Like, it looks like code to me.”
“Tasks and evals are the new world of CI… you must use humans as a necessary judge. There are behavioral nuances that LLMs will not catch.”
¶5Melbourne established that spend is engineering; San Francisco named the market structure underneath it. Notion's Sarah Sachs, reprising and sharpening her Melbourne argument: “your supplier is your competitor” — and the upgrade trap in one line: “you try it out. It uses three times as many output tokens.” Sachs ▸ Her slide carried the quarter's headlines: Uber's 2026 AI budget gone in four months; one company spending half a billion dollars on Claude in a single month. Yaron's survey made it population data: “cost is now a first-class engineering constraint” — three of four engineers adjust usage on price. Yaron ▸
¶6The counterweight is real too. Google's Kevin Hou built an OS kernel from scratch — 93 sub-agents, twelve hours, two billion tokens — “and it was under a thousand dollars.” Hou ▸ DSPy's answer to both sides: hold your spec and evals fixed and swap everything else — “Shopify, 550 times cheaper.” Rivest & Miller ▸ And Anthropic's platform team reframed the whole line item: “tokens are not fungible” — give them jobs, price the true cost of an answer, not the request. Lesse & Jiang ▸
“Your supplier is your competitor… You will always be getting a bad deal on tokens with someone who builds them natively. — Bet on the frontier, not on the lab. If you don't have the capability to walk at any point, you are stuck.”
“It took 93 sub-agents over the course of twelve hours, made 15,000 requests, 2,000,000,000 tokens. And it was under a thousand dollars.” The kernel ran Doom.
“In programming, if we want to repeat a task often, we make it a function. We believe the same should be true for AI programs… Shopify, 550 times cheaper.” Same evals, cheaper model swapped in underneath.
“The thing we want everyone to think about is this idea that tokens are not fungible… You can expect, on average, to have to spend 1,800,000 tokens with the execution strategy to get to your perfect answer.”
¶7Day two's showpieces were honest in a way conference demos rarely are. Prime Intellect set world records on the GPT-2 training speedrun with off-the-shelf agents — and reported: “there was really no novel optimizer or mechanism that was coming from those models.” Eighteen clever constructions, zero new algorithms. Bakouch ▸ WeCo's agent became the top contributor in OpenAI's hiring challenge — “it wasn't a person” — and their conclusion still put people at the top of the stack: “the search is automated. The human would just move up the stack, not out of it.” Jiang ▸ Richard Socher took the maximal position — a Eureka machine to “invent pretty much all future inventions for humanity” — and even he located the constraint in verification: anything you can simulate, you can solve. Socher ▸
¶8Which leaves the bottleneck where Steinberger put it on day one: not tokens, not compute — attention. Ido Salomon: “so in reality, we are the bottleneck.” Salomon ▸ The machines run the loop; deciding what's worth looping on is still ours.
“Claude keeps stopping every nine or ten hours and basically says: I cannot improve the record. It's too hard for me. And Codex [sic], totally the opposite, just worked all the time… But there was really no novel optimizer or mechanism that was coming from those models.” From the slide: “18 clever constructions, zero new algorithms.” Token bill: Codex 21.0B vs Claude 2.8B.
“The top contributor was one candidate that they couldn't hire. It wasn't a person. It's an agent we built… 28% of its submissions made the leaderboard, roughly six times the community average. — The search is automated. The human would just move up the stack, not out of it.”
“…my version of going to Mars: the Eureka machine, a machine that will eventually invent pretty much all future inventions for humanity… Anything you can simulate, you can verify — and you can then solve with AI.”
“Each one of these agents requires you to steer them, direct them, review them — and at scale especially, it's just exhausting… So in reality, we are the bottleneck. But we don't have to be.”
The sharpest fault line of the three days: do you still read the code? HumanLayer said yes, twice — Horthy: “for now we're stuck reading the code, but we can still move pretty fast”; Mistele: “we can design loops and still read the code.” Boundary's Vaibhav Gupta ships a programming language with no code reviews — “slop is just any code you don't read” — and DeepMind's Benoit Schillings predicted that within a year “nobody will actually look at it.” Greptile brought a million PRs of data to the fight: agent PRs revert at human rates, and “humans were actually more likely to make P0 errors.” Anthropic held the middle: review intent, not lines.
Behind it, three more genuinely unsettled arguments. Factories versus orchestras versus “just prompt it” — Tereza Tížková's whole-loop factory against Charlie Holtz (“I honestly kind of hate the term… I want my software to feel human and crafted”) against Theo Browne (“you don't need some fancy software factory. You just need to prompt it to go a little further”). Token-maxing abundance against token discipline. And the golden age against the warning label — OpenAI's “the goal is squarely not to automate engineers” against Erik Meijer's “these models are intrinsically very, very dangerous, and we have to tame them,” with PostHog's Sarah Sanders supplying the security rule of the week: “if it isn't enforced deterministically, it is not enforced. Prompts are not security rules.”
OpenAI, on stage: “At every layer, we actually use the thing that we give to you.” — Notion, same stage, two days earlier: “Your supplier is your competitor.”
Both are true. That's the token market in 2026, and why optionality — the ability to walk — kept coming up as the buyer's only real lever.
Our Melbourne report derived six themes from a conference on the other side of the Pacific, three weeks before this one. San Francisco confirmed them — and in one case, named a day after one. The bridge is literal: AWS's Mike Chambers told the SF audience, “I was at the AI Engineer summit conference in Melbourne, and took a lot of it in.”
Melbourne 01 · The harness beats the model → SF named day three “Harness Engineering” and Grinich defined the discipline on stage. SF's complication: the labs are absorbing the harness, and Horthy says it isn't enough.
Melbourne 02 · Context isn't memory → Sakana's Stefania Druga, from experiments: memory is “not just the database store — it's this control loop around the model,” and naive memory “only added more cost.”
Melbourne 03 · Evals are the new tests → the loudest echo: “tasks and evals are the new world of CI” (Sweeney); Arize runs 100M evals a month; Yaron's survey has evals as the #1 problem, every year.
Melbourne 04 · Spec is the source → DSPy's whole thesis; WeCo: “you design two things — the eval and the abstraction”; radicalised by Theo: the markdown file no longer specifies the product, it is the product.
Melbourne 05 · The economics broke → Melbourne's anecdotes became SF's population data: cost is “a first-class engineering constraint”; three in four engineers adjust usage on price.
Melbourne 06 · The org is the bottleneck → SF's sharper version: the human is — “we are the bottleneck” (Salomon); “bottlenecked on human ability to fully conceptualize what we're doing” (Krieger).
| Engineers | Design loops, not prompts — and borrow control theory: incremental, sensed, damped (Mistele). Put verification inside the loop; Sonar's guide-verify-solve cut issues 92%. |
|---|---|
| Team leaders | Review intent, not lines — Anthropic builds artifacts that surface trade-offs because “2,000 lines… looks like code to me.” Check your velocity at the three-month mark, not the demo (Shaukat). |
| Org leaders | Keep the ability to walk: multi-model, specs and evals held constant while everything underneath swaps (Sachs, DSPy). And note Yaron's gap — nobody has settled the control layer for write-capable agents; yours isn't settled either. |
| Key takeaway | Stop treating prompts as security. Sanders: “if it isn't enforced deterministically, it is not enforced” — her own sub-agents tried to invent secrets and pull them from anywhere in the codebase. |
Read every line, review intent only, or no reviews at all — the World's Fair split three ways, with data on each side. Where is your team, honestly — and is that a decision or a drift?
Thirty-nine Main Stage talks across three days. Each link is the captured transcript our quotes were verified against.
Not captured, and said plainly: the Startup Battlefield finals, Garry Tan's closing keynote, day two's closing block (Addy Osmani, “Trends in AI,” Wei-Lin Chiang), Emil Eifrem's and Kay Malcolm's talks — and everything that happened off the Main Stage, which was most of the conference.
This report is from Web Directions, the team behind AI Engineer in Australia. The next conference: AI Engineer Sydney, 7–8 December 2026 — Hilton Hotel, Sydney.
The conference →