Moving AI agents from prototypes to production requires abandoning general-purpose frameworks in favor of custom, memory-aware harnesses that treat deterministic code, tool retrieval, and token-cost tracking as first-class engineering requirements.
Only the stories worth your time.
Get the next daily digest delivered to your inbox — curated from trusted sources and summarized in minutes. No spam.
Editor's Notes
These videos demonstrate that moving agents into production is less about model intelligence and more about building rigorous, cost-aware systems that treat reliability as a measurable engineering output. By shifting from open-ended chat interfaces to structured harnesses that prioritize deterministic verification and granular token economics, teams are finally moving past the prototype phase where performance is just a guess.
Key Takeaways
Reliability is a multi-layered engineering problem requiring distinct evaluation methods: code checks for deterministic tasks, LLM judges for semantic nuance, and human labels for ground truth.
Trust in financial agents is built by forcing the system to trace every output back to a source and requiring users to clarify ambiguous inputs before the model acts.
Model performance is not a monolith, as upgrading to a more capable model can solve one problem while introducing new, unexpected biases like a tendency to hold rather than act.
Massive cost reductions are possible by splitting workloads into distinct tracks for agentic tasks versus high-volume process automation, rather than running everything through a single, expensive model.
Most organizations fail to optimize their AI spend because they skip the foundational step of measuring token consumption by specific team and use case.
Agent harnesses—not frontier models—determine whether an AI agent actually works well for a given task, and the same model can perform significantly better or worse depending on how the harness manages memory, tools, and context. This talk demonstrates a custom harness for anti‑money laundering, where the agent loop vectorizes skills and tools for dynamic retrieval, keeps the context window small to preserve accuracy, and stores episodic/procedural/semantic memory in a database rather than files. The practical takeaway: building a stripped‑down, problem‑specific harness often beats trying to adapt a general‑purpose agent framework.
A layered eval strategy—code evals for deterministic checks, LLM judges for meaning, and human labels for ground truth—is necessary to reliably improve AI agents. In a financial analyst demo, prompt fixes boosted actionability from 5/16 to 48/48, but faithfulness stayed low; upgrading to Haiku 5.5 improved consistency at 4x cost but introduced a 'hold' bias. The lesson: define 'good' explicitly, measure it, and iterate.
Building a financial analysis agent you can trust requires a harness designed around five principles — shared domain, consistency, evidence, accountability, verification — not just a chat interface with an LLM. The speakers demonstrate this with a wedding budget planning tool that forces users to clarify ambiguous terms, delegate deterministic calculations to code, and trace every number back to its source. The key insight is that trust must be designed into the system from the ground up, especially when money is at stake.
One team cut its AI inference bill from $150k/month to under $10k/month by switching from Opus to a dedicated GRM 5.3 node for running coding agents, with no measurable drop in output quality. The savings came from measuring token spend by team and use case, then splitting workloads into agentic (long context, cache-heavy) and process automation (volume) tracks with different speed and cost profiles. The talk frames this as a four-step process — measure, right-size, buy capacity, check processes — and emphasizes that most organizations skip the first step entirely.
Xiaomi's approach to training MiMo V2.6 suggests that we are entering an era where training efficiency is driven by managing policy staleness across massive GPU clusters, a technique that could soon democratize high-end model training for non-frontier labs.