Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

2026-07-11

Authors: Kalle Kujanpää, Ning Liu, Shahnawaz Alam, Yeshwanth Reddy Sura

ArXiv: 2607.08010v1

PDF: Download PDF

Imagine you hired an assistant who, every single time you asked them to file an expense report, sat down and re-invented the entire expense-filing process from scratch — reading the manual, testing which buttons work, and writing themselves fresh instructions. That's roughly how many production LLM agents behave today: every request triggers a fresh burst of "think, plan, generate code, try it, fix it." It works, but it's slow, expensive, and unreliable.

This paper proposes a smarter arrangement: let the agent make its own tools ahead of time, then just use them at runtime.

The approach works in two phases:

The clever bit is that the tool-maker grounds itself in the actual production environment rather than guessing from documentation. It sees the real schemas, real values, real edge cases. That's the difference between a tool built from a textbook and one built by someone who's actually done the job.

There's also a self-evolution angle: as new failure modes appear, the system can generate new tools or repair existing ones, rather than being frozen forever at deployment time. This addresses a common gripe with agent systems — they either get locked in at training time (brittle) or improvise at runtime (slow and inconsistent). Tool-making sits between those extremes.

For anyone building customer-facing LLM agents, the tradeoff is compelling: you spend compute upfront to bake in reliability, and you get back predictable low-latency responses. It also means your logs become far more auditable — every action maps to a named, versioned tool rather than a fresh blob of generated code.

Why it matters: Shifting repetitive agent work from runtime code generation to pre-compiled, environment-grounded tools is a pragmatic path to making LLM agents fast and reliable enough for production.

All newsletters