Agents and Performance

I didn’t set out to build agent infrastructure. I started with Ryva.

Ryva had a real performance problem. Some of the work was expensive, some of the context was unnecessarily large, and the algorithm behind parts of the agent was simply wrong. The system could produce useful output, but it took too long and did too much work to get there.

That bothered me for a reason beyond the product itself. I want AI to be accessible to more people. That means better models, but it also means lower costs, shorter feedback loops, and systems that do not require a large company to operate. If an agent is technically impressive but too slow or expensive to run, it is not very accessible.

So I started working lower in the stack.

What began as an attempt to make Ryva cheaper turned into a series of projects about local models, memory, tool discovery, scheduling, context, and the infrastructure around an agent. The projects are AGT, RecallBench, and an open-source runtime in my SDK repository.

The main idea I came away with is simple:

Agent performance is not just model quality. It is the quality of the loop around the model.

AGT made the problem obvious

The first project was AGT, a local, defensive autonomous security employee.

AGT inventories an explicitly authorized repository, creates bounded security-review work, runs separate scout, validator, and remediation phases, and records decisions in persistent SQLite memory with a SHA-256 chained audit ledger. It is not a public-target bug-bounty scanner. It only works on a repository the operator has authorized, and it keeps workers away from arbitrary host shell, network, and web access. Proposed patches require a separate operator-controlled apply step and an isolated Git worktree.

The architecture mattered because I wanted the tool to be useful without making the security boundary vague. Deterministic rules seed likely issues such as injection, command execution, traversal, weak cryptography, authentication bypasses, unsafe deserialization, SSRF, XSS, CORS, cookie, permission, and debug-configuration problems. Agents then investigate those slices, a different session validates a finding, and another worker can propose a remediation.

I used local models through Ollama while building it. The tested setup uses a small coordinator model, a worker model, and a 16K context window. The default audit is a bounded 16-worker swarm, although Ollama can serialize inference depending on how its server is configured.

That last part became impossible to ignore.

A swarm of agents does not automatically behave like a swarm. If every worker is waiting for the same local inference server, concurrency at the application layer does not mean concurrency at the model layer. Every worker also carries instructions, safety rules, tool schemas, repository context, and previous results. A model can be small while the overall request is still large.

AGT did actually work. When I ran it on large repositories, including repositories like Supabase, it surfaced real vulnerabilities rather than only producing generic security advice. That was exciting, but it also made the performance problem more interesting. The tool could find meaningful things, yet the path to a result felt slow and unusual.

I also became more careful about what “found a vulnerability” means. A pattern match is not a confirmed issue. A candidate needs source evidence, a credible abuse path, independent validation, and a remediation that does not create a second problem. That is why AGT separates those phases instead of asking one model to declare itself correct.

The question changed from:

Which model should I use?

To:

What does the model need to see, when does it need to see it, and what work should happen outside the model?

I thought RAG was the answer

The next thing I studied was context and memory.

My first instinct was the same one most people have: use a vector database, retrieve the most relevant chunks, and inject them into the prompt.

A basic version looks like this:

const matches = vectorDatabase.search(query);

const prompt = `
Use the following context:

${matches}

Answer the question.
`;

This works surprisingly often. It also fails in a very specific way when the system has state.

Similarity is not the same thing as truth. A retrieved fact can be relevant but outdated. It can describe a canceled decision, an old owner, a previous deadline, or a requirement that was later replaced. A large context has the opposite problem: it contains everything, including contradictions, so the model has to decide which state is current while it is also trying to solve the task.

I started thinking of this as a context selection problem rather than a storage problem.

The important questions are not only:

• Which chunks are closest to the query?
• Which memories have the highest embedding score?

They are also:

• Is this fact still current?
• Does it supersede another fact?
• Is it authorized for this agent?
• Is it a duplicate?
• Does the task need it now, or is it merely interesting?
• Can the answer preserve provenance?
• What should be ignored on purpose?

That led me to build RecallBench, a small research project around selective memory for long-term project support.

Selective memory is not just smaller RAG

RecallBench compares three conditions:

1. A large-context baseline that receives all project updates.
2. A TF-IDF RAG baseline that receives the five highest-scoring updates for each question.
3. A selective-memory condition generated from current, updated, outdated, and canceled facts, with explicit Use and Ignore labels.

The current benchmark contains 90 events and 30 questions about owners, delayed work, blockers, changed requirements, canceled tasks, and conflicts between old and new facts. The timeline is fictionalized as ShieldPath Learning Lab so that the benchmark contains no private project data or credentials.

The result was not that RAG became useless. The result was that relevance alone was not enough.

The publication-facing score came from a manual audit of the saved replay:

Selective memory also used 18.1% fewer estimated total tokens than traditional RAG in that replay. But it was not faster. The selective-memory run took 217.326 seconds end to end, compared with 108.809 seconds for RAG.

That tradeoff is important. I do not want to turn one benchmark into a universal claim that selective memory wins. In this experiment, it was more accurate and used a smaller token proxy, but the memory-generation path added latency. A real production system needs to decide whether the extra accuracy is worth that latency, or whether the memory should be compiled ahead of time, cached, or updated asynchronously.

The chart above is generated from the automated scoring layer, which reports 25/30 for selective memory, 23/30 for large context, and 21/30 for RAG. The text uses the manually audited publication-facing score. I kept that distinction visible because evaluation pipelines can create their own context problems if automated labels and human adjudication are quietly mixed together.

The more durable lesson was this:

Useful memory is not all available information. It is the smallest current state that lets the agent make the right decision, with enough provenance to recover the rest.

That is closer to a compiler or a query planner than to a passive vector store.

The broader performance problem

While working on memory, I started reading more about how agent workloads behave at the systems level.

A 2025 study, The Cost of Dynamic Reasoning, separates an agent into model inference and tool-use phases. In its experiments, model inference accounted for 69.4% of average latency and tool execution for 30.2%. Prefix caching reduced end-to-end latency by an average of 15.7%, while tool-augmented agents used substantially more memory per request than chain-of-thought baselines.

The exact percentages are workload-specific, but the shape of the problem matches what I was seeing. Agent performance is a systems problem with several interacting budgets:

agent performance = success quality
                  + latency
                  + token cost
                  + tool reliability
                  + safety
                  + context freshness

Optimizing only one of these can make the product worse. A faster agent that forgets the current owner is not better. A more accurate agent that takes two minutes for every interaction may not be usable. A cheaper agent that speculates a write or leaks a secret is not an optimization.

The evaluation literature is moving in the same direction. The 2025 survey on evaluating LLM agents argues that agent evaluation needs to cover behavior, capabilities, reliability, safety, interaction mode, planning, tool use, memory, and cost rather than relying on a single model score.

That is why I stopped asking whether an agent was “smart” and started asking whether it was efficient at completing a real task.

From memory to tool discovery

The selective-memory work led to a second realization: the same problem exists in tool use.

Most tool-using agents start with a giant catalog. They receive every tool name, description, parameter schema, and server detail, even when the task needs only two tools. Then they return a large result, put the entire result back into the next model call, and often execute independent reads one by one.

This is an inefficient default:

all tools -> all schemas -> serial calls -> full results -> full context

I wanted a runtime that could make the opposite path easy:

discover -> inspect -> validate -> schedule -> compact -> cache -> measure

That is what I built in the Navo SDK repository. The repository is called sdk, while the runtime is called Navo and the published packages use the @runto/* namespace.

The runtime is framework-neutral. It does not replace the model or force an agent framework on top of an existing application. It sits around the tool loop and lets each optimization be enabled independently.

Install the pieces from npm:

npm install @runto/core @runto/provider-openai @runto/cache @runto/telemetry

The main package links are @runto/core, @runto/provider-openai, @runto/provider-anthropic, @runto/provider-vercel, @runto/mcp, @runto/cache, @runto/context, @runto/scheduler, and @runto/telemetry. They are Apache-2.0 licensed and published at version 0.1.0 right now.

A minimal runtime looks like this:

import { createRuntime } from "@runto/core";
import { createVercelProvider } from "@runto/provider-vercel";

const runtime = createRuntime({
  provider: createVercelProvider({
    model: "openai/gpt-5",
    gateway: true,
  }),
  tools,
  optimizations: {
    toolDiscovery: { enabled: true, strategy: "bm25", topK: 3 },
    schemaCompiler: { enabled: true },
    contextCompiler: { enabled: true, tokenBudget: 500 },
    resultCompiler: { enabled: true, maxInlineBytes: 400 },
    cache: { enabled: true },
    parallelExecution: { enabled: true, maxConcurrency: 4 },
    speculation: { enabled: false, mode: "read-only" },
  },
});

const result = await runtime.run({
  task: "Investigate issue ENG-142 and identify the likely root cause.",
});

console.log(result.output, result.metrics);

The point of this API is that an existing agent can adopt one layer at a time.

1. Discover only relevant tools

The runtime can begin with search_tools, then let the model inspect a specific tool before executing it. The default registry supports BM25-style ranking, and the core also has hooks for keyword, embedding, hybrid, and learned ranking strategies.

The agent does not need to carry the full catalog through every step. It sees a small set of candidates, inspects the exact schema it needs, and executes a validated tool.

2. Compile schemas and context

The schema compiler removes unnecessary shape from tool definitions. The context compiler can exclude unauthorized, stale, duplicate, or low-relevance items, apply a token budget, redact content, and surface conflicts through explicit provenance.

This is the same idea as selective memory, applied to every run instead of only to a long-term memory database.

3. Compile results instead of dumping them

Large tool results are often the hidden source of cost. Navo can project fields, filter rows, sort values, aggregate data, and replace an oversized inline result with a small preview plus a recoverable handle. The full value stays available to the runtime instead of being copied into the next prompt by default.

4. Schedule safe work concurrently

The scheduler infers dependencies between tool calls and runs independent none or read operations together within a concurrency limit. Writes and irreversible operations remain serialized. Timeouts, bounded retries, cancellation, and dependency-cycle checks are part of the scheduler rather than being left to every application.

5. Cache with freshness rules

The cache layer includes memory, filesystem, SQLite, and Redis-compatible adapters. Cache policy can be ttl, versioned, validate-before-use, or never. That distinction matters because “cached” is not automatically “current.”

6. Keep provider choice separate

There are direct OpenAI Responses and Anthropic Messages adapters, a Vercel AI SDK and Gateway provider, and adapters for MCP and telemetry. The runtime loop stays the same while the provider changes.

Telemetry is also part of the design. Events can go to memory, JSONL, Convex, or OpenTelemetry-compatible exporters. Sensitive telemetry is redacted by default, and content capture is an explicit choice rather than an accidental default.

Read-only speculation is available as an opt-in path for historical traces, but writes are never speculated. I would rather leave performance on the table than make a system that silently turns a prediction into an external side effect.

What the Navo benchmarks showed

I built a production-incident benchmark around a task that has twelve independent read-only sources: issues, comments, commits, ownership, logs, deployments, metrics, flags, runbooks, design documents, recent changes, and incident history.

The control was the normal serial model-driven loop. The treatment used validated fan-out for independent reads, result compilation, lower planning output, and lower reasoning effort. The latest live run used Vercel AI Gateway with openai/gpt-5, 20 tasks, 12 tools, a 120 ms tool fixture, and zero failed attempts.

The cost number is an estimate from configured token rates, not an invoice. The run is a local measurement, not a claim about every provider or production workload.

Smaller runs showed a larger spread. One three-task planning run reduced p95 latency by 56.9% and estimated cost by 78.7%. A repeat reduced latency by 61.5% and cost by 82.3%. A realistic, unoptimized comparison did not improve latency, which is useful evidence too. The treatment must actually enable the right optimizations; there is no magic in adding a runtime package.

The tool phase showed the clearest result. With twelve independent reads, measured tool-phase p95 moved from about 1,478 ms in the serial control to 133 ms with bounded parallel scheduling, an 11.1x speedup for that phase. End-to-end latency still includes model time, so the full request improved by less.

This distinction is why I care about instrumentation. If I only measured total response time, I could miss the fact that the tool scheduler was working while model inference remained the main bottleneck. If I only measured token count, I could miss a slow memory compiler. If I only measured success rate, I could miss that the agent is too expensive to run.

The 10x goal is still the right direction for me, but the honest claim today is smaller: this runtime produced roughly 4.1x lower estimated cost per successful task in the larger live run, with unchanged success rate, and much larger improvements in the tool phase. That is promising, not finished.

Why Caveman and Ponytail changed how I thought about this

Two projects made the direction feel more obvious: Caveman and Ponytail.

Caveman asks why an agent should spend tokens saying many words when fewer words will do. The newer versions also compress what an agent reads, including tool payloads and browsing results, while keeping a recovery path. Its README reports 33.2% fewer provider-reported input tokens in a pinned Claude Code benchmark, while also warning that output-token savings and whole-session savings are different measurements.

Ponytail attacks a different waste pattern: agents over-build. Its rule is not “write bad code quickly.” It is “understand the problem, then stop at the smallest solution that is correct, safe, and accessible.” Its agentic benchmark reports 54% less code, 22% fewer tokens, 20% lower cost, and 27% lower time across twelve feature tasks, while documenting that the result varies by task and model.

I like both projects because they challenge the assumption that the only way to make agents better is to give them a larger model. Sometimes the better system is the one that reads less, speaks less, calls fewer tools, reuses more work, and refuses to build what the task does not need.

Where I think agent performance is heading

Over the next year or two, I expect agent performance to move in five directions.

1. Context will become a managed resource

The winning systems will not treat context as a string that gets appended until the window is full. They will track freshness, provenance, sensitivity, permissions, stability, priority, and recovery. Context will have something like a type system.

The best memory will not be the one that remembers the most. It will be the one that knows what changed, what was superseded, what is safe to show, and what the agent can ask for later.

The MemoryAgentBench paper frames memory around accurate retrieval, test-time learning, long-range understanding, and selective forgetting. That is a better description of the problem than “add embeddings.” The 2026 LongMemEval-V2 work pushes the same idea into specialized web environments, testing static state recall, dynamic state tracking, workflow knowledge, environmental gotchas, and premise awareness across very long histories.

I expect selective retrieval, structured state, event logs, runbooks, and asynchronous memory compilation to converge. Plain vector search will remain useful, but it will become one component inside a larger memory system.

2. Tool discovery will matter as much as tool execution

As agents gain access to thousands of tools, exposing every schema on every turn will become obviously wasteful. Tool registries will rank capabilities, compress schemas, track freshness, learn from successful workflows, and attach permission and side-effect metadata.

The agent will not start with a phone book. It will start with a search box, inspect a few candidates, and receive only the contract it needs.

This is also where MCP can become more useful. The protocol gives tools a common shape, but a common shape is not the same thing as an efficient catalog. The next layer is likely to be discovery, policy, versioning, caching, and observability around the protocol.

3. Latency will be optimized across the whole loop

Model inference will get faster, but that alone will not solve agent latency. Systems will overlap safe tool calls with planning, reuse KV and prompt prefixes, compile large results, cache stable reads, route easy steps to smaller models, and reserve expensive reasoning for the parts that need it.

I also expect more bounded speculative execution for read-only work. The safety rule is simple: speculation can prepare a reversible read, but it should not silently perform a write. The runtime should be able to prove when a speculative result matched the authoritative call, and discard it when it did not.

4. Agent evaluation will become product-specific

A single benchmark score will not be enough. Teams will measure successful task completion, factual or procedural accuracy, cost per successful task, p50 and p95 latency, tool-call count, cache hit rate, stale-context rate, recovery rate, permission violations, and human escalation.

The benchmark should look like the real workflow. A coding agent needs code diffs and regression tests. A security agent needs validated findings and false-positive rates. A project agent needs current decisions, ownership, and blockers. An agent that wins a generic QA benchmark can still be a bad product.

The future of agent evaluation is probably closer to a control chart or a flight recorder than to a leaderboard.

5. The runtime will become a product surface

Today, developers still micromanage model keys, provider clients, tool definitions, retries, caches, context windows, and telemetry. That is normal for an early ecosystem, but it is not the final interface.

I can imagine a hosted platform where an agent team uploads tool contracts and policies, then gets model routing, execution scheduling, context compilation, caching, evals, and usage controls without operating every provider integration itself. The platform would not need to hide the model. It would make the model layer replaceable.

That is the direction I may explore next: hosting usage of this SDK so developers can focus on the agent workflow instead of managing every API key and performance detail. The hard part will be trust. A hosted runtime needs clear data boundaries, tenant isolation, policy enforcement, provenance, and an honest answer to where prompts and tool results go.

The goal is not a faster demo

The projects are connected now.

Ryva gave me a real product where the cost and latency were painful. AGT showed me that a local agent can do serious defensive work, but also that model and tool orchestration can dominate the experience. RecallBench showed me that injecting more context is not the same as giving an agent better memory. Navo turned those observations into a runtime that can discover tools, compile schemas and results, schedule safe work, cache with freshness rules, and measure what happened.

I still want the ambitious version: agents that are 10x cheaper, 10x more efficient, and more accurate at the same time.

I do not think that will come from one clever prompt or one giant model. It will come from many small decisions made by the runtime:

• retrieve only the context that can change the answer
• keep stale and canceled facts out of the active state
• discover tools instead of exposing all of them
• validate schemas before execution
• run independent reads together
• serialize writes and irreversible actions
• compile large results into useful summaries with recovery handles
• cache only when freshness and permissions allow it
• route simple work to smaller models
• measure every change against success, cost, latency, and safety

The model is still important. It is just not the whole product.

Agent performance is heading toward systems that feel less like a model with a tool belt and more like an operating system for delegated work. The best agents will not merely reason harder. They will carry less unnecessary context, choose better tools, preserve state more carefully, and spend compute where it changes the outcome.

That is the infrastructure I want to build.

And it is now fully open source in the SDK repository.