{"a-watched-agent-never-finishes":{"slug":"a-watched-agent-never-finishes","kicker":"Notes from the CSO","title":"A Watched Agent Never Finishes","standfirst":"Tokens per second measures the model. Time to done measures the agent.","author":"Lawrence Spracklen","detail":"CHIEF SCIENCE OFFICER · NOVEMBER 2026","avatar":"/blog/assets/lawrence.webp","extra":{"tldr":[{"b":"Time to done.","x":"For an agent, the number to watch is how long it takes to finish the job."},{"b":"Chat and agents.","x":"A rate that easily outpaces a reader becomes the wait when dozens of calls run one after another."},{"b":"Parallel has limits.","x":"Throughput is capped by the engineer's attention, and splitting work is capped by the dependency chain."},{"b":"Efficiency first.","x":"Fewer calls and leaner context save time and money, provided the success rate holds. Pay for a faster tier on what's left."}],"tldrLabel":"In short","tldrEnd":true,"close":{"a":"You can pay to make the tokens come faster.","b":"Or you can need fewer of them.","cta":["Book a Demo","See your AI footprint"]},"heroAlt":"Pen drawing of an engineer on a stool, chin in hand, watching a kettle on a stove with a blue flame, an hourglass half run beside him"},"blocks":[{"t":"lede","x":"Token rates are set by your model, provider and service tier. Short of paying for a faster tier, there isn't much you can do from your desk to make the next token arrive faster."},{"t":"p","x":"Those rates are more than enough for a chatbot, but not for a coding agent. The same hundred or so tokens a second that feels instant in a conversation stretches into minutes, then hours, when an agent has a ticket to close and an engineer is waiting on the result."},{"t":"p","x":"In <a href=\"/blog/amnesia-is-expensive\">Amnesia Is Expensive</a> and <a href=\"/blog/the-price-of-everything\">The Price of Everything and the Value of Nothing</a>, I looked at what wasted agent work costs in dollars. This piece is about the other bill: time."},{"t":"p","x":"I was a performance engineer, and the first lesson of that work is to measure what the user actually waits for. For an agent, that's the time to a successful outcome: how long it takes to finish the job."},{"t":"img","src":"/blog/assets/watched.webp","srcDark":"/blog/assets/watched-d.webp","alt":"An engineer on a stool, chin in hand, watches a kettle on a small stove. An hourglass stands half run on the bench beside him.","cap":"","mw":760,"iw":1100,"ih":642},{"t":"h","num":"01 · Fast enough to read","x":"Fast enough to read"},{"t":"p","x":"In a chat, the model only has to outpace the reader. People read at around 200 to 300 words a minute, roughly five or six tokens a second. A model streaming at a hundred tokens a second is producing text fifteen to twenty times faster than that."},{"t":"fig","type":"bars","capL":"Reader vs model, tokens per second","capR":"Chat","rows":[{"l":"A person reading","left":0,"w":5.5,"v":"~5–6 tok/s","tone":"n"},{"l":"A model streaming","left":0,"w":100,"v":"~100 tok/s","tone":"b"}],"cap":"At 200 to 300 words a minute, the reader is the bottleneck. The model is fifteen to twenty times ahead."},{"t":"p","x":"That is why streaming works. The first words land within a second or two, and from there the text runs ahead of your eyes."},{"t":"p","x":"Chat also has a natural rhythm: a question, an answer, and a pause while the human thinks. The model spends most of the conversation idle, waiting on you."},{"t":"h","num":"02 · Too slow to wait on","x":"Too slow to wait on"},{"t":"p","x":"A coding agent flips that relationship. Nobody reads its output as it streams. The tokens feed the next step of a loop: plan, read a file, run a command, read the result, edit, run the tests, read the failures, edit again."},{"t":"fig","type":"ladder","sets":[{"l":"Each step waits on the last","act":false,"rungs":[{"n":"01","name":"Plan","q":""},{"n":"02","name":"Read a file","q":""},{"n":"03","name":"Run a command","q":""},{"n":"04","name":"Read the result","q":""},{"n":"05","name":"Edit","q":""}]},{"l":"Retry","act":true,"rungs":[{"n":"06","name":"Run the tests","q":""},{"n":"07","name":"Read the failures","q":""},{"n":"08","name":"Edit again","q":""}]}],"cap":"One pass through the loop. The retry cycle (highlighted) is where unnecessary steps pile up."},{"t":"p","x":"A single ticket can take dozens of model calls, and a long session runs to hundreds or thousands. They run one after another, each waiting on the last. At a hundred tokens a second, every call costs seconds, and the seconds add up."},{"t":"p","x":"Now the human is the one sitting idle. An engineer kicks off a task, checks back, finds it still going, and checks back again. A watched pot never boils, and a watched agent never seems to finish. The rate that was plenty for chat is now what everyone is waiting on."},{"t":"p","x":"Every unnecessary step adds to that wait, and unlike the token rate, the number of steps an agent takes is something you can change."},{"t":"h","num":"03 · The limits of parallel","x":"The limits of parallel"},{"t":"p","x":"The natural response is to stop waiting on one agent and run several. Many engineers do. It helps in two ways, and each runs into a limit that has nothing to do with the model."},{"t":"p","x":"The first is throughput: more independent tasks in flight at once. There, the limit is the engineer. Every session needs someone to set it up, steer it, answer its questions and review what it produced. Each switch means reloading context: what this session was for, how far it got, what it is asking now. Past a handful of sessions, the switching starts to cost more than the waiting saved. Agents run in parallel. Attention doesn't."},{"t":"p","x":"The second is finishing one piece of work sooner by splitting it across agents. This time the limit is the dependency chain. The schema change lands before the API that uses it. The failing test gets fixed before the feature built on it can be trusted. At each of those points, you wait for the one agent on the critical path."},{"t":"p","x":"Anyone who has done performance work knows this as Amdahl's law: the speedup from parallelism is capped by the part of the work that must run in sequence. If a fifth of a task is serial, no number of agents finishes it more than 5x faster, and that serial fifth runs at the speed of a single agent. That's before you count the cost of coordinating agents, reconciling their changes and reviewing the result."},{"t":"fig","type":"bars","capL":"Amdahl's law, one fifth serial","capR":"5× max","rows":[{"l":"1 agent","left":0,"w":20,"v":"1.0×","tone":"b"},{"l":"2 agents","left":0,"w":33.3,"v":"1.7×","tone":"b"},{"l":"4 agents","left":0,"w":50,"v":"2.5×","tone":"b"},{"l":"8 agents","left":0,"w":66.7,"v":"3.3×","tone":"b"},{"l":"16 agents","left":0,"w":80,"v":"4.0×","tone":"b"}],"cap":"Speedup = 1 ÷ (0.2 + 0.8 ÷ agents). A full bar is the 5x ceiling. Coordination, reconciling and review costs are not included."},{"t":"h","num":"04 · Paying for speed","x":"Paying for speed"},{"t":"p","x":"If the model is the bottleneck, the obvious answer is to buy a faster one. Providers now sell exactly that: faster inference tiers, at a per-token premium that is often a multiple of the standard price."},{"t":"p","x":"For a blocking task on the critical path, with an engineer watching, the premium can be worth paying."},{"t":"p","x":"It's expensive leverage, though. Every request routed through that tier pays the premium on every token, including the ones the agent never needed to generate: the directory it walked yesterday, the conventions it rediscovered, the approach it tried and abandoned. You pay more to get the same waste sooner. And a faster tier does nothing about the round trips the agent didn't need to make."},{"t":"h","num":"05 · The speedup that pays you","x":"The speedup that pays you"},{"t":"p","x":"There's a cheaper lever, and it's the one this series has been about all along."},{"t":"p","x":"If the rate is largely fixed, the other lever is how much work the agent does. Each unnecessary model call adds its own wait for the first token and for the output, and each unnecessary tool call adds another round trip. Rediscovered conventions, abandoned approaches and repeated test runs all lengthen the path to done. You can't remove every source of delay, but you can stop the agent paying it twice."},{"t":"p","x":"Context adds a quieter cost. As a session's context grows, each call takes longer to produce its first token, even when most of that context is cached, and output can slow down too. An agent that carries everything it has ever read pays that delay on every call that follows. Lean context saves time as well as money."},{"t":"p","x":"Optimizing the agent's trajectory deals with all of this. Give it what you would tell a new team member on day one (the layout, the commands, the conventions, what has already been tried) and it stops rediscovering them. It makes fewer calls and carries less context, and still passes the same tests."},{"t":"p","x":"The earlier pieces counted those savings in dollars. This one counts them in time."},{"t":"p","x":"Speed is only useful if the work is right. The clock stops when the change is done and accepted, which can be a while after the agent stops. A faster model that explores more dead ends can take longer to close a ticket than a slower one that goes straight there, and a short trajectory that ends in a broken patch isn't an optimization. Efficiency only counts if the success rate holds."},{"t":"fig","type":"cols","cols":2,"cells":[{"tag":"Paying for speed","tone":"o","title":"A fast tier","body":"","mono":["Cost ↑","Time ↓"]},{"tag":"The speedup that pays you","tone":"b","title":"An efficient agent","body":"","mono":["Cost ↓","Time ↓"]}],"cap":"Both get the ticket closed sooner. Only one of them lowers the bill, and it makes the other one cheaper."},{"t":"p","x":"Efficiency is the mirror image of a premium tier: a fast tier costs more to finish sooner, while an efficient agent costs less and finishes sooner. It helps parallelism too: sessions that need less steering free up the engineer sooner, and work cut from the critical path shortens the stretch that caps everything else. Make the agents efficient first. Then, if a task still needs to go faster, pay for speed on what's left."},{"t":"p","x":"In the next piece, More Haste, Less Speed, we'll put numbers on this: where an agent's time actually goes, what a skill built from our own telemetry did to wall-clock time, and why a smaller model isn't automatically a faster one."}]}}