Context
In an automated fulfilment setting, a robotic system picks products from stock, and not every product is suitable for it: some are too fragile, oddly shaped, or otherwise better left to a person. Something has to decide, per product, whether the automated system should attempt it. The first version of that decision lived in a single large-language-model prompt that tried to judge everything at once.
The service is event-driven: it reacts to product-update events, makes the call, and emits the result for the rest of the platform to act on. The interesting design question was not the plumbing, it was how the decision itself should be structured.
The problem
One prompt doing the whole job had three problems:
- Opaque. When it judged a product wrong, it was hard to see which part of the reasoning failed, because every kind of product flowed through the same undifferentiated prompt.
- Wasteful. Plenty of products fall into categories whose answer is known in advance, yet every item still cost a model call.
- Hard to keep stable. A language model is non-deterministic, so a small change to the prompt or the underlying model could quietly shift conclusions across the board, with nothing in place to catch it.
Options considered
Keep refining the single prompt
The least work, and the reason it is tempting. But it leaves all three problems in place: still opaque, still one model call for every item, still no guard against drift.
One model, richer context
Feed the prompt more data and force structured output. This helps auditability a little, but it is still one undifferentiated call doing every category's reasoning, and still pays the model on items whose answer was never in doubt.
Classify first, then route
Work out the product's category up front, then hand it to a handler chosen for that category. Categories with a known answer are settled by a simple rule; only the genuinely uncertain categories reach a model. More moving parts, but each part is small and individually testable.
The decision, and why
I went with the routed design, and it is worth being precise about it, because the interesting part is what does not call the model. A classifier stage works out the product's category. A routing layer then hands the product to the right specialist handler. For categories known up front to be unsuitable for the robot, the handler is a plain policy: it returns the answer directly and never calls a model. Only the categories where suitability genuinely depends on the specific product go to a language-model-backed handler.
So it did become a multi-agent workflow, but a disciplined one. The judgment was not "agents good" or "agents bad"; it was matching the cost of a decision to the certainty of its answer. A rule is cheaper, faster, and perfectly reliable when the answer is already known, so the model is reserved for the cases that actually need judgment. Routing by category also made the system legible: each handler is small, owns one kind of product, and can be reasoned about and tested on its own.
Reserving the model this way also helps with a problem it can never fully escape: a large model is not truly deterministic, and even at temperature zero the same product can occasionally come back with a different verdict, a structural quirk of how these models run rather than something a sharper prompt fixes. Research on the non-determinism of "deterministic" LLM settings points to the levers that help, and the most useful one is that shorter, more constrained outputs are the most stable, which is exactly what routing and a compact verdict give you. A regression set of products with known-correct outcomes then runs on every change, so any drift surfaces before it reaches production rather than after.
This one is cleanly mine at the design level: the architecture, the solution-design write-up, and the research behind it are my work. It runs as a team-owned service now, as everything here eventually does.
Outcome
The single prompt became a routed multi-agent classifier where the model is used surgically: deterministic policy settles the categories whose answer is known, and a language model is called only for the products that genuinely need judging. The regression set guards the whole thing against silent drift, so the system can keep being tuned without quietly changing its mind in production.
Reflection
My first instinct, stated honestly, had been to keep the design as small as possible and treat a full multi-agent build as overkill. Following the problem changed my mind: the agents were not the indulgence, the indulgence was spending a model call on a decision a rule could make. The version I am proud of uses more structure, not less, but spends intelligence only where it is actually needed, and can prove it has not regressed. Knowing where judgment is required, and where a rule is plenty, turned out to be the whole game.