Every software team has a job nobody fights over: bug triage. A report comes in, and someone has to read it, figure out which part of the codebase is actually responsible, check whether a fix already exists, and write up what they found. It is detective work, it is repetitive, and it competes for time with actually fixing things.
At Levamo, bug reports flow into our tracker from three directions: our team, our customers through support, and an automated poller that turns real production errors from our monitoring into cards. That last one is great for coverage and rough on the humans doing triage, because error monitoring never sleeps.
So a few weeks ago we gave the triage job to an AI agent. Not as a demo. As a scheduled part of our engineering operations, running every day without anyone's laptop involved.
This is what we built, the guardrails that made us comfortable shipping it, and what actually happened when it started working through real bugs.
What the agent does every morning
Each day, a scheduled Claude agent wakes up in the cloud, clones our repositories, and pulls the list of open bug cards from our tracker. For each card that has not been triaged yet, it runs a fixed procedure:
- Check out the code safely. It creates read-only working copies of our frontend and backend repositories, so it is always looking at the current state of both codebases.
- Localize the root cause. A "localizer" agent reads the bug report and its comments, digs through the code, and produces a structured hypothesis: the suspected root cause, the specific files involved, and a confidence level.
- Try to tear that hypothesis apart. Two independent "verifier" agents then review the localizer's work adversarially. One checks correctness: does the evidence actually support this root cause? The other hunts for alternative causes: could something else entirely explain the same symptom?
- Decide through a deterministic gate. A plain script, not a model, makes the final call. It posts a triage comment only if every verifier confirmed the finding and the localizer was not low-confidence. Anything less and the run holds: it posts nothing and leaves the card for a human.
- Check fix status. For bugs already triaged, it does a cheap deterministic pass instead: has a fix landed on our dev branch, has it shipped to production, or is there no fix yet? If the answer changed since yesterday, the comment updates itself.
The result on a confirmed card is a single comment: here is the suspected root cause, here are the files, here is whether a fix exists yet. The comment carries a marker so re-runs edit it in place rather than piling up duplicates, and it self-corrects if the verdict changes later.
The guardrails matter more than the AI
The honest reason this works is not that the model is clever. It is that we wrapped the model in boring, deterministic constraints:
Detection only. The agent comments. That is all it is allowed to do. It never changes a card's status, never assigns anyone, never pushes code, and never opens a pull request. Repository push access is switched off for the whole routine. A human reads the triage comment and decides what to do with it.
Silence is a valid output. If the verifiers disagree, or confidence is low, the agent posts nothing. Early on this felt like failure. It is actually the whole point. A triage bot that comments on everything trains the team to ignore it. One that only speaks when two adversarial reviewers could not refute the finding earns a different kind of attention.
A cost cap. The full analysis only runs on new cards, at most around ten per run. Already-triaged cards get the cheap fix-status refresh. Anything deferred is logged, never silently dropped.
The LLM never touches the tracker directly. All writes go through small scripts that enforce idempotency and formatting. The model proposes; the pipeline disposes.
What actually happened
The part I find most interesting is what the adversarial gate caught before we ever let the agent post publicly.
During dry runs, the localizer confidently attributed one bug to the wrong cause. The explanation sounded plausible, and if a comment like that had landed on the card, a developer could easily have burned an hour in the wrong corner of the codebase. The verifier pass refuted it, the gate held, and nothing was posted. That single catch converted the team from "interesting experiment" to "turn it on."
Once posting went live, the first real triage comments landed on production bugs with root-cause localization that held up to human review, on cards where the agent had to search two codebases to find the responsible one.
And a lot of cards resolve to no comment at all. Some bugs are not localizable from the report alone. Some already have fixes in flight. Some hypotheses do not survive the verifiers. The agent stays quiet on all of those, which is exactly what we want from a teammate whose job is to save attention rather than consume it.
What we learned
Adversarial verification is the unlock. A single model asked "what causes this bug?" will always produce an answer, and it will sound confident whether it is right or wrong. Making a second and third model attack the answer, from different angles, filters out most of the plausible-but-wrong findings before anyone sees them.
Idempotency is not optional. A daily bot that duplicates its own comments becomes noise within a week. Edit in place, self-correct, and never repeat yourself.
Keep the deterministic parts deterministic. Fetching cards, checking git history for a fix, formatting a comment: none of that needs a model. The AI does the one thing only it can do, reading a vague bug report against a large codebase, and scripts do everything else.
Humans keep the steering wheel. Status changes, assignments, and fixes stay with people. The agent's output is an input to a human decision, not a replacement for one.
Why we are doing this
We run a managed WordPress hosting platform, and our customers feel every bug in our dashboard directly. Faster, more accurate triage means real fixes ship sooner. But there is a bigger reason: we think AI-assisted operations is where serious engineering teams are heading, and the way to learn is to put agents into production with real constraints, real guardrails, and real accountability.
Bug triage turned out to be a nearly perfect first assignment. The work is valuable, the failure mode is safe (a held run just means a human triages the card like before), and the output is easy to audit.
The agent does not get tired, does not resent the work, and does not comment unless it can defend its answer against two skeptics. Honestly, that is a higher bar than most of us hold ourselves to before hitting send.
