Uncovering Masked Failure Modes Before General Rollout Authorization
The engineering team had shipped an autonomous support agent behind a feature flag but could not answer a simple board question: is it safe to expand traffic past 5%? Their only signal was an aggregate helpfulness score from a single LLM judge that masked severe multi-turn refund hallucinations.
The team relied on an aggregate 4.6/5 score from an off-the-shelf LLM judge. In production shadow traffic, the agent appeared helpful, but senior customer support leads suspected hallucinations in complex policy refund queries. Without deterministic traces or agreement metrics, leadership refused to expand rollout.
We rebuilt the evaluation harness around the actual regulatory job. We defined task-grounded rubrics for financial policy compliance, tool-call payload validation, and mandatory refusal behavior on regulated credit queries. We scored a held-out set of 850 complex workflows with calibrated domain specialists, enforcing inter-rater agreement before any score was certified.
The audit revealed that the aggregate score had been masking a severe failure mode in multi-turn refund disputes: roughly one in six conversations produced an unsupported financial commitment. With reproducible regression baselines in place, the team fixed the tool-retrieval path, verified the fix across two model iterations, and secured board authorization to expand rollout to 100% of customers.
