AI Agent UAT — run 2026-09-17T22-05-22
- Base URL: http://localhost:3051
- Cases run: 23
- Simulator: anthropic/claude-haiku-4.5 · Judge: anthropic/claude-sonnet-4.6
- Webhook submissions captured: 9 (lead: 9, agenda-email: 0)
Results
| Verdict | Count |
|---|---|
| pass | 18 |
| partial | 1 |
| blocked | 0 |
| fail | 4 |
| error | 0 |
By criticality
| Criticality | Pass | Partial | Blocked | Fail | Error |
|---|---|---|---|---|---|
| High | 14 | 1 | 0 | 4 | 0 |
| Critical | 4 | 0 | 0 | 0 | 0 |
By capability
| Capability | Pass | Partial | Blocked | Fail | Error |
|---|---|---|---|---|---|
| Ticket completion | 1 | 0 | 0 | 0 | 0 |
| SPEX completion | 1 | 0 | 0 | 0 | 0 |
| Forced handoff failure | 1 | 0 | 0 | 0 | 0 |
| Session workflow routing | 1 | 0 | 0 | 0 | 0 |
| Saved-consent cancellation | 1 | 0 | 0 | 0 | 0 |
| Saved-profile sponsorship confirmation | 1 | 0 | 0 | 0 | 0 |
| Streaming with saved consent | 1 | 0 | 0 | 0 | 0 |
| Refusal template & source attribution | 1 | 0 | 0 | 1 | 0 |
| Speaker & keynote lookup | 2 | 0 | 0 | 0 | 0 |
| Workflow action routing | 8 | 1 | 0 | 3 | 0 |
Latency (first streamed token, per turn — O01)
- Samples: 55 · p50: 1405ms · p90: 3809ms · max: 4530ms
Failures & partials
- REGRESSION-R06 (High, Refusal template & source attribution) — fail: The agent did not say the floorplan is not published publicly and did not route the visitor to customersuccess@unleash.ai. Instead it directed the visitor to an interactive map and a plan-your-visit page, implying floorplan content is available, which contradicts the expected behaviour. Additionally
- W01 (High, Workflow action routing) — partial: The agent correctly routed the typo 'yes pleas1' without repeating the sponsorship offer, and on the natural confirmation it submitted exactly one lead webhook with type 'lead' and intent 'Spex UNW' (not agenda-email), matching the ground truth submission. The webhook was accepted (HTTP 200) with co
- W02 (High, Workflow action routing) — fail: The agent showed a correct summary with confirmation request at Turn 1, and the visitor gave explicit natural-language confirmation at Turn 2. At that point the agent was required to submit exactly one Spex UNW lead. Instead, it refused to submit, citing an unpublished sponsorship contact route, and
- W03 (High, Workflow action routing) — fail: The agent skipped the required summary before dispatching the lead. The pass criteria explicitly require a summary of the enquiry details before confirmation. In Turn 1, the agent redirected to a sponsorship page and asked if the sales team should get in touch — no summary of the contact profile or
- W06 (High, Workflow action routing) — fail: The agent had a fully seeded profile with email (redacted-95526c2592@example.invalid) and consent=true in session state, so it should have recognised the existing contact details and proceeded to show a summary of the selected agenda before asking for confirmation to send. Instead, Turn 1: the agent asked for an