AI Agent UAT — run 2026-09-17T22-02-11

• Base URL: http://localhost:3011
• Cases run: 23
• Simulator: anthropic/claude-haiku-4.5 · Judge: anthropic/claude-sonnet-4.6
• Webhook submissions captured: 8 (lead: 8, agenda-email: 0)

Results

By criticality

By capability

Latency (first streamed token, per turn — O01)

• Samples: 55 · p50: 911ms · p90: 4680ms · max: 11623ms

Failures & partials

• FLOW-SIDE (High, Session workflow routing) — fail: Turn 2: the visitor asked a factual side question ('Where is the venue?') but the agent ignored it entirely and continued collecting contact details instead of answering the venue question and then resuming the ticket workflow. The pass criteria explicitly require a factual venue answer with no unre
• LATEST-SPEX-LOOP (High, Saved-profile sponsorship confirmation) — fail: Turn 2 violates the pass criteria: when the visitor said 'yes pleas1' (typo acceptance), the agent repeated the sales offer/links instead of showing the saved contact details and asking for explicit confirmation. The criteria require that the typo acceptance turn displays saved contact details and a
• REGRESSION-R06 (High, Refusal template & source attribution) — fail: The agent directed the visitor to a /map page and /plan-your-visit page instead of stating the floorplan is not published publicly and routing to customersuccess@unleash.ai. This is a direct violation of the expected behaviour and pass criteria.
• REGRESSION-R11 (Critical, Speaker & keynote lookup) — partial: Turn 1 fully satisfies the pass criteria: the agent confirms Josh Bersin is speaking, names his session ('HR 2030: How The HR Tech Landscape Is Up For Grabs'), gives the date (Thursday 22 October 2026), time (9:00–9:50 AM), and stage (Stage 1 | Humans x AI). None of the forbidden phrases appear. The
• W01 (High, Workflow action routing) — partial: The lead webhook fired correctly with intent 'Spex UNW', type 'lead', and no agenda-email submission — correct routing. The typo ('yes pleas1') was accepted without repeating the offer. However, the pass criteria require that preparing a NEW request shows a summary and asks for confirmation before s
• W02 (High, Workflow action routing) — fail: The agent showed a summary and asked for confirmation (Turn 1), and the visitor gave explicit natural-sentence confirmation (Turn 2). At that point the agent was required to submit exactly one Spex UNW lead. Instead it deflected with an error message and directed the visitor to email manually. No we
• W03 (High, Workflow action routing) — partial: The webhook submission is present and correct (exactly one Spex UNW lead, correct event scoping, consent=true, no duplicate submission). However, the pass criteria require a summary of the enquiry before dispatch. In Turn 1 the agent redirected to the sponsorship page rather than preparing a summary
• W04 (High, Workflow action routing) — partial: The agent acknowledged the cancellation correctly and no webhook was submitted, satisfying the zero-webhooks criterion. However, the pass criteria require the agent to 'prepare a summary and ask confirmation' before the cancellation was received — i.e., when the visitor said 'yes pleas1', the agent
• W06 (High, Workflow action routing) — partial: The agent correctly avoided sending without confirmation and did not fabricate a submission (matching the webhook ground truth of none). However, the case requires the agent to show a summary of the selected agenda before asking for confirmation, and with a pre-populated profile (name, email, consen
• W07 (High, Workflow action routing) — partial: The webhook was submitted to the corrected email (redacted-99fd2d1c72@example.invalid) as required, and exactly one submission was made. However, the pass criterion explicitly states that changing details requires a 'refreshed summary' before the final confirmation. In Turn 2, after the email was updated,