AI fixes — failure-only retest report

Index

• Scope and status
• Original supplied UAT
• Adjusted pass rates and exclusions
• Missing content percentages
• Targeted retest results
• Verified fixes and remaining findings
• Final fix outline and next actions
• Changes
• Local verification
• Latency, streaming and handoffs
• FAQ freshness
• Evidence and limits

Scope and status

Completed 37 unique previously failed scenario occurrences: 19 OpenAI GPT-5.4-mini and 18 Claude Sonnet 4.6. After reviewing the first results, 14 still-failing occurrences affected by additional fixes were rerun. A further nine unresolved occurrences were tested in round three, followed by three unresolved/error occurrences in round four. These are 63 executed scenario occurrences, not 63 distinct cases. Passing, partial and blocked baseline cases were excluded, as were cases reviewed as solely obsolete expectations. No new full-suite run occurred.

The user confirmed Gateway credit with the $40.21 screenshot. An initial suite-file wrapper error prevented acceptance execution; it was repaired. Accidentally copied old baseline files are archived as setup-error evidence and excluded from these totals. The completed Claude sponsorship failure scenario was retained and not repeated.

Both variants use the same profile extractor (OpenAI GPT-5.6-luna), narrow Jev selector, simulator and judge as the original benchmark. No production/staging website or AI configuration changed. At the user’s request, the existing shared Paris agenda and sitemap Pinecone namespaces were refreshed from their current published sources; see the source-refresh evidence below. All submission destinations are local capture receivers.

Original supplied UAT

This block covers only the original 95 cases, C01–C95 under each model. It excludes later additions, regression/workflow cases, scripted submission checks and FAQ freshness tests.

Latest retained combines targeted reruns with untouched original results. Seven original-UAT cases were rerun on OpenAI and five on Claude; some received subsequent failure-only corrections. We did not repeat the whole 95-case block.

Percentages use all 95 supplied cases as the denominator. Partial and blocked outcomes are separate from passes. Raw failures include obsolete expectations and judge evidence gaps, and are not a confirmed application-defect rate.

The 37 selected scenario occurrences below span original UAT and later blocks; do not add their totals to these 95-case totals. Count evidence · Original comparison.

Adjusted scores

These scores remove 33 model/block/case occurrences with confirmed obsolete expectations or unsupported evaluator judgments: 13 OpenAI and 20 Claude. They exclude those occurrences from the denominator; they do not relabel them as passes. The raw results and transcripts above remain intact. This is a conservative, itemised adjustment rather than a complete re-audit or a new test run.

Full passes only; partial and blocked outcomes remain in the denominator. Missing content, mixed failures and unresolved application defects are retained. The same exclusions apply to the before/after figures, so the comparison uses a consistent cohort.

The full benchmark originally contains 279 occurrences per model; the original supplied UAT is a separate 95-case block per model. Adjusted totals must not be added to the raw totals. Variant denominators differ because some evaluator errors are model-specific; these percentages are descriptive, not a controlled model-ranking claim.

What stays in the score

• T02 on both models: historical pricing is wrong in the test, but the response also misses the required concrete pass-preference question.
• OpenAI K17: its old complaints-email expectation is wrong, but the unrelated duplicate-message warning is a real issue.
• Claude C97: final consent is now required, but the transcript has independent agenda/dispatch problems.
• C14 and C01 partials: some summary/qualification rules are stale, but independent plan completeness or eligibility issues remain.
• F08: the old script omits required company/role, but repetitive recovery and malformed-email handling are still concerns.
• C116: missing volunteer content is an unresolved content gap, not a pass.
• C18/C21, C33, OpenAI C54/C68/C83 and other unsupported-fabrication claims: evidence is not sufficient to exclude the whole current outcome without resolving event-scoping, source-publication or independent wording/behaviour questions.
• All other partial/blocked/unreviewed results remain. The separate one-day/two-day agenda defect is still recorded, even though H01 passes its handoff criterion.

Exclusion register

Machine-readable scores and exclusions · OpenAI complete inclusion register · Claude complete inclusion register.

The adjustment does not change application code, expected-case files, deployment, live sources or historical verdicts. Fixed historic prices, roster notes and policy assertions should be rewritten from current source/policy contracts before their next execution; no fresh rerun occurred for this scoring change.

Missing content percentages

Confirmed minimum, based on reviewed source evidence: one missing content area—current Paris volunteering eligibility/application information and the five-team role breakdown—affects two retained case occurrences, C116 on OpenAI and Claude. The published-page snapshot and uploaded FAQs do not provide the required answer. This is a content/publication gap; the current replies safely provide Support rather than inventing roles or linking a removed page. Raw failures stay in the scores.

The original supplied UAT is C01–C95; C116 belongs to the later additions. “0 confirmed” does not mean no missing content exists in the original UAT: the remaining outcomes have not all received a complete source audit. Overall raw denominators give 1/279 (0.4%) per model and 2/558 (0.4%) combined. The retest row covers only the 37 deliberately selected failures, of which six are excluded as outdated/evaluator-only; its 100% figure must not be applied to the full benchmark's remaining failures.

Not counted as confirmed content-only gaps: C18/C21 event-scoping/source concerns, C33 volunteer publication concerns, C24/C26 requested articles/webinars/reports being substituted, and C85 unsupported attendee-industry claims. These may involve source freshness, retrieval or response behavior, and require case-specific source review before assigning a content-only cause. A missing answer in a transcript is insufficient evidence that the content itself is missing. Older volunteer outcomes are not automatically reclassified from the later source snapshot.

Action: editorial owners should confirm whether a current volunteer program should be published; if so, publish approved eligibility, application link and team/role details, sync those pages/FAQs into the existing Pinecone namespaces, and rerun C116 on both models. If the program is intentionally unavailable, update the expected behavior to the approved Support fallback. Do not fabricate or restore unpublished content merely to satisfy a historical test.

Evidence: OpenAI C116, Claude C116, published source snapshot, source-refresh results, adjusted denominators. These percentages describe test occurrences, not the percentage of site pages or knowledge-base documents missing. No new tests or paid calls were run for this breakdown.

Results

All selected baseline verdicts were fail. Raw judge labels below are preserved; they are not a reviewed application-defect count.

OpenAI case detail · Claude case detail · Raw transitions CSV · Detailed transcript review

Verified fixes and remaining findings

The latest retained raw totals are 21 pass / 10 partial / 6 fail, versus 37 selected baseline failures. This is a targeted before/after result, not a full-system pass rate or causal model comparison.

Verified fixes include natural ticket confirmation, saved agenda email with one correct handoff, ticket navigation, failed-sponsorship acknowledgement, headcount handling, retries, stored email masking, complete free-ticket FAQ answers, stand-plan/manual contact routing and conditional onsite registration. The Claude registration detour now collects the selected pass and saved contact details, shows a single consent summary and captures one delegate enquiry after confirmation.

The six remaining raw failures are reviewed separately: two keynote cases use obsolete August ground truth; two volunteer cases require a page currently absent from the published sources; OpenAI C98 uses a stale spelling expectation; Claude C78’s evaluator lacked actual pricing/FAQ evidence. Raw verdicts are preserved. Partial outcomes and broader UAT failures still require separate review; no complete-system certification is claimed.

Source refresh and final focused fixes

The user requested refreshing the existing source-to-Pinecone pipeline instead of moving runtime lookups to event-management/Sanity APIs. Runtime agenda lookups remain in Pinecone. The new exact read-only eventFacts lookup also reads FAQ/uploads and complete event-page bodies from existing Pinecone namespaces.

The refresh wrote 237 Paris agenda vectors and 145 page vectors covering 33 currently published Paris pages, removing 34 vectors belonging to pages no longer present. Before/current-source/after artifacts are retained in refresh evidence. Other event and knowledge namespaces, website deployment and AI switches were unchanged.

The refreshed live event-management source itself lists seven keynote-format sessions, including Brian Glaser and Stephen Childs, while Eric Mosley is absent from its keynote set. The older local agenda JSON and August six-speaker judge note are stale relative to that source. The earlier suspicion that Pinecone alone had the wrong formats was not supported by the live source check. Both C104 answers matched the current source; their raw failures remain because the original judge note was deliberately preserved. Current source snapshot.

No current volunteer page or volunteer FAQ answer is available. The two C116 replies correctly avoid inventing five team names or linking the removed page, and provide Support instead. Restoring that public content is an editorial decision, not something these tests should force into the answer.

OpenAI R01/R02 passed with complete HR programme, group-ticket and sponsor-package guidance. R07 initially produced an empty response observed by the runner, with the underlying cause not established; a focused retry passed with manual instructions and Customer Success. OpenAI C78 initially received a blocked judge label because the evaluator lacked policy evidence; the final retest included the captured current FAQ in evaluator notes, unchanged criteria, and a published Attend link, and passed. These evidence changes are recorded in round-four manifest.

Claude H01 initially reached the ticket workflow but repeatedly requested quantity because model extraction could erase saved booking fields. The fix preserves existing booking values through null extraction, recognises an explicit personal “a ticket” request as one, and clears old fields on event/new-booking changes. The final fixed six-message replay passed, with a contact summary, explicit consent and one HTTP 200 local capture. It does not prove real inbox or Salesforce delivery. The transcript’s initial agenda still spans two days despite the visitor’s one-day request; that separate plan-duration issue is recorded rather than hidden by the handoff verdict.

Final H01 transcript · Onsite transcript · Stand plans transcript · Implementation guide.

Final fix outline and next actions

Implementation and source changes

Deployment status: the application fixes are local and have not been pushed or deployed. The requested content refresh has already been applied to the existing shared Pinecone data. Source-fetching remains in the publishing/refresh pipeline; runtime fact and agenda lookups remain in Pinecone.

The agenda has an existing ten-minute cache per warm application instance. Upload clears the cache in the process performing the upload; that does not establish immediate invalidation of every other instance. The isolated test apps were restarted before the final retests. The 38.22-second refresh duration measures the bulk refresh operation, not FAQ-to-answer propagation or an SLA.

Focused case results

OpenAI C98 and Claude C78 were not repeated in these final rounds: the former has an obsolete spelling expectation; the latter's prior failure conflicts with the recorded badge/FAQ tool evidence. Together with C104/C116, these account for the six remaining raw failures in the 37 selected occurrences. This does not resolve every partial outcome or every untouched original-UAT failure.

Outstanding work and operational limits

Implementation guide · Round-three frozen code · Round-four frozen code and evaluator-evidence changes.

Changes

Additional corrections preserve full natural agenda-email acceptance through later contact collection and produce an application-owned contact summary before a send. Factual ticket navigation/venue/upgrade, free-ticket eligibility and exhibitor logistics questions are protected from new-lead qualification even when the selector chooses a commercial destination.

Local verification

• Latest owning-package checks: 231 web tests passed / 0 failed, and 36 shared AI tests passed / 0 failed. Form tests previously passed 94/94; the full ordinary monorepo suite was not repeated.
• After the second corrections, the focused flow test file passed 19/19, including natural email acceptance, negative/delayed controls and commercial-information versus actual buying requests.
• Biome passed on all 14 final touched source/test files. Web typecheck still fails elsewhere; no diagnostics matched the final changed files. Typecheck log.
• Original ordinary results remain in the baseline ordinary report, with a follow-up link.

Latency, streaming and handoffs

Metrics use the latest retained result for each selected case. Visitor first-token time includes profile extraction. They describe small, selected local concurrent batches and are not an overall model benchmark.

Multi-chunk answers confirm streaming continues in these cases; deterministic short replies use one chunk. HTTP 503 is rejected delivery, never successful submission. Captures remain in each transcript. Compiled metrics also separate all attempts from latest selected results.

The synthetic post-fix Langfuse submission trace masks the email in persisted tool arguments. Names, company and role remain visible; this is not full anonymisation. The existing broad phone masking also masks parts of session timestamps. Stored trace evidence.

FAQ freshness

Six new isolated operational tests are prepared for FAQ create/edit, existing-chat refresh, shorter-upload replacement, deletion and empty-FAQ cleanup while retaining uploads. They were not run under this failure-only instruction; no propagation time or SLA has been measured. Test method. The empty FAQ sync cleanup defect is corrected locally, with the live verification still pending.

Evidence and limits

• Raw original benchmark, all four retest rounds remain separate. Baseline and later runs use adaptive follow-ups, not identical conversation replays.
• The source snapshots are recorded in round-one manifest and round-two manifest, round-three manifest, and round-four manifest.
• Transcript counts and selected case identifiers were checked; no stale setup-error files contribute to totals.
• Real inbox delivery, Salesforce processing, shared Sanity publishing, live FAQ propagation, production kill-switch changes and all previously passing cases remain outside this retest.
• No statistically significant model improvement, full release certification or actual spend total is claimed. The credit screenshot was a point-in-time balance.
• No code push, deployment, real email or CRM write was made.