Skip to content
Gauntlet
SOLARI LIVE

PR #82 @ 9f2d13a

DEMO DATA

Reference Agent · Complete Demo Checkout · run dffba21 · 4d ago

acme/checkout-agent · pr-82-refactor-waits @ 9f2d13a

All runs
Reliability
75%12 / 16 runs passed

95% confidence interval 50.5% – 89.8% over 16 scored runs

Baseline
100%
34%–100%
Perturbed
71%
45%–88%
Runs
16
16 scored
Failed
4
no infra errors
p95 duration
30.7s
p50 14.3s
Avg steps
6.13
avg 17.3s
Run-to-run flips
0%
how often identical repetitions disagreed

Reliability by perturbation

Worst first. The whisker is the 95% Wilson interval — small samples are wide on purpose.

Expired session
State
0%0 / 2
Unexpected modal
UI
0%0 / 2
Baseline
Baseline
100%2 / 2
Cookie popup
UI
100%2 / 2
Mobile viewport
Viewport
100%2 / 2
Network delay
Network
100%2 / 2
Renamed CTA
UI
100%2 / 2
Slow API
Network
100%2 / 2

Reliability by category

State
0%0 / 2
UI
67%4 / 6
Baseline
100%2 / 2
Viewport
100%2 / 2
Network
100%4 / 4

Failure distribution

Only scored failures. Infrastructure errors are excluded.

Auth
250%
Unexpected UI
250%

Run matrix

Every cell is one run. Select one to open its evidence.

Outcome of every run, by perturbation and repetition. Select a cell to open that run.
PerturbationRun 1Run 2Rate
BaselineBaseline, run 1: PassedBaseline, run 2: Passed100%
Cookie popupCookie popup, run 1: PassedCookie popup, run 2: Passed100%
Expired sessionExpired session, run 1: FailedExpired session, run 2: Failed0%
Mobile viewportMobile viewport, run 1: PassedMobile viewport, run 2: Passed100%
Network delayNetwork delay, run 1: PassedNetwork delay, run 2: Passed100%
Renamed CTARenamed CTA, run 1: PassedRenamed CTA, run 2: Passed100%
Slow APISlow API, run 1: PassedSlow API, run 2: Passed100%
Unexpected modalUnexpected modal, run 1: FailedUnexpected modal, run 2: Failed0%

Failure clusters

Failures grouped by category and cause, so a repeated defect reads as one problem.

  • Auth

    The shopping session expired mid-task and the agent did not re-establish it.

    2 runs · dc7abe3, 6912bef

  • Unexpected UI

    An overlay covered the page and 3 clicks never reached the intended element.

    2 runs · b2cb4b6, 3a9a157

Individual failures

Open one to see the assertion that failed and the trace that led there.