main @ a1b4c7e
DEMO DATAReference Agent · Complete Demo Checkout · run 41810d7 · 4d ago
acme/checkout-agent · main @ a1b4c7e
Reliability
88%14 / 16 runs passed
95% confidence interval 64.0% – 96.5% over 16 scored runs
- Baseline
- 100%
- Perturbed
- 86%
34%–100%
60%–96%
Runs
16
16 scored
Failed
2
no infra errors
p95 duration
28.6s
p50 13.5s
Avg steps
6.88
avg 14.7s
Run-to-run flips
0%
how often identical repetitions disagreed
Reliability by perturbation
Worst first. The whisker is the 95% Wilson interval — small samples are wide on purpose.
Expired session
State
0%0 / 2
Baseline
Baseline
100%2 / 2
Cookie popup
UI
100%2 / 2
Mobile viewport
Viewport
100%2 / 2
Network delay
Network
100%2 / 2
Renamed CTA
UI
100%2 / 2
Slow API
Network
100%2 / 2
Unexpected modal
UI
100%2 / 2
Reliability by category
State
0%0 / 2
Baseline
100%2 / 2
UI
100%6 / 6
Viewport
100%2 / 2
Network
100%4 / 4
Failure distribution
Only scored failures. Infrastructure errors are excluded.
Auth
2100%
Run matrix
Every cell is one run. Select one to open its evidence.
| Perturbation | Run 1 | Run 2 | Rate |
|---|---|---|---|
| Baseline | Baseline, run 1: Passed | Baseline, run 2: Passed | 100% |
| Cookie popup | Cookie popup, run 1: Passed | Cookie popup, run 2: Passed | 100% |
| Expired session | Expired session, run 1: Failed | Expired session, run 2: Failed | 0% |
| Mobile viewport | Mobile viewport, run 1: Passed | Mobile viewport, run 2: Passed | 100% |
| Network delay | Network delay, run 1: Passed | Network delay, run 2: Passed | 100% |
| Renamed CTA | Renamed CTA, run 1: Passed | Renamed CTA, run 2: Passed | 100% |
| Slow API | Slow API, run 1: Passed | Slow API, run 2: Passed | 100% |
| Unexpected modal | Unexpected modal, run 1: Passed | Unexpected modal, run 2: Passed | 100% |
Failure clusters
Failures grouped by category and cause, so a repeated defect reads as one problem.
Individual failures
Open one to see the assertion that failed and the trace that led there.