small real gauntlet
SOLARI LIVEReference Agent · Complete Demo Checkout · run 1e6db4f · 3d ago
Reliability
75%3 / 4 runs passed
95% confidence interval 30.1% – 95.4% over 4 scored runs
- Baseline
- 100%
- Perturbed
- 67%
21%–100%
21%–94%
Runs
4
4 scored
Failed
1
no infra errors
p95 duration
1m 53s
p50 1m 23s
Avg steps
10
avg 1m 18s
Run-to-run flips
0%
how often identical repetitions disagreed
Reliability by perturbation
Worst first. The whisker is the 95% Wilson interval — small samples are wide on purpose.
Expired session
State
0%0 / 1
Baseline
Baseline
100%1 / 1
Cookie popup
UI
100%1 / 1
Unexpected modal
UI
100%1 / 1
Reliability by category
State
0%0 / 1
Baseline
100%1 / 1
UI
100%2 / 2
Failure distribution
Only scored failures. Infrastructure errors are excluded.
Auth
1100%
Run matrix
Every cell is one run. Select one to open its evidence.
| Perturbation | Run 1 | Rate |
|---|---|---|
| Baseline | Baseline, run 1: Passed | 100% |
| Cookie popup | Cookie popup, run 1: Passed | 100% |
| Expired session | Expired session, run 1: Failed | 0% |
| Unexpected modal | Unexpected modal, run 1: Passed | 100% |
Failure clusters
Failures grouped by category and cause, so a repeated defect reads as one problem.
- Auth
The shopping session expired mid-task and the agent did not re-establish it.
1 run · dfdbd18
Individual failures
Open one to see the assertion that failed and the trace that led there.