Reference Agent · 2026-09-02 14:53
SOLARI LIVEReference Agent · Complete Demo Checkout · run 04ef9ed · 3d ago
Reliability
89%8 / 9 runs passed
95% confidence interval 56.5% – 98.0% over 9 scored runs
- Baseline
- 100%
- Perturbed
- 88%
21%–100%
53%–98%
Runs
9
9 scored
Failed
1
no infra errors
p95 duration
28.9s
p50 7.1s
Avg steps
8.44
avg 10.4s
Run-to-run flips
0%
how often identical repetitions disagreed
Reliability by perturbation
Worst first. The whisker is the 95% Wilson interval — small samples are wide on purpose.
Expired session
State
0%0 / 1
Baseline
Baseline
100%1 / 1
Cookie popup
UI
100%1 / 1
Mobile viewport
Viewport
100%1 / 1
Network delay
Network
100%1 / 1
Renamed CTA
UI
100%1 / 1
Reordered layout
UI
100%1 / 1
Slow API
Network
100%1 / 1
Unexpected modal
UI
100%1 / 1
Reliability by category
State
0%0 / 1
Baseline
100%1 / 1
UI
100%4 / 4
Viewport
100%1 / 1
Network
100%2 / 2
Failure distribution
Only scored failures. Infrastructure errors are excluded.
Auth
1100%
Run matrix
Every cell is one run. Select one to open its evidence.
| Perturbation | Run 1 | Rate |
|---|---|---|
| Baseline | Baseline, run 1: Passed | 100% |
| Cookie popup | Cookie popup, run 1: Passed | 100% |
| Expired session | Expired session, run 1: Failed | 0% |
| Mobile viewport | Mobile viewport, run 1: Passed | 100% |
| Network delay | Network delay, run 1: Passed | 100% |
| Renamed CTA | Renamed CTA, run 1: Passed | 100% |
| Reordered layout | Reordered layout, run 1: Passed | 100% |
| Slow API | Slow API, run 1: Passed | 100% |
| Unexpected modal | Unexpected modal, run 1: Passed | 100% |
Failure clusters
Failures grouped by category and cause, so a repeated defect reads as one problem.
- Auth
The shopping session expired mid-task and the agent did not re-establish it.
1 run · 6acbe0e
Individual failures
Open one to see the assertion that failed and the trace that led there.