Reliability testing for browser agents

Crash-test your browser agent
before production does.

Benchmarks tell you whether your agent can complete a task. AgentGauntlet measures whether it keeps completing it when the environment changes — across repeated runs, on real cloud browsers, judged from state the agent cannot fake.

Bring any agent. AgentGauntlet doesn't care how it thinks — only whether it survives.

Explore the demo Run a real gauntlet

Exploring costs nothing. Starting a run executes on Solari and spends credits, so it asks for an access code.

Repeat

Run identical tasks many times. One success is an anecdote; a rate is a measurement.

Perturb

Change UI, timing, viewport, locale and session state — deterministically, from a seed.

Debug

Open the failed run: the exact assertion, the action trace, and the session replay.

A benchmark asks whether the agent can do it once.

AgentGauntlet asks whether it still works when a cookie banner appears, when the API is slow, when the button is renamed, when the viewport is a phone, or when the session quietly expires halfway through.

Completion is judged from the benchmark site's own server-side state — never from the agent's report about itself. An agent can say “done”; only the state decides.

The built-in Reference Agent is deterministic and needs no model key, so results are reproducible. Point it at your own agent instead — any framework, any model, running in an isolated sandbox — and the measurement is the same.

Reliability belongs in CI

gauntlet run

AgentGauntlet

  ✓ baseline           2/2
  ✓ cookie popup       2/2
  ✓ slow API           2/2
  ✗ unexpected modal   0/2
  ✓ mobile viewport    2/2
  ✓ renamed CTA        2/2
  ✗ expired session    0/2
  ✓ network delay      2/2

  Reliability   75.0%   (12/16)
  Required      90.0%

  FAIL  exit 1

Recent runs

All runs →