sitelooper

Write browser tests in plain English. Run them in CI as plain Playwright. Repair them automatically when the app changes.

sitelooper is a CLI with a small LLM agent inside. You give it one instruction at a time, such as "sign in as ops@example.com, create a ticket titled 'k7 Bench' and report its id". It works the live browser and returns one verified result. Every step it gets right is compiled into a procedure that replays with no model, no tokens and no cost, and can be exported as a standalone @playwright/test spec.

Recorded once against the in-repo repair-desk app (24s, 28s and 42s with the agent driving). The app is reset, the same flow replays in 16s with zero model calls, and then the app's own state is queried.

How it works

1. Record. A cheap model follows your instructions. As it works, sitelooper records durable locators, where each value came from, and what each step changed on the page.
2. Replay. The recording replays deterministically. If the app has changed under a step, only that step goes back to the model, and a recovery that validates is pinned into the flow, so the flow heals itself over runs.
3. Compile. sitelooper build emits a Playwright spec with effect checks on every step. It runs in CI with no sitelooper daemon and no model.

Results

Ten self-hosted apps, each job done once to record it, then twice more with no model, then as a compiled Playwright spec. The comparator is agent-browser driven by an LLM, which redoes the job on every run. Every run was scored against the app's own database or API, never against the tool's own report. September 2026:

• sitelooper verified every objective in all 30 runs, with GPT-6 Luna writing the instructions and DeepSeek V4.1 Flash driving the browser. Every replay ran with zero model calls. Nine of the ten compiled specs pass; EspoCRM's refuses to compile rather than guess.
• agent-browser on the same model was clean on 3 of 11 runs. It failed outright on Odoo, EspoCRM and Snipe-IT, posted duplicate comments on three apps, and twice reported work as done that the app had not saved.
• On GPT-6.1 Sol, about 20× the per-token price, agent-browser was clean on 9 of 10. It still pays that price on every run, and on Grafana it again reported unsaved settings as verified.

Report-only objectives ask a run to state a value in its final report, which a compiled spec does not write.

• bench/RESULTS.md: the protocol, the models, costs, and what the benchmark does not show
• [bench/MATRI

... View full README on GitHub