An open-source framework for agent-run e2e tests on web and mobile.
Oskar
@o_kwasniewski
e2e + jev from @typesafeai ⚡
I'm building an open-source framework for running e2e tests with agents. supports web, mobile (and more!)
available soon: tester.army/e2e
Oskar is building an open-source framework for running end-to-end tests with agents, on the web, on mobile and more, with Jev in the loop. It is announced at tester.army.
Breaking: Browser Use + Jev = Ultrafast ⚡
Findings flights took 7s and cost only $0.0039 🤯
> new action space every step
> DOM state space
> small LLM fallback to type
(this video is at 1x speed btw)
Built a tiny open source browser agent. try it below ↓
We just ran Jev on our WebMCP benchmark.
The result: basically broke the benchmark.
Jev + Mercury 2.5 (a fast, low-cost LLM) using WebMCP solved 100% of the tasks at roughly 112× lower model cost than GPT-6 Astra using computer use with code execution. Compared to Astra using screenshot-based computer use, the model cost was 245× lower (!).
We also compared Jev operating the browser with and without WebMCP.
We used Browser Use’s open-source Ultrafast, with some improvements to the harness to make it more reliable across the benchmark.
Jev’s browser-control accuracy on its own was not amazing - adding WebMCP nearly doubled the number of solved tasks, from 25/49 to 49/49, while reducing model cost by 18% (more on why below).
The benchmark and methodology are fully open and reproducible.
Full results: https://webmcp.com/benchmark
A few words on how the Jev + WebMCP harness works and why this is exciting:
Jev receives text as input and a set of discrete options it can choose from. With WebMCP, those options are the tools exposed by the website. At each step, Jev sees the task, the available tools and previous results, then picks what to do next.
The limitation is that Jev can’t generate arbitrary text, which you need for tool arguments. For example, it can choose the search_products tool, but it can’t generate the search query itself.
So we split the work: Jev picks the tool and Mercury 2.5 generates the arguments if needed.
This works well because turns out most of the cognitive load in these tasks is around choosing the right action. The argument generation itself is relatively simple, so we can delegate to a small and very fast model. We used Mercury, which outputs 1,000+ tokens/sec and is very cheap.
The result is a pretty simple combination: Jev for tool selection + Mercury for arguments + WebMCP for the interface. It ends up being very reliable, very fast, and very cheap.
A few words about Ultrafast and why do we think it underperforms:
Without WebMCP, Jev chooses from the page’s controls: which button to click, which field to fill, or which option to select.
But choosing a valid button is different from choosing the right next step. The agent still has to navigate menus, understand forms, recover from errors and recognize when the task is actually complete.
Our hypothesis is that WebMCP makes the decision space much simpler. Instead of figuring out a sequence of clicks through a website, Jev chooses explicit actions that directly advance the task.
@typesafeai itself documents weaker accuracy on questions requiring multiple reasoning steps. WebMCP moves much of that complexity into the website’s tools, leaving Jev with clearer decisions and fewer opportunities to go wrong (in a sense WebMCP "compresses" a sequence of clicks into one tool call).
Our modified Ultrafast setup solved 25/49 tasks - that is a result for our particular implementation and benchmark, not a universal limit on Jev or Browser Use. We are open to more harness optimization to get this result to perform better, feel free to directly contribute to the benchmark here: https://github.com/nekuda-ai/WindTunnel
Browser-use ultrafast: https://github.com/browser-use/jev-ultrafast
we built blazing fast computer/browser use with Jev + @Stagehanddev.
this task cost $0.001 and executed at near instant speed (in a remote browser btw)
the loop: observe the page, send a11y tree as state + actions as questions, Jev decides the next action, then Stagehand executes it.
