How it works
Four models, one game, and a number that is only ever the time the model took.
Why Pong
Pong is the Atari game chat models handle worst: below random in the Atari-GPT benchmark, mostly because deciding takes them longer than the game gives them. A live loop is the one job where speed is the whole point, so it is the fair place to measure a model built to answer in milliseconds. Jev is a typed-decision model: a small state goes in, a typed answer comes out.
The rules
- Same serve, same rules, same question in every lane.
- One decision moves the ball one step. 8 steps cross the court.
- The model's paddle moves at most 12 units per decision, which is enough reach over a crossing to get anywhere the ball can go.
- The left paddle is a script, not a model. It always returns the ball and it takes no time to decide, so the only difference between lanes is the model on the right.
- When you play, a step never finishes in under 260 ms. Jev at a couple of hundred is a reflex test rather than a game. The floor paces the ball, not the clock: the number beside the court is still Jev's real round trip.
What the model sees
The whole state, numbers only, 125 bytes:
{"court":{"w":160,"h":100},"ball":{"x":80,"y":50,"vx":19,"vy":4.4},"paddle":{"y":50,"h":20},"interceptY":67.4,"dir":"toward"}The question, in the words every lane is given:
You control the right paddle in a game of Pong. The state is a JSON object of numbers: the court size, the ball position and velocity, your paddle (paddle.y is its CENTRE, paddle.h its height), interceptY, and dir. y = 0 is the top edge and y grows downward, so "up" DECREASES y and "down" INCREASES y. interceptY is the y value where the ball will cross your paddle plane after any wall bounces; it is null when the ball is moving away from you. Move so that paddle.y covers interceptY. Treat the paddle as already covering it when paddle.y is within 5 units of interceptY. Answer with one move only.
Three answers: up · down · stay.
Jev vs the chat models
Jev answers through the AI SDK experimental_evaluate API. The chat models get the identical state and the identical words through structured output, at temperature 0, with reasoning off. Nobody gets a retry. The number on a lane is the round trip of that one call, timed on the server, and nothing else.
Other models, same question
The four lanes were chosen to be simple: one typed-decision model against the three chat models most people recognise, recorded next to the Gateway so nothing on the front page is anyone's broadband. After launch, people asked about the newest frontier models, about Qwen, and about the smallest chat models. So the same question went to them too, on 2026-09-19: 30 real game states, one call per decision, sequential, reasoning switched to the lowest setting each provider accepts. The principle holds: nothing gets near Jev's time without starting to get the question wrong, and nothing is more accurate.
| Model | Setting | Correct | Median | p95 | vs Jev |
|---|---|---|---|---|---|
| Jev | evaluate | 28/29 | 222 ms | 288 ms | 1x |
| Ministral 3B | default | 26/30 | 396 ms | 588 ms | 1.8x |
| Qwen 3 235B | thinking off | 28/30 | 608 ms | 919 ms | 2.7x |
| Gemini 3.5 Flash-Lite | thinking off | 30/30 | 660 ms | 799 ms | 3x |
| Claude Haiku 4.5 | thinking off | 30/30 | 791 ms | 1885 ms | 3.6x |
| GPT-5.4 Nano | default | 29/30 | 799 ms | 1070 ms | 3.6x |
| Qwen 3.8 Max | thinking off | 30/30 | 834 ms | 1222 ms | 3.8x |
| Qwen 3.8 27B | thinking off | 30/30 | 950 ms | 2928 ms | 4.3x |
| Qwen 3.8 Flash | thinking off | 30/30 | 951 ms | 2358 ms | 4.3x |
| GPT-5.6 Sol | default | 30/30 | 1153 ms | 1954 ms | 5.2x |
| GPT-6 Astra-fast | reasoning effort low | 30/30 | 1355 ms | 3158 ms | 6.1x |
| GPT-6 Astra | reasoning effort low | 30/30 | 1542 ms | 2113 ms | 6.9x |
| GPT-6 Astra | reasoning effort high | 30/30 | 1548 ms | 2763 ms | 7x |
| Claude Fable 5.1 | adaptive thinking, effort low | 30/30 | 4530 ms | 6887 ms | 20.4x |
Measured from a Vercel function in iad1, next to the Gateway, which is where the lanes were recorded, so Jev's median here is the same round trip the front page shows. Astra at medium effort landed with low and high, all within 200 ms of each other on a one-token answer, and Fable at high effort came in at 3.6 s with an 11 s p95, so neither setting is listed twice: the setting is not what makes them slow. The chat models answered 383 of 390 correctly between them; Jev missed one. Jev got no answer to one call, which is not counted. Not smarter. Faster.
Where Ably comes in
POST /api/game leaves a worker running on Vercel. It joins the channel pong:game:<id> as an ordinary member, announces itself in presence as the agent, runs the physics and publishes a state snapshot after every decision. Your browser is another member of the same channel: present as a player, publishing input messages carrying one move, drawing what the channel says rather than a simulation of its own.
That is why the watch link works. Anyone who opens it attaches to the same channel with rewind, so the current state arrives immediately instead of at the next decision, and the number of people watching is that presence set, counted.
Read the code
It is all public, and it is meant to be read. Eight files, in order.
- 1lib/game/types.ts · the contract: the court, one decision = one tick, and the bytes a model is given
- 2lib/game/engine.ts · the whole game as pure functions, including the paddle that never misses
- 3lib/decide/prompt.ts · the one question every model is asked, and its answers
- 4lib/decide/jev.ts · evaluate, for Jev. lib/decide/llm.ts is structured output, for the rest
- 5lib/worker/game-worker.ts · the agent as a member of the channel: the decision loop and the wire
- 6lib/worker/referee.ts · when a game stops, with no clock of its own
- 7app/api/game/route.ts · one HTTP request, one function, one game
- 8lib/ably/hooks.ts · everything a browser does: subscribe, be present, publish input