Every frame, every plan, on the same clock

We drove one right turn in Helios eleven times and hashed everything the loop produced. Every decision, every pose and every chase-camera frame matched to the byte. Then we let a 10-billion-parameter learned driver take the wheel twice on two different GPUs. Here is how we checked, a stray 0.0999 that stopped every run at step 101, and where exactness stops on CARLA.

Every simulation team has lived this. A scenario fails, you rerun it to look closer, and it passes. The failure was real, but it is gone, and so is the evidence. In closed-loop driving simulation, where a model sees a frame, plans, and the world moves in response, a run that will not repeat cannot tell a better model from simulator noise.

So we ran an experiment. One right turn, one policy, eleven runs on Helios Engine, our renderer. After each run we hashed everything the loop produced: every steering and throttle command, every vehicle pose, every camera frame.

Every decision, every pose and every chase-camera frame matched in all eleven runs. Then we put a learned driver, Alpamayo 1.5, in the same loop in SimForge Cloud and ran it twice. This post is about how we checked, why that is less common than it sounds, and the places where we do not claim it at all.

Eleven runs, one result

The car queues behind a van, turns right across the junction while a cross-street car passes, and accelerates away past a pedestrian. Every 0.1 s of simulation time Helios renders both cameras and reports the vehicle state, the policy chooses steering and acceleration, and Helios advances the world five engine ticks at 50 Hz. Each run started a fresh renderer on the same Apple M5 laptop GPU.

Below, five of those runs are drawn on top of each other, each at a fifth of full opacity. Any drift in position would show as a ghost; there is one car. The counter in the band does the exact check, pixel by pixel.

"Identical" here is not a visual judgement. Our tooling hashes each step's timing record, action, applied action and full vehicle state with SHA-256 and chained to the step before it, and every camera frame gets its own hash. Eleven runs, one chain digest (first 16 hex characters shown): 1add3829f9364405. Wall-clock time per step varied a lot from run to run, and none of it reaches a digest.

To be clear about the setup: the driver is a deterministic route-following policy that steers from the vehicle state toward the authored route, not a learned model, and it does not look at the camera images. The other road users replay recorded paths without reacting to the car, and the car's reference heading was aligned with its direction of travel so a route follower could track it. The next test uses a learned driver that plans from the camera images.

A learned driver, twice

A route follower is the easy case: it never looks at a pixel. The hard case is a driver that plans from the camera images, because then a single changed pixel can change a plan, and a changed plan moves the car. So we asked SimForge Cloud to let Alpamayo 1.5, a 10-billion-parameter driving model, drive one San Francisco scenario, the approach to a junction where a pedestrian has the walk signal, and then asked it to do exactly the same thing again.

Each run rendered four cameras in Helios, sent them to the model every half second, and drove the car on the returned plan for 200 steps. The two runs landed on different A100 GPUs. When both finished we compared every file they produced. GPU rendering is often named as the layer you cannot make exact; here the frames from two GPUs matched byte for byte.

The engine trace, the replay timeline, the camera frames and both rendered videos match to the byte. What differs is bookkeeping, not driving: the run ids, the GPU serial numbers, a packaging digest, the hashes computed over those, and a closing line in the step log that records how many milliseconds each model call and render took, which is wall-clock time and no part of the simulation. The model's responses differ the same way: session id, latency and memory use, never the plan.

We keep the claim exactly that size: one scenario, two runs, one model on one GPU type. The model's own record says PyTorch's deterministic-algorithms switch was off. Both runs used the same seed, the same bf16 checkpoint and a single trajectory sample. Our CARLA test below needed the model's determinism option to get identical plans, and we have not yet worked out why this pair matched without it. More scenarios, other GPU types and other machines come next.

The 0.0999 that stopped every run at step 101

Early in this work, every closed-loop run on CARLA aborted at step 101. Not most runs: every one, at the same step. The cause was a single hand-typed constant. The camera capture period had been written as 0.0999 s instead of being read from the simulator's own 0.1 s step. The difference is a tenth of a millisecond, and after about 100 steps it had added up to a whole physics substep. The camera fired on the wrong substep, the frame no longer lined up with the step it belonged to, and the run aborted.

That bug is the whole argument in miniature. Time that is approximately right drifts. A hand-typed copy of the clock drifts. The fix was not a better constant but a rule: one component owns simulation time, and everything else asks it.

One clock, five rules

Those rules now form one timing contract with a binding for CARLA and one for Helios. Each rule has a named refusal, so a broken run says why it stopped instead of quietly drifting:

• Integer time. Relative to the run's time origin, step k happens at exactly k × decision_us microseconds. If the engine reports anything else, the run stops with time_drift.
• Whole-number ticks. The decision interval must be a whole number of engine ticks, or the run stops with rate_not_multiple. Helios also checks the tick rate it reports.
• Latency in steps. A model delay is applied a fixed number of steps later, anchored at the pose where the plan was made. A slow machine changes how long a run takes, not what the simulation does.
• Frames by sample time. Each observation takes the frame stamped for its exact simulation time, never the latest one in a buffer. A missing frame stops the run.
• Chained digests. Every step's timing record goes into the run's digest, and SimForge's own engine chains SHA-256 over the world state every tick.

Three more scenarios, with latency

One right turn is one data point, so we ran the contract on three more Richmond Field Station scenarios, this time in the SimForge simulation engine alone, without Helios rendering: the approach to an unprotected left turn against an oncoming car, braking for a red signal, and a hand-over where the car leaves a recording and slows for a parked car. Each ran twice with no model latency and twice with a simulated 200 ms latency.

Same command, different answer

We hold CARLA 0.10 to the same contract, and it is where the interesting failures live. Our connector is the only component that advances time. Each 0.1 s step is ten physics substeps of 0.01 s, the cameras capture on the last one, and every frame is checked against the vehicle pose to 0.01 m and 0.1° before the driver sees it. Here it is with Alpamayo 1.5 driving a San Francisco scenario:

First, replay. We played the same recorded motion five times in each of CARLA's two modes, for 500 ticks. In synchronous mode every run matched: 0 m of difference, and 62 collision events at the same ticks every time. In asynchronous mode, CARLA's default, none of the four run pairs matched. They diverged from the first tick, by up to 0.889 m, with anywhere from 94 to 188 collision events. SimForge only runs synchronous.

Then the closed loop, where it gets subtle. With a scripted driver and the same seed, the commands sent to the car were bit-identical, and so were the traffic, the route and the commanded speed. The car still ended up in a slightly different place: 1e-8 m apart at first, at most 0.87 mm after 200 ticks. A bit-identical command stream still produced a difference, and our diff tool attributes it to the vehicle physics.

The learned-driver runs diverged further. Same-seed runs with Alpamayo 1.5 first differ in the driver's control output at step 17 and by more than 1 cm in position somewhere between steps 34 and 46. In a separate test on identical saved inputs, with its determinism option on, the model returned the same plan in 182 of 182 calls, at 24% more cost per call. With the option off, 0 of 182.

Exact where it can be, measured where it can't

So on CARLA we do not call the full loop deterministic. We say exactly which layer is, and we measure the rest:

This spread is the useful part. It summarizes the variability measured across repeated runs in these scenarios, so we compare models over repeated runs per seed, with an uncertainty estimate for the metric being ranked, never on a single run. In the Helios tests above, actions and poses repeated exactly, and the Alpamayo pair is our first evidence that a learned driver can do the same in this setup. Shrinking that spread is what we are building Helios for.

Side by side

SimForge cells come from our code or our measurement runs. CARLA cells describe its default behaviour, and “Not measured” means we did not measure it.

Bad runs are refused, not counted

A result is only as good as the runs behind it. Every run carries a manifest of digests (driver configuration, renderer bridge, traffic package, simulator image), its seed and per-step records, and a diff tool names the first step where two runs differ and whether the controller output or the vehicle state differed first. Four guards decide what counts:

• No stale frames: a render for a pose the world has not reached is refused and the run stops.
• Pose tolerance: the image pose must match the simulated pose to 0.01 m and 0.1°. It has fired once, on a camera-rig error 5.4 mm and 0.134° out, which is exactly its job.
• Clean exit only: a run counts only with a clean exit, the expected step count and no abort lines.
• Harness faults labelled: a contact caused by our own scripted traffic is marked and kept out of every rate.

What this buys you

• A failure you can keep. In the Helios runs we tested, the same inputs reproduced every action and pose; in CARLA's synchronous replay, every position and collision tick. A failure in a loop like that can be rerun and studied instead of vanishing.
• Diffs you can trust. When two runs disagree, the diff tool names the first differing step and whether the controller output or the vehicle state differed first.
• An honest noise floor. Where a loop is not exact, you know by how much, before you rank anything on it.
• An audit trail. Every counted run carries the manifest, seed and per-step records needed to re-run it and check it step by step.

Next we run learned drivers over many more scenarios, check identity across GPU types and machines, benchmark Helios against CARLA on the same hardware, and work toward running in lockstep with an external ECU. To run one of your scenarios or your own controller on this harness, write to contact@simforge.ai.