Reinforcement learning with SimForge Gym

simforge env serve runs episodes of a scenario on the local runtime, with reactive traffic and optional rendered cameras. SimForge Gym, the open simforge-rl Python package, is the Gymnasium client: it never contains a simulator of its own, so without a running server there is no episode. Reinforcement learning runs on your machine; the cloud does not run it.

Install it from PyPI:

python -m pip install 'simforge-rl[ppo]'

Serve episodes#

simforge env serve s.json --socket /tmp/simforge-env.sock --no-sensors --max-sessions 4

The server prints a ready document and runs until you stop it. --no-sensors serves state only and needs no GPU. For camera observations, serve with --rig rig.json instead (see Preview and render for rigs). --max-sessions must cover every environment you open, including the evaluation one.

On Windows use simforge-rl 0.1.3 or newer and a runtime advertising capabilities.envSocketTransport: "named-pipe". The same --socket path selects a secure local named pipe with a bounded hashed name, so long profile paths do not hit the Unix socket path limit. Older Windows runtimes fail with an upgrade instruction. macOS and Linux use Unix sockets.

Drive it from Python#

from simforge_rl import SimForgeEnv, SocketTransport, WeightedTerms

with SimForgeEnv(
    SocketTransport("/tmp/simforge-env.sock"),
    action_mode="setpoint",
    observation="state",
    reward_fn=WeightedTerms({"progress": 1.0, "proximity": 2.0}),
) as env:
    observation, info = env.reset(seed=7)
    observation, reward, terminated, truncated, info = env.step([5.0, 1.0])

• A setpoint action is [target_speed_mps, target_accel_mps2]; action_mode="control" takes [throttle, brake, steer]; None keeps the scenario's scripted control.
• observation="state" is privileged engine state. "sensors" returns only rendered camera passes keyed <sourceId>/<pass>, with no engine state in info; "state+sensors" returns both.
• reward_fn replaces the reward; info["native_reward"] keeps the runtime's own.
• Reset after an episode terminates or truncates; nothing resets on its own.

Train with PPO#

python -m simforge_rl.examples.ppo --tasks richmond-field-station --runtime --num-envs 8 --total-timesteps 200000 --out run/

State observations need no GPU. --tasks picks a bundled task set: one per released map (belmont-research-center, easterbrook-discovery-school, el-camino-road, richmond-field-station, yale-street), each 24 to 29 weighted, seeded scenario templates (car following, cut-ins, lead braking, crossing pedestrians, unprotected turns, red-light runners) at sites verified on that map. A set is pinned to the map version it was verified on: pull that version. --runtime runs each worker's episodes in its own runtime process; without it the example starts simforge env serve processes instead, with the same observations and rewards. run/ then holds curve.csv, curve.svg, metadata.json and policy.pt, and each evaluation also scores the scenarios' scripted driving as a baseline. python -m simforge_rl.examples.ppo --help lists every flag.

Train on a pulled dataset#

simforge datasets pull <dataset-id> --include scenarios --out data/my-dataset
python -m simforge_rl.examples.ppo --dataset data/my-dataset --split train --runtime --max-open 16 --eval-limit 8 --out run/

--dataset takes a pulled folder or a hosted dataset id, which it pulls first (see Getting data out); every file is checked against its digest, and each episode runs exactly the input its scenario's simulation ran, on the installed map the scenario names. --max-open bounds the sessions each worker keeps open, so a split of thousands of scenarios trains with a bounded number of sessions. In Python, ScenarioSet.from_dataset(...) and ScenarioSet.bundled(...) give the same task sets.

Persistent worlds#

simforge world serve hosts one shared, realtime world that several clients join and drive, rather than isolated episodes. It authenticates clients with a token from a file you create first and keep private (for example umask 077; head -c 32 /dev/urandom | xxd -p > world.token), passed as --token-file; see simforge world serve --help.