# Reinforcement learning with SimForge Gym

> Serve SimForge episodes with simforge env serve and train a policy with SimForge Gym (the open simforge-rl Gymnasium client), on your own machine.

Source: https://simforge.ai/docs/rl

`simforge env serve` runs episodes of a scenario on the local runtime, with reactive traffic and optional rendered cameras. SimForge Gym, the open `simforge-rl` Python package, is the Gymnasium client: it never contains a simulator of its own, so without a running server there is no episode. Reinforcement learning runs on your machine; the cloud does not run it.

Install it from PyPI:

```sh
python -m pip install 'simforge-rl[ppo]'
```

## Serve episodes

```sh
simforge env serve s.json --socket /tmp/simforge-env.sock --no-sensors --max-sessions 4
```

The server prints a ready document and runs until you stop it. `--no-sensors` serves state only and needs no GPU. For camera observations, serve with `--rig rig.json` instead (see [Preview and render](/docs/render) for rigs). `--max-sessions` must cover every environment you open, including the evaluation one.

On Windows use `simforge-rl` 0.1.3 or newer and a runtime advertising `capabilities.envSocketTransport: "named-pipe"`. The same `--socket` path selects a secure local named pipe with a bounded hashed name, so long profile paths do not hit the Unix socket path limit. Older Windows runtimes fail with an upgrade instruction. macOS and Linux use Unix sockets.

## Drive it from Python

```python
from simforge_rl import SimForgeEnv, SocketTransport, WeightedTerms

with SimForgeEnv(
    SocketTransport("/tmp/simforge-env.sock"),
    action_mode="setpoint",
    observation="state",
    reward_fn=WeightedTerms({"progress": 1.0, "proximity": 2.0}),
) as env:
    observation, info = env.reset(seed=7)
    observation, reward, terminated, truncated, info = env.step([5.0, 1.0])
```

- A `setpoint` action is `[target_speed_mps, target_accel_mps2]`; `action_mode="control"` takes `[throttle, brake, steer]`; `None` keeps the scenario's scripted control.
- `observation="state"` is privileged engine state. `"sensors"` returns only rendered camera passes keyed `<sourceId>/<pass>`, with no engine state in `info`; `"state+sensors"` returns both.
- `reward_fn` replaces the reward; `info["native_reward"]` keeps the runtime's own.
- Reset after an episode terminates or truncates; nothing resets on its own.

## Train with PPO

```sh
python -m simforge_rl.examples.ppo --tasks richmond-field-station --runtime --num-envs 8 --total-timesteps 200000 --out run/
```

State observations need no GPU. `--tasks` picks a bundled task set: one per released map (`belmont-research-center`, `easterbrook-discovery-school`, `el-camino-road`, `richmond-field-station`, `yale-street`), each 24 to 29 weighted, seeded scenario templates (car following, cut-ins, lead braking, crossing pedestrians, unprotected turns, red-light runners) at sites verified on that map. A set is pinned to the map version it was verified on: pull that version. `--runtime` runs each worker's episodes in its own runtime process; without it the example starts `simforge env serve` processes instead, with the same observations and rewards. `run/` then holds `curve.csv`, `curve.svg`, `metadata.json` and `policy.pt`, and each evaluation also scores the scenarios' scripted driving as a baseline. `python -m simforge_rl.examples.ppo --help` lists every flag.

## Train on a pulled dataset

```sh
simforge datasets pull <dataset-id> --include scenarios --out data/my-dataset
python -m simforge_rl.examples.ppo --dataset data/my-dataset --split train --runtime --max-open 16 --eval-limit 8 --out run/
```

`--dataset` takes a pulled folder or a hosted dataset id, which it pulls first (see [Getting data out](/docs/datasets)); every file is checked against its digest, and each episode runs exactly the input its scenario's simulation ran, on the installed map the scenario names. `--max-open` bounds the sessions each worker keeps open, so a split of thousands of scenarios trains with a bounded number of sessions. In Python, `ScenarioSet.from_dataset(...)` and `ScenarioSet.bundled(...)` give the same task sets.

## Persistent worlds

`simforge world serve` hosts one shared, realtime world that several clients join and drive, rather than isolated episodes. It authenticates clients with a token from a file you create first and keep private (for example `umask 077; head -c 32 /dev/urandom | xxd -p > world.token`), passed as `--token-file`; see `simforge world serve --help`.