AgentMaze

Overview

AgentMaze is a first-person navigation task and the training stack that goes with it. The agent sees only a RGB camera frame and has to reach a goal in a maze that was regenerated moments earlier. No map, no state vector, no waypoint list, no privileged access to its own coordinates.

Under the hood, I built a game in Unity Engine step deterministically in lockstep with a Python trainer, fast enough that a 10 million step run finishes in a day, and then doing the whole thing a second time in Unreal Engine without changing the trainer.

Metric	Value
Peak success rate on unseen mazes	92%
Training steps, headline run	10M
Parallel environments	20
Simulation speed	70-100x real time
Engines behind one wire protocol	2

The training with Unity game.

Results

The headline run trained PPO from pixels for 10M steps across 20 parallel environments, peaking at roughly 92% rolling success on 5x5 mazes in Unity. Both the layout and the wall and floor textures are regenerated every round, so there is no fixed level to memorize. The policy has to learn corridor-following and dead-end recovery rather than a route.

Configuration	Episode limit	Steps	Success
3x3, fixed maze	300 s	—	sanity check
5x5, randomized	180 s	6.0M	0.75
5x5, randomized	180 s	8.7M	0.80
5x5, randomized	60 s	10M	0.92

These rows are separate runs rather than one continuous curve; the episode time limit tightened as the agent became more reliable. The final row is the headline run, which peaks near 6.5M of its 10M steps.

Policy and Reward

Observation	64x64 RGB, 4-frame stack via VecFrameStack
Action space	Box(-1, 1), 5-D continuous — moveX, moveY, lookX, lookY, jump
Jump	thresholded from the continuous head at 0
Termination	goal trigger reached
Truncation	episode time limit, owned by Python
Parallelism	4 engine processes x 5 agents = 20 environments

Four consecutive observations, at their native 64x64 capture, from a CitySample run in Unreal Engine 5.

The frame stack, oldest to newest: t-3, t-2, t-1, t.

Hyperparameters

Parameter	Value
policy	CnnPolicy
n_steps	1024
batch_size	256
n_epochs	4
target_kl	0.02
learning_rate	2.5e-4
gamma	0.999
log_std_init	-1.0

Three of these are not defaults and were chosen deliberately. target_kl = 0.02 caps the update when a maze rebuild shifts the observation distribution mid-rollout. gamma = 0.999 keeps a long episode’s terminal bonus visible from the start. log_std_init = -1.0 sets the initial exploration standard deviation to roughly 0.36 — and that number nearly sank the entire Unreal effort, for reasons covered further down. Every value is a CLI flag, and the trainer refuses to start if batch_size does not divide n_steps x num_envs.

Shaped, clipped, and knowingly non-invariant

A sparse goal reward does not work here. A random policy in a 5x5 maze reaches the exit rarely enough that PPO sees essentially no signal. So the reward is potential-based, over BFS distance to the goal computed on the maze grid:

$$\phi(s) = -0.1 \cdot d_{\mathrm{BFS}}(s)$$ $$r = \mathrm{clip}(\Delta\phi,\ \pm 0.15) \;-\; 0.01 \;+\; 5.0 \cdot \mathbf{1}[\text{goal}]$$

Two details are deliberate. The shaping delta is clipped to ±0.15, so a mid-episode maze rebuild or a streaming hitch cannot hand the policy a large spurious return that it will happily learn to chase.

And the discount factor is omitted from the shaping term. The textbook form is what makes potential-based shaping policy-invariant, but at gamma = 0.999 the residual left standing still scoring close enough to progress that early training stalled. I traded the exact invariance guarantee for a signal that actually moves the agent, and kept an ablation switch (--reward-mode goal_only) so I could measure what the shaping was buying.

Reward is computed engine-side and mirrored in Python, so the two implementations can be checked against each other.

Simulation Infrastructure

Most game engines’ default relationship with time is optimized for realtime gameplay showed to players. That will cap throughput at real time by default. Therefore the advancing of time in game engine needs to be manual.

physics_fps — default 30 Hz

Fixed simulation substeps. One step command advances an exact substep count, so game time is a pure function of steps taken, never of how fast the machine runs.

sim_fps — default 5 Hz

The decision rate. The agent acts every six substeps, which is what makes a 0.2 s action interval mean the same thing on every machine.

capture_fps — 5 Hz

The render rate. Rendering only at capture ticks is where most of the speedup comes from. Frames the trainer will never see are never drawn and main camera is disabled when running in headless mode.

Divisibility is validated on both sides at startup. It is required that physics_fps % sim_fps and physics_fps % capture_fps must both be zero, so an invalid combination fails at startup instead of quietly desynchronising two processes.

On top of that sits a small binary protocol: newline-delimited JSON commands down, a fixed-size header plus raw RGB24 frames up, with multiple agents multiplexed over a single socket per process. Frames are never encoded because PNG compression on the hot path would cost more than the transfer it saves. Captures come off the GPU through an asynchronous readback, so the render pipeline never stalls waiting for the CPU.

Run headless in batchmode, this sustains 350-500 observations per second across 20 environments, or roughly 70-100 aggregate seconds in game time per second in real world.

Porting to Unreal Engine 5

The Python side did not change, it is built to support multiple game engines as long as they follow the same protocol.

I wrote MazeRLBridge, a roughly 4,000 line C++ plugin that turns any packaged UE5 project into an RL environment with the identical wire protocol as the Unity build. It will spawn a world subsystem and the RL maze manager on begin play, so there are no level edits required, and nothing in the plugin references game project content.

Three engine-level problems had to be solved to get there.

Manual Physics Simulation

FApp::SetUseFixedTimeStep with SetFixedDeltaTime(1/physics_fps) pins one engine tick to one physics substep. Between steps the world is paused entirely, while the bridge manager keeps ticking to serve the socket. Without that pause the world would keep simulating during the Python round trip, and the agent will not perceive the continuity in scene.

Input timing

Agent input are applied in TG_PostPhysics, which lands after physics and is consumed by the character movement component on the next substep, giving a consistent one-substep input pipeline instead of a race against tick order.

World Partition

A headless run has no player pawn, so nothing acts as a streaming source and no cells ever load. Therefore, each agent needs to carry a streaming source component, and the first capture after a reset waits on IsAllStreamingCompleted() and on every agent settling onto the ground. The world is unpaused while it waits, since streaming and gravity only advance while it ticks.

The pose recorded for a reset is the settled one, not the spawn transform. A character spawns slightly above the floor, so restoring the spawn transform would put it back in mid-air and the reset observation would not match the episode’s first observation.

Scaling to CitySample

Before apply the maze RL bridge to CitySample, I built a whitebox project that has minimal map setup to verify correctness of the plugin and protocol. After verification, I add the plugin into CitySample project. The whitebox maze is a controlled environment, but CitySample is not. It has a fully authored, World-Partitioned city built for cinematic rendering rather than machine learning. Almost every convenience the maze provided disappears there, and each absence needed a different answer.

Placement, without a nav mesh

CitySample ships no nav mesh, which rules out GetRandomReachablePointInRadius. Sampling a random point and tracing downward fails too, because under World Partition only cells near an agent are resident, so a trace at an arbitrary point hits nothing at all.

I sampled the ZoneGraph lane network instead, this walkable network is the same as the crowd system spawns pedestrians on, and one that loads independently of cell streaming. The candidate pool is built once per world by walking tag-filtered lanes; only the per-round draw is reseeded, so a round is reproducible from two seeds.

Placement as a difficulty knob

A min/max goal-distance band controls how hard a round is. Lane tags keep agents on pedestrian lanes and off the road, and a minimum separation stops agents spawning on top of each other. Constraints relax rather than fail: if nothing in the pool falls inside the band, the band is dropped for that round with a warning.

Agents get a random yaw on purpose. A start that always faces the goal hands the policy the answer in its first observation.

Distance the agent can actually walk

Straight-line distance measures through buildings, so distance-to-goal resolves in three tiers: a ZoneGraph A* route, then a nav-mesh path length, then straight-line as a last resort. This matters because the naive fallback is not monotone along any route the agent can actually walk, and shaping a reward with it teaches the agent to press itself against a wall. The reported value includes the off-lane legs at both ends, so it keeps changing as the agent moves perpendicular to a lane and shrinks to near zero at the goal.

Honest observations in a temporal renderer

Ideally an observation is a pure function of world state. Unreal’s renderer is temporal, so it is not. Eye adaptation, temporal antialiasing and motion blur all carry history between captures, so all three are disabled on the capture component, followed by explicit warm-up and settle frame counts before a capture is trusted.

That shrinks the problem rather than removing it, so I measured what was left.

Walking an agent 1900 uu away and teleporting it back to a bit-identical pose gives a mean per-pixel difference of 6.1/255 at 30 settle frames and 3.0/255 at 90, against roughly 10/255 for frames the agent genuinely walked to. The pose is restored exactly; the residual is purely the renderer’s caches converging, and it shrinks asymptotically rather than reaching zero. Settle frames trade reset throughput for observation stability, and this measurement is how I chose where to sit on that curve.

Antialiasing, because flicker looks like motion

With TAA off and no MSAA in a deferred renderer, captures have no antialiasing at all. On a whitebox maze that is fine. In a detailed city it corrupts the observation in a specific and nasty way: Nanite and virtual shadow maps render subpixel detail on the assumption that a temporal resolve will integrate it, and at 64 pixels nothing does. A pedestrian twenty metres away covers a few pixels, so whether they register depends on subpixel phase — they flicker between consecutive captures.

Stack four frames to give the policy velocity information and it encodes that flicker as motion, while shimmer on static geometry reads as movement on things that are not moving. The fix renders each agent offscreen at a higher resolution and box-filters back down, in linear light — averaging sRGB bytes directly darkens exactly the antialiased edges the feature exists to produce. It is purely spatial with no history, so the determinism properties above still hold, and the wire format is untouched.

Fix Runs That Learned Nothing

Explained variance sat at roughly zero for 139,000 steps. Every loss curve looked unremarkable. The metrics never told me why, because the problem was not in them.

Symptom

Five consecutive CitySample runs produced no learning at all. Success stayed flat, explained variance stayed pinned near zero, and nothing in the training metrics distinguished a broken run from a merely slow one.

Diagnosis: stop reading curves, look at the observations

I dumped the 64x64 captures the policy was actually receiving and paged through them. The agent was staring at the pavement, and at its own mannequin mesh, for most of every episode.

The look action is applied as an angular rate, not an absolute angle. At 120°/s, a 0.2 s decision interval and a policy standard deviation of about 0.36, every single step applied a random ±8.6° kick, and those kicks accumulated. Pitch random-walked into its ±80° clamp and dwelled there; yaw fully randomised heading within about fifty steps. Since movement is body-relative and zero-mean, displacement was diffusion since it is roughly 4.5 m per episode against goals 6-15 m away.

The agent was not failing to learn navigation. It was being prevented from ever generating a trajectory worth learning from.

Fix

Two values: pin pitch to zero, and cut look speed from 120°/s to 30°/s.

Result

Success went from 6% to 86% over 230k steps, and explained variance held between 0.45 and 0.65 for the rest of the run. No hyperparameter changed.

When an action is an integrator driven by zero-mean exploration noise, the noise does not average out — it accumulates into a random walk, and the agent’s own exploration destroys the trajectories it needs to learn from. No hyperparameter sweep finds this, because every hyperparameter is innocent. The rule I now apply first: when a pixel-based agent shows flat explained variance, render the observations and look at them before touching a single training parameter.

Throughput

Every CitySample run decayed monotonically from about 35 fps to 6-8. At 7 fps a 200k-step run costs a full day, which made throughput — not sample efficiency — the binding constraint on how fast I could learn anything.

The process reached 14-15 GB of private commit against 31.7 GB of RAM, at which point Windows trimmed its working set to around 7 GB and it began paging. Handles and threads stayed flat and the footprint plateaued, which ruled out a leak and pointed at the level itself. Restarting the process reset throughput every time.

Rather than chase a fix inside a shipped sample project, I restructured training as chained checkpoint resumes every 30-40k steps, which holds 20-35 fps instead of averaging about 15 — roughly a 2x improvement in wall-clock iteration speed for an afternoon of work. One trap worth knowing on that path: PPO.load restores the saved hyperparameters and silently ignores the CLI, so only environment-side arguments take effect on a resume.

Engineering Practice

• 68 tests, 881 lines, covering protocol framing, clock validation, frame decoding, action mapping and the VecEnv contract — the places where a silent mismatch between two processes is otherwise invisible
• Layering enforced by a test: core to transport to env to app, with the dependency direction asserted automatically rather than documented and hoped for
• Seeded from one number: per-process RNG streams derived from a single base seed, so a fully randomized run is still reproducible end to end
• Compile-time escape hatches: the three engine APIs whose signatures move between UE versions each sit behind a switch with a working fallback, so an engine upgrade degrades instead of breaking

Next Steps

• Recurrent policy. A 4-frame stack is a crude answer to partial observability. An LSTM policy should handle the maze’s real memory demand — remembering which corridors it has already tried — far better than four frames of history can.
• Shared-memory transport. TCP copies are not the bottleneck at 64x64, but they will be at higher resolutions or agent counts.
• Curriculum over the goal-distance band. The placement system already exposes the knob; nothing yet schedules it.
• Late-training stability. Success dips after its peak in the 10M run. I have a suspicion about the clipped shaping term interacting with target_kl, and no evidence yet.