Why I Race Wildfire Response Strategies on Identical Random Seeds

Wildfire Sentinel has a mode called the Strategy Lab. It pauses a live fire, clones it once per response doctrine, races the clones, and ranks the doctrines by how many structures each one saved. The single design decision that makes the ranking mean anything is that every clone is seeded identically. This piece is about why, and about what the races actually show when you run a few hundred of them on a real fire.

The trick has a name in simulation statistics, common random numbers, and it is older than most of the software that ignores it. The idea is simple. If you compare two strategies against two separately randomised fires, part of the difference in outcome is the strategy and part is the fire: one drew a gust at the wrong hour, one got an ember that happened to land in scrub. Give both strategies the same fire, ember for ember, and that second part cancels. What remains is the strategy.

What “the same fire” requires

Saying the clones share a seed is easy; making it true took three specific commitments in the engine.

1. One random stream, consumed in one place. The fire engine draws randomness for exactly three things: the hour-to-hour wobble in wind direction, a per-neighbour jitter on the spread rate, and ember spotting (the coin flip, the exponential throw distance, and the bearing scatter). All three draw from one 32-bit generator handed to the engine at construction. Nothing else in the simulation is random.
2. The snapshot carries no generator.When the lab clones a paused fire it serialises the grid state and installs a fresh generator at cursor zero, one per clone, all from the same seed. The engine's own comment on the matter is the rule: ensemble member N is the same fire for every configuration being compared.
3. Iteration order is part of the seed.The engine steps the set of burning cells, and JavaScript sets iterate in insertion order, so the order in which random draws are consumed depends on the fire's history. Rebuild that set from the grid and you get a different, equally valid, not identical fire. The snapshot therefore records the active-cell order explicitly. This is the kind of reproducibility bug that bites once and never again, and the code has a four-line comment to make sure it is never again.

// The RNG stream is deliberately NOT part of the snapshot: every consumer
// seeds its own so that ensemble member N is the same fire for every
// configuration being compared.
...
// The active set is iterated to drive spread, so its ORDER determines
// the order in which random draws are consumed.
activeOrder: [...this.active],

One more commitment belongs on the list: the doctrines race using the exact code that runs the live incident. There is one planner, one fleet model with finite retardant loads and turnaround times, and one station layout function. A doctrine that wins an experiment is the doctrine that then takes over the map. Diverging the two would make every result a claim about a system nobody is running.

Measuring what pairing buys

The case study asserts that pairing makes a real effect show up with far fewer replicates. For this article I wanted to measure that rather than assert it, on the real thing rather than a toy grid. The engine and the lab are plain JavaScript with no browser dependencies, so I ran them headless in Node against the January 2025 Palisades fire: the app's own scenario loader with real terrain, 45,692 OpenStreetMap building footprints, hourly ERA5 reanalysis weather, and the NIFC fire perimeter. The single spread multiplier was fitted the way the app fits it, by maximising overlap with that perimeter across a sweep, which landed at 2.1 with an intersection-over-union of 0.55 and 99.5 percent of the real footprint reached.

Then, at several points in the fire's life, the five doctrines (including the no-drones control) were raced on 32 shared seeds. “Unpaired” was emulated honestly: each doctrine draws its own independent subset of seeds, which is exactly what separately randomised fires are. The comparison is the width of the 95 percent confidence interval on structure cells saved versus the control, and how often a subset of the seeds picks the same winner as all 32.

The pattern is what the theory predicts and what the lab's default of eight replicates depends on. Pairing matters most exactly where the interactive tool lives, at a handful of replicates, and stops mattering once you can afford dozens. The lab exists to answer “what should we do right now” with a few seconds of compute, so the small-replicate regime is the one it has to be good at.

There is no best doctrine

The second thing the races show is the reason the lab races the live incident rather than shipping a recommendation. The winning doctrine changes with the stage of the fire.

Downwind intercept, which lays retardant between the flames and the assets immediately in their path, is the right call while the fire has one direction and a manageable front. Once the fire has spread along a ridge and dozens of streets are threatened, a doctrine that spends its finite sorties by value wins, and the interceptor is stretched across too many fronts. Neither result is surprising once you see it. Both would be wrong to hard-code. The doctrine that should take over the map depends on the map, which is the argument for a tool that measures rather than one that advises.

Ties are real, and the lab says so

Six hours in with a 24-hour horizon, cluster shield and value priority finish within two cells of each other on average. With eight paired replicates the race picks the right winner only two thirds of the time, and no amount of pairing fixes that, because there is nothing to fix: they are genuinely tied on this fire. The lab renders that honestly. A doctrine only gets the trophy when its paired interval against the control excludes zero, and the top row can be first and still labelled “no measurable effect” if the interval straddles.

Where the pairing argument is weaker than I expected

I went into this expecting pairing to dominate at every sample size. It does not, and the reason is worth stating plainly because it is a limitation of the model, not of the method. On this fire, under this weather, the control run is close to deterministic: its per-seed losses vary by a few percent. The Santa Ana wind history is so strong that wind-direction wobble and ember luck barely change where the fire goes. Common random numbers cancel the shared fire-to-fire variance, and when that variance is small there is little to cancel. The doctrines' own outcomes vary more than the control's, because their interaction with the fire (which drop landed where, which ember crossed which line) is where the randomness bites.

That means the case study's framing, that pairing lets a real effect show up with far fewer replicates, is true in the small-replicate regime and for weaker fires, and only mildly true for a wind-driven fire raced at 32 replicates. The ensemble itself is another limitation: its members diverge through the engine's internal stochastic channels within a single deterministic weather history. They are not alternative weather forecasts. A forecast ensemble with genuinely different wind timelines would have far more shared variance, and pairing would earn far more.

If you are building a comparison harness

• Put every source of randomness behind one generator and hand it in from outside. Grep for Math.random and make the count zero.
• Exclude the generator from your snapshot and include anything that affects the order draws are consumed in.
• Always run a control, whether or not the user asked for it, and report every number as a paired difference from it.
• Compute significance on the paired differences and show it. Rank last, and never give a trophy to a row whose interval straddles zero.
• Race the same code that will act on the result.

The fire model, the data pipeline that assembles a scenario for any point on Earth, and the validation against the real fire are in the Wildfire Sentinel case study. The engine runs headless in a few seconds per race, which is what made this article possible to write with numbers instead of adjectives.