Why Random Cross-Validation Lies in Geospatial ML

SporeCast, my mushroom-forecasting model for Washington State, reports an AUC of 0.94 for western matsutake. Measured the way most tutorials would have you measure it, the number is 0.99. Measured the way a forager would, it is 0.60. All three numbers come from the same model and the same data. Only the question changed.

This piece is about why the first two numbers are lies, what the honest number costs to compute, and what the gap looks like across twenty species when you actually go and measure it. The short version: on spatial data, random k-fold cross-validation does not test prediction. It tests memory.

What a random fold actually measures

The training set is 24,266 presence records pooled from iNaturalist, GBIF, Mushroom Observer, and iDigBio. Before any modelling, I measured how close those records sit to each other. The median distance from an observation to its nearest neighbour is 137 metres. Eighty-one percent of records have another record within one kilometre, and ninety-seven percent have one within five. Six percent share their coordinates exactly with another record.

Now picture a random 80/20 split. For almost every held-out point there is a training point a few hundred metres away, drawn from the same slope, the same soil, the same 10-metre satellite embedding, very likely the same afternoon and the same person. A model does not need to have learned anything about mushroom habitat to score that point correctly. It needs to have memorised its neighbour, and gradient-boosted trees are extremely good at memorising neighbours.

Spatial autocorrelation is the technical name for this, and it is not a subtle effect at the tails. It is the main effect. The environmental features that make the model useful (elevation, soil moisture, distance to water, satellite texture) are precisely the ones that vary smoothly across space, so any two nearby points share them. Random folds put those pairs on opposite sides of the split and call the resulting score generalisation.

Holding out places instead of rows

The fix is old and simple: partition space, not rows. Every record is assigned to a block on a latitude-longitude lattice, and folds are built from whole blocks, so a test point's neighbours are in the test fold with it. In scikit-learn terms it is GroupKFold with the block id as the group. The whole blocking function is four lines.

def _spatial_blocks(coords, block_deg):
    b = np.floor(coords / block_deg).astype(int)
    return np.array([f"{r}_{c}" for r, c in b])

The interesting question is how big the blocks should be. Too small and you are back to random folds with extra steps. Too big and a four-fold split leaves the model training on three quarters of a state and testing on the other quarter, which is a different and harsher question than the one the map will be asked.

Deriving block size from the data

Rather than pick a number, SporeCast derives one from the empirical variogram of the predictors that carry the most spatial structure: elevation, soil moisture, distance to water, and distance to the two host-tree classes. For each field it standardises the values, bins pairwise distances, computes the semivariance in each bin, and reads off the range, the distance at which semivariance reaches 95 percent of its sill. The block size is the median range across those fields, clamped to a floor and a ceiling of ▓▓▓° so a degenerate variogram cannot produce a block the size of a county or a continent.

Two honest notes on that machinery. First, broad climate fields (annual mean temperature, autumn precipitation) were deliberately removed from the set, because their ranges are so long that they pinned every species to the ceiling and erased the per-species adaptivity the method was meant to add. Second, in the current build every one of the twenty species still lands on the ceiling. The adaptive block size has, for now, degenerated to a constant. It is still the right constant, and the code's own comment on the matter is the one I would write today: the gain is a more conservative, less leakage-inflated estimate, not a measured accuracy jump.

What the gap looks like

There was no random-fold baseline in the project, because I never trusted one enough to run it. For this article I ran it anyway: LightGBM, four folds, the same sixty features and the same class-balanced weighting as the production ensemble, over the cached training frames, for six species spanning the range of sample sizes. The background here is the forested serving grid rather than the regional background, for reasons the next section explains.

The mean gap between random folds and 1.5° blocks is 0.045 AUC, and the pattern underneath it is the important part. The species with thousands of records lose about three hundredths. The two species with a few hundred records lose five to ten. That is exactly what leakage predicts: the thinner the data, the larger the share of a random fold's score that is memorised neighbours rather than learned habitat, so the more the honest number falls when the neighbours are taken away.

Put differently, random cross-validation is most flattering precisely where you most need it to be strict. A rare species is the one whose map you should trust least, and it is the one whose random-fold score looks best relative to reality.

The metric lies too

Blocking the folds fixes half the problem. The other half is what the model is being asked to separate presences from. A species distribution model trains presences against background points drawn from the region, and in the Pacific Northwest the region is mostly places a forager would never look: sagebrush steppe, alpine rock, open water. Against that background, every species scores about 0.97, because the model has learned that mushrooms grow in forests. That is true, and useless. The person opening the map is already standing in a forest.

So the score that ships is computed against the forested part of the serving grid, within reach of known finds, with any cell that holds a find removed. It asks the question the user asks: given that I am in plausible habitat, does this model know where within it to send me? The presences are scored out of fold and the landscape is scored by the final blend, which is the conservative choice.

The median species goes from 0.972 to 0.818. Black morel barely moves, because its habitat signal (recent burns, specific soils) is sharp enough to survive the harder question. Western matsutake goes from 0.940 to 0.596, which is nearly a coin flip inside its own habitat, and it is the species I would most want the map to be good at. That row is why this evaluation exists. Reporting 0.94 for matsutake would not have been wrong on any technicality. It would just have described a model that does not exist.

Two more ways to hold out

Blocked folds test whether the model transfers across space. They do not test whether it transfers across time, or across the quirks of whoever collected the data. SporeCast runs two further holdouts for that.

• Held out by time. Train on the oldest three quarters of dated records, test on the most recent quarter, with the background split disjointly between the two. There is a trap here: foragers revisit patches, so a recent find usually sits within a kilometre of an older find of the same species. The measured median is 1.0 to 1.2 kilometres. Without a spatial exclusion the temporal holdout is just a random fold with a date on it, so test finds within five kilometres of any training find are dropped. At the full block scale nothing survives for any species, which is itself a finding: the data cannot support a fully independent temporal test.
• Held out by platform. Train on iNaturalist alone, test on the other three archives. iNaturalist is 90 percent of the records, so this asks whether the model learned mushrooms or learned iNaturalist users. For one species there are not enough non-iNaturalist records to run it at all, and the model reports that rather than a number.

A single split cannot produce out-of-fold predictions for a stacked meta-learner, so both holdouts score a plain mean of the seven ensemble members rather than the stacked blend that ships. That understates the production model slightly, which is the right direction to be wrong in.

Confidence intervals that respect geography

If the folds are spatial, the uncertainty should be too. A bootstrap that resamples individual records assumes they are independent, which is the exact assumption this whole article is about violating. SporeCast resamples whole blocks with replacement, two hundred times, and reports the 2.5th and 97.5th percentiles of AUC and of the Boyce index. Golden chanterelle, the best-sampled species, comes out at 0.966 to 0.987. The rarer species get intervals wide enough to be embarrassing, which is the point of computing them.

The same logic applies to feature importance. Permutation importance computed in-sample on a tree ensemble is close to meaningless, because a tree that has memorised its training points will report that whatever feature lets it find them again is important. It is computed on held-out folds instead.

The recipe, if you only remember one thing

1. Before you fit anything, compute nearest-neighbour distances between your records. If most of them are small relative to the scale at which your features vary, random folds will lie to you.
2. Block your folds on a spatial lattice, and size the blocks from a variogram of your most spatially structured predictors rather than from habit. Clamp the result, and say so when it hits the clamp.
3. Score against the background your users actually stand in. A high AUC against an easy background is a description of the background.
4. Add a temporal and a source holdout, and expect the temporal one to need a spatial exclusion of its own.
5. Bootstrap blocks, not rows, and publish the interval next to the point.

None of this makes the model better. All of it makes the number honest, and an honest 0.82 has done more for this project than a flattering 0.97 ever could, because it told me which species to keep working on. The full system, including the seven-learner ensemble, the effort correction, and the phenology model, is written up in the SporeCast case study. The observer-bias half of the problem, which is arguably the larger one, gets its own article.