Correcting Observer Bias in Citizen-Science ML Datasets

A citizen-science dataset is a map of where people went, annotated with what they saw. Train a model on it naively and it learns the first thing, not the second. SporeCast is built on 24,266 mushroom sightings from four public archives, and most of the modelling effort went into being honest about who collected them.

The problem has a name in the species-distribution literature, sampling bias, and a standard set of remedies. What I want to do here is show what the bias looks like in an actual dataset, with numbers, and then walk through the four corrections SporeCast applies, including the one that produced a measurable improvement and the one whose effect I still cannot separate from the ecology.

Where people look

Ninety percent of the records are from iNaturalist. A few percent each come from iDigBio (digitised herbarium specimens), GBIF with the iNaturalist feed excluded, and Mushroom Observer. The archives do not just differ in volume; they differ in who is behind them. A herbarium record is a specimen someone vouchered on purpose, often decades ago. An iNaturalist record is a phone photo from a walk. The same species can be common in one archive and absent from another for reasons that have nothing to do with where it grows.

The first thing to measure is clustering, because it is the fingerprint of effort. If sightings were spread according to habitat they would be spread across a lot of habitat. They are not.

On a 1.1-kilometre lattice, the top tenth of occupied cells hold 41 percent of all records; on a 7-kilometre lattice, over half. That is trailheads, popular loops, and a few people's favourite patches. It is also, inconveniently, partly real: good habitat does cluster, and some of those patches are famous precisely because they are productive. There is no clean way to separate the two from the sightings alone. What you can do is stop the model from being rewarded for the part that is effort.

The other axis: when people look

Effort is not just spatial. The whole all-fungi observation record for the region peaks in October and November at four times the mean month, because that is when people go mushroom hunting. A species' monthly sighting curve is therefore the product of two things: when it fruits, and when anyone was outside to notice.

Nothing in the target curve tells you whether a species fruits in November or whether November is simply when the observers are out. The correction is to divide each species' monthly counts by the all-fungi effort curve before smoothing, with a floor so an empty winter month cannot blow up the ratio. The result is closer to when the fungus fruits than to when the foragers do, which matters because the served forecast shifts that curve with the current season's soil moisture, and a curve that was really an effort curve would shift the wrong thing.

Correction one: a background that knows where people went

A presence-only model needs something to contrast presences against. The naive choice is background points sampled uniformly across the region. Against that background, a sighting near a trailhead looks like evidence that trailheads are habitat, because uniform background is thin near trailheads and dense in the backcountry nobody visited.

The standard fix is target-group background: weight background points by how much observation effort the area received, using sightings of the whole taxonomic group as the proxy. SporeCast uses every fungus observation in the region, not just the twenty target species, because that is a far denser and less circular measure of where fungus-noticing people have been. Each background cell's weight is a Gaussian-decayed count of effort points within a few bandwidths, with a bandwidth of ▓▓▓km and a floor of ▓▓▓% so genuinely unvisited habitat still counts for something.

idx_lists = tree.query_ball_point(gx, r=3 * bandwidth_km)
eff = np.zeros(len(grid_coords))
for i, idxs in enumerate(idx_lists):
    if idxs:
        d = np.linalg.norm(tx[idxs] - gx[i], axis=1)
        eff[i] = np.exp(-(d / bandwidth_km) ** 2).sum()
return floor + (1.0 - floor) * (eff / eff.max())

The effect is that a presence near a trailhead is now contrasted with background that is also near trailheads. The model can no longer earn credit for learning the road network, because the road network is on both sides of the comparison.

Correction two: make the comparison hard on purpose

Effort weighting removes one bias and leaves a bigger one in place. Most of the Pacific Northwest is not forest, and a model asked to separate forest mushrooms from a regional background will learn forest-versus-not and stop, scoring 0.99 on the way. So a large share of the total background weight, ▓▓▓%, is concentrated on forested cells within ▓▓▓km of known finds, with the total preserved so class balance downstream does not move. The model is forced to discriminate within habitat, which is the only discrimination the map needs.

This is the correction I can put a number on. In a four-species A/B with the hard stratum switched on and off, within-habitat AUC improved for four species out of four, by a mean of 0.014. Judged by a full-background metric the same change looks like a regression of 0.13 in Boyce index, which is the trap described in the companion piece on cross-validation: a metric that measures the easy split will punish you for getting better at the hard one.

Correction three: throw away almost half the data

Reweighting the background handles where the model looks. It does not handle the fact that fifty sightings from one famous chanterelle patch are not fifty independent pieces of evidence. Spatial thinning addresses that directly: no two retained presences may sit within ▓▓▓km of each other. Records are kept greedily, highest identification confidence first, so when a cluster is collapsed the point that survives is the best-vouchered one.

The cost is stark. Across the twenty modelled species, thinning takes the training set from 21,880 records to 12,479, a 43 percent reduction. Golden chanterelle loses 46 percent of its records, lobster mushroom 53 percent. It is uncomfortable to delete that much data, and it is correct: the deleted records were not telling the model anything the surviving one did not, except that someone had been back.

A small engineering note, because it bit me: the obvious implementation rebuilds a spatial index after every kept point and is quadratic, which was fine at a coarse thinning distance and fell over as the distance shrank. The current version hashes points onto a lattice at the thinning distance and checks the nine surrounding cells, which is linear and made experimenting with the distance cheap enough to actually do.

Correction four: distrust the record itself

Before any of the above, the records are cleaned, and the cleaning is also a bias correction, because the errors are not random. The two that matter most are specific to how citizen scientists behave.

• Coarse coordinates. Any record with a reported positional accuracy worse than a kilometre is dropped. Records with no reported accuracy are kept, which is a judgement call: the herbarium archives rarely report one and dropping them would silently remove the oldest, best-vouchered records.
• The “came back to Seattle and posted” problem. Records that land on open water, or in developed land cover, are dropped for forest-obligate species, because a chanterelle logged in a city is a find that was uploaded from home. The urban filter deliberately excludes the open-space developed class, which legitimately hosts edge-of-forest fungi, and exempts the three species (black morel, oyster, lion's mane) that really do grow in town.

Two smaller pieces of the same idea. Identification confidence is not a filter but a weight: an iNaturalist record with several agreeing identifiers, or a vouchered specimen, trains at full weight, while a bare research-grade record trains at a floor. And de-duplication across archives is necessary because iNaturalist records flow into GBIF; the pipeline excludes the iNaturalist dataset from the GBIF pull entirely and then de-duplicates on species, day, and rounded coordinates for whatever slips through.

What is left when you are done

The corrections do not produce a model with more skill. They produce a model whose skill is about mushrooms. Within-habitat AUC for the median species is 0.82, which is a modest number and a real one, and the map it produces sends people into forest they have not been to rather than back to the trailheads they already know. That second property is the entire value of a forecast built from citizen science, and it is the property a naive model is guaranteed not to have, because the trailheads are where its evidence is.

If you take one method from this: use the whole taxonomic group as your effort proxy, not the species you are modelling. It is the difference between a correction and a tautology. The rest of the system, from the seven-learner ensemble to the weather-shifted phenology, is documented in the SporeCast case study, and the code behind the_effort_weights snippet above is the reference implementation I would point anyone at.