Automating Stock Image Generation End to End

Generating one good stock image with a model takes a minute. Generating a thousand overnight, unattended, with every one of them legal to upload, the right size, correctly labelled, and costed to the cent, is a different job. This is the automation layer of my stock-image pipeline: the loop that runs while I sleep and the checks that stop it from producing a night of waste.

The strategy behind the pipeline, sampling keywords by search demand and treating metadata as the product, is in the companion article. Here I want to stay on the operational side, because it is where most of the engineering hours went and where a pipeline earns the word “automated”.

The batch loop

Every keyword the sampler draws becomes one coroutine, and all of them are scheduled at once. Backpressure comes entirely from two gates that sit in front of every model call: a token bucket that caps requests per second with a small burst allowance, and a semaphore that caps how many calls are in flight. The bucket is acquired before the semaphore, so the rate limit is global and the concurrency limit nests inside it. Nothing is chunked by hand.

class TokenBucket:
    async def acquire(self):
        async with self._lock:
            now = time.monotonic()
            self.tokens = min(self.burst, self.tokens + (now - self.last) * self.rate)
            self.last = now
            if self.tokens < 1.0:
                await asyncio.sleep((1.0 - self.tokens) / self.rate)
                self.tokens = 0.0
            else:
                self.tokens -= 1.0

Retries, and which errors deserve one

Model APIs fail in two ways: transiently (rate limits, timeouts, connection resets) and permanently (a malformed request, a policy refusal). Only the first kind is retried, with exponential backoff, additive jitter, a cap on the delay, and a cap on attempts. Anything outside that allowlist re-raises immediately, because retrying a permanent error five times is a slow way of failing. One small touch that turned out to matter: a success decrements the consecutive-failure counter rather than resetting it, so a flapping upstream does not get treated as healthy after one lucky call.

Skip and continue, at four levels

1. A single variant call fails: that variant is dropped, the others are kept.
2. A candidate fails validation or brand safety: it is logged and dropped.
3. No candidate survives: the keyword is abandoned and the batch moves on.
4. At the top, the whole batch is gathered with exceptions captured rather than raised, so one keyword's failure can never kill the run.

For an unattended generator that is the difference between a partial batch in the morning and a wasted night. It also has a cost, which the run summaries make visible and which I come back to below.

The gates every asset passes

• Schema. The chat model is called with a strict JSON schema (no additional properties, required fields) and a single self-correction retry that appends a corrective note to the system prompt if the first answer does not parse. A balanced-brace scan recovers a JSON object from chatter when the model narrates around it.
• Brand safety. A set of regexes runs over the prompt and the keywords together: consumer brands, franchises, artists' names, “in the style of”, trademark symbols. Any hit kills the variant, because a marketplace rejection costs more than a regeneration.
• Deduplication before spending. A short hash over the prompt, title, and keywords is checked against the run's set under a lock before the image is requested, so a duplicate costs nothing.
• Marketplace limits. Title length and keyword count are enforced in code, not requested in the prompt. The title is cut at a word boundary; keywords are Unicode-normalised, deduplicated, padded to the floor, and trimmed at the cap.
• Megapixels. The marketplace requires four megapixels, and a 16:9 render from the image models lands just under it (2560 by 1440 is 3.69 megapixels). Every asset is upscaled with Lanczos to the smallest size that clears the floor and saved as a high-quality JPEG with no chroma subsampling, in a thread so the event loop keeps going.

Unit economics as an output of every run

The pipeline reports its own cost at the end of each run: tokens in and out, image calls, p95 latency, duplicates, regenerations, cost per image, and a projection table for larger batches. That report is what drove the model choices, because it made one fact impossible to ignore: the image model is 96 percent of the bill and the language model is a rounding error.

Every model is a configuration entry rather than a literal, and each stage picks its own tier, so the migration from one image model to the next was a flag change. That cuts both ways: the prompt guidelines were tuned for one model and inherited by the next without being re-versioned, and the version string in the script still names the old one. The prompt layer outlived the model it was written for, which is a mistake I now check for explicitly.

The failure the artifacts did not show

The run summaries also revealed a number the output folders never would have. About a quarter of the keywords sampled in every run produced nothing. Not duplicates, not JSON failures, not brand-safety hits; those counters were zero. The cause was the keyword count bound: a small language model asked for 45 to 60 keywords under-delivers roughly one time in four, and the validator rejected the row before the padding logic that could have repaired it ever ran. The pipeline was throwing away work it knew how to fix.

Two lessons from that. Instrument the loss, not just the output, because an unattended pipeline's failures are silent by construction. And put normalisation before validation when the normaliser can repair what the validator rejects. Neither is specific to images.

Durability and the human at the end

Two writers own the two output files, each draining an async queue in its own task, so there are no write locks and no interleaved rows. A pending journal is written up front and a completion journal is appended one line per finished asset; resuming a run is a set difference between the two, with no in-place rewrites of anything. A signal handler sets a stop event that every coroutine checks at entry, so an interrupted run finishes its in-flight work and flushes cleanly.

The last step is deliberately not automated. The pipeline produces an upload manifest and a folder of images, and a small desktop gallery lets me page through a run, read each asset's metadata, and delete the bad ones, rewriting the manifest so it stays consistent. A fully autonomous loop between a generative model and a live storefront is a reputational risk with no upside; reviewing a batch costs minutes and prevents the category of mistake that closes contributor accounts.

The economics, the demand model, and the reasons behind the design are in the AI Stock Image Factory case study, and the keyword side of the pipeline has its own article on keyword optimization for Adobe Stock. If you build one of these, start with the run summary and the TokenBucket. The model is the easy part.