Making a long-running bulk job safe to restart

Any job that runs for hours will be interrupted. A deploy, a timeout, a full disk, a supplier going down at 3am, someone closing the wrong terminal. The question isn't whether it stops halfway - it's what happens when you start it again.

We hit this while building a pipeline that pulled a few hundred thousand products, and their images, from a supplier catalog. The download stage was the fragile part. These are the patterns that made it boring.

1. Record progress per item, not per run

The cheapest and highest-value change: give every record a flag that says whether its work is done. In our case a simple boolean on the product row indicating its images had been fetched.

On start, the job selects only the records where that flag is false. A restart then naturally resumes: no offsets to track, no cursor file to corrupt, no assumption that the source ordering is stable between runs. If the process dies after 180,000 items, the next run picks up 180,001 without being told anything.

2. Set the flag after the work, never before

Order matters more than it looks. Write the file, verify it landed, then mark the record complete. If you mark first, a crash in between leaves a record that claims to be done and isn't - and because your resume query now skips it, that gap is permanent and invisible.

Getting this backwards is the single most common way "restart-safe" jobs quietly lose data.

3. Bound your concurrency deliberately

Downloading one file at a time is too slow. Firing off a promise per record exhausts sockets, memory or the supplier's patience - usually all three.

We used a semaphore: a fixed number of concurrent slots, with each worker taking a slot, doing its download, and releasing it. The number becomes a tuning dial you can lower when the source starts throttling and raise when it doesn't.

The important property is that it's a ceiling, not a target. Unbounded parallelism isn't faster - past a certain point it's slower and less reliable, because you spend the throughput on retries.

4. Stream archives, don't land them

Where the supplier offered bulk image archives, we extracted them as a stream rather than downloading the archive, writing it to disk, unzipping it, and cleaning up. Streaming the extraction removes the temporary file entirely - which means no half-written archives to detect and no disk filling up on a long run.

It also fails better. A truncated stream throws immediately, rather than producing a file that looks complete until something tries to open it.

5. Make the job idempotent end to end

• Writing the same file twice should be harmless - overwrite deterministically rather than appending or auto-renaming.
• Re-processing a completed record should be a no-op, not a duplicate row.
• Running the whole job twice in a row should leave the system in exactly the state one run would have.
• Schema changes should be applied through migrations, so a restarted job never meets a table it doesn't recognise.

Once that holds, restarting stops being a decision. You just run it again.

6. Separate entrypoints, one config

As the pipeline grew we split it into independent modules - authentication, crawling, ingestion, image fetching - each runnable on its own, sharing one central configuration.

That structure means you can re-run only the failed stage instead of the whole chain. When the image download breaks, you fix it and run the image module. You don't re-crawl a catalog you already have.

The goal isn't a job that never fails. It's a job where failing costs you the last few seconds of work rather than the last four hours.

A quick self-test

Take any long-running job you maintain and kill it at random, mid-run. Then start it again. If you have to check anything, clean anything up, or reason about where it stopped, it isn't restart-safe yet - and you'll find that out at the worst possible moment instead of on your own terms.

How do you make a long-running batch job safe to restart?

Track completion per record, not per run, using a flag the job checks before starting work. Set that flag only after the work is verified done, never before, and bound concurrency with a fixed-size semaphore rather than unlimited parallelism. Once every step is idempotent, restarting stops being a decision - you just run it again.