Making a long-running bulk job safe to restart

Any job that runs for hours will be interrupted. A deploy, a timeout, a full disk, a supplier going down at 3am, someone closing the wrong terminal. Whether it stops halfway is not really the question. What matters is what happens when you start it again.

We hit this while building a pipeline that pulled a few hundred thousand products, and their images, from a supplier catalog. The download stage was the fragile part. These are the patterns that made it boring.

Record progress per item, not per run

The cheapest change with the highest payoff: give every record a flag saying whether its work is done. In our case that was a boolean on the product row indicating its images had been fetched.

On start, the job selects only the records where that flag is false. A restart then resumes on its own. No offsets to track, no cursor file to corrupt, no assumption that the source ordering stays stable between runs. If the process dies after 180,000 items, the next run picks up at 180,001 without being told anything.

Set the flag after the work, never before

Order matters here more than it looks. Write the file, verify it landed, then mark the record complete. Mark it first and a crash in between leaves a record claiming to be done when it isn't, and because your resume query now skips that record, the gap is both permanent and invisible.

Getting this backwards is the single most common way "restart-safe" jobs quietly lose data.

Bound your concurrency on purpose

Downloading one file at a time is too slow. Firing off a promise per record exhausts sockets, memory or the supplier's patience, usually all three.

We used a semaphore: a fixed number of concurrent slots, with each worker taking a slot, doing its download, and releasing it. The number becomes a tuning dial you can lower when the source starts throttling and raise when it doesn't.

The important property is that it's a ceiling and not a target. Unbounded parallelism doesn't buy you speed. Past a certain point it is slower and less reliable, because the throughput goes on retries.

Stream archives, don't land them

Where the supplier offered bulk image archives, we extracted them as a stream instead of downloading the archive, writing it to disk, unzipping it and cleaning up afterwards. Streaming the extraction removes the temporary file entirely, which means no half-written archives to detect and no disk filling up over a long run.

It also fails better. A truncated stream throws immediately, where a downloaded file looks complete until something tries to open it.

Make the job idempotent end to end

• Writing the same file twice should be harmless. Overwrite deterministically instead of appending or auto-renaming.
• Re-processing a completed record is a no-op, never a duplicate row.
• Two runs back to back should leave the system in exactly the state one run would have.
• Apply schema changes through migrations, so a restarted job never meets a table it doesn't recognise.

Once that holds, restarting stops being a decision. You just run it again.

Separate entrypoints, one config

As the pipeline grew we split it into independent modules: authentication, crawling, ingestion, image fetching, each runnable on its own and sharing one central configuration.

That structure means you can re-run only the failed stage instead of the whole chain. When the image download breaks, you fix it and run the image module. You don't re-crawl a catalog you already have.

Watch for silent stalls, not just crashes

A job that dies loudly is the easy case. The process exits, the deploy tool or cron wrapper notices, and you get a restart. The dangerous failure is a job that is still running but has stopped making progress, stuck waiting on a socket that will never respond or retrying an item that will never succeed. Track a "last progress" timestamp alongside your completion flag, and alert when it hasn't moved in longer than one worker cycle should ever take. A restart-safe job you can't tell is stuck will quietly waste the whole runtime window until something else notices.

Prove restart-safety by testing it, not by hoping

We treat "kill it and restart" as a real test case, not a mental exercise. Before trusting a new stage in production, we run it against a representative slice of data and kill the process at a handful of different points mid-run: during a download, during the completion write, between two records. Then we compare the end state against a clean, uninterrupted run. Any mismatch is a bug to fix before the job ever meets a full dataset, whether that's a missing item, a duplicate, or a record stuck flagged-in-progress forever.

Idempotency has to reach past your own database

Marking a record complete is only half of restart-safety when the work itself has a side effect outside your system. Sending a notification email. Calling a third-party webhook. Pushing a message to a queue another team consumes. A crash between "external call sent" and "flag set" means the retry will send that call again. Where the downstream system supports it, pass an idempotency key so a duplicate call is a no-op on their end too. Where it doesn't, treat side effects that can't be made idempotent as the one place you're willing to accept an occasional duplicate, and say so explicitly instead of assuming the flag alone covers it.

When does a boolean flag stop being enough?

A single "done" column is the right starting point. It strains once a stage needs retry counts, backoff, or a distinction between "hasn't run yet" and "failed three times and needs a human." That's the moment to graduate from a flag on the row to a proper job queue, with per-item retry counts, a dead-letter destination for items that keep failing, and visibility into how many are stuck versus merely queued. Do it when the flag stops answering your questions, not preemptively. The flag-on-a-row pattern handles most bulk jobs for their entire lifetime without ever needing to graduate.

The goal isn't a job that never fails. It's a job where failing costs you the last few seconds of work instead of the last four hours.

A quick self-test

Take any long-running job you maintain and kill it at random, mid-run. Then start it again. If you have to check anything, clean anything up, or reason about where it stopped, it isn't restart-safe yet. You'll find that out at the worst possible moment instead of on your own terms.

How do you make a long-running batch job safe to restart?

Track completion per record, not per run, using a flag the job checks before starting work. Set that flag only after the work is verified done, never before, and bound concurrency with a fixed-size semaphore instead of unlimited parallelism. Once every step is idempotent, restarting stops being a decision. You just run it again.