Ingesting a seven-million-record catalog, twice a day

A wholesale book distributor's catalog runs to over seven million titles. The requirement was to publish that catalog across several international marketplaces, keep inventory and pricing current, and remove discontinued items - continuously, and without someone babysitting it.

The interesting problem here isn't the volume. It's that the volume removes every shortcut you'd normally take.

Why the obvious approach fails

The instinctive design is one job: read the feed, transform each row, push it to the marketplaces. At a few thousand records that works fine. At seven million it breaks in four separate ways.

• Memory - you cannot hold the feed in memory, so every step has to stream.
• Time - a single pass takes long enough that failures mid-run are certain, not hypothetical.
• Coupling - if transformation and publishing share a process, a marketplace rate limit stalls your ingestion.
• Recovery - when something breaks at record five million, you need to resume, not restart.

Stage 1 - Ingest, and nothing else

The first stage reads the vendor CSV feed and does almost no thinking. It streams the file, validates that each row is structurally sound, and writes it into a staging database as close to raw as possible.

Resisting the urge to transform here is the whole point. Staging gives you a checkpoint: if enrichment logic changes next month, you re-run from staging instead of re-downloading seven million records. It also gives you somewhere to look when a downstream value seems wrong - you can compare against what the vendor actually sent.

Stage 2 - Transform and enrich

Vendor data is never in your shape, and it's rarely complete. This stage normalises the raw rows into an internal product schema, then fills the gaps by calling a third-party bibliographic API for records missing key metadata.

Two rules kept this manageable. Enrichment is idempotent - running it twice on the same record produces the same result, so a partial run is safe to repeat. And enrichment failures are non-fatal: a product missing an optional field still moves forward, flagged, rather than blocking the batch.

Stage 3 - Categorisation, handled by machine

Every marketplace wants products mapped into its own category tree, with its own required attributes. Doing that by hand across millions of titles isn't a staffing problem, it's an impossibility.

We ran this as a queue-backed processing stage: jobs distributed through a Redis-backed queue, with a Python service using sentence-embedding models to predict the right category and generate marketplace attributes from the product's text. Semantic similarity handles the long tail far better than keyword rules, which fall apart the moment a title doesn't contain the obvious word.

Queueing it separately matters as much as the model does. Categorisation is the slowest step, so it needs to scale horizontally - add workers, not patience - without holding up ingestion.

Stage 4 - Distribute

The final stage publishes processed products outward and keeps listings, inventory, pricing and deletions in sync. Each destination gets its own workers with its own rate limits and its own retry state, so one slow or unhappy marketplace degrades only its own channel.

Two schedules, not one

The catalog runs on two different rhythms, and conflating them is a mistake we've seen elsewhere.

1. A full refresh - the entire catalog, re-ingested and re-processed, capturing new titles and structural changes.
2. A lighter, frequent pass - every few hours - carrying only price and availability changes, plus additions and discontinuations.

Most of what changes day to day is price and stock. Those are small, cheap updates. Rebuilding seven million full product records to propagate a price change wastes hours and adds risk for no benefit.

What we'd tell anyone starting this

• Stage your raw data before you touch it - the checkpoint pays for itself the first time requirements change.
• Make every stage idempotent, so a retry is always safe.
• Put a queue between anything slow and anything fast.
• Separate your full rebuild from your incremental updates.
• Assume mid-run failure is normal and design for resume, not restart.

None of these are exotic techniques. The discipline is applying them from the start - because at this size, retrofitting them means rewriting the pipeline.

How do you sync a multi-million-record product catalog to several marketplaces?

Split the pipeline into independently restartable stages - ingest, transform and enrich, categorise, distribute - each streaming data rather than holding it in memory, and each idempotent so a crash only costs the current stage. Run a full refresh on one schedule and a lighter, more frequent price and stock sync on another.