The API that returned 200 OK and lost our data anyway

This is the bug we bring up most often when someone asks why integration work takes longer than it looks. It cost real data, it produced no errors, and nothing in the logs was red.

The symptom

We were running a sync pipeline against a large cross-border supplier's B2B API - authenticate, crawl categories, pull product records, write them into the database. It worked. Then, on longer runs, the final product counts started coming in lower than expected. Not zero. Just short.

There were no exceptions. No failed requests. No non-200 responses. Every monitoring signal we had said the run completed successfully. The only evidence anything was wrong was a number at the end that didn't match what we knew was in the source catalog.

What was actually happening

The supplier's API rate-limited us. That part was expected - most do. What was not expected was how it communicated that.

Instead of returning 429 Too Many Requests, or a 503, or an error object with a retry-after header, it returned HTTP 200 with an empty body. As far as our code was concerned, the request had succeeded and the category simply contained no products. So we recorded zero results, moved on to the next page, and never retried.

Why the usual defences didn't catch it

• Status-code checks passed - 200 is 200.
• try/catch caught nothing, because nothing threw.
• Retry logic never fired, because there was no failure to retry.
• Row counts went down, not to zero, so no alarm threshold tripped.
• Re-running a single failing request by hand usually worked, because on its own it wasn't being throttled.

That last point is what makes this class of bug expensive. It only appears under load, and it disappears the moment you try to reproduce it in isolation.

The fix

The principle we settled on: validate the shape of the response, not just the status of the response.

1. Define what a valid response looks like for each endpoint - not just "parsed without error", but "contains the fields we actually need".
2. Treat a structurally empty response to a request that should return data as a failure, regardless of status code.
3. Route those failures into the same retry-with-backoff path as a real network error.
4. Log the distinction, so a throttled-and-recovered request looks different in the logs from a genuinely empty category.
5. Track expected-versus-received counts per run, so a shortfall surfaces as an alert instead of a number nobody reads.

The retry itself was ordinary exponential backoff. The hard part was never the recovery mechanism - it was noticing there was something to recover from.

The wider lesson

HTTP status codes are a convention, not a guarantee. Plenty of third-party APIs - especially older ones, regional ones, and ones not primarily designed for public integration - signal problems in ways that don't map onto the spec. We've seen success codes wrapping error payloads, XML errors returned with a JSON content type, and, as here, throttling expressed as silence.

If you're integrating an API you don't control, assume its error handling is different from what the docs imply, and write your client so that "I got nothing back" is a case you handle explicitly rather than a case you accidentally accept.

A pipeline that fails loudly is a bug. A pipeline that fails silently is a liability - you don't find out until someone downstream asks why the numbers are wrong.

If you're running a sync you don't fully trust

Two checks worth adding this week. First, reconcile counts: whatever your source says it has, compare it to what you actually stored, every run. Second, add a deliberate assertion that a successful response contains the data it's supposed to contain - and make the failure of that assertion noisy.

Neither takes long to build. Both would have caught this on day one.

Why did an API returning 200 OK still lose our data?

The supplier's API responded to rate-limiting with HTTP 200 and an empty body instead of an error code, so status-code checks and try/catch both passed cleanly while records silently went missing. The fix was validating the shape of every response, not just its status, before treating it as successful.