The API that returned 200 OK and lost our data anyway

This is the bug we bring up most often when someone asks why integration work takes longer than it looks. It cost real data. It produced no errors, and nothing in the logs was red.

The symptom

We were running a sync pipeline against a large cross-border supplier's B2B API: authenticate, crawl categories, pull product records, write them into the database. It worked. Then, on longer runs, the final product counts started coming in lower than expected. Not zero. Just short.

There were no exceptions. No failed requests. No non-200 responses. Every monitoring signal we had said the run completed successfully. The only evidence that anything was wrong was a number at the end that didn't match what we knew was in the source catalog.

What was happening

The supplier's API rate-limited us. That part was expected, since most do. How it communicated the throttling was not.

Instead of returning 429 Too Many Requests, or a 503, or an error object with a retry-after header, it returned HTTP 200 with an empty body. As far as our code was concerned the request had succeeded and the category simply contained no products. So we recorded zero results, moved on to the next page, and never retried.

Why the usual defences didn't catch it

• Status-code checks passed. 200 is 200.
• Nothing threw, so try/catch caught nothing.
• Retry logic never fired, because there was no failure to retry.
• Row counts went down but not to zero, so no alarm threshold tripped.
• Re-running a single failing request by hand usually worked, because on its own it wasn't being throttled.

That last point is what makes this class of bug expensive. It only appears under load, and it disappears the moment you try to reproduce it in isolation.

The fix

The principle we settled on: validate the shape of the response, not just its status.

1. Define what a valid response looks like for each endpoint. "Parsed without error" is not the bar; "contains the fields we need" is.
2. A structurally empty response to a request that should return data counts as a failure, whatever the status code says.
3. Route those failures into the same retry-with-backoff path as a real network error.
4. Log the distinction, so a throttled-and-recovered request looks different in the logs from an empty category.
5. Track expected-versus-received counts per run, so a shortfall surfaces as an alert instead of a number nobody reads.

The retry itself was ordinary exponential backoff. Recovery was never the difficult bit. Noticing there was something to recover from was.

The wider lesson

HTTP status codes are a convention, not a guarantee. Plenty of third-party APIs signal problems in ways that don't map onto the spec, especially older ones, regional ones, and ones never primarily designed for public integration. We've seen success codes wrapping error payloads, XML errors returned with a JSON content type, and, as here, throttling expressed as silence.

If you're integrating an API you don't control, assume its error handling differs from what the docs imply. Write your client so that "I got nothing back" is a case you handle on purpose instead of one you accept by accident.

A pipeline that fails loudly is a bug. A pipeline that fails silently is a liability, because you don't find out until someone downstream asks why the numbers are wrong.

Other shapes the same failure takes

Once we started looking for it, the same pattern turned up in other integrations wearing different disguises. Partial writes are one version: a batch endpoint accepts a list of records, returns 200 even when only some of them saved, and buries the failures in a per-item status field nobody was reading. Cursor resets are another. A paginated endpoint silently restarts from page one instead of erroring when a session or auth token expires mid-crawl, so a long-running job re-ingests the start of the catalog instead of continuing where it left off. Then there's the content-type lie: a response declared as JSON that is an HTML error page from a proxy or load balancer sitting in front of the real API, which a lenient parser will happily coerce into an empty object instead of failing to parse.

• Batch endpoints with a top-level 200 and per-item success or failure buried in the body.
• Pagination that resets to page one instead of erroring when a session expires mid-crawl.
• A proxy or load balancer's HTML error page served with a JSON content-type header.
• A field that is present but has quietly changed type or encoding. A price arriving as a formatted string instead of a number parses without error and produces a wrong value forever.

Testing for a failure mode that only shows up under load

You can't reproduce throttling by sending one request by hand, because it only triggers under sustained volume. The assertion itself is testable, though. Feed the parser a response that is structurally valid but semantically empty, shaped the way the real throttled response looked, and confirm it gets treated as a failure instead of a zero-result success. That test needs no access to the real API, only a fixture shaped like the failure you already found.

1. Save a real example of the bad response the first time you find one. It's the most valuable fixture you'll have for this bug.
2. Write a test that feeds that fixture through the parser and asserts it's rejected, not accepted as an empty result.
3. If the supplier offers a staging instance, run a load or soak test against it specifically to reproduce the throttling behaviour on purpose.
4. With no staging environment available, simulate the observed behaviour with a mock server so the retry path gets exercised in CI and not only in production.

What we watch for now

The reconciliation check from the fix section only helps if someone looks at it. So we turned it into a real alert: a per-run comparison of expected against stored counts that pages someone the moment the delta exceeds what we consider normal drift. A report sitting in a dashboard until somebody happens to open it would not have done the job. Just as important, the logs now separate a category that really is empty from one that came back empty after a throttling signal, using the same shape-validation logic as the fix itself. Next time something like this happens it is visible in minutes, instead of being discovered downstream by whoever notices the numbers don't add up.

If you're running a sync you don't fully trust

Two checks worth adding this week. First, reconcile counts: whatever your source says it has, compare it against what you stored, every run. Second, assert that a successful response contains the data it is supposed to contain, and make the failure of that assertion noisy.

Neither takes long to build. Both would have caught this on day one.

Why did an API returning 200 OK still lose our data?

The supplier's API responded to rate-limiting with HTTP 200 and an empty body instead of an error code, so status-code checks and try/catch both passed cleanly while records silently went missing. The fix was validating the shape of every response, not just its status, before treating it as successful.