Designing around third-party rate limits before they design around you

Every integration we've built against a marketplace or supplier API has a rate limit, and no two are the same. Requests per second, per minute, per day, sometimes a concurrency cap sitting on top of all of it. The teams that struggle are the ones that treat the limit as an error to catch. The ones that don't treat it as a fixed input to the design, like disk space or memory.

Catching 429s is the fallback, not the plan

By the time you react to a `429 Too Many Requests`, you have already sent too much. The provider is now refusing you, and depending on how strict they are, the block may outlast the single request that tripped it. Backoff-on-429 has to exist. As your only mechanism, though, it collapses throughput under load, which is precisely when you need throughput.

Meter on your side, under the real limit

Put a limiter in front of the client: a token bucket or leaky bucket sized to something under the published rate, so you're never riding the ceiling. Every outbound call takes a token. No token, the call waits. Your request rate becomes a property you control and can reason about, instead of an emergent result of however many workers happen to be running.

One limiter per provider, shared across workers

• The limit applies per API key or account, so the limiter has to be shared state. Redis, or a single gateway process. Never a per-worker counter that ten workers each think they own.
• Track the different windows the provider enforces, per-second and per-day alike, in the same limiter.
• Holding multiple keys? Key the limiter by credential, and let work spread across them.

Prioritise, because not all calls are equal

When you're limit-bound, what you spend the budget on matters. A customer's order pull or a stock zero-out is worth more than the hourly catalogue refresh. Run separate queues with weights, or a priority field, so time-sensitive work goes first and bulk work fills whatever capacity is left over.

Spread scheduled work out

Cron jobs that all fire at the top of the hour create a spike, trip the limit, then sit idle for 59 minutes. Jitter the start times, chunk large jobs, stretch them across their available window. Smooth and slow finishes ahead of spiky and blocked.

Back off properly when you do get limited

• Honour the `Retry-After` header when the provider sends one. That is them telling you exactly how long to wait.
• Otherwise use exponential backoff with jitter, so a fleet of workers doesn't retry in lockstep and re-trip the limit together.
• Feed the event back into the limiter. Every call to that provider should slow down, not only the one that failed.

The Timestamp Converter helps when you're lining up `Retry-After` values and rate-limit reset headers against your own logs. The JSON Formatter makes a provider's error body readable while you work out which window you hit.

Test the limiter without hammering the real API

You can't validate backoff and queueing behaviour by running it against the live provider and hoping to get rate-limited on demand. We build a stub of the provider that returns 429s and `Retry-After` headers on a schedule we control, then drive the limiter against that. Burst it, starve it, feed it a mixed-priority queue. The scheduling logic gets exercised deterministically long before it meets a real account's real limit. The integration itself still gets tested against the provider's sandbox where one exists, but that is for confirming the request format, not for proving the throttling logic works.

Watch the limiter, not just the 429 count

A 429 count close to zero looks like success. It can just as easily mean the limiter is set so conservatively that real throughput is being left on the table. We track queue depth and wait time per provider alongside the 429 rate, so a growing backlog shows up as a metric before it shows up as a complaint about a stock update that took too long to reach a channel. Wait time that trends upward over days, instead of spiking with traffic, usually means the published limit and the account's real limit have drifted apart. It is not a traffic problem.

A concurrency cap is a different constraint

Some providers cap requests per second. Others cap how many requests can be in flight at once regardless of rate, which has to be modelled separately, because a slow endpoint can exhaust a concurrency cap at a request rate nowhere near the documented per-second number. We size a semaphore for in-flight requests separately from the rate-limiting token bucket. A limiter that only tracks rate will happily fire many concurrent slow requests, get capacity-limited anyway, and misreport it as an unexplained rate-limit hit.

Retries need an idempotency key, or backoff just moves the bug

A request that times out waiting on a rate-limited queue and then gets retried was not necessarily one that never reached the provider. It may have been accepted, with only the response lost. Retrying a stock update is safe, since the second write just repeats the same value. Retrying an order-creation or payment-adjustment call without an idempotency key can create a duplicate. We attach an idempotency key to any retried call with a side effect beyond overwriting a value, which lets backoff logic stay aggressive without turning a slow response into a duplicate order.

When one limiter stops being enough

A single in-process token bucket is fine until more than one service or deploy needs to share the same provider account's budget. At that point the limiter itself has to move into shared state, and not stay bolted onto whichever service happened to own the integration first. We've moved a per-worker limiter into a small dedicated gateway service more than once, whenever enough independent jobs needed the same provider account. Every call routes through it. It owns the token bucket and the priority queue, and the callers just wait on it instead of each re-implementing their own throttling.

A rate limit you designed for is a scheduling parameter. A rate limit you react to is an outage waiting for a busy day.

What's the best way to handle API rate limits?

Meter your own requests below the published limit with a shared token-bucket limiter, instead of waiting for 429 responses. Give each provider account its own limiter, prioritise time-sensitive calls over bulk work, and spread scheduled jobs across their window instead of firing them all at once. Keep backoff-with-jitter and Retry-After handling as the fallback.