A rate limit is a promise in two directions. To you, it promises that one client cannot take down your origin. To the client, it promises a rule they can plan around. Most limiters keep the first promise and quietly break the second: the documented limit says 100 requests a minute, and the client gets throttled at 63 with no hint about when to come back.
Fixed windows burst
The simplest limiter counts requests per key per clock minute and resets at the top of the minute. It is cheap and easy to explain, and it allows double the limit across a boundary. A client can send 100 requests at 12:00:59 and another 100 at 12:01:00, which is 200 in two seconds against a limit that says 100 a minute.
That burst is exactly the traffic shape that hurts an origin, so fixed windows are fine for coarse daily quotas and a poor fit for protection.
Pick the algorithm by the promise
| Algorithm | Good for | Trade-off |
|---|---|---|
| Fixed window | Daily and monthly quotas | Up to 2× bursts at the boundary |
| Sliding window | Per-minute API limits | Slightly more state per key |
| Token bucket | Bursty clients, steady average | Harder to explain in docs |
A sliding window weights the previous window by how much of it still overlaps the last sixty seconds, which removes the boundary burst for the cost of one extra counter. A token bucket refills at a steady rate and lets a client spend saved-up tokens at once, which suits SDKs that batch work. We default to sliding windows because they match what people expect when they read “100 per minute”.
Where you count
Counting at the origin means abusive traffic has already crossed the network and hit your load balancer by the time you reject it. Counting at the edge rejects it in the region it came from. The catch is that a client can hit several regions, and a single global counter would add a cross-region round trip to every request.
Halyard keeps a local counter in each region and reconciles them continuously in the background. Each region also gets a share of the limit based on where that key's traffic has recently come from. The result is a bounded overshoot: in the worst case a client spread evenly across regions gets a few percent over its limit for a few seconds. We think that is the right trade, and we document it rather than pretending the counter is exact.
Tell the client the truth
A bare 429 forces the client to guess, and guessing clients retry in tight loops, which is the traffic you were trying to stop. Send the limit, what is left and when to retry on every response, not only on rejections.
HTTP/1.1 429 Too Many Requests
RateLimit-Policy: "per-key";q=100;w=60
RateLimit: "per-key";r=0;t=18
Retry-After: 18
Content-Type: application/json
{ "error": "rate_limited", "retry_in_seconds": 18 }These headers follow the IETF RateLimit fields draft, so well-behaved HTTP clients can back off without reading your docs. Halyard adds them by default, and rejected requests are never billed.
Roll out in shadow mode
The riskiest moment for a new limit is the deploy. You picked 100 a minute from a dashboard, and it turns out a partner's nightly sync sends 140. Shadow mode evaluates the new limit on live traffic and records who would have been rejected, without rejecting anyone.
await halyard.limits.update("per-key", {
limit: 100,
window: "60s",
mode: "shadow", // log would-be 429s, block nothing
});Run it for a few days, look at the keys that would have tripped it, talk to those customers, then switch the mode to enforce. A limit that rolls out that way protects the origin and keeps its promise to the client.