What Is Rate Limiting, and Does Your Website Need It?

What is rate limiting, and does your website need it?

Rate limiting caps how many requests one client can make in a set window. Your marketing site probably needs it on three things: form submissions, any public API, and site search. Everything else usually does not. Applied too broadly, it blocks search crawlers and costs you far more than the abuse would have.

Most teams meet rate limiting in an emergency. A contact form gets hammered by a bot, a login endpoint gets probed, an AI scraper starts pulling every page twice an hour. Then someone adds a rule in a panic and the rule stays for years.

Here is the calmer version, with the actual specifications and platform behaviour rather than folklore.

What does a rate limit actually do to a request?

It counts requests against a key, and when the count passes your threshold in your window, it applies an action. The key is usually an IP address. The action is usually a block or a challenge.

The correct response when you reject is HTTP 429. MDN describes 429 Too Many Requests as indicating "the client has sent too many requests in a given amount of time," and notes that the specification allows a Retry-After header telling the client how long to wait before trying again.

MDN also notes that implementations vary. Restrictions may be server-wide or per resource, are typically based on the client's IP address, and can be specific to users or authorised applications when requests carry authentication or a cookie. That last option is the one most teams ignore and the one that usually produces the best results.

What should a 429 response actually contain?

A Retry-After header and a human-readable body. MDN's example shows exactly this shape: a 429 status, a Retry-After value of 3600, and a short HTML page saying the client is doing that too often and should try again later.

Retry-After accepts two formats. MDN documents both an HTTP date, such as "Wed, 21 Oct 2015 07:28:00 GMT," and a plain number of seconds, such as 120. The seconds form is easier to get right and easier for clients to handle.

Omitting Retry-After is the most common mistake we see. Without it, a well-behaved client has no idea whether to retry in a second or an hour, so it guesses, usually badly, and your limit produces more traffic rather than less.

Which parts of a marketing site actually need limiting?

Anything that writes. Contact forms, newsletter signups, demo requests, comment fields. These cost you money per submission in spam handling and sales time, and they are the classic bot target. Our piece on stopping form spam covers the layered approach.

Anything that costs you compute per request. Site search, especially AI-powered search, and any endpoint that calls a model or a third-party API on your behalf. Here a limit is cost control as much as security.

Anything that authenticates. Login, password reset, magic link requests. These need limits per account as well as per IP, because a credential-stuffing attack spread across many IPs looks like normal traffic to an IP-based rule.

How do the counting options actually work?

Cloudflare's documentation is a useful reference here because it exposes the choices explicitly. It defines rate limiting rules as letting you "define rate limits for requests matching an expression, and the action to perform when those rate limits are reached."

What you can count by varies by plan. Cloudflare's docs state that the Free tier supports only IP tracking, Pro adds verification of bot status, and Business and Enterprise plans offer IP, IP with NAT support, query, host, headers, cookie, ASN, country, path and JA3 or JA4 fingerprint. Advanced Rate Limiting adds JSON field value, body and form input value.

The windows are similarly tiered. Cloudflare documents supported periods in seconds including 10, 60, 300, 600, 3600 and 86400, and notes maximum periods vary by plan, with Free and Pro capped at one minute and Business and Enterprise reaching from ten minutes up to a day. Knowing your ceiling matters, because a one minute maximum window cannot express "five password resets per hour."

Why is IP-only limiting weaker than it looks?

Because attackers have many IPs and your real users often share one. A corporate office, a university network or a mobile carrier can put hundreds of legitimate people behind a single address. A tight IP rule locks all of them out together.

Meanwhile a distributed bot spreads across thousands of addresses and never trips a per-IP threshold at all. So the rule that annoys your customers is often the same rule that misses the actual attack.

The better pattern is to count by the thing you are protecting. Limit password resets per account, not per IP. Limit form submissions per session and per form. Reserve IP limits for crude volumetric abuse, which they handle well.

How do you avoid blocking search engines?

This is the expensive failure and it is easy to cause. Google's crawl budget documentation states that when a site responds with errors or rate-limiting signals, "the limit goes down and Google crawls less," and it names HTTP 429 alongside 5xx codes explicitly.

Google's status code documentation reinforces it. Google's crawlers "treat the 429 status code as a signal that the server is overloaded, and it's considered a server error." So a rate limit that catches Googlebot does not just slow one crawl. It reduces how much of your site Google visits at all.

The fix is to exclude verified crawlers from your limits, verifying by reverse DNS or the published IP ranges rather than by the user agent string, which anyone can forge. Then check your logs for 429s served to crawlers as a standing health check. Our notes on crawl budget cover why this matters more on large sites.

What about AI crawlers?

They are the reason a lot of teams are looking at rate limiting in 2026, and they need a deliberate decision rather than a reflex. Blocking them entirely removes you from the AI answers your buyers are increasingly using. Letting them run unlimited can cost real bandwidth on a large site.

A middle path works for most B2B sites: allow the crawlers you want to be cited by, apply a generous limit rather than a block, and serve a proper 429 with Retry-After when they exceed it so a well-behaved crawler backs off instead of hammering.

Deciding which ones to allow is a strategy question, not a security one. We covered the trade in our piece on controlling AI crawlers.

How do you pick the actual numbers?

From your logs, never from a blog post. Look at the real distribution of requests per IP per minute for the endpoint you are protecting over a normal week. Find the 99th percentile of legitimate traffic. Set your limit meaningfully above it.

Then watch what you would have blocked before you actually block anything. Most platforms let you log a rule without enforcing it. Run it in that mode for a week and read what it caught. In our experience the first draft of a limit almost always catches something legitimate.

And set different limits per endpoint. One global limit is a blunt instrument that is simultaneously too tight for your search page and too loose for your login. The extra rules cost you nothing to run.

What should you put in place first?

Start with your forms, because that is where the abuse is cheapest for attackers and most expensive for you. Add a per-form, per-session limit, return a real 429 with Retry-After, and exclude verified crawlers from the start.

Then add monitoring before you add more rules. You want to know how many 429s you serve per day, to whom, and on what path. A rate limit you cannot observe is a trap you have set for yourself.

If you want a second pair of eyes on where your site is exposed and where a limit would just create support tickets, let us talk. We are at phoenix.studio.