How Should You Handle URL Parameters and Faceted Navigation?

How Should You Handle URL Parameters and Faceted Navigation?

Block the combinations you do not want crawled, and do it in robots.txt. Google's own guidance names robots.txt as the most effective long term fix for faceted navigation, ahead of canonical tags and nofollow. Filters and sort options multiply URLs faster than any crawler will ever get through them.

This comes up on resource hubs, blog archives, job boards and directories, not just online shops. Any page with tag filters, category dropdowns and a sort control is generating faceted URLs whether the team meant to or not.

Here is what Google actually says, and the order we work through it.

Why Are Filter URLs a Problem At All?

Because they look new to a crawler and there is no way to tell in advance that they are not. Google describes the mechanism plainly: because the URLs created by faceted navigation seem novel and crawlers cannot determine whether they will be useful without crawling them first, crawlers typically access a very large number of faceted navigation URLs.

The cost is not abstract. Google warns that having these URLs crawled means increased resource usage on your server and potentially slower discovery of new URLs on your site. Resources spent on filter combinations are resources not spent on the article you published this morning.

The multiplication is what catches people out. Five filters with six options each, plus three sort orders, is thousands of URLs from one template, and every one of them is a near duplicate of the others.

Does Your Site Even Need to Care?

Probably less than you fear, and Google gives thresholds. It says crawl budget is a concern for large sites with 1 million or more unique pages whose content changes moderately often, meaning about once a week, and for medium or larger sites with 10,000 or more unique pages with very rapidly changing content, meaning daily.

It adds a third category that catches smaller sites: those with a large portion of their total URLs classified by Search Console as "Discovered, currently not indexed". That report is the tell. If it is full of parameter URLs, you have the problem regardless of your page count.

If your pages tend to be crawled the same day they are published, Google's position is that you do not need to worry about this. Most B2B marketing sites are in that group, which is worth saying out loud before anyone spends a sprint on it. We unpack the wider picture in our piece on crawl budget.

What Is the Recommended Fix?

Disallow the parameters in robots.txt. Google explicitly recommends using robots.txt to disallow crawling of faceted navigation URLs, and treats this as the most effective long term solution rather than one option among equals.

The pattern is to block the filtering and sorting parameters while leaving the unfiltered version of the page crawlable. Your category page stays available. The 4,000 permutations of it do not.

Get this right before you deploy it, because a robots.txt line that is slightly too broad can remove pages you wanted indexed, and you will not get an error message. Test the rules against real URLs from your logs, not against examples you wrote by hand.

What About Canonical Tags?

They help, and they are the second best answer here. Google's position is that rel="canonical" may over time decrease the crawl volume of non canonical versions, but it describes the approach as less effective in the long run than blocking.

The reason is timing. A canonical tag is a signal Google reads after fetching the page, so the crawl has already happened. You are telling the crawler which version counts, not saving it the trip. Over many cycles that reduces crawling of the duplicates, but slowly.

Canonicals still matter for consolidating ranking signals, so this is not an argument against using them. It is an argument against treating them as your crawl control, which is a distinction we draw out in canonical URLs and duplicate content.

Is There a Clever Structural Option?

Yes: put the filter state in a URL fragment. Google notes that Search generally does not support URL fragments in crawling and indexing, so a state expressed after a hash is invisible to the crawler by design.

This is elegant because it requires no ongoing maintenance. There is no rule to keep in sync, no parameter list to update when someone adds a new filter, and no risk of a robots.txt pattern drifting out of date.

The trade is that fragment state is client side, so the filtered view cannot be indexed or shared as a distinct page even when you might want it to be, and the unfiltered page must render properly in HTML on its own. For most filter interfaces that is the correct trade anyway.

What About nofollow?

It works in principle and it is fragile in practice. Google says the approach can help, but requires that every anchor pointing to a specific URL must carry the rel="nofollow" attribute.

That word "every" is the problem. One template that forgets it, one hand written link in a blog post, one sitemap entry, and the URL is discoverable again. Then you have a rule that is 95% enforced, which for crawling purposes is close to not enforced.

We treat nofollow here as a supporting measure rather than a mechanism. If a strategy depends on nobody ever making a mistake in a template, it is not a strategy.

Why Does Google Warn Against noindex Here?

Because it does not save any crawling. Google's crawl budget guidance is direct: do not use noindex for this, because Google will still request the page and then drop it, wasting crawling time.

This surprises people, because noindex feels like the strongest tool available. It is strong for keeping something out of the index. It does nothing for the problem you actually have, which is the fetching.

Google also warns against using robots.txt to temporarily reallocate crawl budget to other pages. These rules are structural decisions, not a dial you adjust during a launch week.

What Else Does Google Recommend?

Four things that pair well with parameter handling. Consolidate duplicate content so crawling focuses on unique material rather than on the same content at different addresses. Block crawling of unimportant URLs with robots.txt. Return a 404 or 410 status code for pages that are permanently gone. Keep your sitemaps up to date.

The one teams most often miss is supporting 304 Not Modified responses. If your server can tell Google that nothing has changed, the crawler spends far less on your unchanged pages and has more left for the ones that did change.

These are unglamorous server behaviours rather than SEO tactics, which is exactly why they tend to be nobody's job and stay broken for years.

Should Any Filtered Pages Be Indexed?

A small number, chosen deliberately. If a filter combination matches something people genuinely search for, it deserves a real page with its own title, its own copy and its own place in your navigation.

The key word is deliberately. That page should be a considered landing page, not the accidental output of a filter interface, and there should be perhaps a dozen of them rather than a thousand. Everything else stays uncrawled.

Getting that distinction right is really a URL design question, and it is much easier to decide before launch than to untangle afterwards, which is why we cover it in URL structure for SEO.

What Would We Do This Week?

Open the "Discovered, currently not indexed" report in Search Console and look at what is in it. If it is dominated by parameter URLs, you have confirmed the problem in five minutes without any tooling.

Then write down every parameter your templates can emit, decide which ones should ever be crawled, and block the rest in robots.txt. Test the rules against real URLs from your server logs before shipping, and check the report again in a few weeks.

If your filter pages are eating your crawl and you are not sure which rules are safe to write, we are happy to look at the URL patterns with you. You can find us at phoenix.studio.