Crawl Budget and Index Hygiene for Large Sites
For a two-hundred-page site, crawl budget is a non-issue. For a catalogue with filter combinations generating millions of URL permutations, it is the dominant technical concern. The failure mode is subtle: nothing breaks, important pages simply get crawled less often and updated more slowly in the index.
Where the waste comes from
| Source | Typical symptom | Fix |
|---|---|---|
| Faceted navigation | Millions of parameter URLs crawled | Disallow non-valuable parameters, link only curated facets |
| Internal search results | Query URLs appearing in the index | Robots disallow plus noindex on already-indexed ones |
| Session or tracking parameters | Duplicate content at many URLs | Strip at the server, canonical to the clean URL |
| Pagination chains | Deep pages crawled endlessly | Shorter chains, better hub linking |
| Redirect chains | Crawl spent on hops | Flatten to a single redirect |
Disallow and noindex are different tools
A robots disallow prevents crawling. A noindex tag prevents indexing but requires the page to be crawled to be seen. Combining them on a page already in the index is a common mistake: the crawler is blocked, never reads the noindex, and the URL lingers. Remove it from the index first with noindex, then disallow once it has dropped.
You cannot tell a crawler something on a page it is not allowed to read.
Canonicals are advisory
A canonical tag expresses a preference. Search engines frequently ignore it when the signals disagree — different content, different internal links, different inbound links. If two URLs must be one page, make them one page: redirect, or remove the duplicate route entirely. Reserve canonicals for cases where both URLs genuinely need to exist.
Sitemaps as a diagnostic
Split sitemaps by page type - products, categories, articles, landing pages. The indexation rate per sitemap then tells you which type of page has a problem, instead of giving you one blended number that hides everything.
Read the logs
Server logs are the only source that shows what crawlers actually did rather than what tools estimate. A month of logs answers the real questions: which sections get crawled most, which valuable pages are visited rarely, and how much crawl activity lands on URLs you never wanted indexed. It is unglamorous work and it routinely finds problems no audit tool surfaces.
Related articles
Topical Maps: Turning Keyword Lists Into Territory
Search engines reward demonstrated coverage of a subject, not isolated pages. A topical map is the plan that turns scattered posts into an ownable territory.
Content Briefs That Make Freelancers Unnecessary to Manage
Editing a bad draft costs more than writing a good brief. Most teams keep choosing the expensive option.
Writing for AI Answer Engines Without Abandoning Search
Answer engines reward clarity, structure and verifiable claims. So did search, we just got away with less.