What is Crawl Budget?
Crawl budget is the number of URLs Googlebot is willing and able to fetch from your site over a given period. Two forces set it: crawl capacity, meaning how much your server takes before responses slow, and crawl demand, meaning how badly Google wants your pages. Small sites rarely hit the ceiling. Large catalogues and property portals hit it constantly.
Crawl Capacity and Demand
How crawl budget works
Googlebot watches your server response times and adjusts accordingly, so a fast and stable host earns heavier crawling while repeated timeouts or 5xx errors make it back off within hours. That is the capacity half, and it is mostly a hosting question rather than an SEO one.
Demand is editorial. Pages that get updated, linked to and clicked pull crawlers back more often, while a property listing untouched for fourteen months slides steadily down the queue as your blog index gets fetched every single day.
The waste happens in between, because faceted navigation on a Shopify store can generate tens of thousands of near-duplicate parameter URLs, and Googlebot will happily spend its whole allowance there instead of on the forty product pages that actually sell.
Common ways teams waste it
Uncontrolled filter parameters top the list, since colour, size, price band and sort order combine into a URL explosion where every single combination looks crawlable unless you deliberately block it. Sort order is the worst offender.
Then there are the long redirect chains left behind by two site migrations nobody cleaned up after. Each hop is a separate fetch. Three hops means Googlebot burns three requests to reach one page.
Soft 404s are quieter but just as costly, because a sold-out project page returning a 200 status alongside a message saying nothing was found will keep getting recrawled forever. Return a real 404 or 410, or redirect it to the parent category with some intent behind the choice. Also check for an open staging subdomain.
What this looks like in practice
For a Delhi NCR property portal carrying 30,000 listings the work is triage: block filter parameters in robots.txt, keep an XML sitemap containing only live inventory, and split that sitemap by city so you can read index coverage per market inside Search Console. Drop sold units the day they go. Same day. No exceptions.
Watch the Crawl Stats report closely, because a rising average response time sitting alongside falling pages crawled per day almost always points at your hosting rather than at anything in your tags. Moving off shared hosting has fixed more crawl problems than tag tinkering ever has.
Related terms: Creative Fatigue · CRO · Domain Authority.
Where this shows up in the work: SEO & AI Search · Full glossary.
Crawl Budget — questions, answered.
Probably not, because if you run under a few thousand URLs and pages get indexed within a week or two then Google already has enough capacity for you. Spend the effort on content and internal linking.
Yes, in practice, since requests Googlebot never makes to blocked parameter URLs become available for pages that matter. Remember a blocked URL can still appear in results if other sites link to it, so use noindex for genuine removal.
Last updated 2026-08-08
Ready to replace guesswork with a growth engine?
Book a 30-minute strategy call. We’ll show you exactly where your funnel is leaking, before you spend a dollar.