Crawl Budget in 2027: What It Is and When You Need to Care
crawling, explainedAlright, let’s sort this out. Crawl budget is the set of URLs Google can and wants to crawl on your site. Google says its own guide is mainly for very large sites, so most small business owners can relax. This guide shows who should care, what wastes crawl budget and how to manage it.
- Google’s size thresholds quoted
- what wastes crawling
- no ranking promises
// example data in the shape of Search Console’s Crawl Stats report. Not a real site. The report’s fields come from Google’s help page.
What Crawl Budget Means
Crawl budget is how much of your site Google can crawl and chooses to crawl. It only becomes a problem when you have far more URLs than Google will fetch.
Google’s guide defines it in one line: “Google defines a site’s crawl budget as the set of URLs that Google can and wants to crawl.” Each unique hostname counts as a separate site.
Two forces set the budget. Google calls them the crawl capacity limit and crawl demand. Capacity is how much your server can handle. Demand is how much Google wants to crawl.
Now the reassuring part. Google says the guide is for large sites, with rough thresholds: 1 million or more unique pages that change about weekly, or 10,000 or more unique pages that change daily. It calls those numbers a rough estimate. If you run a plumber’s site with 40 pages, this is not your bottleneck.
Should You Worry About Crawl Budget?
Answer three questions. Most small sites stop at the first.
How many unique pages?
count real, different pages
Under the thresholds
focus on quality and links
1 million+ or 10,000+ daily
this guide applies
Many URLs not crawled?
check the Page indexing report
Server and waste
errors, duplicates, chains
Trim and speed up
then watch the stats
// built from Google’s crawl budget guide. The thresholds are Google’s rough estimate. My flow around them is a simplification.
Capacity and Demand, Side by Side
You can raise the budget only by helping both halves.
| Crawl capacity limit | Crawl demand | |
|---|---|---|
| What it is | Limits the time your server spends holding connections open for Google | How much Google wants to crawl |
| What raises it | A healthy, fast server | Popularity, fresh content, a clean list of URLs |
| What lowers it | Slowdowns, 5xx errors and 429 responses | Duplicates, stale pages, low-value URLs |
// based on Google’s crawl budget guide. For Googlebot, it lists site size, update frequency, page quality and relevance as demand factors, and says the main thing you control is the URLs Google knows about, including duplicates.
Server health matters in a very practical way. Google says 5xx and 429 errors make its crawlers slow down, and that once the server answers with success codes again, it gradually increases the crawl rate.
URLs That Burn Crawl Budget
Google’s guide names several habits. Here they are with the fix.
| waste | what Google advises |
|---|---|
| duplicate pages | consolidate duplicate content so crawling focuses on unique pages |
| soft 404 pages | fix them, because they keep getting crawled. See soft 404 errors |
| long redirect chains | avoid them. See redirect chains and loops |
| removed pages | return 404 or 410 for permanently removed pages |
| pages you never want crawled | block them with robots.txt, which prevents crawling |
| noindex used to save crawling | do not. Google still requests the page and then drops it |
// the right column is paraphrased from Google’s guide. Not every site has every problem.
On a small shop, the usual offenders are filter and sort URLs that create thousands of near-duplicate pages. That example is my own experience, not Google’s. Google says the URLs it knows about, duplicates included, are the main thing you can control.
How to Manage Crawl Budget
Work through these in order. Stop when the waste is gone.
- Count your unique pages. If you are far below Google’s thresholds, make sure the basics are clean and read no further.
- Open the Crawl Stats report. In Search Console, go to Settings and choose Crawl stats. Google says it is available only for root-level properties and is aimed at advanced users.
- Read host status and response codes. Google says host status should ideally be green, and most responses should be 200 or other good types. Errors such as 5xx need fixing.
- Find the waste. Look for duplicate URLs, parameter pages, soft 404s and redirect chains in a crawl of your site.
- Consolidate and block. Merge duplicates, use robots.txt for areas Google should not crawl at all, and return 404 or 410 for pages gone for good.
- Keep the sitemap current. Google says to keep it up to date and include lastmod for updated content.
- Speed up the server. Google says improving response time lets it read more of your content. If server capacity is the bottleneck, Google says adding server resources is an option.
Good internal links also help Google reach what matters. See internal linking and orphan pages, and tidy broken URLs with fix broken links. Redirect choices are explained in 301 vs 302 redirects.
Mistakes That Waste the Effort
Crawl budget is one of the most over-diagnosed topics in SEO.
- Blaming crawl budget on a small site. If a 40-page site has pages not indexed, the cause is almost always quality, links or a blocking rule. Read the Page indexing report first.
- Using noindex to save crawl budget. Google says it still requests the page and then drops it, so you save nothing.
- Blocking pages in robots.txt and expecting removal. Robots.txt stops crawling. It is not a removal tool, and Google’s Page indexing report notes that a block does not guarantee the page will not be indexed through some other means.
- Letting soft 404s pile up. Google says they keep getting crawled and consume budget.
- Expecting more crawling to mean better rankings. Crawling is a step before indexing and ranking. Google says indexing is never guaranteed.
How to Check the Fix Worked
Compare the Crawl Stats report before and after, over the same time span.
- Watch the response code table. The share of 200 responses should grow as waste shrinks.
- Check average response time. A lower number means Google can fetch more in the same time.
- Look at the crawl purpose split between discovery and refresh. Google defines discovery as a URL never crawled before and refresh as a recrawl of a known page.
- Give it time. Google says some changes take a few hours and others several months.
Questions People Ask Me
What is crawl budget?
It is the set of URLs that Google can and wants to crawl on your site. It depends on how much your server can handle, called the crawl capacity limit, and on how much Google wants to crawl, called crawl demand.
Does my small site need to worry about crawl budget?
Probably not. Google’s guide is for sites with 1 million or more unique pages that change weekly, or 10,000 or more that change daily, and it calls those numbers a rough estimate.
How do I increase crawl budget?
Google names two routes: add server resources if capacity is the limit, and improve content quality, such as popularity, user value and uniqueness. Cleaning up wasted URLs helps too.
Do 404 errors waste crawl budget?
Not in the way people fear. Google says 4xx codes, except 429, have no effect on crawl rate, and that crawling of a 404 page gradually decreases. Soft 404s are different, since Google says they keep getting crawled.
Where can I see how Google crawls my site?
In the Crawl Stats report in Search Console. It shows total crawl requests, download size, average response time and responses by code. Google says it is aimed at advanced users.
Send Me the URL. I’ll Read the Crawl Stats.
I review crawl health as part of my technical SEO service. Message a cheap SEO company run by one person for a first look.
Sources and Last Check
- Google Search Central: Managing crawl budget for large sites (definition, thresholds, advice)
- Search Console Help: Crawl Stats report (fields, host status, audience)
- Google Crawling Infrastructure: HTTP status codes (4xx and crawl rate)
- Google Search Central: HTTP status codes and network errors (5xx and 429 slow crawling)
- Google Search Central: How Google Search works (indexing is not guaranteed)
- Search Console Help: Page indexing report (blocked by robots.txt note)
// last checked 11 Nov 2026. The small-shop filter example and my flow chart are my own.