Crawl Stats Report: How to Read It in 2026

Administrator in Search Engine Optimization Sep 26 / 3 weeks ago

Imagine it is the Monday after Black Friday at a US retailer, where Googlebot requests jumped over the weekend, response times climbed, and nobody can explain why. The first place to look costs nothing. It is the Crawl Stats report in Google Search Console.

The Search Console Help page states: "If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail." This guide explains who should care, what every panel means, and how to respond when crawling spikes or drops, and it argues that Crawl Stats is a health monitor, not a growth lever.

Who actually needs the Crawl Stats report?

Websites with more than about a thousand pages benefit most, especially large retailers, publishers, marketplaces and brands operating several country subdomains. Google's Help documentation says a site with fewer than a thousand pages should not need it. If your pages get crawled on the day you publish them, I think a quarterly review is enough, and your time is better spent elsewhere.

Google's guide to optimizing crawl budget sets an even higher bar, since it targets sites with 1 million+ unique pages that change weekly, sites with 10,000+ pages that change daily, and sites with many URLs classified as discovered but not indexed. Google describes those numbers as "a rough estimate." The realistic audience includes large retailers in the United States, news publishers in Germany and France, and international marketplaces. Smaller websites can keep their sitemap current, monitor the Page Indexing report, and read our practitioner guide to the core Search Console reports instead.

How do you open the Crawl Stats report in Search Console?

Open Search Console, choose a root-level property, click Property settings, then select Crawl stats. The report is available only for a Domain property such as example.com or a URL-prefix property at the root, such as https://example.com, so a property scoped to a folder like example.com/blog/ will not display it. Every subdomain rolls up into the parent property.

The report counts the actual URLs Google requested rather than canonical URLs, and it includes both HTTP and HTTPS requests. The trap is subdomain scope. A parent property shows every child host, but a child never shows its siblings, as the FAQ below explains.

For example, imagine a publisher in France with fr. and de. subdomains, where the fr. chart looks calm while the real spike sits on de. The same applies to a ko. subdomain for South Korea, so click into each child. If you are still choosing an international structure, read our guide to hreflang and picking subfolders, subdomains or country domains.

What do the three over-time charts tell you?

The top of the report displays three charts: total crawl requests, total download size and average response time. Requests measure how busy Googlebot was, download size shows how many bytes it retrieved, and response time shows how quickly your server answered. Read them together, because a slower server can explain fewer requests.

Total crawl requests counts every request for URLs on your site, successful or not, including page resources on your domain, duplicate requests, and even fetches skipped because robots.txt was unavailable. Download size has a quirk, since a cached resource used by many pages is requested only the first time. Response time counts each linked resource as a separate response. Google's crawl budget documentation says the crawl capacity limit increases when response times, including latency and Time-to-First Byte, remain stable or improve, which is why I would monitor the response time chart first.

What does host status mean, and when should it worry you?

Host status tells you whether Google encountered availability problems while crawling, and it has three states. No significant issue in the past 90 days is the goal. An issue in the last 90 days but more than a week ago may be transient or already resolved. An issue in the last week means you should check whether it is a recurring problem.

Google evaluates three categories: robots.txt fetching, DNS resolution and server connectivity. Each category chart has a dotted red line, and if a metric crosses it on a given day, that counts as an issue. Google's example is DNS resolution failing for more than 5% of requests on a day, and for DNS errors Google recommends checking with your registrar. My recommendation is to also review the firewall configuration on any CDN such as Cloudflare, and our look at what edge SEO fixes and what it breaks at the CDN layer shows how one bad rule can misfire across a whole site.

What happens when Google cannot fetch your robots.txt file?

Google needs a successful robots.txt response before it crawls. A 200 with any file, even an empty or invalid one, counts as success, and so does a 403, 404 or 410, but a 429 or any 5XX does not. If the fetch fails, Google stops crawling for 12 hours, then uses the last successful file for up to 30 days.

Google's Help documentation describes the complete timeline.

1. Google looks for a successful robots.txt response less than 24 hours old, and if it has one, it uses that file.
2. Otherwise it requests the file again, and a successful response lets the crawl start.
3. If that request fails, Google stops crawling for the first 12 hours but keeps requesting the file.
4. From 12 hours to 30 days, it uses the last successfully fetched file and keeps requesting.
5. After 30 days, it crawls as if no file exists if your homepage is available, and otherwise it stops crawling the site.

Google also warns against cloaking robots.txt or varying it by user agent. For the directives themselves, see our guide to robots.txt syntax, meta robots and noindex.

How should you read the crawl responses table?

The responses table groups every crawl by response type as a percentage of all requests, not of individual URLs. If Google requests one URL twice and receives a 500 then a 200, the table shows 50% server error and 50% OK. Most responses should be 200 or other good codes, unless you are in the middle of a site migration.

Good codes include 200, 301 or 308, 302 or 307, a meta refresh and 304. The possibly good category is 404, and Google says: "It's not possible, worthwhile, or even desirable to fix all 404 errors on your site." Bad codes require fixing, including robots.txt not available, 401 or 407, 5XX, DNS errors, page timeouts and redirect errors. A "Page could not be reached" error never reached your server, so your logs will not record it. For the meaning of each code, read our guide to how HTTP status codes shape crawling and indexing, and for long chains, use our playbook for redirects and site migrations.

What do file type and Googlebot type reveal?

File type shows what Google fetched, such as HTML, images, JavaScript, CSS, PDF or JSON. Googlebot type identifies which crawler made the request, such as Googlebot Smartphone, Googlebot Desktop, Page resource load, AdsBot or StoreBot. Together they answer one diagnostic question: is the crawl load coming from your pages, from their resources, or from an advertising or shopping crawler?

File types also include video, audio, other XML, RSS or Atom feeds, and geographic data such as KML. Page resource load covers the images and CSS files Google fetches to render a page, so blocking those resources can break rendering, as our explainer on how Google renders JavaScript shows. The Help page says an AdsBot spike likely means many new Dynamic Search Ads targets, and AdsBot crawls those URLs about every 2 weeks, so paid and organic teams should coordinate. See our playbook for running PPC and SEO as one team. StoreBot is the product shopping crawler, and Google's crawl budget documentation notes that Google Shopping has higher demand for products in your merchant feeds, so a clean Google Merchant Center product feed matters too.

What is the difference between discovery and refresh crawls?

A discovery crawl is a request for a URL Google has never crawled before, while a refresh crawl revisits a page Google already knows. After you add a large new section or submit a sitemap, Google says you should ideally see a bump in discovery crawls. If you see none, my guess is that your new URLs are hard to find.

For pages that change frequently but are not recrawled often enough, Google recommends including them in a sitemap, and our guide to XML sitemaps and the Indexing API covers accurate lastmod dates. If a US retailer's new Black Friday hub gets no discovery bump after a sitemap update, the pages may sit too deep. This guide to site architecture and crawl depth on large sites explains why.

Why did my crawl requests suddenly spike?

Spikes usually follow a change you made. Google lists three common causes: you unblocked a large section, you added a large new section, or you created many new Dynamic Search Ads targets through page feeds or URL_Equals rules. First identify which crawler is responsible using Googlebot type, then decide whether your server has the capacity to cope or needs relief.

For immediate relief, Google offers two options. You can block the overloading agent in robots.txt, although that can take up to a day to take effect, or you can return HTTP 503 or 429 when you approach your serving limit. Google warns: "Be sure not to return 503 or 429 for more than two or three days, though, or it can signal Google to crawl your site less frequently in the long term."

Why is Googlebot crawling my site less than before?

Google names four possible reasons for a drop: a new or very broad robots.txt rule, a slower server that makes Googlebot throttle back, a rising server error rate, or content that changes rarely or lacks quality. Google adds that crawl rate should stay relatively stable over a week or two, so a sudden decrease deserves investigation.

Start with recent robots.txt edits, and confirm you did not block CSS or JavaScript. Then review the response time chart and the 5XX rows. Google's status code documentation says a 500 reduces crawl rate in proportion to the individual URLs returning errors, and the rate climbs back gradually once 2xx responses return. For the quality cause, Google suggests you "get neutral feedback from people not associated with your site." If organic traffic fell too, use our decision tree for finding the real cause of a traffic drop.

How does Crawl Stats compare with server logs and URL Inspection?

Use all three for different jobs. The Crawl Stats report is a free, aggregate health check with host status built in, while server logs record every request your server received, URL by URL. URL Inspection answers questions about one page at a time. I think most teams should start with the report and move to logs only when it cannot explain a pattern.

If report totals run much higher than your logs, the Help page points to robots.txt. When the file is unavailable for too long, Google counts crawls it might have made but never sent. Google also admits that some requests are not counted, so small differences from your logs are normal.

The table below uses only Google's documentation plus common facts about server logs and URL Inspection.

My contrarian view is that the report is a health monitor, not a growth lever. For Google Search, the crawl budget documentation names two ways to get more budget: add server resources, and improve content quality. For deeper work, move to our hands-on guide to log file analysis and crawl budget, then reduce waste with this guide to taming URL parameters and crawl traps, and fold the fixes into a technical SEO audit your developers will actually ship.

What should your monthly Crawl Stats check include?

A monthly check is quick if you follow the same procedure each time: review host status, the three charts, the responses table, Googlebot type, crawl purpose and each child host. Document anything new, compare it with your release notes, and escalate only what repeats. For a team in the United Kingdom or elsewhere in Europe, a shared checklist keeps reviews consistent.

1. Open the parent Domain property and read host status first.
2. Click any red category and note the dates above the dotted line.
3. Compare total crawl requests with average response time.
4. Scan the responses table for 5XX errors, timeouts and robots.txt not available.
5. Check Googlebot type for AdsBot or StoreBot surges after campaign or feed changes.
6. Confirm a bump in discovery after any launch or sitemap update.
7. Open each child host, such as de., fr. and ko., and document what you find.

Go back to that Monday after Black Friday. With this routine, the team reads host status, checks Googlebot type for an AdsBot surge, and quickly decides whether to add capacity, reduce ad targets or return a short 503, which is the real job of Crawl Stats. Google launched the current version on Tuesday, November 24, 2020, in a Search Central Blog post by Hillel Maoz, Engineering lead for Search Console. My advice is simple. Open the parent property and read host status first.

Frequently asked questions about Crawl Stats

Does the Crawl Stats report show data for subdomains?

Yes, but only from the parent property. If you view the Domain property for example.com, you see requests to every child domain below it, such as en.example.com or ko.example.com, and you can scope the view to one subdomain by clicking it in the Hosts list. A sibling view will not show its neighbors, so de.example.com cannot see requests for en.example.com. Google's Help documentation adds that only the top 20 child domains that received traffic in the past 90 days appear in that list.

Can I ask Google to crawl my site more often?

No. The Search Console Help page states: "You can't tell Google to increase your crawl rate." What you can do is remove the reasons Google holds back, so keep your server fast and stable, fix 5XX errors, and avoid wasting crawls on duplicate URLs. Google's crawl budget documentation says the capacity limit increases when a site stays healthy and there is demand. For small or medium websites, Google suggests updating your sitemaps and confirming you are not blocking pages you want crawled.

Is it a problem if my site has no robots.txt file?

No, because your site does not need a robots.txt file at all. Google's Help documentation treats a 403, 404 or 410 for robots.txt as a successful response, and a 404 means Google may crawl any URL on the site. The real danger is a robots.txt URL that returns a 429 or a 5XX error, which counts as unsuccessful, so Google may stop crawling for the first 12 hours while it keeps requesting the file. A missing file is safe, but a broken one is not.

Can I use a 401 or 403 to slow Googlebot down?

No. Google's documentation on HTTP status codes says not to use 401 and 403 codes to limit crawl rate, because 4xx codes, except 429, have no effect on crawl rate. If you need temporary relief, Google suggests returning 503 or 429 while you are near your serving limit. Keep that to two or three days at most, since a longer period can signal Google to crawl your site less often in the long term. A robots.txt block is the other option, but it can take up to a day to take effect.

Why does the report show so many AdsBot requests?

AdsBot requests come from Google's advertising crawlers, and the Help documentation says a spike in AdsBot requests likely means you recently created many new targets for Dynamic Search Ads, often through page feeds or URL_Equals rules. AdsBot crawls those URLs about every 2 weeks, so the load returns unless you change the configuration. If your server struggles, Google suggests limiting ad targets, adding URLs in smaller batches, or increasing serving capacity. Because crawl capacity is shared across crawlers, heavy AdsBot demand can squeeze Googlebot.

Does a noindex tag stop Google from crawling a page?

No. Google's crawl budget documentation says Google will still request a noindex page, then drop it when it sees the noindex tag or header, which wastes crawling time on large sites. If you never want a page crawled, block it in robots.txt instead. For pages you removed permanently, return a 404 or 410, because the documentation calls a 404 a strong signal not to crawl that URL again. Blocked URLs, by contrast, stay in the crawl queue much longer and get recrawled when you remove the block.

What does Page resource load mean in Googlebot type?

Page resource load is a secondary fetch for resources your page uses. When Google crawls a page, it fetches important linked resources, such as images or CSS files, so it can render the page before trying to index it. If an image is loaded as part of a page, the report counts it as Page resource load, not as Image. In my view, a large share here is not a warning sign on its own, but make sure robots.txt does not block the CSS or JavaScript that Google needs to understand your content.

Why is a URL missing from the example lists?

Because the lists are samples. Google's Help documentation says example URLs are not comprehensive, just a representative example, so a missing URL in the list does not mean Google did not request it. The number of examples can be weighted by day, so some request types may show more examples than others, and this should balance out over time. For proof about one individual URL, use the URL Inspection tool in Search Console or search your server logs for that exact path.

Do redirects count as extra crawl requests?

Yes. Each hop in a server-side redirect counts as its own request, so if page1 redirects to page2, which redirects to page3, the report shows three separate requests. Redirect responses fall under the Other file type category, and client-side redirects are not counted. Google's status code documentation says its crawlers follow up to 10 redirect hops, and its crawl budget documentation warns that long redirect chains have a negative effect on crawling. My recommendation is to point every old URL straight at its final destination in one hop.

Does returning 304 Not Modified help crawling?

It can. Google's crawl budget documentation recommends supporting 304 responses, because if a page has not changed since Google last crawled it, a 304 tells Google to reuse its cached copy and saves server bandwidth and resources. The Crawl Stats report lists 304 as a good response, so a healthy share of 304s is not a problem. Google's status code documentation adds that a 304 has no effect on indexing beyond letting the pipeline recalculate signals for the URL, so it simply saves work on both sides.