Google recently introduced new and improved crawl stats for websites in the Search Console tool, to help site owners better understand how Googlebot crawls their sites.
The new Crawl Stats report – which can be viewed from the ‘Settings’ option in the menu bar – includes the following new features:
- Total number of requests grouped by response code, crawled file type, crawl purpose, and Googlebot type
- Detailed information on host status
- URL examples to show where in your site requests occurred
- Comprehensive summary for properties with multiple hosts and support for domain properties.
The Crawl Stats report enables website owners to see Google crawl data totals and overtime charts for: total requests, total download size and average response time.
Grouped crawl data
The new version of the report also provides data on crawl requests broken down by response, file type of the fetched URL, purpose of the crawl request, and Googlebot agent. See example URLs of each type by clicking on a row in the grouping table.
High level & detailed information on host status issues
Host status details in the report let you check the site’s general availability to Google in the last 90 days.
Host status details
For domain properties with multiple hosts, you can check the host status for each of the top hosts presented in the report summary view. This can help evaluate performance of all hosts under the domain in one place.
In summary, although this new set of reports are quite technical, they will help you understand how Googlebot crawls your site in these ways:
- See Google’s crawling history in the overtime charts
- See the file types and file sizes returned by your site
- See crawl requests details in the example lists
- Track your site’s availability issues in the host status view.
It’s also possible to check the Crawl stats report documentation which has lots of details on what can be done with the new report.