Server logs record every request to your website, including visits from search engine crawlers. They show what Googlebot actually crawls.
What logs reveal
- Which pages Googlebot crawls most and least.
- Crawl errors and server problems.
- Crawling of useless URLs, such as filters and parameters.
- Bots hammering your site.
Who needs it
Large sites, online stores with many URLs, and sites with indexing problems. Small brochure sites rarely need log analysis; Search Console’s crawl stats are enough.
Getting logs
Hosting control panels often provide access logs. Some hosts or CDNs offer analytics on crawler activity.
Analysing
- Filter for verified Googlebot requests.
- Group by URL type and status code.
- Compare crawled URLs with important pages.
Break down URLs with our URL parser; developers may export data as JSON, readable with our JSON formatter.
Read crawl budget.
Want the full picture? This article is part of SEO: the complete guide, our in-depth guide with everything in one place.
