Web Crawler: Googlebot or Not?

Shaikat Ray

Google has multiple web crawlers, Googlebot is the most known one. Google also has some special-case-crawlers that do not respect the global robots.txt user agent (*), and they need to call out explicitly.

Google’s Crawlers:

  • Googlebot Smartphone
  • Googlebot Desktop
  • Googlebot Image
  • Googlebot News
  • Googlebot Video
  • Google-InspectionTool
  • Google-Extended

Google’s Special Case Crawlers

  • AdsBot Mobile Web
  • APIs-Google
  • AdSense
  • Mobile AdSense
  • Google