Googlebot user agent and robots.txt rules

Googlebot is Google Search’s crawler. Google’s AI Overviews and AI Mode are Search features, so Googlebot’s robots.txt rules are the control for them too.

AI search Follows robots.txt Google

User agent

As documented by Google. Version numbers change; match on the token.

User agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

robots.txt rules for Googlebot

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.

Block Googlebot

Disallows every path for Googlebot. That keeps your pages out of Google’s AI search index, so its answers can’t cite them.

robots.txt
User-agent: Googlebot
Disallow: /

Allow Googlebot

Lets Googlebot fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: Googlebot
Allow: /

Good to know

Blocking Googlebot also removes you from regular Google Search. To limit what AI features show, Google points to nosnippet, data-nosnippet, max-snippet or noindex instead.

Googlebot FAQ

What is Googlebot?

Googlebot is Google Search’s crawler. Google’s AI Overviews and AI Mode are Search features, so Googlebot’s robots.txt rules are the control for them too.

How do I block Googlebot?

Add “User-agent: Googlebot” and “Disallow: /” to your robots.txt. That keeps your pages out of Google’s AI search index, so its answers can’t cite them.

Does Googlebot respect robots.txt?

Yes. Google says Googlebot follows robots.txt rules written for it.

What is the Googlebot user agent?

Google documents it as “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”. Version numbers change, so match on “Googlebot”.

How can I verify Googlebot requests?

Match the request IP against Google’s published list (https://developers.google.com/static/search/apis/ipranges/googlebot.json). The user agent alone can be faked.

Source: developers.google.com, checked Nov 17, 2026. AI Crawler Directory

Files are the start. Answers are the result.

Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.