Meta-ExternalFetcher user agent and robots.txt rules
Meta-ExternalFetcher fetches individual links at a user’s request, for example to help Meta AI complete a task on a website.
User agent
As documented by Meta. Version numbers change; match on the token.
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
robots.txt rules for Meta-ExternalFetcher
Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.
Block Meta-ExternalFetcher
Disallows every path for Meta-ExternalFetcher. That asks Meta not to fetch your pages when its users ask, though Meta says user-triggered fetches may not always follow robots.txt.
User-agent: meta-externalfetcher Disallow: /
Allow Meta-ExternalFetcher
Lets Meta-ExternalFetcher fetch every path, whatever “User-agent: *” says.
User-agent: meta-externalfetcher Allow: /
Good to know
Meta says it may bypass robots.txt, because the fetch was requested by a user.
Meta-ExternalFetcher FAQ
What is Meta-ExternalFetcher?
Meta-ExternalFetcher fetches individual links at a user’s request, for example to help Meta AI complete a task on a website.
How do I block Meta-ExternalFetcher?
Add “User-agent: meta-externalfetcher” and “Disallow: /” to your robots.txt. That asks Meta not to fetch your pages when its users ask, though Meta says user-triggered fetches may not always follow robots.txt.
Does Meta-ExternalFetcher respect robots.txt?
Meta says that because a person starts these requests, robots.txt rules may not apply to Meta-ExternalFetcher.
What is the Meta-ExternalFetcher user agent?
Meta documents it as “meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)”. Version numbers change, so match on “meta-externalfetcher”.
How can I verify Meta-ExternalFetcher requests?
Meta publishes no IP list for Meta-ExternalFetcher. Treat the user agent as a hint, not proof.
Source: developers.facebook.com, checked Nov 17, 2026. AI Crawler Directory
Files are the start. Answers are the result.
Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.