What AnswerHawkBot is
AnswerHawk tracks how AI engines answer questions about brands. Answers come from the engines through data vendors and official APIs; AnswerHawkBot is not used for that. It only fetches a few things from websites, and every request carries the user agent above.
When it visits and what it reads
| Request | When | What it reads | robots.txt | Rate limit |
|---|---|---|---|---|
| Cited page check | After an AI answer cites a page, at most once a week per page and brand | The cited page, to see if it lists the tracked brand | Followed | 10 / min per site |
| Favicon | For domains that appear in a project, every 30 days | The home page <head> and /favicon.ico | Standard file | 6 / min per site |
| Site details | When someone enters your domain in onboarding or the free AI Visibility Checker | The home page title and description | On request | One request each time |
| Free tools | When someone runs the llms.txt Checker, llms.txt Generator or AI Crawler Checker on your domain | robots.txt, llms.txt, llms-full.txt, sitemaps and a few pages | On request | 20 / min per site |
| llms.txt directory | Sites listed in the directory, every 30 days | robots.txt, then /llms.txt | Followed | 20 / min per site |
How it behaves
- only
GETrequests for public URLs; it never submits forms, signs in, keeps cookies or runs JavaScript; - explicit timeouts (8 seconds per request), at most 4 redirects, and a size cap on every response;
- per-site rate limits (see the table), so a site never receives bursts from us;
- only public internet addresses: private, local and reserved addresses are refused on every redirect;
- results are cached: a cited page is checked at most once a week for each brand.
What we keep
- cited pages: only the yes/no answer to "does this page list the brand?", for 7 days. Page content is not stored;
- favicons: the icon file, refreshed every 30 days, shown next to your domain in the app;
- llms.txt directory: a copy of public llms.txt files and their history, shown on the directory page;
- nothing we fetch is sold or used to train AI models.
Control access with robots.txt
The cited-page check and the llms.txt directory follow robots.txt. They read the group for AnswerHawkBot (or * when there is none). To stop them on your whole site:
User-agent: AnswerHawkBot
Disallow: /
To block only part of the site:
User-agent: AnswerHawkBot
Disallow: /private/
Favicons are standard public files, and the free tools read robots.txt, llms.txt and sitemaps only when someone asks for them. If you want us to stop every request to your domain, contact us and we will exclude it.
Verify a request
Real requests send exactly the user agent above and come from AnswerHawk servers. We do not publish a fixed list of IP addresses yet. If you see traffic that uses our name but ignores these rules, tell us with the time, URL and IP address.
Looking for the crawlers of AI companies (GPTBot, ClaudeBot, PerplexityBot…)? See the AI crawler directory or test your site with the AI Crawler Checker.