What is Meta-WebIndexer?
Meta-WebIndexer is the Meta crawler for Meta AI search. Meta says it navigates the web to improve Meta AI search result quality, and that allowing it in robots.txt helps Meta cite and link to your content in Meta AI's responses.
- Operator
- Meta
- Purpose
- Crawls pages to improve the search results of Meta AI.
- robots.txt token
meta-webindexer- Example user agent
meta-webindexer/1.1- How to verify
- Meta's crawler page gives no IP list and no reverse DNS domain for Meta-WebIndexer.
- Operator documentation
- https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- Last verified
How often is Meta-WebIndexer impersonated?
We have not measured requests claiming to be Meta-WebIndexer yet, so we show no share.
This directory measures requests between 3 July 2026 and 28 September 2026. The fake crawler report covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
Have a request that claims to be Meta-WebIndexer? Check its IP address.
Should you block Meta-WebIndexer?
- Allow it if
- you want Meta AI to cite and link to your pages.
- Block it if
- you do not want your pages indexed for Meta AI's answers.
- What blocking changes
- Meta states only the allow side: allowing Meta-WebIndexer helps Meta cite and link to your content in Meta AI's responses.
To block it, add this group to your robots.txt. Meta says to add a disallow for the crawler to robots.txt to block it. Its crawlers may cache robots.txt for up to 24 hours.
User-agent: meta-webindexer
Disallow: /robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Meta-WebIndexer. Stopping those takes verification at your edge.
Related bots
- facebookexternalhit (Meta): Builds link previews when content is shared on Meta's apps.
- Meta-ExternalAgent (Meta): Crawls for training AI models or improving Meta products.
- Meta-ExternalFetcher (Meta): Fetches individual links on behalf of Meta product users.
- Claude-SearchBot (Anthropic): Indexes pages to improve Claude's search results.
- DuckAssistBot (DuckDuckGo): Crawls in real time to support DuckDuckGo's AI-assisted answers.
- MistralAI-Index (Mistral AI): Indexes pages for Mistral search.
See which bots reach your site, and which of them are who they claim to be.
Request a site audit