What is SemrushBot?
SemrushBot is Semrush's crawler. It discovers and collects new and updated web data for Semrush's tools, including its backlink index. Semrush runs several variants with their own robots.txt tokens, such as SemrushBot-BA and SiteAuditBot.
Last verified 2026-09-30 against the operator's documentation. All bots
SemrushBot at a glance
- Operator
- Semrush
- Type
- SEO and link-index crawlers
- Purpose
- Collects web data for Semrush's SEO tools.
- robots.txt token
SemrushBot- How to verify
- Semrush publishes no IP list and says it does not use consecutive IP blocks, so it asks you not to block it by IP.
- Operator documentation
- https://www.semrush.com/bot/
How often is SemrushBot impersonated?
Our verification data for SemrushBot was incomplete during the measurement window, so a share would overstate impersonation. We show none until it is complete.
This directory measures requests between 3 July 2026 and 28 September 2026. The fake crawler report covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
Have a request that claims to be SemrushBot? Check its IP address.
Should you block SemrushBot?
- Allow it if
- you or your partners use this tool's backlink or audit data, or the crawl load is not a concern.
- Block it if
- the crawl load is noticeable and you have no use for the tool's data. Blocking it does not affect search rankings.
- What blocking changes
- Semrush stops collecting data about your site for its tools. Each variant needs its own robots.txt line.
To block it, add this group to your robots.txt. Semrush says SemrushBot obeys robots.txt and honors Crawl-delay up to 10 seconds.
User-agent: SemrushBot
Disallow: /robots.txt only asks. It does nothing against a client that ignores it or only pretends to be SemrushBot. Stopping those takes verification at your edge.
SemrushBot: common questions
See which bots reach your site, and which of them are who they claim to be.
What is SemrushBot?
SemrushBot is Semrush's crawler. It discovers and collects new and updated web data for Semrush's tools, including its backlink index. Semrush runs several variants with their own robots.txt tokens, such as SemrushBot-BA and SiteAuditBot.
How do I verify that a request is really SemrushBot?
Semrush publishes no IP list and says it does not use consecutive IP blocks, so it asks you not to block it by IP. The user agent alone proves nothing: any client can send it.
How do I block SemrushBot in robots.txt?
Add a group for User-agent: SemrushBot with Disallow: /. Semrush says SemrushBot obeys robots.txt and honors Crawl-delay up to 10 seconds. robots.txt only asks. A client that ignores it, or pretends to be SemrushBot, has to be stopped at your edge.
What happens if I block SemrushBot?
Semrush stops collecting data about your site for its tools. Each variant needs its own robots.txt line.
Related bots
- AhrefsBot (Ahrefs): Builds the link database behind Ahrefs and the Yep search engine.
- MJ12bot (Majestic): Maps links between websites for Majestic's link index.
- DotBot (Moz): Gathers data for the Moz Link Index.
- DataForSeoBot (DataForSEO): Adds links to DataForSEO's backlink database.
- Barkrowler (Babbar): Collects links and page metadata for Babbar.