1592+ Crawler profiliert
Crawler-Verzeichnis
Alle Web-Crawler und KI-Bots, die wir verfolgen. Was sie tun, wer sie betreibt und wie Sie Ihre Inhalte davor schützen.
B2B Bot — What It Is and How to Handle It
B2B Bot collects business information from company websites for B2B data services and lead generation.
AggregatorAggregator
awesomecrawler — What It Is and How to Handle It
awesomecrawler is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
AwarioRssBot — What It Is and How to Handle It
AwarioRssBot fetches RSS feeds for Awario's social listening platform to monitor brand mentions and track content updates across blogs and news sites.
Feed ReaderFeed Reader
AwarioBot — What It Is and How to Handle It
AwarioBot crawls over 13 billion web pages daily for Awario's social listening platform, tracking brand mentions, sentiment, and online conversations for reputation management.
MonitoringMonitoring
Awario — What It Is and How to Handle It
Awario is a social media monitoring bot that crawls over 13 billion web pages daily to track brand mentions, conversations, and social media content across platforms for reputation management and market intelligence.
MonitoringMonitoring
Automattic Feed Fetcher — What It Is and How to Handle It
Automattic Feed Fetcher is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
OtherOther
Authory — What It Is and How to Handle It
Authory is an automated content archiving crawler that systematically searches for and backs up published articles, podcasts, and videos by journalists and content creators to create secure portfolios and prevent content loss.
ArchiverArchiver
auramundi — What It Is and How to Handle It
auramundi is a media monitoring crawler that tracks online content and brand mentions.
MonitoringMonitoring
Augure — What It Is and How to Handle It
Augure is a media monitoring and PR analytics crawler for tracking brand mentions and media coverage.
MonitoringMonitoring
AudigentAdBot — What It Is and How to Handle It
AudigentAdBot is a web crawler operated by Audigent that collects metadata from HTML page headers to support targeted advertising solutions. The bot gathers information from websites where Audigent or its advertiser partners may serve ads, helping them bid on ad space and deliver relevant advertisements.
ResearchResearch
Atom Feed Robot — What It Is and How to Handle It
Atom Feed Robot fetches and indexes RSS/Atom feeds for RSSMicro, a feed search engine and aggregation service.
Feed ReaderFeed Reader
atlassian-bot — What It Is and How to Handle It
atlassian-bot is Atlassian's AI web crawler that indexes custom websites for Rovo, allowing the indexed content to appear in Rovo Search results and be used by Rovo Chat and Agents.
AI SearchAI Search
asterias — What It Is and How to Handle It
asterias is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
AspiegelBot — What It Is and How to Handle It
AspiegelBot is a web mirroring and caching crawler that creates snapshots of web pages.
PreviewPreview
asknread.com — What It Is and How to Handle It
asknread.com is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
Ask n read — What It Is and How to Handle It
Ask n read is an advertising technology crawler that analyzes web content for ad targeting, verification, or competitive intelligence.
AdvertisingAdvertising
ArchiveBot — What It Is and How to Handle It
ArchiveBot is a web archiving crawler that saves endangered websites to the Internet Archive, operated by Archive Team volunteers to preserve digital content at risk of deletion.
ArchiverArchiver
archive.org_bot — What It Is and How to Handle It
archive.org_bot is the Internet Archive's web crawler for the Wayback Machine, systematically crawling and preserving publicly accessible web pages for historical record and research.
ArchiverArchiver
Archive-It — What It Is and How to Handle It
Archive-It is a web archiving crawler operated by Internet Archive that preserves copies of web pages for long-term digital preservation and historical record-keeping.
ArchiverArchiver
arabot — What It Is and How to Handle It
arabot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
Aqua_Products — What It Is and How to Handle It
Aqua_Products is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
AppleNewsBot — What It Is and How to Handle It
AppleNewsBot is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
OtherOther
Applebot-Extended — What It Is and How to Handle It
Apple-Extended is used to train Apple’s foundation LLM models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.
AI CrawlerAI Training
AppInsights — What It Is and How to Handle It
AppInsights is Microsoft Azure's application performance monitoring crawler that checks website availability and performance metrics.
MonitoringMonitoring
Apercite — What It Is and How to Handle It
Apercite is a web screenshot and thumbnail generation service that creates visual previews of web pages.
PreviewPreview
AnyEvent — What It Is and How to Handle It
AnyEvent is a Perl-based HTTP client library used for web crawling and async requests.
OtherOther
antibot — What It Is and How to Handle It
antibot is a bot detection and verification crawler that checks for anti-bot measures on websites.
SecuritySecurity
Anomura — What It Is and How to Handle It
Anomura is Direqt's web crawler that discovers and indexes links and metadata from websites for inclusion in Direqt's AI search results.
AI SearchAI Search
Andibot — What It Is and How to Handle It
Andi search engine crawler
AI CrawlerAI Search
AndersPinkBot — What It Is and How to Handle It
AndersPinkBot curates and aggregates content for Anders Pink, a content curation platform that helps teams discover and share relevant industry content.
AggregatorAggregator