1592+ Crawler profiliert
Crawler-Verzeichnis
Alle Web-Crawler und KI-Bots, die wir verfolgen. Was sie tun, wer sie betreibt und wie Sie Ihre Inhalte davor schützen.
Arquivo Web Crawler — What It Is and How to Handle It
Arquivo-web-crawler is the Portuguese web archive's bot that systematically crawls and preserves Portuguese websites for historical research, creating a comprehensive digital heritage of Portugal's web presence.
ArchiverArchiver
Algolia — What It Is and How to Handle It
Algolia crawler extracts website content to power Algolia's search-as-a-service platform, commonly used for site search, documentation search (DocSearch), and e-commerce.
Search EngineSearch Engine
MailRUBot — What It Is and How to Handle It
MailRUBot is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.
PreviewPreview
ICC Crawler — What It Is and How to Handle It
ICC-Crawler automatically crawls the Internet and collects web pages. ICC-Crawler is operated by the Universal Communication Research Institute at the National Institute of Information and Communications Technology (NICT).
AI CrawlerAI Training
uptimedoctor — What It Is and How to Handle It
uptimedoctor is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
SilktideBot — What It Is and How to Handle It
SilktideBot is an accessibility and website quality crawler by Silktide that audits sites for WCAG compliance and content issues.
AccessibilityAccessibility
Quantcastbot — What It Is and How to Handle It
Quantcastbot is Quantcast's web crawler used for advertisement quality assurance and analyzing page content to build Interest-Based Audiences for targeted advertising and audience measurement services.
AdvertisingAdvertising
Bibliotheque Nacional de France Crawler — What It Is and How to Handle It
Bibliotheque Nacional de France Crawler is the web archiving crawler for the National Library of France (BnF), preserving French web heritage.
ResearchResearch
Doctom Monitor — What It Is and How to Handle It
Doctom Monitor is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
Reelevant — What It Is and How to Handle It
Reelavant allows customers to dynamically update content inside their emails. Needs to fetch images of their product at runtime and send them back to the users.
AdvertisingAdvertising
UptimeBot — What It Is and How to Handle It
UptimeBot is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
Jetpack — What It Is and How to Handle It
Jetpack is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
rss2tg_bot — What It Is and How to Handle It
rss2tg bot is an automated service that monitors RSS feeds and sends content updates to Telegram channels or users, enabling real-time notifications for feed content.
Feed ReaderFeed Reader
WOVN Crawler — What It Is and How to Handle It
WOVN Crawler is a website localization crawler by WOVN Technologies that helps translate and localize web content.
AccessibilityAccessibility
Jagged Pixel UptimeBot — What It Is and How to Handle It
Jagged Pixel UptimeBot is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
Bravebot — What It Is and How to Handle It
Bravebot is Brave Search's independent web crawler that discovers and indexes new pages for Brave's search engine, building an independent search index free from Big Tech dependencies.
AI CrawlerAI Training
FlipboardRSS — What It Is and How to Handle It
FlipboardRSS is a web crawler used by Flipboard to discover articles from RSS feeds and generate article summaries for content curation on the Flipboard platform.
Feed ReaderFeed Reader
Bushbaby — What It Is and How to Handle It
Bushbaby is a security scanner that checks websites for vulnerabilities, misconfigurations, and security issues.
SecuritySecurity
Freespoke — What It Is and How to Handle It
Freespoke is a crawler for Freespoke, a US-based search engine focused on unbiased news search and free speech.
Search EngineSearch Engine
Termly — What It Is and How to Handle It
Termly is a compliance crawler that scans websites for cookie usage, privacy policy requirements, and GDPR/CCPA compliance.
MonitoringMonitoring
AdsTxtCrawler — What It Is and How to Handle It
AdsTxtCrawler verifies ads.txt files from publishers to confirm authorized digital advertising seller relationships.
AdvertisingAdvertising
BugsNag — What It Is and How to Handle It
Bugsnag is an error tracking and performance monitoring service that automatically detects application crashes, exceptions, and performance issues, providing developers with detailed diagnostics to improve app stability.
MonitoringMonitoring
Muck Rack Scraper — What It Is and How to Handle It
Muck Rack uses many approaches for source discovery such as RSS feeds, sitemaps, and other structured formats
Feed ReaderFeed Reader
ChatGPT Operator — What It Is and How to Handle It
Agent that can use its own browser to perform tasks for user.
AI CrawlerAI Assistant
WPMUDEV Uptime Monitor 5.0 — What It Is and How to Handle It
WPMUDEV Uptime Monitor 5.0 is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
HostTracker — What It Is and How to Handle It
HostTracker is a website monitoring bot that checks website availability and performance from 265+ monitoring locations worldwide. It monitors websites 24/7 for issues like downtime, slow loading, and broken links.
MonitoringMonitoring
Cloudflare SSLDetector — What It Is and How to Handle It
Cloudflare SSLDetector checks SSL/TLS certificate configurations and validates secure connections for Cloudflare-proxied domains.
SecuritySecurity
ClearscopeBot — What It Is and How to Handle It
Clearscope is an AI-driven SEO content optimization platform developed by Mushi Labs. It assists content creators, marketers, and SEO professionals in producing high-quality, search-optimized content by providing real-time keyword recommendations, content grading, and insights into search intent. By analyzing top-performing content, Clearscope offers actionable suggestions to enhance content relevance and visibility in search engine results.
SEO ToolSEO Tool
Panopta — What It Is and How to Handle It
Panopta (now FortiMonitor) provides digital experience monitoring with end-to-end network and application performance monitoring to improve customer and employee digital experience through uptime checks.
MonitoringMonitoring
Splunk — What It Is and How to Handle It
Splunk is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring