1592+ Crawler profiliert
Crawler-Verzeichnis
Alle Web-Crawler und KI-Bots, die wir verfolgen. Was sie tun, wer sie betreibt und wie Sie Ihre Inhalte davor schützen.
Download Ninja — What It Is and How to Handle It
Download Ninja is an aggressive download manager that scrapes websites for files and content.
ScraperScraper
Download Express — What It Is and How to Handle It
MetaProducts download manager that fetches files from web and FTP sites
ScraperScraper
Dow Jones Searchbot — What It Is and How to Handle It
Dow Jones Searchbot is a crawler operated by Dow Jones for indexing financial and business news content.
Search EngineSearch Engine
DotBot — What It Is and How to Handle It
DotBot is the crawler for Moz (formerly SEOmoz), collecting data for their SEO tools including link analysis and domain authority metrics.
SEO ToolSEO Tool
DomainStatsBot — What It Is and How to Handle It
DomainStatsBot collects domain metrics and website analytics for DomainStats, a domain research and valuation service.
SEO ToolSEO Tool
Domains Project — What It Is and How to Handle It
Domains Project is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
OtherOther
DomainCrawler — What It Is and How to Handle It
DomainCrawler is an SEO crawler that analyzes domain information, backlinks, and website structure.
SEO ToolSEO Tool
Domain Re-Animator Bot — What It Is and How to Handle It
Domain Re-Animator Bot scrapes website content, often associated with expired domain recovery services.
ScraperScraper
Do Not Track Verifier — What It Is and How to Handle It
Do Not Track Verifier is operated by the Electronic Frontier Foundation to test whether websites properly implement Do Not Track (DNT) browser signals. The bot checks if sites honor user privacy preferences by respecting DNT headers sent by web browsers.
OtherOther
DnyzBot — What It Is and How to Handle It
DnyzBot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
DNSScanner — What It Is and How to Handle It
DNSScanner is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
OtherOther
DnBCrawler-Analytics — What It Is and How to Handle It
DnBCrawler-Analytics is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
OtherOther
DMBrowser — What It Is and How to Handle It
DMBrowser is Dotcom-Monitor's web monitoring bot that checks websites for uptime and performance issues, simulating real user interactions to detect problems and ensure optimal site functionality.
MonitoringMonitoring
dlvr.it — What It Is and How to Handle It
dlvr.it is a content delivery and social media automation service that fetches content from websites to automatically share and distribute it across various social media platforms and RSS feeds.
OtherOther
DLC — What It Is and How to Handle It
DLC is the web crawler for Dr. Link Check service that automatically scans websites to identify broken links, malicious links, and other link-related issues, helping website owners maintain link integrity and SEO performance.
MonitoringMonitoring
DittoSpyder — What It Is and How to Handle It
DittoSpyder is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
Disqus — What It Is and How to Handle It
Disqus is the crawler for the Disqus commenting platform, fetching page metadata for comment embedding and social features.
Social MediaSocial Media
Discordbot — What It Is and How to Handle It
Discordbot fetches URL metadata to generate link embeds when users share links in Discord servers and direct messages.
PreviewPreview
discobot — What It Is and How to Handle It
discobot is a crawler related to Discourse forum software or Discord chat platform integrations.
Social MediaSocial Media
Digincore bot — What It Is and How to Handle It
Digincore bot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
Digimind — What It Is and How to Handle It
Digimind is a social listening and competitive intelligence crawler for brand monitoring and market analysis.
MonitoringMonitoring
DigiCert DCV Bot — What It Is and How to Handle It
DigiCert DCV Bot performs domain control validation checks to verify domain ownership during TLS/SSL certificate issuance and renewal processes for DigiCert certificates.
SecuritySecurity
Digg Deeper — What It Is and How to Handle It
Digg Deeper was the crawler for Digg, a social news aggregation website.
OtherOther
Diffbot — What It Is and How to Handle It
Diffbot is an intelligent web crawler used to understand, aggregate, and ultimately sell structured website data for real-time monitoring and training other AI models.
AI CrawlerAI Assistant
DeuSu — What It Is and How to Handle It
DeuSu is a German search engine crawler focused on privacy-respecting search.
Search EngineSearch Engine
Determ — What It Is and How to Handle It
Determ is a media monitoring crawler (formerly Mediatoolkit) that tracks brand mentions across web and social media.
MonitoringMonitoring
DeepSeekBot — What It Is and How to Handle It
DeepSeekBot is the crawler for DeepSeek, a Chinese AI company that develops large language models for search and reasoning tasks.
AI CrawlerAI Assistant
deepnoc — What It Is and How to Handle It
deepnoc is a data collection crawler of unknown purpose.
ScraperScraper
deepcrawl.com — What It Is and How to Handle It
deepcrawl.com is Lumar's website crawler bot that analyzes customer websites to collect performance analytics, identify technical SEO issues, and provide optimization recommendations for improved search engine visibility.
SEO ToolSEO Tool
dcrawl — What It Is and How to Handle It
dcrawl is an open-source distributed web crawler written in D programming language for custom web scraping.
ScraperScraper