1592+ Crawler profiliert
Crawler-Verzeichnis
Alle Web-Crawler und KI-Bots, die wir verfolgen. Was sie tun, wer sie betreibt und wie Sie Ihre Inhalte davor schützen.
OpenindexSpider — What It Is and How to Handle It
OpenindexSpider is the crawler for Openindex, a search indexing service for website search functionality.
Search EngineSearch Engine
OpenHoseBot — What It Is and How to Handle It
OpenHoseBot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
OpenGraphCheck — What It Is and How to Handle It
OpenGraphCheck is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
OpenGraph.io — What It Is and How to Handle It
OpenGraph.io is a web crawler that extracts Open Graph metadata, titles, descriptions, and images from web pages to generate rich link previews for social sharing platforms.
PreviewPreview
Openfind data gatherer — What It Is and How to Handle It
Openfind data gatherer is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
openfind — What It Is and How to Handle It
openfind is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
Openbot — What It Is and How to Handle It
Openbot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
OnlineOrNot — What It Is and How to Handle It
OnlineOrNot monitors website uptime and performance, conducting regular checks to detect outages and page speed issues, providing immediate alerts when websites go down or experience problems.
MonitoringMonitoring
OneTrust — What It Is and How to Handle It
OneTrust is a privacy compliance crawler that scans customer websites to identify and categorize cookies, tracking technologies, forms, tags, and storage mechanisms for privacy management purposes.
SecuritySecurity
OnCrawl Mobile — What It Is and How to Handle It
OnCrawl Mobile is an SEO crawler operated by Oncrawl, a technical SEO platform. This mobile variant crawls websites to collect data and analyze site structure, performance, and other SEO-related metrics for website optimization purposes.
SEO ToolSEO Tool
omgilibot — What It Is and How to Handle It
omgilibot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
omgili — What It Is and How to Handle It
omgili is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models.
AI CrawlerAI Training
OhDear.app — What It Is and How to Handle It
OhDear.app is an all-in-one website monitoring bot that checks uptime, SSL certificates, broken links, performance, DNS records, and application health to ensure optimal website availability and functionality.
MonitoringMonitoring
OhDear — What It Is and How to Handle It
OhDear is a website monitoring bot that checks websites for uptime, performance, and other health metrics. It visits websites to perform regular checks and alert website owners about potential issues or downtime.
OtherOther
Offline Explorer — What It Is and How to Handle It
Offline Explorer by MetaProducts downloads websites for offline viewing, supporting scheduled downloads and site mirroring.
ArchiverArchiver
officestorebot — What It Is and How to Handle It
officestorebot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
OdklBot — What It Is and How to Handle It
OdklBot is a crawler that fetches link preview data for the Odnoklassniki social network, extracting Open Graph metadata (titles, descriptions, images) from external links to generate rich previews.
PreviewPreview
Odin — What It Is and How to Handle It
Odin is a web crawler operated by ODIN, an internet scanning search engine platform. The bot scans websites as part of indexing operations that power searches across various datasets including hosts, exposed buckets, exposed files, and subdomains.
SecuritySecurity
Observer — What It Is and How to Handle It
Observer is a web security crawler that scans websites to identify third-party risks, security vulnerabilities, and performance issues. It detects compromised external resources like images, scripts, and links that could turn websites into attack vectors, as well as expired domains purchased by malicious actors and slow-loading resources.
SecuritySecurity
ObjectsSearch — What It Is and How to Handle It
Objects Search Spider for web indexing
Search EngineSearch Engine
nyt_scraping — What It Is and How to Handle It
nyt_scraping is a web scraping bot operated by The New York Times. It is used to collect content from websites for The New York Times' purposes.
ScraperScraper
Nuzzel — What It Is and How to Handle It
Nuzzel is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
Nutch — What It Is and How to Handle It
Apache Nutch is an open-source web crawler used for search engine development, data mining, and large-scale web crawling.
Search EngineSearch Engine
NTENTbot — What It Is and How to Handle It
NTENTbot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
NPBot — What It Is and How to Handle It
NPBot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
NovaAct — What It Is and How to Handle It
Nova Act is an AI agent created by Amazon that can use a web browser. It can intelligently navigate and interact with websites to complete multi-step tasks on behalf of a human user.
AI CrawlerAI Assistant
NostoCrawlerBot — What It Is and How to Handle It
NostoCrawlerBot is operated by Nosto, a company that provides ecommerce personalization solutions including product recommendations, search personalization, and merchandising tools.
ResearchResearch
noorobot — What It Is and How to Handle It
noorobot from Noor Digital Agency periodically crawls customer websites to provide SEO recommendations, identify potential optimization issues, and enhance organic search presence.
SEO ToolSEO Tool
Noibu — What It Is and How to Handle It
Noibu monitors ecommerce websites for performance issues, JavaScript errors, API failures, and checkout disruptions, providing real-time error detection and debugging insights to prevent revenue loss.
MonitoringMonitoring
NoahBot — What It Is and How to Handle It
NoahBot is an uncategorized agent operated by NoahWire. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
OtherOther