1592+ Crawler profiliert
Crawler-Verzeichnis
Alle Web-Crawler und KI-Bots, die wir verfolgen. Was sie tun, wer sie betreibt und wie Sie Ihre Inhalte davor schützen.
AfterShip Magento 2 Connector — What It Is and How to Handle It
Use Magento 2 connector for AfterShip to track your orders and get delivery
OtherOther
MagnetmeBot — What It Is and How to Handle It
MagnetmeBot is the crawler for Magnet.me, a career platform that matches professionals with companies.
AggregatorAggregator
Tumblr — What It Is and How to Handle It
Tumblr is Automattic's link unfurling bot that fetches URL content to create rich Link Blocks in Tumblr posts, parsing responses to display titles, descriptions, and preview images for shared links.
PreviewPreview
WPMU DEV Broken Link Checker — What It Is and How to Handle It
WPMU DEV Broken Link Checker is a WordPress plugin crawler that performs comprehensive site scans to identify and report broken links, helping developers maintain site quality and user experience.
MonitoringMonitoring
SEO Audit Check Bot — What It Is and How to Handle It
seo-audit-check-bot is a web crawler used by SEO audit platforms like SEO Audit Check and WebCEO to perform comprehensive SEO analysis and audits on websites for optimization insights.
SEO ToolSEO Tool
upday — What It Is and How to Handle It
upday crawls news websites and sources to aggregate articles for its news app platform, collecting and indexing content to provide personalized news recommendations.
AggregatorAggregator
WebsitePulseBot — What It Is and How to Handle It
WebsitePulseBot is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
Watchbot — What It Is and How to Handle It
Watchbot is a monitoring service that performs automated checks on website availability and SSL certificate expiration dates, alerting users to potential downtime or security certificate issues before they affect end users.
MonitoringMonitoring
ActiveComply Bot — What It Is and How to Handle It
ActiveComply Bot is a crawler for a service that monitors social media for regulatory compliance. It scans for specific keywords and content for businesses in regulated industries.
MonitoringMonitoring
NewsBlur — What It Is and How to Handle It
NewsBlur is a web crawler that fetches RSS feeds and web content for its personal news reader service, collecting articles from subscribed websites and blogs to deliver real-time updates to users' newsfeeds.
Feed ReaderFeed Reader
Modular DS — What It Is and How to Handle It
Service to manage WordPress websites via the WP JSON API.
OtherOther
Omnisend — What It Is and How to Handle It
Bot for checking omnisend integration in woocommerce shops and shopify
OtherOther
Drata Autopilot — What It Is and How to Handle It
Drata Autopilot is a security scanner that checks websites for vulnerabilities, misconfigurations, and security issues.
SecuritySecurity
Sentry — What It Is and How to Handle It
Sentry is a security scanner that checks websites for vulnerabilities, misconfigurations, and security issues.
SecuritySecurity
PingPing Bot — What It Is and How to Handle It
PingPing Bot is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
MonitoRSS — What It Is and How to Handle It
This is for the official public hosting of the open-source project https://github.com/synzen/MonitoRSS so that the bot may poll for RSS feeds of Cloudflare-protected sites to deliver news articles. Feeds are chosen by paid users, and the bot adds them to a schedule to be polled at a regular interval of 2-10 minutes.
Feed ReaderFeed Reader
LinkChecker Bot — What It Is and How to Handle It
LinkChecker Bot validates links and finds broken URLs across websites for link auditing and SEO analysis.
SEO ToolSEO Tool
SecurityHeaders — What It Is and How to Handle It
SecurityHeaders is a free security scanning service that analyzes HTTP security headers on websites, providing grades and recommendations to improve web security posture and protect against common vulnerabilities.
SecuritySecurity
Instapaper — What It Is and How to Handle It
Instapaper is a content fetcher that crawls and downloads web articles to make them available for offline reading. It processes and parses web pages to extract readable text content for the Instapaper read-later service.
OtherOther
Jumio — What It Is and How to Handle It
Jumio provides AI-driven digital identity verification and fraud detection services with biometric technology, document verification, and liveness detection for secure user onboarding and compliance.
SecuritySecurity
Funnelback — What It Is and How to Handle It
Funnelback is an enterprise search platform, and its crawler indexes content from an organization's websites and data repositories. This powers the organization's internal search function.
Search EngineSearch Engine
Skroutz ImageBot — What It Is and How to Handle It
Skroutz ImageBot crawls and fetches product images from e-commerce websites to display in Skroutz's price comparison platform, enabling visual product browsing and comparison for Greek consumers.
OtherOther
PressEngine Bot — What It Is and How to Handle It
The PressEngine Bot verifies coverage created by video games press as genuine and their own creation. When a member of the video games press is granted a review key for a video game they will create an article, known in the industry as "coverage". When they submit a URL to us as "coverage" we automatically verify this URL exists and is viewable. This automated code announces itself as the PressEngine Bot.
PreviewPreview
Thousand Eyes Cloud Agent — What It Is and How to Handle It
Thousand Eyes Cloud Agent is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
Parse.ly — What It Is and How to Handle It
Parse.ly is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
Devin — What It Is and How to Handle It
Devin is a software engineering AI assistant that can browse websites and perform web-based tasks, functioning as a collaborative AI teammate for engineering teams.
AI CrawlerAI Assistant
BinaryCanary — What It Is and How to Handle It
BinaryCanary monitors website uptime and availability, sending alerts when sites go down or experience performance issues.
MonitoringMonitoring
SiteUpTimeBot — What It Is and How to Handle It
SiteUpTimeBot is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
Cloudflare Stream Webhook — What It Is and How to Handle It
Cloudflare Stream Webhook delivers video processing notifications for Cloudflare Stream, a video hosting and streaming service.
OtherOther
Bazqux — What It Is and How to Handle It
The Bazqux Fetcher is how BazQux Reader grabs RSS/Atom feeds and comments when users choose to subscribe to your blog in BazQux Reader. Fetcher collects and periodically refreshes these user-initiated feeds.
Feed ReaderFeed Reader