1592+ crawlers perfilados
Directorio de Crawlers
Todos los crawlers web y bots de IA que rastreamos. Qué hace cada uno, quién lo opera y cómo proteger su contenido.
HetrixTools — What It Is and How to Handle It
HetrixTools offers an Uptime Monitoring service, where our monitoring locations (bots) will check whether our users' websites are online or not. Similar to more known services such as Pingdom or UptimeRobot.
MonitoringMonitoring
Seznam — What It Is and How to Handle It
Seznam is the crawler for Seznam.cz, the Czech Republic's largest domestic search engine with 50%+ market share.
Search EngineSearch Engine
Bing Ads — What It Is and How to Handle It
AdIdxBot is the crawler used by Bing Ads. AdIdxBot crawls ads and follows the websites from those ads for quality control. Just like Bingbot, AdIdxBot has both “desktop” and “mobile” variants.
AdvertisingAdvertising
catchpoint — What It Is and How to Handle It
Catchpoint is a synthetic monitoring agent that tests website performance, availability, and functionality from global monitoring locations as part of Catchpoint's Internet Performance Monitoring platform.
MonitoringMonitoring
UptrendsBot — What It Is and How to Handle It
UptrendsBot is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
SiteAuditBot — What It Is and How to Handle It
SiteAuditBot crawls websites to analyze over 130 SEO and technical issues, examining crawlability, markups, internal linking, performance, HTTPS, and international SEO factors for optimization insights.
SEO ToolSEO Tool
DuckDuckBot — What It Is and How to Handle It
DuckDuckBot crawls websites to supplement DuckDuckGo's privacy-focused search results, working alongside data from Bing and other sources.
Search EngineSearch Engine
Seekport — What It Is and How to Handle It
Seekport is an internet search engine. Originally founded in 2003, the search engine has been operated by SISTRIX, a platform intelligence provider from Bonn (Germany), since December 2014. The search engine is a public, free and independent alternative to Google. Seekport does not store user data and does not profile users. Seekport is also operated without advertising and has no conflicts of interest in the display of search results.
Search EngineSearch Engine
Cloudflare-Traffic-Manager — What It Is and How to Handle It
Cloudflare-Traffic-Manager monitors and routes traffic for Cloudflare's load balancing and traffic steering services.
MonitoringMonitoring
StatusCake — What It Is and How to Handle It
StatusCake monitors website uptime, page speed, SSL certificates, and domain expiration from 48 global locations.
MonitoringMonitoring
elmah.io Uptime Monitoring — What It Is and How to Handle It
elmah.io Uptime Monitoring is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
Google-CloudVertexBot — What It Is and How to Handle It
Google-CloudVertexBot is Google's AI/ML platform crawler for Vertex AI, used for training and fine-tuning machine learning models.
AI CrawlerAI Training
LeikiBot — What It Is and How to Handle It
Leikibot is a web crawler operated by DoubleVerify that analyzes webpage content for digital advertising verification, brand safety, and ad fraud detection services.
AdvertisingAdvertising
Better Uptime — What It Is and How to Handle It
Better Uptime by Better Stack monitors website availability and performance, providing instant alerts, status pages, and incident management for DevOps teams.
MonitoringMonitoring
Perplexity‑User — What It Is and How to Handle It
Perplexity-User supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response.
AI CrawlerAI Assistant
HubSpot Page Fetcher — What It Is and How to Handle It
HubSpot Page Fetcher crawls web pages to collect content and metadata for social media posting and content optimization within the HubSpot platform.
PreviewPreview
Taboola — What It Is and How to Handle It
Taboola crawler analyzes page content and metadata for Taboola's content recommendation and native advertising platform.
AdvertisingAdvertising
Twilio Proxy — What It Is and How to Handle It
Twilio webhook requests triggered by Twilio when there are incoming SMSes, calls, etc.
OtherOther
Siteimprove Crawl — What It Is and How to Handle It
Siteimprove Crawl analyzes websites for accessibility, SEO, and content quality as part of Siteimprove's digital optimization platform.
SEO ToolSEO Tool
Slackbot — What It Is and How to Handle It
Slackbot fetches URL metadata to generate rich link previews (unfurls) when users share links in Slack channels and direct messages.
PreviewPreview
Cookiebot — What It Is and How to Handle It
Cookiebot is a compliance scanner that automatically crawls websites to detect and categorize cookies and tracking technologies, ensuring GDPR compliance by identifying data collection practices for consent management.
SecuritySecurity
PerplexityBot — What It Is and How to Handle It
PerplexityBot crawls websites to provide real-time answers for Perplexity AI, an AI-powered search engine that cites sources in its responses.
AI CrawlerAI Training
YahooMailProxy — What It Is and How to Handle It
Yahoo Mail Proxy is a content fetch proxy that retrieves the page content of URLs that are embedded within emails sent to Yahoo Mail users. Having the content displayed through the proxy improves the security for email users while reducing overall network usage.
PreviewPreview
HubSpot Feed Fetcher — What It Is and How to Handle It
HubSpot Feed Fetcher is a feed reader that fetches and processes RSS, Atom, and other content feeds.
Feed ReaderFeed Reader
Google Site Verification — What It Is and How to Handle It
Verification is the process of proving that you own the property that you claim to own. Search Console needs to verify ownership because verified owners have access to sensitive Google Search data for a site, and can affect a site's presence and behavior on Google Search and other Google properties. A verified owner can grant full or view access to other people.
SecuritySecurity
Make Platform — What It Is and How to Handle It
Make is a no-code platform to help with task and workflow automation.
OtherOther
Ahrefs Site Audit — What It Is and How to Handle It
Ahrefs Site Audit is a diagnostic crawler that analyzes websites for technical SEO issues, broken links, performance problems, and on-page optimization opportunities.
SEO ToolSEO Tool
Stripe — What It Is and How to Handle It
The Stripe Webhooks service allows Stripe to push real-time event data to customers' application webhook endpoint when events happen in their Stripe account.
OtherOther
Site24x7 — What It Is and How to Handle It
Site24x7 monitors websites, servers, and applications for uptime and performance, providing synthetic monitoring and real user monitoring.
MonitoringMonitoring
Sucuri — What It Is and How to Handle It
Sucuri crawls websites to detect malware, security vulnerabilities, and blacklist status as part of its cloud-based website security platform and malware removal services.
SecuritySecurity