1592+ crawlers perfilados
Directorio de Crawlers
Todos los crawlers web y bots de IA que rastreamos. Qué hace cada uno, quién lo opera y cómo proteger su contenido.
Pocket Casts Feed Parser — What It Is and How to Handle It
Pocket Casts Feed Parser fetches and parses podcast RSS feeds to deliver content for playback in the Pocket Casts mobile and web applications.
Feed ReaderFeed Reader
Google Schema Markup Testing Tool — What It Is and How to Handle It
The Google Schema Markup Testing Tool bot, now part of the Rich Results Test, crawls pages to validate their structured data. This helps webmasters check if their schema markup is correctly implemented for Google Search.
MonitoringMonitoring
Slack Image Proxy — What It Is and How to Handle It
Slack Image Proxy is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.
PreviewPreview
Iframely — What It Is and How to Handle It
Iframely is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.
PreviewPreview
seo4ajax — What It Is and How to Handle It
The seo4ajax bot is used by a service that helps make single-page applications (SPAs) crawlable by search engines. It pre-renders JavaScript-heavy pages into static HTML so they can be indexed.
SEO ToolSEO Tool
SparkpostBot — What It Is and How to Handle It
Sparkbot webhook integration is used for automating email transactions on web server events.
OtherOther
Uptimia — What It Is and How to Handle It
Uptimia is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
Rackspace — What It Is and How to Handle It
Rackspace is a monitoring crawler that checks website availability, performance, and health.
MonitoringMonitoring
DataForSEO — What It Is and How to Handle It
DataForSEO is using RSiteAuditor to scan websites for critical on-site SEO errors and provides aggregated data in a structured form to its customer through a RESTful API.
SEO ToolSEO Tool
Metorik — What It Is and How to Handle It
Analytics and email automation service used by eCommerce businesses. Metorik syncs data from customer sites by making API requests to their sites.
OtherOther
Clickagy — What It Is and How to Handle It
Clickagy is a B2B intent data crawler that analyzes website visitor behavior for advertising targeting and lead generation.
AdvertisingAdvertising
GTmetrix — What It Is and How to Handle It
GTmetrix is a website performance analysis bot that crawls and analyzes web pages to measure speed performance. Using PageSpeed and Lighthouse, GTmetrix generates performance scores and provides actionable recommendations to optimize website loading times.
MonitoringMonitoring
Cloudflare Custom Hostname Verification — What It Is and How to Handle It
Cloudflare Custom Hostname Verification validates SSL certificates and DNS settings for Cloudflare for SaaS custom hostnames.
MonitoringMonitoring
Bluesky Link Preview Service — What It Is and How to Handle It
Bluesky Link Preview Service is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.
PreviewPreview
Skype — What It Is and How to Handle It
Skype is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.
PreviewPreview
NETVIGIE — What It Is and How to Handle It
NETVIGIE monitoring bot checks website availability and performance metrics for clients, providing uptime monitoring and performance analysis services.
MonitoringMonitoring
Feeder — What It Is and How to Handle It
Feeder is a feed reader that fetches and processes RSS, Atom, and other content feeds.
Feed ReaderFeed Reader
Hotjar — What It Is and How to Handle It
Hotjar analyzes website user behavior through heatmaps, session recordings, surveys, and feedback tools to help optimize user experience and conversions.
MonitoringMonitoring
Protopage — What It Is and How to Handle It
Protopage is an RSS reader and web portal service that crawls and indexes RSS news feeds from various sources to create personalized dashboard start pages and aggregated content for users.
Feed ReaderFeed Reader
Google Inspection Tool — What It Is and How to Handle It
Google-InspectionTool is the crawler used by Search testing tools such as the Rich Result Test and URL inspection in Search Console. Apart from the user agent and user agent token, it mimics Googlebot.
SecuritySecurity
Nodeping — What It Is and How to Handle It
NodePing is an uptime monitoring service that performs HTTP checks and other server monitoring tests to verify website availability and alert customers when services go down.
MonitoringMonitoring
SiteLock — What It Is and How to Handle It
SiteLock is a security scanner that checks websites for vulnerabilities, misconfigurations, and security issues.
SecuritySecurity
Factset_spyderbot — What It Is and How to Handle It
factset_spyderbot is a web crawler operated by FactSet that gathers financial and business data from websites to support their comprehensive financial data platform and analytics services.
AI CrawlerAI Training
ChargeBee — What It Is and How to Handle It
Chargebee provides a webhooks integration to notify web severs of payment events.
OtherOther
Svix Webhooks — What It Is and How to Handle It
Scalable webhook platform featuring automatic retries, signature verification, deep observability, and a static-IP delivery bot—deploy hosted or self-hosted.
OtherOther
Google Videos — What It Is and How to Handle It
Google Videos indexes video content for Google Video Search and YouTube, extracting thumbnails, metadata, and structured data.
Search EngineSearch Engine
MSN — What It Is and How to Handle It
MSNBot was the web crawler for Microsoft's MSN Search, which has since been replaced by Bing. Its purpose was to index web pages for inclusion in the MSN search engine.
Search EngineSearch Engine
klaviyo — What It Is and How to Handle It
Klaviyo is a web tracking script that collects visitor behavior data, product views, and site interactions to enable personalized email and SMS marketing campaigns through its B2C customer relationship management platform.
AdvertisingAdvertising
BlogVault — What It Is and How to Handle It
BlogVault is a WordPress backup and monitoring service bot that performs automated backups, security scans, and site monitoring to protect WordPress websites from data loss and downtime.
MonitoringMonitoring
Outbrain — What It Is and How to Handle It
outbrain is Outbrain's web crawler that analyzes content on publisher websites to understand context and topics for content recommendation and advertising purposes.
AdvertisingAdvertising