Directorio de Crawlers

Detectify is an application security testing platform that continuously scans web applications and attack surfaces for vulnerabilities, combining automated DAST methods with crowdsourced ethical hacker research to identify security flaws.

CriteoBot is Criteo's web crawler that visits and analyzes web pages to understand content and user behavior patterns for serving personalized advertising campaigns.

Google AdSense crawler verifies website content for AdSense eligibility and checks that ad placements comply with Google's publisher policies.

Pingdom monitors website uptime, performance, and page speed from global locations, part of SolarWinds' application performance monitoring suite.

Onetrust CMP Scanner is a monitoring crawler that checks website availability, performance, and health.

A Twitter bot is a type of bot software that controls a Twitter account via the Twitter API. The bot software may autonomously perform actions such as tweeting, re-tweeting, liking, following, unfollowing, or direct messaging other accounts.

DataForSEO Bot is a driving force of our leading product - Backlinks API, which has been developed with a single purpose: providing website owners, webmasters, and SEO professionals with opportunities to analyze the key component of website optimization – backlink analytics. You can learn more about the DataForSEO Bot on this dedicated page: https://dataforseo.com/dataforseo-bot

SEO PowerSuite Link Explorer (webmeup.com) is the world's freshest backlink index, and the primary source of backlink-related data for the SEO PowerSuite tools. We're dedicated to providing SEOs with the most comprehensive, up-to-date backlink data on the Web.

Microsoft Preview is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.

botify's SiteCrawler analyzes website structure and content like a search engine, giving enterprise SEO teams detailed insights into how their sites are crawled and indexed.

Baiduspider is the main crawler for Baidu, China's largest search engine with 60-80% market share, essential for visibility in the Chinese market.

Shopify-Captain-Hook is a system used by Shopify to deliver webhooks. It sends automated messages to a user's server when specific events occur within their Shopify store.

ContentKing is a cloud-based service that monitors websites from a digital marketing perspective. We monitor the websites for customers such as Netflix, Atlassian, Fedex and IBM and alert their digital marketing teams whenever a technical issue or content change is detected.

Tools and product functions where the end user triggers a fetch.

proximic is a contextual advertising crawler that analyzes website content to provide advanced contextual intelligence for programmatic advertising targeting and media optimization.

SemrushBot is the web crawler and search bot software that Semrush sends out to discover and collect web data for their SEO platform including backlink analytics, site audit, content research and SEO tools.

Barkrowler is an SEO web crawler operated by Babbar that analyzes websites to identify page popularity, topical relevance, and authority metrics for search engine optimization and competitive analysis research.

OAI-SearchBot is OpenAI's web crawler that indexes websites for SearchGPT, collecting and analyzing web content to power AI-driven search results and real-time information retrieval in OpenAI's search applications.

Deepcrawl (now Lumar) is an enterprise SEO crawler that audits websites for technical SEO issues.

MJ12bot is the web crawler for Majestic. MJ12Bot does not currently cache web content or personal data. Instead it maps the link relationships between websites to build a search engine. This data is available to technologies and the public, either by searching for a keyword or a website at Majestic.

Cloudflare Healthchecks monitors origin server health to route traffic away from unhealthy servers in load balancing configurations.

APIs-Google is the user agent used by Google APIs to deliver push notification messages. Application developers can request these notifications to avoid the need for continually polling Google's servers to find out if the resources they are interested in have changed. To make sure nobody abuses this service, Google requires developers to prove that they own the domain before allowing them to register a URL with a domain as the location where they want to receive messages.

The Google StoreBot is a search-engine-based program that automatically 'crawls' through web pages to gather and analyse data. Google uses crawlers that go through product pages and checkout processes using machine learning algorithms to fill in forms with information such as delivery addresses, and help compile other information on price, delivery, payments and more.

Prerender is a web crawler that pre-renders Single Page Applications (SPAs) into static HTML to improve SEO visibility for JavaScript-heavy websites and search engine indexing.

GoogleUserTriggeredFetcher is triggered when a user requests Google to translate or cache a page. It acts on behalf of real human users, not autonomous crawling.

Applebot data is used to power various features, such as the search technology that is integrated into many user experiences in Appleʼs ecosystem including Spotlight, Siri, and Safari.

Google-Read-Aloud fetches web content to provide text-to-speech functionality, enabling users to have web pages read aloud through Google's accessibility services.

Google Images (Googlebot-Image) indexes images for Google Image Search, analyzing alt text, surrounding content, and image metadata.

ChatGPT-User fetches web content in real-time when ChatGPT users browse the web or request current information, distinct from training data collection.

Dotbot is Moz's web crawler, it gathers web data for the Moz Link Index. This data we collect through Dotbot is available in the Links section of your Moz Pro campaign, Link Explorer, and the Moz Links API.