Directorio de Crawlers

AwarioSmartBot is a web crawlers sent by Awario to discover and collect new and updated web data (that is further used by Internet marketers from all over the world).

The Library of Congress Web Archive manages, preserves, and provides access to archived web content selected by subject experts from across the Library, so that it will be available for researchers today and in the future. More information on the programme here: https://www.loc.gov/programs/web-archiving/about-this-program/ And information about crawling policy here: https://www.loc.gov/programs/web-archiving/for-site-owners/

FlipboardProxy crawls and fetches content from publisher websites to format articles for display in the Flipboard social magazine app.

Coveo provide services to website, customer service and commerce solutions so they can feature relevant experiences to their end users; said services are based on a unified index which crawls websites when configured so by our customers.

Telegram Bot is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.

Caliperbot crawls Conductor clients' and prospects' websites for HTML feature extraction to power Content Analytics features within our Searchlight web application.

Ghost Inspector is an automated browser testing bot that crawls websites and web applications to perform end-to-end testing, monitoring functionality and detecting bugs through scheduled test runs.

Qwantbot is the web crawler for Qwant, the French privacy-focused search engine. Based in Europe, it indexes websites while prioritizing user privacy and data protection, offering search results without tracking users.

Cxense crawler collected content for Cxense's content recommendation and advertising platform (now part of Piano).

SearchAtlas Bot crawls customer websites to audit technical SEO issues, analyze content optimization opportunities, and provide AI-powered SEO recommendations for improving search engine rankings.

MonSpark monitors website availability, network conditions, SSL certificate validity, and server performance, providing uptime checks and page speed tests to ensure optimal website functionality.

HubSpot offers a full platform of marketing, sales, customer service, and CRM software — plus the methodology, resources, and support — to help businesses grow better. Get started with free tools, and upgrade as you grow.

Oh Dear is a monitoring crawler that checks website availability, performance, and health.

Autopilot is an SEO marketing automation tool that includes features for internal linking and image optimization. We crawl customer sites so that we can determine the best links to use on the site and to find images that need to be optimized.

Sentry Uptime Monitoring is a monitoring crawler that checks website availability, performance, and health.

Yahoo Ad monitoring crawls webpages where Yahoo advertisements are served to monitor content quality, ad placement compliance, and advertiser policy adherence.

Mediatoolkitbot is Determ's media monitoring crawler that scans the open internet searching for brand mentions, keywords, and phrases that users track, helping marketers identify relevant opportunities and monitor brand reputation.

ias_crawler from Integral Ad Science performs digital ad verification, brand safety monitoring, and fraud detection across digital advertising channels to ensure quality media measurement.

Google Feed Fetcher is a feed reader that fetches and processes RSS, Atom, and other content feeds.

Feedly is an RSS reader bot that fetches and aggregates content from blogs, news sites, and newsletters. It provides threat intelligence, market intelligence, and news reading services by crawling and collecting articles from various sources.

Freshping is a website uptime monitoring service by Freshworks that checks site availability and response times.

NixStatsMonitoringBot is a monitoring crawler that checks website availability, performance, and health.

Mojeek is an independent UK-based search engine crawler with its own index, focused on privacy and unbiased results.

Innologica is a feed reader that fetches and processes RSS, Atom, and other content feeds.

Yahoo! Slurp crawls and indexes web content for Yahoo's search engine, collecting data to power search results and web discovery features.

VaultPress is a subscription service developed by Automattic, the company behind WordPress, that offers automated daily and real-time backups of WordPress websites onto WordPress.com's cloud servers. It is known for its ease of use, secure backups, and proactive security scanning.

The PayPal webhooks is part of Paypal's Instant Payment Notification message service, automatically notifying merchants of events related to Paypal transactions.

LinkedInBot fetches page metadata to generate link previews when users share URLs on LinkedIn, extracting titles, descriptions, and images.

DuckAssistBot is a web crawler that scans websites to collect content for DuckDuckGo's AI-assisted answers feature, which generates brief responses to search queries using natural language technology and Wikipedia sources.

Monsidobot is a website scanning crawler that analyzes web content for accessibility compliance, quality assurance, and SEO issues as part of Acquia Optimize platform.