Directorio de Crawlers

The Adyen webhooks integration sends HTTP requests to inform web servers about payment-related events.

Cludo is a monitoring crawler that checks website availability, performance, and health.

MirrorWebCrawler is a web archiving crawler by MirrorWeb that captures and preserves website content for compliance and records management.

Audisto Crawler fetches all accessible URLs of a website. Audisto provides a service to audit and monitor websites for its customers. More information about the crawler is available here: https://audisto.com/bot

Google-AdWords-Express is an automated crawler that analyzes advertiser websites to assist with Google Ads creation and campaign optimization, specifically designed for small business advertising needs and site verification.

Feedbin is an RSS reader service that fetches and aggregates web content, RSS feeds, email newsletters, podcasts, and YouTube videos to provide users with a centralized reading and content consumption experience.

deadlinkchecker is a web crawler service operated by DLC Websites that crawls customer websites to identify and report broken links (404, 500 errors, etc.). It helps website owners maintain site quality and SEO rankings by detecting problematic links.

Overcast is a podcast player application, and its bot fetches RSS feeds and audio files from podcast hosting servers. This keeps the podcast directory and episodes updated for its users.

Yext crawler verifies business listings and local SEO data across their network of over 200 publisher sites and directories.

Retool platform user agent for the business application development platform that helps build internal software, AI agents, workflows and apps using databases, APIs and LLMs.

Let's Encrypt is a security scanner that checks websites for vulnerabilities, misconfigurations, and security issues.

Pro-Sitemaps is a web crawler that automatically scans websites to generate XML sitemaps, analyze site structure, detect broken links, and provide SEO insights for search engine optimization and website maintenance.

Sansec Security Monitor is a specialized e-commerce security scanner that monitors online stores for malware, vulnerabilities, and digital skimming attacks, providing real-time threat detection and forensic analysis.

Payment confirmation callbacks to ecommerce backends

Accessible Web Bot crawls customer websites to discover pages and monitor for accessibility violations on regular basis. Crawls are initiated for Accessible Web's "Page Monitoring" SaaS product.

SeobilityBot crawls websites to gather SEO information and provide comprehensive SEO analysis including website audits, ranking monitoring, and optimization insights to its customers.

netEstate Imprint Crawler extracts legal imprint and contact information from websites for business data collection and compliance analysis.

Blogtrottr is the RSS feed crawler that fetches content from blogs, news feeds, and websites to deliver updates directly to users' email inboxes, providing automated content aggregation and email delivery services.

InternetArchiveBot is a monitoring crawler that checks website availability, performance, and health.

eMoney Advisor is a financial planning platform crawler that aggregates account data for wealth management.

EasyCron is an online cron job service. Users can schedule an HTTP request to be made at a specific date and time.

Blockaid is a Web3 security crawler that scans websites and smart contracts for phishing, scams, and malicious activity to protect cryptocurrency users.

Foregenix perform security and risk scanning on the web sites of eCommerce merchants for a number of banks and card brands globally. The service assists these organisations in controlling and identifying fraud and financial losses, with a particular focus on trying to identify compromised merchants before they end up in the card brand's compromise investigation process. Early detection (prior to fraud losses escalating) can save the banks and merchants alike considerable sums. The solution has two primary modes of operation Scanning for active malware, this normally entails pulling a very limited number of pages within a sandboxed context for analysis at various stages of DOM initialisation. From the target sites perspective, the operation is simply another browser requesting a small number of pages as normal. Scanning for known publicly exploitable vulnerabilities and outdated software solutions as these attributes are frequently exploited by threat actors to introduce malware targeting financial information. Typically a complete scan comprises less than one hundred requests and is already rate limited on our side. Scanning is always "passive" in nature, relying on GET, HEAD and OPTIONS requests only. The scanning heads by default abide by the "robots.txt" file but this can be overridden by the scan initiator (usually one of our banking clients). This override, to force a scan/assessment is not actioned all that frequently.

BestChange Bot is the crawler for BestChange, a cryptocurrency and e-currency exchange rate aggregator and monitoring service.

QualifiedBot is Qualified's web crawler that analyzes customer websites to provide contextual information for their AI-powered chatbots and conversational marketing platform.

WormlyBot is the HTTP monitoring probe and web crawler used by Wormly's uptime monitoring service to check website availability, performance, and server health for continuous monitoring and alerting.

The ManageWP webhooks integration to manage mulitple Wordpress websites with a single dashboard.

Moz rogerbot is Moz's SEO crawler that indexes web pages for link analysis, Domain Authority metrics, and SEO audits.

Marginalia Search is a noncommercial niche search engine focusing on old websites, personal websites, and blogs that suffer crippling discoverability problems in today's fiercely SEO-optimized lanscape.

Easy automation for busy people. Zapier moves info between your web apps automatically, so you can focus on your most important work.