Blog

The Adyen webhooks integration sends HTTP requests to inform web servers about payment-related events.

Cludo is a monitoring crawler that checks website availability, performance, and health.

MirrorWebCrawler is a web archiving crawler by MirrorWeb that captures and preserves website content for compliance and records management.

Audisto Crawler fetches all accessible URLs of a website. Audisto provides a service to audit and monitor websites for its customers. More information about the crawler is available here: https://audisto.com/bot

Google-AdWords-Express is an automated crawler that analyzes advertiser websites to assist with Google Ads creation and campaign optimization, specifically designed for small business advertising needs and site verification.

Feedbin is an RSS reader service that fetches and aggregates web content, RSS feeds, email newsletters, podcasts, and YouTube videos to provide users with a centralized reading and content consumption experience.

deadlinkchecker is a web crawler service operated by DLC Websites that crawls customer websites to identify and report broken links (404, 500 errors, etc.). It helps website owners maintain site quality and SEO rankings by detecting problematic links.

Overcast is a podcast player application, and its bot fetches RSS feeds and audio files from podcast hosting servers. This keeps the podcast directory and episodes updated for its users.

Yext crawler verifies business listings and local SEO data across their network of over 200 publisher sites and directories.

Retool platform user agent for the business application development platform that helps build internal software, AI agents, workflows and apps using databases, APIs and LLMs.

Let's Encrypt is a security scanner that checks websites for vulnerabilities, misconfigurations, and security issues.

Pro-Sitemaps is a web crawler that automatically scans websites to generate XML sitemaps, analyze site structure, detect broken links, and provide SEO insights for search engine optimization and website maintenance.

Sansec Security Monitor is a specialized e-commerce security scanner that monitors online stores for malware, vulnerabilities, and digital skimming attacks, providing real-time threat detection and forensic analysis.

Payment confirmation callbacks to ecommerce backends

Accessible Web Bot crawls customer websites to discover pages and monitor for accessibility violations on regular basis. Crawls are initiated for Accessible Web's "Page Monitoring" SaaS product.

SeobilityBot crawls websites to gather SEO information and provide comprehensive SEO analysis including website audits, ranking monitoring, and optimization insights to its customers.

netEstate Imprint Crawler extracts legal imprint and contact information from websites for business data collection and compliance analysis.

Blogtrottr is the RSS feed crawler that fetches content from blogs, news feeds, and websites to deliver updates directly to users' email inboxes, providing automated content aggregation and email delivery services.

InternetArchiveBot is a monitoring crawler that checks website availability, performance, and health.

eMoney Advisor is a financial planning platform crawler that aggregates account data for wealth management.