Blog

Funnelback is an enterprise search platform, and its crawler indexes content from an organization's websites and data repositories. This powers the organization's internal search function.

Skroutz ImageBot crawls and fetches product images from e-commerce websites to display in Skroutz's price comparison platform, enabling visual product browsing and comparison for Greek consumers.

The PressEngine Bot verifies coverage created by video games press as genuine and their own creation. When a member of the video games press is granted a review key for a video game they will create an article, known in the industry as "coverage". When they submit a URL to us as "coverage" we automatically verify this URL exists and is viewable. This automated code announces itself as the PressEngine Bot.

Thousand Eyes Cloud Agent is a monitoring crawler that checks website availability, performance, and health.

Parse.ly is a monitoring crawler that checks website availability, performance, and health.

Devin is a software engineering AI assistant that can browse websites and perform web-based tasks, functioning as a collaborative AI teammate for engineering teams.

BinaryCanary monitors website uptime and availability, sending alerts when sites go down or experience performance issues.

SiteUpTimeBot is a monitoring crawler that checks website availability, performance, and health.

Cloudflare Stream Webhook delivers video processing notifications for Cloudflare Stream, a video hosting and streaming service.

The Bazqux Fetcher is how BazQux Reader grabs RSS/Atom feeds and com­ments when users choose to sub­scribe to your blog in BazQux Reader. Fetcher col­lects and pe­ri­od­i­cally re­freshes these user-ini­ti­ated feeds.

Arquivo-web-crawler is the Portuguese web archive's bot that systematically crawls and preserves Portuguese websites for historical research, creating a comprehensive digital heritage of Portugal's web presence.

Algolia crawler extracts website content to power Algolia's search-as-a-service platform, commonly used for site search, documentation search (DocSearch), and e-commerce.

MailRUBot is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.

ICC-Crawler automatically crawls the Internet and collects web pages. ICC-Crawler is operated by the Universal Communication Research Institute at the National Institute of Information and Communications Technology (NICT).

uptimedoctor is a monitoring crawler that checks website availability, performance, and health.

SilktideBot is an accessibility and website quality crawler by Silktide that audits sites for WCAG compliance and content issues.

Quantcastbot is Quantcast's web crawler used for advertisement quality assurance and analyzing page content to build Interest-Based Audiences for targeted advertising and audience measurement services.

Bibliotheque Nacional de France Crawler is the web archiving crawler for the National Library of France (BnF), preserving French web heritage.

Doctom Monitor is a monitoring crawler that checks website availability, performance, and health.

Reelavant allows customers to dynamically update content inside their emails. Needs to fetch images of their product at runtime and send them back to the users.