Blog

EasyCron is an online cron job service. Users can schedule an HTTP request to be made at a specific date and time.

Blockaid is a Web3 security crawler that scans websites and smart contracts for phishing, scams, and malicious activity to protect cryptocurrency users.

Foregenix perform security and risk scanning on the web sites of eCommerce merchants for a number of banks and card brands globally. The service assists these organisations in controlling and identifying fraud and financial losses, with a particular focus on trying to identify compromised merchants before they end up in the card brand's compromise investigation process. Early detection (prior to fraud losses escalating) can save the banks and merchants alike considerable sums. The solution has two primary modes of operation Scanning for active malware, this normally entails pulling a very limited number of pages within a sandboxed context for analysis at various stages of DOM initialisation. From the target sites perspective, the operation is simply another browser requesting a small number of pages as normal. Scanning for known publicly exploitable vulnerabilities and outdated software solutions as these attributes are frequently exploited by threat actors to introduce malware targeting financial information. Typically a complete scan comprises less than one hundred requests and is already rate limited on our side. Scanning is always "passive" in nature, relying on GET, HEAD and OPTIONS requests only. The scanning heads by default abide by the "robots.txt" file but this can be overridden by the scan initiator (usually one of our banking clients). This override, to force a scan/assessment is not actioned all that frequently.

BestChange Bot is the crawler for BestChange, a cryptocurrency and e-currency exchange rate aggregator and monitoring service.

QualifiedBot is Qualified's web crawler that analyzes customer websites to provide contextual information for their AI-powered chatbots and conversational marketing platform.

WormlyBot is the HTTP monitoring probe and web crawler used by Wormly's uptime monitoring service to check website availability, performance, and server health for continuous monitoring and alerting.

The ManageWP webhooks integration to manage mulitple Wordpress websites with a single dashboard.

Moz rogerbot is Moz's SEO crawler that indexes web pages for link analysis, Domain Authority metrics, and SEO audits.

Marginalia Search is a noncommercial niche search engine focusing on old websites, personal websites, and blogs that suffer crippling discoverability problems in today's fiercely SEO-optimized lanscape.

Easy automation for busy people. Zapier moves info between your web apps automatically, so you can focus on your most important work.

Pocket Casts Feed Parser fetches and parses podcast RSS feeds to deliver content for playback in the Pocket Casts mobile and web applications.

The Google Schema Markup Testing Tool bot, now part of the Rich Results Test, crawls pages to validate their structured data. This helps webmasters check if their schema markup is correctly implemented for Google Search.

Slack Image Proxy is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.

Iframely is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.

The seo4ajax bot is used by a service that helps make single-page applications (SPAs) crawlable by search engines. It pre-renders JavaScript-heavy pages into static HTML so they can be indexed.

Sparkbot webhook integration is used for automating email transactions on web server events.

Uptimia is a monitoring crawler that checks website availability, performance, and health.

Rackspace is a monitoring crawler that checks website availability, performance, and health.

DataForSEO is using RSiteAuditor to scan websites for critical on-site SEO errors and provides aggregated data in a structured form to its customer through a RESTful API.

Analytics and email automation service used by eCommerce businesses. Metorik syncs data from customer sites by making API requests to their sites.