Blog

Baiduspider is the main crawler for Baidu, China's largest search engine with 60-80% market share, essential for visibility in the Chinese market.

Shopify-Captain-Hook is a system used by Shopify to deliver webhooks. It sends automated messages to a user's server when specific events occur within their Shopify store.

ContentKing is a cloud-based service that monitors websites from a digital marketing perspective. We monitor the websites for customers such as Netflix, Atlassian, Fedex and IBM and alert their digital marketing teams whenever a technical issue or content change is detected.

Tools and product functions where the end user triggers a fetch.

proximic is a contextual advertising crawler that analyzes website content to provide advanced contextual intelligence for programmatic advertising targeting and media optimization.

SemrushBot is the web crawler and search bot software that Semrush sends out to discover and collect web data for their SEO platform including backlink analytics, site audit, content research and SEO tools.

Barkrowler is an SEO web crawler operated by Babbar that analyzes websites to identify page popularity, topical relevance, and authority metrics for search engine optimization and competitive analysis research.

OAI-SearchBot is OpenAI's web crawler that indexes websites for SearchGPT, collecting and analyzing web content to power AI-driven search results and real-time information retrieval in OpenAI's search applications.

Deepcrawl (now Lumar) is an enterprise SEO crawler that audits websites for technical SEO issues.

MJ12bot is the web crawler for Majestic. MJ12Bot does not currently cache web content or personal data. Instead it maps the link relationships between websites to build a search engine. This data is available to technologies and the public, either by searching for a keyword or a website at Majestic.

Cloudflare Healthchecks monitors origin server health to route traffic away from unhealthy servers in load balancing configurations.

APIs-Google is the user agent used by Google APIs to deliver push notification messages. Application developers can request these notifications to avoid the need for continually polling Google's servers to find out if the resources they are interested in have changed. To make sure nobody abuses this service, Google requires developers to prove that they own the domain before allowing them to register a URL with a domain as the location where they want to receive messages.

The Google StoreBot is a search-engine-based program that automatically 'crawls' through web pages to gather and analyse data. Google uses crawlers that go through product pages and checkout processes using machine learning algorithms to fill in forms with information such as delivery addresses, and help compile other information on price, delivery, payments and more.

Prerender is a web crawler that pre-renders Single Page Applications (SPAs) into static HTML to improve SEO visibility for JavaScript-heavy websites and search engine indexing.

GoogleUserTriggeredFetcher is triggered when a user requests Google to translate or cache a page. It acts on behalf of real human users, not autonomous crawling.

Applebot data is used to power various features, such as the search technology that is integrated into many user experiences in Appleʼs ecosystem including Spotlight, Siri, and Safari.

Google-Read-Aloud fetches web content to provide text-to-speech functionality, enabling users to have web pages read aloud through Google's accessibility services.

Google Images (Googlebot-Image) indexes images for Google Image Search, analyzing alt text, surrounding content, and image metadata.

ChatGPT-User fetches web content in real-time when ChatGPT users browse the web or request current information, distinct from training data collection.

Dotbot is Moz's web crawler, it gathers web data for the Moz Link Index. This data we collect through Dotbot is available in the Links section of your Moz Pro campaign, Link Explorer, and the Moz Links API.