Crawler-Verzeichnis

Anchor Browser is Anchor's cloud-hosted browser designed to enable AI agents to interact with websites in a human-like manner while maintaining verified, legitimate automated access.

AmiSoftware is a media monitoring crawler that collects news and content for reputation management.

AmazonProductDiscovery is a web crawler operated by Amazon that discovers and collects product information across the web. This bot is likely used to identify products available on various websites for Amazon's retail and vendor programs.

AmazonBuyForMe is an Amazon bot that crawls websites as part of the Amazon Buy For Me service. This bot visits product pages and e-commerce websites to gather product information and make purchases on behalf of human users.

AmazonAdBot is the web crawler used by Amazon's advertising services to visit and analyze advertiser landing pages to ensure compliance with advertising policies and optimize ad campaign performance.

AlphaBot is a web crawler. Its specific purpose and operator are not publicly documented.

AllAfrica NewsBot is a content fetching bot operated by AllAfrica, a pan-African news aggregation service. The bot visits websites to collect and aggregate news content from various sources across the African continent for distribution on their news platform.

AliyunSecBot is a security scanner operated by Alibaba, a major Chinese technology company. This bot crawls websites to scan for potential security vulnerabilities, threats, or compliance issues as part of Alibaba's cloud security services.

Algolia Crawler is a web crawler that indexes website content for Algolia's search-as-a-service platform, enabling fast and relevant search functionality for applications.

Alexibot is a known aggressive scraper that often ignores robots.txt and consumes excessive server resources.

AlexandriaOrgBot is a web archiving crawler operated by Alexandria.org, a nonprofit focused on preserving digital knowledge and web content.

Alexabot is Amazon's crawler for Alexa web information services, collecting data for Alexa rankings and site analysis tools.

alexa site audit was Amazon Alexa's SEO analysis crawler that evaluated websites for search optimization. Alexa Internet was discontinued in May 2022.

Alexa Archive was Amazon Alexa's web archiving crawler that captured website snapshots for historical records and the Wayback Machine integration. Alexa Internet was discontinued in 2022.

AlertSite is SmartBear's synthetic monitoring service that uses automated probes to monitor website availability, performance, and functionality from global monitoring locations.

AISearchBot is an AI-powered search crawler that indexes content for AI search services. Purpose and operator are not well documented.

aiHitBot crawls websites for aiHit, a company data provider that extracts business information, technologies used, and company profiles for B2B intelligence.

Ai2Bot-Dolma is operated by Ai2, a non-profit AI research institute. It's used to download data to train open source AI models.

Ai2Bot-DeepResearchEval is operated by Ai2, a non-profit AI research institute. It's used to collect and scan resources used in deep research queries performed by Ai2's open source AI models.

AhrefsSiteAudit is a technical SEO crawler that analyzes websites to identify and flag over 170 technical and on-page SEO issues, providing detailed recommendations and performance metrics for website owners.

AdvBot is an advertising technology crawler that analyzes web content for ad targeting, verification, or competitive intelligence.

AdsTxtCrawler-CyberAgent is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.

adscanner is an advertising verification crawler that scans websites for ad placement and compliance.

AdsBot-Google-Mobile checks mobile landing pages for Google Ads quality, evaluating load speed, mobile usability, and ad policy compliance to determine Quality Scores.

AdsBot-Google checks landing pages for Google Ads quality, evaluating load speed, usability, and ad policy compliance to determine Quality Scores and cost-per-click rates.

The Admantx web robot indexes and analyzes website content.

AdkernelTopicCrawler is a web crawler operated by Adkernel, a white-label ad tech solutions provider. This bot gathers information to support the company's ad network, DSP, SSP, and trading desk platforms.

adidxbot is Microsoft's Bing Ads crawler that monitors ads and follows ad destination websites for quality control to ensure advertising standards and policy compliance.

adequat-systems is an advertising technology crawler that analyzes web content for ad targeting, verification, or competitive intelligence.

adequat is an advertising technology crawler that analyzes web content for ad targeting, verification, or competitive intelligence.