Crawler-Verzeichnis

GoogleImageProxy fetches and caches images and link previews for Gmail email rendering, enabling safe display of external content within Google's email interface.

GoogleDocs is a fetcher bot operated by Google that visits websites when users embed or link to external content within Google Sheets and other Google Docs applications.

Googlebot-Video is Google's specialized crawler for video content discovery and indexing. It crawls websites to find and analyze video files, metadata, and thumbnails for Google Video Search results.

Googlebot-News indexes news articles for Google News, prioritizing timely content from verified news publishers.

Googlebot-Mobile indexes mobile-optimized web pages for Google's mobile-first search index.

Googlebot-Image is Google's specialized search engine crawler that discovers and indexes images across the web for inclusion in Google Images search results, analyzing visual content and metadata.

Googlebot-IA is Google Scholar's specialized crawler that indexes scholarly literature from academic publishers, repositories, and university websites to populate the academic search engine with research papers and publications.

GoogleAssociationService is a verification bot that checks Digital Asset Links statements on websites to validate relationships between web properties and mobile apps for features like app linking and credential sharing.

GoogleAgent-Mariner is an AI agent created by Google that can use a web browser. It can intelligently navigate and interact with websites to complete multi-step tasks on behalf of a human user.

google-xrawler is a Google experimental crawler used for testing and research purposes.

Google-Trust-Services is a certificate authority bot that helps build a safer Internet by providing transparent, trusted, and reliable TLS certificates. It crawls websites to verify certificate authenticity and manage secure connections for Google services and users.

Google-Structured-Data-Testing-Tool validates structured data markup (Schema.org) on web pages for Google Search features.

Google-Structured-Data-Testing bot validates structured data markup on web pages, helping developers ensure proper schema implementation for rich search results and SEO optimization in Google Search.

Google Sitemaps crawler

Google-Safety is a specialized security crawler that performs abuse-specific scanning to detect malware and other security threats on publicly posted links across Google properties, operating independently of robots.txt restrictions.

Google-PhysicalWeb was Google's crawler for the Physical Web project, which allowed Bluetooth beacons to broadcast URLs.

Google-NotebookLM is an AI-powered research and note-taking assistant that helps users synthesize information from their own uploaded sources, such as documents, transcripts, or web content.

Google-Extended controls whether your content is used to train Google's AI models (Bard/Gemini), separate from Google Search indexing.

Google-Display-Ads-Bot is operated by Google and crawls websites to analyze their content for contextual ad targeting and placement in the Google Display Network.

Google-Certificates-Bridge is a Google crawler that verifies SSL/TLS certificate deployment and configuration.

Google-Calendar-Importer is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.

Google-Ads-Creatives-Assistant is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.

Google Search Console inspection tool

Google Web Preview is a feed reader that fetches and processes RSS, Atom, and other content feeds.

Google Favicon fetches website favicons to display in Google Search results, Chrome tabs, and bookmarks.

Goodreads is a social cataloging website that allows users to create reading lists, rate and review books, and connect with other readers. The bot fetches content from web pages to generate book previews and metadata when users share links on the platform.

GnowitNewsbot is a news aggregator crawler that collects and indexes news articles.

gnam gnam spider is a search engine crawler that indexes web content for search results.

Gluten Free Crawler is a web crawler. Its specific purpose and operator are not publicly documented.

GingerCrawler is a web crawler. Its specific purpose and operator are not publicly documented.