Blog

GoogleImageProxy fetches and caches images and link previews for Gmail email rendering, enabling safe display of external content within Google's email interface.

GoogleDocs is a fetcher bot operated by Google that visits websites when users embed or link to external content within Google Sheets and other Google Docs applications.

Googlebot-Video is Google's specialized crawler for video content discovery and indexing. It crawls websites to find and analyze video files, metadata, and thumbnails for Google Video Search results.

Googlebot-News indexes news articles for Google News, prioritizing timely content from verified news publishers.

Googlebot-Mobile indexes mobile-optimized web pages for Google's mobile-first search index.

Googlebot-Image is Google's specialized search engine crawler that discovers and indexes images across the web for inclusion in Google Images search results, analyzing visual content and metadata.

Googlebot-IA is Google Scholar's specialized crawler that indexes scholarly literature from academic publishers, repositories, and university websites to populate the academic search engine with research papers and publications.

GoogleAssociationService is a verification bot that checks Digital Asset Links statements on websites to validate relationships between web properties and mobile apps for features like app linking and credential sharing.

GoogleAgent-Mariner is an AI agent created by Google that can use a web browser. It can intelligently navigate and interact with websites to complete multi-step tasks on behalf of a human user.

google-xrawler is a Google experimental crawler used for testing and research purposes.

Google-Trust-Services is a certificate authority bot that helps build a safer Internet by providing transparent, trusted, and reliable TLS certificates. It crawls websites to verify certificate authenticity and manage secure connections for Google services and users.

Google-Structured-Data-Testing-Tool validates structured data markup (Schema.org) on web pages for Google Search features.

Google-Structured-Data-Testing bot validates structured data markup on web pages, helping developers ensure proper schema implementation for rich search results and SEO optimization in Google Search.

Google Sitemaps crawler

Google-Safety is a specialized security crawler that performs abuse-specific scanning to detect malware and other security threats on publicly posted links across Google properties, operating independently of robots.txt restrictions.

Google-PhysicalWeb was Google's crawler for the Physical Web project, which allowed Bluetooth beacons to broadcast URLs.

Google-NotebookLM is an AI-powered research and note-taking assistant that helps users synthesize information from their own uploaded sources, such as documents, transcripts, or web content.

Google-Extended controls whether your content is used to train Google's AI models (Bard/Gemini), separate from Google Search indexing.

Google-Display-Ads-Bot is operated by Google and crawls websites to analyze their content for contextual ad targeting and placement in the Google Display Network.

Google-Certificates-Bridge is a Google crawler that verifies SSL/TLS certificate deployment and configuration.