1592+ Crawler profiliert
Crawler-Verzeichnis
Alle Web-Crawler und KI-Bots, die wir verfolgen. Was sie tun, wer sie betreibt und wie Sie Ihre Inhalte davor schützen.
SputnikBot — What It Is and How to Handle It
SputnikBot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
spotter — What It Is and How to Handle It
spotter is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
Spinn3r — What It Is and How to Handle It
Spinn3r is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
SpiderLing — What It Is and How to Handle It
SpiderLing is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
OtherOther
Spider — What It Is and How to Handle It
Spider is a web crawler designed for AI projects, including AI agents, LLMs, RAG systems, and data analysis. It collects and converts web data into multiple formats including markdown, HTML, and text for AI training and fine-tuning purposes.
AI TrainingAI Training
SpeedyIndexBot — What It Is and How to Handle It
SpeedyIndexBot is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
OtherOther
special_archiver — What It Is and How to Handle It
special_archiver is a web crawler operated by Internet Archive as part of their Archive-It service, which creates and preserves collections of web content. This bot visits websites to capture and archive web pages for long-term preservation and historical research purposes.
ArchiverArchiver
spbot — What It Is and How to Handle It
spbot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
spanner — What It Is and How to Handle It
spanner is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
SpankBot — What It Is and How to Handle It
SpankBot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
SottopopNone — What It Is and How to Handle It
SottopopNone is a web crawler operated by UpContent, a content curation and discovery platform that helps users find and share relevant content for their marketing and social media needs.
OtherOther
Sosospider — What It Is and How to Handle It
Sosospider is a search engine crawler that indexes web content for search results.
Search EngineSearch Engine
Sonic — What It Is and How to Handle It
Sonic is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
Sogou web spider — What It Is and How to Handle It
Sogou web spider indexes content for Sogou, a major Chinese search engine owned by Tencent with 10+ billion pages indexed.
Search EngineSearch Engine
Sogou spider2 — What It Is and How to Handle It
Sogou web spider is the crawler for Sogou, a major Chinese search engine now part of Tencent, with over 10 billion Chinese pages indexed.
Search EngineSearch Engine
Sogou Push Spider — What It Is and How to Handle It
Sogou Push Spider is a web crawler operated by Sogou, a Chinese search engine company. This bot crawls and indexes web pages for inclusion in Sogou's search results, serving primarily Chinese-language content.
Search EngineSearch Engine
Sogou inst spider — What It Is and How to Handle It
Sogou web spider is the crawler for Sogou, a major Chinese search engine now part of Tencent, with over 10 billion Chinese pages indexed.
Search EngineSearch Engine
SocialRankIOBot — What It Is and How to Handle It
SocialRankIOBot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
Snapchat — What It Is and How to Handle It
Snapchat is a feed reader that fetches and processes RSS, Atom, and other content feeds.
Feed ReaderFeed Reader
Snap URL Preview Service — What It Is and How to Handle It
Snap URL Preview Service is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.
PreviewPreview
SMTnetPMBot — What It Is and How to Handle It
SMTnetPMBot crawls partner company websites to index and include their content in SMTnet's on-site search engine for electronics manufacturing industry resources.
Search EngineSearch Engine
SMTBot — What It Is and How to Handle It
SMTBot is a monitoring and analytics crawler that collects web data for business intelligence and competitive analysis.
MonitoringMonitoring
SmarshBot — What It Is and How to Handle It
SmarshBot is an archiving bot operated by Smarsh that crawls and captures web content for compliance management and regulatory recordkeeping purposes. Organizations use this service to maintain archives of web pages and online communications to meet legal and regulatory requirements.
ArchiverArchiver
slurp — What It Is and How to Handle It
Yahoo! Slurp is Yahoo's search robot for crawling and indexing web content. Now part of Verizon Media, Yahoo search results are primarily powered by Bing.
Search EngineSearch Engine
SlickBot — What It Is and How to Handle It
SlickBot is Slickstream's web crawler that indexes customer websites to power engagement features like site search, content recommendations, and personalization for improved visitor experience on publisher sites.
Search EngineSearch Engine
Sleuth — What It Is and How to Handle It
Sleuth is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
OtherOther
Slackbot-LinkExpanding — What It Is and How to Handle It
Slackbot-LinkExpanding is Slack's link unfurling bot that fetches web page content when users post links in channels, extracting meta tags like oEmbed, Twitter Cards, and Open Graph data to display rich media previews.
PreviewPreview
Slack-ImgProxy — What It Is and How to Handle It
Slack-ImgProxy is a web crawler used by Slack to fetch, process, and cache images that are shared in Slack channels, ensuring they can be displayed properly within the messaging platform.
Feed ReaderFeed Reader
SkypeUriPreview — What It Is and How to Handle It
SkypeUriPreview is Skype's link preview bot that fetches page content when users share URLs in Skype messages, extracting metadata to display rich previews with titles, descriptions, and thumbnails.
PreviewPreview
SiteUptime — What It Is and How to Handle It
SiteUptime monitoring bot performs HTTP monitoring probes to check website availability and uptime status for the SiteUpTime.com monitoring service.
MonitoringMonitoring