1592+ Crawler profiliert
Crawler-Verzeichnis
Alle Web-Crawler und KI-Bots, die wir verfolgen. Was sie tun, wer sie betreibt und wie Sie Ihre Inhalte davor schützen.
SearchmetricsBot — What It Is and How to Handle It
SearchmetricsBot is the crawler for Searchmetrics, an enterprise SEO and content optimization platform.
SEO ToolSEO Tool
SearchAtlas — What It Is and How to Handle It
SearchAtlas is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
search.marginalia.nu — What It Is and How to Handle It
search.marginalia.nu is a web crawler for Marginalia Search, a noncommercial search engine that focuses on indexing old websites, personal websites, and blogs that struggle with discoverability in the SEO-optimized web.
Search EngineSearch Engine
Scrunchbot — What It Is and How to Handle It
Scrunchbot is a web crawler operated by Scrunch AI, a company that provides analytics and optimization tools for brand presence across large language models and AI search platforms.
OtherOther
Scrubby — What It Is and How to Handle It
Scrubby is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
scribdbot — What It Is and How to Handle It
scribdbot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
Screaming Frog SEO Spider — What It Is and How to Handle It
Screaming Frog SEO Spider is a desktop crawler that audits websites for technical SEO issues, broken links, and on-page optimization.
SEO ToolSEO Tool
Scrapy — What It Is and How to Handle It
Scrapy is a popular open-source Python web scraping framework by Zyte, used for data extraction and web crawling at scale.
ScraperScraper
scraping@nytimes.com — What It Is and How to Handle It
NYTimes.com newsroom scraping bot collects publicly available, non-copyrighted data for journalistic projects including election result tracking, COVID-19 data aggregation, and other news analytics initiatives.
ResearchResearch
ScoutJet — What It Is and How to Handle It
ScoutJet is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
ScourRSSBot — What It Is and How to Handle It
ScourRSSBot is a fetcher bot operated by Scour that retrieves RSS feed content from websites. The bot is likely used to collect and aggregate syndicated content for the Scour service.
OtherOther
scour.ing — What It Is and How to Handle It
scour.ing is Scour's AI-powered web crawler that fetches content from RSS feeds and websites to create personalized content recommendations based on user interests.
Feed ReaderFeed Reader
score3 — What It Is and How to Handle It
score3 is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
scope3 — What It Is and How to Handle It
The scope3 crawler indexes publicly available web content and paywalled media to assess content classification and brand safety.
MonitoringMonitoring
scoopit-crawler — What It Is and How to Handle It
scoopit-crawler is a web crawler operated by Scoop.it, a content curation platform that helps users discover, curate, and share content on various topics. The bot fetches and indexes relevant content to power Scoop.it's content discovery and recommendation features for content curators.
OtherOther
scoop.it — What It Is and How to Handle It
scoop.it is a monitoring and analytics crawler that collects web data for business intelligence and competitive analysis.
MonitoringMonitoring
SBL-BOT — What It Is and How to Handle It
SBL-BOT is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
SBIntuitionsBot — What It Is and How to Handle It
SB Intuitions AI bot
AI CrawlerAI Agent
SafeSearch microdata crawler — What It Is and How to Handle It
SafeSearch microdata crawler extracts structured data and microdata for safe search filtering and content categorization.
Search EngineSearch Engine
SafeDNSBot — What It Is and How to Handle It
SafeDNSBot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
s4a — What It Is and How to Handle It
s4a is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
OtherOther
RyteBot — What It Is and How to Handle It
RyteBot is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
RuxitSynthetic — What It Is and How to Handle It
RuxitSynthetic is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
RukiCrawler — What It Is and How to Handle It
RukiCrawler is a web crawler. Its specific purpose and operator are not publicly documented.
OtherOther
RSSingBot — What It Is and How to Handle It
RSSingBot is an RSS feed aggregator crawler that collects and processes content feeds.
Feed ReaderFeed Reader
RSSBot — What It Is and How to Handle It
RSSBot is a generic RSS feed crawler that fetches and processes RSS/Atom feeds.
Feed ReaderFeed Reader
RSSAPI — What It Is and How to Handle It
RSSAPI is a service that periodically monitors RSS, ATOM, and JSON feeds for customers, parsing and normalizing content, then delivering updates via webhooks in near real-time for feed subscription applications.
Feed ReaderFeed Reader
rss-is-dead.lol web bot — What It Is and How to Handle It
rss-is-dead.lol web bot is an uncategorized agent. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
OtherOther
RSiteAuditor — What It Is and How to Handle It
RSiteAuditor is a website crawling tool that performs comprehensive on-page SEO audits, scanning websites for technical errors, analyzing 60+ SEO parameters, and providing structured data through RESTful APIs.
SEO ToolSEO Tool
rogerbot-crawler — What It Is and How to Handle It
rogerbot-crawler is Moz's automated site audit crawler that analyzes websites for SEO issues, technical problems, and optimization opportunities as part of Moz Pro Campaigns SEO analysis suite.
SEO ToolSEO Tool