Blog

SearchmetricsBot is the crawler for Searchmetrics, an enterprise SEO and content optimization platform.

SearchAtlas is a web crawler. Its specific purpose and operator are not publicly documented.

search.marginalia.nu is a web crawler for Marginalia Search, a noncommercial search engine that focuses on indexing old websites, personal websites, and blogs that struggle with discoverability in the SEO-optimized web.

Scrunchbot is a web crawler operated by Scrunch AI, a company that provides analytics and optimization tools for brand presence across large language models and AI search platforms.

Scrubby is a web crawler. Its specific purpose and operator are not publicly documented.

scribdbot is a web crawler. Its specific purpose and operator are not publicly documented.

Screaming Frog SEO Spider is a desktop crawler that audits websites for technical SEO issues, broken links, and on-page optimization.

Scrapy is a popular open-source Python web scraping framework by Zyte, used for data extraction and web crawling at scale.

NYTimes.com newsroom scraping bot collects publicly available, non-copyrighted data for journalistic projects including election result tracking, COVID-19 data aggregation, and other news analytics initiatives.

ScoutJet is a web crawler. Its specific purpose and operator are not publicly documented.

ScourRSSBot is a fetcher bot operated by Scour that retrieves RSS feed content from websites. The bot is likely used to collect and aggregate syndicated content for the Scour service.

scour.ing is Scour's AI-powered web crawler that fetches content from RSS feeds and websites to create personalized content recommendations based on user interests.

score3 is a web crawler. Its specific purpose and operator are not publicly documented.

The scope3 crawler indexes publicly available web content and paywalled media to assess content classification and brand safety.

scoopit-crawler is a web crawler operated by Scoop.it, a content curation platform that helps users discover, curate, and share content on various topics. The bot fetches and indexes relevant content to power Scoop.it's content discovery and recommendation features for content curators.

scoop.it is a monitoring and analytics crawler that collects web data for business intelligence and competitive analysis.

SBL-BOT is a web crawler. Its specific purpose and operator are not publicly documented.

SB Intuitions AI bot

SafeSearch microdata crawler extracts structured data and microdata for safe search filtering and content categorization.

SafeDNSBot is a web crawler. Its specific purpose and operator are not publicly documented.