Overview

A large German news publisher with a high-traffic, multi-domain digital portfolio partnered with Centinel to better understand and govern automated access to its content.

Facing a rapid increase in AI crawler and scraper traffic, the publisher required a solution that would deliver full transparency, zero SEO risk, and no operational disruption—without immediately enforcing blocking decisions.

The Challenge

Like many digital publishers, the organization experienced a sharp rise in automated traffic driven by:

• AI crawlers and LLM agents

• Headless browsers and large-scale scrapers

• Undocumented or opaque automated clients

While basic controls such as robots.txt were in place, they proved insufficient for:

• Measuring actual crawler activity

• Distinguishing between different AI systems

• Enforcing governance beyond voluntary compliance

Critically, the publisher lacked reliable answers to fundamental questions:

• Which AI crawlers are accessing our content?

• How frequently and at what scale?

• Are declared opt-outs being respected in practice?

Requirements

The publisher defined clear constraints for any solution:

• No SEO or availability risk

• No impact on latency or page delivery

• No forced blocking or enforcement during initial rollout

• Compatibility with existing CDN and infrastructure

• Clear, actionable insights—not just raw logs

The Centinel Approach

Centinel was deployed in monitor-only mode, enabling passive analysis of all automated traffic without blocking or altering request flows.

This allowed the publisher to:

• Observe AI crawler activity in real time

• Classify automated clients by behavior and fingerprint

• Quantify request volumes across domains

• Identify non-compliant or previously unknown crawlers

Importantly, the deployment required no changes to SEO configuration and no active intervention during the pilot phase.

Key Insights

Within the first weeks, Centinel uncovered several critical findings:

• A meaningful share of automated traffic originated from AI crawlers that were not explicitly authorized

• Multiple crawlers showed inconsistent or selective adherence to robots.txt

• Distinct AI systems exhibited repeatable technical fingerprints, enabling reliable classification beyond user-agent strings

For the first time, the publisher had verifiable data rather than assumptions about AI access to its content.

Outcome

Centinel enabled the publisher to:

• Establish a single source of truth for AI crawler activity

• Move internal discussions from speculation to evidence

• Prepare differentiated allow, deny, or monitor policies with confidence

• Support legal, technical, and commercial teams with shared visibility

The result was not immediate enforcement, but strategic clarity—a prerequisite for any future governance, licensing, or monetization decisions.

Why It Matters

As AI systems increasingly consume publisher content at scale, visibility becomes the foundation for control.

Centinel does not force publishers into premature blocking decisions. Instead, it provides the observability and governance layer required to make informed, defensible choices—on technical, legal, and commercial grounds.

Next Steps

Following the successful monitoring phase, the publisher is evaluating:

• Selective enforcement by crawler type

• Policy alignment with TDM opt-out strategies

• Long-term AI access governance across all properties