Skip to content
Cloudflare Docs

Get started

This guide instructs you on how to:

  • View AI crawlers that are interacting with pages in your domain (a Cloudflare zone).
  • Use AI Crawl Control to block individual crawlers from accessing your content.

Prerequisites

  1. Sign up for a Cloudflare account ↗.
  2. Connect your domain to Cloudflare.
  3. Make sure your domain is proxying traffic through Cloudflare.

1. Monitor AI crawler activity at a glance

  1. Go to AI Crawl Control.

    Go to AI Crawl Control

  2. Review the snapshot of your AI crawler activity in the Overview tab.

  3. Use the filters to view activity by specific date ranges, crawlers, operators, hostnames, or paths.

2. Block specific AI crawlers

To block specific AI crawlers:

  1. Go to AI Crawl Control.

    Go to AI Crawl Control

  2. Review which AI crawlers are accessing your domain in the Crawlers tab.

  3. In the Action column, select Block.

For more information, refer to Manage AI crawlers.

You can also create more complex rules when taking action on AI crawlers, using Cloudflare WAF. For more information on creating more specific rules, refer to Create a custom rule in the dashboard.

3. Explore detailed metrics

For more detailed analytics, use the Metrics tab.

  1. Go to AI Crawl Control.

    Go to AI Crawl Control

  2. Review detailed breakdowns by date range, crawler, operator, status code, hostname, or path in the Metrics tab.

Note that on free plans, the Metrics tab only displays metrics for the past 24 hours.

Plan comparison

All plansEnterprise plans with Bot Management
AI crawler detection via user agent strings ↗Advanced AI crawler detection via Bot Management detection ID
Maximum 24-hour analytics windowConfigurable analytics timeframes
Allow/block controlsAllow/block controls, and the ability to charge AI crawlers using pay per crawl

Next steps

Refer to the following related resources: