Glossary

Review the definitions for terms used across Cloudflare's AI Crawl Control documentation.

Term	Definition
AI crawler	A bot which scrapes content from websites in support of an AI model, including by scraping content for indexing, retrieval augmented generation, or training.
In-band pricing	A pricing model where pricing information is transmitted within the same communication channel as the content itself. In Pay Per Crawl, in-band pricing means the price is included in the HTTP response headers (crawler-price) alongside the actual content, rather than being retrieved from a separate system or API call. When custom pricing is enabled, Cloudflare sets the cf-pay-per-crawl header to pricing=in-band, indicating that the origin server or Worker can dynamically set prices by including a crawler-price header in the response.
Merchant of Record	The entity who facilitates "buying and selling". For pay per crawl, Cloudflare is the merchant of record.
Operator	The company or organization that owns and operates an AI crawler. Examples include OpenAI, Microsoft, Google, ByteDance, Anthropic, and Meta. In AI Crawl Control, crawlers are grouped by their operators.
Referrals	Traffic referred to your website by AI services, identified by the referrer URL and whether it is owned by an AI operator.
robots.txt	A text file which lists pages in your website that are off-limits for bots. Well-behaved bots respect this file, but some bots may violate it. You can enforce robots.txt with Cloudflare WAF custom rules.