How AIPREF Works

Choose preferences for four categories: bots (automated processing), train-ai (AI training), train-genai (generative AI training), and search (search indexing). Each can be set to Allow (y), Disallow (n), or Unstated.

Use the AIPREF Generator to create properly formatted Content-Usage headers, robots.txt directives, or JSON configuration files that comply with RFC 9651 Structured Fields specification.

Add the Content-Usage header to your HTTP responses via web server configuration (Nginx, Apache) or application middleware. Alternatively, add Content-Usage directives to your robots.txt file following RFC 9309 Robots Exclusion Protocol.

1. HTTP Headers (Preferred Method)

When a system requests content from your server, the Content-Usage header is included in the HTTP response. This applies to the specific resource being accessed and takes precedence over robots.txt.

HTTP/1.1 200 OK
Content-Type: text/html
Content-Usage: train-ai=n, train-genai=n
...

2. robots.txt Discovery

Before crawling, compliant systems check your robots.txt file at the root of your domain. Content-Usage directives in robots.txt apply to URL patterns specified in the User-Agent section.

User-Agent: *
Allow: /
Content-Usage: train-ai=n, train-genai=n

When preferences conflict, the most restrictive preference takes precedence. Disallow (n) overrides Allow (y), and explicit preferences override unstated ones.

More specific preference categories take precedence over general ones. For example, train-genai=n overrides train-ai=y for generative AI training.

Content-Usage headers attached to HTTP responses take precedence over Content-Usage directives in robots.txt for the same resource.