Add an llms.txt File and AI-Crawler Controls with AI

Publish an llms.txt and set per-bot crawler rules so AI search sees your site the way you want.

Difficulty Beginner

Time ~10 min

Tools run_wp_cli

Prerequisites

A connected site with an SEO plugin that supports llms.txt and a robots.txt editor (Rank Math).

The prompt

Make my site legible to AI search engines. Set up an llms.txt file and AI-crawler rules through my SEO plugin.

First, turn on the llms.txt generator and include my [pages and posts]. Add this description of my site so AI models cite it correctly: [one or two sentences about what your site is and your preferred citation]. Then show me the live /llms.txt.

Next, set my robots.txt AI-crawler rules: allow [GPTBot, ClaudeBot, Google-Extended, PerplexityBot] so AI search can index me, and block [CCBot and any training-only scrapers I name]. Keep all my existing robots.txt rules intact. Show me the final robots.txt and confirm both files load.

Note for me that robots.txt is honored by well-behaved crawlers but is not an enforced block.

Already connected to WPVibe?

Skip the copy-paste. Just ask your AI for the “Add an llms.txt File and AI-Crawler Controls with AI” recipe, or describe the task, and WPVibe runs these steps on your own site.

AI search engines and chat assistants are turning into a real source of traffic, and they read your site differently from Google. They look for an llms.txt file, a plain summary of your most important pages, and they honor separate crawler rules for bots like GPTBot and ClaudeBot. Most WordPress sites publish neither, so AI tools either skip past them or scrape blindly, and the owner gets no say in which one happens.

Ask your AI assistant and WPVibe turns on your SEO plugin’s llms.txt generator, which publishes a clean, auto-updating /llms.txt that lists your key pages, posts, and sitemap for AI models to read. Then it sets the crawler rules in your robots.txt: allow the AI search engines you want indexing you, and block the scrapers you do not. It shows you both files so you can see exactly what an AI tool will pick up before and after the change.

Paste the prompt and decide your stance first: welcome AI search for the visibility, or keep training crawlers out. Add a sentence describing your site and how you want to be cited, since that goes straight into the llms.txt. This corner of SEO moves fast, so recheck which bots you allow every few months as new ones show up, and keep in mind that robots.txt is a request good crawlers respect, not a hard lock.

Outcome

Your site serves an auto-updating /llms.txt of key content plus a robots.txt with explicit allow and block rules for AI crawlers.