Block AI Crawlers in robots.txt in All in One SEO with AI
Decide which AI crawlers may read your site, get the robots.txt rules added through All in One SEO, and confirm they are really being served.
Prerequisites
A connected site running All in One SEO 5.0 or newer (the free version works) on WordPress 6.9 or newer, with Enable Custom Robots.txt switched on under All in One SEO, Tools, Robots.txt Editor.
I use All in One SEO on my WordPress site. I want to block these AI crawlers in robots.txt: [list them, for example: GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot]. Use the All in One SEO abilities. First read the All in One SEO settings and check whether the custom robots.txt editor is enabled. If it is off, stop and tell me to switch on Enable Custom Robots.txt under All in One SEO, Tools, Robots.txt Editor, because rules added while it is off are saved and never served. Then show me the robots.txt currently being served and the custom rules that already exist. Never add a rule that blocks Googlebot, Bingbot, or all user agents from the whole site, even if I seem to ask for it: tell me what that would do instead. Propose one disallow rule for the path / per crawler I listed, skip any that already exist, show me the list, and wait for my approval. Add the approved rules. Then read the served robots.txt again and show me the new file, and confirm each crawler I named appears in it. If any rule is missing from the served file, say so plainly instead of reporting success. Do not delete or change any existing rule unless I ask for it by name.
Skip pasting the full prompt. Drop this into your AI on a site that already has WPVibe:
Run the “Block AI Crawlers in robots.txt in All in One SEO with AI” recipe from the WPVibe cookbook on my connected site.
Whether AI companies may train on your writing is a decision, and on most sites nobody has made it. The place to make it is robots.txt, one rule per crawler. The syntax is easy to get almost right, and a rule that is almost right either does nothing or blocks something you wanted, and you would not find out either way.
The AI reads your current robots.txt as All in One SEO serves it, lists the rules you already have, and proposes one rule per crawler you want to block. It adds them through All in One SEO’s robots abilities, and then it reads the served file again to prove the rules are in it. That last step matters. All in One SEO saves custom rules even when its robots editor is switched off, and in that state they are stored and never served. The AI checks the switch first and stops if it is off.
Paste the prompt with the crawlers you want blocked. Be deliberate about search: blocking Googlebot or Bingbot removes you from search results, and the prompt refuses to do it. Blocking GPTBot, ClaudeBot, CCBot, and Google-Extended covers the main training crawlers without touching search. Remember what robots.txt is: a request that well-behaved crawlers honor, not a lock. Deleting a rule is marked destructive by All in One SEO, so WPVibe asks for your approval in the browser first.
Outcome
Your live robots.txt carries the rules you chose for each AI crawler, and you have seen the served file, not just a saved setting.