Announcing the Magic Pages Open House. Quarterly calls with our team!

AI crawler controls Included with every site

Decide which AI crawlers reach your Ghost® site

Ghost’s robots.txt says nothing to AI crawlers, and even if you add a line, it’s only a request they can ignore. On Magic Pages, you choose which AI crawlers get in, and the ones you block are turned away before they reach your site.

The Ghost site Seaside Notes with AI crawler controls set to “Block AI training”. Of five requests that just arrived, the training crawlers GPTBot and ClaudeBot were turned away with a 403, while Googlebot, ChatGPT’s search crawler and a reader on an iPhone were let in.

Where do you stand on AI crawlers?

Some publishers want their writing kept out of AI training. Others want to be found and quoted when someone asks an AI assistant. Both are fair positions, and the setting works for either.

Time to switch if…

  • You don’t want your posts used to train AI models.
  • You’d rather not be quoted in AI answers that don’t send readers your way.
  • You wrote a robots.txt for your theme and want the block enforced, not just requested.
  • You want to know how many AI crawlers your site turns away each week.

Stay with the default if…

  • You want AI assistants to find, quote and link to your posts.
  • You don’t mind your public posts being used to train AI models.
  • You haven’t made up your mind yet. A new level takes effect within a minute, whenever you decide.

Every Magic Pages site starts with AI crawlers allowed, so there’s nothing to change if that’s your choice.

Keep them out, or let them in

AI crawler controls are part of the Magic Pages plan. You pick a level in your customer portal, and we handle the rules at the edge and the lines in your robots.txt. Nothing to add to your theme.

  1. Four levels, one choice

    Allow everything, block AI training, block training and AI answers, or block everything, including search. Each level adds to the one before, so you decide where the line is.

    Blocking training keeps you in search engines and in AI answers. Blocking answers also turns away the crawlers behind ChatGPT search, Claude’s search and Perplexity. Only the last level touches search.

  2. Turned away, not just asked

    A robots.txt line asks a crawler to stay away. A blocked crawler gets a 403 before the request reaches your site, on every page, paid posts included. Readers browse as normal.

    We match Cloudflare’s verified AI crawler category and a list of named crawlers from the open-source ai.robots.txt project, refreshed every week. Your portal counts what was turned away in the last 7 days.

  3. A robots.txt line for Google and Apple

    Google and Apple don’t send a separate crawler for AI training. Googlebot and Applebot fetch for search, and two robots.txt lines, Google-Extended and Applebot-Extended, decide whether that content may train their models.

    Blocking those crawlers would cost you search. So we add the two Disallow lines to the robots.txt Ghost already serves. Both companies document this as their opt-out, but it’s a request, not a block.

  4. Search and link previews keep working

    Googlebot and Bingbot are never blocked, at any level, so your posts stay in Google and Bing. The fetchers behind link previews on Facebook, WhatsApp and Messenger get through too, so a shared post keeps its title and image.

    The one exception is the last level. Apple and Huawei use one crawler for search and AI, so blocking everything takes you out of Apple and Petal search. The portal says so right under that option.

  5. Or welcome them in

    If you want AI tools to find and quote you, pick Allow everything. Your robots.txt then carries a Content-Signal line that says yes to search, to AI answers and to training.

    Ghost’s own llms.txt setting, under Settings → Meta data, adds a plain index of your public posts for AI tools to read. Members-only posts appear there by title and excerpt only.

Your options for AI crawlers on Ghost

There’s more than one way to do it. Here’s what each one asks of you, and what it actually stops.

Swipe the table sideways to compare all four.

Ways to control AI crawlers on a Ghost site, compared on robots.txt requests, blocking at the edge, the Google and Apple training opt-out, keeping the crawler list current, set-up and cost. Checked November 2026.
Ghost’s default robots.txt A robots.txt in your theme Cloudflare AI Crawl Control Magic Pages
Asks AI crawlers to stay away No AI lines The lines you write Yes, a managed robots.txt Yes, written for your level
Turns them away at the edge No No, crawlers can ignore it Yes, per crawler Yes, per level
Google and Apple training opt-out No If you add the lines Yes, in its robots.txt Yes, in your robots.txt
Keeps the crawler list current No list You, by hand Cloudflare Our team, every week
What it takes Nothing A theme edit and upload Your domain on your own Cloudflare account One choice in your customer portal
Extra bill None None None, on every Cloudflare plan None, it’s included

“Ghost’s default robots.txt” is the file every Ghost site serves, including on Ghost(Pro); a theme can replace it with its own. “Cloudflare AI Crawl Control” means putting your domain on a Cloudflare account of your own. Magic Pages sites already run behind Cloudflare, and our setting uses Cloudflare’s verified AI crawler category plus the open-source ai.robots.txt list, so you don’t need that account. We checked each product’s own documentation in November 2026 (Ghost themes, Cloudflare AI Crawl Control, Cloudflare’s managed robots.txt). Cloudflare’s AI Crawl Control goes further than we do, with a choice per crawler, a report of which crawlers ignore your robots.txt, and pay per crawl in private beta.

Easy to check, either way

Our own robots.txt

We let AI crawlers in on magicpages.co. Open our robots.txt and you’ll see the line our setting wrote: search, AI input and AI training all set to yes.

See our robots.txt →

One request from your terminal

Send a request to your site as GPTBot and it comes back 403. The same request as a normal browser comes back 200. The portal’s Test it button does the same for you.

How to test it →

1,600+ publishers host with us

Writers, newsletters and small businesses run their Ghost sites on Magic Pages. Every one of them decides for their own site.

Choosing takes one click

No firewall rules to write and no robots.txt to upload.

  1. Open Configuration

    It’s a tab in your Magic Pages customer portal, with an AI crawlers section.

  2. Pick a level

    It saves as soon as you choose and reaches the edge within 60 seconds.

  3. Press Test it

    The portal sends requests as AI crawlers and as a normal visitor, and shows you which got through.

What each level blocks

Every crawler on each level, and what blocking can’t do, is in the help article.

Read the help article →

Already running Ghost somewhere else?

We move your posts, members and paid subscriptions for you. Most moves are free on any plan, and big ones are free with a yearly plan.

How migration works →

Questions you might have

Something else on your mind? Email us. A real person reads every message, usually the same day.

Last updated

Ghost serves a robots.txt with no AI lines, and a theme can replace it with your own. Either way it’s a request, so it only works on crawlers that read and respect it. AI crawler controls write those lines for you and also turn away the crawlers that name themselves, with a 403 at the edge.

Ready to Publish? A snap of your fingers – and you're live!

Join 1,600+ publishers who chose Magic Pages as the home for their Ghost site.

Jannis - Founder of Magic Pages
Live in 5 minutes
Personal support from our team
All Ghost features included
14-day free trial, no credit card
Start Your 14-Day Free Trial

No credit card required

I personally guarantee you'll love the experience, or I'll help you find a better solution.

— Jannis, Founder of Magic Pages