AI crawler controls Included with every site
Decide which AI crawlers reach your Ghost® site
Ghost’s robots.txt says nothing to AI crawlers, and even if you add a line, it’s only a request they can ignore. On Magic Pages, you choose which AI crawlers get in, and the ones you block are turned away before they reach your site.
No credit card required
Where do you stand on AI crawlers?
Some publishers want their writing kept out of AI training. Others want to be found and quoted when someone asks an AI assistant. Both are fair positions, and the setting works for either.
Time to switch if…
- You don’t want your posts used to train AI models.
- You’d rather not be quoted in AI answers that don’t send readers your way.
- You wrote a robots.txt for your theme and want the block enforced, not just requested.
- You want to know how many AI crawlers your site turns away each week.
Stay with the default if…
- You want AI assistants to find, quote and link to your posts.
- You don’t mind your public posts being used to train AI models.
- You haven’t made up your mind yet. A new level takes effect within a minute, whenever you decide.
Every Magic Pages site starts with AI crawlers allowed, so there’s nothing to change if that’s your choice.
Keep them out, or let them in
AI crawler controls are part of the Magic Pages plan. You pick a level in your customer portal, and we handle the rules at the edge and the lines in your robots.txt. Nothing to add to your theme.
-
Four levels, one choice
Allow everything, block AI training, block training and AI answers, or block everything, including search. Each level adds to the one before, so you decide where the line is.
Blocking training keeps you in search engines and in AI answers. Blocking answers also turns away the crawlers behind ChatGPT search, Claude’s search and Perplexity. Only the last level touches search.
-
Turned away, not just asked
A robots.txt line asks a crawler to stay away. A blocked crawler gets a 403 before the request reaches your site, on every page, paid posts included. Readers browse as normal.
We match Cloudflare’s verified AI crawler category and a list of named crawlers from the open-source ai.robots.txt project, refreshed every week. Your portal counts what was turned away in the last 7 days.
-
A robots.txt line for Google and Apple
Google and Apple don’t send a separate crawler for AI training. Googlebot and Applebot fetch for search, and two robots.txt lines, Google-Extended and Applebot-Extended, decide whether that content may train their models.
Blocking those crawlers would cost you search. So we add the two Disallow lines to the robots.txt Ghost already serves. Both companies document this as their opt-out, but it’s a request, not a block.
-
Search and link previews keep working
Googlebot and Bingbot are never blocked, at any level, so your posts stay in Google and Bing. The fetchers behind link previews on Facebook, WhatsApp and Messenger get through too, so a shared post keeps its title and image.
The one exception is the last level. Apple and Huawei use one crawler for search and AI, so blocking everything takes you out of Apple and Petal search. The portal says so right under that option.
-
Or welcome them in
If you want AI tools to find and quote you, pick Allow everything. Your robots.txt then carries a Content-Signal line that says yes to search, to AI answers and to training.
Ghost’s own llms.txt setting, under Settings → Meta data, adds a plain index of your public posts for AI tools to read. Members-only posts appear there by title and excerpt only.
Your options for AI crawlers on Ghost
There’s more than one way to do it. Here’s what each one asks of you, and what it actually stops.
Swipe the table sideways to compare all four.
| Ghost’s default robots.txt | A robots.txt in your theme | Cloudflare AI Crawl Control | Magic Pages | |
|---|---|---|---|---|
| Asks AI crawlers to stay away | No AI lines | The lines you write | Yes, a managed robots.txt | Yes, written for your level |
| Turns them away at the edge | No | No, crawlers can ignore it | Yes, per crawler | Yes, per level |
| Google and Apple training opt-out | No | If you add the lines | Yes, in its robots.txt | Yes, in your robots.txt |
| Keeps the crawler list current | No list | You, by hand | Cloudflare | Our team, every week |
| What it takes | Nothing | A theme edit and upload | Your domain on your own Cloudflare account | One choice in your customer portal |
| Extra bill | None | None | None, on every Cloudflare plan | None, it’s included |
“Ghost’s default robots.txt” is the file every Ghost site serves, including on Ghost(Pro); a theme can replace it with its own. “Cloudflare AI Crawl Control” means putting your domain on a Cloudflare account of your own. Magic Pages sites already run behind Cloudflare, and our setting uses Cloudflare’s verified AI crawler category plus the open-source ai.robots.txt list, so you don’t need that account. We checked each product’s own documentation in November 2026 (Ghost themes, Cloudflare AI Crawl Control, Cloudflare’s managed robots.txt). Cloudflare’s AI Crawl Control goes further than we do, with a choice per crawler, a report of which crawlers ignore your robots.txt, and pay per crawl in private beta.
Easy to check, either way
Our own robots.txt
We let AI crawlers in on magicpages.co. Open our robots.txt and you’ll see the line our setting wrote: search, AI input and AI training all set to yes.
One request from your terminal
Send a request to your site as GPTBot and it comes back 403. The same request as a normal browser comes back 200. The portal’s Test it button does the same for you.
1,600+ publishers host with us
Writers, newsletters and small businesses run their Ghost sites on Magic Pages. Every one of them decides for their own site.
Choosing takes one click
No firewall rules to write and no robots.txt to upload.
-
Open Configuration
It’s a tab in your Magic Pages customer portal, with an AI crawlers section.
-
Pick a level
It saves as soon as you choose and reaches the edge within 60 seconds.
-
Press Test it
The portal sends requests as AI crawlers and as a normal visitor, and shows you which got through.
What each level blocks
Every crawler on each level, and what blocking can’t do, is in the help article.
Already running Ghost somewhere else?
We move your posts, members and paid subscriptions for you. Most moves are free on any plan, and big ones are free with a yearly plan.
Questions you might have
Something else on your mind? Email us. A real person reads every message, usually the same day.
Last updated
Ghost serves a robots.txt with no AI lines, and a theme can replace it with your own. Either way it’s a request, so it only works on crawlers that read and respect it. AI crawler controls write those lines for you and also turn away the crawlers that name themselves, with a 403 at the edge.
A scraper that pretends to be a normal browser gets through, and posts collected before you switched it on may already sit in trained models. Google’s AI Overviews and AI Mode are part of Google Search, so no robots.txt line removes you from them; Google offers its own switch in Search Console.
Not below the last level. Googlebot and Bingbot get through at every level. The last level, Block everything, including search, also turns away Applebot and PetalBot, which removes your site from Apple and Huawei Petal search.
That’s Ghost’s own setting, under Settings → Meta data, and it works on Magic Pages like on any Ghost site. It lists your public posts for AI tools, and members-only posts by title and excerpt only. If you block AI crawlers, you’ll probably want to leave it off.
No. Magic Pages has one plan, from $14.50 a month, and AI crawler controls are part of it.
We move your site for you: posts, pages, members and paid subscriptions, which keep billing through your own Stripe account. Most moves are free on any plan, and big ones are free with a yearly plan. Start the trial, then tell us where your site lives. How migration works
Ready to Publish? A snap of your fingers – and you're live!
Join 1,600+ publishers who chose Magic Pages as the home for their Ghost site.
No credit card required
I personally guarantee you'll love the experience, or I'll help you find a better solution.
— Jannis, Founder of Magic Pages