From June 20 to June 21, 2025, Magic Pages experienced an issue where ActivityPub posts from Ghost sites were not properly federating to other ActivityPub instances (e.g. other Ghost insances, but also Mastodon and all other platforms).
While posts continued to federate between Ghost sites on the same the Magic Pages server, they weren't reaching the broader fediverse. Here's what happened and how it was resolved.
What happened?
The issue began on June 20, 2025, when I deployed an update to the ActivityPub server codebase, which introduced a commit that looked harmless, but had quite some consequences.
Commit 362c9cb (https://github.com/TryGhost/ActivityPub/commit/362c9cbd5063ab6ec707771679a56e6c8e1cb246) removed the legacy /mq endpoint in favor of the new /pubsub/fedify/push endpoint structure. There were two issues though:
The first one was the Magic Pages infrastructure. Since it is run on Kubernetes, there are ingresses for every single Ghost site. These ingresses also define certain endpoints (like /mq) that should not be handled by the Ghost sites, but by the ActivityPub server.
With this new commit, the ActivityPub integration expected a /pubsub/fedify/push endpoint, but that was not defined in the ingress.
I thought fixing that solved the issue. But no.
The second problem was the GCP PubSub queue that processes messages for the ActivityPub server to federate to other intances (e.g. Mastodon). Its configuration contained an endpoint that it sends messages to: /mq. Well, the commit removed that endpoint. The messages were rejected with a classic 404 error.
I have updated the configuration to use the new /pubsub/fedify/push endpoint and well...
Within a few minutes, all messages (basically posts) that were in the PubSub backlog were processed and things started working again.
The only issue: this created massive noise for all followers of ActivityPub profiles on Magic Pages. All posts and notes since yesterday were essentially sent to these followers within a few minutes. Ouch.
So, apologies for that from my side. If your followers are complaining, send them my way. Totally my bad for not spotting that commit!
Going forward, I'll make sure to check commits more thoroughly. In this particular case it is a bit hard, since the ActivityPub repository does not have classic releases or version numbers. Hopefully that will come soon. Until then, I'll go over every commit that goes out 🙃
