On the evening of 20 October, between 19:04 and 20:46 UTC, a network problem at our hosting provider Hetzner disrupted Magic Pages sites in several waves. 84 sites were completely offline for between 12 and 56 minutes. Another 260 couldn't load uncached pages or Ghost Admin for 19 minutes. And about 1,320 sites restarted once and were unavailable for about two minutes each.
I want to take a few moments to write down what happened why it came in waves, and what we changed during the evening.
What happened?
Magic Pages runs on dedicated servers at Hetzner. A bunch of them run the Ghost sites. We call them workers. They talk to our database, our storage cluster and other server that manage them over a private network that Hetzner provides, called a vSwitch.
At 19:04 UTC, that private network started dropping connections between some of our servers. A database server got lost first. At 19:12, a storage server was cut off, and at 19:13 one worker lost the servers that manage it.
Hetzner has an open incident for this, Technical issues with vSwitch MAC learning. Other Hetzner customers reported the same outages on Reddit. Similar to people on Reddit, I have noticed short issues throughout the whole day, but nothing really affected our production sites.
The actual impact this evening depended on which worker your site runs on:
- On one worker about 260 sites couldn't reach the database from 19:37 to 19:56 UTC, plus a minute and a half at 19:07. Pages already in Cloudflare's cache still loaded. Everything else failed with an error after about 10 seconds. That includes logging into Ghost Admin, publishing, member sign-ups and pages that weren't cached yet.
- On four other workers, about 1,060 sites lost the database for two to three minutes between 19:45 and 19:50.
- Some other sites kept working throughout, because their network paths weren't impacted.
The database cluster and the storage cluster recovered on their own. Nothing that was already saved was lost.
What have we already changed?
Since we saw some instability with Hetzner's vSwitch in October in general, we already prepared a plan on how we can prevent this in the future. This plan was meant to be implemented when things were a bit calmer, but since sites were already impacted, another 2-3 minutes of downtime didn't matter too much, in that particular moment.
All workers can now reach the servers that manage them, as well as database and storage clusters, over a secondary encrypted network as well, in addition to the vSwitch. If the vSwitch drops again, the servers should keep in touch with each other over this secondary network.
What are we still doing?
As a next step, we will be rolling out this secondary network to our entire fleet and will transform it into a mesh. If one worker loses connection to a database cluster, but another server still has that, the broken worker can use the connection of the other one. It's basically an alternative route to the destination – the database server.
Hetzner's incident has been open since 19 October and is still open as I write this. It's also the second time this month that their vSwitch has taken our sites down, after a shorter incident on 1 October. Should this happen again, we expect that the impact on our infrastructure will be minimal.
If you have any questions, send me a quick email or message in our chat.
Jannis
