Create: Aloft
On 15 June, Create: Aloft went offline for roughly five hours. The SMP, the Hub, Creative, and the proxy were all down for the duration. We want to be straight with you about what happened, what we did about it, and what is changing so it does not happen the same way twice.
What went wrong
Our server runs on mirrored drives (RAID1). That mirror protects us if a physical drive dies, but it does not protect against data corruption. Anything written to one drive is instantly written to its mirror as well. During maintenance on the host, a low-level storage operation corrupted the filesystem that holds all of our server data. Because the damage was reflected across both drives, the system could no longer read its own data, and every instance went offline at once.
In short: the hardware was fine. The data structure sitting on top of it was not, and drive redundancy alone cannot protect against that.
How we got back
No world data was lost. The actual world files were intact the whole time. What broke was the filesystem index that points to them. Recovery meant rebuilding that index from backup structures buried deeper on the disk, then carefully reassembling the directory tree until every instance’s world reappeared. Once each one was verified, we restored all four instances and brought them back online.
Total downtime was about five hours, and most of that was spent on slow, careful recovery. We worked on copies of the data rather than touching the originals, so at no point were we risking what we had.
What we are changing
Off-site backups. This is the most important one. We have added automated nightly backups of every world to storage that lives completely off the server. RAID guards against a dead drive. Backups guard against corruption, human error, and worst-case total loss. We should have had this from the start. Now we do.
What this means for downtime going forward
The five hours this time went into reconstructing data that had no external backup. With verified off-site backups now in place, an incident of this kind becomes a straightforward restore, measured in minutes rather than hours, followed by a quick integrity check. We are also writing a recovery runbook so the steps are documented and known ahead of time, not figured out under pressure.
Thank you
Thanks for your patience while we worked through this carefully. Your builds, your contraptions, and your worlds are safe, and we are back up and flying. See you on the server.