Well… we made it in the end, but was a bit of a struggle to get here. So here’s a brief run down of what happend:

• At 10am I turned outside access to the servers off and started backing up the database (just in case anything happend). Did a quick tidy of loose files and started off the data hosting migration process. From previous upgrades and resizes I assumed that this would be relatively quick (hence my initial 1 hour guesstimate)

• The first server (Mint – our Search/Jobs/Backup/Email server) started the moment I pressed the button (Which was a good sign). Then nothing happened… for 2 hours.

• 2 hours in the second server started the migration process (Fennel – our Database server). The second server took roughly 4 hours to migrate, and is one of our largest servers, the Folia database alone is 4Gb in size, so this became our yard stick for how long the following two servers would take.

• From here on we started asking our data providers what was happening, and it became apparent that we were in a very long queue of other server migrations. It was an open chatroom so a lot of other people were coming in and asking similar questions. Some of them had been waiting for hours (even days) to have their migrations prioritised. It was this time we feared that we would be offline for the whole weekend.

• After 13 hours we were in a bit of a huff and managed to get one of the engineers to “force migrate” our two remaining servers. Much to no avail. They remained in the queue for another 10 hours.

• 17 hours in Mint initiated an automatic shut down (suggesting it completed). Apparently this was a false alarm and was just entering a mysterious “Phase 2”

• 23 hours into pressing the migration button Thyme (our proxy server) and Basil (our application server) started migrating.

• 25 hours in Mint performed another reset and had indeed finished the migration. We were just waiting for Basil to finish.

• 28 hours in Basil finally migrated. A few tests and we’re back online.

So that’s the rundown – mostly us waiting for things to happen. We updated the maintenance page to highlight the progress better and to explain the situation a little clearer. We didn’t want people to think that with the site being offline for so long that we somehow lost your data.

It was a migration process completely out of our control and so poorly handled by our hosting provider. Which is terrible considering that it was a migration that we didn’t have a choice for. I would have been happy to not migrate, but if that was the case it would have happened automatically in a few days time with unknown consequences.

I was hoping that but setting the migration day that we would have been better accommodated for but that wasn’t the case. We pride ourselves on good support and accessible site with very little down time, but this weekend’s outage has dented that.

All we can do is apologise and hope to make it up to you in the near future.