Server Blues

For those who have followed this site for more than a few years the Server Blues will be a well known theme from our repetoire. We sometimes play the variation Disk Space Blues.

As you might know The Global FlyFisher is 100% non-commercial. In other words: this is yours for free and we don't make any money doing it. The only economy involved in this is the expenses for web servers and traffic. Yes, the more you use GFF the more we pay! True!

Never mind that... We have decided to keep it that way. We took a short trip around ads a few years back. The was quite truoblesome and still no source of income.

So GFF is basically expenses.

In order to keep those as low as possible, GFF piggybacks on a server that I bought and now run for my business. The server runs mine and my Danish partner's common business site and a few client's sites. And it's used heavily as a development and demo server.
It's hosted in a server camp in Denmark.
Since it's our server it is configured and maintained by us, and since we haven't implemented a smart backup scheme, there's no smart backup scheme.

That's not really a good idea, but having a backup or mirror server is beyond our needs and economic capability.

When a hard disk breaks down, you kind of regret dispositions like that. The price of not paying is that you pay in hours when the thing breaks down—which it did a couple of days back.

On the evening of the 16th it ran smoothly, but a couple of hours after midnight, server time, the disk crashed. Not completely, but enough to draw a lot of database stuff into the grave. And none of our sites work without the database.

A phone call to our Linux hacker Thomas in the morning of the 17th gave the sentence: Disk I/O error. He suggested a cold reboot, which could result in three things:
- Cure
- Status Quo
- Death

We left the server on all day the 17th. A few sites were still useable. In the morning the 18th we had our provider flip the big red switch. Unfortunately the server decided not to start after having been shut down.

So we spent the 18th getting new disks, installing Linux, salvaging files from the old disk, configuring the server and patching up dead files.

Now, when I say we, I should in reality say Thomas the Linux shark and Allan, a supporter from the hosting company. My Danish business partner Steven and I myself did a bit, but without the wizards, we would have been lost.

Yesterday at midnight all sites were running again. Some perfectly, others with a few quirks. That was taken care of by all of us during the morning today and now almost everything is back to normal.

A few logs and counters had to be reset, but basically we're OK.

GFF runs as well as ever. And we seem to be heading for a record 24 hours! There are still a couple of hours left before the server turns the 19th into the 20th and we already clocked more than 12,000 individual users today! Our record until now was 12,604 and that seems about to be outdone.

We're back in "business"

PS: Did I mention that I bought two new disks while I was at it? So now the server has 2*160 Gb in stead of 1*80. The extra space and the extra disk will be used for backups!

Me and the beast: our server before it was shipped to the server camp.