DSLReports down for a day and counting

DSL Reports has been down for a day with SQL server problems following a power failure.  This should serve as a warning to everyone who has a web site or blog to make sure they are protected.

Make sure you have adequate UPS support to give the server enough time to shut down in an orderly manner.  Make sure there is a connection between the server(s) and the UPS to start an orderly shutdown.

Make sure you have adequate backups and test them by restoring to an alternate server for testing.

I do not think you want to be the person who has to post something like this on their home page.  As of this post, DSLReports is still down.

Mon Apr 16 13:52:50 EDT 2012

dslreports.com is offline

————————-

Update Tue Apr 17 04:14:29 EDT 2012

Things have turned out to be more complicated. Although no hardware was fried, our large storage array has decided, probably as a result of going dark in mid write, that our two sql servers can no longer have access to one of the small, but important, partitions that makes up the larger SQL data area. All data is there, but it refuses to offer it.

At this point all courses of action involve delays. When I know more I’ll post 🙁 -Justin

Update 15:35 EDT

Our data center experienced a power outage which affected our entire system. We are in the process of recovering everything and turning a few cranky servers into happy ones.

Apologies for the inconvenience, we expect to be up in a few hours

Update 16:18 EDT

————————-

The power outage affected the entire data center and we are still awaiting our turn to power up some machines.

We will keep this page updated.

Update 19:20 EDT

————————-

Unfortunately, a database server doesn’t take kindly to being taken down without proper shutdown procedures and the

restoration efforts are still under way. No ETA at this time.

Print Friendly, PDF & Email
This entry was posted in Technology. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *