DSL Reports has been down for a day with SQL server problems following a power failure. This should serve as a warning to everyone who has a web site or blog to make sure they are protected.
Make sure you have adequate UPS support to give the server enough time to shut down in an orderly manner. Make sure there is a connection between the server(s) and the UPS to start an orderly shutdown.
Make sure you have adequate backups and test them by restoring to an alternate server for testing.
I do not think you want to be the person who has to post something like this on their home page. As of this post, DSLReports is still down.
Mon Apr 16 13:52:50 EDT 2012
dslreports.com is offline
Update Tue Apr 17 04:14:29 EDT 2012
Things have turned out to be more complicated. Although no hardware was fried, our large storage array has decided, probably as a result of going dark in mid write, that our two sql servers can no longer have access to one of the small, but important, partitions that makes up the larger SQL data area. All data is there, but it refuses to offer it.
At this point all courses of action involve delays. When I know more I’ll post 🙁 -Justin
Update 15:35 EDT
Our data center experienced a power outage which affected our entire system. We are in the process of recovering everything and turning a few cranky servers into happy ones.
Apologies for the inconvenience, we expect to be up in a few hours
Update 16:18 EDT
The power outage affected the entire data center and we are still awaiting our turn to power up some machines.
We will keep this page updated.
Update 19:20 EDT
Unfortunately, a database server doesn’t take kindly to being taken down without proper shutdown procedures and the
restoration efforts are still under way. No ETA at this time.