Server Outages

What a day! our main clustered fileservers at the office up and died on us today. These two servers are the core of business operations, and they were down from about 9:15 AM until 8:00 PM. We were able to keep them up for a maybe a total of an hour during that time. Everything started with some drive anomalies on one of RAID arrays, then it was all down hill. What finally worked was removing our network teaming software from the servers, reconfiguring all of the networking parameters, and slowly bringing the boxes back online one at time.

We’ll have to see if things hold for tomorrow (we did do some serious stress testing once we got things back up). Lets hope that we don’t have a day like this for a long, long time.

Advertisements
Previous Post
Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: