start rant
it started sunday night around 9pm, i got an email from my boss that our datacenter was 90 degrees, and things were starting to act funny, so he was going to let some fans run all night and i told him that i would be there by 7am, since i live an hour away there was no way i was going in last night....
i get to work this morning at 700am, my phone looks funny. i go into our datacenter and find that our phone server has taken a nice steaming shit! Our company gets approximately 1100 calls per day so our phones being out is a HUGE deal. start diagnosing the problem and find out that the OS has gotten all fuckered up, and our last good backup was..... 3 weeks ago! (fucking backup admins!) so, we spent 2 hrs reinstalling the OS, and now its up, however the 3rd drive in the raid array is starting to flake out, so we've been copying data off, including 45GB of recorded calls, transferring across the network - that still has another hour to run approximately. once its up we have 3 spares that we are going to put in place, then recopy data back...
oh least i forget, during all of this our email server crashed. we have a simple raid array inside the server with just the os, and all of the database information (mailstore, ect) is on an external raid array box, and now the two devices dont want to communicate! we just replaced the scsi controllers in the external device as well as replaced the back plane. we have a guy now running to get a scsi card for the actual server.... its 8pm as i'm writing this and i dont know when i'll be getting home....
end rant
it started sunday night around 9pm, i got an email from my boss that our datacenter was 90 degrees, and things were starting to act funny, so he was going to let some fans run all night and i told him that i would be there by 7am, since i live an hour away there was no way i was going in last night....
i get to work this morning at 700am, my phone looks funny. i go into our datacenter and find that our phone server has taken a nice steaming shit! Our company gets approximately 1100 calls per day so our phones being out is a HUGE deal. start diagnosing the problem and find out that the OS has gotten all fuckered up, and our last good backup was..... 3 weeks ago! (fucking backup admins!) so, we spent 2 hrs reinstalling the OS, and now its up, however the 3rd drive in the raid array is starting to flake out, so we've been copying data off, including 45GB of recorded calls, transferring across the network - that still has another hour to run approximately. once its up we have 3 spares that we are going to put in place, then recopy data back...
oh least i forget, during all of this our email server crashed. we have a simple raid array inside the server with just the os, and all of the database information (mailstore, ect) is on an external raid array box, and now the two devices dont want to communicate! we just replaced the scsi controllers in the external device as well as replaced the back plane. we have a guy now running to get a scsi card for the actual server.... its 8pm as i'm writing this and i dont know when i'll be getting home....
end rant