Hardware failure/site crash (2/6/09)

[Clarity]

Five weeks since the last one. Could be worse, I guess.

Some crashed tables and issues in the database, both here and on RP, but I think I've completely repaired and optimized all of that.

As always, please watch for any errors or unusual problems and report them here in this thread. I'll be watching it closely for a while and don't expect any issues (until the next crash) but we'll see how it goes.

Wish I knew which piece of hardware was causing these problems.

 

[Dryden]

Any symptoms? Does the machine lock up and require a hard reset?

I can't believe it's been about five years already on the "new" server ... time flies.

 

[MililaniBuckeye]

That was actually a pretty fast recovery. Good job, C-Dog...

 

[Clarity]

Yeah, they find it running, but when they hook a crash cart up there's just a blank screen and it's non-responsive -- nothing to indicate why it locked up. They power it down and back up in person. Then I get a shell up after and start trying to stumble through the database problems.

When we were suffering crashes like this a couple years ago, removing all but the first two sticks of ram put an end to that. We do have 8 sticks in there again, and maybe doing the same thing would resolve the issue, but it would also take us down to a quarter of our memory, and after dropping the cash I did on that last year (and our existing performance issues during heavy loads as-is) I'm not excited about doing that.

I guess that makes it sound like a motherboard issue, where sticks 3-8 connect to the board, but I can't even swear to that.

Looking back, we crashed twice in September, twice in December, and now once today. Each time, it's crashed tables in the various databases and done some carnage within, but (so far) it's all been recoverable/repairable. At some point it's going to be too ugly to do so, and I'll have to roll back to the last hotcopy (made every morning at 4am ET), but the whole thing is what it is.

5 years ago? Really? Wow.

 

[DaytonBuck]

site seems like it's loading faster now on my computer

 

[Deety]

Dammit. I know that's unsettling.

 

[Dryden]

5 years ago? Really? Wow.

Built it in the summer of '05. Well, I guess that makes it ~four years since it's only Feb ... We'll be in the fifth year of its duty cycle before this year is over.