• New here? Register here now for access to all the forums, download game torrents, private messages, polls, Sportsbook, etc. Plus, stay connected and follow BP on Instagram @buckeyeplanet and Facebook.

Downtime today (5/30/07)

Jump to latest Follow Reply
Sort by date Sort by votes
Clarity

Clarity

Will Bryant
Staff member
Don't know why yet. They worked on it from 7pm ET through 11:30pm, and don't just have a lot of answers. Ugly stuff.

What we do know is there was a hard crash caused by some unknown event. The crash caused serious mysql problems that are 'fixed' for now, but it's still not quite right. We're going to have to just watch it. There could be more problems in the short-term, or it could be okay.

Site performance may not be up to par, as we've had to normalize all the settings in order for it to run at all. Obviously that will have to be dealt with well before the season hits.
 
Last edited:
Clarity;854003; said:
Don't know why yet. They worked on it from 7pm ET through 11:30pm, and don't just have a lot of answers. Ugly stuff.

What we do know is there was a hard crash caused by some unknown event. The crash caused serious mysql problems that are 'fixed' for now, but it's still not quite right. We're going to have to just watch it. There could be more problems in the short-term, or it could be okay.

Site performance may not be up to par, as we've had to normalize all the settings in order for it to run at all. Obviously that will have to be dealt with well before the season hits.
Click to expand...

OCBucksFan;854007; said:
Server down or network issues? I didn't think to tracert, I just pinged and got no response.
Click to expand...


I will stay out of this conversation for fear of sounding like a fucking idiot. That is all.
 
Upvote 0
Well, yes. Server and network issues. Hardware and software issues.

It started with a hard server crash. But when we called the NOC, they couldn't connect to the IP or the router, it's like they weren't there. They went to the machine, found it running, but nothing was responding. They rebooted it, and at some point thereafter the router started responding again, but the machine wasn't showing up on the network. Don't know what they did, but they managed to open it up for at least local connections, and that's when we discovered extensive issues on the machine itself.

It may or may not be tied to some sort of hardware failure. That was one of the early guesses, but the bottom line is there's just not enough forensic data to figure out what caused it.

We're still not 100%, but we're up, and I've backed up everything, and am downloading it all to my laptop. Can't promise there won't be more problems, and right now, there are no answers whatsoever regarding what it's going to take to properly 'fix' mysql.

Just to make mysql work at all, they had to move all of our data to a different part of the drive. Almost makes it sound like the HDD itself has bad sectors or something, but they don't think that's the case. Either way, moving it allowed it to work, and mysql stopped spawning endless mysqld's making it crash.

Eh, I only half understand any of this stuff. I just know a lot of really knowledgeable people were saying things like "I've never seen this before," and it seemed for a couple hours that it could have been a total loss. :(

It's hard to imagine this just happened for no reason. The logs show nothing, which I'm told isn't uncommon for a hard crash. Unnerving though, and tonight sucked.
 
Upvote 0
OregonBuckeye;854085; said:
What movie is that from again? "Dude Where's My Car?"?
Click to expand...

bttfdoc.jpg
 
Upvote 0
Clarity;854027; said:
Well, yes. Server and network issues. Hardware and software issues.

It started with a hard server crash. But when we called the NOC, they couldn't connect to the IP or the router, it's like they weren't there. They went to the machine, found it running, but nothing was responding. They rebooted it, and at some point thereafter the router started responding again, but the machine wasn't showing up on the network. Don't know what they did, but they managed to open it up for at least local connections, and that's when we discovered extensive issues on the machine itself.

It may or may not be tied to some sort of hardware failure. That was one of the early guesses, but the bottom line is there's just not enough forensic data to figure out what caused it.

We're still not 100%, but we're up, and I've backed up everything, and am downloading it all to my laptop. Can't promise there won't be more problems, and right now, there are no answers whatsoever regarding what it's going to take to properly 'fix' mysql.

Just to make mysql work at all, they had to move all of our data to a different part of the drive. Almost makes it sound like the HDD itself has bad sectors or something, but they don't think that's the case. Either way, moving it allowed it to work, and mysql stopped spawning endless mysqld's making it crash.

Eh, I only half understand any of this stuff. I just know a lot of really knowledgeable people were saying things like "I've never seen this before," and it seemed for a couple hours that it could have been a total loss. :(

It's hard to imagine this just happened for no reason. The logs show nothing, which I'm told isn't uncommon for a hard crash. Unnerving though, and tonight sucked.
Click to expand...

Hrm... Yeah, it sounds to me like a hard drive failure is about to occur, that can cause a lot of just freezes too. The part that I don't get is how the router is failing, since it's a completely different device there's no reason why it shouldn't respond unless it was just the sheer number of people attempting to connect through it, but it would take millions of connections to kill a decent router.

Is the drive in a raid setup? if not I would talk to whoever is hosting about getting a current image of the system in case of a full hardware failure, this will protect your data and assure you get back up quickly, if it is in raid I would recommend rotating the drives out.

In my past experiences just crashing is usually one of three areas, disk storage, memory or a simple fan, but since the system had to have data relocated I am leaning towards a disk failure.
 
Upvote 0
MililaniBuckeye;854111; said:
A lot of IP hosting services will block exterior-source pings to prevent ping attacks.
Click to expand...

Pings to BP are not denied, that is done at the firewall level, also I think it's a safe deduction that if I have been able to ping in the past and that the web services (apache) are not responding then there's a problem...
 
Upvote 0
OCBucksFan;854280; said:
Pings to BP are not denied, that is done at the firewall level,...
Click to expand...

You can specify which IPs/sites are denied, and thus you can block specific protocols/services to specific sites. Therefore you can indeed block pings to BP...whether or not C-Dog's provider is doing that or not is another matter.
 
Upvote 0
You must log in or register to reply here.
Back
Top