Okay, we're fixed.
First and foremost, I would have been at this much much much longer without the efforts of Deety and OCBucksFan. So take a sec, and send them some greenies, a kind PM -- or if you're OCBW, perhaps something more involved (like a pat on the back or something).
Secondly, this was not the direct result of a malicious attack. There are a variety of reasons it seemed to be. For one, the site we were connecting to is a documented spam pit. As demonstrated by the multitude of domains OCB rattled off above, it has both legitimate users (mili) and illegitimate users. BP is constantly under a sort of low-threat-level attack. Indeed, as I type this, some mooch is trying password after password against known logins, trying to get into the machine. Normal stuff. We're regularly scanned and searched for known security holes -- in the OS, in the software behind the site, in other pieces of software that run on the machine, in vBulletin, and even in the modifications we run on vB. I try to stay on top of all of that as best I can, but as demonstrated by the redirect fun we had last year, the tools sometimes get the word before the good guys do.
The problem itself was caused by a single line of code, that existed only as a single cell in the database. The code itself was something I created, months ago, to be called whenever a flag I'd place in one or more of our templates was identified as a PHP script was run. Which is to say, I'd put this flag in a template, and when one of us would load the page with that template, this code would run.
Thing is, the template it was going to go into was never active, and the flag itself, never used. So clearly, something somewhere isn't working properly. It remains to be seen if this is a problem with the core vBulletin code, or if it's related to a properly functioning modification, or an improperly functioning or otherwise compromised mod (such as the plaza).
So, as I said in another post, "I found the culprit, and he is me." At least in the sense that I wrote the code.
I believe the actual slowdown only started once the poor site on the other end actually firewalled against us. That caused time outs, the time outs bogged PHP down since nothing over here said "well hell, this isn't working, I should just stop, at least for a while," and the bog boggled me.
The "what" has been answered. The line of code. Now set ablaze, stomped upon, verbally abused, and otherwise maligned.
The "why" and/or "how" remains a bit of a puzzle, but I'll continue to poke around there as possible. I do feel confident the horrific load times won't return (at least not as a result of the same series of problems). Even if the potential is still there somewhere, the 'tool' it used (that code) is dead.
Again, thanks to Deety and OCB.
Finally, I blame NOTREDAMECHIEF for all of this -- add to that the Steve Erkel look and GAWDDDD! Not for any good reason -- just because it's good to have someone to point towards when rousing rabbles. With his 0-3 touchdown-less start to the season, he's probably in too deep a peach margarita-induced stupor to care.
