I'm collecting data that Jelsoft has asked for. It's purely a server issue, and I've already been told we're right about at the point where web needs to be on one box, and the mysql database handling on another. Which feels a bit like one of those road construction scenarios, where, by the time they're done widening, it needs to be widened again. :(
My goal right now is to maximize production of what we have. There's no good reason why we should have 20+ second wait times at moments when there are 500-600 users on at once.
This may well mean a downgrade or removal of some services. Full-text boolean searching for example. Across a 1.7gb database, if 10-12 users are doing complex searches at once, it's a database spike. Searching is already off for unregistered users, so I may simply have to undermine our search handling a bit to offset the impact on the server.
We're also being hurt a bit by the fact that our old machine (hopkins.buckeyeplanet.com/bryants.net) is dying. The power supply is failing, and that server was handling some of the heavier image requesting, which means those tasks are falling back to labounty.
Anyway, it's something I'm actively working on. I should be turning data over to Jelsoft tomorrow or Thursday at the latest, which means I hope to have solutions in place by Thursday or Friday.
I believe those solutions will resolve the worst of the delays during peak load times (mid-day), but understand that with more users (and a metric shit ton of more unregistered browsers/lurkers/readers) the baseline for this situation is going to continue to get worse, even if the worst of it should get much better in the short term.