So, the tale of my day so far. First off, I had to get up early because our telemarketing had a phone issue, so at 6:00 AM, my phone rings. So I walk in this morning, take care of the phone issue, and I notice that I have a lot less spam than usual, so, being proactive, run a few reports and realize that internet mail usage was at 2% of it's usual flow after 10:00 PM last night. Hrm... curiously, I hop over to our website, or try to, only to find out our name isn't resolving.
Crap, this can't be good. So, I call our ISP who hosts our website and pretty much has my DNS records hostage. My favorite line ever, "well, I can go to your site just fine from here" Yeah you can, ya fucking moron, you're using your internal DNS that has my records in it.
So, going back and forth from 6:00 AM this morning, I decide around 1 that it's time to get ahold of a supervisor, I have no email, our web site is having issues and my DMZ is useless because of cert errors.
They inform me that they lost our DNS records....
How the hell does an ISP lose our DNS records, they host our damn website, yet suddenly, just our records vanish, no one elses, gah, I friggin' hate ISP's.
So, after a rather long discussion, they tell me that they have copied the zone records from their internal DNS server and have added our changes, so a whole fucking day of downtime because some friggin idiot in a noc can't figure out how to do the basic network fucking task without researching it for 8 hours.
Hey, I guess I should look at the brightside, they won't bill us for this month. That's good, I won't be with them next month,.