The nightmare of the day
by blackhatgk on Dec.21, 2003, under My personal dribble
Ok, so I woke up this morning, went upstairs to get some information off the computer to take with me to work and what do my eyes see?
My dedicated server is dead…. This is very, very heart wrenching. Not only because it is not responding at this moment, but also because I do not have a) physical access to this leased box or b) console access to this box.
I was already running close to time to leave for work so I head on out to work. When I get there I start to research the issue and discover that the last couple of messages I recieved from it the cpu load was pegged out.
So i contact the company I lease my server with and submit a reboot ticket. After 30 minutes they confirm they couldn’t get ssh access to it either and reboot the box. It comes back up and after some quick investigation I found the culprit……
Drum roll Please……
Over 90,000 email messages.
End Drum Roll……
This is what happened. I installed a script that will take email messages and throw them into a database for me to do with as I please.. This script also sent out an email to the sender saying “Thank you and this will be taken care of”.. Well, at 4 o’clock this morning, my credit card processing software sent me an email of today’s invoices. The to and from field for that email were the same email address as my little perl script. So the perl script kept emailing itself over and over again… This in itself was not enough to kill the server. What killed the server in though was the fact that the database tables got corrupt from this processes speeding up exponentially. Once the tables were corrupt, it was only a matter of time before the server finally gave up.
The fix for all this madness was pretty simple:
1) Disabled the automatic script insertion thing until I can put other safety precautions in place.
2) Disable all outgoing email being generated by this script.
3) Move the to: address of the invoice generator to a non-forward so that it will not do this again
Now I am stuck with trying to download and delete over 90,000 emails off the server. I could wipe out the entire email account and start over, but being that the server was offline, I want to make sure I get all the emails I can out of it to reply back to people.
