Our company had grown quickly and our server room was less than organized. It consisted primarily of two gorilla racks packed with dozens of custom built servers, and at the far end switches, hubs, routers, and a fiber drop. Getting to the machines was tricky, involving careful navigation of all the cables, UPSs, and other miscellaneous network hardware that cluttered the approximately three foot clearance behind the gorilla racks. Anyone who had anything to do with the server room was exceptionally aware of our two main servers, Solo and Chewy, which housed our client's most important sites and databases. These two machine were sacrosanct, and were monitored by the second.
Our main networking ace had recently left the company, and we were coping with his replacement - his previous apprentice. She might have known the technology, but in other areas she was a bit lacking. I was trying to cut her some slack since she was inexperienced, young, and the rare woman in technology (my personal version of Affirmative Action that I've since abandoned). Eventually it became obvious she was in the wrong line of work.
One afternoon I get a call from a client: their site's down. It's a site on Solo. I do the required panic, and with a quick check, sure enough their site is down. Wait, no, every site I try on Solo is down.
I IM networking:
"Do you know that Solo is down?"
"No, I was just in the server room and everything seemed fine."
"I'm pretty sure it's down."
"Maybe the outgoing DNS is messed up again and we can't get to our own sites."
"No, a client called. They noticed it was down. I just verified it."
"Just a sec."
I get up and walk over to networking. On her screen, next to our IM conversation, is "Sid Meier's Pirates!" She's piloting a ship around.
"I found this on Underdogs. I love this game. I haven't played it in so long. I'm pretty sure Solo is fine. It's something wrong with our internal network."
"Yeah, I like that game, too. Could you let me in the server room?"
We head over to the server room, she unlocks it, and we go in. It's freezing cold and loud. Solo and Chewy are on, but with little disk IO, and Solo's networking light isn't flashing.
I switch the KVM to Solo and check all the obvious things. Nothing seems amiss, but it's off the network. I check the back of the machine, everything is plugged in, but I reseat the connections anyway. Still nothing.
"What did you do in here just a bit ago?"
"I plugged a new machine into the switch."
"It's voodoo, but let's disconnect it and see if that changes anything."
I stretch through to the back of the switch rack, and with some serious contortions, peek behind to find the right cable to disconnect. Thank God she labeled it. More nothing. I'm looking at the serious snarl of cables and hardware and begin to realize I'm in trouble. I start grasping at straws.
"Okay, could you show me exactly what you did when you were in here?"
"Sure. I brought in the machine and put it up here. I plugged it into the UPS, and got a long network cable for it. It's a long way from the switch. It's headless, so that's all I did. It's not even on yet."
"How did you choose the port to connect the plug into?"
"I just chose an open one. I'll plug it back in and show you."
She squeezes behind the gorilla rack and blithely proceeds to tread on the cables lying on the floor, sometimes yanking a machine's connections so hard that the case wiggles. She notices my whole-body cringe and quickly backs out.
I then squeeze behind the gorilla rack and start pushing in every single network connection I can see, be it machine, hub, switch, or router. She starts pushing them in from the front. I do my best to step in the little areas of exposed concrete amongst the tangles. A few minutes later Solo was back online.
I never knew which connection it was that was loose. Later we cleaned up the cabling and it wasn't so much a concern. Our Peter Principal network admin didn't stay around much longer. She had other telling mishaps: database backups that failed due to the drive being filled with her MP3 collection; setting up IIS and leaving the default SMTP relay open getting us blacklisted; and sometimes assigning the same address to two different machines or an address from the DHCP pool, sending us on a company wide game of IP address Whack-a-Mole. Worst of all was coming in late, being unreachable in emergencies, or completely out of it because she had a good night partying. I think she too eventually realized she was in the wrong line of work. Last I heard she was managing a bar.