I’ve been working in BzzAgent’s Technology Department for just under a year and a half now. As part of a small team, I get to have a hand in everything and it’s been an incredible learning experience. Despite working in a support role, my day-to-day tasks are surprisingly low-stress. Sure, there’s the occasional fire drill when thousands of Agents flock to the site at once (hi, Sonicare!) or—horror of horrors—Dave’s laptop tweaks out. Thankfully, these “fire drills” have only been metaphorical. Until this week, that is…
Among many other perks, our new office came equipped with a real, honest to God server room. Air conditioning, doors that lock, important-looking flashing lights, the whole nine yards. I wish I had photos of our old setup, because it was a sight to behold. Servers were stashed in whichever closet had space (preferably, but not always, ones with ventilation). Racks? Who needs ‘em, we’ve got some cardboard boxes. I should add that this is not at all a knock on the tech team, but rather a testament to how ready BzzAgent was for this new place!
Most mornings, I am one of the first five or so people into the office. I’m not actually awake (never been a morning person), but I’m here. This is important, because the hours of 7-9am are like the BzzAgent Bermuda Triangle. The strangest, most disastrous problems inevitably crop up when I’m the only person around to deal with them. It’s sort of like being 24’s Jack Bauer, but with less explosions.
On this particular day, in my pre-caffeine haze, I noticed a peculiar beeping coming from the server room. Now, the Third Law of Tech Support (right behind “is it plugged in?” and “reboot til it goes away”) is “loud beeping = bad”. I forced myself away from my coffee mug and ventured into the closet.
I was greeted by a blast of hot air that was easily 100 degrees. For reference, we normally keep this room in the mid 60’s. The thermostat was beeping frantically, proclaiming “HIGH TEMPERATURE ALARM” on the screen. At least half the servers had shut themselves off, with several others adding to the chorus of beeps.
It was going to be one of those days.
We wound up being very lucky; nothing was permanently damaged, and all of the servers came up once the room aired out a bit. If this had happened over the long weekend, we could have been (to use the technical term) Royally Boned. The whole incident was caused by the air conditioner simply breaking down, nothing we could have foreseen. It was quite a way to start the day.
Have any of you BeeLog readers gotten to work and discovered a disaster in progress? What’s your best “worst case scenario” story?
PS: I’d like to thank our heroic Office Manager, Sue, for getting the air conditioning repaired ASAP. She’s the best.
ReTweet
Move over Charlie! Make room for John Henry!
When I owned my own Laundry business, I came in to the shop one day with water all over the floors and customers with rolled up pants wading through all the chaos. It was a blessing really, the tanks needed cleaning out and the drains needed cleaning. If this would have happened on the weekend when I wasn’t around geesh it would have been disastrous, the weekends are the busiest days for Laundry.
Wow if Melissa isn’t a “glass is half full” kinda person, then no such person exists!
Ah, Grasshopper, you want to share high tech war stories?
After 25 years in the computer business, I’ve got a TON of ‘em!
Let’s see, how about the time that I spent days rebuilding a UNIX/Oracle development system time and time again with programmers and managers breathing down my neck. After about the fourth re-build, I finally call in the hardware guys only to find out the reason the hard drives kept misreading was a very slight (as in microvolts) high reading on a controller board that sent it out of spec enough to cause the corruption and misreads on the drives.
Or another job where the company had skipped regular maintenance as being an unnecessary expense to the point the read heads on their minicomputer’s system drive had so much corrosion on them it was physically damaging the drive platters and wiping out such minor things as compilers and program files.
Or when a client decided that I had made a typo in written step-by-step instructions I had made up for an overseas location so he “corrected” it before sending it on. After all, the instruction couldn’t have been “umount”‘; it must really be “unmount.” Result being said overseas client couldn’t get the instructions to work, so it must be the CONTRACTOR’S (me) fault, not the non-tech, pencil-pushing moron who changed the instructions.
I could go on, but I’m sure you get the gist. What you had was a minor blip in the wonderful world of Computer Science.