[Leica] Outage

[reid at mejac.palo-alto.ca.us (Brian Reid)]

On Monday May 20 I got a call from someone I didn't know saying that my late 
best friend's widow was in the emergency room in Princeton and had just been 
diagnosed with a spinal tumor from diffuse large B cell lymphoma and would 
after palliative treatment be released to two friend-of-a-friend types. They 
called me as a last resort because they couldn't find anyone else to whom to 
hand her off, and she wasn't in any shape to take care of herself.

I instantly got on an airplane to go deal with it. I'm pretty much all she's 
got left. To Burlington, New Jersey, which is a far suburb of Philadelphia.

Before the airplane reached cruising altitude, a critical component in my 
server complex failed that resulted in my entire internet operation being 
offline. Not only was the LUG down, but all of my personal and family stuff 
was offline as were the mailboxes of the other 1900 people for whom I run 
email service (not everyone trusts gmail).

Juggling my time between dealing with the cancer patient and remote 
diagnosis of the network problem, by Wednesday evening I determined that the 
problem was the HP2848 switch that connects my house to its fiber optic link 
to the internet exchange. These devices are not supposed to fail, enough so 
that HP offers an unconditional lifetime warranty should one ever fail. Fat 
lot of good that does at 2am when you have a broken one.

You can't just get a new device and stick it in. These things have to be 
configured. And of course all of the configuration information is on servers 
that I can't reach because of the outage.

By various processes too complex to explain here, a team of 6 people 
coordinated by my daughter Elizabeth managed to locate a replacement and get 
it configured and installed.

This is by far the worst outage I've had in the 20 years I've been doing 
this. I couldn't even read my own email. Now that I can read it again, it 
has hundreds of "hey, is the server down?" messages. Forgive me if I don't 
answer them individually.

I'm leaving my cancer patient in paid-for hands and heading home tomorrow.  
None of that gear would *dare* to fail when I'm home, though.

Whew.

Brian Reid
LUG Saloon Keeper and part-time unlicensed oncology nurse