Acu vreo cateva zile, pe NANOG s-a apucat lumea sa se certe daca e bine ca intr-un datacenter sa ai un buton de EPO (Emergency Poweroff , pentru profani) sau nu. Discutia a evoluat si s-a ajuns la povesti cu ce experiente au mai avut unii si altii cu incendii, inundatii si alte calamitati in datacentere, si mai ales ce se intampla cand stiinta da nas in nas cu religia:

Povestea #1

While working at a distinguished university with a religious affiliation, I learned, as did one of the priest-biologists, not to refer to a piece of instrumentation as possessed. While one of the priest-theologians meant well, we learned what happened when holy water is sprinkled into the high voltage supply of a gas chromatograph. Beckman Instruments was so amused they didn’t charge for equipment abuse not under maintenance contract.

Povestea #2

One of the server rooms becomes unusable and needs to be rebuilt[0], so everything needs to be migrated out of the existing room and into new space — this includes a large APC Symmetra UPS. We shut down the UPS and pull all of the batteries out of both it and the expansion shelves so that we can move it with a pallet lift. We move everything into the new space and its time to put the UPS back together. I quickly decide that lifting large numbers of heavy batteries into the shelves is not fun, so I show the random helper dude what to do… “You pick up this big, heavy thing and put in into this cubbyhole type spot, then you connect this large connector and slide the battery back, lather, rinse, repeat…”.

I watch him do the first one and he seems to have it figured out… I wander off to go hook up some fiber or something and peer down the corridor every now and then to make sure he still has this under control. Surprisingly enough he is managing ok and hasn’t wandered off to take a nap or anything. He gets down to the last few batteries and seems to be having some issues, but I figure he’ll work it out, so I carry on with what I am doing… I peer down the corridor again and he is sitting on the floor with his back braced against something, pushing the battery into place with his feet… “Whoa, this can’t be good”, I think, just as there is a LARGE bang, a big flash and much smoke and fire….

Turns out that for the last battery he managed to get the cables caught between the side if the battery and the side of the (sheet-metal) case. When it didn’t just slide easily back, he pushed it really hard and the edge of the case chomped through the cable creating a dead short — this literally vaporized a crescent of metal from the case around 5 inches in radius, flung bits of molten case and battery leads all over the place and ignited the cardboard that we put on the pallet to soften it…

Povestea #3

So I’m working at this place that is really cheap… Our CTO believes that it is stupid to pay for electricians that have experience working in datacenters, because after all, power is power, right?

So, he calls a bunch of people in the Yellow Pages and hires the cheapest guy he can find. Said person arrives and looks a little goggle eyed at all the power stuff — I wander back in a few hours later and he is sitting in the middle of the floor reading the Users Manual for the UPS..

Anyway, he manages to run the three new circuits for us without killing himself (although for some reason keeps switching the UPS between online and bypass), and then starts walking out the door… He stops at the door, looks at the big red glowing switch marked “Emergency Power Off” — and then pushes it….. Everything goes quiet, apart from Rob got startled and dropped the shelf he was mounting onto his foot.

After we got things turned back on we ask the electrician what exactly he was thinking… “Well, I figured the light was on because you were running on Emergency Power…”

Sper sa va fi placut povestile :)