data center disaster

I work in the electric grid industry with an ISO/RTO.  We have outgrown our currect datacenters and are in the midst of building new ones.  A lot of time and trouble went in to building these datacenters (needless to say).  Sometime during the night on Thursday, one of the intake fans for cooling had a bit of a meltdown.  It came lose from its frame and started to grind it's blades on the outer manifold.

What this did in essence is spray our brand new datacenter with small particles of metal.  They were very small.  You could run your hand down the ledge of equipment and have sparkles on your hand, but no cuts.  Of course, you could feel the cutting action after some time if you didn't wash your hands!

We were not in production as the datacenter is not fully complete, but we had moved in quite a bit of infrastructure.  We had a Hitachi VSP which is now trash.  We had a Cisco 9513 SAN director which is probably trash.  We had a NetApp 6240 which is probably trash as well.  There was also Cisco network equipment and an IBM Power7 system.

We equiped the datacenter due to a timeline which was pushed back for multiple reasons, but the datacenter just wasn't ready to be populated.  At a minimum, this is easily $10m worth of eqiupment which is either trash or questionable.  Not too mention all the ports within the patch panels and structured cabling that are populated throughout the datacenter that are infested with tiny particulate metal.

Now I am not saying this would have been caught within a break-in period.  We could have fully well went into production following all recommended procedures and this disaster could have still met us.  I just find it interesting that we spend so much time on making sure there is no single point of failure on our designs of systems and networking, but something outside the scope of what we do can just wreck havock on the best designed system.

I have heard of plenty of datacenters that have taken on water and such.  This isn't a large disaster, but it is one that I have heard of or thought of before.  I thought I would share!