On Saturday September the 10th, it was reported that ING Bank in Romania had an entire regional datacentre taken offline for ten hours, resulting in the loss of all banking services. A disaster of this size might have been expected to be caused by a long term regional blackout, earthquake, or possibly a terrorist bombing, but no, it was caused by sound… a very loud sound.
ING was performing a proactive test of their fire suppression system, and the high intensity of sound, reported to exceed 130db, disabled many of the hard disks in their datacentre. Although this is something completely new to most of us in I.T., apparently it has been observed and studied by the Fire Suppression community for many years. I apparently did not get that memo, and I doubt many others in I.T. did as well.
So what is it about loud sound that interferes with the operation of hard disks? There appear to be two related effects that sound has on hard drive operation. The first is that the disk platters and heads are vibrating, making it difficult to impossible to accurately read and write to the disk platter(s). If you are reading data, it means that the drives will perform very poorly, but will recover their performance when the sound stops. If you are writing data, performance will also be poor, with the possibility that data will not be written to the correct location, corrupting your data on disk. This corruption of data is permanent.
The second effect of the loud sound is that one of the disk heads may actually crash into a disk platter. Heads are said to be “flying” as they skim across the platter, on a thin layer of air. When a head is said to “crash”, it touches the platter surface, creating a divot and releasing metal particles into the previously clean environment of the hard disk. After a head crash, even if all of your data is recovered, the metal particles released into the drive cause it to become a “ticking time bomb” just waiting to cause future data loss.
Now don’t assume that a datacentre affected by such an event just needs a few hard disks replaced, and perhaps a little data restored. The reality is the integrity of every hard disk in the datacentre is now questionable, and it would be prudent to replace every single drive.
In the studies linked below, it was found that it was both the sound from the high pressure release of fire retardant, and the alarm horns, that caused the drive damage. If you have a datacentre, or use a datacentre, now is the time to ask what steps have been taken to minimize the risk of damage being caused by the fire suppression system.
Additional Sources:
