The Golden 1 Credit Union's data center experienced a complete loss of power last February. This resulted in a hard crash of the mainframe system as well as all networks and other in-house systems. The cause of the power failure, which was isolated to our data center, was ultimately determined to have been a faulty circuit board in the emergency power off system. This system serves as a life-safety measure to override all power sources, including emergency back-up generators, in the event of an actual fire in the data center.
I hope this type of experience never occurs at other credit unions. However, the lessons learned may enable others to be better prepared for a similar situation.
In an emergency situation, the command and control approach to leading and managing is key. Swift, decisive and focused actions are critical to manage and mitigate the numerous priorities and impacts inherent in a crisis environment. Proactive and prompt one-on-one board communication resulted in full support for management's handling of the situation and allowed management to remain focused on crisis management during this difficult period.
Recommended For You
In addition, proactive, honest and succinct communication with key media contacts facilitated appropriate messages, eliminated media sensationalism and created numerous valuable member communication channels (TV, radio and print).
As a result of recent bank failures, some members panicked when the credit union suddenly lost connectivity with its Web site, telephone system, ATMs and the like. In the future, management will share press releases about emergency situations with our primary regulator to ensure peace of mind for all involved.
While social networks included ill-informed commentary from a few bloggers who made incorrect and derogatory assumptions regarding the crisis situation, management was prudent to not waste valuable time responding. Supportive and well-informed members ultimately set these misinformed individuals straight without any encouragement or help from management or staff.
We found that early assumptions regarding the cause of the incident turned out to be incorrect. Fact finding and validation was necessary to ultimately determine the true cause of the loss of power-without which appropriate corrective measures would not have been possible.
The process of validating the integrity of the mainframe database system went flawlessly and was a great example of how business continuity measures can be properly executed. Management knowledge of appropriate actions to take in the event of a disaster situation helped facilitate an orderly process that was efficient and effective with respect to systems restoration despite a complex environment.
The immediate inclusion of key business partners to help troubleshoot the situation proved invaluable. Management should never be too proud to ask for outside assistance early on. Strong business partner relationships are an invaluable resource.
The information technology team members remained focused and level-headed during an extremely intense and stressful situation. As president/CEO, it was essential that I balance a need to remain informed on a periodic basis without impeding the IT team's focus or efforts. In a crisis mode, the president/CEO must be able to trust and rely upon the executive team members' strengths and competencies, including sound decision making and risk management.
With systems fully down, e-mail and Blackberry communications did not function. Cell phone and wireless communications were the critical link for crisis management and the related communications. In retrospect, broader internal deployment of wireless communications would have been beneficial. Similarly, while electronic copies of business continuity plans are nice to have, hard copies are a must in this type of situation and save a lot of valuable time.
A critical judgment-to migrate to our backup site or remain focused on restoring data center power and systems at headquarters-had to be made quickly based on limited information. While this decision was being carefully weighed by IT, key staff were dispatched to the backup site to provide maximum flexibility. The ultimate decision to restart systems at headquarters and not migrate to our local backup site ensured no further disruption in services and did not lengthen our downtime. IT's focus on carefully thinking through the pros and cons of our particular situation ensured the best path was followed for this incident.
The local telephone company messages of "the number is disconnected" or "out of service" triggered needless member concerns. If possible, make advance arrangements for a substitute emergency-situation message to be played.
While no scenario planning would have anticipated the situation as it actually occurred, the business continuity plans management had prepared and periodically reviewed proved to be effective and useful in this real disaster recovery situation.
In addition to having the emergency power off system redesigned to avoid the potential of a repeated failure in the future, management has now experienced a live test of our business resumption plan.
While we would never have endorsed intentional infliction of a hard system power crash, the successful outcome resulted in our having an even higher confidence in the business continuity and disaster recovery plans and processes that had been developed, documented and previously rehearsed.
I hope some of our lessons learned will help others recognize the importance of advance thought and planning related to business continuity planning, the value of command and control leadership in this situation, and some areas they may want to review in advance to be better prepared in the unlikely event their business continuity plans must be invoked.
© Touchpoint Markets, All Rights Reserved. Request academic re-use from www.copyright.com. All other uses, submit a request to [email protected]. For more inforrmation visit Asset & Logo Licensing.
