The post Hurricane Sandy lessons learned keep coming in as it becomes plain that some credit unions had woefully inadequate disaster recovery and business continuity plans, but other institutions kept operating despite the devastation that hit much of New Jersey and New York City.

“For the credit unions that already were prepared, with solid plans in place, everything was relatively calm,” said Scott Collins, president of Grand Rapids, Mich.-based CUSO Xtend Inc., which provided disaster recovery services to five New York City-based credit unions and one in New Jersey. “None of our credit unions lost a minute of uptime on electronic channels,” said Collins, who acknowledged they all had limited or no branch services, primarily due to lack of power.

For one credit union, Progressive Credit Union, the $540 million Manhattan institution known for its portfolio of taxi medallion loans, “We provided phone answering services,” said Collins who indicated Progressive has a contract in place with Xtend where the CUSO, as needed, steps in and answers inbound phone calls with the aim of providing members with questions full and accurate answers.

But “not every credit union follows best practices for disaster recovery,” said Matt Gerber, CEO of IT-Lifeline, a Liberty Lake, Wash., provider of disaster recovery services. “Some don't even comply with the FFIEC guidelines,” he added.

Exactly what counts as state of the art for disaster recovery? Plymouth, Minn.-based TruStone Financial, a $840 million credit union, said it has implemented what might count as gold- standard protection.

“All our data are replicated in real time to computers 300 miles away in Milwaukee,” said Bob Thompson, a senior vice president for information technology at TruStone. He said that no matter what happened to the Minnesota-based computers, “we could be back up and running in two hours.”

Key is that TruStone has created what Thompson describes as “a hot standby site that could take over in a minute.” Data flows from TruStone Minnesota computers to the Milwaukee site continuously.

Of course, TruStone is not in the Sandy impact area but, shrugged Thompson, if it were hit with an event of similar magnitude, he believed the institution would operate essentially without break.

Compare that to a long list of New York and New Jersey institutions that struggled mightily after Sandy.

The TruStone solution is not cheap. Thompson put a ballpark figure on the cost of $1 million, but he said TruStone members could rest easy knowing that no matter what is thrown at the institution, from tornados to floods and ice storms, “we will be able to operate.”

After Sandy or any catastrophe, from Katrina in New Orleans through earthquakes in California, experts probe what went right with existing recovery scenarios and what did not. The goal, they stressed, is to keep perfecting responses.

One loud take-away from Sandy: Every credit union needs a disaster recovery plan.

“There have been, what, three or four major storm events on the eastern seaboard in the past four years. Many credit unions have invested in business continuity and disaster recovery, and the ones that have kept operating,” said Collins.

The frightening news for credit unions that stumbled and stayed down due to Sandy: their days may be numbered.

“Prolonged events like this may put some credit unions out of business,” said Collins. “You may not know why members are leaving. But some will.”

“You have to invest, test and document your disaster recovery plans, and you have to come to the realization that regardless of size, this is something you have to invest in. You will lose members if their accounts aren't available.”

“I would agree, some institutions will lose customers. When disaster hits, you need the infrastructure to deal with it to provide services. Smaller financial institutions will take the biggest hit,” said John Reeder, a consultant with Foundstone, a company that helps customers manage vulnerabilities.

A fact is that while rigorous disaster recovery and business continuity planning get expensive, much can be done with little or no money. Collins, for instance, urges Xtend customers to form informal mutual assistance networks, where if one institution goes down, it may be able to open a teller station at another institution. He also knows cases where a credit union let another temporarily house its executive team in their building. “A credit union needs to know who its partners are, who will help in the event of an emergency,” said Collins.

The experts are unanimous in believing that the Sandy aftermath will trigger greater vigor on the part of NCUA examiners in looking into an institution's disaster planning. The expectation also is that examiners will look for more tangible proof that boards of directors have informed themselves about their institution's disaster readiness.

Gerber said that FFIEC currently provides benchmarks for recovery.

“FFIEC is clear that an institution has to have a plan for resuming operations 'in a reasonable amount of time'–that usually is defined as 24 hour for core systems and 72 hours for all systems.”

There also is room to debate what constitutes adequate protection. A particular thorn of contention is how far away a backup system should be. The current minimum number thrown out by many experts is 200 miles, although TruStone's Thompson said his institution is more comfortable with a 300 miles distance. The underlying idea is that backups have to be far enough away not to be engulfed by the same event that knocked out an institution's main systems and, in the case of a super storm such as Hurricane Sandy, that means backups have to be very far away indeed.

Examiners, added Gerber, already have been saying they want to see more data from full tests of disaster systems–usually generated in simulated system failures where the institution goes through a mock hurricane, then details how long recovery takes in each sector of the operation.

“Anybody can back up your data. The question is, how long does it take to retrieve when you need it?”The answer, for all except systems that undergo frequent and rigorous tests, is that restoration typically takes much longer than anticipated.

“You just need to invest,” said Collins. “You need to make business resiliency a priority. You need to test it. And you need to test it again. When you need it, you want to know your system will be there.”

Instant Insights /

Hurricane Sandy Lessons in Disaster Recovery: Print Preview

Related Stories

Recommended For You