The post Hurricane Sandy lessons learned keep coming in as itbecomes plain that some credit unions had woefully inadequatedisaster recovery and business continuity plans, but otherinstitutions kept operating despite the devastation that hit muchof New Jersey and New York City.

“For the credit unions that already were prepared, with solidplans in place, everything was relatively calm,” said ScottCollins, president of Grand Rapids, Mich.-based CUSO XtendInc., which provided disaster recovery services to five New YorkCity-based credit unions and one in New Jersey. “None of our creditunions lost a minute of uptime on electronic channels,” saidCollins, who acknowledged they all had limited or no branchservices, primarily due to lack of power.

For one credit union, Progressive Credit Union, the $540 millionManhattan institution known for its portfolio of taxi medallionloans, “We provided phone answering services,” said Collins whoindicated Progressive has a contract in place with Xtend where theCUSO, as needed, steps in and answers inbound phone calls with theaim of providing members with questions full and accurateanswers.

But “not every credit union follows best practices for disasterrecovery,” said Matt Gerber, CEO of IT-Lifeline, a Liberty Lake,Wash., provider of disaster recovery services. “Some don't evencomply with the FFIEC guidelines,”he added.

Exactly what counts as state of the art for disaster recovery?Plymouth, Minn.-based TruStone Financial, a $840 million creditunion, said it has implemented what might count as gold- standardprotection.

“All our data are replicated in real time to computers 300 milesaway in Milwaukee,” said Bob Thompson, a senior vice president forinformation technology at TruStone. He said that no matter whathappened to the Minnesota-based computers, “we could be back up andrunning in two hours.”

Key is that TruStone has created what Thompson describes as “ahot standby site that could take over in a minute.” Data flows fromTruStone Minnesota computers to the Milwaukee sitecontinuously.

Of course, TruStone is not in the Sandy impact area but,shrugged Thompson, if it were hit with an event of similarmagnitude, he believed the institution would operate essentiallywithout break.

Compare that to a long list of New York and New Jersey institutions that struggledmightily after Sandy.

The TruStone solution is not cheap. Thompson put a ballparkfigure on the cost of $1 million, but he said TruStone memberscould rest easy knowing that no matter what is thrown at theinstitution, from tornados to floods and ice storms, “we will beable to operate.”

After Sandy or any catastrophe, from Katrina in New Orleansthrough earthquakes in California, experts probe what went rightwith existing recovery scenarios and what did not. The goal, theystressed, is to keep perfecting responses.

One loud take-away from Sandy: Every credit union needs adisaster recovery plan.

“There have been, what, three or four major storm events on theeastern seaboard in the past four years. Many credit unions haveinvested in business continuity and disaster recovery, and the onesthat have kept operating,” said Collins.

The frightening news for credit unions that stumbled and stayeddown due to Sandy: their days may be numbered.

“Prolonged events like this may put some credit unions out ofbusiness,” said Collins. “You may not know why members are leaving.But some will.”

“You have to invest, test and document your disaster recoveryplans, and you have to come to the realization that regardless ofsize, this is something you have to invest in. You will losemembers if their accounts aren't available.”

“I would agree, some institutions will lose customers. Whendisaster hits, you need the infrastructure to deal with it toprovide services. Smaller financial institutions will take thebiggest hit,” said John Reeder, a consultant with Foundstone, acompany that helps customers manage vulnerabilities.

A fact is that while rigorous disaster recovery and businesscontinuity planning get expensive, much can be done with little orno money. Collins, for instance, urges Xtend customers to forminformal mutual assistance networks, where if one institution goesdown, it may be able to open a teller station at anotherinstitution. He also knows cases where a credit union let anothertemporarily house its executive team in their building. “A creditunion needs to know who its partners are, who will help inthe event of an emergency,” said Collins.

The experts are unanimous in believing that the Sandy aftermathwill trigger greater vigor on the part of NCUA examiners in lookinginto an institution's disaster planning. The expectation also isthat examiners will look for more tangible proof that boards ofdirectors have informed themselves about their institution'sdisaster readiness.

Gerber said that FFIEC currently provides benchmarks forrecovery.

“FFIEC is clear that an institution has to have a plan forresuming operations 'in a reasonable amount of time'–that usuallyis defined as 24 hour for core systems and 72 hours for allsystems.”

There also is room to debate what constitutes adequateprotection. A particular thorn of contention is how far away abackup system should be. The current minimum number thrown out bymany experts is 200 miles, although TruStone's Thompson said hisinstitution is more comfortable with a 300 miles distance. Theunderlying idea is that backups have to be far enough away not tobe engulfed by the same event that knocked out an institution'smain systems and, in the case of a super storm such as HurricaneSandy, that means backups have to be very far away indeed.

Examiners, added Gerber, already have been saying they want tosee more data from full tests of disaster systems–usually generatedin simulated system failures where the institution goes through amock hurricane, then details how long recovery takes in each sectorof the operation.

“Anybody can back up your data. The question is, how long doesit take to retrieve when you need it?”The answer, for all exceptsystems that undergo frequent and rigorous tests, is thatrestoration typically takes much longer than anticipated.

“You just need to invest,” said Collins. “You need to makebusiness resiliency a priority. You need to test it. And you needto test it again. When you need it, you want to know your systemwill be there.”