BAYTOWN, Texas — A good New Year's resolution for credit unions in hurricane country, which as Ike showed in September can extend a thousand miles inland, is to put some serious distance between your home base and your backup center.
That was one lesson learned by Community Resource Credit Union in Baytown, one of the Gulf Coast communities squarely in Ike's path as it roared ashore on Sept. 12.
While the $242 million institution was prepared and managed to re-open a full week before most of the other financial institutions in the area, it could have been worse, said Mike Smith, vice president of marketing and business development.
With home base in an area prone to storms and flooding, the 28,000-member credit union had created at a backup center at its Crosby branch, about 13 miles north of its main Decker Branch.
Sitting 23 feet or so higher above sea level than the Decker branch, and with a backup generator and the technology infrastructure and other equipment needed to swing into action as a backup site, the Crosby branch seemed ready to go.
The problem was after the storm, when things like getting gas for the generator and water and food for the staff, and even simply being able to get to the building, became problematic, Smith said.
“I waited in line for two hours at one point to get gas for one of our branch managers,” he said.
The crucial technology infrastructure held, said Tamara Hudson, the credit union's vice president of information technology, but not without some scrambling.
A Symitar core processing customer, Community Resource used a Citrix virtual private network (VPN) connection from the Crosby branch to the data center operated by Centurion, another Jack Henry & Associates company, in Angola, Ind.
Call center operations, meanwhile, were moved in the hours before the storm to another credit union, EECU hundreds of miles away in Fort Worth, where they fielded an average of 1,200 calls a day in the first few days after the Friday night storm.
By Monday morning, the Crosby branch was up and running “with no interruption” from the storm that hit with enough force to leave some people in the area without electricity for up to three weeks.
The decision to create an operational hub at the Crosby branch was made after Hurricane Rita in 2005. Now, building on the experience of Ike, one thing the credit union will do differently next time is immediately flip the switch to the core processing connection to Angola instead of taking that crucial function to the Crosby branch first, avoiding the scramble when landfall looms, Hudson said.
“We're still going to use Crosby for backup to Decker, though, so we have two different locations for connectivity if we're operating from Angola. As long as we have one of those facilities up and running and remote access to the people in Fort Worth, we'll have full access to serve our members no matter where they are,” Hudson said.
And while the intricate network of core processing in Indiana, a call center working through a shared branching network in Fort Worth and the people back home in Baytown worked, there were some new lessons learned.
“What we learned out of this situation was that when a hurricane is coming in, you probably don't want to be in a position to lose your backup location, too,” Smith said.
Indiana-based Bradford-Scott Data Corp., another core processor, was learning that same lesson from Ike as the storm moved hundreds of miles inland, still a surprisingly potent tropical system. That outcome was a bit different, since the predetermined backup site more successfully weathered the storm.
With the quick help of technicians who drove down from Indianapolis, Louisville Metro Police Officers CU quickly moved its operation about 12 miles away to the Kentucky Credit Union League office.
“Having never been through a situation like this, I didn't know what to expect,” said Gale Stivers, CEO of the $9.5 million CU. “I can't say enough about Bradford-Scott. They showed up just a few hours after I reported the disaster” at 10:30 a.m., and had the credit union up and running before 5 p.m. that day.
As it turns out, the league office never lost power and was not hit by the severe, widespread flooding. But it could have been different.
“The importance of having a remote backup was obvious after seeing the flooding,” said Matt Marra, hardware service manager at Bradford-Scott. “It reinforced the point that data backups need to be sent to a secure remote facility.
“Our response time is irrelevant if the customer site and the backup site are both flooded. We need to encourage our customers to utilize an online backup solution that puts distance between their live data and their backup,” Marra said.
Dan Jorna and Chris McWilliams couldn't agree more. They're responsible for business continuity for credit union clients of USERS, a Fiserv core processing unit with data centers at its Valley Forge, Pa., headquarters, in the Detroit suburb of Troy, Mich., and in Honolulu.
USERS began offering a product it calls Real-Time Vaulting in the past couple years. The system eliminates the need to ship backup tapes, using shadow-server technology and a VPN to backup data as it's recorded and store it at one of the remote sites.
“One of the biggest lessons we learned this hurricane season from a client standpoint was the difference between those using Real-Time Vaulting for their database and those that did not,” said McWilliams, product manager for business recovery services at USERS.
Two of the three client credit unions were using Real-Time Vaulting and were able to restore services within an hour or two. The third client was not a vaulting client, and it took a good 12 to 15 hours to put a tape on a plane and get it to us, since they were having a hard time even getting to a plane in that area,” he said.
One of those two Real-Time Vaulting clients was $292 million FivePoint Credit Union in Nederland, Texas.
There, the decision was made to take the system down in the hours before the storm hit and switch over to the data center, at the same time as most of the credit union's members were presumably following evacuation orders and leaving the area.
That was at 11 p.m. on Sept. 12. During the night, the core system, home banking, audio response and shared branching systems were switched over to the USERS data centers and were again running by 6 a.m. the next day.
“It's like we never missed a beat,” said Marla Rucker, FivePoint CU's assistant vice president of information technology.
Currently about 23 of 138 credit unions using the data centers as backup sites are using Real-Time Vaulting, with another 25 or so committed to coming online.
“Every time we go through one of these seasons, we see a little more interest in it,” said Jorna, USERS' vice president of data center services.
Jorna said he's seen other improvements in disaster recovery since the fierce hurricane years of 2004 and 2005.
“Dealing with third-party vendors has gotten better, and it seems like ATM and shared-branch vendors have really come up to speed in being able to contact them and connectivity,” he said. “They've become very flexible. Before you might have to go to batch with them, but that doesn't seem to be the case anymore.”
McWilliams, meanwhile, added another lesson learned, especially from hurricanes Katrina and Rita in 2005.
“It's really better if you don't host your Web site for members on-site,” he said. “That's a real big one. If you're building gets destroyed, so does your information hub, and we can't really be that much help.”
–[email protected]