Red River CU Uses New Technology for Disaster Recovery, Business Continuity

Traditional backup tape storage also continues, Haley said, with tapes taken each day to yet a third facility a few miles from the main office. And, the credit union has purchased a used RS6000 production box for training and testing at the backup center. It's the same as the machine that runs the XP2 core platform at the main center and also serves as a backup in case of emergency.

"We used to pay an outside group to basically just keep equipment on standby," the Red River CIO said. "That's one reason we still send the tapes to a third facility."

"We feel like with the geographic spread of the three facilities, if we had a disaster that took out all three of them…Well, I don't believe anyone here would be concerned about it at that point," he said. "It would take a disaster of much larger scale than even what hit New Orleans with Katrina to do that."

Connecting the two data centers–about 15 miles apart as the crow flies–is a high-speed wireless data link, an encrypted connection that zips data along at 160 megabits per second and costs less than a comparable traditional phone data circuit, Haley said.

"I wasn't surprised at the cost savings, but I was surprised that we could get that kind of speed at that kind of distance," he said, adding that other area financial institutions have asked to take a look. He also said that XP Systems has told him his credit union is the only one of its client base configuring its backup infrastructure this way.

Expanding beyond the core processor's own backup offering, in fact, was what prompted the credit union to move in this direction, Haley said, citing the need to expand beyond backing up just the core system.

He said the SAN work also is part of an increased emphasis on disaster recovery and business continuity planning under way for the past year or so, both in response to NCUA directives and to internal policies set by the 59,000-member credit union's board.

"We're extremely satisfied with how far we've come," Haley said. "We consider this a much better plan than we would have had, since it covers more than just our core system, and it actually is cheaper over a three-year amortization than the cost of the disaster recovery option that XP Systems offers."

EMC specialists installed the SAN, but "it still has taken a few hundred hours of effort on our part to complete all this," Haley said.

The effort also remains a work in progress.

Next up is using VMWare software to "virtualize" the majority of the servers in the data center, allowing "us to be able to recover more of our services in a much shorter period of time in the event of a disaster," Haley said, since entire databases can be mirrored and moved without having to, for instance, install a new operating system in the next server.

Cisco CallManager systems also are being installed in both data centers, allowing one to roll over to the other in case of a failure and ensuring phone service at the branches. Similar measures, using redundant LynxGate boxes, will be used to ensure ATM service at the credit union's 24 machines.

"We're also equipping many of our key employees with laptops and setting them on with the ability to work at home through a VPN [virtual private network] in the event of a facility loss," Haley said.

"We're also working on the possibility of sending call center reps to work at home, so we can distribute them and not lose that service, either," he said.

Extending the wireless data transfer system as a point-to-point to all the branches also is under way, Haley said.

Overall, the Red River CIO said, "I'm a whole lot more confident than I was six months ago."

"The reality is that before we did this, if a disaster struck, it could be days or a week or more before we were really up and running and serving members. Now, we have the potential to restore all our core services within an hour."

–mrapport@cutimes.com