You wouldn’t think a single camera pointing down an empty hallway would be much of a problem. But when it’s constantly feeding images through the same data pipe as everything else in a big credit union’s network, it certainly can.
That’s what the $3.8 billion ESL Federal Credit Union found out after it used a new piece of software–the Scrutinizer from Plixer International Inc. of Sanford, Maine–to determine what was clogging up the works at ESL’s new headquarters in Rochester, N.Y.
The five-story structure was built with all-new infrastructure designed for the facility to serve as the nerve center for a system of 20 or so branches and recovery centers–complete with high-end Cisco routers and switchers all connected by T-1 broadband lines.
But soon after things were up and running, the physical security department started complaining of slow network connectivity. So what was slowing it down? Acting on a recommendation from ESL’s moving consultant, ESL network engineer Chris Lomonaco and his colleagues began using Cisco’s NetFlow software and Plixer’s Scrutiziner to monitor network flow and troubleshoot the network noise and fluctuating response times.
They immediately began detecting a tremendous increase in bandwidth and Internet traffic from the physical security department’s network of 130 digital cameras and then isolated that to a single camera streaming nonstop images of an empty hallway through the network pipes and into storage.
Turns out other cameras had similar issues–capturing images of inactive hallways and stairwells–but now they’ve all been configured to record what they’re capturing only when there’s motion, Lomonaco said.
He said that simple change reduced video traffic by more than 25%, improved Internet connectivity speed and decreased storage demands.
"We bought them 24 terabytes to store all their pictures, enough to give them up to 30 days of pictures, and they were running out of space after 10, and because they were sharing bandwidth with everything else, we had to quit pulling pictures during production time and wait until after hours," Lomonaco said. "That is, until we figured out what was going on."
The analysis software also helps the IT staff figure out other bandwidth problems–such as a staffer printing a 55-meg document through the Citrix thin-client server and onto the T1 line.
"That really choked the pipe," Lomonaco said. "We were able to pinpoint it, though, and now when people need to do that kind of thing, they know not to and that we can help them figure out a different way of getting that information to where it needs to go."
The Scrutiziner system also includes historic analysis and reporting functions that Lomonaco has found useful both for spotting trends and specific problems, he said.
"Now I can go back and look at what was affecting traffic at a given time," he said. "Before, when someone would say to me, ‘The network was really slow for a couple hours yesterday afternoon, why was that?’ I couldn’t tell them. Now I can go back and find out."
"It’s nice to be able to give them a good explanation, not just say, ‘I don’t know.’"
Michael Patterson, Scrutinizer product manager at Plixer, said his company’s solution is now in use by more than a thousand customers in a variety of industries and that he often hears of it being used in proactive ways as well as to react to known problems.
That includes one scenario where an Internet IP address was detected to have made it through security measures and was streaming video from a camera to a Web-based application. "We could do something about it because we were able to see it," Patterson said.
Lomonaco at ESL said the software is now a regular part of the IT staff’s security arsenal. "Monitoring is a huge part of our network and this is now part of the repertoire we use to do that. If you know what’s going on in the network, on your servers, in your hardware and your software, you can be a lot more proactive when it comes to handling any issues that come up," he said.