As far as I can tell, one of the few network security tools getting better at the speed of Moore’s Law is network monitoring and audit. Modern networks throw off vast amounts of data as users go about their daily business. It is often possible to find the telltale signs of network intrusion by watching for activity that is anomalous or that fits the signature of network attacks elsewhere.
But finding those faint signals in a sea of noise isn’t easy. No one wants to sit and read logs.
The good news is that tools to analyze Big Data are improving and getting cheap at a great rate, and companies like Zions Bank have begun using open source programs like Hadoop to analyze their networks. 
Around 2005, [Zions Bank Chief Security Officer] Wood said, his team made the move to a massively parallel processing system that was designed for log management but that his team bent and hammered into a data warehouse for analytics. “We adopted a business intelligence mindset,” he said, “but slanted toward security.” He brought in some data-analysis specialists, and they started mining data and searching for patterns, a process made easier by the new higher-powered and more scalable system. But it, too, reached its limits as unstructured data from myriad new sources began streaming in.
In 2010, Zions started its transition to Hadoop and has been running its big data workloads exclusively on that platform since late 2011. Wood said he’s loading about 130 data sources into Hadoop, including server logs, web logs and customer transactions. Now, he explained, his team is able to analyze massive amounts of data — and fast — to detect everything from malware and spear phishing attempts to account takeovers. The latter is similar to credit card theft, only instead of discovering anomalous spending, Zions is able to detect anomalous transfers from customers’ bank accounts.
And Wood doesn’t worry about outgrowing his Hadoop cluster, which means his team can keep innovating on new ways to detect criminal behavior. If you’re monitoring network traffic, for example, Wood said, you “have to get down to 0s and 1s in packets to look for the needle in the haystack.” That means storing and analyzing everything in its raw form.
This approach to security is gaining traction, but once again, it looks as though the financial sector, rather than government, is pioneering a network security technology. In fact, this is going to be a tough act for government to follow. Just read that last sentence again. “That means storing and analyzing everything in its raw form.”
Just as they stalled government network intrusion prevention technology for a decade, privacy advocates are likely to trash any government security system that depends on storing and analyzing everything, even everything on the government’s own networks. Which means that security will likely remain Mission Impossible for most government information security officers.