We tested the intrusion prevention capabilities of each of the next-generation firewalls to determine how well they work and how the IPS integrates with system management.
We were especially concerned with the IPS workflow for false positives, taking the network manager from a logged IPS event to the particular IPS signature triggering the event, to the ability to disable or modify the IPS signature to reduce problems.
We started by using our Mu Dynamics Studio Security test tools to check how well each firewall's IPS would catch Mu's list of published vulnerabilities. We tested the firewalls in two different configurations, one optimized to protect end users, and a second one optimized to protect servers. For each configuration, we sent a different set of about 1,000 vulnerabilities.
For each vulnerability set (server attacking and client attacking), we created two policies for each firewall. One policy included all of the IPS signatures and the other just had the subset of signatures marked as highest priority. We were thinking that the "all" signature set would have more false positives, and most network managers would want to only block the most critical vulnerabilities.
In most products, we saw less than two percentage points of difference between the two sets, meaning that there's very little tuning of the IPS possible. Fortinet's FortiGate was the exception, showing a 10% to 25% difference in attacks blocked, offering the network manager more tools to match the IPS to their network.
When protecting clients, we found that the Check Point Security Gateway, Fortinet FortiGate, and Barracuda NG firewall all outperformed SonicWall SonicOS. However, when we tested server-protecting IPS configurations, SonicWall and Fortinet performed significantly better than Check Point and Barracuda.
We believe that most enterprises deploying next generation firewall functionality will be doing it to protect end users rather than servers, so the client-protecting IPS coverage is more important than server-protecting coverage.
While we think that testing with the Mu Dynamics tester helps to keep IPS vendors on their toes with vulnerability signatures, it's important not to read too much into efficacy tests like these. Since the Mu Dynamics tester is a standard product, there's always the possibility that IPS vendors will tune their systems to increase their scores — even if they don't agree that a particular attack or vulnerability is important or correctly crafted. The Mu Dynamics tester is also useful because it can do mutation testing, which can stress the software in next generation firewalls, although only the Barracuda NG Firewall had a crash during our test runs.
Because IPSes all can trigger false positives, management is an important concern. We found Check Point Security Gateway and SonicWall SonicOS the easiest to work with, although this can be a matter of personal preference. Both devices only allow for a single IPS policy per device, which means that you're managing a single large policy on the firewall. That's limiting, but it is an appropriate limitation when you're managing a firewall and not a dedicated IPS device.
Sign up for Computerworld eNewsletters.