Questions arise on PC World testsA sweeping review of 10 security suites published in a major computer magazine last month featured some very unlikely rankings for this crucial category of products. After examining the evidence, I’ve found that some material facts were omitted from the article, rendering its ratings useless.
The cover of the July 2006 PC World Magazine promised a review of security suites that would give readers “total protection against spyware, hackers & spam.” Inside the magazine, a lengthy article summarized extensive test results by AV-Test.org, a respected antivirus research group based in Magdeburg, Germany. The magazine’s product rankings, however, seemed inexplicable.
When good software ratings go bad
I reported on July 27 that CNET had given its Editors’ Choice award in a June 4 review of security suites to Zone Alarm Security Suite (ZASS). PC Magazine’s Editors’ Choice went to the same product in a June 13 article. But PC World’s ratings, which were first posted online in May, dropped ZASS to 6th place out of 10 products reviewed. The magazine’s top honors went to Symantec Norton Internet Security 2006. I promised in my previous article to find out why and report the answer to you today.
I have no love of any hardware or software vendor. If a product drops from being top-rated to merely mediocre, I’ll say so in my Security Baseline section, below, or my Reviews Overviews, which I update online.
PC World’s ratings, however, are so puzzling that I immediately suspected something was wrong. After looking at some of the raw data, I believe AV-Test did provide PC World with accurate figures on the security suites that the German lab tested. Essential tests, however, were left out. The errors fall into three broad categories:
1. The review ignored behavior-based protection. Behavior-based protection, which stops suspicious activity, was left out of the tests. Signature-based virus scanning is declining in effectiveness, but at this point only a few of today’s security suites include behavior-based protection. This crucial feature, which could represent a huge difference in malware detection, was simply left out of PC World’s scoring.
2. The review omitted complete leak-test results. Leak tests rate a security suite’s abilities to prevent malware that somehow sneaks into your PC from successfully sending your personal data to a remote server. AV-Test’s findings revealed widely divergent scores for the tested suites. But the results for most vendors were left out of PC World’s ratings.
3. The review turned off some suite features. Integrated security suites should be tested with all features turned on. PC World, however, chose to disable some capabilities in order to run tests aimed at other capabilities.
Consumer Reports backs up CNET and PC Mag
The well-regarded U.S. product-testing magazine, Consumer Reports, hit the newsstands last week with its own ratings of PC security programs. The lab’s testing separately rated the antivirus, antispyware, and antispam programs available from each vendor. In addition, the magazine contracted with security experts to generate 5,500 original virus variants to test behavior-based protection. CR also monitored how quickly the companies released updated signatures in real time over a period of weeks as new threats emerged on the Net.
Zone Alarm Security Suite received Consumer Reports’ Quick Picks award — the magazine’s version of Editors’ Choice — for “the best all-around protection.” Perhaps because it’s well known that security suites haven’t yet mastered the latest spyware, CR also gave Quick Picks awards to Webroot Spy Sweeper and PC Tools Spyware Doctor in the antispyware category (with the free Spybot as a complement).
These ratings make sense. They dovetail with CNET and PC Magazine’s latest findings, both in the rankings and the award winners. Besides PC Magazine’s Editors’ Choice for the Zone Alarm Security Suite, for example, Editors’ Choice awards also went to Webroot and PC Tools in the magazine’s latest, July 2006 reviews of antispyware apps.
To be sure, it’s not unusual for magazines to differ in their ratings of computer products. For one thing, PC World’s tests were conducted in April using ZASS version 6.0 and the then-current versions of competing products. The other publications’ latest awards are based on the newer ZASS 6.5.
But when a category is as important as security suites, and when one magazine’s rankings deviate so much with no logical basis, I look for a reason.
I found the answer in personal interviews with principals at AV-Test, Symantec, McAfee, and Zone Labs. To solicit comments, I provided AV-Test and PC World with draft copies of this story. I then participated in a telephone conference call with PC World editor-in-chief Harry McCracken, test center director Ulrike Diehlmann, and senior associate editor Narasu Rebbapragada.
The review ignored behavior-based protection
Near the middle of PC World’s July 2006 article, I found a few sentences that related to nothing else in PC World’s review:
- “AV-Test.org found that Panda TruPrevent will block up to 90 percent of network and e-mail worms and that Zone Labs’ OSFirewall will stop up to 70 percent of network and e-mail worms.”
Behavior blocking isn’t a panacea. But when combined with traditional signature scanning it’s a major enhancement. It should hardly be ignored. (Behavior-based protection should not be confused with heuristics, a technique that looks for suspicious patterns in executable code. See TechTarget’s Apr. 12 article on antivirus trends.)
I arranged an interview with Andreas Marx (photo, right), co-manager of AV-Test.org. I was one of the first American journalists to write about this university-based antivirus research group in my Executive Tech column back on Feb. 23, 2004. At that time, the lab’s ratings of antivirus programs were being used by German publications, but its work wasn’t yet widely reported by U.S. magazines.Explaining the value of behavior blocking to stop new malware variations, Marx told me by telephone:
- “We’re seeing at least 200 to 300 new variants a day. The malware writers are using optimizations … They’re not only doing the modifications, they’re also creating several variants. This can’t be detected any more by virus scanners without antivirus [signature] updates. …
“Only ZoneAlarm and Panda have behavior-based solutions that block malware by its bad behavior. That kind of advanced protection is not just relying on traditional signature-based solutions but also mechanisms to protect the user against unknown malware as well.
“Most of the other companies — like Symantec, McAfee, F-Secure, Trend Micro — they will include such behavior-based solutions in their software as well in two to three months, as soon as the new 2007 editions come out.”
- “We agree that behavior-based protection is becoming increasingly instrumental in fighting zero-day threats, for which no signature-based patch is yet available. Eventually PC World security reviews will thoroughly test a product’s behavior-based protection. During the testing period for this particular story, however, we were not able to test behavior-based protection in a manner that was fair, defensible, and repeatable during our testing window for the story, which was well before the July 2006 publication date. Rather than conduct unsatisfactory tests, we chose to focus on features included in all programs.
“We make clear in the story that behavior-based protection was not included in our testing and that it could have an impact on overall results. Later in the process we were able to get some top-level statistics on Panda’s TruPrevent and Zone Labs OS Firewall, two behavior-based technologies. We included that information in the story.”
The review omitted complete leak-test results
Another omission involves leak tests. Let’s say that a Trojan horse somehow manages to install itself on your PC. A leak test determines how many little critters are able to defeat a security suite and slip your data out to a hacker’s server.
PC World’s Rebbapragada, the author of the piece, mentioned the leak-test scores in just a single paragraph near the end of her article:
- “Zone Labs’ firewall was again 100 percent successful, passing all 17 leak tests, with Microsoft’s in second place, passing 7 tests. The other products earned very low scores, and Panda’s passed none of the leak tests. … [Panda] says that it doesn’t optimize its software for leak tests, instead relying on its TruProtect behavior-based technology to decide whether a piece of code is malicious.”
Table 1, below, shows the percentage of leak tests that each security suite passed, according to raw data sent to me by AV-Test. Most of the products passed only one or two of the 17 tests. Aside from the single paragraph cited above, none of this was mentioned in PC World.
Table 1. Percentage of 17 leak tests passed by security
software. Higher numbers are better. Source: AV-Test.org
PC World’s editors say:
- “Our evaluation of security suite firewalls included seven tests for blocking malware already on the system (inside attacks) and four tests for blocking malware outside the system (outside attacks). Leak tests represented one of the seven inside attack tests. While we felt it was worthwhile to include leak test results as a portion of our overall rating, we chose not to weight it heavily or to report on these tests in detail. Leak tests are standardized, publicly available tests for which companies can optimize their firewalls. We believe that AV-Test.org’s other inside attack tests were most representative of a product’s ability to fight real-world malware.”
Unfortunately, there’s no way a reader could know, based on the information in PC World’s article.
The review turned off some suite features
The third concern about PC World’s ranking of security suites is the magazine’s practice of turning off some features during testing. This is intended to allow the magazine to use existing tests that are specific to adware, spyware, virus detection, and the like.
But does testing one security component while other components are turned off actually reflect the real-world performance of an integrated suite?
Vendors are increasingly combining all of their individual security products into a single, integrated package. Representatives of Symantec, the company that won PC World’s Best Buy award, explained to me how two separate software components can strengthen each other when brought together into a single product.
“The firewall might detect some activity independently,” said Kraig Lane, Symantec’s group product manager of consumer Internet security products. “Then it can say that the antivirus [component] should quarantine some file.” In other words, each component can use the strengths of the others.
Providers of security suites say they want real-world testing. McAfee’s suite did extremely well in PC World’s ranking, receiving almost the same overall scoring as Symantec. (The two suites were rated 83 and 84 points, respectively, out of a possible 100.) Even having received such a high rating from PC World, McAfee’s director of product management, Marc Solomon, expressed concerns about testing new products with older routines.
“I’d really like to know how they tested this, to see if they turned off the antivirus in order to test the firewall,” Solomon said in a telephone interview.
PC World’s editors tell me:
- “PC World’s philosophy in testing security suites is to test the strength of individual components of that suite and then combine the results in an overall PCW Rating. To run some of these component tests, it is sometimes necessary to disable a product’s malware detection capabilities in order to get the malware samples onto the test PC. In some cases, this involves altering default settings.
“However, this approach tests several scenarios that exist in real world, including situations in which the malware is already on a PC before the software is installed and ones in which a user has, for whatever reason, turned off detection features. In either of these scenarios, a user may need to use one component of a security suite to get rid a PC of malware even if another component of the suite might have been able to detect it.”
Moving toward 100% protection, all the time
Today’s worms and rootkits can be difficult or impossible for Windows users to remove. Once the devious little critters have snuck into a system, they can be devilishly hard to detect and eradicate.
For this reason, it’s important for security suites to be installed before a PC is set up and exposed to the Internet. Gateway computers, for example, now ship with a 90-day free version of McAfee Internet Security Suite automatically enabled. In my opinion, most such vendors’ annual subscription fees to continue the protection are reasonable.
The question is, How much protection does the best security suite provide? Users want to know how often a real-world threat can slip through the automatically updated armor of these suites. (Every six days? Every six months? Almost never?)
For his part, AV-Test’s Marx says he’s satisfied with PC World’s article. In an e-mail after reading a draft of this story, he noted:
- “Security suites are integrated products with many features, like virus scanning and personal firewall protection. For example, in case of ZoneAlarm, the firewall was top-class, but the virus detection was rather poor, so they lost some points here. Even then, the ranking was still ‘Good,’ as it’s a good product, so the rating is perfectly fine. (Maybe it should even be mentioned that ZoneAlarm confirmed the problems we have seen in their antivirus product. We have also supplied the missed viruses, worms and bots to them, so they can add detection for them with the new few updates — and they did! So we were even helping improving their product, as well as all others.)
“1. Again, the behaviour-based features of all 10 products were reviewed, but only two of them actually included something we could test. Almost all 2007 products will include behaviour-based warnings, so we can review it in more detail than now.
“2. Leak tests: The prevention of firewall leaks is just one of many different tests we have performed. We attacked the firewall against a set of inside and outside attacks, against real malware. Leak tests are (as the name is saying) special test programs which are not reflecting the real-world protection in a proper way. As I said, the protection against real-world threats is much more important and this was included in the ranking with much higher weights, as the user wants to be protected against keyloggers, backdoors and bots/zombies (real-world malware!) and not necessarily against leak tests. Leak tests are harmless, but malware is not harmless.
“3. We tested both the on-demand scanner protection and the protection by the real-time/on-access virus guard. In order to check the guard, we need to access malware in some way (e.g. by copying files or double-clicking on it) and see if it’s blocked or not blocked. This test was included in this review — and some products performed not so well here, but this is included in detail at the Web page. In addition to this, we have also tested the on-demand scanner as a separate feature. In order to test the on-demand scanner, you need to switch off the real-time protection mechanisms, as you want to test the scanner, not the guard. So the test was simply split into two parts which have to be tested separately and independently from each other.”
PC World has a reputation for excellence in its technical material. Disclosure: I myself was once a contributing editor, writing a monthly column for the magazine for a couple of years in the 1990s. The publication’s quality has steadily improved since then, in my opinion. But mistakes can hurt a publication, even if most of its work is solid.
I request that PC World retract its ratings of security suites. This topic is important enough to warrant spending the money to write up a new set of real-world tests.
In our conference call, PC World editor-in-chief McCracken told me, “We won’t retract that. We feel we made the right decisions.” He also said, however, “I think you will see us do behavior-based testing in the next few months.”
The online version of the security-suite review is posted at PC World’s site. For details on Marx’s antivirus testing group, visit AV-Test.
Readers, I leave it up to you at this point. I welcome your expertise on how security suites should be tested — and whose tests you find to be the most dependable.
Many subscribers have asked me whether installing separate programs to handle firewall, virus, spam, and spyware duties wouldn’t be superior to installing an integrated security suite. That’s certainly true for large enterprises. Corporations with IT staff capable of evaluating these programs will always put together their own layers of protection.
Many home users and small businesses, however, don’t have this luxury. They need to run one or two products that they can understand. Security vendors — and the test labs that review their products — will inevitably concentrate more and more on integrated suites to meet this demand.
My hope is that all the competing suites will improve enough that their detection of malware becomes virtually foolproof. Then these products can compete over which one is easiest to use, has excellent customer support, and is affordably priced. We won’t know when that day has come, however, unless the major test labs convince us that their methods reflect real-world protection.
To send us more information about security suites, or to send us a tip on any other subject, visit the WindowsSecrets.com contact page. You’ll receive a gift certificate for a book, CD, or DVD of your choice if you send us a comment that we print.
