Headline Feeds
When consumers buy a top-rated software product, are they really getting what they pay for? That's what Sunbelt Software's Chief Scientist Joe Wells is addressing in the wake of anti-spyware and antivirus testing methodologies that appeared in Consumer Reports, the magazine published by the non-profit Consumers Union.
The independent testing organization's results have come under fire for questionable testing criteria. Some are even calling the company's review techniques scandalous. Indeed, professional security researchers have called the magazine to the carpet for testing antivirus programs using 5,500 fake viruses.
The event raises questions about the art and science of testing these security programs and Wells is evangelizing the importance of correct methodologies to ensure consumers are really getting what the box claims it delivers.
TechNewsWorld caught up with Wells to discuss what consumers should look for in security software and how they can protect their data safety interests in a world where testing is not always a clear indication of effective protection.
TechNewsWorld: When consumers buy a top-rated software product, are they really getting what they paid for?
Joe Wells: It depends on the reliability of the rating method. On one hand, the more the method depends on precise testing the better; on the other hand, the more the method depends on the tester's personal opinion, the worse it is. So tests that emphasize look and feel tend to be less dependable.
TNW: What is your philosophy on quality assurance and testing for anti-malware software?
Wells: We test our software in the same basic way all software is tested. But in addition, we must test against real, active threats, including detection, remediation, correct information, as well as false positive testing.
TNW: In the wake of this Consumer Reports incident, what can we learn about the art and science of testing anti-malware software?
Wells: The CU testing is a simple example of a testing body not researching to find out what the current state of the art in security testing actually is. There are papers available on well-established scientific procedures for testing antivirus and anti-other malware products.
TNW: What are the correct methodologies for testing anti-malware software?
Wells: Correct methodologies are based on reality. Testing the "real" clear and present danger to "real" end users must be tested. To do otherwise is a disservice to readers of the testing, to the product developers and to the product users. Testing simulated or out-of-date threats is a benefit only to one group -- the makers of malware.
TNW: Why do some tests fall short of the scientific method?
Wells: Testing that falls short is usually because correct testing is very time-consuming. Testers take short cuts. Magazines on deadline often doom themselves to poor quality testing by not taking the time necessary to do the job right. Such tests would best be done by established, bona fide, independent testing bodies.
TNW: What is the biggest challenge anti-malware software developers face in today's threat environment?
Wells: The biggest problem for anti-malware developers is glut. To illustrate, imagine if the developer of a dictionary program had to deal with receiving and writing definitions for 200 to 250 new words per day.
TNW: What is the potential risk to end users if anti-malware software is not properly tested?
Wells: Anything short of comprehensive and correct testing is not only a disservice to users, but a very real danger. Faulty testing gives misleading results. The CU test of anti-spyware products used only the SpyCar simulator. One could easily write a program that would pass such a test with flying colors, but at the same time, not detect any real threats. So such a program could take first place. What if that program was actually a rogue spyware application?
TNW: What should consumers be looking for -- what criteria -- in anti-malware software?
Wells: Look for reality. Are real, current threats used? Are the computers actively infected? Is remediation verified? Some testers stoop to scanning a disk full of threats. But some anti-spyware products specifically look for threat traces where they actually exist in an infection; such products may not even look at the disk full of inert threats.
TNW: In light of all the recent hubbub, how can consumers be sure they are getting what they pay for?
Wells: Don't base buying on a single test, even if that test is by an organization considered to be independent and exemplary. Look at other tests by bona fide testing bodies. Prefer testing bodies with experience in and focus upon computer security products. In fact, it may be a good idea to avoid generalized testers altogether; no matter how good their testing is on cereals, interior paints and upscale hatchbacks.
Talkback: 
