"By using the API instead of the user interface, AV-TEST bypassed our warning system designed to keep customers from being harmed by malware," said Felstead. "Bing actually does prevent customers from clicking on malware infected sites."
Felstead said that users see the warning only once in every 10,000 searches, or 0.01% of the time, a number close to AV-TEST's 0.012%. "In any case, the overall scale of the problem is very small," Felstead asserted.
AV-TEST confirmed today that it relied on a Bing API (application programming interface) to collect search results from Microsoft's engine.
"No links were clicked/followed through the search engine," Andreas Marx, CEO of AV-TEST, said in a Monday email reply to questions. "We simply grabbed the URLs and downloaded them on our own systems for further analysis. We didn't want to test the warnings from the search engine but simply how many potentially malicious websites are returned by the search engine."
Microsoft cited vacationhotline.net as an example of how Bing warns users. But the site's owners denied the infection allegation. Computerworld confirmed that Bing shows this warning. (Image: Microsoft.)
Marx acknowledged that some search engines ward off users from suspicious links with warnings, but of those, not all are as clear as Bing's, and could be easily ignored or dismissed by users.
He also cited Felstead's claim that "Our data shows that these warnings block 94% of clicks to malicious sites" to defend AV-TEST's approach.
"Microsoft argues that their warning is 94% effective, so 'only' 6% of the people will click on the malicious link anyway," Marx countered. "Still, that's a lot of people."
Additional protective measures, including Bing's in-results warnings, those displayed by browsers -- all the major Web browsers have mechanisms for warning users of potential danger when they click on some links -- and others generated by security and antivirus software, were "out of scope for this study," Marx said.
Marx stood by the study, and said it would not be revised to take Microsoft's complaints into account. "The report was NOT designed to be a 'safety comparison' for search engines," he said.
However, he confirmed that AV-TEST was considering revamping its methodology for future tests. "In the next report, we might be able to report that search engine A warns [of] 30% of the malicious links, or if it's just 1%, or more like 70-80%, plus how many false positives we've seen," Marx said.
As to why Bing indexes suspicious links and shows them in its results, Felstead contended that "most are legitimate sites that normally don't host malware but have been hacked."
He also called on other arguments to defend Bing's approach, including one related to competition with Google and other engines. "We warn our customers rather than suppressing the result [because] if a user searches for 'vacation hotline' and doesn't get the site they're looking for, they perceive Bing to be an incomplete index of the Web which impacts their confidence of the engine," Feldstead said.
Sign up for Computerworld eNewsletters.