Independent antivirus testing organization AV-TEST found that the version of the security product submitted for evaluation by Chinese company Tencent included optimizations specifically for achieving a better performance score.
Test labs check the detection rate of the submitted antivirus products with their default configuration, since most users are likely to leave the settings unchanged. A different setup could have a notable impact on overall results, creating a false perception of the detection capabilities.
Qihoo 360 sanctioned for changing the default settings
Tencent’s product is not the only one discovered by testing bodies to attempt cheating the performance tests. Last week, AV‐Comparatives announced that it was investigating another security solution for trying to influence evaluation results.
It was later revealed that the product was submitted by Qihoo 360, who did not provide for testing the exact same variant offered to its customers. The difference consisted in having a different antivirus engine enabled by default, which influenced the amount of false positives detected.
According to a joint statement from AV‐Comparatives, AV‐TEST and Virus Bulletin, Qihoo provided them a software release that had its own engine disabled, while the one licensed from Bitdefender was turned on.
The variant available to users in Qihoo’s main market came with an opposite configuration, leading to “a considerably lower level of protection and a higher likelihood of false positives,” the statement informs.
Following this discovery, the three testing laboratories revoked all certifications and rankings awarded this year to Qihoo 360 products.
Tweaks put account for degraded detection
As far as Tencent is concerned, although the tweaks were present in widely distributed versions of the antivirus, the benefits emerged during AV-TEST’s assessment and carried no significant advantages to normal users.
On the contrary, the modifications would be detrimental to users, as the level of protection under a real-life scenario would be lower.
All three security testing bodies were involved in this investigation and have taken the same stance against vendor practices that would influence the test results.
“The labs will be imposing stricter controls on participants to reduce opportunities for such actions, and will revoke all affected certifications and awards granted so far in 2015,” the three organizations say in separate posts on Facebook.
Tencent’s product came in fifth position in the real-world protection test performed in March, recording a detection rate of 99.6%, surpassed by Panda, Bitdefender, Kaspersky and Avira.
