Did Samsung just get caught gaming benchmarks?
A report by Anandtech, a site specializing in benchmarks and other performance-based analysis, uncovered a whopper on Tuesday: the site found code that apparently triggers the international version of the Samsung Galaxy S4 to run at a faster speed when certain benchmarks are performed.
Specifically, Samsung--which manufactures the phone itself as well as the 8-core Exynos 5 Octa chip within it--apparently programmed the chip to run at its full rated speed of 533MHz when in the presence of four benchmarks: AnTuTu, Linpack, Benchmark Pi, and Quadrant. Otherwise, the site claimed, Samsung programmed the Exynos 5 Octa to run at 480MHz during other times, even while playing games.
TechHive tested the U.S. version of the Galaxy S4, which uses a Qualcomm APQ8064T chip, and found no such discrepancies.
The kicker, however, is a string of code that the site discovered, which the code identified as "Benchmark Booster." In the presence of the benchmarks, the code instructed the processor to run at the higher 533MHz speed. The site also reported that the string appeared to be designed so it could be called by other benchmarks--running at the higher frequency in the presence of those other benchmarks in the future.
Why it matters
The upshot? If benchmarks are used to compare the performance of the Galaxy S4 against rival phones, Samsung could receive an unfair advantage. (TechHive this year stopped using synthetic benchmarks to evaluate phones.)
Unfortunately, the gaming of benchmarks is a shameful tradition within the PC industry. For instance, companies like ATI and Nvidia have been accused of altering their driver software to skew test results favorably. However, FutureMark, the organization in question, later retracted its allegations. Nevertheless, it's a surprise, and a sad one, to see those tactics in use in the mobile space.
Anand Lal Shimpi and Brian Klug, the authors of the Anandtech piece, said as much.
"We've said for years now that the mobile revolution has/will mirror the PC industry, and thus it's no surprise to see optimizations like this employed," the two wrote. "Just because we've seen things like this happen in the past, however, doesn't mean they should happen now."
Samsung couldn't be reached for comment at press time.
Sign up for Computerworld eNewsletters.