Android OEMs almost all cheating in benchmarks

Posted on Thursday, October 03 2013 @ 12:43 CEST by Thomas De Maesschalck
AnandTech published an article about the state of cheating in Android benchmarks, you can read it over here. The site notes that almost all Android OEMs are cheating in benchmarks and points out that what these firms are doing is quite absurd due to the relatively small performance gains - it's just not worth it considering all the bad press that follows when these cheats are discovered. The impact on CPU tests is between 0 and 5 percent, and a bit over 10 percent on GPU benchmarks.
The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors. Whether the OEMs choose to change or not however, we’ve seen how this story ends. We’re very much in the mid-1990s PC era in terms of mobile benchmarks. What follows next are application based tests and suites. Then comes the fun part of course. Intel, Qualcomm and Samsung are all involved in their own benchmarking efforts, many of which will come to light over the coming years. The problem will then quickly shift from gaming simple micro benchmarks to which “real world” tests are unfairly optimized which architectures. This should all sound very familiar. To borrow from Brian’s Galaxy Gear review (and BSG): “all this has happened before, and all of it will happen again.”


About the Author

Thomas De Maesschalck

Thomas has been messing with computer since early childhood and firmly believes the Internet is the best thing since sliced bread. Enjoys playing with new tech, is fascinated by science, and passionate about financial markets. When not behind a computer, he can be found with running shoes on or lifting heavy weights in the weight room.



Loading Comments