Yogi Berra once opined “Baseball is 90% mental – the other half is physical.” Yogi missed a starring role in performance profiling! Batting average is a good benchmark of performance for a baseball player, but it may be more useful to know how well he bats in night games with runners in scoring position playing on artificial turf while facing a left handed pitcher from Mississippi.
The Nexus One has generated quite a bit of media interest. A recent search in YouTube returned over 3,000 Nexus One-related videos. This Smartphone has generated many positive and not so positive reviews as well. However, nothing caught my attention as much as Engadget’s review of Nexus One, which provided a movie demo of Nexus One, Motorola’s Droid and Apple’s iPhone3GS simultaneously loading Engadget’s home page. This singular example showed the iPhone3GS as having the fastest web page download performance among the three Smartphones, but in our comparisons, we never saw a situation where Nexus One was slower than any other Smartphone. In our page download benchmarks, the Nexus One was 50% faster on average, and over 2x faster in some cases.
Our i-Bench and SunSpider benchmarks for Nexus One, iPhone3GS and Droid are shown below (lower means faster).
Furthermore, in our standalone testing of google.com, sandiego.craigslist.org, mobile.washingtonpost.com, amazon.com, and nytimes.com, the Nexus One was fastest in every case. However, Engadget’s web page did indeed demonstrate a case where iPhone3GS usually (but not always) performed faster than Nexus One. We ran several runs, and on average, the iPhone3GS downloaded the page ~1 second faster than Nexus One. My colleagues Rajiv and Zein captured the video below for reference.
This situation represents a perfect example of content variability and why fair comparisons should represent more than a single download case. Whereas some web pages download in 1-2 seconds, some may take 20 seconds or longer. Even if you download an uncached web page many times in a row, the content on that single page could change due to active content. As I said, benchmarking is a tricky business!