Google Chrome in a runaway lead for browser performance supremacy
By Scott M. Fulton, III | Published October 31, 2009, 1:51 PM
Print ArticleE-mail Article41 Comments
If Apple's Safari is going to make any kind of a challenge for best performing Windows-based Web browser moving into next year, it needs to be now. In Betanews' most extensive testing to date, involving tests that by anyone's guess should not have given it any special advantages, the latest stable edition of Google Chrome runs away with a three point lead over the latest stable Safari -- a lead that now grows by one-half point with each point release.
Chrome now posts test scores in certain heats of our revised CRPI 2.2 test battery that are virtually obscene -- so far ahead of competition that we have to validate our results on various machines to make sure we're not generating false results. For example: On the control flow element of the SunSpider test, both Chrome 3 and the dev build of Chrome 4 post record low time scores of 2.6 ms. This is an element that tests the JavaScript interpreter's capability to keep track of nested loops and its location in a twisted program. By comparison, the latest public Firefox 3.6 Beta 1 -- released late yesterday afternoon after a flurry of delays totaling over one month -- posts a score of 38.2 ms in that category.
Mozilla's team has been making efforts to better Firefox' control flow scores, evidently knowing how much they influence test results like ours. The evidence comes from the latest daily builds of Firefox 3.7 Alpha 1, on the "Minefield" track, whose control flow scores recently quantum-leaped down to 8 ms. That's almost a 5x improvement, but it will need another 3x blast to catch up with Chrome. The latest stable Apple Safari scores 3.4 ms on this element.
Another example: A factorial is the result of multiplying together all positive integers that are less than or equal to a number, and a new element of our testing battery includes a classic algorithm for obtaining the set of all factorials. On this heat, the higher number is better since the objective is to obtain as many factorials as possible over a set period of time, so the score is a relative index. While Firefox 3.6 Beta 1 scores a 17 on this heat, Chrome 3 scores a 150.4, and the latest dev build of Chrome 4 scores 164.5.
There are certain things that Chrome does where the score differences are factors of 10, where one might get the impression that Google is improving Chrome just to score better with Betanews (the company has been expressing its interest to us directly in recent days) or more likely, with its own internal test suite. Indeed, the company's V8 benchmark suite would have users thinking the browser is hundreds of times more capable than its competition -- a claim for which we just don't see the practical evidence just yet, which is why V8 isn't part of our tests.
That's why we've made an effort to pack our CRPI test suite with examinations of a multitude of real-world characteristics in various categories, now with 69 different "heats" in ten separate batteries, plus a multitude of derivative scores (e.g., average of 50 iterations, consistency between the fastest and slowest run, etc.) for each browser.
Once we include all these different elements, we get a much more practical and believable result. There are many reasons why a person chooses a favorite Web browser, with JavaScript performance being just one of them. But that factor is becoming more important now with the onset of applications delivered to you from the Web rather than your hard drive. So with regard to that factor alone, Betanews can say that Google Chrome delivers almost twice the performance (not quite 2x) of the latest stable Mozilla Firefox builds, with Apple Safari probably pulling close, and Opera being left in the dust until it comes time to be thinking about Opera 11...if not Opera 12.
Click here for a comprehensive explanation of the Betanews CRPI index version 2.2.
"Probably" pulling close? What, Betanews can't do better than that with respect to Safari? Yea, unfortunately there's still trouble with that: Apple's test builds of Safari come by way of grafts of its daily WebKit engine onto the existing 531.9.1 browser. Usually, after applying one of these grafts, the updated Safari displays better if not superior rendering performance than even Chrome. But its ability to serve as a full-scale browser for other tests vanishes. However, over the last week while Betanews has been trying to resolve this problem, it actually only got worse: Windows testers reported through Apple's forums that not even the grafting mechanism was working.
We validated those claims, discovering that the manifest which the replacement executables were being shipped with (the embedded XML files that point to proper COM components in the System Registry) were pointing instead to incompatible versions of Windows Common Controls, versions that may have worked back in the 1990s. Apple is apparently already aware of this, but as is the company's policy with regard to any kind of problem with its software or hardware, will not publicly comment.
The runaway train you see at the bottom of your screen there is Google Chrome. And yes, that splash of orange at the bottom of the platform-by-platform breakdown very graphically illustrates what we've been saying for the last few months: Chrome's platform of choice is Windows XP, the one with the smallest footprint that's deployed on the greatest number of netbooks -- the hardware platform where Google is working to make a breakthrough.
Usually Microsoft Internet Explorer isn't even a topic of conversation with regard to performance tests, except to point out that the older IE7 is our baseline. But over the past few weeks, IE8 has made news by marching the opposite direction, with security fixes and a Jscript update bogging down IE8 speeds quite noticeably. It's also created the only situation where the Windows 7 version of a browser is faster than the XP version; as you can see elsewhere, XP typically provides the much faster platform, although with Opera of late the difference is more minor.
Nowhere is Google's concentration on Windows XP more pronounced than in the Nontroppo CSS rendering test, where Chrome's XP scores continue to double its scores on Vista. Another trend worth pointing out here, though, is how well both Firefox and Opera perform on this test on Windows 7 (the blue line in the middle) compared with the other platforms -- for Firefox 3.6 Beta 1 and both stable and beta versions of Opera, XP is actually the slowest of the three Windows versions.
Chrome's performance on classic benchmark scores in the new JS Benchmark battery is phenomenal; and here, Chrome 4 on Windows 7 posts a higher score than on XP, which is not Chrome's usual pattern. The new JS Benchmark is about the interpreter's capability to handle complex, highly nested mathematical problems, which is more related to how the browser will run Web apps than render Web pages.
Mozilla's highest score yet on the Acid3 compliance test from the Web Standards Project comes from the latest Firefox 3.7 Alpha preview build: a 96%. Before long, we could be seeing all the non-IE browsers posting perfect 100% scores, making Acid3 almost a non-factor.
Safari's most impressive scores have always come from the SunSpider test on Windows XP, which are much better than even Chrome's scores on Vista and Windows 7. But it's Chrome's scores here on XP that are the true phenomenon, and the real indicator that Google builds Chrome to fly on the most popular Windows for netbooks.
We added the TestCube 3D battery earlier this month in deference to a number of Betanews readers who pointed out that we weren't paying enough attention to the browser's capability to plot and work geometry simultaneously. This test remains Opera's forte, especially on XP where Opera 10 Beta 1's numbers are better than double those of Google Chrome. There's something here about parallelism between rendering and math that Opera still gets and the others don't.
The SlickSpeed test measures how well each browser manages a variety of different JS libraries, especially for managing layout elements in a page and arrays full of content strings. Firefox's scores for stable and beta versions are improving here, especially for XP, although it's interesting how Vista and Win7 scores are very close to one another. Safari is the king, for now, with JavaScript library management.
The Nontroppo table rendering test examines how well the browser manages the <TABLE> element -- the part of HTML that CSS proponents would just as soon everyone forget about. But many old-style Web sites prefer the old HTML way of dividing pages over the more modern CSS, maybe just because CSS is just one more thing to have to learn. The surprise here is how Chrome 4's performance in this test on Windows 7 is actually quite poor; we ran this test several times, and the amount of time Chrome 4 consumed in attempting to find the start of the table sequence, was much slower than for Chrome 3. Version 4 made up for this by rendering the table faster, but it could not make up for the time lost.
The third Nontroppo test in our suite evaluates how fast each browser renders a very ordinary page that does not use CSS -- a majority of Web pages still out there (though a declining one with Yahoo's recent closure of GeoCities). We just can't figure out the scores from Mozilla's latest preview build of Firefox 3.5.5 (the "Shiretoko" track), but there it is: a 9.90 on XP versus a 2.34 on Windows 7, even slower than Vista. Both 3.5.5 and 3.6 Beta 1 are actually slower at accessing the first element of the page in memory than even 3.5.4, but 3.6 makes up for this by rendering the page almost one-third faster.
One of the two new additions to our CRPI suite is Delgado's Canvas rendering test, which evaluates how well the browser uses its internal Canvas object to render graphics in memory, rather than rely on the server to provide graphics for it. It's a two-part test where the outlines of the 48 contiguous United States are rendered in the first, and a detailed outline of Alaska is rendered in the second. Firefox is actually quite good with this -- we expected Firefox to be bested by Opera here, and it wasn't. But Chrome still has a clear advantage.
The second addition is actually a very old library that tests how well the interpreter executes individual JavaScript instructions in a sequence of 1,000,000 iterations. There were two surprises for us here: First and most obvious is the poor performance of Opera, a clear indicator that Opera slows down exponentially as a task gets heavier and heavier. If Opera's developers can concentrate on this alone, Opera 11's performance could start to rival at least Firefox's. The second surprise was Firefox itself, whose scores were right in line with those of Safari and Chrome -- quite competitive, in fact -- except for Chrome on XP, where Google once again runs away with the show.
Continue at
http://www.betanews.com/article/Goog...acy/1257011509
Re: Google Chrome 
Linear Mode
RSS Feeds |
Podcasts
