Update: Retested Mozilla Firefox w/ proper trunk build, thanks Kirkburn for the heads up (and schooling).
Shortly after the release of the Windows Internet Explorer Testing Center – a site housing a comprehensive browser feature implementation test suite – there have been some grumblings regarding the accuracy of the data within. More specifically, Microsoft performed a comparison of its Internet Explorer 9 browser technology – currently in developmental stages – to stale builds of Mozilla’s Firefox, Apple’s Safari, Google’s Chrome, and … that browser no one cares about (sadly) – Opera.
Sounds like a valid argument to me. I decided to re-test using builds of Mozilla Firefox “Namoroka” (1.9.2.5pre), Mozilla Firefox “Minefield” 3.7a5pre, Google Chromium (6.0.397.0/46552), and Apple Safari w/ a newer WebKit engine (r58804) that matched release dates with Internet Explorer 9 (May 5, 2010). After clicking around the site a hundred or so times in each browser, the results… changed. Each browser made noticeable improvements in areas like CSS3 and DOM; Apple barely beat out Chrome for the most improvement, while Firefox proved to be a bit slower (but error free). The numbers, Ed Bott’s favorite meal, are below. The takeaway here isn’t the numbers – forgive me if I made a slight error – but the fact that you can’t compare bleeding edge browser builds with stale release builds. That’s just not fair.
Actually it might be even more fair to compare against trunks for Chrome and Firefox. That might be what you did for Firefox, but the Chromium build you tried looks out of date, the dev channel of Chrome is newer and that is like 2 weeks old.
I think I’ll try the benchmark myself (didn’t realize it was public when I made that tweet, my eyes went straight for the charts) with the newest Chrome since it’s relatively easy to do (http://build.chromium.org/buildbot/snapshots/chromium-rel-xp/ for nightlies) and they don’t conflict with any currently installed Chrome (Chrome and Chromium can coexist).
In honesty that tweet was just a knee-jerk reaction from the claim that IE is somehow magically more compatible than the other browsers after one release, when the others have been working at it harder for years. My suspicions are partly rooted int he fact that many of their fancy demos that perform better in IE are suspiciously graphically intensive… and they do tout DirectX rendering as being speedier than other browsers (which I don’t contest… yet. Firefox was looking into a similar feature). I suspect they are doing the same here… showcasing tests they know IE will ace.
So I just popped on over to a few independent benchmarking/testing sites.
IE9 preview scores a 19/160 at http://html5test.com/, while Chrome trunk gets 135 (Chrome Dev, being older, somehow gets 142). To be fair they’re going to update their tests there so we’ll see. But IE does clearly fail to skew the “Coming Soon” box at the bottom as Chrome correctly does.
I also hopped over to Peacekeeper (google it, it’s a browser benchmark suite). Chrome Dev scores 9106 for me, Chromium trunk 8395 (both setting high scores for my machine), and IE9 2357. IE9 even has a handicap applied; Peacekeeper does not run canvas tests for IE and doesn’t even factor in the 0 scored in to the results, which I really think they should do, since otherwise IE’s scores are inflated. Of course for IE9 it should actually run the tests, it doesn’t look like it did.
Both Chrome Dev and Chromium trunk pass Acid 3 flawlessly with 100% and perfect rendering, although Acid 3 complains test 69 was “less than perfect” though it passed. IE9 scores 68. A big improvement from IE8 but still not close. An iframe brielfly appears and is replaced by a white rectangle over the Acid text. “YOU SHOULD NOT SEE THIS AT ALL” is shown prominently in the corner. The white on purple X, which I think is a web font test, shows (it should not). The final report on failed tests exceeds IE9′s character length for alert()s and is truncated.
Oh and as for my Chromium tests with the IE test site, it was looking the same as yours and counting by hand was getting tedious (I see no “Run all” button) so I quit and did the other benchmarks instead.
For Firefox, it would probably be better to use the Minefield builds (3.7a5pre) to run the tests. You compared 3.6 to itself, since the version you used is identical to the one Microsoft used + security and stability fixes + out-of-process plugins, while the 3.7 alphas have progressed much further.
Guys, the point of the re-testing was to level the play field. I tested with browsers -all- dated May 5, 2010.
Rafael, and Firefox 3.6.5 is not the next major version, 3.7a5pre is. 3.6.5 would be a security and stability fix release only, and is not where the main development occurs. I’m surprised you don’t know this.
It’s not glacial, you just looked in the wrong place.
Thanks for the heads up, I’ll re-test and correct my mistakes… but how the hell do you find a May 10th build of 3.7x in this rat nest? http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/
Re-tested and updated accordingly. Thanks for the heads up and sorry for the screw up.
Why is Firefox still failing some Acid 3 tests?
Is the latest nightlies for Firefox failing acid 3 as well?
In MS’s own graphics tests, Opera performed far smoother (higher FPS) without sacrificing image quality than IE9 on Windows 7. IE9 sux. No XP support = EPIC FAIL.
@ Mitchel – Mozilla just kinda refuses to spend money implementing SVG Fonts, the last 3 points they need on Acid3: http://weblogs.mozillazine.org/roc/archives/2010/06/not_implementin.html
someone, sacrificing what image quality? Opera does indeed perform impressively, no doubt. Why does that make IE9 “bad”?
DigDug, spend money? You realise it’s mostly a volunteer effort? Did you even read the post you linked, which pointed out it’s basically nothing to do with time or money?
Rafael, thanks for the update. Gotta say though, Chrome/Webkit’s rate of advance truly is impressive.
Dan the reason chrome scores higher than chromium is that google bundles more video/audio tag formats with the chrome releases, things like H.264 are not in Chromium but are available in Chrome
@someone: Dunno how you got your results, but on my Win7 machine IE9 significantly outperforms Opera on the graphics test suite. As DirectX rendering is wont to do, vs. software rendering.
Should do Opera 10.6 and Opera 10.5.
10.6 blows everything else out of the water in terms of performance and compatibility. It mightn’t be the most popular browsers, but in my eyes, it is the best.
Oh and my own peacekeeper benchmarks:
http://service.futuremark.com/peacekeeper/results.action?key=2bJb
Fastest system:
Opera (v10.60)
16931 Points
Processor
Intel Core i7-920 Processor
Graphics card
ATI Radeon HD 5850
Memory
6144 MB
OS
Windows 7 Beta
Form factor
Desktop
@Rafael Rivera
You might want to confirm with some experts on the quality of all of those test suite then, I just got this off of twitter.
http://isgeolocationpartofhtml5.com/
The test suite has been updated, making most of these results inaccurate: IE 9 fails at some of the tests which had read the specification backward.
Gotta run’em all over again…
Safari is good, I’m using the new Safari 5, and it’s fast. But Safari don’t have much extensions and plugins.
I would love to see this update, both for the new tests as well as most current verision of chrome. I wish the testing center had a ‘run test’ button and would display the results for your browser. But until then, I will keep coming back here for updates. thanks.
That makes IE bad because IE cuts off XP compatibility for hardware acceleration yet Opera with software rendering outperforms IE9 with hardware rendering in their OWN tests.