Update: The image alt tag says "Interop 2022 scores. Chrome/Edge 71, Firefox 74, and Safari 73." It also should be in the body text, and is out of 100 I guess.
back
1 comments
Furthermore, I think the meaning of these numbers is a bit hard to grasp because the three numbers are not and cannot be independent from each other.
Just as a thought experiment, imagine you saw Chrome/Edge and Safari being 0, and Firefox being 100? What would that even mean?
This was a great question, but the article implies that the numbers are in fact mostly independent of each other:
> Interop 2022 has ten new focus areas:
> There are also five additional areas that have been adopted from Google and Microsoft’s “Compat 2021” effort:
> A browser’s test pass rate in each area contributes 6% — totaling at 90% for fifteen areas — of their score of Interop 2022.
> We believe these are areas where the standards are in good shape for implementation, and where improving interoperability will directly improve the lives of developers and end users.
A "test pass" could just be a test on which all three browsers give exactly the same result. But that would immediately imply that all three browsers would have the same Interop score, which isn't true. So I would guess that the number refers to "compliance with a standard", where you can be objectively right or wrong, rather than strictly to "interoperability", where all that matters is whether you agree with everyone else.
(The other 10% of the score really does appear to be graded on interoperability:
> Interop 2022 has three investigate areas:
> These are areas in which we often see complaints from end users, or reports of site breakage, but where the path toward solving the issues isn’t clear. Collaboration between vendors is essential to working out how to fix these problem areas, and we believe that Interop 2022 is a unique opportunity to make progress on historically neglected areas of the web platform.
> The overall progress in this area will contribute 10% to the overall score of Interop 2022. This score will be the same across all browsers. This reflects the fact that progress on the web platform requires browsers to collaborate on new or updated web standards and accompanying tests)
What I meant is that, in the areas that are not well standardized, and where the browsers don't agree with each other, who is to say that Chrome is right and that Firefox is the one being incompatible/non-conformant? In this sense, the scores are not independent.
Among the areas listed, I understand that at least 10 focus areas are well-standardized and the major browser vendors agree amongst themselves regarding the interpretation of the standard. So that means 60% of the score should be independent. There are 5 other areas that were pulled from Google's and Microsoft's Compat 2021; those might not be well standardized, in which case 30% of the score might be disputable. (UPDATE: I think the 5 additional areas are also well standardized, so I guess you could also say they are independent and not subject to dispute.)
And then the remaining 10% investigative areas are quite probably disputable.
You are correct; the real success in terms of improving the web for users and authors will be if all the browser engines end up aligning on behaviour and getting a good score. Nevertheless having per-browser scores is useful for the browser teams themselves to help them prioritise their own contribution to that overall success.
You're also correct that there are cases where the behaviour as expected by current specs / tests might turn out to not be web compatible and so alignment will require changing the expectations and might reduce scores for implementations that were previously following the spec. That is already a thing that happens (although it would be great if it didn't). In practice, most of the focus areas are quite new parts of the platform with well written standards and not too much existing content. So I don't expect this to be a big effect. The "web compat" area has some cases where this outcome is more likely, but it's comprised of user-experienced problems in real sites, so the incentive to fix the problem is much greater than the incentive to avoid a temporary drop in the interop score.
The investigate areas are specifically those where it's unclear what the path to interoperability looks like, or how to measure interoperability. In those areas, figuring out what we can realistically do to ship the feature in a way that works the same across browsers is the point. Hopefully, if we run this kind of thing again in the future, we have a clear picture of what's web compatible, and can turn those investigate areas into focus areas that are scored on the basis of a testsuite that everyone agrees is useful.