back

by bikeshaving·5y ago·view on hn ↗
UTF-32 is a bit like giving up and saying code units equal code points right? I’m more interested in the comparison between UTF-8 and UTF-16, where UTF-8 requires 1 to 3 bytes just in the BMP, with 3 bytes for CJK characters. I’m saying that as a quick measure of the actual length of a string, UTF-16 is much more predictable and provides a nice intuitive estimation of how long text actually is, as well as providing a fair encoding for most commonly used languages.

RE the UTF-16 vs UCS-2 stuff, that’s probably a distinction which has technical meaning but will collapse at some point because no one actually cares, much like the distinction between URI and URL.

1 comments
> I’m saying that as a quick measure of the actual length of a string, UTF-16 is much more predictable

Meaning the software will deal much less well when it's wrong.

> and provides a nice intuitive estimation of how long text actually is

Not really due to combining codepoints, which make it not useful.

> as well as providing a fair encoding for most commonly used languages.

Which we know effectively doesn't matter: it's essentially only a gain for pure CJK text being stored at rest, because otherwise the waste on ASCII will more than compensate for the gain.