▲ 140 points
back
131 comments
If only Windows, Java and JavaScript could also move away from internal usage of UTF-16, it's purely a legacy format and the worst of both worlds (UTF-32 and UTF-8). Even worse is that unicode itself, which should in theory be a list of codes for glyphs, modifiers and other script related values, that's independent of encoding, had to have some codes reserved for "surrogates" for the UTF-16 encoding anyway. UTF-8 doesn't need such a thing...
I have an old saw about UTF-16 not being an irredeemable format and UTF-8 eating the world being bad, and I'm happy to dig it out again.
UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings.
The main benefit of UTF-8 if you're say, Chinese, is interop. Everything else is worse.
You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy. Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too.
You might also think "what about ASCII compatibility?" ASCII compatibility is an anti-feature. You should never be indexing into UTF strings (you always have to iterate, or save the results of an iteration), upper/lowercasing isn't addition/subtraction, etc. etc. You also can't just forget about encodings as a result--you can store ASCII in something expecting UTF-8, but you definitely can't store UTF-8 in something expecting ASCII. So if you're sniffing/decoding/tagging a format anyway, you may as well be agnostic.
You might also think "OK OK, you could be right, but what about HTML, which is mostly ASCII and would nearly double in size if it went from UTF-8 to UTF-16." Practically all HTML is gzipped, so the difference is pretty small, plus the majority of text isn't HTML (almost anything stored in a database, almost anything in a file on your computer, etc.)
Different encodings are good at different things. There's no one superior encoding for all uses. What we need is text encoding agnosticism.
---
In fairness, I will say I've heard that UTF-8 is pretty popular in countries with exactly the kind of languages I'm talking about, so the issue is mostly moot at this point. I just think UTF-16 gets a really bad rap, and I think we shouldn't just gloss over UTF-8 having won because it's good for European languages.
The main evil of UTF-16 is that it's usually not UTF-16, it's UCS-2, which is to say, a fixed-width format which has been shoehorned into a varible-width format to support non-BMP code points. As a consequence, systems that are built on "UTF-16" (i.e., really UCS-2) use designs centered around the ability to random-access Unicode strings despite the fact that doing so reliably isn't possible.
> ASCII compatibility is an anti-feature.
Actually, it's not. One of the main reasons it's important is...
> What we need is text encoding agnosticism
... that bit that you want. If you're supporting multiple encodings, you need a reliable way of signalling that encoding. And file metadata tends to be insufficiently reliable, so it ends up being in-band for text (e.g., HTML files). If all the encodings you need to support are ASCII-compatible, and the directive to set the charset is itself pure ASCII, that means it is perfectly safe to parse the file as ASCII to figure out how to parse the file. If the charset isn't ASCII-compatible... well, that historically has led to security vulnerabilities. Ergo, a non-ASCII-compatible charset has heightened security risks.
That said, my own experience when dealing with the mess that is charsets is that supporting multiple charsets is itself a painful process that isn't worth it.
> systems that are built on "UTF-16" (i.e., really UCS-2) use designs centered around the ability to random-access Unicode strings despite the fact that doing so reliably isn't possible.
100% this. In javascript you can accidentally cut characters in half. This lets you accidentally construct strings which have invalid encodings - and that can lead to all sorts of weird problems. In comparison, using rust's safe API, its impossible to construct a string which contains invalid unicode.
For example, if I put an emoji like this U+1F970 in javascript, I can then split it in half and put one half into a JSON string - like this: "\"\\ud83e\"". Javascript will treat this as "one character". Rust will treat this as an encoding error, and will refuse to parse it back into a string.
Another implication is that javascript's .length field looks useful, but its almost never what you actually want. How many unicode characters in this string: "\ud83e\udd70"? Trick question - even though javascript says the string's length is 2, there's only 1 unicode character in that string. How many bytes does this string (with JS length 1) "\u270C" take up over the wire? Javascript says the length is 1, but it takes 2 bytes to send in any encoding (utf8 or utf16).
What a mess.
Without some specific way of knowing what encoding you're dealing with, you can have this problem with any number of encodings you're working with (including ASCII suddenly having UTF-8 bytes in it). As soon as you're in a "assume an encoding and maybe be surprised", you're asking to be surprised.
Totally agree these discussions circle around to "you need some kind of OOB way to know what encoding you're dealing with", which puts us back to everything being encoding agnostic, with some way to configure it. Maybe that's locales, maybe that's a type 32-byte integer at the beginning, dunno.
If you care about text size, you should compress your text; that'll save much more space, since it can optimize for what's actually used in the document.
> ASCII compatibility is an anti-feature.
ASCII compatibility is extremely useful if you're working with, for instance, filenames or programming languages. You can lex UTF-8 and handle separators like `/` or quotes like `"` and `'`, because those bytes can never occur otherwise.
Sure it's useful, but it's not at all a guarantee. You can't assume the byte stream is UTF-8, and UTF-8 has a NUL in it.
Reading about it, filename encoding looks like a mess, GLib apparently assumes UTF-8 unless you set it with an environment variable, which I think is wild. Qt uses the locale, which feels reasonable (ignoring that locales are themselves bananas). But assuming UTF-8 and relying on ASCII compatibility and no NULs feels risky.
I'm Chinese, and I prefer UTF8 to UTF16.
> You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy. Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too.
BOM breaks substring. Everything you take a substring, you need to prepend a BOM if you want to serialize it to somewhere else. If you want to concat two strings with BOM, you need to remove one of them. All of these are unnecessary pains.
> You might also think "OK OK, you could be right, but what about HTML, which is mostly ASCII and would nearly double in size if it went from UTF-8 to UTF-16." Practically all HTML is gzipped, so the difference is pretty small, plus the majority of text isn't HTML (almost anything stored in a database, almost anything in a file on your computer, etc.)
This just contradicts your own reasoning that UTF-16 is better for East Asians due to size savings.
That is a minor nuisance when dealing with raw bytes. Any sane language representation would probably store it as a separate field egen not on the wire.
I would probably do BOM, LUT, BYTES. Where LUT is a lookup table of indices for every 64 chars so that you could do o(1) random access for the vast majority of cases.
Substrings just get the BOM too. EZ.
Re: HTML, people usually bring it up because it's a lot of ASCII that'll all double up under UTF-16. Mostly my argument is that most text isn't HTML, so size savings on other text files still matter.
> In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings.
Meanwhile in lots of other countries, UTF-8 lets you use 1 byte to store each character instead of 2; which is also a huge savings.
The question is, is it better for the world to pick one encoding and have all programs use it, or pick several encodings and switch between them depending on the context / country? Picking one encoding means we can reuse our code more easily. Picking multiple encodings means we get slightly smaller file sizes.
For my money, the best approach is to use UTF-8 everywhere. This lets us reuse code. Then if you're worried about file size on disk or over the wire, compress your text content with LZ4 or snappy or something. That'll halve the size of text for everyone and LZ4 is so fast its essentially free from a computational standpoint.
As this blog post demonstrates, quite a lot of code needs to deeply understand text encoding to work (be that UTF-8 or UTF-16 or whatever). Its a big savings in programming hours if we only have to write this code once.
I mostly disagree. We already have lots of code that deals with tons of different encodings, UTF-16 is pretty easy and there's lots of libraries already, etc. But if we ignore all that, your argument is against UTF-8, which succeeded UTF-16.
> UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings.
"Only ever" is an exaggeration. The vast majority of standard Chinese (aka. Putonghua) characters are in the BMP, but occasionally there are characters outside of it. In particular a lot of Hong Kong characters are outside of the BMP, and before emoji forced software to fix things, systems claiming to support UTF-16 (but were instead UCS-2) kept failing for say 1% of the characters.
I've had much better luck with UTF-8 systems than purported UTF-16 ones. The problem with UTF-16 is not necessarily in its inherent technical issues, but rather, it's how often systems with UCS-2 support just claim they support UTF-16, and leave users/developers with a crappy experience when working with characters outside the BMP. There's also the encoding agnosticism thing -- with UTF-16 you run into issues about byte order marks (with or without), endian problems, and generally incompatibility with ASCII.
As I said, the situation probably improved quite a bit after emoji became popular. But if I had the choice, I'd choose UTF-8 over UTF-16 any time.
Also, although it's generally true that you "can't store UTF-8 in something expecting ASCII", in practice a lot of systems are somewhat more graceful than that. For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string. Similarly it's possible to edit a UTF-8 text file as an ASCII file if the editor preserves the "binary" bits.
In conclusion, "different encodings are good at different things" is only true when considered in isolation. The historical baggage of UCS-2, and separately, of ASCII-compatibility tilts the balance strongly in favor of UTF-8 IMHO. The alleged 33% saving in Chinese text doesn't really matter that much in the grand scheme of things -- which is perhaps why people use UTF-8 in spite of the "advantages" you mentioned.
Yeah I mean, I broadly agree with what you're saying. Text encoding is a mess, and pragmatic solutions are needed. And when jamming text in various places, jamming UTF-8 is typically more successful than jamming in UTF-16, because of NULL bytes and the shortsightedness of UCS-2.
But I think that 99% of the things you have to do for UTF-8 (validate, encode/decode, no indexing) you have to do for UTF-16, and if we're arguing for systems that are UTF-aware, UTF-16 does great. If we're arguing for trying to jam encoded bytes into systems, well it's a crapshoot. UTF-8 being better at winning that crapshoot doesn't feel like a recipe for robustness to me, and certainly doesn't strike me as a perfect solution. OOB configuration does, though.
> For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string.
Well, this is a weird example. grep uses locales, so yeah you're probably fine if you're grepping for ASCII using an ASCII-compatible locale. But if you're not, then what you feed to grep has to be encoded in your locale's encoding (I think). So I think this is actually an example in favor of "this should be OOB configurable", and an example of "ASCII-compatibility lulls into a false sense of security".
> I have an old saw about UTF-16 not being an irredeemable format and UTF-8 eating the world being bad, and I'm happy to dig it out again.
> UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings.
That doesn't make UTF-16 not a terrible format, though. East Asian countries use better formats instead. For example, in China they use GB18030.
When your case for "UTF-16 is not an irredeemable format" is "it's better than UTF-8 at a task where nobody would use either format anyway", you're not making a strong case.
> When your case for "UTF-16 is not an irredeemable format" is "it's better than UTF-8 at a task where nobody would use either format anyway", you're not making a strong case.
Haha this is pretty fair; I know I'm being pedantic. But I wouldn't say any GB* encoding is better than UTF-16. Another commenter pointed out that you really want the ability to--say--paste arbitrary text into an editor and for that editor to be using an encoding that can handle it.
Or even for the web, if we actually took the Accept-Language header [0] seriously, that could also be a big savings.
> You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy.
Checking the BOM is not enough, you also have to handle it. Non-LE BOMs (like surrogates, at least before the emojipocalypse) are rare enough that many "UTF-16" based tools simply only support UTF-16LE. They are also stupid because you need to know that you are dealing with UTF-16LE or UTF-16BE in the first place - either through heuristics or because it is specidified and then you can also guess/specify the byte order.
And like surrogates, the BOM is another wasted Unicode character that is only needed for the UTF-16 mess.
> Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too.
You don't need validation for most UTF-8 tasks, GIGO is often more reasonable for things entered by humans.
> you definitely can't store UTF-8 in something expecting ASCII.
Unix tools would disagree.
$ echo "Hello Wörld!" | tr '!' '?' Hello Wörld?
Many tools only care about finding substrings. UTF-8 gurantees if the byte sequence making up the encoding of a Unicode character (or sequence) appears in valid UTF-8 encoded text then it also decodes to that Unicode sequence.
> Practically all HTML is gzipped
Not when it is being parsed it isn't.
> Non-LE BOMs (like surrogates, at least before the emojipocalypse) are rare enough that many "UTF-16" based tools simply only support UTF-16LE.
Gymnastics like this always occur when you don't know the encoding. If you didn't know you were getting UTF-8, welcome to heuristics or whatever.
> And like surrogates, the BOM is another wasted Unicode character that is only needed for the UTF-16 mess.
This isn't a big deal at all.
> You don't need validation for most UTF-8 tasks, GIGO is often more reasonable for things entered by humans.
You can't build robust systems this way. You always have to validate, and in order to do that, you gotta know what encoding you're getting.
> Unix tools would disagree.
Most of those use locales, or they don't expect ASCII they expect bytes. Again you can get lucky with this, but you can't build a robust system with luck.
> Many tools only care about finding substrings.
You have to scan through the string or save offsets (after an initial scan) either way, UTF-8 or 16.
> Not when it is being parsed it isn't.
Fair point!
> The main benefit of UTF-8 if you're say, Chinese, is interop. Everything else is worse.
Well, for HTML it turns out to be a wash.
While Han characters are typically three bytes in UTF-8, markup is ASCII and so only one byte — whereas in UTF-17, both the character data and the markup are two bytes. When you add them together the average cost per character is more or less two bytes, whether your Chinese-content HTML is encoded as UTF-8 or UTF-17.
This is probably true for HTML, but not true for lots of other text like whatever's stored in databases or on hard drives.
(LOL. How did I type "UTF-17", twice?)
> UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings.
Not really. That's just an excuse to be contrarian.
Any significant length of text will be compressed and/or have lots of latin characters.
> you definitely can't store UTF-8 in something expecting ASCII
That's not true.
> Any significant length of text will be compressed and/or have lots of latin characters.
A reasonable counterargument is Asian books on Project Gutenberg [0]. I guess you can say they're compressed on the wire, but they're not compressed in my RAM or cache.
>> you definitely can't store UTF-8 in something expecting ASCII
> That's not true.
Well alright, you can jam bytes pretty much into anything. And you get luckier with UTF-8 than you do with probably any other encoding. But lucky isn't robust, is all I'm saying.
(author here)
You are absolutely right! If your use case demands storage of CJK strings, UTF-8 is probably not your best bet.
But our data (mostly stringly typed, think product descriptions or links) is >99% Basic Latin characters and we are getting to a point where memory is actually becoming an issue. So it's neat that Haskell allows us to "upgrade" to UTF-8. With the data being in memory I think compression would not be very helpful either.
Edit: I also kind of agree that UTF-16 gets to much hate :D
Besides the surrogate characters there are also some other noncharacters: https://www.unicode.org/faq/private_use.html#noncharacters
Because of modifier characters, control characters like for bidi, stuff like soft-hyphens and ligatures, locale-dependent semantics (upper/lowercase, collation etc.), the general discordance between glyphs and characters, and so on and so forth, Unicode is so complex, and in general always requires careful processing of code point (or code unit) sequences, that honestly the surrogate encoding doesn’t make that much of a difference. It’s just an additional wrinkle in a sea of wrinkles.
I still find the surrogates different. Bidi, private use, ligatures, ... are script or locale related.
Unicode uses numeric values from 0 to 1112063. You can invent all kinds of methods to encode numbers from 0 to 1112063 (variable length, fixed length, decimal, hexadecimal, anything else). But most ways I can think of to encode these numbers, including variable length ones that would use 8 bit or 16 bit primitives, don't require me to actually reserve some of those to-be-encoded numbers themselves for a special meaning. Yet for UTF-16 they managed to do it. Imagine that all other encodings out there would also want to reserve some Unicode values for their own purpose!
You always have to work with sequences of code units anyway (instead of just single code points), so the individual reasons for that doesn’t make much of a difference. It seems your rejection is more on aesthetic than on practical grounds.
As far as valid Unicode is concerned, you care about the distinction between code points and scalar values, and surrogates are the only difference.
Noncharacters can be represented in any Unicode encoding. Surrogates code points cannot, but can be found in unvalidated UTF-16 (which is most UTF-16).
Dealing with Unicode text semantically requires that you be aware of a great many factors such as those you name, but you don’t need to be aware of those for just storing and transferring Unicode text. But with surrogates, UTF-16 managed to break it for everyone: any part of the system that uses unvalidated UTF-16 can introduce errors that you must care about. Hence surrogates are a special kind of atrocity.
Windows is slowly but surely moving to UTF-8.
https://docs.microsoft.com/en-us/windows/apps/design/globali...
> As of Windows Version 1903 (May 2019 Update), you can [...] use UTF-8 as the process code page.
> Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps.
Of course if you use this, your app will only run on very recent Windows versions. But that's how it goes with OS features. We'll start reaping the benefits 10-20 years from now.
Microsoft is making improvements in their UTF-8 support. Getting rid of the `W` APIs will take forever. Java and JavaScript are even more stuck with UTF-16.
UTF-8 support for filenames would be a great start, to support windows filenames in a multiplatform way in C!
Filenames are tricky. You need to interpret them for display and when interfacing with other systems but fudamentally, they are not Unicode strings on either Windows or Linux but rather sequences of WCHAR or char respectively (with some restrictions). That means that a UTF-8 API can never support the full range of filenames that might be present on a valid Windows filesystem. But you can have eat your cake and have it to by having that API accept WTF-8 [0] which is a superset of UTF-8 that is specifically designed for this kind of interopability.
But what do you care how they file names are stored on disk, as long as you can read directories and traverse paths using UTF-8?
Is UTF-16 part of the JVM or the Java language, or both?
Oh wow. That is really not very much pain, as described.
I have to say, I never thought that the benefit of Haskell having a horrible native string type would be "you can just upgrade strings like any other dependency," which is really kinda slick. You think about how much pain there was for Py2 -> Py3 where one of the big sticking factors was all of the distinctions around strings and encoding and byte arrays... this is comparatively quite nice. Makes me wonder how much of a programming language can be hotswappable.
> Makes me wonder how much of a programming language can be hotswappable.
For a research language that can make a lot of sense, not so much for a language to be used in industry.
The downside is that different libraries will have different string representations, so you can end up being forced to do a lot of conversion if you're using different libraries that have made different choices from each other, or from your own code.
There are at least 5 commonly used string types - String (linked list of Char), ByteString lazy & strict, and Text lazy and strict. The latter two have a good rationale for being different - byte strings are not necessarily text - but, for various reasons, they're often used to represent text anyway.
These five also have corresponding `readFile` functions - see https://www.snoyman.com/blog/2016/12/beware-of-readfile/ . As Snoyman recommends in that post, it's probably best to "Stick with Data.ByteString.readFile for known-small data, use a streaming package (e.g, conduit) if your choice for large data, and handle the character encoding yourself. And apply this to writeFile and other file-related functions as well."
The first comment on that post starts out with "This problem extends well beyond readFile." Having the string handling more standardized at the language level can make life quite a bit simpler for developers.
> String (linked list of Char)
That... sounds awful for performance. Is that a real thing?
Utf8 vs utf16 as the internal representation of the Unicode string type is mostly just an implementation detail.
This is very different from going from python2, which conflated bytes and ascii strings, to python3, which intentionally changed the api to propely distinguish sequences of bytes and strings.
This site saves 26 "statistics" cookies and 99 "marketing" cookies.
Really? Is all that necessary?
Your user agent saves the cookies. If you don't like it, change it.
Ignoring the privacy bit - 125 cookies is quite a bit of per request overhead, especially in http/1.1 where they are not compressed. I would say its poor website design.
Heh, so I actually do this.
An incredible amount of the web just breaks. Twitter, Reuters, Imgur. Like it's one thing if, when I attempt to log in, your log in fails (and usually, logins fail to handle the error & will just loop back to the start, that's at least a start) but a lot of the web will have a flash-of-text and then nothing, & JS has crashed.
I do think cookies get unfair treatment.
They are things that your browser happily rebroadcasts back to the server with no real UI for it outside of the shitty devtool bar made for devs, even after all this outcry about cookies.
It reminds me of the meme of the guy riding a bicycle, throwing a branch into the spokes (rebroadcasting cookies), and then roaring in pain on the ground about how evil websites/advertisers are tracking him with cookies.
That said, what a lame HN thread on a post about Haskell.
Why shift the burden on the user and the user agent? The website is the only one to blame here.
I don't understand how are surrogates in UTF16 a problem solved by UTF8.
From the article it seems that two main improvements were smaller memory footprint due to mostly ASCII data and using better algorithm which resulted in even better performance in UTF16.
I had to deal with surrogates in UTF16 and it is much easier than dealing with variable encoding in UTF8. Naive UTF8 decoder is easy and fast but if one takes time to fully validate each UTF8 code point then it becomes much more difficult and a lot slower...and that is going forward, trying to move backwards in UTF8 is again much harder than in UTF16.
The main disadvantage of UTF16 is memory usage when dealing with ASCII data but if that is the case then just use array of bytes and don't worry about unicode.
I am always extremely doubtful of these types of blogposts that take a well-known algorithm and somehow beat all others (including academia, bioinformatics tools, etc.) with a fancy implementation in <insert cool programming language 2022>
The article is about how they moved an existing (fast) implementation in Haskell in UTF-16 to an even faster implementation in Haskell by switching to UTF-8. This is stated in the first paragraph.
The post they reference, is also very honest: ..., the fastest Haskell implementation of the Aho–Corasick string searching algorithm, which powers string search in Channable.
Basically the blog posts show that if you want to program in Haskell and still optimise, this is how you can do it. I think both posts are great resources and don't overstate their claims.
(author here)
I wrote this article during a short internship at Channable. Not to be apologetic but I think these kind of articles are so prevalent because young or unpopular languages usually have worse documentation than established ones (naturally). I basically wrote down the things I learned during my internship that I found noteworthy.
I was taught Haskell at university and I'm old. Looking at it's wiki page it's a 32 year old language not that much younger than 37 year old C++.