▲ 155 points
back
118 comments
Funny(???) warstory:
1. Back in the days, we were using a Linux NFS server, with NFSv3, and out-of-the-box locale was iso-8859-1 (latin1). Life was good, except for occasional problems with people with strange non-latin1 names, or documents with non-latin1 names etc.
2. At some point, we switch to using UTF-8 by default. Telling users to use convmv to rename their files when they are ready to switch to the new defaults. Most people ignored this, of course, but files with now invalid utf-8 were mostly fine, just with the occasional "?"'s in the names.
3. Switch to NFSv4. Invisible to end users. NFSv4 per se requires that paths are UTF-8 encoded, but in practice the Linux NFS server and client just pass along a bag of bytes, so invalid UTF-8 just worked as fine as it did previously.
4. Switch from a Linux NFS server to a netapp.
5. User complains that files are missing. Initial comparison with the old Linux NFS server, which was still online, shows no problems. Problem occurs only on user workstation, not on admin box which has both the old Linux NFS and netapp directory trees mounted. Investigation on users workstation shows that in some cases lots of files appear to be missing, including ones which plain ASCII names.
- Turns out that the admin box had the netapp mounted with NFSv3, and thus everything appeared Ok there, including the rsync from Linux NFS -> netapp in the first place.
- However, when mounted using NFSv4, netapp follows the spec and does not like non-utf8 paths. Does it report an error then? Hell no, the NFS READDIR (READDIRPLUS?) message reply just stops returning directory entries when it hits the first one with invalid UTF-8. And thus you get a partial directory listing. GAAAH!
- So the solution was to run convmv centrally (from the admin box which had the netapp mounted with NFSv3) for the entire directory tree which had been moved.
Ah yes, had the same fun problem at a customer's facility last week. Moving 350 TB of data from an old DDP storage server to a Linux one. Mounting with CIFS (no other option available), an copying using "cp -a".
The file names look OK after the copy on the Linux machine. However, when exporting the directory through Samba, the Macs Finder doesn't display files with accents in the names (though they appear correctly with "ls", weird...).
So the user copies the files again, using the Finder. Now I have files with exactly the same name (uhhhhh???):
# ls -l Mmo-1. -rw-rw-rw- 1 root root 8417218 6 sept. 2013 Mémo-1.aif -rwxr--r-- 1 test test 8417218 6 sept. 2013 Mémo-1.aif -rw-rw-rw- 1 root root 363175 6 sept. 2013 Mémo-1.m4a -rwxr--r-- 1 test test 363175 6 sept. 2013 Mémo-1.m4a
Yes, it looks like two files have exactly the same name, but actually they're different: one as "é" encoded as 0xCC81, and the other one (the "good one") as 0xC3A9. Why is that? Why does one work with the Finder, and the other doesn't? who knows.
A similar thing happens with Java's file and directory APIs on Linux. IIRC in Java filenames are Strings. If your vm is configured with utf8 as "file.encoding" and you have non-utf8-compliant filenames on your filesystem, those files are completely inaccessible to Java!
Useless non-sense like this is the main reason why i desperately want to move away from software engineering.
> on Windows, paths are fundamentally text
They were back when there were less than 2^16 characters in the Unicode standard. Back then each two-byte word in a filename corresponded exactly with a Unicode code point.
Now there are more than 2^16 but well under 2^32, Windows uses UTF-16 in filenames. That is, Unicode code points above 2^15 are obtained by a pair of special Unicode code points in the range 2^15-2^16 called surrogates; surrogate pairs need to be collapsed into a single code point when decoding the file name. Surrogates are exactly those things that Python uses on Linux to hide bytes that are not valid UTF-8. Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16).
In summary, on Windows, you end up with effectively the same situation as Linux: file names that are supposed to be in one encoding (UTF16) but contain invalid data for that encoding.
Funnily, because of the 2/3 transition, Python became a language obsessed with encoding correctness. Hence the team spent a great deal improving the situation, from version to version during the last 10 years.
For a fantastic read (from Victor Stinner himself) on all the work done and get how twisted this get when you want to be cross plateform and abstract it:
https://vstinner.github.io/python37-new-utf8-mode.html
Windows and Linux FS encoding, of course, are at the center of the challenge.
It's also the reason why we now have a __fspath__ protocol that allows any object to be converted to a file system path instead of having pathlib.Path inheriting from string.
> Here's the problem: it is possible to have unmatched surrogates in a file name (or in other places that Windows accepts UTF-16).
NTFS (and Windows as a whole) does not use UTF-16, it uses UCS-2. It is a subtle difference, but surrogate pairs didn't exist in UCS-2.
> file names that are supposed to be in one encoding (UTF16) but contain invalid data for that encoding
You just described my music collection, aggregated over two decades and haphazard successive migrations. I've given up salvaging the corrupted names in an automated manner...
Rust `std::path` [1] has two representations under the hood for Windows (UTF-16 plus lone surrogates) and non-Windows (bytes) for the exactly same reason. Paths are not strings nor texts.
Emphasis on the "plus lone surrogates" part. Like on Unix, Windows does not require a path to be valid Unicode.
That is, on Windows, paths are fundamentally sequences of 16-bit words, just like on Unix paths are fundamentally sequences of 8-bit bytes. On neither system are paths fundamentally text.
They're not strings nor texts right up until the point you need to display them to users.
Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites with UX from the 90s.
Has anyone tried this? Is it possible with FUSE? I would love to hear from people who know about this stuff - what are the obstacles? Or do you think FSs are fine the way they are?
As far as I can tell, file systems are inherently broken by design; this isn't an implementation issue. For example, the notion of finding a file by its path is just riddled with race conditions. If you create files /a/b/c and /a/b/d, are c and d necessarily in the same directory? Not really, because someone could have moved around the parent directories in between. But we conveniently assume paths stay the same... except, of course, when they change. Now try actually formalizing what exactly that means from a global (multi-program) perspective!
ZFS handles internationalization about as well as could be hoped for. You can forbid non-UTF-8 strings (note that ZFS doesn't know if some string that is valid UTF-8 is actually encoded in UTF-8 -- it might not be), and ZFS does form-insensitive directory lookups, so if you copy some normalized-to-NFD files from OS X, it will work out fine.
Making filesystems codeset-aware is not worth the trouble. It's best instead to just use UTF-8 locales everywhere. If you need to deal with other codesets, convert as needed, but don't use non-UTF-8 locales.
On ZFS with a fast ZIL fsync()/sync() function a lot like write barriers, which is what we really need. Actually, what we really need is for all filesystem operations to be available with async system calls, write barriers included.
What is sane path handling?
Alone on windows, the maximum path length is 260 characters, except when you use extended-length paths which have a 4 character prefix and a maximum length of 32,767 characters. A sane API for reading files probably converts your paths to extended-length paths. But if you do the same thing for writing files your users start calling you insane again, because most of Windows (including Windows Explorer) can't open extended-length paths. So you would be creating files that only select software can even open, and which the user can't browse without third party software.
(An easier to ignore fun fact is that NTSF and Windows also support case sensitive names if you set the right flags in the file APIs. But nobody uses that, so it's probably save to ignore that (until somebody mounts EXT3 partitions in windows...))
ZFS has a `utf8only` property that constrains filenames to be UTF-8 only... :)
The simple thing to go about this is to make assumptions that do not always hold technically. And to be ok when these assumptions break - in this case you simply can't deliver your promises anymore.
Another way to put this is "shit in, shit out".
So: When receiving a filename that is not UTF-8, one could just emit a warning and ignore the file. That's what I would do if I wrote a music tagger, at least.
When someone else modifies a directory tree at the same time, we're bound to run into problems. That's just how it is. Actually there are synchronization facilities (e.g. flock()) for some of these types of problems. But these are seldom used because they lead to other problems.
I think they knew all that in the 70s, and just chose to be pragmatic about it.
I think Java does that. Thanks to its built-in libraries, you can write code that works on multiplie platforms with different file systems. However, you still need to know yourself which characters are legal in which system (for example, ‘:’ is not a legal character in file names on windows, but it is on OS X.
What are you trying to achieve? If you want to store data for your own application, you'd use something higher-level than a filesystem (e.g. a database); those exist and offer sane semantics. The only reason to interact with the OS-level filesystem is to use it to communicate with other programs, in which case the insanity is necessary: if you want to e.g. read files created by other programs, you have to be prepared to deal with malformed names, because other programs will create files with malformed names.
sqlite?
The uncomfortable generic answer to this: https://xkcd.com/927/
I work on another music file management system, my personal special hell is playlist files. An m3u playlist file is just a new-line separated list of file paths, which can be relative or absolute, and potentially encoded in whatever locale is set on the users computer. Some fun issues:
* Windows and Mac filesystems are generally case-insensitive, so some users will have the file names in the playlist file in one case and the actual file names on disk in another format * Sometimes file paths cross between two different filesystems, because one is mounted in the other with a USB drive or over CIFS or similar. Sometimes these two different filesystems have different case sensitivities * There's no way to know how the playlist file was encoded * HFS+ normalizes file paths to Unicode NFD, but there's no guarantee that the paths in a playlist file will be normalized. Also, sometimes users generate an m3u file on a Windows system and expect it to just work on a Mac. Also, the filesystem nesting problem with network or USB mounts can happen this way too.
Sounds a lot like my life a couple of years ago (and intermittently since). I don't get the bug reports any more because I think customer service has learned that file name problems can be fixed by renaming the files. Not fun for the user, but a sure fix.
Ya know what kind of file names work virtually everywhere? ASCII ones.
It misses a even more complex, I'd say insane, encoding problem: on HFS+ (or even APFS now?) filenames are unicode normalized.
HFS+'s use of NFD made even more insane by the fact that OS X's input modes prefer to produce NFC anyways. And besides, so do other OSes' input modes, so in any heterogeneous system this is a nightmare.
This is why ZFS does form-insensitive directory lookups (and hashing)[0] rather than normalize-on-CREATE! I'm so glad ZFS got it right, and can stand as a model for all. (I implemented none of that functionality, though I code-reviewed some of it, specifically the u8_* functions in Solaris/Illumos, but I remember it took some doing to convince others that this was the correct approach.)
[0] https://cryptonector.com/2006/12/filesystem-i18n/ [1] https://cryptonector.com/2010/04/on-unicode-normalization-or...
Not only are they normalized unicode, they're normalized decomposed, and not only that, but slightly non-standard (does not conform to standard Unicode "NFD" form). (Or at least, this was the case with HFS. I haven't followed APFS closely enough to say for that.)
"NTFS allows any sequence of 16-bit values for name encoding (file names, stream names, index names, etc.) except 0x0000. This means UTF-16 code units are supported, but the file system does not check whether a sequence is valid UTF-16 (it allows any sequence of short values, not restricted to those in the Unicode standard). "
- from wikipedia NTFS page [1]
So if you assume that NTFS filename is valid UTF-16 and convert it to UTF-8 there might be a problem. Basically they can be any sequence of 16-bit values.
[1] https://en.wikipedia.org/wiki/NTFSThere was a time when some of our customers had lots of problems with gigantic files on their drives that was impossible to delete with windows explorer. I would come home to them and help them delete the files with the command line using filename*.ext to catch them. My guess was that the filename had some protected characters that windows explorer didn't allow. Don't remember how they ended up with the files but most likely some download program and someone having a laugh :-)
Doesn't it (or Windows) also disallow the path component separator character(s) ('/' and '\')?
Unix and alike disallow NULs and /, for obvious reasons.
The built-in file system libraries for many languages are total footguns. Using strings as paths is a great example. I’ve had a few recent bugs around case sensitive vs case insensitive file systems because a lot of code assumes that when pathA != pathB then it must be dealing with two different resources. Not to mention the classic “doesn’t work on windows” problem: newPath = pathA + “/“ + pathB
AFAIK, Windows understands / as a directory separator.
That's why it's good to have a cross-platform path datatype/literal baked into the language.
For eg. in Rebol/Red - http://www.rebol.com/r3/docs/datatypes/file.html
Most decent stdlibs have a path.join or an path.separator you can use instead. Just that people don't bother and hardcode...
tangentially: Tool I use to test my stuff when I expect it to handle all valid filenames:
I found a similar problem with my backups (Ugh. :-)
A year ago I went on a trip and some combination of the humidity, the travel, and the 6 year old Thinkpad resulted in my laptop not booting.
I had been experimenting with Borg to backup the system, and so I tried using Borg to restore the latest copy onto the new laptop. Turns out that I have a bunch of files on my laptop that have names with weird characters in them: rips of my CD collection. I couldn't find any combination of settings and environment and locale that would allow borg to recover or skip these files and recover everything else.
Now, I had 2-3 other copies of the data (my pre-borg backups, the original SSD which was still readable, a few other rsync copies), so it wasn't a big deal.
But, as always, test your recoveries!
I think you could have mounted your borg backup as a FUSE filesystem, and then used rsync to restore your files.
now sing along with me children: /none of this matters to me/ because I live in eight dot three/
I wrote an essay a while ago about fixing Unix/Linux filenames here: https://www.dwheeler.com/essays/fixing-unix-linux-filenames....
This is a big disconnect between "what most users expect" and "what systems actually do". Usually generally expect that filenames are sequences of characters - and today almost everyone expects that they must be in UTF-8 on a Unix-like system. That is not, of course, what most systems actually do.
I wrote about this eons ago: https://cryptonector.com/2006/12/filesystem-i18n/ and https://cryptonector.com/2010/04/on-unicode-normalization-or... -- these might still be available on https://blogs.oracle.com/, though these are from my days at Sun.
TL;DR, basically, the lack of ability to tag strings in the system call API with codesets means that UTF-8 is the only plausible answer, and the ends (C library system call stubs, filesystems) have to apply whatever codeset conversions. But there's practically zero chance of C library system call stubs (and related functions) performing codeset conversions (can you imagine readdir(3) doing it?), which means that the only reasonable answer is to use UTF-8 locales and be done.
Even shorter: just use UTF-8 locales and be done.
You cannot use UTF-8 locales on Windows though.
In which the author takes a long and winding path to what most of us already know, "paths are fundamentally bytes".
… but also, sort-of, but not really, text: they get displayed to the user, they get input from the user, and they get emitted in logs, messages, etc. All as text. And that rub between where they're bytes but they should have been text, that's the problem and that's the complexity.
Just dealing with file extensions is enough of a head spin. We stopped trying to differentiate between .xls, .xlsx, *.xlst... etc. to show an Excel icon for a file uploaded to our SaaS and just went with a generic file icon in the end.
IBM's backup software TSM/Spectrum Protect messes this up as well.
If the machine has a UTF-8 encoding (like, say, every modern system), it will try to treat filenames as valid UTF-8 strings and fail to back up files which don't fulfill that assumption. The "solution" is to run the TSM software with a single-byte locale like en_US.
I've seen a number of shops that were silently missing files from backup from old systems because of this problem.
And me thinking that I would see mainframe and other non-POSIX systems, only to see the usual Linux/Windows dichotomy.
And a slow black snake slithers out of your computer's USB ports, made of billowed smoke and unrealized dreams. It sticks its tongue out; "typesssss," it whispers, "typessssss."
A way to stay sane seems to use Python3 pathlib and drop Python2 development.