▲ 484 points
back
197 comments
The title is "Why Google Pixel lags 10x more than Moto Z" but the content is just a microbenchmark of fsync() without anything resembling a measurement of UI lag or that the 10x slowdown in fsync() actually means a 10x (or any) increase in said lag.
The author's previous blogpost (http://taras.glek.net/post/Laggy-phones-and-misleading-bench...) contends that 4K writes are a good proxy for phone lag, but has no evidence or measurements either, just the author's contention.
It could all well be true! And it's great the author's dug up some concrete areas the pixel team could potentially improve at (dumb question: couldn't the pixel be updated to use most of these FS tweaks? could an in-use pixel FS be converted to use f2fs?). But I don't think the author has done a good job demonstrating an actual relation to UI lag or anything to do with the phone's perceived performance.
disclaimer: googler, but I do iOS things.
So much clickbait :/ Under no condition the Pixel lags "10x as much as Moto" when actually using it. It's a single benchmark of an FS performance for applications that do I/O on main thread. Something that is actively discouraged by things like StrictMode checks.
> but has no evidence or measurements either, just the author's contention
He both produces some measurements he has made, and cites an academic paper that corroborates. It's not certain proof but it's a long way from 'no evidence'.
> couldn't the pixel be updated to use most of these FS tweaks
That would require reformatting, so probably not.
fsync benchmarks are stupid.
(neither a Pixel fan or hater here)
> The key option is nobarrier. This effectively makes fsync() a no-op and explains most of the difference in performance.
In other words, because the Moto Z cheats.
I suppose it's understandable... nobody should ever be blocking GUI animations on fsync, much less two of them in a row, but here we are.
Not exactly a cheat. A calculated risk. A friend of mine were having the storage discussion on phones and realized that phones never really experience a 'power outage' in normal use. People use their phones, they get low in power, and then they plug them in. Or the battery gets so low the phone shuts itself off in an orderly manner.
In the 'computer is a phone' perspective there is no sudden power loss unless the user yanks the battery out of the back. That is presumably a rare occurrence for your typical user. As a result, the big risk for phone file systms is that the phone crashes, but nearly every phone I've seen preserves memory contents as long as power is applied, so even a crashed phone, on reboot can pick apart the previous bits in memory and reconstruct what was going on before it crashed.
Given that fairly unique to phone criteria I could see nobarrier as a legit option.
fsync() has never been as reliable as the name implies, since flushing the disk cache kills performance system-wide. Fuzziness on what fsync does has... side effects: https://danluu.com/file-consistency/ http://blog.httrack.com/blog/2013/11/15/everything-you-alway...
Filesystems optimized for flash, and for battery-backed systems in general, (laptops, phones) have some history: https://en.wikipedia.org/wiki/Flash_file_system
> I suppose it's understandable... nobody should ever be blocking GUI animations on fsync, much less two of them in a row, but here we are.
this is the thing that confused me.. why are apps doing disk i/o on the GUI thread?
So from reading the XFS FAQ linked in the article(http://xfs.org/index.php/XFS_FAQ#Write_barrier_support.), does this mean that Moto Z never flushes the cache to disk?
Wouldn't the cache HAVE to be flushed as it fills up at some point to maintain coherence.
>In other words, because the Moto Z cheats.
If you're not cheating, you're not doing it right.
I wouldn't really say Moto Z cheats so much as that Google designed the entire software ecosystem of Android without ever talking to a hardware engineer.
They simply do not care about optimizing the performance of the phone running their software. Otherwise they never would have chosen a garbage-collected language like Java for the platform in the first place (I understand they've ameliorated this concern recently). Or used ext4 like here.
This articles conflates a lot of things but it also has the priorities somewhat wrong.
1) fsync cost. Yes, fsyncs are dangerously slow in any Android app. (SQLite for example is a common culprit. Shared Prefs are another). HOWEVER, it's possible that flushes cause reads to be queued behind them (either in the kernel or on the device itself) which is even worse because
2) Random read cost is super super important. Android mmap's literally everything and demand paging is particularly common AND horrendous as a workflow. To add insult to injury, Android does not madvise the byte code or the resources as MADV_RANDOM, so read-ahead (or read-around) kicks in and you end up paging in 16KB-32KB where you only wanted 4KB.
Also, history has shown custom flash-based file system on Android to be a world of pain. yaffs, jffs have some pretty atrocious bugs/quirks. I'd much rather see the world unify on common file systems, optimized for flash-like storage, rather than OEMs shipping their own in-house broken file "systems" (I'm looking at you, Samsung).
Why can't f2fs be that common file system?
While it no doubt impacts performance in some cases, MADV_RANDOM probably isn't the correct choice for a lot of circumstances (the classloader tends to do a linear scan of JAR files, for example).
> Android mmap's literally everything
Why is that so?
This makes very little sense. The author shows a fsync() benchmark comparing an actualy fsync (Pixel on ext4 without nobarrier) with a nobarrier (no-op fsync) alternative. The only thing this benchmark shows is that a no-op is faster than the real thing.
The title of the article is "Why [the] Google Pixel lags 10x more than [the] Moto Z." The author showed that the Moto Z is using a no-op fsync while the Pixel is doing the real thing. I think the author appropriately explained and answered the title prompt.
Saying "no-op is faster than fsync" is just not as click-baity a title.
There is a good question though... do you really need fsync on Android? If you don't, why are you calling it?
"try to avoid buying devices that will slow down to point of being unusable as NAND wears out (ala Nexus 7, Nexus 6)"
Has anyone experienced this issue with their Nexus 6? My phone is more than 2 years old and I have no noticeable slowdown. The pixel might have the slower storage option but it has no effect on usability. From what I have read its UI performance is the best of any Android phone yet.
"The Pixels are fast — noticeably faster than Samsung's Galaxy S7. On performance alone, these are easily the best Android phones you can buy." http://www.theverge.com/2016/10/18/13304090/google-pixel-pho...
This slowly happened to my original nexus 7 tablet. Sad. Eventually its just too slow to use. Ten minutes or longer to bootup, etc. Of course its a gamble, will people crack the screen or kill the battery before the memory is destroyed, if so who cares ship it.
You can play games for awhile stripping stuff out of memory so it can live in the fast area but eventually its speed drops to approximately zero even if all you have installed is the kindle app at which point I tossed it.
There is a lot of cargo cult about there about wiping the cache over and over supposedly helps in the sense that the owner feels they're doing something, although all they're doing is wasting the last few R/W cycles left.
Actually, yes. At work we have a Nexus 6 testing device. It started off strong but a few months ago it started lagging badly on videos. It even struggles with gif length mp4s. The problem is notable because it is the only phone with this problem. Even our old Galaxy S4 performs better.
My nexus 7 became utterly unusable over time, even after a complete factory reset.
The only way I was able to make it remotely usable was to install an old version of cyanogen mod and install only bare minimum apps.
I certainly have - I just replaced my Nexus 6 with a Moto G4, which somehow seems faster with the exact same set of apps and prefs.
As an aside, having a phone that costs $150 is sort of amazing. It's removes the fetishism of "oh god, what if this breaks?" that I had with my iPhones and Nexii. It's a new and interesting feature that isn't discussed enough.
I have to wipe out my nexus 6 every couple of months for it to be bearable. My last experiment took me to franco.Kernel and that have proven itself good during last couple of months, but I'll keep watching.
I perked up when I read this part of the article. I've had my Nexus 6 for nearly two full years and it's not fun to use. Very laggy and I missed many moments because the camera wouldn't load. I ended up getting a Pixel and I'm worried that in a year I'll see this same phenomenon again.
From Tim Murray, performance engineer on Pixel:
>that fsync blog post floating around is pretty much bogus. also nobody should use nobarrier, it's not safe at all
I don't think this could be interpreted as a visual lag. fsync() is usually used by databases, not UI libraries. So the title is misleading.
My chinese Android phone has a more annoying hardware lag - a delay between touching the screen and touch event processing is over 100 ms. Any drum app is unusable. And if you try to scroll something up and down fast it is easy to see how the content on the screen lags behind finger movements.
Offtopic: I love my old Moto G, but I've been reluctant to buy any new phones from Motorola since its purchase by Lenovo, due to Lenovo's history of shipping computers with rootkits and malware. Should I suspect that Motorola phones are compromised, or am I just being unreasonably paranoid?
"They drove development of the filesystem specifically by Twitter/FB/etc workloads captured from the phone."
Why would Twitter or Facebook apps need to write much to persistent storage? They don't do so in a browser.
Other comments are querying the relevance of fsync() as a 'lag' benchmark, but I want to query whether fsync() is even meaningful at multiple calls per second.
I know fsync() ensures data gets written to disk, but why does anyone care that it can happen so often? When a device crashes, some data (prior to the sync) may be lost, but do we really need multiple checkpoints per second to ensure only sub-second data loss?
I'd be content with a couple of minutes worth of loss even on my main PC, with its lack of battery backup. To enforce rapid syncs on a phone seems utterly pointless.
Keep the syncs for meaningful checkpoints, like buying something in an app or marking a message as sent. Multiple fsync() calls per second are a total waste.
Any real proof that this actually causes user visible issues at all?
I don't see the hard link between GUI smoothness and io. You can have perfectly smooth 12000fps rendering and crappy IO, everything will just be "loading...".
My old 4S runs annoyingly slow. According to the article,the NAND, as it ages will slow down the phone, so will replacing my storage make my iPhone fast again?
The fundamental problem sounds like it's that SQLite insists on not running in its own thread. This, coupled with the fact that Linux has no way to issue a true (i.e., non-blocking) write barrier, means that there is no way to implement write barriers in user space.
Whereas, if SQLite did issue I/O from a separate thread, one could easily implement an "async commit" function which guaranteed consistency and ordering but not necessarily durability (i.e. a write barrier). This would suffice I suspect for 95% of usage in applications: users will probably be OK if their phone loses the last few seconds of user input before an OS crash, so long as everything else is left intact.
EDIT: In fact, Postgresql has an option to permit exactly this behavior: https://www.postgresql.org/docs/9.6/static/wal-async-commit.... This is possible in Postgresql because, unlike SQLite, I/O does not run in the client thread.
If f2fs is so superior to ext4 then why doesn't its creator, Samsung, use it on their phones?
I hadn't heard about f2fs before. Sounds useful even on a notebook with SSD.
I always wondered why my wife's Moto X is still buttery smooth after 3 years!! I love that phone. I have a Pixel but she still has her Moto X and it is really great at being smooth, plus the wave to wake and other gestures are really unmatched by the Pixel.
Love this write up/research! Hopefully it will teach the Pixel team a few things, or maybe they already knew but will now have the ammo to take to Product and change things!!
I suspect it is not the fsync directly that leads to lag. But that when a fsync comes through, Linux stops doing anything else for the duration.
If you're calling fsync on your painting thread, you're going to have trouble hitting 60FPS no matter what. What a clickbaity post.
So this might be the reason why some mobile devices (like my ASUS TF700T tablet) degrade in performance so badly over time. Interesting.
Is there an Android sample app that show how to do I/O properly (off the main thread, and so on)?
I know there's some kind of system to prevent network I/O on the main thread... w... why... I dare to ask the obvious, w-wwhy isn't there a warning for simple disk I/O too?
I wonder if this is why my Sony Xperia Z5 is so sluggish sometimes, despite having a Snapdragon 810 8-core CPU...
What does it matter?
> ... noatime,nodiratime, ...
noatime implies nodiratime. I shrug off when I see newbies copy pasting this to their /etc/fstab but this is in a mainstream Android device??
Nexus 5X running rooted AOSP 7.1.1
/storage/emulated fuse /dev/block/dm-0 /data ext4 rw,seclabel,nosuid,nodev,noatime,noauto_da_alloc,errors=panic,data=ordered,inode_readahead_blks=8 (no nobarrier mount option)
In theory Google should be able to easily change the /data ext4 mount option, why didn't Google?
Kinda makes me regret getting a Pixel to replace my Moto Z. Except the reception was terrible on the Z. Oh well.
How is it that no one has made a worthy successor to the OG Turbo. That was a monster of a phone across the board.