▲ 455 points
back
466 comments
I have a bunch of privacy-enhancing addons installed, which have now all been disabled. If I hadn't read HN this morning, I wouldn't even have known why. Until now, I had no idea that it was even possible to remotely disable my addons.
And now Mozilla are saying that the "fix" is to allow them to install & run "studies" on my machine? What are they smoking? I'm having a hard time trusting a company that randomly & remotely disabled all my addons, regardless of the cause.
This is not entirely accurate. Nothing was done remotely to disable the add-ons. It happened locally. A certificate that's on your machine as part of the Firefox install expired. When that happened, add-ons that were signed via a cert chain that included the expired one started appearing to be invalidly signed. And that's why it requires an update to completely fix. That part is remote, because they need to push a new valid certificate to you to replace the old one.
I do think that the UX should ideally be a bit more graceful; one of my add-ons is Multi-account Containers and its being disabled suddenly caused the window I was actively browsing in to just close, among other side effects.
But that kind of UX polish for what should be an exceptional case is obviously not going to be super-high priority, unfortunately.
I enjoy a nice cup of outrage in the morning just like the next guy, but this one is really weak and lacks that fresh taste of evil conspiracy that I really crave.
You use a browser that has remote update capability, which allows them to install and run new software on your machine all the time. There is a whole separate section of the Preferences that says "Privacy" in large print that has a section that clearly identifies the Studies feature and lets you turn it off. And you use a browser that lets you install privacy-enhancing add-ons in the first place, and in fact which invented the whole concept of add-ons. When the browser discovered that it couldn't verify the add-on integrity with a valid cert, it did what it's supposed to do, it disabled them to protect you from someone backdooring these add-ons.
Someone at Mozilla fucked up, and they're trying in good faith to fix it. I don't know what else people are expecting them to do, putting on sackcloth and ashes won't resolve the problem.
Your addons have not been remotely disabled. They were marked as trustworthy by a certificate that expired and thus are no longer considered trustworthy. The effect is similar, the mechanism is different. You could also enable loading of unsigned extensions, that would “fix” the issue, too.
No one remotely disabled anything. There's a certificate deployed with Firefox. The certificate Firefox used to check addons was only valid till yesterday. So, when the browser started next time it couldn't validate the addons and disabled them. That all happened locally.
> And now Mozilla are saying that the "fix" is to allow them to install & run "studies" on my machine? What are they smoking?
Can you elaborate what's your concern with "studies"? By installing Firefox that updates automatically, the user is already giving control of the software and letting Mozilla decide what's the best. How is modifying software logic using studies different than modifying logic by updating the binary?
They have not remotely disabled addons. The certificate expired and the addons did the correct thing when connection couldn’t be established. Nobody triggered a switch to disable addons.
Due to them easily being able to push code without much hastle using Studies, I think this is an elegant-ish solution to a problem that shouldn't even have happened (expired certs are something that's entirely avoidable), but errors happen.
Eorum est humanum.
Not initiated by user != remotely initiated. And agreed no the fix side, they should have posted how to do it yourself as well.
The problem is that Firefox does not have sufficient built in privacy settings by default. Users shouldn't have to crawl the internet for lists of recommended addons, then have to trust such a variety of authors, to have basic privacy. Like I said elsewhere, I'm using Brave because of this.
At the minimum they should add a testsuite that runs at least a month into the future to catch these kinds of things.
There was a similar issue[0] a few years ago that was only caught a month in advance.
Even better would be to set things up to only do a verify on install instead on every startup.
IMHO it seems problematic, that they can remotely push code changes, including replacement of trusted certificate, and bypass package managers.
I don't expect software to (significantly?) change during runtime, outside of what was packaged, signed, distributed and installed as part of apt/yum/pacman/etc.
I understand (not that I like or agree with) that some apps are just embedded web browsers, and load everything externally, and that Firefox extensions are in the end just some JS/CSS/HTML loaded outside of system's package manager. However, extensions have limited API they can interact with, and you need to allow permissions for each extension. Having Mozilla owned extension, that can modify core functionality, seems a bit scary.
I wonder if there is someone out there in the middle of the ocean with a browser extension based communication and navagation system which is dead in the water?
It sounds to me that the real headline here is that every copy of firefox out there was timebombed and we only noticed because someone forgot to elongate the fuse.
I hate to say all these things because I use Firefox all the time, but...the communication around the add-ons issue has been poorly handled by Mozilla. I only learned of the problem by visiting HN. But what of the thousands of other users who don't visit HN?
If you visit the Mozilla homepage, there is nothing to acknowledge the problem (at least at the time of writing this message). Let's try the Support page. Where is it? Scroll down to the bottom of the lengthy Mozilla homepage to the page footer to find the link. (How many visitors will make it to the bottom?)
When you click through to the Support page, an easy-to-miss banner in tiny text appears at the top of the page that mentions the problem - screenshot here: https://imgur.com/a/TAHZSWa
Additionally, when the add-ons are disabled, Firefox misleading says: "These extensions do not meet current Firefox standards so they have been deactivated". This is probably a generic message but it's also an example when a generic message is misleading.
Finally, poorly-named settings like "Normandy" and "studies" that give no hint of their meaning only adds to the confusion.
I'm interested in the general writeup what went wrong that they missed this certificate expiring. That's a structural problem.
Also why it took 6 hrs to assign P1 to the bug
I know Firefox isn't being malicious, but ugh, this seems like the worst possible PR move for this, optics wise. "Hey so uh, we accidentally broke your browser, so you need to opt-in to becoming a guinney pig. But don't worry! You probably were already opted in anyway and just didn't realize it! Also it might take six hours to work."
An article that mentions a timeframe of “the next few hours”, but doesn't have any timestamp besides a date without a timezone.
Granted I'm using Nightly (and previously disabled extension signing in about:config), but now all my themes are disabled, even the default one apparently, though that is what it is using. Cannot be re-enabled. When do I get my dark theme back?
Also...my default search engine is now Amazon.com?? WTF is going on.
EDIT: Also my only search engine. Heck of a job Mozilla.
For me (repository firefox on ArchLinux), the temporary fix was setting devtools.chrome.enabled = true in about:config, and running this small JavaScript snippet in Chrome DevTools (Ctrl+Shift+J):
https://wiki.archlinux.org/index.php/Firefox#Firefox_disable...
AFAIK, this will enable all the disabled add-ons until the next check, which is in 24 hours. This will be hopefully enough time for Mozilla to release a stable channel update, instead of the "Studies hack".
At least for me, fiddling with the Studies settings had no effect; the about:studies page remained empty regardless of what I did. I've also seen multiple reports from people who got the Studies hack working that the fix actually failed to address the issue properly.
Very curious how the decision to use the Studies program happened. Why not just roll the version early and include the fix in the new version - isn't Firefox an evergreen browser now? Maybe there is extra bureaucracy to roll a new version, or the hotfixers didn't have permission to do it. Either of which I can understand, being that they made the fix late on a Friday night - so huge kudos to those who worked hard to get this fix started. Seems like a good case study for lessons learned here, I'm eagerly anticipating the postmortem and follow-ups.
Certificate expirations show up on causes of outage lists so frequently, yet little has been done to address the underlying issues of how PKI works to address it. The core issue of time limits on trust and no specs and requirements on how to handle the most common case of saying “yes, this is still trusted” is oddly absent.
Could we have for example a publicly verifiable ledger that can be used to verify a cert chain with not only a defined workflow to answer if a cert is still trusted but also a requirement for the workflow to be fully implemented? Seems quite doable, vs sort of hacks of auto-renew which are hit and miss depending on the CA.
In other words, when do we fix the sport rather than the players here?
Interesting. Sadly, I imagine many users will have studies disabled since the Mr. Robot incident. I've re-enabled it but there does not appear to be a way to force it to check for updates. Guess it will show up in the next 6 hours.
The most frustrating part of this for me is there is no (relatively) easy way to override this behavior. Its fine to disable the addons, but please allow me to "understand the risks" and continue against Firefox's recommendations.
The feeling of no control over my web browser was why I left Chrome in the first place.
On Android I get this:
>We rolled out a hotfix that re-enables affected add-ons. The fix will be automatically applied in the background within the next few hours. For more details, please check out the update at https://support.mozilla.org/en-US/kb/add-ons-failing-install...
Which is like "we did something we shouldn't have causing unauthorised changes to your computer, so we're going to make unauthorised changes to fix it".
Quite telling is that this is supposed to protect us from other developers. On the add-on screen "Enable" is greyed out, there's no "Enable even though Mozilla doesn't like it".
The UX is just like the "fuck you this computer isn't yours it belongs to Microsoft and we'll do what we like with it" that I thought I'd left in the past decades ago.
It's not your computer Mozilla, you fucked it up, you don't get to mess around with it without asking the owner.
My understanding is that this is literally illegal in the UK.
Mozilla barely had any trust left to burn IMO but they sure went all out.
Can we take a moment and consider the side effects?
This is a once in a lifetime chance for Google & Co. to get a glimpse of all those sly fuckers hiding behind adblockers.
This effectively uncloaked a very specific subset of Internet users and exposed them to the very companies that they've been actively trying to avoid. Not just those who avoid Chrome, but those who take extra steps to explicitly evade the tracking.
Surely Mozilla, the privacy advocate, must understand the impact of this fuck up, and yet the offered "fix" doesn't even mention a one-click .xpi install, but rather asks to enable a mechanism that, if left enabled, will grant unnecessary control to Mozilla over people installs.
This ain't right.
Mozilla decided to make signing mandatory, then screwed up and now they're trying to fix it by making use of a feature that basically allows them to remotely execute code on all their users silently?
I checked Mozilla's main site again, and it still has this ironic statement in its description tag (it's been there for many years):
https://www.mozilla.org/en-US/firefox/new/
Firefox is created by a global non- profit dedicated to putting individuals in control online.
...I guess it's more dedicated to putting Mozilla in control now. Something about this whole incident brings up a point that just feels very wrong to me --- it's not a Google or Microsoft, but the fact that Mozilla also seems to have this large amount of control over its users is unsettling.
I'm not gonna bother with 'studies' or manual workaround - I'm just going to wait for an update.
In the meantime I'm enjoying trying out Vivaldi[1] - really reminds me of opera 3/4, that I loved.
Instead of enabling studies just click on this link. It installs that specific "study" (hotfix) without installing anything else.
https://storage.googleapis.com/moz-fx-normandy-prod-addons/e...
This one will be emotional as this destroyed some of my today's work.
F you Mozilla. I lost all my tabs opened in other containers. The containers don't work too, so I cannot reopen them.
This bug has been known for 3 years, and you did nothing to fix it. You get so much money, and what you do is basically provide a pathetic software (thunderbird) and a nice browser (which you just stopped from working) and you show me banners asking for more money.
You should be ashamed. 3 years. And no, I'm not going to listen things like "this is open source, you are free to fix it". I will just go and switch to another browser. I need a browser which works, not a one which suddenly decides that my stuff should be broken because all the developers and managers have been ignoring a critical issue for a couple of years.
:( I know, this will be flagged, and removed. I don't care, I just need to get all my tabs in containers back. I have never thought that a browser can just close my tabs because a certificate expired.
This is why you always have one of your employees keep their clock 30 days in the future.
Preferably someone who doesn't go to meetings and installd updates.
Another workaround if you don't want to enable "studies" is to manually re-load the add-ons in Debug Mode. I don't know the full consequences of this, but Firefox seems to be behaving normally having done it.
Go to about:debugging from the address bar. Right at the top is a button to "Load Temporary Add-on", with a checkbox "Enable add-on debugging". (On a Mac, the add-ons are in ~/Library/Application Support/Firefox/Profiles/«ID».default/extensions (assuming that you have only a single profile).) They should stay enabled until Firefox relaunches.
From a UX point of view I don’t know why tools don’t have these two features:
1. “Warning, a critical method for verifying authenticity is set to expire in X days. Please visit <Y> to update now.”
2. “A critical verification certificate has expired; while you should immediately go to <Z> to obtain an update, you may defer authentication for up to 5 more days.”
...or in other words, why can’t tools cut us some slack on either side of a deadline? Security for most things is not going to fall apart just by giving people a little room to deal with issues on their own schedule.
How long until heads roll?
There's something really wrong with the organization.
And I thought it was only their marketing/pr that was bad.
> We can't afford to lose Mozilla and Firefox.
Sadly, this removed my settings for multi-account containers extension :(
(disclosure: I am a Mozilla employee but not commenting in any official capacity)
"Give me control over what code I run on my computer" (meaning "provide a switch to disable the requirement that extensions be signed") keeps coming up over and over. And perhaps it hasn't been clearly stated but the problem is this: if there's a switch that a user can flip, the browser has to record the state of that switch somewhere (presumably on disk). If such a switch becomes available, we'll quickly be flooded with malware that flips that switch without users' consent. At that point, there's no way to tell the difference between savvy users making an informed choice to enable unsigned extensions and malware doing it behind their backs. The browser can do various things to obscure the way that setting is stored, but ultimately any method the browser uses to read and write the state of that switch is something that other software can easily mimic.
This is not a theoretical concern, a modern web browser target is an irresistible target for all sorts of get-rich-quick scammers -- if you don't experience this day-to-day its due in no small part to the fact that browser vendors among others are constantly working to keep the bad guys at bay. But make no mistake: the bad guys are out there and they quickly find and exploit any opportunities that are available to them.
So as to the problem of how to let users disable signing but ensure that they have made a conscious decision to do so, there is a stark tradeoff here: giving the most savvy users that switch necessarily makes other users less safe. The solution that Firefox has opted for here is to handle this tradeoff differently on different channels. The release channel (aka the stable channel, or the thing you get by default when you download Firefox) is intended for a very wide audience, and so it handles this tradeoff by favoring safety for all users regardless of their level of technical knowledge. The developer edition and nightly channels are intended for more technically savvy users and they handle this tradeoff differently; specifically they do provide a switch for disabling extension signing.
If there are other (practical and effective) ways to solve this problem of determining true user intent, I (and I'm sure many many others) would be very interested in hearing about them. In the mean time, using the mass-market versus developer-focused channels as a signal for users' preferences on the risk-configurability continuum seems like a reasonable way to handle this.
So the studies update has hilariously enabled my essential legacy extensions while leaving my more modern WebExtensions still disabled. Way to go, Mozilla.
From ghacks (comment section): https://www.ghacks.net/2019/05/04/your-firefox-extensions-ar...
This should allow the extensions to work until the next check (Verified locally):
1) Shut down Firefox
2) Open extensions.json (located by about:profile -> Root Directory)
3) Replace all instances of “appDisabled”:false to “appDisabled”:true
4) Replace all instances of “signedState”:-1 to “signedState”:2
5) Save and close extensions.json
6) Start Firefox
7) Close Firefox
8) Open extensions.json
9) Replace all instances of “appDisabled”:true to “appDisabled”:false
10) Start Firefox
11) Disable and re-enable all extensions in about:addons
No Firefox Studies on Android. Maybe they'll release a whole new version.
The study pushed, but all my installed plugins are listed as "unsupported" and says "<plugin> could not be verified for use in Firefox and has been disabled."
It's incredibly strange that this blog post doesn't even link to the bug description https://bugzilla.mozilla.org/show_bug.cgi?id=1548973 Imagine someone stumbling upon this post and trying to find more technical information... Poor communication.
I'm trying to be angry at Mozilla for messing up, but cant force myself to do it. They ended up to be that adorkable kid that spilled the paint all over the carpet and you just sigh and start cleaning up the mess.
This isn't even working reliably. The hotfix study is showing up completed, not active, on my Firefox at home.
No idea why, there's no information about how to reactivate it. No, re-installing it didn't help.
Well, I'm about to make a lot of people happy with this info. I was researching this today as I'm using FF 56.0.2 and found the solution on this discussion thread. Leave it to an end user to do the job the professionals either failed or refused to due. It worked for me on 3 different machines. Go to this link and follow the instructions detailed:
If this hadn't happened, I probably wouldn't have discovered Brave. Noticeably faster and the ad-blocking is built in. Thanks Mozilla
hmm, i don't seem to have been affected by this bug somehow (my extensions are all still working). i turn off as much phoning home as i can (including turning studies off) and block connections to *.services.mozilla.com
any idea why i might not be affected? it may help others who might want to retain control of their firefox browser (chromium-based browsers being non-sequiturs).
Patching security bugs using Shield is utterly stupid.
What would happen if they found a Zero Day - would they use the same method?
If anyone is running firefox-esr on debian (I am), there is a separate discussion happening here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928415