▲ 378 points
back
116 comments
Author here if anyone has Pentium questions :-)
My Mastodon thread about the bug was on HN a few weeks ago, so this might seem familiar, but now I've finished a detailed blog post. The previous HN post has a bunch of comments: https://news.ycombinator.com/item?id=42391079
In my view, this $475M was perhaps the best marketing spend for Intel. Because of the bug and recall, everyone including those not in tech knew about Intel. Coming from the 486 when people were expecting 586 or 686 but then suddenly "Pentium", this bug and recall built a reputation and good will that carried on later with Pentium MMX.
Great article and analysis as always, thanks! Somewhat crazy to remember that a (as you argue) minor CPU erretum made world wide headlines. So many worse ones out there (like you mention from Intel) but others as well, that are completely forgotten.
For the Pentium, I'm curious about the FPU value stack (or whatever the correct term is) rework they did. It's been a long time, but didn't they do some kind of early "register renaming" thing that had you had to manually manage doing careful fxchg's?
> The bug is presumably in the Pentium's voluminous microcode. The microcode is too complex for me to analyze, so don't expect a detailed blog post on this subject.
How hard is it to "dump" the microcode into a bitstream? Could it be done programatically from high resolution die photographs? Of course, I appreciate that's probably the easy part in comparison to reverse engineering what the bitstream means.
> By carefully examining the PLA under a microscope
Do you do this stuff at home? What kind of equipment do you have in your lab? How did you develop the skills to do all this?
I was about to ask if the explanation of floating point numbers was using Avogadro's number on purpose, but then I realized the other number was Planck's constant.
Thank you very much for this detailed article.
I never realised this is how floating point division can be implemented. Actually funny how I didn't realise that multiple integer division steps are required to implement floating point division :-)
In hindsight one could wonder why the unused parts of the lookup table were not filled with 2 and -2 in the first place.
Tour de force, truly. Amazing work!
The bug is super fun, but I also find the Intel response to be fascinating on its own. They apparently didn’t replace everyone’s processor with a non faulty version who wanted it, resulting in a ton of bad press.
To contrast, I’ve been thinking a lot about the Amazon Colorsoft launch, which had a yellow band graphics issue on some devices (mine included). Amazon waited a bit before acknowledging it (maybe a day or two, presumably to get the facts right). Then they simply quietly replace all of them. No recall. They just send you a new one if you ask for it (mine replacement comes Friday, hopefully it will fix it). My takeaway is that it’s pretty clear that having an incredibly robust return/support apparatus has a lot of benefits when launches don’t go quite right. Certainly more than you’d expect from analysis.
Similarly I haven’t seen too many recent reports about the Apple AirPod Pros crackle issue that happened a couple years ago (my AirPods had to be replaced twice), but Apple also just quietly replaced them and the support competence really seemed something powerful that isn’t always noticed.
Colorsoft: https://www.tomsguide.com/tablets/e-readers/amazon-kindle-co...
AirPods Pro: https://support.apple.com/airpods-pro-service-program-sound-...
The Kindle and AirPod cases are not really comparable since those are relatively minor products for the respective companies.
On the Apple side the iPhone 4 antennagate is a better comparison since the equivalent fix there would have involved free replacements for a flagship and revenue-critical product which Apple did not offer.
Intel on the other hand did eventually offer free replacements for anybody who asked and took a major financial hit.
I had the first gen white MacBook with the magnetic closure that resulted in chipped, discoloured topcases. I had it replaced for free like three or four times over the lifespan of that computer, including past the three year AppleCare expiry.
I really respected Apple’s commitment to standing behind their product in that way.
I thought the response from intel was to invest a lot in correctness for a while and then deciding that AMD were not being punished for their higher defect rate and so, more recently, investing in other things to try to compete with AMD on other metrics than how buggy the cpu is.
For the most part, this wasn't an individual problem. Corporations purchased these pretty expensive Pentium computers through a distributor, and just got them replaced by the vendor, per their support contract.
I've been in some consumer Apple "shadow warranty" situations, so I know what you are talking about, but IMO very different than the "IT crisis" that intel was facing. "IBM said so" had a ton of IT weight back then.
That is default Amazon - you can return stuff no hassle for almost any reason.
> Intel's whitepaper claimed that a typical user would encounter a problem once every 27,000 years, insignificant compared to other sources of error such as DRAM bit flips.
> However, IBM performed their own analysis,29 suggesting that the problem could hit customers every few days.
I bet these aren’t as far off as they seem. Intel seems to be considering a single user, while I suspect IBM is thinking in terms of support calls.
This is a problem I’ve had at work. When you process a 100 million requests a day the one in a billion problem is hitting you a few times a month. If it’s something a customer or worse a manager notices, they ignore the denominator and suspect you all of incompetence. Four times a month can translate into “all the time” in the manner humans bias their experiences. If you get two statistical clusters of three in a week someone will lose their shit.
No, IBM's estimate is for a single user. IBM figures that a typical spreadsheet user does 5000 divides per second when recalculating and does 15 minutes of recalculating a day. IBM also figures that the numbers people use are 90 times as likely to cause an error as Intel's uniformly-distributed numbers. The result is one user will have an error every 24 days.
> It appears that only one person (Professor Nicely) noticed the bug in actual use.
I recall a study done years ago where students were supplied calculators for their math class. The calculators had been doctored to produce incorrect results. The researchers wanted to know how wrong the calculators had to be before the students noticed something was amiss.
It was a factor of 2.
Noticing the error, and being affected by the error, are two entirely different things.
I.e. how many people check to see if the computer's output is correct? I'd say very, very, very few. Not me, either, except in one case - when I was doing engineering computations at Boeing, I'd run the equations backwards to verify the outputs matched the inputs.
I used to tutor physics in college. My students would show a problem they worked and ask for feedback, and I’d tell them that they definitely went wrong somewhere since they calculated that the rollercoaster was 23,000 miles tall.
Which is to say, it will depend a lot on the context and the understanding of the person doing the calculation.
> Noticing the error, and being affected by the error, are two entirely different things.
Only somewhat true. Take any consumer usage here for example. If you're playing a game and it hits this incorrect output but you don't notice anything as a result, were you actually affected?
How much usage of FDIV on a Pentium was for numerically significant output instead of just multimedia?
I remember that bug. Because I could not control what CPU my customers were running on, I had to add special code in the library to detect the bad FPU and execute workaround code (this code was supplied by Intel).
I.e. Intel's problem became my problem, grrrr
Reminds me of a joke floating around at the time that captures a couple different 90s themes:
I AM PENTIUM OF BORG.
DIVISION IS FUTILE.
YOU WILL BE APPROXIMATED.pretty sure that was in my tagline generator...
Another great article from Ken. I remember this particularly because the first PC that I bought with my own money had an affected CPU. Prior to this era I hadn't been much interested in PCs because they couldn't run "real" software. But Windows NT changed that (thank you Mr. Cutler), and Taiwanese sourced low cost motherboards made it practical to build your own machine, as many people still do today. Ken touched on the fact that it was easy for users to check if their CPU was affected. I remember that this was as easy as typing a division expression with the magic numbers into Excel. If MS had released a version of Excel that worked around the bug, I suspect fewer users would have claimed their replacement device!
Couldn’t these PCs run 386BSD?
What an interesting and utterly dedicated analysis. Thank you so much for all your work analysing the silicon and sharing your findings. I particularly like how you’re able to call out Intel on the actual root cause, which their PR made sound like something analogous to a trivial omission. But, in fact, was less forgivable and more blameworthy, ie they stuffed up their table generation algorithm.
>Smith posted the email on a Compuserve forum, a 1990s version of social media.
I hate how this sentence makes me feel.
I like to use the 1900s instead of the 1990s.
My initial feeling is: that data is probably mostly unmined and lost. Lucky bastards!
He sent it via his Personal Computer, a precursor to the smartphone.
Given that the fixed table is a much simpler one (by letting out-of-bounds just return 2, rather than adding circuitry to make it return 0), I wonder why they didn't just do it that way in the first place?
It feels like the kind of optimization that gets missed because the task was split between multiple people, and nobody had complete knowledge of the problem.
The person generating the table didn't realize filling the out-of-bounds with two would make for a simpler PLA. And the person squishing the table into the PLA didn't realize the zeros were "don't care" and assumed they needed to be preserved.
It's also possible they simply stopped optimizing as soon as they felt the PLA was small enough for their needs. If they had already done the floorplanning, making the PLA even smaller wasn't going to make the chip any smaller, and their engineering time would be better spent elsewhere.
"Make it work first before you make it work fast". Fundamentally this is a software problem solved with software techniques. And like most software there's some optimization left on the table just because no one thought of it in time. And you can't patch a CPU of this era.
Returning 0 for undefined table entries is the obvious thing to do. Setting these entries to 2 is a bit of a conceptual leap, even though it would have prevented the FDIV error and it makes the PLA simpler. So I can't fault Intel for this.
That must have been such a satisfying fix for the engineers though!
More engineering time resulted in a more efficient solution.
>Since only one in 9 billion values caused the problem, Intel's view was that the problem was trivial: "This doesn't even qualify as an errata."
This sounds utterly insane. You are making a CPU, if any calculations are wrong it needs to be fixed ?? I supposed this only came to light very late into testing and it was very impractical to bin every cpu, so they rolled the dice.
> Curiously, the adder is an 8-bit adder but only 7 bits are used; perhaps the 8-bit adder was a standard logic block at Intel.
I believe this is because for any adder you always want 1 bit extra to detect overflow! This is why 9 bit adders are a common component in MCUs
The weird thing is that I traced out the circuitry and the bottom bit of the adder is discarded, not the top bit where overflow would happen. (Note that you won't get overflow for this addition because the partial remainder is in range, just split into the sum and carry parts.)
I'm surprised they took the risk of extending the lookup table to have all 2's in the undefined region. A safer route would have been to just fix the 5 entries. Someone was pretty confident!
It actually seems like it becomes much easier to reason about because you remove a ton of (literal in the diagram) edge cases.
How did idiv work on the pentium. Was it also optimized, or somehow connected to fdiv, or just the old slow algorithm?
At the 2012 Turning Award conference in San Francisco, Prof William Kahan mentioned that he had a newer test suite available in 1993 that would have caught Intel's bug. Still, Intel did not run that.. Prof. Kahan was actively involved in its analysis and further testing. (I'm stating this just from memory).
> The explanation is that Intel didn't just fill in the five missing table entries with the correct value of 2. Instead, Intel filled all the unused table entries with 2.
I wonder why they didn't do this in the first place.
From someone who had to mentally let go once you started talking about planes crossing each other, thank you for such an amazingly detailed writeup. It's not everyday that you learn a new cool way to divide numbers!
Intel $475B error: not building a decent GPU
Didn't Intel have floating point division issues more recently as well?
> He called Intel tech support but was brushed off
I laughed when I read this. It’s hard enough to get support for basic issues, good luck explaining a hardware bug.
Reminds me of part 2 of day24. Some wrong wirings. ;-)
"At Intel, Quality is job 0.9999999999999999762"