▲ 116 points
back
100 comments
If you're like me and struggled to understand the traditional textbook floating point formula explanation, Fabien Sanglard (of doom/wolf3d black book fame) wrote a page that finally made sense for me: http://fabiensanglard.net/floating_point_visually_explained/...
Can I ask an embarassing question to verify I'm not misunderstanding that page? Does this mean that floats are less precise the larger they are? You have as many bits to describe the "position of the number" between 0-1 as you do between 2048-4096
This is precisely what the term floating point means. The decimal point "floats" right or left depending on magnitude due to the mantissa-exponent representation. The opposite is fixed point where you have a fixed number of bits reserved for the whole and fractional parts. A 32-bit floating point can represent a vastly greater range of magnitudes than a 32-bit fixed point, with a vastly greater precision for values near zero, but at the expense of absolute precision at large magnitudes.
They are less precise in the absolute sense (the absolute error is larger) but equally so in the relative sense (the relative error is the same).
Well, essentially, of course.
The author's presentation of the traditional explanation makes sense only if M is visualized as binary, while E remains decimal. But the reader coming to the subject for the first time is in no position to guess this, and would need a clue like subscript notation: B_2, E_10.
Traditional explanation done well:
A one pager explanation I wrote after it finally clicked for me -- floating point is just binary scientific notation! Not sure why more places don't explain it like this. https://github.com/dasl-/floating-point-numbers#floating-poi...
Plugging my own explanation because HN seemed to like it:
Although understanding floating point is important, it is often ignored in Data Science schools.
The title reminds me a friend of mine who attends a data science master course without computer background. He asked me for help for his numpy homework on Bayesian Probability theory. The number he got was so small that he got weird result. I immedieatly thought it was because float64 couldn't cover that number so I told him to use longdouble dtype (it's 80 bits in Intel I think) and the problem solved.
Later he told me even the professor didn't expect such case when announcing the homework and many students couldn't finish the hw on time because of this.
Would the log-sum-exp trick have helped by any chance? https://en.wikipedia.org/wiki/LogSumExp
> I immedieatly thought it was because float64 couldn't cover that number so I told him to use longdouble dtype (it's 80 bits in Intel I think) and the problem solved.
This is a very dangerous habit. If 64 bits isn't enough you are generally using the wrong representation.
That's a difficulty that exists for the simulations, optimization problems and various algorithms. Some need so many bits that they don't work at all.
No such thing as “Data science” - just a fancy term for trade school computer skills. Thats why.
according to this document the people who implemented java did not:
"How Java’s Floating-Point Hurts Everyone Everywhere"
https://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf
p.s.: it's by William Kahan, who the primary architect behind the IEEE 754-1985 standard for floating-point computation (and its radix-independent follow-on, IEEE 854)
Would be great to see similar critiques for Python and Go. Found only this for Matlab, on Dr. Kahan's page:
no language has a good support for IEEE754. The only doable thing is using some weird macros in C/C++, but the rest of the languages ignore the issue altogether. And LLVM too.
Not developers. This survey included mostly PhD students (73), faculty (49), but only 23 software engineers. The rest are others. Only 1/4 see software engineering as their main role according to the paper.
A PhD student is very different from a professional developer with years of real-world experience in industry. It's a completely different job.
The title is highly misleading.
It's OK, not misleading at all, and the abstract spells out what they're studying and the motivation behind it clearly.
A "developer" is just someone who writes software and not necessarily a "professional developer" with years of experience in a particular kind of job like what you have in mind.
Moreover, I would even expect that the vast majority of "professional software engineers" who make a living strictly writing software without any particular domain in focus, might actually fare worse in floating point gotchas than scientific domain workers who are "software engineering dilettantes" but nonetheless do computation as their job.
I'd wager the vast majority of "professional developers with years of real-world experience in industry" have a very superficial understanding of IEEE floating point behavior beyond "sometimes you get inaccurate results due to something-something-rounding-errors".
I agree that there are definitely a lot of differences between developers in an academic environment to the ones in a corporate environment. I would expect that this counts in favor of the title and not against it. How many of the "professional developers" do you think even know(or remember) what "IEEE floating point" is?
I once encountered an old database using floats as its primary keys. This was in a fairly popular piece of manufacturing software from the 90s.
I sometimes, in Python, cautiously use floating point numbers as dictionary keys. I do this when what I mean is "this exact floating-point number" and not "the result of an equivalent computation." It's almost never a good idea, because it's really easy to trip yourself up by assuming the latter when only the former actually holds.
What type of data would that be? I once did this when I did my own primitive version of a geospatial database in the 90s.
Wow, I'd love to hear more about this.
Useful in case you want multiple entries with an id of approximately 1?
I doubt most developers really understand how floating points work ( as in how it is represented in memory and why and the mantissa, bias, etc ). Most understand that it is just an approximation and go on about their day.
When using floating point numbers, that's usually as far as you need to know. That and the range of safe integers for langauges like JS that only do float. (Though now there's BigInt at stage 3)
Yes with http://0.30000000000000004.com/
This is a pet subject of mine since as an FPGA developer a huge amount of my time is spent focusing on bit-precision and Fixed-Point and reduced precision floating point implementations of algorithms. In fact one of the interview questions I ask focuses on how to do different mathematical operations in floating point using only logical/integer operations.
On the one hand I find the results unsurprising - even people I know to have worked a lot on numerics often have only a rudimental understanding of the intricacies of corner case behaviour for floating point, and yes that absolutely manifests in hitting a wall when something curious goes wrong. Mostly this results in head banging until you find the piece of code going wrong is the numerical piece and then very quickly you start looking at possible floating point gremlins.
Having said that though this paper seems to have a very academic view of what HPC is. Even for people designing HPC systems numerical optimizations are rarely a huge chunk of their job so it's probably not actually that important - I think the fact few people have a good understanding is a reflection of the fact it's not often necessary.
Finally while we're on the topic: Does anyone know a good tool that can allow me to write an equation, specific the precision of inputs/outputs and get the required precision of all the operators? I know Matlab have the DSP toolbox but it has some serious limitations, I'm still in search of something fantastic.
Herbie is not an exact match for your need it might be useful nevertheless : https://herbie.uwplse.org/
I understand floating point representation, but beyond that, it is a bit murky. I find floating point arithmetic to be very hard to really implement. For example, a basic question I have involves scaling: suppose I want to perform an operation involving large numbers like 13000.01 + 13000.01 + 13000.01. Do I get better precision results if I do 0.01 + 0.01 + 0.01 in floating point, then do the 3*13000 in integer arithmetic, and add the results?
A more general way to do that is Kahan summation[1], where you keep track of the rounding errors made so far.
[1]: https://en.wikipedia.org/wiki/Kahan_summation_algorithm
Significand * 2^exponent so yes. Another way to think of it more naturally is just remember it's a floating point in a fixed number of bits. Since base doesn't really matter for theory you can just think of a decimal point instead of binary. If you move it over to put more digits/information on the left side then you have to take digits/information from the right side.
If you were going to do this though I'd just do 2 ints acting as a single fixed point. Less glue logic and likely more precision. A float using all of the bits would likely be better than int + float as well depending how you configured it.
It's interesting to point out that you can compute with exact real numbers. But this is of course going to be less efficient than using something crude and fast like floating point numbers. There's also an argument that imprecision from your physical sensors is probably going to be more significant than the rounding errors that come with floating point.
Apart from greater speed, floating point numbers don't really have any advantages over exact real numbers. Problems like the inability to compute discontinuous functions over the real numbers manifest as even worse problems over the floating point numbers.
I'm also a believer that anything beautiful is probably going to end up useful. I think exact real computation (as described in Type Two Computability Theory) is one such thing. Automatic differentiation was another thing I found beautiful, before I found out about its applications to neural networks.
Type-two theory has a huge disadvantage: it elevates the type of all operations by a level. For example, real numbers become functions. Hence addition becomes a type 2 functional. Integration becomes type 3. It is very hard to reason about the efficiency (and computability, come to think of it) of type 3 functionals. There's some esoteric example involving the fan functional [1].
The complexity theory of type-2 functionals is a theory which is natural to ask for, but after results by Cook, Bellantoni, Kapron, Seth, Pezzoli and others, has come to an impasse and now seems abandoned.
[1] https://pdfs.semanticscholar.org/3888/cc2d3c44fb051067581e39...
> floating point numbers don't really have any advantages over exact real numbers.
With floating point numbers, you can solve quadratic equations, calculate sin/cos, calculate log/exp. These things have lots of applications.
I have interviewed a lot of developers, and floating point precision is one of my common phone screen questions.
My experience is; those who should know about it, do. Of course, a lot of apps or web developers don't. And I think that is okay.
The bigger issue is code designed for float32 (for performance reasons) being assumed valid for float64 data. And in these cases the issue is always, what is the appropriate way to communicate to your user the intended use case of the function? When the writer is long gone, a user will not intuitively know. Whenever I have to use a 3rd party library (ahem, PCL). I peruse the code and grep for float variables... If I see too many, I simply assume I need to recenter my float64 to be near 0.0.
Not everybody has the experience for this diligence, thus it appears as if developers don't understand.
A lecture on IEEE floating point from CMU including most of the gotchas the pdf contains https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=...
This is an active research area, ENS de Lyon has a lot of material available if interested implementing floating point on FPGAs, plus there is Vol 2 of The Art of Computer Programming where Knuth shows the reader how Babylonian mathematicians made extensive use of a sexagesimal (radix sixty) positional notation that was unique in that it was actually a floating point form of representation with exponents omitted.
Even if you understand them... in a polyglot world, you're still going to get burned.
It is so easy--especially when you're jumping from a NUMBER language like SQL or Scheme to a floating-point one like C or JavaScript--to backslide into boners like
for(i = 0.0; i < 1.0; i += 0.1)
edit: My first Clojure program was the last time this got me--LISP on the outside, Java floats/ints/bigints on the inside.
Yeah, and that’s why we have int for counting.
Obligatory link to the classic paper on the subject, from 1991:
https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...
How many software engineers had to study IEEE floating point in (any amount of) detail? Part of my formal education is in computer (hardware) engineering, and we studied hardware number representations and manipulations quite extensively in my microprocessors courses, so I developed some facility with the topic once upon a time.
I've met plenty of coders who aren't even aware of the problem, and I spent 13 years maintaining and extending an accounting system written by people who couldn't tell a floating point issue from their ass on a good day.
It's not going to happen much until we stop pushing floats as the standard solution for dealing with decimals. I'm pretty sure they are more commonly abused than used at this point, and most cases would be better off with fixpoints, rationals or bignums in increasing order of flexibility and complexity.
Lisp and Perl6 get it more or less right. My own baby, Snigl [0], only supports integers and fixpoints so far. I might well add floats eventually, but I intend on hiding them well enough to not encourage anyone who isn't motivated.
> (...) until we stop pushing floats as the standard solution for dealing with decimals.
I'm curious, in that sentence, who is "we"? I cannot imagine a scenario where someone would be so technically involved to be aware of the difference between a floating-point and fixed-point decimal, but still decide to use floating-point in an accounting environment.
Are you talking about project managers? Architects?
Do you mean binary fixed point, or decimal? Because for money matters I'll take decimal floating point over binary fixed point.
Crikey, floating point in accounting software :(
Integers with implied decimals is the way to go.
> Zero Divide By Zero:
> This question asks whether in floating point 0.0/0.0 is a non-NaN value, which it is not. NaN generation is desirable here, since it will propagate to the output as a NaN and thus make the user suspicious.
Horrible nonsense. What is desirable is an exception so that it's loudly confirmed to the user that something has screwed up. We don't want to lead the user on with subtle hints and suspicions.
Even a dollar store calculator from 1985 gets this right by locking up with an E on he display, requiring a clear.
This is configurable. Sometimes you want to see 99% of your results and know those were weird ones, and you can deal later.
60% of the people who took the survey develop software to support their main role. So, 60% of the people are not really software engineers but coders by necessity: could this explain the results easily?
60% of the time, developers understand floating point 100% of the time.