▲ 334 points
back
160 comments
Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect.
(2) it's not clear what the "error bars" should mean. One is a confidence interval[1] (e.g. model gives 95% chance the output will be within these bounds). Another is a standard deviation (i.e. you are pretty much predicting the squared difference between your own point forecast and the outcome).
[1] acknowledged: not the correct term
Recently someone on hacker news described statistics as trying to measure how surprised you should be when you are wrong. Big fat error bars would give you the idea that you should expect to be wrong. Skinny ones would highlight that it might be somewhat upsetting to find out you are wrong. I don't think this is an exhaustive description of statistics but I do find it useful when thinking about forecasts.
A position espoused by Bill Phillips [1], and to which I now adhere:
"You should be willing to take either side of the bet that confidence interval implies." (paraphrasing; he says it better).
For a concrete example, with a 95% confidence interval, you should be as willing to accept the 19:1 odds that the true value is outside the interval as you are the 1:19 odds that the true value is inside the interval.
Aside from being generally correct, this approach is immediately actionable by making the meaning more visceral in discussions of uncertainty. Done right, it pushes you to assign uncertainties that are neither too conservative nor too optimistic.
If the notion of letting your reader take either side of the bet makes your stomach a little queasy, you're on the right track. The feeling will subside when you're pretty sure you got the errorbar right and your reasoning is documented and defensible.
Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course)
[1] https://www.nobelprize.org/prizes/physics/1997/phillips/fact...
Error bars in forecasts can only mean uncertainty your model has. Without error bars over models, you can say nothing about how good your model is. Even with them, your hypermodel may be inadequate.
That's not what a confidence interval is. A confidence interval is a random variable that covers the true value 95% of the time (assuming the model is correctly specified).
Error bars (either confidence interval or standard deviations) are of little use because they do not tell you how the probability is distributed within the confidence interval band. The holy grail of forecasting is the Probabilistic Forecast that predicts the entire posterior distribution so that you can sample from it generating scenarios or realizations of the underlying random process.
> Another is a standard deviation (i.e. you are pretty much predicting the squared difference between your own point forecast and the outcome).
What you probably want is the standard error, because you are not interested in how much your data differ from each other but in how much your data differ from the true population.
As far as error bars are concerned, you could report some% credible intervals calculated from taking the some%tile out of your results. It’s somewhat Bayesian thinking but it will work better than confidence intervals.
The intuition would be that some% of your forecasts are between the bounds of the credible interval.
Yes, please! I was part of an org that ran thousands of online experiments over the course of several years. Having some sort of error bars when comparing the benefit of a new treatment gave a much better understanding.
Some thought it clouded the issue. For example, when a new treatment caused a 1% "improvement", but the confidence interval extended from -10% to 10%, it was clear that the experiment didn't tell us how that metric was affected. This makes the decision feel more arbitrary. But that is exactly the point - the decision is arbitrary in that case, and the confidence interval tells us that, allowing us to focus on other trade-offs involved. If the confidence interval is 0.9% to 1.1%, we know that we can be much more confident in the effect.
A big problem with this is that meaningful error bars can be extremely difficult to come by in some cases. For example, imagine having something like that for every prediction made by an ML model. I would love to have that, but I'm not aware of any reasonable way to achieve it for most types of models. The same goes for online experiments where a complicated experiment design is required because there isn't a way to do random allocation that results in sufficiently independent cohorts.
On a similar note, regularly look at histograms (i.e., statistical distributions) for all important metrics. In one case, we were having speed issues in calls to a large web service. Many calls were completing in < 50 ms, but too many were tripping our 500 ms timeout. At the same time, we had noticed the emergence of two clear peaks in the speed histogram (i.e., it was a multimodal distribution). That caused us to dig a bit deeper and see that the two peaks represented logged-out and logged-in users. That knowledge allowed us to ignore wide swaths of code and spot the speed issues in some recently pushed personalization code that we might not have suspected otherwise.
> This makes the decision feel more arbitrary.
This is something I've started noticing more and more with experience: people really hate arbitrary decisions.
People go to surprising lengths to add legitimacy to arbitrary decisions. Sometimes it takes the shape of statistical models that produce noise that is then paraded as signal. Often it comes from pseudo-experts who don't really have the methods and feedback loops to know what they are doing but they have a socially cultivated air of expertise so they can lend decisions legitimacy. (They used to be called witch-doctors, priests or astrologers, now they are management consultants and macroeconomists.)
Me? I prefer to be explicit about what's going on and literally toss a coin. That is not the strategy to get big piles of shiny rocks though.
> That caused us to dig a bit deeper and see that the two peaks represented logged-out and logged-in users.
This is extremely common and one of the core ideas of statistical process control[1].
Sometimes you have just the one process generating values that are sort of similarly distributed. That's a nice situation because it lets you use all sorts of statistical tools for planning, inferences, etc.
Then frequently what you have is really two or more interleaved processes masquerading as one. These distributions generate values that within each are sort of similarly distributed, but any analysis you do on the aggregate is going to be confused. Knowing the major components of the pretend-single process you're looking at puts you ahead of your competition -- always.
[1]: https://two-wrongs.com/statistical-process-control-a-practit...
Completely agree with this idea. And I would add a corollary...date estimates (i.e. deadlines) should also have error bars. After all, a date is a forecast. If a stakeholder asks for a date, they should also specify what kind of error bars they're looking for. A raw date with no estimate of uncertainty is meaningless. And correspondingly, if an engineer is giving a date to some other stakeholder, they should include some kind of uncertainty estimate with it. There's a huge difference between saying that something will be done by X date with 90% confidence versus three nines confidence.
The problem is that date estimates for deadlines are NOT a standard distribution and everybody's normal statistical tools do not work.
They are pretty much a one sided distribution power law. Deadlines almost never come in early and, when they do, it's rarely by much. On the other hand, deadlines can come in late by wild amounts.
Generating confidence intervals on that is really hard.
So much this. I've written about it before, but one of the big bonuses you get from doing it this way is that it enables you to learn from your mistakes.
A date estimation with no error bars cannot be proven wrong. But! If you say "there's a 50 % chance it's done before this date" then you can look back at your 20 most recent such estimations and around 10 of them better have been on time. Otherwise your estimations are not calibrated. But at least then you know, right? Which you wouldn't without the error bars.
A deadline implies the upper limit of error bar cannot exceed it. That means you need to appropriately buffer to hit the deadline.
Uncertainty quantification is a neglected aspect of data science and especially machine learning. Practitioners do not always have the statistical background, and the ML crowd generally has a "predict first and asks questions later" mindset that precludes such niceties.
I always demand error bars.
You can demand error bars but they aren't always possible or meaningful. You can more or less "fudge" some sort of normally distributed IID error estimate onto any method, but that doesn't necessarily mean anything. Generating error bars (or generally error distributions) that actually describe the common sense idea of uncertainty can be quite theoretically and computationally demanding for a general nonlinear model even in the ideal cases. There are some good practical methods backed by theory like Monte Carlo Dropout, but the error bars generated for that aren't necessarily always the error you want either (MC DO estimates the uncertainty due to model weights but not say, due to poor training data). I'm a huge advocate for methods that natively incorporate uncertainty, but there are lots of model types that empirically produce very useful results but where it's not obvious how to produce/interpret useful estimates of uncertainty in any sort of efficient manner.
Another, separate, issue that is often neglected is the idea of calibrated model outputs, but that's its own rabbit hole.
So is it really science? These are concepts from stats 101. And the reasons and need, and the risks of not having them are very clear. But you have millions being put into models without these pre-requisites, and being sold to people as solutions, and waved away as "if people buy is it's bc it has value". People also pay fraudsters.
Error bars are important. But most people misinterpret their meaning, see https://errorbars.streamlit.app/
Also, error bars qua statistics can indicate problems with the underlying data and model, eg. if they're unrealistically narrow, symmetric etc.
I'm reminded of Walter Lewin's analogous point about measurements from his 8.01 lectures:
any measurement that you make without any knowledge
of the uncertainty is meaningless
You could say that forecasts are measurements you make about the future.
To that point, similarly:
"Being able to quantify uncertainty, and incorporate it into models, is what makes science quantitative, rather than qualitative. " - Lawrence M. Krauss
I really thought that this was going to be about the weather.
Me too, and I was looking forward to the thread that talks about error bars in weather models, which is totally a thing!
It turns out the ECMWF does do an ensamble model where they run 51 concurrent models, presumably with slightly different initial conditions, or they vary the model parameters within some envelope. From these 51 models you can get a decent confidence interval.
But this is a lower resolution model, run less frequently. I assume they don't do this with their "HRES" model (which has twice the spacial resolution) in an ensemble because, well, it's really expensive.
[1]: https://en.wikipedia.org/wiki/Integrated_Forecast_System#Var...
I've been using meteoblue for a while now and they tell you how sure they are of their predictions. Right now I can see that they rate their predictability as medium for tomorrow, but high for the day after.
https://content.meteoblue.com/en/research-education/specific...
Same, but in a human context, are mundane atmospheric events so far off today that error bars would have any practical value and/or potentially introduce confusion?
The interesting example in this article is nowcasting! The art of forecasting the present or past while you're waiting for data to come in.
It's sloppy science / statistics to not haven error ranges.
Not easy to always say what the benefit is: if you present in-model uncertainty from a stochastic model that might still say nothing about an estimation error vs the actual process. For forecasting to show actual uncertainty you need to be in a quite luxurious position to know the data generating process. You could try to fudge it with a lot of historical data where available - but still...
Every estimate/prediction/forecast/interpolation/extrapolation should have a confidence/prediction/ or tolerance interval (application dependent) that incorporates the assumptions that the team is putting into the problem.
Reminds me of this paper[1]
> An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability
> Traditionally, scientists have placed more emphasis on communicating inferential uncertainty (i.e., the precision of statistical estimates) compared to outcome variability (i.e., the predictability of individual outcomes). Here, we show that this can lead to sizable misperceptions about the implications of scientific results. Specifically, we present three preregistered, randomized experiments where participants saw the same scientific findings visualized as showing only inferential uncertainty, only outcome variability, or both and answered questions about the size and importance of findings they were shown. Our results, composed of responses from medical professionals, professional data scientists, and tenure-track faculty, show that the prevalent form of visualizing only inferential uncertainty can lead to significant overestimates of treatment effects, even among highly trained experts. In contrast, we find that depicting both inferential uncertainty and outcome variability leads to more accurate perceptions of results while appearing to leave other subjective impressions of the results unchanged, on average.
[1] https://www.microsoft.com/en-us/research/publication/an-illu...
Forecasts can also be useful without error bars. Sometimes all one needs is a point prediction to inform actions.
But sometimes full knowledge of the predictive distribution is helpful or needed to make good decisions.
"Point forecasts will always be wrong" - true that for continuous data but if you can predict that some stock will go to 2.01x it's value instead of 2x that's still helpful.
If you are forecasting both "Crime" and "Economy", it's VERY likely you have domain expertise for neither.
This is a great advantage of Gaussian Process Regression aka. Kriging.
https://en.wikipedia.org/wiki/Gaussian_process#Gaussian_proc...
Looking at the graph, changes in this decade are noise. But what happened back in 1990?
Doesn't work.
For instance in a business setting, if I say "it'll be done in 10 days +/- 4 days", they'll immediately say "ok so you're saying it'll be done in 14 days tops then".
More effective to sound as unsure as possible, disclaim everything in slippery language, and promise to give updates to your predictions as soon as you realise they've changed (granted this wouldn't work as well for an anonymous reader situation like in this article).
Not just forecasts.
Accounting should do it too in their reporting.
I would love to see a balance sheet with a proper 'certainty range' around the values in there.
Take the error bars for Scrum estimations, 3, 5, 8 - people treat them as real things although they have huge errors and are very course.
For prediction intervals that are guaranteed coverage, check out conformal prediction [1]. Works great especially for time series data.
Not only forecast need error bars. Every statistic needs error bars. But even then most people interpret error bars wrongly, see e.g. https://errorbars.streamlit.app/
I'm just imagining adding error bars to my schedule forecasting (with schedules that are typically one the optimistic side thanks to management), with bars pointing in the bad direction, and seeing management still insist it'll take too long.
Let me suggest a solution https://github.com/valeman/awesome-conformal-prediction
And also claims that say "x improves y", should include std and avg in the title.
two inches deeper, is too much money!
So Alphabet prefer to loose the idea, the contract, the business plan rather than make the things solid and works for a long time.
a $ is a dollars, a project, is just a piece of fun.
Alphabet LOL XD
What is the best explanations to error bars ?
Linear error rate
I have, in my life as a web developer, had multiple "academics" urgently demand that i remove error bands, bars, notes about outliers, confidence intervals etc from graphics at the last minute so people are not "confused"
Its depressing