5

I am a little confused by Mathematica's use of the term prediction interval in NonlinearModelFit. We are given two kinds of prediction bands as an option: SinglePredictionBands and MeanPredictionBands

As I understand a prediction band, this is the region where one might expect the next data points, based on the result of the fit and the data. Where as a confidence band relates more to the likely location of your estimate/fit based on the data available. Predcition bands -- tell you about the future. Confidence bands -- tell you about now.

See here, here, and here for more information on confidence and prediction bands (this also gives some justification of my own understanding on the distinction between them).

In the documentation, for NonlinearModelFit it says:

"MeanPredictionBands" confidence bands for mean predictions
"SinglePredictionBands" confidence bands based on single observations

However this seems contradictory to me, as I have understood prediction bands to be distinct from confidence bands.

Is this sloppy documentation, or am I wrong in my understanding between the two? Is this a semantic issue?

Finally, if these are indeed prediction bands, can we plot confidence bands in Mathematica? Often I am far more interested in how well my model holds up to the measured data, rather than making predictions about the future.

J. M.'s missing motivation
  • 124,525
  • 11
  • 401
  • 574
user27119
  • 2,500
  • 13
  • 34
  • It might be helpful to rewrite this question for CrossValidated (https://stats.stackexchange.com) as it appears that you want to know about two statistical concepts that Mathematica doesn't explain so well (and that site has more "real" statisticians than here). As a hint, a confidence interval is for a parameter (or set of parameters). Here one can figure out the difference in the two (MeanPredictionBands and SinglePredictionBands) by determining the parameter(s) of interest. You might even want to throw in a question about "tolerance intervals" for completeness. – JimB Sep 30 '21 at 20:29
  • I was hoping you might appear. I think I understand the difference between prediction intervals/bands and confidence intervals/bands. After further research and reading this: https://rip94550.wordpress.com/2010/10/04/regression-1-%E2%80%93-two-kinds-of-prediction-bands/ I am becoming more convinced that MeanPredictionBands is indeed what I understand to be a Confidence Band. – user27119 Sep 30 '21 at 20:34
  • ...it sort of makes sense, if a single prediction refers to the future data point, then mean prediction I take to be where future fits would lie assuming the same distribution and number of points. The confusion for me is that I have always seen the two concepts labelled differently, e.g.: https://en.wikipedia.org/wiki/Confidence_and_prediction_bands – user27119 Sep 30 '21 at 20:37
  • The Wiki page also mentions pointwise (e.g, confidence interval for a particular $a+b\times x$) and simultaneous (95% confidence for a collection of $x$'s in $a+b\times x$ which could be a finite number or an infinite number of confidence intervals). – JimB Sep 30 '21 at 20:42
  • I feel you're trying to prompt me with the pointwise, but I'm not sure why. Isn't that its the confidence interval is just calculated discretely, at the $x$-values of the data? It's still pertaining to the distribution of the parameter estimate and not where future data points may end up, right? – user27119 Sep 30 '21 at 20:57
  • I don't think so. It's about if one needs to account for multiple comparisons: https://en.wikipedia.org/wiki/Multiple_comparisons_problem. – JimB Sep 30 '21 at 22:09

1 Answers1

1

It does seem a little ambiguous. However, the prediction bands are calculated using a confidence interval, so the ambiguity is understandable.

For example, in calculating the single prediction bands (Econometric Methods, J.Johnston, page 43):

enter image description here

As used in a code example here:

(* Estimated variance of a new observation *)
ev[X0_] := t s Sqrt[1 + 1/n + (X0 - Xmean)^2/Σx2]

On the other hand, I note on Wikipedia - Prediction bands it says:

Just as prediction intervals are wider than confidence intervals, prediction bands will be wider than confidence bands.

Chris Degnen
  • 30,927
  • 2
  • 54
  • 108
  • I have removed the MMSE examples, in which the confidence bands are somewhat unrigorous. However, I will leave the comment that prediction bands are calculated using confidence intervals, which explains why they can be called confidence bands. There appears to be numerous versions of confidence bands, which supports idea that it is a vague terminology. I.e. at least two on the Wikipedia link. – Chris Degnen Sep 30 '21 at 20:13
  • I've removed the down vote. In Sjoerd Smit's article: https://resources.wolframcloud.com/FunctionRepository/resources/BayesianLinearRegression. What I understand to be a confidence band he calls a "...distribution of the fit line...". But again, confidence bands pertain to the fit result itself, and prediction bands to future data points. Perhaps they mean this is where future (predicts) fits will lie, assuming the same data distribution -- as suggested by this article: https://rip94550.wordpress.com/2010/10/04/regression-1-%E2%80%93-two-kinds-of-prediction-bands/. – user27119 Sep 30 '21 at 20:29
  • 1
    @Q.P. I was looking at this and I think the main difference between the two bands is that one of them includes the error term in the regression line (y == f[x] + error) while the other only shows the error in the fit of f[x]. But I don't dabble too much in frequentist statistics, so I could be wrong. See also my blog post. – Sjoerd Smit Mar 01 '22 at 14:08