Why is random noise assumed to be normally distributed?

Question

From residual in the linear regression to noise in signal processing are assumed to be normally distributed? By considering them as normally distributed we are kind of telling the pattern in the noise but shouldn't noise be considered random. This seems contradictory to me as on one side it is random then on the other side their distribution is considered normally distributed. Shouldn't the noise distribution be just random?

I believe there is some lacking in my understanding of the concept of statistical distribution which has lead me to this confusion, or I am looking at it all wrong.

One more example- when one augment data by adding Gaussian noise then it is not expected to change the overall distribution of data, why?

Answer 1

Starting at an even more basic level than the other (much smarter) answers, I'd like to pick up on this part of the question:

This seems contradictory to me as on one side it is random then on the other side their distribution is considered normally distributed.

Perhaps the issue here is what ‘random’ means?

To be clear: ‘random’ and ‘normally-distributed’ do not contradict each other.  ‘Random’ simply means that we can't predict exactly what the next value will be.  But that doesn't mean we can't make probabilistic statements about it.

Consider two experiments:

1. If you throw a (fair) die, then it could show any number from 1 to 6.  We can't tell which number will come up, but we can say that all numbers are equally likely (i.e. the distribution is uniform).

2. If you throw two dice and take their sum, that can be any number from 2 to 12.  Again, the sum is still random — we can't predict what it will be — but we can say that those values are not equally likely.  (For example, 7 is six times more likely than 12.)  So in this case it has a non-uniform distribution.  (You can plot all the probabilities; they take on a peaked shape a bit like a normal distribution.)

So there's no contradiction: both cases are random and have a known distribution.

In fact, most things that are random tend to have a non-uniform distribution: electrical noise, weather, the wait for the next bus, voting patterns…  Being able to make general statements about them without being able to predict the exact values is one of the strengths of statistics.

(As for why you often end up with a normal distribution, that's a result of the Central Limit Theorem, which says that when you combine many independent random variables, the result tends towards a Gaussian (normal) distribution.  So you see that crop up a lot.)

Answer 2

the place to look are the weak and strong law of large numbers, which is the basis of the central limit theorem, which states that if you add a large number of independent random variable with some mild conditions on the variance of those random numbers, the sum will become indistinguishable from a Normal Distribution.

A Normal Distribution also has the property of the maximum entropy of all distributions with bound variance.

The Normal Distribution is key in linear estimation but it should be noted that it isn’t the only distribution considered in Signal Processing while it may seem so to a newcomer.

The Normal is often a good model. Many physical noise mechanisms are Normally distributed. It also tends to admit closed form solutions.

One also encounters situations where the Normal assumption works despite not be a fully accurate assumption.

I don’t understand your last statement. Data has a distribution and adding Normal noise doesn’t change that distribution. The Signal and Noise distribution reflects both.

There are are also “refinements” or corrections to Normal Distributions like Gram Chalier series.

Answer 3

normal distribution (i like to call it "gaussian") remains normal after addition of normally distributed numbers. so if gaussian goes into an LTI filter, a gaussian distribution comes out. but because of this central limit theorem, even if uniform p.d.f. random process goes into an LTI filter with a long and dense impulse response, what will come out tends to be normally distributed. so the LTI system really only changes some parameters, like the power spectrum or autocorrelation of the signal. an LTI filter can turn a uniform p.d.f. white random process into gaussian p.d.f. pink noise.

Answer 4

I'll try to clear one possible source of confusion. If picking each sample value from a single distribution feels "not random enough", then let's try to make things "more random" by adding another layer of randomness. This will be found to be futile.

Imagine that for each sample the noise is random in the sense that it comes from a distribution that is randomly selected for that sample from a list of possible distributions, each with their own probability of occurrence and a list of probabilities for the possible sample values. Keeping it simple with just three distributions and four possible sample values:

$$\begin{array}{l|llll}&\rlap{\text{Sample value and its prob-}}\\ \text{Probability}&\rlap{\text{ability in the distribution}}\\ \text{of distribution}&-2&-1&0&1\\ \hline \color{blue}{0.3}&0.4&0.2&0.3&0.1\\ \color{blue}{0.2}&0.5&0.1&0.2&0.2\\ \color{blue}{0.5}&0.1&0.4&0.4&0.1\end{array}$$

Here we have actually a distribution of distributions. But there is a single distribution that says everything about the probabilities of the values for that sample:

$$\begin{array}{llll}\rlap{\text{Sample value and}}\\ \rlap{\text{its total probability}}\\ -2&-1&0&1\\ \hline 0.27&0.28&0.33&0.12 \end{array}$$

The total probabilities were obtained as sums of conditional probabilities of the sample values over the possible distributions:

$$0.4\times\color{blue}{0.3} + 0.5\times\color{blue}{0.2} + 0.1\times\color{blue}{0.5} = 0.27\\ 0.2\times\color{blue}{0.3} + 0.1\times\color{blue}{0.2} + 0.4\times\color{blue}{0.5} = 0.28\\ 0.3\times\color{blue}{0.3} + 0.2\times\color{blue}{0.2} + 0.4\times\color{blue}{0.5} = 0.33\\ 0.1\times\color{blue}{0.3} + 0.2\times\color{blue}{0.2} + 0.1\times\color{blue}{0.5} = 0.12$$

The laws of probability that were applied:

$$P(A_i\cap B_j) = P(A_i|B_j)\color{blue}{P(B_j)}\quad\text{conditional probability}$$ $$P(A_i) = \sum_jP(A_i\cap B_j)\quad\text{total probability}$$

where $A_i$ are the events of the $i\text{th}$ sample value occurring, and $B_j$ are mutually exclusive and exhaustive events of choosing the $j\text{th}$ distribution.

With continuous distributions, similar things would take place, because those can be modeled as discrete distributions in the limit that the number of possible events approaches infinity.

Answer 5

Noise is not random. It is fractal in nature.

Mandelbrot discovered that while working at IBM. And knowing that led to the improvement of dial-up modems, among other things. Before that, 9600 baud was out of reach.