What Is Continuous White Noise in The Context of Signal Processing and Broadly

Question

How can one define Continuous White Noise in a coherent way?
Is there a way to derive it Mathematically?

Specifically, is there a way to define it which will works as the model in Signal Processing yet will obey all Mathematical properties of a Random Process?

Answer 1

In the context of Signal Processing White Noise is usually defined using a single intuitive property: A random process with constant magnitude, for any frequency, of its power spectrum:

If $ w \left( t \right) $ is a white noise then its power spectrum is given by $ {S}_{w} \left( f \right) = {\sigma}^{2} $. In communication it is common to use $ {S}_{w} \left( f \right) = \frac{ {N}_{0} }{2} $.

Using Wiener Khinchin Theorem one would conclude that the Auto Correlation of the process, which is assumed to be a Stationary Process in the wide sense is given by $ {R}_{ww} \left( \tau \right) = {\sigma}^{2} \delta \left( t \right) $.

In the context of Signal Processing we only need the model of Continuous White Noise in order to analyze the output of Linear System with finite frequency support. For this context, the above model is enough.

Yet, if one wants to be rigorous, mathematically, the above model is not enough.
For instance, if one wants the Wiener Khinchin Theorem to hold one must have, arbitrarily small, correlation in the auto correlation function.
So the move from the power density to the auto correlation, as done above, is not justified Mathematically (See for instance "Roy M. Howard - On Defining White Noise").
So the beast, called White Noise, is not plausible both Physically (Which is OK, we are dealing with Math), yet it is not coherent Mathematically, it requires more delicate work. Yet, again, in the context of Signal Processing, this is enough as only deal with it in the context of going through a Linear System with limited bandwidth support.

A more rigorous derivation can be done by defining the (Gaussian) White Noise as the derivative of Wiener Process.
Wiener Process has some properties which makes it interesting:

1. The difference of 2 samples is Gaussian.
2. The samples / realization path is continuous (In the almost surely sense).

Then we can define Gaussian White noise as the derivative of the Wiener Process. This will yield a more coherent definition (For instance, it indeed defines a distribution).

Remark I'd be happy if those who -1 will specify why.
I will try to explain the answer.

The answer has the following logic:

• We want a random process. Specifically a stationary random process (Even if only in the wide sense, there are subtleties to that, but let's skip this).
• It should obey Wiener Khinchin Theorem we need to be able to analyze it using spectral methods.

A reasonable way to do so is define a random process as following.

• Let $ v \left( t \right) $ be a stationary random process.
• For any pair $ {t}_{i} \neq {t}_{j} $ we would like to have $ \mathbb{E} \left[ v \left( {t}_{i} \right) v \left( {t}_{j} \right) \right] = 0 $.

If we can have that, great, we have a White Noise.
It turns out we can't! Why? Read the reference.
It has to do with convergence of a limit and order of integration and the limit operator.

So, what can have?
We can define $ \epsilon > 0 $ arbitrary small for which the auto correlation function is not zero.
It will give us a support, arbitrary large, in the PSD with constant value.
This model of White Noise will obey the needs of Signal Processing, namely for any Linear System with finite bandwidth the properties will be as we know them yet indeed our process will obey the properties of a valid random process, stationarity included.

Answer 2

Aaagh, it's 2 in the morning. I'll answer Royi's question sincerely tomorrow.

But it's all gonna boil down to how engineers and physical scientists think about the dirac delta function, $\delta(t)$, versus how the mathematicians think about the same. And this SE is not the same as the math SE. So there is a qualitative difference in the nature of rigor between the two communities.

Strictly speaking, the math guys are right. But it's totally useless in the context of signal processing or even in physical modeling, as per the wonderful quote from Richard Hamming:

Does anyone believe that the difference between the Lebesgue and Riemann integrals can have physical significance, and that whether say, an airplane would or would not fly could depend on this difference? If such were claimed, I should not care to fly in that plane.

Answer 3

Well, there's the way that Athanasios Papoulis defines it in Probability, Random Variables, and Stochastic Processes, McGraw-Hill, 1984:

We shall say that a process $\mathbf v(t)$ is white noise if its values $\mathbf v(t_i)$ and $\mathbf v(t_j)$ are uncorrelated for every $t_i$ and $t_j \ne t_i$.

Historically, I'm not sure if this was derived from a spectral point of view (i.e., starting out with a power spectrum and working backwards), or if the concept of "noise that is always different no matter how fine the time interval" came first, and broadband noise came later.

Given that signal processing evolved from pioneer radio technology, and that came from sticking bits of wire and metal plates together first, then identifying inductors, capacitors, and antennas later, I suspect that white noise was first observed and described in terms of spectra, and only later described in the time domain.

If you start with a signal that has a power spectral density $S(\omega) = 1$, then that pretty much defines that it's autocorrelation will be $R(\tau) = \delta(\tau)$. That definition for $R(\tau)$ leads directly* to Papoulis's time-domain definition.

---

* After, if you're picky about such things, you work through all the infinities involved with trying to apply $\delta(\tau)$ in multiple places.

Answer 4

So, this answer is going to need installments. There needs to be a few chapters to lead up to it. Before getting to the definition of white noise, we gotta begin with stochastic processes (a fancy name for "random processes"), specifically processes that are strictly stationary, more specifically Markov processes.

I am going to make lotsa assumptions at the outset to make my life easier:

1. This is about finite power signals and not about finite energy signals: $$ 0 \ < \ \lim_{T \to \infty} \frac{1}{T}\int\limits_{-\frac{T}2}^{\frac{T}2} \big|x(t)\big|^2 \ \mathrm{d}t \ < \ \infty $$
2. So then the appropriate definition of cross-correlation is: $$ R_{xy}(\tau) \triangleq \lim_{T \to \infty} \frac{1}{T}\int\limits_{-\frac{T}2}^{\frac{T}2} x(t+\tau)y^*(t) \ \mathrm{d}t $$
3. Autocorrelation of $x(t)$ is simply $R_{xx}(\tau)$ and $R_{xx}(0)$ is the mean power of $x(t)$.
4. $x(t)$ is ergodic in every sense which means that any expressible average or mean w.r.t. time (like those expressed above) can be expressed as a probabilistic average or mean. This is the Expected value (sometimes called "Expectation operator"): $$ \operatorname{E}\Big\{ g(x) \Big\} \triangleq \int\limits_{-\infty}^{\infty} g(\alpha) p_x(\alpha) \ \mathrm{d}\alpha $$ where $p_x(\alpha)$ is the probability density function (p.d.f.) of the random variable $x$ and $g(\cdot)$ is some well-defined function. A random process $x(t)$ sampled at any given time $t$ is a random variable.

So (with another assumption that $x(t)$ and $t$ are both real) the implication that ergodicity has on the autocorrelation is:

$$\begin{align} R_{xx}(\tau) &= \lim_{T \to \infty} \frac{1}{T}\int\limits_{-\frac{T}2}^{\frac{T}2} x(t+\tau)x(t) \ \mathrm{d}t \qquad \qquad & \text{(average w.r.t. time)}\\ \\ &= \operatorname{E}\Big\{ x(t+\tau)x(t) \Big\} \qquad \qquad & \text{(probabilistic average)} \\ \\ &= \int\limits_{-\infty}^{\infty}\int\limits_{-\infty}^{\infty} \alpha\beta \ p_{x(t+\tau)x(t)}(\alpha,\beta) \ \mathrm{d}\alpha \, \mathrm{d}\beta\\ \end{align}$$

where $p_{x(t+\tau)x(t)}(\alpha,\beta)$ is the joint p.d.f. for the two random variables $x(t+\tau)$ and $x(t)$.

$ R_{xx}(0) = \operatorname{E}\Big\{ \big|x(t)\big|^2 \Big\}$ is the mean-square of $x(t)$ and, if the mean $\operatorname{E}\Big\{ x(t) \Big\}=0$, the mean-square and variance are equal.

Now let's assume, additionally, that this random process is Gaussian and zero mean:

$$ p_{x(t)}(\alpha) = \frac{1}{\sqrt{2 \pi} \sigma_x} e^{-\frac12 \left(\frac{\alpha}{\sigma_x}\right)^2} $$

Being zero-mean, then the mean-square, $\operatorname{E}\Big\{ \big|x(t)\big|^2 \Big\}$, and variance, $\sigma_x^2$, are the same and larger than zero.

Additionally, we'll assume a Markov process. So the value of the previous states may affect the current state. The value of the state at $x(t)$ may have an effect on the probability of the state at a later time, $x(t+\tau)$.

As a judiciously-arranged example, consider this class of Markov process. Given the known earlier value $x(t)$, the dependent p.d.f. for the later random value $x(t+\tau)$ is:

$$\begin{align} p_{x(t+\tau)}\big(\alpha|x(t)\big) &= \frac{1}{\sqrt{2 \pi \left(\sigma_x^2 - R_{xx}(\tau)\right)}} e^{-\frac12 \frac{\left(\alpha-x(t) \sigma_x^{-2} R_{xx}(\tau)\right)^2}{\sigma_x^2 - R_{xx}(\tau)}} \\ \\ &= \frac{1}{\sqrt{2 \pi} \sigma(\tau)} e^{-\frac12 \left(\frac{\alpha-\mu(\tau)}{\sigma(\tau)}\right)^2} \\ \end{align}$$

where the dependent (on $x(t)$ and $\tau$) mean and variance are:

$$\begin{align} \mu(\tau) &= x(t) \sigma_x^{-2} R_{xx}(\tau) \\ \\ \sigma(\tau) &= \sqrt{\sigma_x^2 - R_{xx}(\tau)} \\ \end{align}$$

I think that the joint p.d.f. is related to the conditional p.d.f. as:

$$\begin{align} p_{x(t+\tau)x(t)}(\alpha,\beta) &= p_{x(t+\tau)}\big(\alpha|\beta \big) \cdot p_{x(t)}(\beta) \\ \\ &= \frac{1}{\sqrt{2 \pi \left(\sigma_x^2 - R_{xx}(\tau)\right)}} e^{-\frac12 \frac{\left(\alpha-\beta\sigma_x^{-2}R_{xx}(\tau)\right)^2}{\sigma_x^2 - R_{xx}(\tau)}} \ \cdot \ \frac{1}{\sqrt{2 \pi} \sigma_x} e^{-\frac12 \left(\frac{\beta}{\sigma_x}\right)^2} \\ \end{align}$$

Then the autocorrelation of this Markov random process is:

$$\begin{align} R_{xx}(\tau) &= \int\limits_{-\infty}^{\infty}\int\limits_{-\infty}^{\infty} \alpha\beta \ p_{x(t+\tau)x(t)}(\alpha,\beta) \ \mathrm{d}\alpha \, \mathrm{d}\beta \\ \\ &= \int\limits_{-\infty}^{\infty}\int\limits_{-\infty}^{\infty} \alpha\beta \ p_{x(t+\tau)}\big(\alpha|\beta \big) p_{x(t)}(\beta) \ \mathrm{d}\alpha \, \mathrm{d}\beta \\ \\ &= \int\limits_{-\infty}^{\infty} \beta \, p_{x(t)}(\beta) \ \left[ \int\limits_{-\infty}^{\infty} \alpha \ p_{x(t+\tau)}\big(\alpha|\beta \big) \ \mathrm{d}\alpha \right] \ \mathrm{d}\beta \\ \\ &= \int\limits_{-\infty}^{\infty} \beta \, p_{x(t)}(\beta) \ \Big[ \beta \sigma_x^{-2} R_{xx}(\tau) \Big] \ \mathrm{d}\beta \\ \\ &= \left[ \int\limits_{-\infty}^{\infty} \beta^2 \, p_{x(t)}(\beta) \ \mathrm{d}\beta \right] \ \sigma_x^{-2} R_{xx}(\tau) \\ \\ &= \Big[ \sigma_x^2 \Big] \, \sigma_x^{-2} R_{xx}(\tau) \\ \\ &= R_{xx}(\tau) \qquad \qquad \checkmark \\ \end{align}$$

This confirms the choice of dependent mean $\mu(\tau) = x(t) \sigma_x^{-2} R_{xx}(\tau)$.

The choice of dependent variance $\sigma^2(\tau)$ is, perhaps, less critical but we want $\sigma(0)=0$, and when $R_{xx}(\tau)=0$ we want $p_{x(t+\tau)}\big(\alpha|x(t)\big)$ to be independent of $x(t)$ which would mean that $\mu(\tau)=0$ and $\sigma^2(\tau)=\sigma_x^2$. Perhaps, the dependent variance should be $\sigma^2(\tau)=\sigma_x^2-|R_{xx}(\tau)|$. I dunno.

If $R_{xx}(\tau) = \sigma_x^2 \, e^{-|4 \nu \tau|}$ and $\nu > 0$, then this might be the Ornstein-Uhlenbeck process (essentially white noise filtered through a first-order RC low-pass filter) and it might approach Brownian motion (essentially white noise passed through an integrator) as $\nu \to 0$. I would say it approaches white noise as $\nu \to \infty$.

If, instead, $R_{xx}(\tau) = \sigma_x^2 \operatorname{sinc}(2 \nu \tau)$, it might be what I call "bandlimited white noise" (and the bandlimits are $\pm \nu$). Again, send $\nu \to \infty$ and you have a definition for white noise.

So the power spectrum is

$$ S_{xx}(f) \triangleq \mathscr{F}\Big\{ R_{xx}(\tau) \Big\} $$

For the Ornstein-Uhlenbeck process, then $$R_{xx}(\tau) = \sigma_x^2 \, e^{-|4 \nu \tau|}$$ and $$ S_{xx}(f) = \frac{\sigma_x^2}{2\nu} \ \frac{1}{1 + \left( \frac{\pi f}{2 \nu} \right)^2} $$

For the bandlimited white noise process, then $$R_{xx}(\tau) = \sigma_x^2 \operatorname{sinc}(2 \nu \tau)$$ and $$ S_{xx}(f) = \frac{\sigma_x^2}{2\nu} \ \Pi \left( \frac{f}{2\nu} \right) $$

Both of these have $S_{xx}(0) = \frac{\sigma_x^2}{2\nu}$ and an effective noise bandwidth (one-sided) of $\nu$ (two-sided bandwidth is $2\nu$). If the variance $\sigma_x^2$ is fixed to a non-zero value, the value of the power spectrum $S_{xx}(f)$ goes to zero (for all $f$) in the limit as $\nu \to \infty$.

If $S_{xx}(0)$ were to remain constant, say $S_{xx}(0) = \frac{\eta}{2}$, then the variance is $\sigma_x^2 = \eta \nu$ and would increase without bound as the effective noise bandwidth, $\nu$, goes to infinity. Note that, in both processes, as the noise bandwidth $\nu$ increases without bound, the autocorrelation becomes a dirac impulse in the limit.

Ornstein-Uhlenbeck to white noise: $$\begin{align} R_{xx}(\tau) &= \lim_{\nu \to \infty} \eta \nu \, e^{-|4 \nu \tau|} \\ \\ &= \frac{\eta}{2} \delta(\tau) \\ \\ \end{align}$$

bandlimited white noise to white noise: $$\begin{align} R_{xx}(\tau) &= \lim_{\nu \to \infty} \eta \nu \, \operatorname{sinc}(2 \nu \tau) \\ \\ &= \frac{\eta}{2} \delta(\tau) \\ \\ \end{align}$$

And the power spectra are both

$$ S_{xx}(f) = \frac{\eta}{2} \qquad \qquad \forall f \in \mathbb{R} $$

if $S_{xx}(0) = \frac{\eta}{2}$ is held to a constant and bandlimit $\nu$ goes to $\infty$. This is what decorrelates $x(t_1)$ from $x(t_2)$ if $t_1 \ne t_2$, but again, it requires infinite variance, $\sigma_x^2 \to \infty$.

---

Appendix:

Some definitions so that we can make sure we're all on the same page.

Continuous Fourier transform and inverse: $$\begin{align} \mathscr{F}\Big\{ x(t) \Big\} \triangleq X(f) &= \int\limits_{-\infty}^{\infty} x(t) \, e^{-j 2 \pi f t} \, \mathrm{d}t \\ \\ \mathscr{F}^{-1}\Big\{ X(f) \Big\} \triangleq x(t) &= \int\limits_{-\infty}^{\infty} X(f) \, e^{+j 2 \pi f t} \, \mathrm{d}f \\ \end{align}$$

Rectangular function (sometimes "$\operatorname{rect}(u)$"):

$$\Pi(u) \triangleq \begin{cases} 1 \qquad & \text{ if } |u| < \tfrac12 \\ \tfrac12 \qquad & \text{ if } |u| = \tfrac12 \\ 0 \qquad & \text{ if } |u| > \tfrac12 \\ \end{cases}$$

The inverse Fourier transform of the rectangular function is:

$$ \mathscr{F}^{-1} \left\{ \Pi\left( \tfrac{f}{2\nu} \right) \right\} = 2\nu \, \operatorname{sinc}(2\nu t) $$

The sinc function:

$$\operatorname{sinc}(u) \triangleq \begin{cases} \frac{\sin(\pi u)}{\pi u} \qquad & \text{ if } u \ne 0 \\ 1 \qquad & \text{ if } u = 0 \\ \end{cases}$$

---

More definitions and notation:

The probability density function (p.d.f.) of a random variable $x$ is denoted as

$$ p_x(\alpha) \triangleq \lim_{\Delta x \to 0} \frac{1}{\Delta x} \operatorname{P} \Big\{ \alpha-\tfrac12 \Delta x < x < \alpha+\tfrac12\Delta x \Big\} $$

where $\operatorname{P} \Big\{ \alpha-\tfrac12 \Delta x < x < \alpha+\tfrac12\Delta x \Big\}$ is the probability that $x$ exists between $\alpha-\tfrac12 \Delta x$ and $\alpha+\tfrac12 \Delta x$.

The Expected value of some function of single random variable $x$ is

$$ \operatorname{E}\Big\{ g(x) \Big\} \triangleq \int\limits_{-\infty}^{\infty} g(\alpha) \, p_x(\alpha) \ \mathrm{d}\alpha $$

And for a function of two random variables, $x$ and $y$, the Expected value is

$$ \operatorname{E}\Big\{ g(x,y) \Big\} \triangleq \int\limits_{-\infty}^{\infty} \int\limits_{-\infty}^{\infty} g(\alpha,\beta) \, p_{xy}(\alpha,\beta) \ \mathrm{d}\alpha \, \mathrm{d}\beta $$

where $p_{xy}(\alpha,\beta)$ is the joint p.d.f. for random variables $x$ and $y$ and is defined as

$$ p_{xy}(\alpha, \beta) \triangleq \lim_{\Delta x \to 0} \lim_{\Delta y \to 0} \frac{1}{\Delta x} \frac{1}{\Delta y} \ \operatorname{P} \left\{ \begin{matrix} \alpha-\tfrac12 \Delta x < x < \alpha+\tfrac12\Delta x \\ \text{and} \\ \beta-\tfrac12 \Delta y < y < \beta+\tfrac12\Delta y \\ \end{matrix} \right\} $$