Criteria to compare two frequency responses

Question

I am designing a FIR-low pass filter using freqz function in matlab. I have generated numerator coefficients using one technique(lets call it original response). Also I am trying to come up with these coefficients using some other technique(lets call it approximate response). I want to compare how close is my approximate response to the original response. These two responses I have plotted on matlab(one over another).

Visually its difficult to see the difference. Let me paste zoomed-in picture below. Here we can see there is difference in both.

Is there a way to quantify how much is approximated frequency response is off from actual response?

Answer 1

Typically you design a lowpass filter to certain specifications: pass band ripple, stop band attenuation, transition band width, phase distortion, latency, transient preservation, causality, computational complexity, etc.

When you compare two filter designs, you can evaluate them individually against all these criteria and compare the results. Often, one design will be better for some criteria and worse for other. In this case you need to do some "weighted" average of the metrics, where the weighting represents the priorities for your specific application.

It's entirely possible, that your "approximation" turns out to be "better" than the original.

If you want to figure out how close the approximation is to the original, things get a little more complicated. You can plot the relative error in dB (by simply dividing the approximation by the reference). However that can be misleading: 1 dB error in the pass band is typically much more of a problem than a 1 dB error in the stop band. So again, you need to define a metric for "better" in the context of your specific application and requirements.

Answer 2

There are two common criteria for comparing accuracy for filter approximations: least squares and peak error.

For either we determine the error between the two by subtracting their complex result $E(\omega) = H_1(\omega) - H_2(\omega)$ over the frequency range of interest where $H_1(\omega)$ and $H_2(\omega)$ represent each filters complex response (not in dB!).

From that the error criteria for least squares or peak error which can then be compared to other implementations is determined as follows:

Least Squares: $$\int|E(\omega)|^2d\omega$$

Peak Error: $$\max|E(\omega)|$$

For practical determination of least squares in the OP's case, I would suggest a discrete approximation over $N$ with $k=1...N-1$ samples in the frequency response in steps of $\Delta \omega$ using the following, providing an rms error result:

$$\epsilon_{rms}= \sqrt{\frac{1}{N}\sum_k |E(k \Delta \omega)|^2}$$

If only the magnitude response is of interest, the the error function can be modified to be:

$$E(\omega) = |H_1(\omega)| - |H_2(\omega)|$$

And if preference is given to either passband or stopband, weighting factors can be added. Ultimately the two common linear phase FIR optimum filter design algorithms, least squares and Parks-McClellan, will design FIR filters to target responses in the passband and stopbands providing the solution that would minimize the least squares and peak error respectively. Least squares filter coefficients can be readily computed using firls in MATLAB, Octave and Python scipy.signal, and Parks-McClellan can be computed using firpm in MATLAB and remez in Octave and Python scipy.signal.