Why I only care about amplitude response
I'm working on equalizing a DAC where I only care about the flatness of an AWGN spectrum. Since AWGN has random phase for any frequency bin, I do not care about correcting phase response at all; only "flatness".
Calibration Setup
I'll be playing a CAZAC training sequence out of the DAC and noting the magnitudes on a spectrum analyzer. It will be fairly easy to determine what the desired gain for some frequency $D(f_\xi)$ should be.
Problem formulation
I am trying to find time domain taps $h$ such that the error between the desired frequency response gain ($D$) and the tap frequency response ($H$) is minimized after neglecting any error due to phase ($e^{j \phi(\xi)}$) since I only care about $\left| H(f_\xi) \right|$ vs. $\left| D(f_\xi) \right|$.
$$\left[ \begin{array}{c} H(f_0) \\ H(f_1) \\ \vdots \end{array} \right] \overset{\textrm{LS}}{\textrm{=}} \left[\begin{array}{c} D(f_0) \\ D(f_1) \\ \vdots \end{array} \right] \cdot \left[ \begin{array}{c} e^{j \phi(f_0)} \\ e^{j \phi(f_1)} \\ \vdots \end{array} \right]$$
The final thing I want to solve for is the time domain tap values:
$$\left[ \begin{array}{c c} W_{0,0} & W_{0,1} \\ W_{1,0} & W_{1,1} \\ W_{2,0} & W_{2,1} \end{array} \right] \cdot \left[ \begin{array}{c} h_0 \\ h_1 \end{array} \right] \overset{\textrm{LS}}{\textrm{=}} \left[\begin{array}{c} D(f_0) \\ D(f_1) \\ D(f_2) \end{array} \right] \cdot \left[ \begin{array}{c} e^{j \phi(f_0)} \\ e^{j \phi(f_1)} \\ e^{j \phi(f_2)} \end{array} \right]$$
where $W_{\xi,m} = \exp\left(- j 2 \pi f_\xi m \right)$ with $N=2$ in this example.
Note the definition of the DFT in the $W$ matrix:
$$H(f_\xi) = \sum_{m=0}^{N-1} \exp\left(- j 2 \pi f_\xi m \right)$$
The difficulty in solving this problem is in determining what $\phi(f_\xi)$ should be in order to make the Least Squares (LS) algorithm work best. I am already useing LS equalization (to great effect) elsewhere in this design to correct for phase and amplitude error.