Defined as "negative DFT bins zero", when are such filters suboptimal for AM/FM extraction or related filtering? This answer reads,
[nulling] also has the worst performance compared to other methods such as windowing or using the optimized algorithms known as "least-squares" and "equiripple" (Parks-McLellan) that are used in the 'firls' and 'firpm' or 'remez' commands;
with an explanation I don't quite follow,
this is because the Frequency Sampling method will result in a response that is exact at the bin frequencies used, but have more error compared to the other methods at all other frequencies that are in between the FFT bins, and more error in the time domain responses as well
Firstly, I strongly disagree with "worst performance", as such nulling is considered optimal in time-frequency analysis: the Generalized Morse Wavelets are built with such nulling being explicit. Obvious case: if any negative bin is non-zero, it can always be exploited to make a pure sine appear as AM. Example is STFT, which is pseudo-analytic near Nyquist:
This remains the case with higher sampling rates and observation intervals. What do the alt methods (remez etc) have to say about this?