There is an entire research field on system theory. The problem is well known but that doesn't mean the answer is simple.
Some basics
First, let me write your problem in a more common notation. In general, a discrete linear system is described as:
$x_{k+1} = Ax_k + Bu_k $
$y_{k} = Cx_k + Du_k $
We call $x$ the internal states and $y$ the observable output. I can be shown that for this system to be stable, it is sufficient that the internal states are stable. This is the case if the eigenvalues $e_i$ of $A$ lie within the unit circle ($|e_i| < 1$).
Alternatively, you can also use a continuous linear system:
$\dot{x} = Ax + Bu $
$y = Cx + Du $
Here, the stability condition is that the real part of the eigenvalues must be smaller than zero. You appear to have mixed up these stability conditions.
Analysis of nonlinear systems
In general, analysing nonlinear systems is pretty hard. That's why we usually calculate a linearisation around some fixed points. Unless the eigenvalues are zero, this linearisation is accurate around the fixed point, so if the linear approximation is stable, the nonlinear system is locally stable around this fixed point.
Black box methods
If the system you are studying is a black box, you will have to estimate the system somehow. The method you have described is not sufficient. Even a simple linear system can have varying responses to signals of different frequencies.
You will need to sample multiple frequencies, either by applying signals separately, or by applying a multi frequency signal like white noise and applying some processing. You can use this to estimate the linear system and draw stability conclusions from there.
Edit: Specifics about feedback and your problem
I didn't understand from your question that you have access to the linear response functions. For brevity, I'm going to be working with a continuous time system from now on. I'll also be use the laplace transform representation of the transfer function.
Let's say that we have a first system (the microphone) with transfer function $A(s)$ and second system with transfer function $B(s)$. Putting these two after eachother in a feedback loop results in the following transfer function: $\frac{A(s)B(s)}{1 - A(s)B(s)}$ (you can find this derivation online, or i can write it out for you if you are interested). This transfer function should be a rational function for a linear system, with zeros $z_i$ and poles $p_i$. The stability of the system depends on the poles $p_i$, if their real part is positive, we have instability.
The problem is, i don't think we have access to the full complex transfer functions here. We only have the Fourier transform, which (if it exists) amounts to the laplace transform with imaginary $s$. The Fourier transform doesn't tell you everything about the system. It ignores any transient behaviour and gives you the frequency response after everything has settled down. If we get instability, the signal is constantly changing so this transient behaviour is important.
Small example
Let's say that AB combined (without feedback) amounts to the following system:
$x_{k+1} = au_k\\
y_k = x_k$.
Then the z transform (a discrete equivalent to laplace) of the open loop system is $az^{-1}$. The closed loop system then has a transfer function $\frac{az^{-1}}{(1-az^{-1})}$
The open loop system just copies the original signal one unit of delay later (and slightly amplified). But the closed loop system behaves differently. An impulse will result in a step function that decays according to $a$: $y_k = a^ku[k-1]$, with $u$ the unit step function.
In this simple system, you get your intuitive solution, if $|a| > 1$, you get instability. You can also see that $a$ is a pole of the closed loop transfer function.
In a more complicated example, you will have more memory in the system opening up the potential for destructive interference of the signal. However, the intuitive rule should still work most of the time.