So that we are on the same page, the basic relations are:
\begin{equation} \tag{1}
P_\text{dBm} = 10\log\left (\frac{P_{\text{mW}}}{0.001} \right)
\end{equation}
\begin{equation} \tag{2}
P_{\text{mW}} = \frac{10^{P_\text{dBm}/10}}{1000}
\end{equation}
Now imagine you wanted to do a normalisation with miliwatts, you'd simply do:
\begin{equation} \tag{3}
P_{\text{Normalized}}=\frac{P_\text{mW out}}{P_\text{mW in}}
\end{equation}
So that when there is no loss in the experiment you get $P_{\text{Normalised}}=1$, and when all the light is lost you get $P_{\text{Normalised}}=0$ . Now if we sub in equation 2 into equation 3 we get:
\begin{align}
P_{\text{Normalized}}&=\frac{10^{P_\text{dBm out}/10}}{10^{P_\text{dBm in}/10}} \\
&=10^{P_\text{dBm out}/10-P_\text{dBm in}/10} \\
&=\left ( 10^{P_\text{dBm out}-P_\text{dBm in}} \right )^{1/10}\\
\tag{4}
\end{align}
Now this suggests to us, that if we want something which is related to a normalised value in logarithmic units, we need to subtract instead of divide. This is actually the great thing about logarithmic units, instead of dividing you can subtract, which is much easier. So for example if you input 0dBm into your fibre and collect -10dBm on the other side, you know that you have a 10dB loss (notice that loss is in dB not dBm as it is effectively a normalised value).
Carrying on with the derivation for completeness, if we define the loss $L$ to be:
\begin{equation}
L = P_\text{dBm in}-P_\text{dBm out} \tag{5}
\end{equation}
Then
\begin{equation}
L = -10\log(P_{\text{Normalized}}) \tag{6}
\end{equation}
So in conclusion subtract your logarithmic values and divide your linear ones, use the above expression to switch between their respective normalised quantities.