In short
Consider you are working on the ring $R_Q = \mathbb{Z}_Q[X] / \langle X^N + 1 \rangle$. As a rule of thumb, you have to consider that the noise after modulus switching is larger than $N$. In particular, it will never have only 5 bits, as in your example, because $N$ is typically larger than $2^{13}$ in the FV scheme.
In more detail.
Let's say you have an RLWE ciphertext $c = (a, b) \in R_Q^2$, with $b = a\cdot s + e + (q / t) \cdot m$, as in the FV scheme.
Similarly to what is explained in this answer, but using polynomials instead of vectors, after we perform a modulus switching from $Q$ to some $q$, we get a new ciphertext with noise term given by
$$e' := e \cdot q / Q + \epsilon' + \epsilon \cdot s$$
where both $\epsilon'$ and $\epsilon$ are polynomials with coefficients in the interval $[-1/2,\, 1/2]$.
Usually, it is true that the new error $e'$ is close to the scaled error $e \cdot Q / q$ because the other terms are small compared to this one. However, when the scaled error becomes too small, that is no longer true, as $\epsilon \cdot s$ starts to dominate the norm of $e'$, and this is the "limit of modulus switching". In more detail, the norm of $\epsilon \cdot s$ can be as big as $N \cdot || \epsilon || \cdot || s ||$. So, even using binary or ternary keys (thus $|| s || = 1$), we have
$N \cdot || \epsilon || \cdot || s || = N \cdot || \epsilon || \approx N/2$.
