The Fourier transform doesn't move only from time to frequency (or space etc.), it moves between these domains. The only difference between the Fourier transform and its inverse is a sign in the exponent and possibly the scaling. But these are conventions, and you may have noticed that the Fourier transform can be defined differently in different fields.
With the definition of the Fourier transform and its inverse as it is common in signal processing
\begin{align*}
X(\omega) &= \int_{-\infty}^{\infty}x(t)e^{-j\omega t}dt \\
x(t) &= \frac{1}{2\pi} \int_{-\infty}^{\infty}X(\omega)e^{j\omega t}d\omega \\
\end{align*}
we have
\begin{align*}
\mathscr{F}\big\{X(\omega)\big\} &= \int_{-\infty}^{\infty}X(\omega)e^{-j\omega t}d\omega \\
&= 2\pi x(-t)
\end{align*}
which shows that applying the Fourier transform twice gets us back to the time domain.
Also take a look at this related question.