Why is xcolor causing a layout change in a tabularx column?

Question

I've made a MWE below. It's a simple two column {r X} table in tabularx. In the first row, without any xcolor commands, the first lines start in line with each other but in the second row, with colours added, the second cell starts a line below. Why is this happening?

\documentclass{article}
    \usepackage{lipsum}
    \usepackage{lua-visual-debug}
    \usepackage{tabularx}
    \usepackage{xcolor}

\begin{document}
    \centering
    \begin{tabularx}{\textwidth}{r X}
        First row & \lipsum[1] \\
        {\color{red} Second row} & {\color{blue}\lipsum[2]}
    \end{tabularx}
\end{document}