You need no outer tabular (even less tabularx).
\documentclass{article}
\usepackage{graphicx}
\usepackage{subfig}
\begin{document}
\begin{figure}[htp]
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image-a}}\hfill
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image-b}}\hfill
\begin{minipage}[b]{.3\textwidth}
\subfloat[caption]{\includegraphics[width=\textwidth,height=1cm]{example-image}} \\
\subfloat[caption]{\includegraphics[width=\textwidth,height=1cm]{example-image}}
\end{minipage}
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}
\caption{A selection of the slides made during the user study}\label{slides}
\end{figure}
\end{document}
The trick is to use a suitable width, namely slightly less than 1/3 of the text width.
\documentclass{article}
\usepackage{graphicx}
\usepackage{subfig}
\begin{document}
\begin{figure}[htp]
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image-a}}\hfill
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image-b}}\hfill
\begin{minipage}[b]{.3\textwidth}
\subfloat[caption]{\includegraphics[width=\textwidth,height=1cm]{example-image}} \\
\subfloat[caption]{\includegraphics[width=\textwidth,height=1cm]{example-image}}
\end{minipage}
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subfloat[caption]{\includegraphics[width=.3\textwidth]{example-image}}
\caption{A selection of the slides made during the user study}\label{slides}
\end{figure}
\end{document}

If you're not tied to subfig for specific reasons of a document class not compatible with caption, I recommend using caption together with subcaption.
\documentclass{article}
\usepackage{graphicx}
\usepackage{caption,subcaption}
\begin{document}
\begin{figure}[htp]
\subcaptionbox{caption}{\includegraphics[width=.3\textwidth]{example-image-a}}\hfill
\subcaptionbox{caption}{\includegraphics[width=.3\textwidth]{example-image-b}}\hfill
\begin{minipage}[b]{.3\textwidth}
\subcaptionbox{caption}{\includegraphics[width=\textwidth,height=1cm]{example-image}}\\[1ex]
\subcaptionbox{caption}{\includegraphics[width=\textwidth,height=1cm]{example-image}}
\end{minipage}
\medskip
\subcaptionbox{caption}{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subcaptionbox{caption}{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subcaptionbox{caption}{\includegraphics[width=.3\textwidth]{example-image}}
\medskip
\subcaptionbox{caption}{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subcaptionbox{caption}{\includegraphics[width=.3\textwidth]{example-image}}\hfill
\subcaptionbox{caption}{\includegraphics[width=.3\textwidth]{example-image}}
\caption{A selection of the slides made during the user study}\label{slides}
\end{figure}
\end{document}
