There are two problems here:
Nested table environments: Not in outer par mode
I suggest dropping the inner table environment. (How should the concept of floating be applicable inside a tabular where the contents are fixed, anyway?)
Nested tabularx environments:
Extra }, or forgotten \endgroup,
Missing \endgroup inserted and
Missing } inserted
The solution is to enclose the inner tabularx by a pair of braces.
The the tabularx manual states:
tabular and tabular* environments may be nested with no restriction,
however if one tabularx environment occurs inside another, then the inner
one must be enclosed by { }.
Now the inner caption does work without problems and results in a enumerated caption:
Table 1: Inner table
If this is not preferred, one can use the starred version of \caption* for the inner caption from the caption package.
I also replaced \textwidth with \linewidth so that inner tabularx uses the correct remaining horizontal space (→ Difference between \textwidth, \linewidth and \hsize)
Further improvements could be
- enclosing the inner
tabularx and its caption inside the center environment or
- providing more vertical space before the inner
tabularx, i.e. \\[2ex].
Code
\documentclass{article}
\usepackage{blindtext}
\usepackage{tabularx}
\begin{document}
\begin{table}
\begin{tabularx}{\linewidth}{X}
\blindtext \\
{\begin{tabularx}{\linewidth}{X}
\blindtext
\end{tabularx}}
\caption{Inner table}
\\
\blindtext
\end{tabularx}
\caption{Outer table}
\end{table}
\end{document}
Output

tableenvironment (how should floating work inside atabularxanyway?) and drop the inner\caption(no enumeration) command. – Qrrbrbirlbel Dec 17 '12 at 04:17