The problem with the tabularx environment is, that it is not an actual environment. It is a wrapped command somehow denying any \end after the actual \endtabularx. The solution is to use the command form for both table and tabularx.
\newenvironment{xtable}[1]{
\table
\tabularx{\linewidth}{#1}}
{\endtabularx
\endtable}
This is a complete Tex Sample for the table below.
\documentclass[a4paper,11pt]{report}
\usepackage[english]{babel}
\usepackage[latin1]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[table]{xcolor}
\usepackage{tabularx}
%Colors
\definecolor{colorTabHead}{gray}{0.3}
\definecolor{colorTab1}{gray}{0.92}
\definecolor{colorTab2}{gray}{0.88}
\definecolor{white}{gray}{1.00}
%Table header
\newcommand{\sthead}[1]
{\cellcolor{colorTabHead}
\textcolor{white}{\sffamily\bfseries #1}}
%New centered X column named Y
\newcolumntype{Y}{>{\centering\arraybackslash}X}
%Table environment
\newenvironment{xtable}[3]{
\table
\renewcommand{\arraystretch}{1.20}
\caption{#2 \label{#3}}
\bigskip
\rowcolors{1}{colorTab1}{colorTab2}
\small\tabularx{\linewidth}{#1}}
{\endtabularx
\endtable}
\begin{document}
\begin{xtable}{XY}{XTable}{tab:fox}
\sthead{X Column} & \sthead{Y Column}\\
The quick brown fox jumps over \newline the lazy dog & The quick brown fox jumps over the lazy dog\\
\end{xtable}
\end{document}
Results in:

captionattached to the table and not have the stragely floatingcaptionof. – Xerusial Aug 22 '18 at 09:18