Here are two solutions that use a tabularx environment to eliminate pointless guess-work as to how wide the 3rd and 4th columns should be. Both solutions make use of a macro called \breakingcomma, obtained from an earlier answer by @egreg, to permit TeX to break lines immediately after commas.
The former solution uses your setup of vertical and horizontal rules; the second uses no vertical rules and only well-spaced horizontal rules.

\documentclass{article}
\usepackage{tabularx,booktabs,ragged2e,breqn}
\renewcommand{\tabularxcolumn}[1]{m{#1}}
\newcolumntype{C}{>{\Centering\arraybackslash$}X<{$}}
\usepackage[skip=0.333\baselineskip]{caption}
% see https://tex.stackexchange.com/a/60761/5001 for source of \breakingcomma
% (requires the "breqn" package; that's why it's loaded above)
\newcommand{\breakingcomma}{%
\begingroup\lccode`~=`,
\lowercase{\endgroup\expandafter\def\expandafter~\expandafter{~\penalty0 }}}
\begin{document}
\begin{table}
\caption{Using both vertical and horizontal rules}
\breakingcomma
\begin{tabularx}{\textwidth}{|*{2}{>{$}c<{$}|}*{2}{C|}}
\hline
A & B & A.B & B.A\\
\hline
\{x^2,x^3,y,xy\} &
\{1,x,x^2y,x^3y\} &
\{x^2,x^3,y,xy,x^3,1,xy,x^2y,y,x^3y,x^2,x,xy,y,x^3,x^2\} &
\{x^2,x^3,y,xy,x^3,1,xy,x^2y,y,x^3y,x^2,x,xy,y,x^3,x^2\} \\ \hline
\end{tabularx}
\end{table}
\begin{table}[h]
\caption{Using only well-spaced horizontal rules}
\breakingcomma
\begin{tabularx}{\textwidth}{@{}*{2}{>{$}c<{$}}*{2}{C}@{}}
\toprule
A & B & A.B & B.A\\
\midrule
\{x^2,x^3,y,xy\} &
\{1,x,x^2y,x^3y\} &
\{x^2,x^3,y,xy,x^3,1,xy,x^2y,y,x^3y,x^2,x,xy,y,x^3,x^2\} &
\{x^2,x^3,y,xy,x^3,1,xy,x^2y,y,x^3y,x^2,x,xy,y,x^3,x^2\} \\
\bottomrule
\end{tabularx}
\end{table}
\end{document}
A.BandB.Amaybe be typeset asA\cdot BandB\cdot A? Please advise. – Mico Nov 30 '17 at 17:36