The example you showed has unbalanced braces and $.
Generally, if you want to to type multiline maths, it is not sufficient to separate the lines with blank line as you do in ordinary text. You need to use special environments like align, gather, split, aligned, gathered etc. The new lines / rows are signalled with \\ instead of blank lines (like you do in tables). New columns are signalled with the & (like you do in tables).
Is this what you wanted? I am guessing that you need a multi line environment inside a math environment in order to split the argument of the max into three lines.
(https://tex.stackexchange.com/a/112528/95229 and AMS math documentation -- 3.7
Alignment building blocks)
I have made only minimal adjustments since the meaning of the symbols in the equation are not known to me.
\documentclass{article}
\usepackage{amsmath}
\begin{document}
\begin{align}
Table[i,j] \gets \max\left(
\begin{gathered}[c]
Table[i-1, j]\\
\mathrm{AND}\\ %mathrm because It is an operator?
U[i] + Table\Big[{i-1}, \mathit{taskarea\textnormal{-}HS[i]}\Big]
\end{gathered}
\right)
\end{align}
\begin{align}
Table[i,j] \gets \max\left(
\begin{aligned}
& Table[i-1, j]\\
& \mathrm{AND}\\ %mathrm because It is an operator?
& U[i] + Table\Big[{i-1}, \mathit{taskarea\textnormal{-}HS[i]}\Big]
\end{aligned}
\right)
\end{align}
\begin{align}
Table[i,j] \gets \max\left(
\begin{aligned}
Table[i-1, j]\\
\mathrm{AND}\\ %mathrm because It is an operator?
U[i] + Table\Big[{i-1}, \mathit{taskarea\textnormal{-}HS[i]}\Big]
\end{aligned}
\right)
\end{align}
\end{document}

\documentclass{...}and ending with\end{document}. – AJN Apr 27 '16 at 06:15