The first argument of wrapfigure requires curly brackets {}, so it must be {#1}, and the same is true in the new environment. So it becomes \begin{wraptikz}{r}{5}{20}{2}.
The square brackets [] are reserved for optional arguments, but you're not loading any. If you want an optional argument, change the environment definition to
\newenvironment{wraptikz}[4][*]{%
\begin{wrapfigure}{#1}{#2cm}
But instead of the asterisk, you must enter the default value. For example [r] makes r the default value of the first argument. In this case, you can use square brackets when entering the new environment in the document. Note that wrapfigure still uses the braces in the environment definition.
By the way, wrapfigure requires the wrapfig package.
Output

Code
\documentclass{article}
\usepackage{wrapfig}
\usepackage{tikz}
\usepackage{lipsum}
\newenvironment{wraptikz}[4]{%
\begin{wrapfigure}{#1}{#2cm}
\vspace{-#3pt}
\begin{tikzpicture}[scale=#4]
}{%
\end{tikzpicture}
\end{wrapfigure}
}
\begin{document}
\begin{wraptikz}{r}{5}{20}{2}
\draw (0, 0) -- (5, 0);
\end{wraptikz}
\lipsum[1]
\end{document}