This is a problem with the fpu library that is used by pgfplots: The ifthenelse command is not implemented in fpu, so it falls back to the normal pgfmath routine, which then stumbles over the floating point format of the arguments, because numbers are handled in the form 1Y1.0e0].
To circumvent this, the fpu library can be disabled in the newly defined math function using
\pgfkeys{/pgf/fpu=false}
However, the x-value handed over by pgfplots is still in the floating point format, so it has to be converted into a fixed point format:
\pgfmathfloattofixed{\x}
\let\x=\pgfmathresult
In case the fpu library is not active, this will fail because \pgfmathfloattofixed{\x} doesn't know what to do with a fixed point number. So we need to use \pgflibraryfpuifactive to test whether fpu is active, and convert the number only if it is.
Now x is a normal fixed point number, and all the common pgfmath functions should work.
However, the pgfmath engine is less precise than fpu, so it might not be desirable to fall back to it.
As ifthenelse is not a very complicated function, we can just define it ourselves:
\pgfmathdeclarefunction{ifthenelsefpu}{3}{ %}
\pgfmathparse{#1*#2 + !#1*#3} %
}
Which can be used like the original function.
I've put both approaches into the example document below:
\documentclass{article}
\usepackage{pgfplots}
\pgfmathdeclarefunction{ifthenelsefpu}{3}{%
\pgfmathparse{#1*#2 + !#1*#3}%
}
\pgfmathdeclarefunction{f}{1}{%
\pgfmathparse{ifthenelsefpu(#1<0,#1^2,#1)}%
}
\pgfmathdeclarefunction{g}{1}{%
\pgflibraryfpuifactive{%
\pgfkeys{/pgf/fpu=false}%
\pgfmathfloattofixed{#1}%
\let\x=\pgfmathresult%
}%
{%
\pgfmathparse{#1}%
\let\x=\pgfmathresult%
}%
\pgfmathparse{ifthenelse(\x<0,(\x)^2,\x)}%
}
\begin{document}
\begin{tikzpicture}
\begin{axis}[every axis plot post/.append style={
mark=none,domain=-3:3,smooth},
axis x line*=bottom, axis y line*=left, enlargelimits=upper]
\addplot {f(x+0.5)};
\addplot {g(x-1)};
\end{axis}
\end{tikzpicture}
%Code for showing that the functions work outside pgfplots
\pgfmathf{-3}\pgfmathresult
\pgfmathg{-3}\pgfmathresult
\end{document}

\pgfmathdeclarefunctionrequire a number? – Alan Munn Apr 10 '11 at 16:32