The problem is in the implementation of the mathematical function veclen. In short, it doesn't just do (x^2 + y^2)^{1/2} (How could it? Taking square roots isn't even possible theoretically let alone the issues of precision.) but does something more complicated that at some stage involves dividing by one of the components. You can test this by trying a simple:
\pgfmathparse{veclen(0.00006,0.00006)}
(the number was chosen as that happens to be the number that TeX barfs on when trying Altermundus' example. Multiply by 10 and it all works again.)
Looking at the implementation of veclen I noticed that it did some initial scaling to take into account large numbers. So I added some lines to test also for small numbers as well. And that appears to solve the problem.
(Unfortunately, I couldn't get \pgfmathredeclarefunction to work properly. Rather than debug that, I just \let the old function to \relax and overwrote it.)
Code:
\documentclass{standalone}
\usepackage{tikz}
\usetikzlibrary{decorations.markings}
\makeatletter
\let\pgfmath@function@veclen\relax
\pgfmathdeclarefunction{veclen}{2}{%
\begingroup%
\pgfmath@x#1pt\relax%
\pgfmath@y#2pt\relax%
\ifdim\pgfmath@x<0pt\relax%
\pgfmath@x-\pgfmath@x%
\fi%
\ifdim\pgfmath@y<0pt\relax%
\pgfmath@y-\pgfmath@y%
\fi%
\ifdim\pgfmath@x=0pt\relax%
\pgfmath@x\pgfmath@y%
\else%
\ifdim\pgfmath@y=0pt\relax%
\else%
\ifdim\pgfmath@x>\pgfmath@y%
\pgfmath@xa\pgfmath@x%
\pgfmath@x\pgfmath@y%
\pgfmath@y\pgfmath@xa%
\fi%
% We use a scaling factor to reduce errors.
% First, see if we should scale down
\let\pgfmath@tmp@scale=\divide
\let\pgfmath@tmp@restore=\multiply
\ifdim\pgfmath@y>10000pt\relax%
\c@pgfmath@counta1500\relax%
\else%
\ifdim\pgfmath@y>1000pt\relax%
\c@pgfmath@counta150\relax%
\else%
\ifdim\pgfmath@y>100pt\relax%
\c@pgfmath@counta50\relax%
\else%
% Not scaling down, should we scale up?
\let\pgfmath@tmp@scale=\multiply
\let\pgfmath@tmp@restore=\divide
\ifdim\pgfmath@y<0.00001pt\relax%
\c@pgfmath@counta1500\relax%
\else%
\ifdim\pgfmath@y<0.0001pt\relax%
\c@pgfmath@counta150\relax%
\else%
\ifdim\pgfmath@y<0.001pt\relax%
\c@pgfmath@counta50\relax%
\else
\c@pgfmath@counta1\relax%
\fi%
\fi%
\fi%
\fi%
\fi%
\fi%
\pgfmath@tmp@scale\pgfmath@x\c@pgfmath@counta\relax%
\pgfmath@tmp@scale\pgfmath@y\c@pgfmath@counta\relax%
\pgfmathreciprocal@{\pgfmath@tonumber{\pgfmath@y}}%
\pgfmath@x\pgfmathresult\pgfmath@x%
\pgfmath@xa\pgfmath@tonumber{\pgfmath@x}\pgfmath@x%
\edef\pgfmath@temp{\pgfmath@tonumber{\pgfmath@xa}}%
%
% Use A+x^2*(B+x^2*(C+x^2*(D+E*x^2)))
% where
% A = +1.000012594
% B = +0.4993615349
% C = -0.1195159052
% D = +0.04453994279
% E = -0.01019210944
%
\pgfmath@x-0.01019210944\pgfmath@xa%
\advance\pgfmath@x0.04453994279pt\relax%
\pgfmath@x\pgfmath@temp\pgfmath@x%
\advance\pgfmath@x-0.1195159052pt\relax%
\pgfmath@x\pgfmath@temp\pgfmath@x%
\advance\pgfmath@x0.4993615349pt\relax%
\pgfmath@x\pgfmath@temp\pgfmath@x%
\advance\pgfmath@x1.000012594pt\relax%
\ifdim\pgfmath@y<0pt\relax%
\pgfmath@y-\pgfmath@y%
\fi%
\pgfmath@x\pgfmath@tonumber{\pgfmath@y}\pgfmath@x%
% Invert the scaling factor.
\pgfmath@tmp@restore\pgfmath@x\c@pgfmath@counta\relax%
\fi%
\fi%
\pgfmath@returnone\pgfmath@x%
\endgroup%
}
\makeatother
\begin{document}
\pgfmathparse{veclen(0.00006,0.00005)}
Vector length is: \pgfmathresult
\begin{tikzpicture}[decoration={markings, mark = at position .5 with
{\draw (-2pt,-2pt) -- (2pt,2pt) (2pt,-2pt) -- (-2pt,2pt);}}]
% wrong
\draw [postaction={decorate}] (0,0) -- ++(146:1) arc (146:157:1) -- (0,0);
%fine
\draw [postaction={decorate}] (2,0) -- ++(146:1.2) arc (146:157:1.2) -- (2,0);
\end{tikzpicture}
\end{document}
Result:

As can be seen, the accuracy is not great! However, at that level of precision then it's probably not all that important. Perhaps a better implementation would be to test if the components of the vector are less than some small number and then simply return the maximum of the two: at that level, the difference between the sup norm and the l^2 norm is not a lot! (A slightly more sophisticated version would have a switch that returned the sup norm or the l^1 norm depending on whether it was acceptable to underestimate or overestimate.)
fixedpointarithmeticand the problem is similar. The little difference is that I want to put a mark in the middle of the arc. Perhaps I can avoiddecoration? – Alain Matthes Jun 15 '11 at 16:42fixedpointarithmetic? – Alain Matthes Jun 16 '11 at 04:43scale=2as an option to thetikzpictureto see it better, and then it works, but not without thescale. – Jake Jun 16 '11 at 05:30scale=.8the second line gives an error ! – Alain Matthes Jun 16 '11 at 06:19