Ok, I think I came to some sort of a workaround - which looks OK (except the dimension line endpoints don't seem to be exactly at node centers):

... achieved by trying to throw in any \pgf* command I could find in the manual, related to coordinate system transformations, randomly in the code in brute force fashion, and then seeing their effect (see comments in MWE below)
The trick is, essentially, to do this:
decoration={markings,
mark=at position 0 with {%
\pgftransformresetnontranslations %
\pgftransformscale{#4} %
\arrow[sloped,rotate=0,scale=1,fill=#5]{#1}; %
},
mark=at position 1 with {%
%\pgftransformresetnontranslations % NOT HERE!
\pgftransformscale{#4} %
\arrow[sloped,scale=1,fill=#5]{#2};
},
},
Why? Beats me - I have no idea; especially since if I print out the transformation matrices from these points in the code, they are exactly the same (the scaling part, not the translation one), no matter where they're called from; the MWE below prints out:
pos 0: pgfgettransform: {-1.0}{0.0}{0.0}{-1.0}{90.95824pt}{62.30548pt}; pgf@pt@
aa, pgf@pt@ba, pgf@pt@ab, pgf@pt@bb: -1.0, 0.0, 0.0, -1.0
pos 1: pgfgettransform: {-1.0}{0.0}{0.0}{-1.0}{33.65276pt}{62.30548pt}; pgf@pt@
aa, pgf@pt@ba, pgf@pt@ab, pgf@pt@bb: -1.0, 0.0, 0.0, -1.0
pos 0: pgfgettransform: {-1.0}{0.0}{0.0}{-1.0}{90.95824pt}{42.30545pt}; pgf@pt@
aa, pgf@pt@ba, pgf@pt@ab, pgf@pt@bb: -1.0, 0.0, 0.0, -1.0
pos 1: pgfgettransform: {-1.0}{0.0}{0.0}{-1.0}{33.65276pt}{42.30545pt}; pgf@pt@
aa, pgf@pt@ba, pgf@pt@ab, pgf@pt@bb: -1.0, 0.0, 0.0, -1.0
Well, hope someone will eventually clarify this properly - in the meantime, I hope this was it with dimension arrow problems.
The MWE:
\documentclass[varwidth,tightpage,border=1bp]{standalone}
\usepackage{tikz}
\usetikzlibrary{decorations}
\usetikzlibrary{decorations.markings}
\usetikzlibrary{shapes}
\usetikzlibrary{calc}
\pagecolor{yellow!15} % ignored with preview, but not w/ varwidth
\begin{document}
Hello:
\begin{tikzpicture}
\makeatletter
% http://tex.stackexchange.com/questions/185444/
\gdef\tout{}\gdef\trfm{}
\gdef\typeoutMyInfoString#1{%
\pgfgettransform{\trfm}%
\edef\tout{#1 %
pgfgettransform: \trfm ; %
pgf@pt@aa, pgf@pt@ba, pgf@pt@ab, pgf@pt@bb: \pgf@pt@aa, \pgf@pt@ba, \pgf@pt@ab, \pgf@pt@bb %
}
\typeout{\tout} %
}
\makeatother
\gdef\myset{
\tikzset{ %
%instead of dimen/.style={, using tconarrC for dimen!
declare function={multip(\a,\b)=\a*\b;}, % otherwise cannot use pgfmathparse here
tconarrC/.style n args={5}{
line width=##3,
draw=##5,
decoration={markings,
mark=at position 0 with {%
\typeoutMyInfoString{pos 0:}%
% (c0)
\pgftransformresetnontranslations %
% (c1)
\pgftransformscale{##4} % this works the same for both;
% so don't scale the arrow
\arrow[sloped,rotate=0,scale=1,fill=##5]{##1}; % (c3)
},
mark=at position 1 with {% 0.999999
% here we seemingly have correct orientation between M and A;
% just the scale is different; so only scale needed.
\typeoutMyInfoString{pos 1:}%
%\pgftransformresetnontranslations % causes both arrows to
% rotate/orient wrongly!
% without it, all is correct
\pgftransformscale{##4} % this works the same for both;
% so don't scale the arrow
\arrow[sloped,scale=1,fill=##5]{##2};
},
},
shorten >={multip(##4,##3)},
shorten <={multip(##4,##3)},
reset transform/.code={},%\pgftransformreset}, %is called anyway
%decorate, % cannot, no line is drawn
%preaction={reset transform,decorate}, % meh
postaction={reset transform,decorate},
},
tconarrC/.default={latex}{latex}{3pt}{1.3}{black},
dimen/.style={|-|,
tconarrC={latex}{latex}{\pgflinewidth}{2.0}{gray},
every rectangle node/.style={midway,},
},
minbox/.style={shape=rectangle,inner sep=0pt,outer sep=0pt,minimum size=0pt},
} % end tikzset
} % end gdef
\myset % execute
\makeatletter
\pgfdeclareshape{myshape}{ %
\inheritsavedanchors[from={rectangle}] %
\inheritbackgroundpath[from={rectangle}] %
\inheritanchorborder[from={rectangle}] %
\foreach \x in {center,north,north east,north west,south,south east,south west,east,west}{ %
\inheritanchor[from={rectangle}]{\x} %
} %
%\myset % ! Package pgfkeys Error: I do not know the key '/tikz/dimen'
\backgroundpath{ %
\edef\ename{\tikz@fig@name} %
\node[rectangle,minimum size=10pt,draw] (\ename-A) at (\ename.east) {};
\node[rectangle,minimum size=10pt,draw] (\ename-B) at (\ename.west) {};
}
\foregroundpath{ %
%\myset % ! Package pgfkeys Error: I do not know the key '/tikz/dimen'
% here, minbox must have a shape (e.g. rectangle) defined!
% else ! TeX capacity exceeded, sorry [grouping levels=255].
\draw[dimen] (\ename-A.center) -- (\ename-B.center)
node[minbox,above=0pt] (\ename-L) {M}
;
}
} % declareshape
\makeatother
\node[fill] (origin) at (0,0) {};
\node[myshape,minimum size=2cm,anchor=south west] (tshape) at (1,1) {};
\draw[dimen]
($(tshape-A.center)-(0,2em)$) -- ($(tshape-B.center)-(0,2em)$)
node[above=0pt] {A};
\end{tikzpicture}
\end{document}
% (c0)
% \pgfgettransformentries{\ta}{\tb}{\tc}{\td}{\tx}{\ty} %
% \pgfusepath{draw,stroke} %
% \pgfinterruptpath % nothing
%\tikz\draw[-latex](\tx,\ty) ; % completely messes up sizing
% \draw[-latex](0pt,0pt) ; % sizing presserved; \tx,\ty moves to bottom left corner; 0,0 is relative; arrows points "up" like this, unscaled! ,rotate=90 is ignored
% \node[draw=none,rotate=180,inner sep=0pt] (ti) {\tikz\draw[-latex](0pt,0pt);}; % somewhat messed up; same at at (0pt,0pt); M leaks?
% (c1)
% \node[rotate=-90,inner sep=0pt,outer sep=0pt,minimum size=0pt,anchor=west] (ti) {\tikz\draw[-latex](0pt,0pt);}; % rotate works here, but opposite directions A and M? with \pgftransformresetnontranslations they finally orient the same!
% \node[draw=none,rotate=180,inner sep=0pt,anchor=center] (ti) {\tikz\arrow{latex};}; % somewhat messed up
% \draw[-latex,rotate=180](0pt,0pt); % same rotation, but rotate ignored!
% (c3)
% with rotate=180, A is correct, M ignores; with rotate=0, A only rotates, M stays same (bad) ; with \pgftransformresetnontranslations and rotate=0, finally both arrows point the same way correctly
% \endpgfinterruptpath %