Here's one approach using decorations. For each segment, the decoration automaton checks whether the target radius of the rounded corners is larger than half the length of the segment. If it is, the corner radius is set to half the length of the segment, otherwise it is left untouched. In some cases, this approach will reduce the corner radius unnecessarily, but it avoids having to analyse the complete path in order to determine the maximum possible radius for each corner:

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{decorations}
\pgfdeclaredecoration{dynamic rounded corners}{initial}{
\state{initial}[width=\pgfdecoratedinputsegmentlength,
next state=middle,
persistent postcomputation=\pgfmathsetmacro\previousroundedendlength{min(\pgfdecorationsegmentlength,\pgfdecoratedinputsegmentlength)}
]{}
\state{middle}[width=\pgfdecoratedinputsegmentlength,next state=middle,
persistent precomputation={
\pgfmathsetmacro\roundedstartlength{min(\previousroundedendlength,\pgfdecoratedinputsegmentlength/2)}
},
persistent postcomputation=\pgfmathsetmacro\previousroundedendlength{min(\pgfdecorationsegmentlength,\pgfdecoratedinputsegmentlength/2)}
]{
\pgfsetcornersarced{\pgfpoint{\roundedstartlength}{\roundedstartlength}}
\pgfpathlineto{\pgfpoint{0pt}{0pt}}
}
\state{final}[if input segment is closepath={\pgfpathclose}]
{
\pgfpathlineto{\pgfpointdecoratedpathlast}
}
}
\tikzset{
dynamic rounded corners/.style={
decorate,
decoration={
dynamic rounded corners,
segment length=#1
}
}
}
\begin{document}
\begin{tikzpicture}[every node/.style={below right, align=center}]
\draw [xshift=-1.5cm, gray!50] (0,0) node {Original\\Path} -- (2,0) -- (1,1) -- (1,3) -- (3,3);
\draw [rounded corners=1cm] node {Normal\\rounded\\corners} (0,0) -- (2,0) -- (1,1) -- (1,3) -- (3,3);
\draw [dynamic rounded corners=1cm,red,xshift=1.5cm] (0,0) node {Dynamic\\rounded\\corners}-- (2,0) -- (1,1) -- (1,3) -- (3,3);
\end{tikzpicture}
\end{document}
[rounded corners]and not specifying a value? This seems to work for this particular example. – Peter Grill Oct 23 '11 at 02:46rounded cornersis4pt, which is small enough. But I'd like to be able to specify a value for the corners without having to worry about whether there are points on my path that are too close together. – Jake Oct 23 '11 at 02:57\pgfprocess@specialroundinbasiclayer/pgfcorepathprocessing.code.tex. I guess you need some calculation in there to prevent the curve becoming too big. – Andrew Stacey Oct 25 '11 at 10:52