As it happens, a complete and sufficiently general (for my purposes) answer was quite involved. There are a couple of nuances to deal with.
I provide in this extra answer both some of the problems I found and cut-and-paste code for the unlikely case that someone else is interested in this particular style, but I still think @Jake correctly answered the original question.
When using shorten with a negative value, the axis indeed do overshoot. However, the overshoot does not count toward the bounding box. If you compile a standalone figure, it may well happen that the overshot axis ends up on top of something else. This is dealt with by inserting two “spurious” nodes with before end axis. The x and y labels have to be shifted accordingly.
For this to work properly, you have to choose tick lines at xmin, xmax, ymin, ymax. However, if grid lines and/or ticks lie at the edge of the clip area, they are “half-clipped”, meaning they are half the width they sould be. (see the answer and comments to this question). I found a workaround by removing completely the clipping path from ticks and grids. This is not pretty.
As a consequence of 2, line cap should be round for grids, lest you end up with “ugly corners“ (see here).
As seen below, I implemented three keys overshoot amount, x overshoot and y overshoot. overshoot amount=<length> defines how large should be the overshoot. x overshoot can be set to left, right, noneor both , in case you want overshooting to happen at one side only (see second graphic below). Similarly, y overshootcan be set to top, bottom none or both.
Finally, I am still learning my tikz/pgfplots skills; I am sure my code is ugly. Someone with more pgfplots-fu will surely mend the code.
Code
\documentclass[border=0cm]{standalone}
\usepackage[]{pgfplots}
\makeatletter
\def\pgfplots@drawticklines@INSTALLCLIP@onorientedsurf#1{}%
\def\pgfplots@drawgridlines@INSTALLCLIP@onorientedsurf#1{}%
\makeatother
\pgfplotsset{compat=1.8,
%%%%
% Overshooting arrows from grid and ticks
overshoot amount/.store in=\overshootAmount,
overshoot amount=4pt,
%
x overshoot left/.store in=\xOvershootLeft,
x overshoot right/.store in=\xOvershootRight,
x overshoot/.is choice,
x overshoot/both/.style={x overshoot left=\overshootAmount, x overshoot right=\overshootAmount},
x overshoot/left/.style={x overshoot left=\overshootAmount, x overshoot right=0pt},
x overshoot/right/.style={x overshoot left=0pt, x overshoot right=\overshootAmount},
x overshoot/none/.style={x overshoot left=0pt, x overshoot right=0pt},
x overshoot=both,
%
y overshoot top/.store in=\yOvershootTop,
y overshoot bottom/.store in=\yOvershootBottom,
y overshoot/.is choice,
y overshoot/both/.style={y overshoot top=\overshootAmount, y overshoot bottom=\overshootAmount},
y overshoot/top/.style={y overshoot top=\overshootAmount, y overshoot bottom=0pt},
y overshoot/bottom/.style={y overshoot top=0, y overshoot bottom=\overshootAmount},
y overshoot/none/.style={y overshoot top=0pt, y overshoot bottom=0pt},
y overshoot=both,
%%%
enlargelimits=false,
every non boxed x axis/.style={
x axis line style={->,
black!60!white, thin, line cap=round,
shorten >=-\xOvershootRight,
shorten <=-\xOvershootLeft,
},
every axis x label/.style={},
xlabel style={anchor=west,
at={(ticklabel* cs:1.0)},
black!60!white,
xshift=\xOvershootRight
},
},
every non boxed y axis/.style={
y axis line style={->,
black!60!white,thin,
shorten >=-\yOvershootTop,
shorten <=-\yOvershootBottom,
},
every axis y label/.style={},
ylabel style={
rotate=0,
anchor=south,
at={(ticklabel* cs:1.0)},
black!60!white,
yshift=\yOvershootTop
},
},
before end axis/.code={
\node[inner sep=0pt,at={(rel axis cs:1,1)},xshift=\xOvershootRight, yshift=\yOvershootTop] {};
\node[inner sep=0pt,at={(rel axis cs:0,0)},xshift=-\xOvershootLeft, yshift=-\yOvershootBottom] {};
},
tick style={line cap=round},
minor tick style={line cap=round},
%
grid style={line cap=round},
minor grid style={line cap=round},
%
set layers=standard,
clip=false,
}
\begin{document}
\pgfplotsset{overshoot amount=10pt}
\begin{tikzpicture}[baseline]
\begin{axis}[
axis equal image=true,
axis x line=middle,
axis y line=middle,
xmin=-1, xmax=1,
ymin=-1, ymax=1,
%
xtick={-1,0,1}, %xticklabels={$-a$,$O$,$a$}, %(if needed)
ytick={-1,0,1},
minor tick num=3,
grid=both,
xlabel={$x$}
]
\addplot+[smooth] coordinates{ (-.8,.2) (-.3,-.6) (.3,.9) (.8,.7)} ;
\end{axis}
\end{tikzpicture}
%
\begin{tikzpicture}[baseline]
\begin{axis}[
axis equal image=true,
axis x line=middle,
axis y line=middle,
xmin=0, xmax=1, x overshoot=right,
ymin=0, ymax=1, y overshoot=top,
%
xtick={0,1}, %xticklabels={$-a$,$O$,$a$}, %(if needed)
ytick={0,1},
minor tick num=3,
grid=both,
xlabel={$x$}
]
\addplot+[smooth] coordinates{ (.1,.6) (.2,.2) (.3,.9) (.8,.7)} ;
\end{axis}
\end{tikzpicture}
\end{document}
Output

tick style={red,thick}shows tick at bottom (as expected) but not at top. – Koji Sep 30 '13 at 13:08grid=bothmeans put grid lines at each major and minor ticks. so you can force some of the ticks to have no line. – percusse Sep 30 '13 at 13:37enlarge x limitsto the grid? – Koji Sep 30 '13 at 13:40