First answer
\documentclass[a4paper]{book}
\usepackage{xparse,refcount,etoolbox}
\newcounter{comparerefs}
\newcommand{\comparerefs}[2]{%
\begingroup\edef\x{\endgroup
\noexpand\docomparerefs{\getrefnumber{#1}}{\getrefnumber{#2}}}\x}
\NewDocumentCommand{\docomparerefs}{ >{\SplitArgument{2}{.}}m >{\SplitArgument{2}{.}}m }{
\xcomparerefs#1#2}
\newcommand\xcomparerefs[6]{%
\ifboolexpr{ test {\ifstrequal{#1}{0}} or test {\ifstrequal{#4}{0}} }% reference is undefined
{\setcounter{comparerefs}{-1}}
{\ycomparerefs{#1}{#2}{#3}{#4}{#5}{#6}}%
}
\newcommand{\ycomparerefs}[6]{%
\ifboolexpr{ test {\ifstrequal{#3}{\NoValue}} and test {\ifstrequal{#6}{\NoValue}} }
{\zcomparerefs{#1}{#2}{#4}{#5}}
{\subseccomparerefs{#1}{#2}{#3}{#4}{#5}{#6}}%
}
\newcommand{\zcomparerefs}[4]{%
\ifboolexpr{ test {\ifstrequal{#2}{\NoValue}} and test {\ifstrequal{#4}{\NoValue}} }
{\chapcomparerefs{#1}{#3}}
{\seccomparerefs{#1}{#2}{#3}{#4}}%
}
\newcommand{\chapcomparerefs}[2]{%
\ifnum#1=#2\relax
\setcounter{comparerefs}{0}%
\else
\ifnum#1=\numexpr#2-1\relax\relax
\setcounter{comparerefs}{1}%
\else
\ifnum#1=\numexpr#2+1\relax\relax
\setcounter{comparerefs}{2}%
\else
\setcounter{comparerefs}{3}%
\fi
\fi
\fi
}
\newcommand{\seccomparerefs}[4]{%
\ifnum#1=#3\relax
\chapcomparerefs{#2}{#4}%
\else
\setcounter{comparerefs}{3}%
\fi}
\newcommand{\subseccomparerefs}[6]{%
\ifnum#1=#4\relax
\seccomparerefs{#2}{#3}{#5}{#6}%
\else
\setcounter{comparerefs}{3}%
\fi
}
\begin{document}
\chapter{Abc}\label{x}
\section{Abc}
\subsection{Def}\label{A}
\subsection{Ghi}\label{B}
\section{XXX}
\subsection{UUU}\label{C}
\chapter{YYY}\label{y}
\chapter{ZZZ}\label{z}
%\comparerefs{x}{x}\showthe\value{comparerefs}
%\comparerefs{x}{y}\showthe\value{comparerefs}
%\comparerefs{y}{x}\showthe\value{comparerefs}
%\comparerefs{x}{z}\showthe\value{comparerefs}
\comparerefs{A}{A}\showthe\value{comparerefs}
\comparerefs{A}{B}\showthe\value{comparerefs}
\comparerefs{B}{A}\showthe\value{comparerefs}
\comparerefs{A}{C}\showthe\value{comparerefs}
\end{document}
I use the \SplitArgument feature of xparse to get the references already split in components. I support only up to three levels (subsections) and there is no check whether the references are not compatible.
If one of the references is not already resolved, the counter will be set to -1.
First we decide whether the references are to chapters, sections or subsections. So \chapcomparerefs is the main one and sets the counter to the requested value. In case of \seccomparerefs we check if the first components are equal and then we pass control to \chapcomparerefs on the second components, otherwise we set the counter to 3. Similarly for \subseccomparerefs which sets the counter to 3 if the first components are different, otherwise calls \seccomparerefs.
In case you compare different kinds of references you get a "Missing number" error.
Second answer
\documentclass[a4paper]{book}
\usepackage{xparse,refcount,etoolbox}
\newcounter{comparerefs}
\newcommand{\comparerefs}[2]{%
\begingroup\edef\x{\endgroup
\noexpand\docomparerefs{\getrefnumber{#1}}{\getrefnumber{#2}}}\x}
\NewDocumentCommand{\docomparerefs}{ >{\SplitArgument{20}{.}}m >{\SplitArgument{20}{.}}m }{
\setcounter{comparerefs}{0}\begingroup
\def\NoValue{-1000}\edef\x{\endgroup
\noexpand\xcomparerefs#1\relax#2\relax}\x}
\def\xcomparerefs#1#2#3\relax#4#5#6\relax{%
\ycomparerefs{#1}{#2}{#4}{#5}{{#2}#3\relax{#5}#6\relax}}
\def\ycomparerefs#1#2#3#4#5{%
\ifboolexpr{ test {\ifnumcomp{#1}{=}{#3}} and
not ( test {\ifnumcomp{#2}{=}{-1000}} and test {\ifnumcomp{#4}{=}{-1000}} ) }
{\xcomparerefs#5}
{\decide{#1}{#2}{#3}{#4}}%
}
\def\decide#1#2#3#4{%
\ifboolexpr{ test {\ifnumcomp{#2}{=}{-1000}} and test {\ifnumcomp{#4}{=}{-1000}} }
{\ifnumcomp{#1}{=}{#3}{}{\xdecide{#1}{#3}}}
{\ifnumcomp{#1}{=}{#3}{\xdecide{#2}{#4}}{\setcounter{comparerefs}{3}}}%
}
\def\xdecide#1#2{%
\ifnumcomp{#1}{=}{#2-1}
{\setcounter{comparerefs}{1}}
{\ifnumcomp{#1}{=}{#2+1}
{\setcounter{comparerefs}{2}}
{\setcounter{comparerefs}{3}}%
}
}
\begin{document}
\chapter{Abc}\label{x}
\section{Abc}
\subsection{Def}\label{A}
\subsection{Ghi}\label{B}
\section{XXX}
\subsection{UUU}\label{C}
\chapter{YYY}\label{y}
\chapter{ZZZ}\label{z}
\comparerefs{x}{x}\showthe\value{comparerefs}
\comparerefs{x}{y}\showthe\value{comparerefs}
\comparerefs{y}{x}\showthe\value{comparerefs}
\comparerefs{x}{z}\showthe\value{comparerefs}
\comparerefs{A}{A}\showthe\value{comparerefs}
\comparerefs{A}{B}\showthe\value{comparerefs}
\comparerefs{B}{A}\showthe\value{comparerefs}
\comparerefs{A}{C}\showthe\value{comparerefs}
\comparerefs{A}{x}\showthe\value{comparerefs}
\end{document}
These macros allow for a maximum of twenty levels (which should be more than sufficient). Again xparse is used to get two lists of arguments from the references. I assume that no reference value is -1000. The result of the test is stored in the counter comparerefs.
Third answer
This is the expandable version of the second answer (the \edef will testify this assertion). As a byproduct, the length of the strings is unlimited, provided they are all of the stated form, that is, numbers separated by periods. None of the numbers should be -1000, the special value used for terminating the recursion.
\documentclass[a4paper]{book}
\usepackage{refcount,etoolbox}
\newcommand{\comparerefs}[2]{%
\expandafter\comparerefsA\expandafter{\romannumeral-`a\getrefnumber{#2}}{#1}}
\def\comparerefsA#1#2{%
\expandafter\comparerefsB\expandafter{\romannumeral-`a\getrefnumber{#2}}{#1}}
\def\comparerefsB#1#2{%
\xcomparerefs#1.-1000\relax#2.-1000\relax}
\def\xcomparerefs#1.#2.#3\relax#4.#5.#6\relax{%
\ycomparerefs{#1}{#2}{#4}{#5}{#2.#3.-1000.-1000\relax#5.#6.-1000.-1000\relax}}
\def\ycomparerefs#1#2#3#4#5{%
\ifboolexpe{ test {\ifnumcomp{#1}{=}{#3}} and
not ( test {\ifnumcomp{#2}{=}{-1000}} and test {\ifnumcomp{#4}{=}{-1000}} ) }
{\xcomparerefs#5}
{\decide{#1}{#2}{#3}{#4}}%
}
\def\decide#1#2#3#4{%
\ifboolexpe{ test {\ifnumcomp{#2}{=}{-1000}} and test {\ifnumcomp{#4}{=}{-1000}} }
{\ifnumcomp{#1}{=}{#3}{0}{\xdecide{#1}{#3}}}
{\ifnumcomp{#1}{=}{#3}{\xdecide{#2}{#4}}{3}}%
}
\def\xdecide#1#2{%
\ifnumcomp{#1}{=}{#2-1}
{1}
{\ifnumcomp{#1}{=}{#2+1}
{2}
{3}%
}
}
\begin{document}
\chapter{Abc}\label{x}
\section{Abc}
\subsection{Def}\label{A}
\subsection{Ghi}\label{B}
\section{XXX}
\subsection{UUU}\label{C}
\chapter{YYY}\label{y}
\chapter{ZZZ}\label{z}
\comparerefs{x}{y}\par
\comparerefs{y}{x}\par
\comparerefs{x}{z}\par
\comparerefs{A}{A}\par
\comparerefs{A}{B}\par
\comparerefs{B}{A}\par
\comparerefs{A}{C}\par
\comparerefs{A}{x}\par
\end{document}
Comments on the "expandable" solution
The \comparerefs macro reads its arguments and first of all tries to get the expansions of \getrefnumber from them. I use the \romannumeral-`a trick, but twice: the \expandafter will reach \romannumeral, so from \comparerefs{X}{Y} we get
\comparerefsA{<expansion of \getrefnumber{Y}}{X}
that will be expanded to
\expandafter\comparerefsB\expandafter
{\romannumeral-`a\getrefnumber{X}}
{<expansion of \getrefnumber{Y}}
that will become
\comparerefsB{<expansion of \getrefnumber{X}}{<expansion of \getrefnumber{Y}}
Say that \getrefnumber{X} expands to 1.1.1 and \getrefnumber{Y} expands to 1.2 (different length); in this case the successive expansion will be
\xcomparerefs 1.1.1.-1000.-1000\relax 1.2.-1000.-1000\relax
The macro \xcomparerefs just picks up the numbers to compare and we'll get
\ycomparerefs{1}{1}{1}{2}{1.1.-1000\relax 2.-1000\relax}
The fifth argument will be used in case the recursion is started, to be fed again to \xcomparerefs (we add a -1000 at the end to never run out of numbers, as the two sequences can be of different length).
The recursion is started if #1=#3 and both #2 and #4 are different from -1000: in this case we can't decide the lexicographic order, so we chop off the first component and restart.
The final stage will happen when the above condition is not satisfied. If both #2 and #4 are -1000, we output 0 if #1 and #2 are equal (the two original sequences are indeed equal). Otherwise we choose what number to output by comparing #1 and #3.
If #2 and #4 are not both -1000 we again compare #1 and #3: if they are different the two sequences are "far away from each other" and we output 3; otherwise we choose the output by comparing #2 and #4.
The decision macro looks at the two numbers and outputs 1 if the first is one less than the second, 2 if the first is one more than the second and 3 in all other cases.
All is expandable thanks to the \ifboolexpe macro provided by etoolbox.
pgfkeys, I'm giving up. Are you sure it has to be expandable? Doing something like\comparerefs{refone}{reftwo}{\result}and then working with the expandable macro\resultmay be enough unless this is nested deeply in some other expansion-only context. – Ryan Reich Nov 12 '11 at 21:31