Accepting special characters in arguments passing them further

Question

I would like to define a macro very similar to \url, that accepts more or less arbitrary strings (including hashes/#) and passes them to other commands (i.e., \href, \url, \nolinkurl). I have tried to apply some other solutions (e.g., changing catcodes locally etc.), but failed. I would also like my new macro to be passable to other macros like \footnote.

The following non-working MWE shows an implementation, taking arbitrary strings as inputs, converts them to escaped tex-compatible strings (by use of \tl_to_str:n as an xparse command processor), and passes them along without errors. However, this completely breaks the meaning of the macros, which just output the escaped text (I don't really understand why).

Ideally, the macro would even detect hashes and escape them when passing the respective argument to \href, because that's required (but not for the original \url).

\documentclass{article}
\PassOptionsToPackage{hyphens}{url}
\usepackage[hidelinks]{hyperref}

\usepackage{xparse}
\usepackage{letltxmacro}

\ExplSyntaxOn
\LetLtxMacro\oldurl\url
\RenewDocumentCommand\url{>{\tl_to_str:n}m>{\tl_to_str:n}g}{%
    \IfNoValueTF{#2}{%
      \oldurl{#1}%
    }{%
      \href{#1}{#2}%
    }%
}
\ExplSyntaxOff

\setlength{\parindent}{0pt}
\begin{document}

\smallskip
\noindent\rule{\linewidth}{1ex}

\smallskip
url\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break/#me}

\smallskip
2url\#1\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break/me/please}

\smallskip
2url\#2\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/please}{www.yahoo.com/break/me/#please}

\smallskip
2url\#3\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break-me/#please}

\noindent\rule{\linewidth}{1ex}

footnote: \footnote{\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break-me/#please}}
\end{document}

Yes (but also without functionality, i.e. without clickable links). This is with pdflatex from TL 2016. Do you get any errors? I have also attached a screenshot of the output. — stefanct, Jul 12 '19 at 16:07
But it does no longer in TL 2018 but stops with Illegal parameter number in definition of \Hy@tempa. — stefanct, Jun 27 '20 at 07:12

Phelype Oleinik · Accepted Answer · 2020-06-27T13:42:54.823

You can't place arbitrary commands in argument processors. An argument processor must leave its results in \ProcessedArgument, so instead of:

\RenewDocumentCommand\url{>{\tl_to_str:n}m}{...}

you'd need:

\cs_new_protected:Npn \Detokenize #1
  { \tl_set:Nx \ProcessedArgument { \tl_to_str:n {#1} } }
\RenewDocumentCommand\url{>{\Detokenize}m}{...}

That said, \detokenize doubles the hashes, so you need to un-double them to get the right URL. I defined an \UndoubleHashes argument processor which turns the argument into a string with \tl_to_str:n, and then loops through it looking for double hashes and replacing them by single ones.

Also, you have to make sure that special characters have the catcode that \url expects, so you need to make \^^M, \%, \#, \&, \_, \~, \$ active (but “other” would also work).

With this change, the tooltip of the PDF viewer shows the correct URL:

\documentclass{article}
\PassOptionsToPackage{hyphens}{url}
\usepackage[hidelinks]{hyperref}
\usepackage{xparse}
\usepackage{letltxmacro}
\makeatletter
\DeclareRobustCommand*{\breakhref}[3][]{%
  \begingroup
    \setkeys{href}{#1}%
    \hyper@normalise\breakhref@{#2}{#3}}
\begingroup
  \lccode\~=#
  \lowercase{\endgroup
  \def\breakhref@#1{\expandafter\breakhref@split#1~~\}%
  \def\breakhref@split#1~#2~#3\#4{%
    \hyper@@link{#1}{#2}{#4}%
    \endgroup}}%
\makeatother
\ExplSyntaxOn
\cs_new_protected:Npn \UndoubleHashes #1
  {
    \tl_set:Nx \ProcessedArgument
      { \stefanct_undouble_hashes:n {#1} }
  }
\cs_new:Npx \stefanct_undouble_hashes:n #1
  {
    \exp_not:N \exp_last_unbraced:No
      \exp_not:N __stefanct_undouble_hashes:w { \exp_not:N \tl_to_str:n {#1} }
        \c_hash_str \c_hash_str \exp_not:N \q_nil \s_stop
  }
\use:x
  {
    \cs_new:Npn \exp_not:N __stefanct_undouble_hashes:w
        ##1 \c_hash_str \c_hash_str ##2
      {
        ##1
        \exp_not:N \quark_if_nil:nTF {##2}
          { \exp_not:N \use_none_delimit_by_s_stop:w }
          { \c_hash_str ##2 }
        \exp_not:N __stefanct_undouble_hashes:w
      }
  }
\tl_new:N \l__stefanct_tmpa_tl
\cs_generate_variant:Nn \tl_replace_all:Nnn { Nx }
\LetLtxMacro\oldurl\url
\RenewDocumentCommand\url { }
  {
    \group_begin:
      \tl_map_inline:nn { ^^M % # &amp; _ ~ $ }
        { \char_set_catcode_active:n { ##1 } } \char_set_catcode_other:n {# }
      \urlaux
  }
\group_begin:
\char_set_catcode_active:n { \# } \char_set_catcode_parameter:n {$ }
\cs_new_protected:Npn \stefanct_url:nn $1 $2
  {
    \tl_set:Nn \l__stefanct_tmpa_tl {$2}
    \tl_replace_all:Nxn \l__stefanct_tmpa_tl { \c_hash_str } { # }
    \char_set_active_eq:NN # __stefanct_breakable_hash:
    \IfNoValueTF {$2}
      { \oldurl {$1} }
      { \exp_args:NnV \breakhref {$1} \l__stefanct_tmpa_tl }
    \group_end:
  }
\group_end:
\cs_new_protected:Npn __stefanct_breakable_hash:
  {
    \penalty \UrlBreakPenalty
    \c_hash_str
  }
\NewDocumentCommand\urlaux{>{\UndoubleHashes}m>{\UndoubleHashes}g}
  { \stefanct_url:nn {#1} {#2} }
\ExplSyntaxOff
\setlength{\parindent}{0pt}
\begin{document}
\smallskip
\noindent\rule{\linewidth}{1ex}
\smallskip
url\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break/#me%too}
\smallskip
url\hfill\hspace{0.6\linewidth}\url{httpswwwyahoocombreak#me%too}
\smallskip
2url#1\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break/me/please}
\smallskip
2url#2\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/please}{www.yahoo.com/break/me/#please}
\smallskip
2url#3\hfill\hspace{0.6\linewidth}\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break-me/#please}
\noindent\rule{\linewidth}{1ex}
footnote: \footnote{\url{https://www.yahoo.com/break-me/#please}{www.yahoo.com/break-me/#please}}
\end{document}

The code allows breaking the URL before # signs. To have it break after a #, change the order of \penalty \UrlBreakPenalty and \c_hash_str in the definition of \__stefanct_breakable_hash:.

Wow, thank you, bounties FTW. I understand what you explained but I would not have been able to solve that in any realistic time. The only problem I still have is that it won't break the line on # but oh well. Thanks! — stefanct, Jun 27 '20 at 09:56
@stefanct \oldurl will break at the # (try url\hfill\hspace{0.6\linewidth}\url{httpswwwyahoocombreak#me%too} in the code). \href doesn't break anywhere but hyphens. — Phelype Oleinik, Jun 27 '20 at 12:19
Yes, unfortunately. There are certainly reasons for all this but sane URL rendering in latex is really harder than it should be. :/ — stefanct, Jun 27 '20 at 12:57
@stefanct I changed it to allow breaking before hashes. To have it break after a #, change the order of \penalty \UrlBreakPenalty and \c_hash_str in the definition of \__stefanct_breakable_hash:. — Phelype Oleinik, Jun 27 '20 at 13:43

Accepting special characters in arguments passing them further

1 Answers1

Linked