soul package command \hl can not work with escaped space

Question

It seems that \hl command in the soul package can not work with escaped space. A minimal non-working example is

\documentclass{article}
\usepackage{xcolor, soul}

\colorlet{mycolor}{red!30}
\sethlcolor{mycolor}

\begin{document}

\hl{can\ we\ highlight}

\end{document}

The error message is

Package soul Error: Reconstruction failed.

Some posts (for example, here and here) suggest using \mbox(or \hbox) to wrap the text or using \protect. While these methods work, they have some drawbacks:

If \mbox is used, long text will not wrap normally.
If \protect is used, the background color for text will disappear.

How to solve this problem more elegantly?

Edit

The solution shared by Skillmon works for English, but not for Chinese. The following MWE will illustrate that.

\documentclass{ctexart}
\usepackage{xcolor, soul}
\usepackage{xparse}

\let\hlORIG\hl

\colorlet{mycolor}{red!30}
\sethlcolor{mycolor}

\ExplSyntaxOn
\tl_new:N \l_jdhao_hlx_tl
\RenewDocumentCommand \hl { +m }
  {
    \tl_set:Nn \l_jdhao_hlx_tl { #1 }
    \tl_replace_all:Nnn \l_jdhao_hlx_tl { \  } { ~ }
    \exp_args:NV \hlORIG \l_jdhao_hlx_tl
  }
\ExplSyntaxOff


\begin{document}

\hl{test\ test}  %% pass!
\hl{测试\ 测试} %% fail!

\end{document}

It is interesting that English test passed, but the Chinese test failed. The error message is the same.

Out of interest: Why do you want to escape spaces after normal words? — TeXnician, Sep 04 '18 at 10:30
Actually, I was trying to convert Markdown to PDF using Pandoc. For space in inline code, Pandoc will escape it, which unfortunately conflicts with soul. — jdhao, Sep 04 '18 at 10:32

Skillmon · Answer 1 · 2018-09-11T09:51:00.620

6

With a helper macro which converts all the explicit spaces (\) into a normal space, this is possible (I redefined the \hl macro so that you don't have to change anything else):

\documentclass{article}
\usepackage{xcolor, soul}
\usepackage{xparse}
\let\hlORIG\hl

\colorlet{mycolor}{red!30}
\sethlcolor{mycolor}

\ExplSyntaxOn
\tl_new:N \l_jdhao_hlx_tl
\RenewDocumentCommand \hl { +m }
  {
    \tl_set:Nn \l_jdhao_hlx_tl { #1 }
    \tl_replace_all:Nnn \l_jdhao_hlx_tl { \  } { ~ }
    \exp_args:NV \hlORIG \l_jdhao_hlx_tl
  }
\ExplSyntaxOff

\begin{document}

\hl{can\ we\ highlight}

\end{document}

EDIT: Working with Chinese input, but breaking automatic hyphenation (you can still provide manual hyphenation with \-):

\documentclass{ctexart}
\usepackage{xcolor, soul}
\usepackage{xparse}
%\usepackage[chinese,main=english]{babel}
\let\hlORIG\hl

\colorlet{mycolor}{red!30}
\sethlcolor{mycolor}

\ExplSyntaxOn
\tl_new:N \l_jdhao_hlx_tl
\RenewDocumentCommand \hl { m }
  {
    \tl_set:Nn \l_jdhao_hlx_tl { #1 }
    \tl_replace_all:Nnn \l_jdhao_hlx_tl { \  } { ~ }
    \regex_replace_all:nnN { [^\s\c{-}] } { \c{hbox} \0 } \l_jdhao_hlx_tl
    \exp_args:NV \hlORIG \l_jdhao_hlx_tl
  }
\ExplSyntaxOff

\begin{document}
% for comparison of hyphenating 
\rule{.9\textwidth}{1pt}possibility\ to\ highlight

\rule{.9\textwidth}{1pt}\hl{possibility\ to\ highlight}

\rule{.9\textwidth}{1pt}\hl{pos\-sibility\ to\ highlight}

\rule{.9\textwidth}{1pt}\hlORIG{possibility to highlight}

\hl{汤面}
\hl{汤面}
\hl{测试\ 测试}
%\hlORIG{汤面}

\end{document}

EDIT2:

Yet another version which allows you to use braces and other stuff in the argument of \hl. Note that both this version and the second version break maths in the argument of \hl. You can change the regular expression used as the first argument of \regex_replace_all:nnN to fix more issues you're facing, I'm not willing to put more time into fiddling with it.

\documentclass{ctexart}
\usepackage{xcolor, soul}
\usepackage{xparse}
%\usepackage[chinese,main=english]{babel}
\let\hlORIG\hl

\colorlet{mycolor}{red!30}
\sethlcolor{mycolor}

\ExplSyntaxOn
\tl_new:N \l_jdhao_hlx_tl
\RenewDocumentCommand \hl { m }
  {
    \tl_set:Nn \l_jdhao_hlx_tl { #1 }
    \tl_replace_all:Nnn \l_jdhao_hlx_tl { \  } { ~ }
    \regex_replace_all:nnN { \c[^CBEMTPUDA]\S } { \c{hbox} \0 } \l_jdhao_hlx_tl
    \exp_args:NV \hlORIG \l_jdhao_hlx_tl
  }
\ExplSyntaxOff

\begin{document}
% for comparison of hyphenating 
\rule{.9\textwidth}{1pt}possibility\ to\ highlight

\rule{.9\textwidth}{1pt}\hl{possibility\ to\ highlight}

\rule{.9\textwidth}{1pt}\hl{pos\-sibility\ \textbackslash{} to\ highlight}

\rule{.9\textwidth}{1pt}\hlORIG{possibility\textbackslash{} to highlight}

\hl{汤面}
\hl{汤面}
\hl{测试\ 测试}

\end{document}

edited Sep 11 '18 at 09:51

answered Sep 04 '18 at 10:49

Skillmon

60,462

It seems that this solution does not work for Chinese text, it only works for English. I have tried to test it with Chinese text and failed. – jdhao Sep 04 '18 at 13:10
@jdhao can you please update your question then, to include Chinese as well? This way any future answerer knows that this has to work for Chinese. And please also add information which engine you use to compile (there might be a clever solution for LuaTeX if you use it). – Skillmon Sep 04 '18 at 13:59
@jdhao I can add code which works with chinese text, but breaks hyphenation (line breaking still works, though). – Skillmon Sep 04 '18 at 17:11
I have updated the question to include the fail case for Chinese. – jdhao Sep 05 '18 at 02:44
@jdhao my updated answer does work with your updated MWE (though breaking hyphenation -- but I don't know how hyphenation works in Chinese anyway). – Skillmon Sep 05 '18 at 06:45
You can use the test case I used, i.e., \hl{测试\ 测试}. On my machine, error will occur. – jdhao Sep 05 '18 at 07:25
@jdhao I did and it works (with the second code block of my answer) on my machine. Did you try the correct code block? Sadly atm hyphenation isn't working with that approach. I'll look into that later this day. – Skillmon Sep 05 '18 at 10:26
Thanks, the second code works. I have been using the first code block. – jdhao Sep 05 '18 at 10:38
@jdhao hyphenation could be possible but needs at least 2 runs with parsing the log file for the output of \showhyphen with some additional stuff around it (as markers). That's nothing I want to do right now. I hope the above is enough. I've made manual hyphenation with \- possible. – Skillmon Sep 05 '18 at 14:21
@jdhao does this answer satisfy your needs? If so, could you please accept it? If not, could you please say what's still needed? – Skillmon Sep 07 '18 at 15:14
It seems that \textbackslash{} can not work with the new \hl command. – jdhao Sep 11 '18 at 01:16
@jdhao see my edit for a version which solve that. Note however, that you don't have to use \textbackslash{} but could also use \textbackslash\ to force a space after \textbackslash. – Skillmon Sep 11 '18 at 09:51

soul package command \hl can not work with escaped space

1 Answers1