Check if macro is fully expandable

Question

Question

Is there a way to check whether a macro is fully expandable (or rather "safe in an expansion-only context" [1])?

Consider this code:

\def\a{Just a string}
\def\b{\a}
\def\c{\def\unsafe}
\def\d{\c}

How could I check which of the macros (a-d) are safe in an expansion-only context? By looking at them I know that a and b are whereas c and d are not but if I wanted to know the same for a macro I haven't written myself this could get quite useful.

Background

I am working on a way to detect whether some input is a valid number in PGF. For this I developed this approach which makes use of passing the input into \pgfmathfloatparsenumber.
The problem I have run into is that said macro appears to somehow manages to expand the input until there is an error (if the input is in fact not safe in an expansion-only-context). I tried using protected, noexpand and similar but somehow PGF manages to circumvent those.

So the idea is to check whether the input is safe before actually passing it to PGF. The problem is: I don't know how I'd go about that...

well you can always do something of course, in the example you give it would be hard in general to avoid an undefined command error on that input, if you defined \unsafe first then the edef would give something bad but would probably not give an error during the actual edef, such cases you can probably detect. similarly if you have \edef\foo{ {\mbox} } you are going to get a low level parse error if you expand \mbox and it hits the } it woul dbe verh hard to avoid such errors if you allow bad input — David Carlisle, Dec 13 '18 at 08:11
if you know in advance that the thing must expand say to some digits, you could try \romannumeral-`0 triggered expansion, then examine first token if a digit ok remove and repeat and do repetitively until either nothing is left or you hit some unexpandable token which is not a digit. You have to detect case of braces etc... The idea here is that \edef can cause errors if your material is not expandable, but "full-first" expansion will not. — , Dec 13 '18 at 08:46
@jfbu If the input consisted of digits only I could use \IfDecimal from the xstring decimal and I'd be good. The problem is that there might be letters as well (scientific number notation)... — Raven, Dec 13 '18 at 08:49
No, you misunderstood. I said "if the thing must expand say to some digits". If letters also are allowed then you only have to take that into account. The point is that if you know in advance what must be the full expansion outcome, then you can check it. — , Dec 13 '18 at 08:50
For example xintfrac package allows a much wider notion of input than the ones for which \IfDecimal (applied to full expansion) will test positive. And xintfrac proceeds purely expandably. If you hack into it you can add branches for the cases it detects something went awry and you could convert it into something which detects if input expands to the expected format. Even more powerful is the xintexpr parser mechanism. — , Dec 13 '18 at 08:53
Since one could argue that the only fully expandable macro must eventually expand to nothing, because a character token is not expandable, the test could be fairly easy with \romannumeral :) — Skillmon, Dec 13 '18 at 08:54
hmm, forget it about xintfrac input processing because at some locations it uses \numexpr hence unrecoverable low-level errors, thus more something in the style of xintexpr (although at some locations it applies \string and also must act expandably). Anyway, the idea of repeated \romannumeral-`0 expansion is workable. — , Dec 13 '18 at 08:56
Does the test have to be expandable or is non-expandable approach fine? — Skillmon, Dec 13 '18 at 09:01
However \romannumeral-`0 will expand macros which have been defined via \protected\def, so this is not really same as doing what would happen in an \edef. Also careful with spaces when doing this things. One can not simply do the expansion then grab a token. If you don't need to work expandably then \futurelet can help. — , Dec 13 '18 at 09:03
@jfbu one could also check the \meaning of each token whether it is \protected. — Skillmon, Dec 13 '18 at 09:07
@Skillmon yes, right. As per the technique of repeated expansion the ~~"x"~~ "e" type recently added to LaTeX3 is surely a reference implementation, and the code comments will be helpful. But in a way any parser such as xfp or xintexpr basically has the query already implemented. They proceed purely expandably and I am sure xfp raises more readable errors in case of problems... (by the way I know my comments go in various directions but it is hard to stay focused if the exact context is not known: does the thing have to work expandably for example?) — , Dec 13 '18 at 09:10
@jfbu additionally to spaces one must check for braced groups when grabbing tokens (so essentially after each \romannumeral expansion step one would first check for spaces and braces, then for \protected and then for valid numbers, discarding them and relooping until either one of the tests is false or the full string was discarded as valid numbers -- additionally testing for at most one decimal marker and an optional lower or upper case e which could be of category code 11 or 12). — Skillmon, Dec 13 '18 at 09:18
@Skillmon if test does not have to be expandable that is less a problem... as per spaces you need a "space upfront" test before applying a \romannumeral-`0 which would gobble it and stop. This is also why context is important: xintexpr does not care and ignore spaces, but perhaps here you don't want to allow them in input. Or you do... — , Dec 13 '18 at 09:21
@jfbu I wasn't precise, I meant to test for spaces before the first \romannumeral, too. — Skillmon, Dec 13 '18 at 09:24
@Skillmon ok, I read too fast your comments anyhow :). I think you are ready to implement it! — , Dec 13 '18 at 09:26
@jfbu we have another problem, which is quite hard to detect: What if a macro expands to something with a space stopping the \romannumeral expansion. That space would be gobbled but we would like to say that this is a case where the test should result in false. So we'd have to parse the \meaning of each token not only for \protected, but also for a space. — Skillmon, Dec 13 '18 at 09:46
@jfbu And we'd have to discard tokens which are read in by any macros in the argument, since this could be discarded by the macro, so they should not result in a false result. — Skillmon, Dec 13 '18 at 09:53
@Skillmon we can not use \romannumeral expansion easily if we care for spaces. \Foo may expand to \foo\bar with \foo expanding to a space token. The \meaning of \Foo (and imagine if \escapechar is -1...) will at best tell us there is \foo. If we blindly apply romannumeral expansion, it will expand the foo and silently gobble the space (we can not distinguish if \foo expand to 1 or to <space>1). Thus we need to check the meaning of \foo too. And we have to do this recursively until a non-expandable token. Then we go back and apply romannumeral to force expansion of \Foo — , Dec 13 '18 at 10:08
(btw, \expandafter\baz\foo does expand \foo even if was \protected so we can't blindy repeat \expandafter if we care for that, meaning (sic) we must use the \meaning, which requires to properly grab a token i.e. check for braces, spaces...) — , Dec 13 '18 at 10:16
@jfbu yes, that's correct. The reason why I won't code this. It'd take forever because we'd have to implement (a great part of) TeX in TeX, preferably in TeX's mouth (because we like to be expandable where possible). I think that Bruno once did/tried something along those lines. — Skillmon, Dec 13 '18 at 12:54

Henri Menke · Accepted Answer · 2018-12-16T01:34:16.053

6

Did I hear someone say that it's not possible?

The following defines \ifexpandable which checks whether a token is expandable (actually you can also give it a list of tokens and it checks whether all of them are expandable). I don't know whether this has any side effects. Requires LuaTeX.

\documentclass{article}

\def\ifexpandablelua{%
  \directlua{
    local t = token.scan_toks()
    local b = true
    for n,v in ipairs(t) do
        local is_assign =
            string.find(v.cmdname, "assign") \string~= nil or
            string.find(v.cmdname, "def") \string~= nil or
            string.find(v.cmdname, "let") \string~= nil or
            string.find(v.cmdname, "box") \string~= nil
        local is_call = string.find(v.cmdname, "call") \string~= nil
        print(v.cmdname, is_assign, is_call)
        b = b and (not is_assign) and (is_call and v.expandable or true)
    end
    if b then
        tex.sprint("\string\\iftrue")
    else
        tex.sprint("\string\\iffalse")
    end
  }%
}

\def\ifexpandable#1{%
  \expandafter\ifexpandablelua\expandafter{\romannumeral-`0#1}%
}

\begin{document}

\def\a{Just a string}
\def\b{\a}
\def\c{\def\unsafe}
\def\d{\c}

\edef\isexpandable{%
  \ifexpandable\section
    expandable
  \else
    not expandable
  \fi
}
\isexpandable

\end{document}

edited Dec 16 '18 at 01:34

answered Dec 13 '18 at 09:01

Henri Menke

109,596

What about something like \def\foo{{\hbox}}\ifexpandable\foo? – Joseph Wright Dec 13 '18 at 09:03
@JosephWright \foo is expandable, i.e. you have to expand it once and ask \ifexpandable again. I tried to expand step-wise with token.expand but quickly realised that I actually have no idea what this function is doing. – Henri Menke Dec 13 '18 at 09:03
1

I think the OP query is not testing one token but testing an entire input which may conceivably be \foo E\bar for example. – Dec 13 '18 at 09:06
@jfbu Fixed. I just found that scan_toks takes arguments, one of them is to expand the tokens. – Henri Menke Dec 13 '18 at 09:13
Hm, it doesn't always work. Things like \ifexpandable\section in a LaTeX document blow up. – Henri Menke Dec 13 '18 at 09:18
Thanks for the answer. However this really checks for pure expandability in TeX's-sense and not whether it is "safe in an expansion-only context". E.g. it considers macros containing only letters as "not expandable" (which strictly speaking is correct but not what I am looking for). – Raven Dec 13 '18 at 09:23
1

@Raven You could additionally check for v.cmdname. This field holds information about the token, i.e. whether it is a def or let or letter or other_char etc. – Henri Menke Dec 13 '18 at 09:29
ping @Skillmon How about using \expanded in this LuaTeX context? I tried \def\foo{{\hbox\the\numexpr2+3}}, \expandafter\def\expandafter\bar\expandafter{\expanded{\foo}}, and the meaning of \bar is then {\hbox 5} showing in passing no expansion error was raised from incongruous \hbox usage. If it is guaranteed that \expanded never generates errors, then this makes the whole thing much much much much easier, because we only test the result of already expanded thing. And I understand TL2019 will have \expanded added to pdftex and to xetex. – Dec 13 '18 at 16:38
@jfbu haven't played with \expanded yet, so I'm not sure what the rules are. But it could be promising. But I won't get anything done today, if you have time, you should test it :) And pinging people where they didn't already made a comment doesn't work, so I only saw this by chance, not because of the @Skillmon. – Skillmon Dec 13 '18 at 18:04
@Skillmon sorry for pinging you from here (and apologies to Henri for abusing the comment space of his answer). in my comment, I expressed myself poorly but anyway the problem is that with \def\foo{\def\error{\undefined}} there is no miracle (or surprise) and \expanded{\foo} will of course try to expand \error hence generates an error. So no, it does not help, it is no better than using an \edef, except that it works expandably. – Dec 13 '18 at 18:29
1

@Raven I have improved my answer. It is now using the \romannumeral trick to expand everything up to the first unexpandable token. Then I feed that result into Lua to check whether there is any unexpandable call or assignment inside. – Henri Menke Dec 16 '18 at 01:21
@HenriMenke Why \romannumeral rather than \expanded? – Manuel Dec 16 '18 at 11:48
@Manuel (not entirely sure about this:) It seems that \expanded is doing the same as \edef in an expandable manner. For this reason, it is not the best option (read: no option at all) to test whether something is safe in an \edef context. For a draft how to do all these tests without invoking Lua, look at the comments below the question, made by me and jfbu. The code would be huge and I will not do this (at least not as of now -- maybe in some years because I have much spare time and want a TeX challenge). – Skillmon Dec 16 '18 at 18:40

Check if macro is fully expandable

Question

Background

1 Answers1

Linked