(Too long a reply to your comment) Others have already showed you how to make it work. I’ll try to explain why your approach doesn’t.
TeX doesn’t work as other languages, with “return values” for functions. TeX is a macro expansion language, and that’s basically what it does. When TeX reads your code it splits the processing in two main stages: expansion and processing. There are situations (most of them) where TeX does both of them: if there’s something to expand, it expands, if there’s something to process, it processes, and all is good. There are situations, however, sometimes called “expansion-only contexts”, where it only expands things and will not do anything else. One of this expansion-only contexts is when TeX is scanning a number, for example, after an \ifnum (there are many others, and the expansion rules slightly differ between them).
Now let us pause the explanation above for a moment to differentiate between things that expand and things that are processed. Any token (let’s focus on control sequence tokens, i.e., \<something>) can be either a macro or a primitive. A macro is anything you (or some package) defined with \def\foo{bar}, and a primitive is built into TeX (for example \ifnum). A macro will always expand (in the example above, \foo expands to bar). A primitive can either be expandable or not (that depends on the primitive; see a list here). \def, for example, is not expandable, while \ifnum is. As a rule of thumb, if the primitive does an assignment, it is not expandable. Everything else depends.
— But why on Earth, then, isn’t \IfStrEq expandable? It’s a macro, after all, so from the definition above, it must expand, right?
— Well, yes, it does expand, but not all the way through. What makes a macro be “safely expandable” or not depends on how it is implemented. If the implementation of certain macro contains only expandable primitives or other (expandable) macros, then that macro is said to be expandable. Otherwise it is not.
Now let’s get back to your example and the expansion-only context of \ifnum. When TeX sees \ifnum it starts expanding tokens ahead, looking for a number, then for either <, =, or >, and then another number (this imposes another restriction in the expansion rules of \ifnum: whatever you put there has to expand to these tokens). Your working example does that:
\IfStrEq{#1}{\empty}
{\def\tmp{0}}
{\ifnum#1>0 \def\tmp{1}\else\def\tmp{0}\fi}
\ifnum\tmp=1
true
...
Here \IfStrEq is used in a normal (not expansion-only) context, and does its thing, defining \tmp to expand (a macro, thus it expands) to either 0 or 1. Then TeX sees \ifnum, which starts by expanding \tmp once, so it ends up like \ifnum1=1 (or \ifnum0=1, depending on #1) and all goes well.
The second code, however, puts \IfStrEq right after \ifnum:
\ifnum\IfStrEq{#1}{\empty}
{0}
{\ifnum#1>0 1\else 0 \fi}=1
true
...
so TeX will expand \IfStrEq (a macro) looking for a number. After one expansion \IfStrEq becomes \xs_ifstar{<code>}{<more code>} (you can see that with \show\IfStrEq or \texttt{\meaning\IfStrEq}), so you have \ifnum\xs_ifstar<code>.... \ifnum keeps expanding things, and now \xs_ifstar (another macro) expands to \xs_ifnxttok*<more code>..., then \xs_ifnxttok (yet another macro) expands to \xs_deftok<code code>..., which finally expands to \let\xs_toksmatch=*\relax.
So after four expansion steps, \ifnum\IfStrEq becomes \ifnum\let\xs_toksmatch=*\relax, and then \ifnum sees the primitive \let (which does an assignment so it is not expandable) and stops looking for a number. But no number was found, so TeX complains about a missing number and shows you what it saw instead of the number, \let:
! Missing number, treated as zero.
<to be read again>
\let
l.28 \TFb{}
?
and that’s why \IfStrEq will not work in an expansion-only context (i.e., it’s not expandable). Different usages of the command (that is, not after a \ifnum) will break in different ways, because the expansion rules will differ, but it will break nonetheless (here, for example).
I hope this rather lengthy explanation sheds some light on how expansion works and why some commands cannot be used in certain places.
xstring's commands are not expandable, which means that when\ifnumis expanding tokens looking for a number, the\IfStrEqmacro leaves things which aren't numbers, thus you get some error likemissing number treated as zeroormissing = inserted for \ifnum.\tmp, on the other hand, is defined to expand to a number, so it works as expected. – Phelype Oleinik Jan 11 '20 at 15:39\IfStrEqwas defined. You were lucky that Phelype recognised it but a test document that included\usepackage{xstring}would have been clearer – David Carlisle Jan 11 '20 at 15:47IfStrEqdoesn't always returns a boolean but e.g. an empty character if the input is also empty? – Max Jan 12 '20 at 09:00%in\newcommand{\TFa}[1]{%and\newcommand{\TFb}[1]{%, and the%in\else 0 \fi}=1%(first line of the definition of\TFb) shouldn't be there (see: https://tex.stackexchange.com/q/7453/134574). – Phelype Oleinik Jan 12 '20 at 15:56