I recently read answer that discouraged the use of \ensuremath on the grounds that one should know whether each command is meant to be used in math mode or text mode. Given a command, how can you tell whether it's meant for use in math mode, text mode, or both? (Note that I'm asking for a practical way to tell this for commonly used LaTeX commands, not something that works in general.)
Edit: I meant to say, 'other than trial and error'.
I seem to remember having something to do with that answer :). The short answer to your query is that there is no a priori way of deciding based on a random macro whether it's "math mode" or "text mode", since of course a macro could be named anything. It is possible (see this question) to write a test that detects whether a macro enters math mode. Otherwise, the following general guidelines exist:
The most obvious difference between math mode and text mode is font-related: math mode uses special fonts, and even for letters, it uses a slanted font. Correspondingly, in LaTeX many font commands for which there is a difference in behavior have math- and text-mode variants, e.g. \mathbf versus \textbf. This is the exception that proves the rule that you can't tell by the name.
The most obvious similarity between math and text modes is that TeX doesn't make a distinction until the typesetting starts, i.e. until it passes to the "stomach". Thus, plain macro expansion will be the same in both, unless the result contains $, ^, _, or one of the \math font commands (but still, the expansion part of the macro will be unaffected by math mode, except for \ifmmode, of course). There are probably dozens of special cases in addition to these, of course.
Math is full of special symbols and constructions; if you have a "macro" that produces a symbol, then you should be aware of the intent of that symbol. In common practice it is actually math-mode: for example, \alpha is not a Greek letter but is actually the mathematical symbol of that shape. See this question for the corresponding text-mode variants (which actually are not macros).
The problem that concerns you, based on the reference to my other answer, is how to tell whether an innocuous macro that takes some argument, i.e. \mymacro{stuff}, will typeset stuff in math or text mode. There's no way, short of looking up what the author intended. Fortunately, the situation is twofold: either you are the author, in which case you're in luck; or the author was a responsible LaTeX package writer and produced a lovely PDF manual that can be obtained via the texdoc program or, failing that, CTAN. No third alternative exists, naturally.
In one of your comments to the question, you ask how to look up an unknown command. First, you should know which package provides the command; if you are using a "boilerplate" preamble without knowing why the packages are included, you should go to texdoc (or CTAN) and read up on all of them :). If the macro isn't part of any package then it's part of TeX or LaTeX itself, in which case, this site mirrors a document whose name I forget that contains all the basic ones provided by LaTeX. If you are truly unfortunate and some command appears to have no documentation at all on the internet, then it is a TeX primitive and you should read TeX By Topic (available free via texdoc) or the TeXbook (not free, but worth it).
\ifmmode expansion differs for any character with mathcode "8000 and catcode 11 or 12 which will be a normal character in text mode but active and (typically) expand to a macro definition in math mode. ' being the canonical example. also (in theory if rarely seen in the wild) etex \ifnum\currentgrouptype or \ifnum\lastnodetype can be used similarly to ifmmode
– David Carlisle
Dec 05 '12 at 01:37
As Ryan mentioned there is no real way to tell, however you can peek at the definition and normally there are enough clues to tell:
\documentclass{article}
\begin{document}
\texttt{\meaning\alpha}
\show\alpha
\end{document}
You can use either \meaning to typeset the result or \show to see them in the console. The
\texttt in MWE is just to get the right font for the backslash.
For a discussion on using \meaning see Introspection and reflection with LaTeX/TeX macros and Bruno's \show with fewer lines?.
If the command you're concerned about is a LaTeX symbol, there's a good chance it's included in "The Comprehensive LATEX Symbol List", which lists almost 6000 symbols and categorizes many of them as either "body-text symbols" or "mathematical symbols".
