Counting words and characters, with or without whitespace with texcount

Question

The goal is a flexible scheme for automatic character counting using texcount. Specifically, the user should be able to select each permutation of the the following options (characters | words) and (including whitespace | not including whitespace)

The options:

1. characters & not including whitespace
2. words & not including whitespace

are already accommodated by the code shown here. The key lines are:

\immediate\write18{texcount -char -merge -tex -sum \jobname.tex | grep -i section > \jobname Count.txt} % counts characters
%\immediate\write18{texcount -merge -tex -sum \jobname.tex | grep -i section > \jobname Count.txt} % counts words

Question: Is there a simple way to give the user the option to include whitespace in the count? (the motivation being that some webforms evaluate the count based on including whitespace)

Proposed approach

If we let a = non-whitespace character count, b = word count, and we possess the ability to determine both a and b, it may be possible to execute both of these counts via two separate texcount calls. Naturally, c = word count - 1 may be an okay approximation of the whitespace character count. We should then be able to determine the total (non-whitespace + whitespace) character count by d = a + c.

The key will be to modify these blocks:

\newcommand{\processCount}{%
 \newread\counts
 \def\zpar{\par}
 \openin\counts=\jobname Count.txt
 \loop
 \read\counts to \sectioncount
 \ifx\sectioncount\zpar\else
 \showcount{\sectioncount}\\
 \fi
 \ifeof\counts
 \else
 \repeat
}
\newcommand*{\showcount}[1]{%
 % e.g. 67+18+0 (1/0/0/0) S[ubs]ection: The first subsection
 \StrBehind{#1}{ection: }[\sectiontitleplusspace]
 \StrGobbleRight{\sectiontitleplusspace}{1}[\sectiontitle]
 \StrBefore{#1}{+}[\thiscount]
 \expandafter\ifcsname\sectiontitle limit\endcsname%
  \renewcommand{\limitcount}{\csname\sectiontitle limit\endcsname}%
 \else%
  \renewcommand{\limitcount}{-1}%
 \fi%
 \sectiontitle:
 {%
  \ifthenelse{\thiscount>\limitcount}{%
   \textcolor{red}{\thiscount/\limitcount}%
   \ifthenelse{\limitcount>-1}{%
    \ (over by \number\numexpr\thiscount-\limitcount\relax)%
   }{}%
  }{%
   \textcolor{green}{\thiscount/\limitcount}%
  }%
 }
}
\makeatletter
\newcommand*{\withlimit}[1]{%
 \expandafter\newcommand\csname@currentlabelname limit\endcsname{#1}
}
\makeatother

We will also need a mechanism for the user to indicate which count type the wish to display in / the implicit units of the limits being set in each section.

Answer 1

TeXcount does not support counting whitespace, at least not in a way that is remotely accurate. The main reason is that it would have difficulty identifying that multiple whitespaces in sequence would result in a single whitespace in the output.

The simple way to bypass this shortcoming is to count the number of words, and use that as an estimate of the number of whitespaces under the assumption that words are separated by single whitespaces.

Answer 2

TeXcount doesn't know how to expand macros, for example to include words from the glossary using \gls{} or glossary descriptions.

So I guess there is no "universal answer" this way, unless you narrow the boundaries of your question (a lot).

See the extended answer https://tex.stackexchange.com/a/312299/161015 from Einar Rødland, the maker of TeXcount.

There are alternatives to the automatic character counting without using TeXcount.

Once you have the pdf generated you can use ms word or chrome to do the counting: words, characters (no spaces), characters (with spaces).

See my answer https://tex.stackexchange.com/a/595275/161015 for more details.

Another approach is through lualatex. See https://gist.github.com/phi-gamma/2622252

It will count words using the callback "pre_linebreak_filter", so every macro is already expanded. I suppose the lua code could be generalize to count whitespace. (It will need to discount the extra whitespace to be discarded after each line).

Answer 3

This answer is a direct extension of my answer at texcount mwe no longer functional after tex system update, which was also previously asked by the OP. That answer is insufficient to this question, because it did not have the ability to apply the counting principals to externalized data (in a separate file). That answer provides the \countem...\endcountem pseudo-environment for counting letters and words contained in the environment.

To that end, I have added the macro \countemfile which will apply the \countem methodology developed in the prior answer to LaTeX code found in an external file. That macro, in turn, calls on the \filedef macro that takes the external-file data and reads it into a \def. I do not know what sort of file-size limitations may exist in this regard, in terms of consuming TeX engine memory/resources etc.

The same modes of counting available in the prior answer on environment content are now available for document text found in external files. However, it is recommended that, unless you know the file is text and/or macros that take only text arguments, it is probably best not to use the \runningcounttrue and \contentlimit features, as these could inject extra tokens into macro arguments, thus breaking them. The use of \summarycountrue is the better alternative in this regard for external files with general document content. Even without employing that, the file counts still contribute to the global count, which can be accessed at the discretion of the user.

As with the prior solution, \countspacestrue is available to add the counting of spaces and cat-12 (punctuation and numbers) to the letter count. Further, \obeyspaces may be employed, if extended white space should be part of the count. As before, the proviso exists that code which changes catcodes (notably verbatim) can not be run through \countem because the method previews tokens prior to execution, thereby setting their catcodes before verbatim has the chance to set them otherwise.

One other thing to note is that the "escape" feature of tokcycle, which bypasses processing with the use of delimiters, perhaps should be disabled (or the default | delimiters changed), since the use of this token in regular LaTeX use (for example, the | in tabular column formats) would be instead taken as a \countem escape token. That disabling can be effected with \settcEscapechar{\empty}, or it can simply be changed to another token that the user can assure does not appear in \countem input data.

Best of all, uncommenting the line

%\disablecountem% TO TURN OFF COUNTING, WITHOUT CHANGING DOCUMENT

will disable the \countem mechanism, with no other changes necessary to the source code.

The two files operated on by the MWE are given below. As you can see, the input files can contain sectioning, macros, etc. The one huge limitation that, I would think, applies to this and other methods, is that only the tokens presented can be counted. Tokens that expand to textual content, ranging from \today to \tableofcontents, can be processed by \countem; however, they will not increase the word or letter count because the count occurs before they expand into their final text.

\begin{filecontents*}[overwrite]{smallexternal}
This is  a    test.
\end{filecontents*}
\begin{filecontents}[overwrite]{chapterexternal}
\section{External-file word count}
Note that the above section heading is part of the count.
Also, one should set content-limit to 0 and runningcount false
  so that superscripts and colors don't interfere with, for example,
  macro arguments or starred argument invocation.
Use those with care, primarily with simple text files.
Macro output, like that from \today{}'' or\rule{3ex}{4pt}''
  is not part of the count, but the macro arguments are.
Unless escaped, things like environment names, column specifications,
  etc. count as letters and words:
\begin{tabular}{|c|c|}
  \hline
  aa & bbb\
  \hline
  cccc & d\
  \hline
\end{tabular}
\end{filecontents*}

Here is the MWE

\documentclass{article}
\usepackage{tokcycle}[2021-03-10]
\usepackage{xcolor}
\newcounter{wordcount}
\newcounter{lettercount}
\newcounter{wordlimit}
\newif\ifinword
% USER PARAMETERS
\newif\ifrunningcount
\newif\ifsummarycount
\def\limitcolor{red}
\setcounter{wordlimit}{0}
%%%%%%%%%%%% ENHANCED COUNT ALGORITHM MORE STREAMLINED THAN
% https://tex.stackexchange.com/questions/577276/
% texcount-mwe-no-longer-functional-after-tex-system-update/591949#591949 
\makeatletter
\newcommand\changecolor[1]{\tctestifx{.#1}{}{\addcytoks{\color{#1}{}}%
  \tc@defx\currentcolor{#1}}}
\makeatother
\newcommand\dumpword{%
  \ifinword\stepcounter{wordcount}
    \ifrunningcount\addcytoks[x]{$^{\thewordcount,\thelettercount}$}\fi
    \ifnum\thewordcount=\value{wordlimit}\relax\changecolor{\limitcolor}\fi
  \fi%
  \inwordfalse
}
\newcommand\addletter[1]{%
  \tctestifcatnx A#1{\stepcounter{lettercount}\inwordtrue}{\dumpword}%
  \ifcountspaces\tctestifcatnx .#1{\stepcounter{lettercount}}{}\fi% NEW!!
  \addcytoks{#1}}
\newif\ifcountspaces% NEW!!
\xtokcycleenvironment\countem
  {\addletter{##1}}
  {\dumpword\groupedcytoks{\processtoks{##1}\dumpword\expandafter}\expandafter
    \changecolor\expandafter{\currentcolor}}
  {\dumpword\ifactivetok\ifnum\number`##1=32\relax
    \stepcounter{lettercount}\fi\fi\addcytoks{##1}}
  {\dumpword\ifcountspaces\stepcounter{lettercount}\fi\addcytoks{##1}}% NEW!!
  {\stripgroupingtrue
    \def\currentcolor{.}
    \setcounter{wordcount}{0}\setcounter{lettercount}{0}}
  {\dumpword\retainsum\ifsummarycount\tcafterenv{%
    \par(Wordcount=\thewordcount, Lettercount=\thelettercount)}\fi}
\newcommand\contentlimit[1]{\setcounter{wordlimit}{#1}}
\newcounter{globalwordcount}
\newcounter{globallettercount}
\newcommand\retainsum{%
  \addtocounter{globalwordcount}{\thewordcount}%
  \addtocounter{globallettercount}{\thelettercount}%
}
\newcommand\processCount{\ifnum\thegloballettercount>0%
  \par Global Wordcount=\theglobalwordcount\
  Global Lettercount=\thegloballettercount
\fi}
\newcommand\disablecountem{\let\countem\empty\let\endcountem\empty}
%%%%%%%%%%% EXTENDED FUNCTIONALITY BASED ON BUT BEYOND
% https://tex.stackexchange.com/questions/577276/
% texcount-mwe-no-longer-functional-after-tex-system-update/591949#591949
\makeatletter
\newcommand\xappendto[2]{\expandafter\tc@defx\expandafter
  #1\expandafter{\expandafter#1#2}}
\makeatother
\newread\srcfile
\newcommand\filedef[2]{%
  \def#2{}%
  \def\srcfileline{}%
  \openin\srcfile=#1%
  \loop\unless\ifeof\srcfile%
    \read\srcfile to\srcfileline % Reads line into \srcfileline
    \ifx\srcfileline\empty\else\xappendto#2\srcfileline\fi
  \repeat%
  \closein\srcfile%
}
\newcommand\countemfile[1]{%
  \filedef{#1}\mytextinput
  \expandafter\countem\mytextinput\endcountem
}
% EXTERNAL INPUT DATA
\begin{filecontents}[overwrite]{smallexternal}
This is  a    test.
\end{filecontents}
\begin{filecontents}[overwrite]{chapterexternal}
\section{External-file word count}
Note that the above section heading is part of the count.
Also, one should set content-limit to 0 and runningcount false
  so that superscripts and colors don't interfere with, for example,
  macro arguments or starred argument invocation.
Use those with care, primarily with simple text files.
Macro output, like that from \today{}'' or\rule{3ex}{4pt}''
  is not part of the count, but the macro arguments are.
Unless escaped, things like environment names, column specifications,
  etc. count as letters and words:
\begin{tabular}{|c|c|}
  \hline
  aa & bbb\
  \hline
  cccc & d\
  \hline
\end{tabular}
\end{filecontents*}
\settcEscapechar{\empty}% TURN OFF ESCAPING, SO THAT | TOKENS ARE LEFT ALONE!
\begin{document}
%\disablecountem% TO TURN OFF COUNTING, WITHOUT CHANGING DOCUMENT
\section*{Local word count}
\contentlimit{12}\runningcounttrue\countem
In this MWE, we will show both local and file counting combined in
  the global counts.
\endcountem
\contentlimit{0}\runningcountfalse\summarycounttrue
\countemfile{chapterexternal.tex}
\runningcounttrue\contentlimit{3}
\subsection*{Small file with running count, word limit}
\countemfile{smallexternal.tex}
\runningcountfalse\contentlimit{0}
\subsection*{Small file with no space counting}
\countemfile{smallexternal.tex}
\subsection*{Small file with space/punctuation counting}
\countspacestrue
\countemfile{smallexternal.tex}
\subsection*{Small file with space/punctuation counting AND obeyspaces}
\begingroup
\obeyspaces
\countemfile{smallexternal.tex}
\endgroup
\summarycountfalse\countspacesfalse
\section*{Document analysis}
\processCount
\end{document}