Managing hundreds of separate components in a large document

Question

I am making a workbook containing hundreds activities. Each activity appears as its own section and generally contains instructions, spaces to write or draw, and sometimes illustrations or diagrams to complete.

In past projects, I have found such documents end up quite long and difficult to manage. I have tried a couple of solutions. Initially, I just created one macro per activity in an external file, then simply retrieving the macros within the main file:

\define\activityone {
    \section{Bird Watching}
        How many birds can you find near your home? Draw a picture of them in the space below.
}

\define\activitytwo{
    \section{What kind of bird?}
        Use the bird guide to find the birds. What are they called? Write the names near your picture above.
}

As each is unique, but there might be some which appear with identical formatting, I also tried making different macros for each format, which are placed in the main file like this:

 \activitystyle{Bird Watching}{How many birds can you find near your home? Draw a picture of them in the space below.}

Both of these solutions help somewhat to separate content from format, but, in the end, they do not seem to make the document any cleaner to manage. Are there any other methods for managing many pieces of text in a larger document?

• I am using ConTeXt, but sometimes solutions from on variety of TeX are applicable to other varieties.

Answer 1

The best option is obviously to create some form of a Database to Manage the content and keep the main file short and readable. As strange as it sounds TeX is very well suited for this. One would need a bit more detail to define the structure of the database, so I will demonstrate the technique using a list of activities only such as (bird watching, swimming, drawing a picture, cross word puzzles,...). An activity can have sub-activities etc.

Each activity is best to be saved in its own file or you can create macros for it. In the example below I used both. The first one uses a macro (\lipsum[1]) and the second a file astro.tex, which we created in the example using filecontents.

\AddActivityDetail{Swimming}{\lipsum[1]}
\AddActivityDetail{Astro surfing}{\input{astro}}

\documentclass[11pt]{book}
\usepackage{lstdoc,lipsum,filecontents}

\begin{filecontents*}{astro.tex}
\lipsum[5]
\end{filecontents*}
\begin{document}
\makeatletter
\def\alist{}

\let\sort\lst@BubbleSort 
\def\addtolist#1#2{
  \lst@lAddTo\alist{#2}
}

\long\gdef\addActivity#1#2{\addtolist\alist{#1,}}

\def\AddActivityDetail#1#2{%
\long\expandafter\def\csname#1\endcsname{\textbf{#1}: #2}
\addActivity{#1}{#2}
\sort\alist
}

\def\PrintActivity{%
  \@for \i:=\alist\do{%
  \csname\i\endcsname}
}
%example
\AddActivityDetail{Bird Watching}{\lipsum[2]}
\AddActivityDetail{Stone Throwing}{\lipsum[3]}
\AddActivityDetail{Swimming}{\lipsum[1]}
\AddActivityDetail{Astro surfing}{\input{astro}}
% print the activities
\PrintActivity
\makeatother
\end{document}

The activities are automatically sorted out in an alphanumeric order. If you do not need this you can comment out the \sort and associated macros.

The code should work in ConTeXt as well with the exception of the packages, which I am not sure (lipsum and filecontents, lstdoc), but I am sure ConTeXt has equivalent macros. Post a comment or two if this worked out please.

Answer 2

What I would do is use LuaLatex. You can have a lot more control and do "programming" related tasks a lot more efficiently in LuaLatex than TeX. Even if you found a TeX solution you should expect it to be much slower. When I "ported" my code from TeX to Lua I went from about 1 min compile time to almost instantaneous.

What you can do in Lua is basically create your macros programmatically.

Since you want separation of content and style you'll need to figure out exactly what kind of control you want in your formatting. e.g., Do you need to have control at the character level or will your styles "wrap" the content. In the first case you'll have to develop your own markup language or mix TeX in your content. If you are just "wrapping" content then you can easily do this efficiently in latex already using macros.

I am really not sure what you exactly are trying to do though. It sounds like you are just wanting to include certain text blocks without cluttering up the main text.

In this case you could use a database and have lua retrieve the data from it. For example, you could store your exercises in it. You could use text markup in the database strings and lua will just pass it on to tex. You could also, in the database, have "styles" and even have lua preprocess the strings to insert these styles.

For example

Instead of

long text

long example

you could have

long text

\example{example324.5}

where \example is a macro that calls a lua function which inserts example324.5... which comes from somewhere else.

Now, if your examples/excessive follow a very regular pattern(for example, examples at end of every section and excessive at end of every chapter you could create your own custom environments that add these things automatically at the end(again, something will have to retrieve them from somewhere(database/textfile)

Could you explain how exactly you want to separate the formatting and content? Can it be done easily for you case?

Do you want to separate the main content from the examples/exercises/etc or do you want to separate the formatting from the examples/exercises/etc? The later seems like it would be the harder of the two unless you have very specific and common "styles" that they all will use. (in which case you can use lua to apply these styles quite easily)

Example:

ExampleStyle1

My Example Title

Example 243) My example text is an example of an example with an image: \img{myexample.png}

And you can make a lua processor that automatically apply ExampleStyle1 to this example which could format the title, the example number, the text that follows and even the image... But if if you have many different example formats then this would not be a good method since you'll have about just as many styles as examples.

Answer 3

There are three parts to manage your material: creating, search, and retrieval. I am not sure what exactly you want, so am simply writing some possible ideas.

For creation, I think that the best solution is to use buffers (either directly or behind a macro). Buffers are better than macros because anything can be a part of a buffer. For example, you could use something like:

\startactivity[title={What kind of bird}, reference={bird:type}]
Use the bird guide to find the birds. What are they called? Write the names near your picture above.
\stopactivity

where a \startactivity uses \grabcontent (or \dostartbuffer) internally (See m-r.tex in the distribution for examples). To separate questions from answers, you can use blocks inside an activity to define an answer. That way you can control whether to print an answer at the end of an activity or print all answers together at the end. See the context manual for details about blocks.

The above can be implemented in such a way that you can write \getactivity[bird:type] to retrieve the activity. (Basically, you just a wrapper around \getbuffer).

I don't have a good suggestion regarding search. For images, I have found ConTeXt image database to be useful. It takes some effort to set up, but then you can easily search for the images by looking at the overview page. According to the wiki, in MkIV the image database can also handle other resources (like files), but I don't know how the new resource management feature works.