4

I use the package pdfpages to include scanned and ocr'd pdf's in my LaTeX document, and I need to index them (over 200 pages). I have a plain .txt file with all keywords that I'd like to find my ocr'd pdf, and those keywords that are present in my ocr'd pdf should then be indexed.

What would be the best way to do this?

lockstep
  • 250,273
Joe
  • 41
  • 1
    Does the .txt file also contain the whereabouts of the keywords? By including the pages via pdfpages LaTeX is technically not aware of the "source code" of these particular pages, so there seems to be no easy way to add index markers to certain words. Seeing that your pdfs are OCR'd you might consider using the text file output by the OCR programme and either try to use this text to produce a new TeX file or run some kind of script to mark the position the keyword is on in the file. A genuine LaTeX-only solution might be quite hard to come by. – moewe Oct 02 '13 at 14:48
  • 1
    If you converted the PDF to plain text using OCR software and include that text in the .tex file, you could write a script that looks for the keywords in the .tex file and replaces each occurrence with a command like \gls{keyword} from the glossaries package. Then you could compile the .tex file and get an index within the document. It might also be possible to use LuaLaTeX for both the script and the document, but I'm not an expert on this. – SFAB Oct 02 '13 at 15:18
  • 1
    Generally, an automatic index is a bad index. – Paul Gaborit Oct 02 '13 at 15:30

1 Answers1

6

As others said, this cannot lead to good index, because not every term usage is important, this cannot find concepts, there will be no cross references, you cannot deal with synonymy, homonyms etc.

But if you really want, there is simple script in lua, autoindex.lua:

#!/usr/bin/env texlua
local indexterms = arg[1]
local file = io.open(indexterms,"r")
if not file then 
    print("Cannot load index terms: ".. inputfile)
    os.exit()
end
local terms = {}
for line in file:lines() do
    terms[#terms + 1] = line
end
file:close()
local text = io.read("*all")
local page = 1 
local words = {}
-- Process pages
for t in text:gmatch("[^\f]*") do
    --tokenize words
    --add more characters which can't be part of words
    for x in t:gmatch("([^%s%.,!%?%(%)%-i@%$]+)") do
        local x = string.lower(x) -- normalize strings. note that this doesn't handle unicode
        local w = words[x] or {}
        w[page] = true
        words[x] = w
    end
    page = page + 1
end
for _, term in pairs(terms) do
    local match = words[term] or {}
    for page,_ in pairs(match) do
        print('\indexentry{'..term..'}{'..page..'}')
    end
end

You must first convert your pdf file to text with pdftotext utility:

pdftotext filename.pdf outputfile.txt

this will preserve page breaks. Then call this script like:

texlua autoindex.lua filewithterms < outputfile.txt > indexfile.idx

This will write entries in standard makeindex format to indexfile.idx:

indexentry{hello}{13}
indexentry{hello}{9}
indexentry{hello}{7}
indexentry{world}{7}
indexentry{world}{3}
indexentry{world}{13}
indexentry{world}{9}
indexentry{world}{5}

you can make index with makeindex or xindy then.

michal.h21
  • 50,697