As Egreg said, makeindex, which is used with imakeidx to produce index by default, doesn't support unicode. There is support for vietnamese in xindy, the other index processor. Unfortunately, there is some problem with pdflatex, Vietnamese language and xindy, because index entries are written as TeX sequences, not utf-8 codes, and xindy is unable to sort them correctly. This doesn't happen with other languages, so it is probably a bug. Nevertheless, you can switch to xelatex, which is good idea for Asian language anyway, and the index is then processed correctly, I hope.
Modified preamble:
\documentclass{book}
\usepackage{polyglossia}
\setmainlanguage{vietnamese}
\usepackage[xindy]{imakeidx}
\makeindex[columns=3,options={-M lang/vietnamese/utf8-lang}]
For xelatex, use polyglossia package instead of vietnam, we use xindy option for imakeidx to avoid makeindex usage and options={-M lang/vietnamese/utf8-lang} will use Vietnamese sorting.
compile with xelatex -shell-escape filename. the -shell-escape option enables compilation of the index directly when your document is compiled. if you are not using the command line to compile your document, you must configure your editor to use this option.
The result:

Edit:
As workaround for pdflatex, I wrote simple filter which convert control sequences generated by inputenc with utf8 option to utf8 characters.
iec2utf.lua:
kpse.set_program_name("luatex")
local cmd_arg = [[
iec2utf.lua - Script for converting LaTeX LICR codes to utf8 characters
Usage: cat filename | texlua iec2utf.lua > newfilename
Parameters:
comma separated list of LaTeX font encodings used in the document
]]
local enc = {}
local licrs = {}
local codepoint2utf = unicode.utf8.char
function load_encodings(f)
local file= io.open(f,"r")
local encodings = file:read("*all")
file:close()
for codepoint, licr in encodings:gmatch('DeclareUnicodeCharacter(%b{})(%b{})') do
local codepoint = codepoint2utf(tonumber(codepoint:sub(2,-2),16))
local licr= licr:sub(2,-2):gsub('@tabacckludge','')
licrs[licr] = codepoint
end
end
function sanitize_licr(l)
return l:gsub(" (.)",function(s) if s:match("[%a]") then return " "..s else return s end end):sub(2,-2)
end
if arg[1] == nil then
enc = {"T1"}
else
for _,n in pairs(arg) do
enc[#enc+1] = n
end
end
for _,e in pairs(enc) do
local filename = e:lower() .. "enc.dfu"
local dfufile = kpse.find_file(filename)
if dfufile then
load_encodings(dfufile)
end
end
local input = io.read("*all")
local cache = {}
local output = input:gsub('\\IeC[%s]*(%b{})',function(iec)
local code = cache[iec] or licrs[sanitize_licr(iec)] or '\\IeC '..iec
-- print(iec, code)
cache[iec] = code
return code
end)
io.write(output)
save this script to the same directory as your tex file. This script takes as parameter used font encoding, which is T5 in the case of Vietnamese.
if you are using Windows, you can create batch script to use this filter:
vietxindy.bat:
@echo of
type %~n1.idx | texlua iec2utf.lua T5 | texindy -i -M lang/vietnamese/utf8-lang -o %~n1.ind
for Linux, create shell script vietxindy:
#!/usr/bin/env sh
texlua iec2utf.lua T5 < `basename $1 .tex`.idx | texindy -i -M lang/vietnamese/utf8-lang -o `basename $1 .tex`.ind
And then compile the document with:
pdflatex filename
vietxindy filename
pdflatex filename
from the command line