In theory, a character that is special for MakeIndex, for instance !, |, @ should be quoted with ". However, this confuses hyperref when | is used.
The workaround is simple:
\index{$\vert F\vert$}
or, better, loading amsmath and using an alphabetizing string:
\index{Fabs@$\lvert F\rvert$}
Using \lvert and \rvert is recommended anyway.
You can also load mathtools and do
\DeclarePairedDelimiter{\abs}{|}{|}
and use $\abs{F}$ (look at the package documentation for more information).
The \abs strategy is better for another reason: if \index appears in the argument to another command, such as
\newcommand\keyword[1]{\underline{\bfseries #1}\index{#1}}
and then called as
\keyword{$\vert F\vert$}
the \vert command is expanded (see Symbol index sorted by occurrence for a similar case). If you use \abs, this won't happen.
Note: \bf is a deprecated command; I changed the definition you showed in the comment to be in line with the recommended practice.
\index{$"|F"|$}should do. – egreg May 13 '14 at 13:18hyperref, the *.idx file looks like:\indexentry{$"|hyperindexformat{\F"}}{1}, which results in an error. – Martin Wang May 13 '14 at 13:33