Customising ToC appearance of chapter titles in front and back matter

Question

This might be a duplicate, but I am interested in customising the ToC appearance of chapter titles for \chapter commands placed:

1. in the front and back matter, and
2. after the \appendix command in the main matter.

I only wish to change font series (to medium) and shape (to italic) as shown below. (Vertical spacing between front matter chapter titles probably needs to be reduced as well.)

At the moment I'm using the standard document class book and:

\usepackage[nottoc]{tocbibind} % nottoc option for inclusion of LoT/LoF/Bib in ToC
\usepackage{hyperref}

Answer 1

One possibility using the titletoc package; the etoolbox package was used to patch \frontmatter, \mainmatter, and \appendix, so as to automatically use the corresponding style:

\documentclass{book}
\usepackage[nottoc]{tocbibind}
\usepackage{titletoc}
\usepackage{etoolbox}

\newcommand\frontformat{%
\titlecontents{chapter}[0em]
  {\itshape}{\contentslabel{0em}}
  {}{\normalfont\titlerule*[1pc]{.}\contentspage}}
\newcommand\mainformat{%
\titlecontents{chapter}[1.4em]
  {\addvspace{10pt}\bfseries}{\contentslabel{1.15em}}
  {}{\normalfont\titlerule*[1pc]{.}\bfseries\contentspage}
}
\newcommand\backformat{%
\titlecontents{chapter}[1.5em]
  {\addvspace{10pt}\itshape}{\contentslabel{1.5em}}
  {\hspace*{-1.5em}}{\normalfont\titlerule*[1pc]{.}\contentspage}}

\titlecontents{section}[3.8em]
  {}{\contentslabel{2.3em}}
  {\hspace*{-2.3em}}{\titlerule*[1pc]{.}\contentspage}

\apptocmd{\frontmatter}{\frontformat}{}{}
\apptocmd{\mainmatter}{\mainformat}{}{}
\apptocmd{\appendix}{\backformat}{}{}

\begin{document}

\frontmatter

\tableofcontents
\chapter{Preface}
\listoftables
\listoffigures

\mainmatter

\chapter{Introduction}
\chapter{Results and Discussion}
\section{Section}
\section{Another Section}
\chapter{Conclusion}

\appendix

\chapter{My Appendix}
\cite{testa}

\backmatter

\begin{thebibliography}{9}
\bibitem{testa} Author A, Title A, 2012
\end{thebibliography}

\end{document}

Answer 2

The solution of Gonzalo Medina uses package titletoc that does not work well with hyperref, because it extracts the heading number of the entry in the table of contents (and LOF, LOT) and sets it outside the link.

The following solution patches \l@chapter, \l@section, …, which are responsible for the formatting of the entry in the table of contents to add \TocTitleFormat. Depending on the switch \ifTocMainMatter, \TocTitleFormat does nothing or adds \textit for the title. If \itshape is used instead, also the page number would be in italic.

Similar to Medina's solution \frontmatter and friends are patched to add the setting of the switch \ifTocMainMatter into the .toc file.

Since hyperref is used, likely also bookmarks are generated. The same style can be applied to them via package bookmark.

\documentclass{book}
\usepackage[nottoc]{tocbibind}
\usepackage{etoolbox}
\usepackage{hyperref}
\usepackage{bookmark}
\bookmarksetup{numbered,open}

\makeatletter
\newcommand*{\PatchTocEntry}[1]{%
  \expandafter\@PatchTocEntry\csname l@#1\expandafter\endcsname
  \csname org@l@#1\endcsname{org@l@#1}%
}
\newcommand*{\@PatchTocEntry}[3]{%
  \@ifdefinable{#3}{%
    \let#2#1%
    \renewcommand*{#1}[1]{%
      #2{\TocTitleFormat{##1}}%
    }%
  }%
}
\@for\@tmp:=chapter,section,subsection,subsubsection,paragraph,subparagraph\do{
  \PatchTocEntry{\@tmp}%
}
\newif\ifTocMainMatter
\let\ifTocMainMatter\if@mainmatter
\newcommand*{\SetTocMainMatter}[1]{%
  \addtocontents{toc}{%
    \expandafter\protect\csname TocMainMatter#1\endcsname
  }%
  \begingroup
  \edef\x{\endgroup
    \noexpand\bookmarksetup{%
      italic=\csname if#1\endcsname false\else true\fi
    }%
  }\x
}
\apptocmd{\frontmatter}{\SetTocMainMatter{false}}{}{}
\apptocmd{\mainmatter}{\SetTocMainMatter{true}}{}{}
\apptocmd{\appendix}{\SetTocMainMatter{false}}{}{}
\apptocmd{\backmatter}{\SetTocMainMatter{false}}{}{}
\newcommand*{\TocTitleFormat}[1]{%
  \ifTocMainMatter
    #1%
  \else
    \textit{#1}%
  \fi
}
\makeatother

\begin{document}

\frontmatter

\tableofcontents
\chapter{Preface}
\listoftables
\listoffigures

\mainmatter  

\chapter{Introduction}
\chapter{Results and Discussion}
\section{Section}
\section{Another Section}
\chapter{Conclusion}

\appendix

\chapter{My Appendix}
\section{Section in Appendix}
\cite{testa}

\backmatter

\begin{thebibliography}{9}
\bibitem{testa} Author A, Title A, 2012
\end{thebibliography}

\end{document}