You can have thick \hlines with the makecell package and its \Xhline{width} and \Xcline{col1-col2}{width} commands. You can replace the vertical spacing added by booktabs around horizontal rules (that's responsible for non joining vertical rules) with the cellspace package; it lets you define minimal vertical spacingabove and below cells of a given column, prefixing its specifier with the letter S.
I also took the opportunity to define a set command, based on an example in the documentation of the mathtools package, that produces better horizontal spacing and variable-sized braces
\documentclass{article}
\usepackage{makecell}
\usepackage{multirow}
\usepackage{hhline}
\usepackage{cellspace}
\setlength\cellspacetoplimit{5pt}
\setlength\cellspacebottomlimit{5pt}
\usepackage{mathtools}
\DeclarePairedDelimiterX\set[1]\{\}{\nonscript\,#1\nonscript\,}
\begin{document}
\begin{table}[!htb]
\centering
\small
\begin{tabular}{| Sc | >{\itshape}l | Sc c c |}
\hline
{\bfseries HD Level} & & & & \\
\Xhline{2pt}
\multirow{2}{*}{1} & Rep-tile: & $\set{1,2}$ & $\set{1,3}$ & $\set{2,3}$ \\
\cline{2-5}
& Encoding:\ \ \ \ & 1 & 2 & 3 \\
\Xhline{2pt}
\multirow{2}{*}{2} & Rep-tile: & $\set{2}$ & $\set{3}$ & \\
\cline{2-5}
& Encoding: & 1 & 2 & \\
\hline
\end{tabular}
\caption{\small\bf Bla bla}
\label{tab:SMuTexEncoding}
\end{table}
\end{document}

If you want to have coloured rules, it's more complelex: the \arrayrulecolor command (from the colortbl package) doesn't work. A workaround consistsin using the hhline package, setting \arrayrulewidth to 0pt and colouring the interrule space of a double rule. Demo:
\begin{table}[!htb]
\centering
\small\arrayrulecolor{red}
\begin{tabular}{| Sc | >{\itshape}l | Sc c c |}
\hline
{\bfseries HD Level} & & & & \\
\Xhline{2pt}
\multirow{2}{*}{1} & Rep-tile: & $\set{1,2}$ & $\set{1,3}$ & $\set{2,3}$ \\
\cline{2-5}
& Encoding:\ \ \ \ & 1 & 2 & 3 \\
\Xhline{2pt}
\multirow{2}{*}{2} & Rep-tile: & $\set{2}$ & $\set{3}$ & \\
\cline{2-5}
& Encoding: & 1 & 2 & \\
\hhline{>{\arrayrulewidth = 0pt\doublerulesep = 2pt \doublerulesepcolor{red}}=====}
\end{tabular}
\caption{\small\bf Bla bla}
\label{tab:SMuTexEncoding}
\end{table}

booktabs! – Oct 16 '14 at 16:47booktabs, you need to consider reading the documentation as well. It states: "You will not go far wrong if you remember two simple guidelines at all times: 1. Never, ever use vertical rules. 2. Never use double rules." – Werner Oct 16 '14 at 16:48booktabs. I invite you to look at my realization of your table in this picture. I would probably remove the\cmidrule{2-5}commands too. The superheavy rules are out of the question. – egreg Oct 16 '14 at 16:55booktabs. – Mico Oct 16 '14 at 17:15booktabs. Basic LaTeX providescline. Please note, that\itis deprecated for about 20 years now. – Johannes_B Oct 16 '14 at 17:16booktabsavoids the use of vertical lines in a table, is because they are redundant most of the time. Especially if it comes to the outer edges of the table. From the point of view of visual design they are not necessary because the vertical margins of the book or the page containing this table serve as such and are therefore unnecessary. – Aradnix Oct 16 '14 at 17:16LaTeX. Internally, depending on the type of information contained in the table may require an internal vertical line or some other typographical distinction as the case may.But anyway if good design doesn't mind, and you want yes or yes use vertical lines, my suggestion is to don't use
– Aradnix Oct 16 '14 at 17:18booktabsand instead you can manually setting the stroke width of the lines, just as egreg or Johannes_B suggested.\clineif you are using vertical rules. Also\scalebox{.8}\smallmakes a box with just the argument\smallwhich is odd?? – David Carlisle Oct 16 '14 at 17:36