How to format this formula, in particular, the parentheses better?

Question

 $$\mathrm{err} \left( h(\vec{\theta}, \vec{x}), y \right)
= \delta_{0y} ~ H \left( h(\vec{x}; \vec{\theta}) - \frac{1}{2} \right)
+ \delta_{1y} ~ H \left( \frac{1}{2} - h(\vec{x}; \vec{\theta}) \right)$$

Answer 1

1. You should never use $$ in LaTeX, see Why is \[ … \] preferable to $$?

2. “err” should be a math operator.

3. Never use a ~ in math mode; multiplication doesn't want space.

4. \left and \right add unwanted space when they surround the argument to a function.

5. A \vec near a closed parenthesis needs a thin space.

6. There's no need to have big parentheses for the argument to err; I'd use normal size, but I show also how to get slighly bigger ones.

\documentclass{article}
\usepackage{amsmath}

\DeclareMathOperator{\err}{err}

\begin{document}
First version
\[
\err( h(\vec{\theta}, \vec{x}\,), y )
= \delta_{0y} H \biggl( h(\vec{x}; \vec{\theta}\,) - \frac{1}{2} \biggr)
+ \delta_{1y} H \biggl( \frac{1}{2} - h(\vec{x}; \vec{\theta}\,) \biggr)
\]

Second version
\[
\err\bigl( h(\vec{\theta}, \vec{x}\,), y \bigr)
= \delta_{0y} H \biggl( h(\vec{x}; \vec{\theta}\,) - \frac{1}{2} \biggr)
+ \delta_{1y} H \biggl( \frac{1}{2} - h(\vec{x}; \vec{\theta}\,) \biggr)
\]

\end{document}

Other options include not having \DeclareMathOperator{\err}{err} in the preamble but the equivalent \operatorname{err} in the formula and using \tfrac{1}{2} as suggested by daleif, although I'm not sure about this. Here's the example.

\documentclass{article}
\usepackage{amsmath}

Third version
\[
\operatorname{err}\bigl( h(\vec{\theta}, \vec{x}\,), y \bigr)
= \delta_{0y} H \bigl( h(\vec{x}; \vec{\theta}\,) - \tfrac{1}{2} \bigr)
+ \delta_{1y} H \bigl( \tfrac{1}{2} - h(\vec{x}; \vec{\theta}\,) \bigr)
\]

\end{document}

Answer 2

Supposing it is a LaTeX document, first replace $$ … $$ with \[ … \]. Also define err as a math operator to have a correct horizontal spacing. Then you can use the nccmath package to use medium sized fractions, and replace \left … \right with the manually adjusted \Bigl … \Bigr:

\documentclass[12pt,a4paper]{article}

\usepackage[utf8]{inputenc}

\usepackage{mathtools}
\usepackage{nccmath} 

\DeclareMathOperator{\err}{err}

\begin{document}

\[ \err\bigl( h(\vec{\theta}, \vec{x}), y \bigr) = \delta_{0y}\,H\Bigl(h( \vec{x}; \vec{\theta}) - \mfrac{1}{2} \Bigr) + \delta_{1y}\, H\Bigl( \mfrac{1}{2} - h(\vec{x}; \vec{\theta}) \Bigr) \]

\end{document}