mathspec is a clever attempt to make it possible to use math in XeLaTeX documents, with system fonts for the letters and symbols from standard math fonts.
With unicode-math one can basically use only specially tailored OpenType math fonts such as Latin Modern Math, TeX Gyre Termes Math, TeX Gyre Pagella Math, XITS Math, Asana Math (among the free ones) or Cambria Math and Lucida Bright Math (not free).
mathspec
With this package you can do limited math with the main system font. Just a silly example:
\documentclass{article}
\usepackage{mathspec}
\setmainfont[Ligatures=TeX]{Old Standard}
\setmathsfont(Digits,Latin,Greek){Old Standard}
\begin{document}
Some text and a formula $a+b=\int_{\xi}^{\theta} f(x)\,dx$.
\end{document}

The emulated math is not bad, and the package has features for manually correcting the possible bad spacings, which happen because the fonts have really no support for math. For instance, inputting the formula as
$a+b=\int_{\xi}^{\theta} "f(x)\,dx$
would give the better result

The " means ”add some space on both sides of the following letter”.
Digits, Latin and Greek letters can be taken from different fonts.
unicode-math
The unicode-math is compatible both with XeLaTeX and LuaLaTeX. Here's the same example with Lucida Bright:
\documentclass{article}
\usepackage{unicode-math}
\setmainfont[Ligatures=TeX,Scale=0.85]{Lucida Bright OT}
\setmathfont[Scale=0.85]{Lucida Bright Math OT}
\begin{document}
Some text and a formula $a+b=\int_{\xi}^{\theta} f(x)\,dx$.
\end{document}

Spacing is correct without manual intervention, because the font used for math is a real math font. Here's the same with TeX Gyre Pagella:
\documentclass{article}
\usepackage{unicode-math}
\setmainfont[Ligatures=TeX]{TeX Gyre Pagella}
\setmathfont{TeX Gyre Pagella Math}
\begin{document}
Some text and a formula $a+b=\int_{\xi}^{\theta} f(x)\,dx$.
\end{document}

Unicode input
One can input math with Unicode symbols, but this is not mandatory. It's possible both with mathspec and unicode-math. For instance, the integral with mathspec could be input as
$∫_ξ^θ "f(x)\,dx$
provided that ∫ is made known to the environment, for instance with
\usepackage{newunicodechar}
\newunicodechar{∫}{\int}
Some symbols (in particular letters) are already known.
Nothing particular is needed for inputting the integral as
$∫_ξ^θ f(x)\,dx$
with unicode-math. However the traditional syntax is understood by the package.
\setmathfont{}do? I read something aboutmathspecbeing XeTeX only, whileunicode-mathalso supports LuaLaTeX; is that true? I think having portable documents is always better. – Henri Menke Jun 08 '13 at 14:31mathspecis not compatible with LuaTeX. (The extra\setmathfontwas a typo). – Alan Munn Jun 08 '13 at 14:33\innot negated in libertine; "slash-space" before\forallnot observed in libertine; spacing around colons not the same (different unicode?); spacing/kerning around finalfin libertine is terrible. – barbara beeton Jun 08 '13 at 16:04mathspecpackage unless you're using a complete unicode math font like STIX or Asana Math, Cambria Math, etc. – Alan Munn Jun 08 '13 at 16:10unicode-math; it works perfectly with traditional input. – egreg Jun 08 '13 at 17:09unicode-mathis right, the Unicode input is merely a side effect of the main goal; supporting Unicode math fonts, and you can still use csnames for math inout, which is actually what most people do. – خالد حسني Jun 08 '13 at 17:27