YAML header
---
mainfont: Font-Regular.otf
mainfontoptions:
- BoldFont=Font-Bold.otf
- ItalicFont=Font-Italic.otf
- BoldItalicFont=Font-BoldItalic.otf
---
command line
$ pandoc in.md --pdf-engine=xelatex \
-V 'mainfont:Font-Regular.otf' \
-V 'mainfontoptions:BoldFont=Font-Bold.otf, ItalicFont=Font-Italic.otf, BoldItalicFont=Font-BoldItalic.otf'
-o out.pdf
Since my answer to the original question, a lot has changed. So here is an update on how to set fonts with pandoc. There are several ways to change the font in pandoc and they are quite well documented nowadays. Here is a brief overview:
1. Changing fonts for XeLaTeX and LuaLaTeX:
You can specify any font on your system that can be used with fontspec
In the YAML header of your markdown file, set the variables like this for XeLaTeX:
---
mainfont: DejaVuSerif.ttf
sansfont: DejaVuSans.ttf
monofont: DejaVuSansMono.ttf
mathfont: texgyredejavu-math.otf
---
or, for LuaLaTex:
---
mainfont: DejaVuSerif
sansfont: DejaVuSans
monofont: DejaVuSansMono
mathfont: TeXGyreDejaVuMath-Regular
---
If you need more fine grained control, you can specify options:
---
mainfont: DejaVuSerif
mainfontoptions:
- Extension=.ttf
- UprightFont=*
- BoldFont=*-Bold
- ItalicFont=*-Italic
- BoldItalicFont=*-BoldItalic
---
Call pandoc with --pdf-engine set to xelatex or lualatex:
$ pandoc in.md --pdf-engine=xelatex -o out.pdf
You can also set the fonts on the command line:
$ pandoc in.md --pdf-engine=xelatex \
-V 'mainfont:DejaVuSerif.ttf' \
-V 'sansfont:DejaVuSans.ttf' \
-V 'monofont:DejaVuSansMono.ttf' \
-V 'mathfont:texgyredejavu-math.otf' \
-o out.pdf
or
$ pandoc in.md --pdf-engine=lualatex \
-V 'mainfont:DejaVuSerif' \
-V 'sansfont:DejaVuSans' \
-V 'monofont:DejaVuSansMono' \
-V 'mathfont:TeXGyreDejaVuMath-Regular' \
-o out.pdf
And again, with options:
$ pandoc in.md --pdf-engine=xelatex \
-V 'mainfont:DejaVuSerif' \
-V 'mainfontoptions:Extension=.ttf, UprightFont=*, BoldFont=*-Bold, ItalicFont=*-Italic, BoldItalicFont=*-BoldItalic' \
-V 'sansfont:DejaVuSans.ttf' \
-V 'monofont:DejaVuSansMono.ttf' \
-V 'mathfont:texgyredejavu-math.otf' \
-o out.pdf
Note, that xelatex wants the filename, while lualatex is content with the fontfamily name. You can use otfinfo to get more information on names and features of the font you are going to use.

2. Change the template
Some packages like dejavu-otf or libertinus-otf do the fontspec setup for you, but you have to modify the template in order for them to work.
Get the default template and save it to some file:
$ pandoc -D latex > template.latex
Open the file and remove all the font configuration code and replace it with, e.g.:
\usepackage{dejavu-otf}
3. Setting the fontfamily for pdflatex
If you are using pdflatex, you can use:
---
fontfamily: dejavu
---
or
$ pandoc in.md --pdf-engine=pdflatex \
-V 'fontfamily:dejavu' \
-o demo.pdf
The option fontfamily tries to load a package with the specified name. This usually applies for Type 1 fonts and pdflatex but there are also packages that use fontspec internally when called from lualatex or xelatex.
4. Setting the font for context
---
mainfont: DejaVuSerif
sansfont: DejaVuSans
monofont: DejaVuSansMono
mathfont: TeXGyreDejaVuMath
---
or
$ pandoc in.md --pdf-engine=context \
-V 'mainfont:DejaVuSerif' \
-V 'sansfont:DejaVuSans' \
-V 'monofont:DejaVuSansMono' \
-V 'mathfont:TeXGyreDejaVuMath' \
-o out.odf