You can do this using XeTeXinterchartoks to insert the font change between characters of defined classes. I've adapted the code here from the replace-numbers.sty which is part of xelatex.
Unfortunately the XeTeX documentation on this is a bit out of date, and the character class of the boundaries has been changed to 4095. This is set to \e@alloc@intercharclass@top in the LaTeX kernel, so I've used this instead of a hard coded number. Thanks to the experts in chat for explaining this change.
In the example below I've added a \color{red} to the code to show the change clearly. This obviously should be removed for your actual use case.
% !TEX TS-program = xelatex
\documentclass{article}
\usepackage{xcolor}
\usepackage{fontspec}
\setmainfont{Linux Libertine O}
\newfontfamily\parenfont{Linux Biolinum O}
\makeatletter
\XeTeXinterchartokenstate=1
\chardef\CharNormal=0
\ifx\e@alloc@intercharclass@top\@undefined
\chardef\CharBound=255
\else
\chardef\CharBound=\e@alloc@intercharclass@top
\fi
\newXeTeXintercharclass\CharParens
\XeTeXcharclass`(=\CharParens
\XeTeXcharclass`)=\CharParens
\newtoks\TokSetfont
\TokSetfont={\begingroup\parenfont\color{red}} % remove \color{red} for actual use
\XeTeXinterchartoks\CharNormal\CharParens=\TokSetfont
\XeTeXinterchartoks\CharBound\CharParens=\TokSetfont
\XeTeXinterchartoks\CharParens\CharNormal={\endgroup}
\XeTeXinterchartoks\CharParens\CharBound={\endgroup}
\makeatother
\begin{document}
Proin quis (tortor) orci. Etiam at risus et $(a + b)x$.
\end{document}

Problems with this code
This code checks for older versions of the latex kernel, and sets the boundary char class appropriately for that. However, there was one version of the kernel released in which \e@alloc@intercharclass@top was not defined although the char class was changed. If the above code doesn't run, try replacing the following code:
\ifx\e@alloc@intercharclass@top\@undefined
\chardef\CharBound=255
\else
\chardef\CharBound=\e@alloc@intercharclass@top
\fi
with this:
\chardef\CharBound=4095