You have to set the transitions correctly and never use \fontspec in the document body.
\documentclass{article}
\usepackage[no-math]{fontspec}
\usepackage{ucharclasses}
\setmainfont{Khmer MN} % the only one I have supporting Khmer
\newfontfamily{\lmodern}{Latin Modern Roman}
\newfontfamily{\lucidacal}{Lucida Calligraphy OT}[Scale=MatchUppercase]
\setTransitionTo{Khmer}{\normalfont}
\setTransitionFrom{Khmer}{\lmodern}
\begin{document}
អត្ថបទជាភាសាខ្មែរ English {\lucidacal English text in Lucida
Calligraphy} and again Latin Modern
\end{document}

The rendering for Khmer is not good, but I know nothing about the language.
More complex setups are possible; here's an example.
\documentclass{article}
\usepackage[no-math]{fontspec}
\usepackage{ucharclasses}
\usepackage{etoolbox}
\setmainfont{Khmer Sangam MN}
\newfontfamily{\khmeraltfam}{Code2000}[Script=Khmer]
\newfontfamily{\lmodernfam}{Latin Modern Roman}
\newfontfamily{\lucidacalfam}{Lucida Calligraphy OT}[Scale=MatchUppercase]
\newrobustcmd{\lmodern}{\lmodernfam\let\currentenglish\lmodernfam}
\newrobustcmd{\lucidacal}{\lucidacalfam\let\currentenglish \lucidacalfam}
\newrobustcmd{\khmernormal}{\normalfont\let\currentkhmer\normalfont}
\newrobustcmd{\khmeralt}{\khmeraltfam\let\currentkhmer\khmeraltfam}
\lmodern \khmernormal % initialize
\setTransitionTo{Khmer}{\currentkhmer}
\setTransitionFrom{Khmer}{\currentenglish}
\begin{document}
អត្ថបទជាភាសាខ្មែរ English {\lucidacal English text in Lucida
អត្ថបទជាភាសាខ្មែរ
Calligraphy} and again Latin Modern
{\khmeralt អត្ថបទជាភាសាខ្មែរ}
\end{document}

\fontspecto execute font changes, have you considered issuing the directive\newfontfamily{\lm}{Latin Modern Roman}in the preamble and then typing something like\title{\lm Hello World}? – Mico Jan 06 '15 at 18:46