I would like to know what is the right way to define all the text font as a specific size. I use \documentclass[20pt]{article} but it won't change the font size in xelatex. And these symbols (" and ') are not Times fonts and now matter how I changed the fonts , they look the same. Here is the min working code. I would like to know what is the right way to address font selection in xelatex. Can I get like this ?
Instead of this ? 
%!TEX TS-program = xelatex
\documentclass[20pt]{article}
\usepackage{geometry}
\usepackage{mathspec}
\usepackage{lipsum}
\setmainfont{Times}
\setmathsfont(Digits,Latin,Greek){Times}
\usepackage{titling}
\usepackage{titlesec}
\usepackage{setspace}
\usepackage[colorlinks,urlcolor=black]{hyperref}
\doublespacing
\titleformat{\section}{\singlespacing\fontsize{20}{24}\bfseries\rmfamily\uppercase} {\thesection.}{0.5em}{}
\begin{document}
\section{"check"}
"just checking" this is a "check" and 'check'.
\end{document}


Mapping=tex-textto your\setmainfontcommand. Opening quotes are typed with\`` and```; closing quotes are typed with'and''. Don't use", since it has incorrect spacing. – ChrisS Feb 25 '14 at 08:54