Unfortunately, this is how \pagestyle works. \pagestyle{plain} executes all the commands given in its defintion (in addition to the fancyhdr setup), so \fancyhf{} is executed in the global context.
A later \pagestyle{fancy} doesn't undo this. Only when the \pagestyle{plain} is given in a local context, its changes are restricted to that context. This happens for example with \thispagestyle, or when you give \pagestyle{} in a group (see solution 1). Otherwise you must put the header definitions explicitely in pagestyle fancy (see solution 2).
In the next version of fancyhdr I will have a better solution for this, but for now you can use one of the solutions below.
Solution 1: Use a group. Note the \clearpage must be inside the group, otherwise the original headers will leak into the previous page.
\documentclass{article}
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\fancyhead[R]{\rightmark}
\fancyhead[L]{\leftmark}
\fancypagestyle{plain}{\fancyhf{}}
\begin{document}
\begingroup
\pagestyle{plain} % with this, '\pagestyle{fancy}' below doesn't work
\section{Test}
\clearpage
\endgroup
\pagestyle{fancy} % doesn't work
\subsection{test}
\end{document}
Solution 2: Put the header definitons inside the pagestyle fancy. Note This will only work in fancyhdr version 4, otherwise you will have to use a different pagestyle name.
\documentclass{article}
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\fancypagestyle{fancy}{
\fancyhead[R]{\rightmark}
\fancyhead[L]{\leftmark}
}
\fancypagestyle{plain}{\fancyhf{}}
\begin{document}
\pagestyle{plain} % with this, '\pagestyle{fancy}' below doesn't work
\section{Test}
\clearpage
\pagestyle{fancy} % doesn't work
\subsection{test}
\end{document}

fancyhdr, and it is not until now that I realize that it may cause trouble. – Jinwen Mar 09 '21 at 08:42\pagestyle{fancy}should be placed after\fancypagestyle{fancy}?\fancyhf{}doesn't seem to get executed if\pagestyle{fancy}is in front of\fancypagestyle{fancy}. – Jinwen Mar 09 '21 at 08:55