Your write-up of the substitution rules isn't very clear. I've interpreted them as follows:
If the letters F, G, and H are followed by a lowercase vowel (i.e., one of aeiou), the uppercase-lowercase letter pair should be replaced with P, Q, and R, respectively.
If the letters F, G, and H are followed by the uppercase letter Y, they (including the letter Y) should be replaced with S, T, and U, respectively.
If neither case 1 nor case 2 applies, the letters F, G, and H should be replaced with Px, Qx, and Rx, respectively.
All other material in the document should not be be affected by the preceding three substitution rules.
As long as the body of your document doesn't contain macros or environments whose names contain the letters F, G, and/or H, you can simply "turn on" the Lua function -- more precisely, assign it to LuaTeX's process_input_buffer callback, making the function act like a preprocessor on the input stream -- by running \fghOn after \begin{document}. It will then keep on running until it gets to \end{document}. The Lua function can be switched off via \fghOff. If you need to perform the substitution on just one string at a time, you could use the macro \fghSub -- cf. its use in the final row of the following working example.

\documentclass{article}
\usepackage{luacode} % for '\luaexec' and '\luastringN' macros
\luaexec{ % set up Lua function called 'fhg_sub':
function fgh_sub ( s )
s = s:gsub ( "F[aeiou]" , "P")
s = s:gsub ( "G[aeiou]" , "Q")
s = s:gsub ( "H[aeiou]" , "R")
s = s:gsub ( "FY" , "S" )
s = s:gsub ( "GY" , "T" )
s = s:gsub ( "HY" , "U" )
s = s:gsub ( "F" , "Px" )
s = s:gsub ( "G" , "Qx" )
s = s:gsub ( "H" , "Rx" )
return s
end
}
%% Three LaTeX macros to interface with the Lua function:
\newcommand\fghOn{\directlua{luatexbase.add_to_callback(
"process_input_buffer", fgh_sub, "fgh_sub" )}}
\newcommand\fghOff{\directlua{luatexbase.remove_from_callback(
"process_input_buffer", "fgh_sub" )}}
\newcommand\fghSub[1]{\directlua{tex.sprint(fgh_sub(\luastringN{#1}))}}
\begin{document}
\obeylines % just for this example
\fghOn % assign 'fgy_sub' Lua function to preprocessor stack
FGH
FaGaHeF
FYGYHY
\fghOff % remove Lua function from preprocessor stack
Figure, Garage, Hotel
\fghSub{Figure, Garage, Hotel} % run the function on a one-off basis
\end{document}
Figure,Garage, andHotel? Or should they be converted toPgure,Qrage, andRtel, respectively? Please advise. – Mico Jul 17 '19 at 16:47