Besides Andrew's fine answer, it may be useful to know why you don't get the expected result.
When you do
\newenvironment{foo}{<begin code>}{<end code>}
and TeX expands \begin{foo}, it does several things, among which recording what environment it has begun (so to avoid bad nesting) and after this and other bookkeeping activity, it puts the <begin code> in the input stream, starting to expand or execute commands.
Similarly, when TeX expands \end{foo} it will do some other bookkeeping and then put <end code> in the input stream.
In your case, the <begin code> is just \ifhint. When this returns true because of a previous \hinttrue, the effect is that TeX records the presence of the conditional (which is internally \iftrue) so that, when it finds the matching \fi it will be satisfied (basically, an internal counter is stepped, although the inner working is a bit more complex). Macros and commands will be expanded or executed normally.
However, when \ifhint returns false, the same internal counter is stepped in order to look for the matching \fi. However tokens will be discarded with no interpretation whatsoever; only (primitive) conditionals will be taken note of, for the purpose of matching the corresponding \else and \fi.
In particular, the tokens \end{myhint} (in your case) will be discarded and not expanded, so the \fi will never be seen. The LaTeX run will go to the end of the file, unless some stray \fi comes along (which is impossible with well formed code).
Why does
\NewDocumentEnvironment{myhint}{+b}{\ifhint#1\fi}{}
work? Because the b argument type tells TeX to look for \end{myhint} storing everything that comes along in memory. When it finds \end{myhint}, it will stop storing and will deliver the whole content in place of #1. Thus
\begin{myhint}
Some text
\end{myhint}
will deliver
\ifhint Some text\fi
in the input stream. The matching \fi is no longer hidden even if \ifhint returns false.
Actually something more is done, for instance the endline after \begin{myhint} will not add a space in the output, nor will an endline before \end{myhint} cause a space in the output.
commentpackage? It does something similar to what you seem to be doing. – Teepeemm Sep 05 '20 at 22:13