You got an error with \newcounter{example}[section*] because section* is not a counter in LaTeX. section is, but not section*.
First method: patching \@startsection
You can patch \@startsection so that when \section* is called (which is a sectioning command of level 1) and only in this case, your example counter is reset to 0. The following works because \@ssect is only called for unnumbered sectioning commands, and we only reset the counter when the level of the sectioning command is 1 (\section or \section*).
\documentclass{article}
\usepackage{etoolbox}
\newcounter{example}[section]
\newenvironment{example}[1][]
{\refstepcounter{example}\par\medskip
\subsection*{Example~\theexample. #1}%
\rmfamily}
{\medskip}
\makeatletter
\patchcmd{\@startsection}
{\@ssect}{\ifnum#2=1 \setcounter{example}{0}\fi\@ssect}
{}{\FAILED}
\makeatother
\begin{document}
\section*{Section}
\begin{example}
A
\end{example}
\begin{example}
B
\end{example}
\section*{Another section}
\begin{example}
C
\end{example}
\begin{example}
D
\end{example}
\section{Numbered section}
\begin{example}
E
\end{example}
\begin{example}
F
\end{example}
\end{document}
Second method: using xparse to redefine \section
Another way to achieve the same result is to use xparse in order to redefine \section, as egreg did in this answer. The advantage of this technique is that it doesn't rely on implementation details of \section and \@startsection, therefore it should always work even if \section has been redefined and is not LaTeX2e's \section command anymore—this may be the case if you redefine \section using a package such as titlesec, or simply if you try to pile up several redefinitions done with the first technique. Thus, I would say that this second method is better from an engineering point of view.
\documentclass{article}
\usepackage{xparse}
\newcounter{example}[section]
\newenvironment{example}[1][]
{\refstepcounter{example}\par\medskip
\subsection*{Example~\theexample. #1}%
\rmfamily}
{\medskip}
% Save the original \section command
\let\latexsection\section
% Simple redefinition of \section (including \section*)
\RenewDocumentCommand{\section}{sO{#3}m}{%
\IfBooleanTF{#1}
{\setcounter{example}{0}%
\latexsection*{#3}}
{\latexsection[#2]{#3}}%
}
\begin{document}
\section*{Section}
\begin{example}
A
\end{example}
\begin{example}
B
\end{example}
\section*{Another section}
\begin{example}
C
\end{example}
\begin{example}
D
\end{example}
\section{Numbered section}
\begin{example}
E
\end{example}
\begin{example}
F
\end{example}
\end{document}
Both techniques lead to the following output:

Remarks on implementation of user-defined environments
In your real environment, you may want to do something like this (or a variant) in order to be sure that your input doesn't introduce unwanted spaces. This is a general remark regarding user-defined environments; since \subsection* typically starts a new paragraph—unless you are using the run-in layout type—the \ignorespaces is probably not needed here, for example. Similarly, \ignorespacesafterend shouldn't be necessary after \par\medskip, since \par puts TeX in vertical mode where spaces are ignored—but it shouldn't harm, nevertheless.
\newenvironment{example}[1][]
{\refstepcounter{example}\par\medskip
\subsection*{Example~\theexample. #1}%
\rmfamily
\ignorespaces}
{\unskip
\par\medskip
\ignorespacesafterend}
\section*{...}\setcounter{example}{0}– Fran Aug 11 '19 at 06:39