Memoir: Decorate chapter/section/... with a star in title and TOC, but not \ref{}

Question

I'd like to be able to mark chapter/section/... with a star (i.e., "difficult/optional material"), but I consider the "*" as a decoration only, it should not affect references (if chapter 1 is optional, it is marked as such in the heading and in TOC, but it is still just chapter 1 for references).

The solution using xparse to question 450666 here is pretty close to what I'm looking for, but \ref{opt} produces "*1.2", I'd like plain "1.2". And I'd like to extend to chapter/section/subsection/... (I think that should be easy, once the basic mechanism is ironed out).

Update: The referenced partial solution to the problem (by @egreg) is:

\documentclass{memoir}

\usepackage{xparse}
\usepackage{kantlipsum}

\NewDocumentCommand{\optionalsection}{O{#3}om}{%
\renewcommand{\thesection}{\optasterisk\standardthesection}%
   \optzerotrue
   \IfNoValueTF{#2}{\section[#1]{#3}}{\section[#1][#2]{#3}}%
   \optzerofalse
   \renewcommand{\thesection}{\standardthesection}%
}
\AtBeginDocument{\let\standardthesection\thesection}

\NewDocumentCommand{\optasterisk}{}{%
   \ifoptzero\makebox[0pt][r]{*}\else*\fi
}
\newif\ifoptzero

\begin{document}

   \frontmatter

   \optzerotrue
   \tableofcontents*
   \optzerofalse

   \mainmatter
   \chapter{Test}

   \section[TOC title][Header title]{Main title}

   \optionalsection[TOC title opt][Header title opt]{Main title opt}\label{opt}

   This is a reference to \ref{opt}.

   \kant % to see the header title

   \end{document}

I'd like to extend this to \chapter, \subsection, ... (I believe that should be easy to manage). But I dont want the references to show up with "*", just titles and TOC.

As a comment says, yes, I do need hyperref. It is math-heavy text, several of the ams packages are loaded too. In any case, I think I can fix any fallout from those; if not, I'll be back here ;-)

A short test including hyperref seems to work fine.