In the solution below I've used
\numberwithin{equation}{section} % (1.1), (1.2), etc
\newtheorem{mytheorem}[equation]{Theorem} % Theorems share equation numbers
to get numbering (1.1), (1.2), etc. and then mytheorem gets the same numbers as equations.
I've borrowed some important code from Command behavior depending on current environment so that my newcounter subequations can be incremented, but only when within mytheorem.
I've used the etoolbox for its important \appto command.

Complete MWE:
% arara: pdflatex
\documentclass{article}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{etoolbox}
\numberwithin{equation}{section} % (1.1), (1.2), etc
\newtheorem{mytheorem}[equation]{Theorem} % Theorems share equation numbers
% https://tex.stackexchange.com/questions/39738/command-behavior-depending-on-current-environment
\makeatletter
\def\Mycurrentvir{document}
\def\ifenv#1{
\def\@tempa{#1}%
\ifx\@tempa\Mycurrentvir
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
}
\makeatother
\newcounter{subequation}
\appto\mytheorem{\setcounter{subequation}{0}%
\def\Mycurrentvir{mytheorem}%
\renewcommand{\theequation}{\arabic{section}.\arabic{equation}.\arabic{subequation}}}
\AtBeginEnvironment{equation}{\ifenv{mytheorem}{\refstepcounter{subequation}\addtocounter{equation}{-1}}{}}
\begin{document}
\section{My section}
\begin{equation}\label{eq:first}
y=mx+b
\end{equation}
\begin{mytheorem}\label{thm:test}
\begin{equation}\label{eq:test}
y=ax^2+bx+c
\end{equation}
Another equation
\begin{equation}
y=ax^2+bx+c
\end{equation}
\end{mytheorem}
\begin{equation}
y=x^3
\end{equation}
\begin{itemize}
\item First equation: \eqref{eq:first}
\item Equation within theorem: \eqref{eq:test}
\item Theorem reference: \ref{thm:test}
\end{itemize}
\section{another section}
\begin{equation}
y=mx+b
\end{equation}
\begin{mytheorem}
\begin{equation}
y=ax^2+bx+c
\end{equation}
Another equation
\begin{equation}
y=ax^2+bx+c
\end{equation}
\end{mytheorem}
\begin{equation}
y=x^3
\end{equation}
\end{document}