As David Carlisle says, the answer is to use \frac instead of \dfrac, which will make the inner denominator fraction shrink appropriately. That's the correct answer to the question, but the question is wrong because the desired result looks poor. A better result is
\begin{equation}
\min \left [ \frac{3}{\mu(S) + D_{\psi}(S,\beta) + \xi(S)} \right ]
\label{eq_1}
\end{equation}
I make this answer not to force a particular style, but to point out that typographic needs come into play and should be given some weight when deciding how to present material. Avoiding tiny text is still important, even when most people read pdfs on screen rather than photocopies on paper. Deeply or asymmetrically nested fractions can often benefit from reformatting -- using mathematical rearrangement, inline "slash" fractions, or the occasional ^{-1} for a reciprocal. Nested exponents or exponents with fractions can benefit from using the \exp function instead of e^.
\dfracjust use\fracand suitable sizes will be chosen – David Carlisle Oct 27 '20 at 19:03bmatrixby Bernard. – Joao Ricardo Braga Paiva Oct 28 '20 at 14:09