Why does minipage have different top/bottom margins for math environments?

Question

My goal is to create a framed minipage environment that will contain text and math environments in varying combinations. I noticed that the top (and bottom) margin of the minipage is quite a bit larger if the first (and last) entry is a math environment (than it is if the first thing is normal text).

I tried to extract the relevant parts into a MWE:

\documentclass{article}
\usepackage[fleqn]{amsmath}

\begin{document}
\setlength{\mathindent}{-.15em}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\setlength{\abovedisplayshortskip}{0pt}
\setlength{\belowdisplayshortskip}{0pt}
  \fbox{
    \begin{minipage}{.1\linewidth}
      XXX \\
      XXX
    \end{minipage}
  }
  \fbox{
    \begin{minipage}{.1\linewidth}
      \begin{align*}
        \textnormal{XXX}
      \end{align*}
      \begin{align*}
        \textnormal{XXX}
      \end{align*}
    \end{minipage}
  }
\end{document}

Which results in:

So why are there larger top/bottom margins for math environments and how do I get rid of them?