TikZ image of Masked Autoencoder for Distribution Estimation (MADE)

Question

I'm trying to recreate this image of a MADE net in TikZ.

Here's what I have so far.

Any advice on how to draw the mask matrices and perhaps how to incorporate the numbers inside the neurons of the MADE net into the drawLayers macro would be much appreciated. (Those numbers indicate the maximum number of input units that affect the neuron in question.)

\documentclass[tikz]{standalone}
\usetikzlibrary{positioning}
\newcommand\drawLayers[2]{
  % #1 (str): namespace
  % #2 (list[int]): number of nodes in each layer
  \foreach \neurons [count=\lyrIdx] in #2
    \foreach \n in {1,...,\neurons}
      \node[neuron] (#1-\lyrIdx-\n) at (1.5*\lyrIdx, \neurons/2-\n) {};
  }
\newcommand\denselyConnectNodes[2]{
  % #1 (str): namespace
  % #2 (list[int]): number of nodes in each layer
  \foreach \n [count=\lyrIdx, remember=\lyrIdx as \previdx, remember=\n as \prevn] in #2 {
    \foreach \y in {1,...,\n} {
      \ifnum \lyrIdx > 1
        \foreach \x in {1,...,\prevn}
          \draw (#1-\previdx-\x) -- (#1-\lyrIdx-\y);
      \fi
    }
  }
}
\newcommand\connectSomeNodes[2]{
  % #1 (str): namespace
  % #2 (list[list[list[int]]]): for each node in each layer, list all connected nodes in the next layer
  \foreach \layer [count=\lyrIdx, evaluate=\lyrIdx as \nextLyr using int(\lyrIdx+1)] in #2
    \foreach \neuron [count=\nIdx] in \layer
        \foreach \edge in \neuron
          \draw (#1-\lyrIdx-\nIdx) -- (#1-\nextLyr-\edge);
}
\begin{document}
\begin{tikzpicture}[
    shorten >=1pt, shorten <=1pt, ->,
    neuron/.style={circle, draw, minimum size=4ex, thick},
    legend/.style={font=\large\bfseries},
  ]
% Fully-connected neural net
  \drawLayers{fcnn}{{3, 4, 4, 3}}
  \denselyConnectNodes{fcnn}{{3, 4, 4, 3}}
\path (fcnn-1-1) -- (fcnn-2-1) node[midway, above=1ex] {$W_1$};
  \path (fcnn-2-1) -- (fcnn-3-1) node[midway, above=1ex] {$W_2$};
  \path (fcnn-3-1) -- (fcnn-4-1) node[midway, above=1ex] {$V$};
% MADE net
  \begin{scope}[xshift=10cm]
    \drawLayers{made}{{3, 4, 4, 3}}
    \connectSomeNodes{made}{{
          {{}, {1,2,3,4}, {1,3,4}},
          {{2,3}, {1,2,3,4}, {2,3}, {2,3}},
          {{1,3}, {1}, {1}, {1,3}},
        }}
  \end{scope}
% Input + output labels
  \foreach \label [count=\c] in {{$p(x_1|x_2,x_3)$}, $p(x_2)$, $p(x_3|x_2)$} {
      \node[left=0 of fcnn-1-\c] {$x_\c$};
      \node[right=0 of fcnn-4-\c] {$\hat x_\c$};
      \node[left=0 of made-1-\c] {$x_\c$};
      \node[right=0 of made-4-\c] {\label};
    }
\node[legend, below=0.5 of fcnn-3-4] {autoencoder};
  \node[legend, below=0.5 of made-2-4] {MADE};
\end{tikzpicture}
\end{document}

You're asking too many different questions at once. The principle here is to ask only one problem per question. What is the question being asked here? the mask or the numbers in the neurons? When this question has an answer, you can ask another one. — AndréC, Aug 14 '20 at 05:46
@AndréC Only asking two questions actually and the second was marked as optional by the word "perhaps". In any case, I found a solution to both questions. Didn't want to answer my own questions right away in case someone had already put in work towards an answer but will post it now. — Janosh, Aug 14 '20 at 08:21

Janosh · Answer 1 · 2020-08-14T11:19:06.563

Answering my own questions, here's the completed image:

% TikZ-reproduction of fig. 1 from the paper MADE: Masked Autoencoder for Distribution Estimation (https://arxiv.org/abs/1502.03509).
\documentclass[tikz]{standalone}
\usepackage{xstring}
\usetikzlibrary{calc,positioning}
\newcommand\drawNodes[2]{
  % #1 (str): namespace
  % #2 (list[list[str]]): list of labels to print in the node of each neuron
  \foreach \neurons [count=\lyrIdx] in #2 {
    \StrCount{\neurons}{,}[\arrlength] % uses the xstring package
    \foreach \n [count=\nIdx] in \neurons
      \node[neuron] (#1-\lyrIdx-\nIdx) at (\arrlength/2-\nIdx, 1.5*\lyrIdx) {\n};
  }
}
\newcommand\denselyConnectNodes[2]{
  % #1 (str): namespace
  % #2 (list[int]): number of nodes in each layer
  \foreach \n [count=\lyrIdx, remember=\lyrIdx as \previdx, remember=\n as \prevn] in #2 {
    \foreach \y in {1,...,\n} {
      \ifnum \lyrIdx > 1
        \foreach \x in {1,...,\prevn}
          \draw[->] (#1-\previdx-\x) -- (#1-\lyrIdx-\y);
      \fi
    }
  }
}
\newcommand\connectSomeNodes[2]{
  % #1 (str): namespace
  % #2 (list[list[list[int]]]): for each node in each layer, list all connected nodes in the next layer
  \foreach \layer [count=\lyrIdx, evaluate=\lyrIdx as \nextLyr using int(\lyrIdx+1)] in #2
    \foreach \neuron [count=\nIdx] in \layer
        \foreach \edge in \neuron
          \draw[->] (#1-\lyrIdx-\nIdx) -- (#1-\nextLyr-\edge);
}
\begin{document}
\begin{tikzpicture}[
    shorten >=1pt, shorten <=1pt,
    neuron/.style={circle, draw, minimum size=4ex, thick},
    legend/.style={font=\large\bfseries},
  ]
% Fully-connected neural net
  \drawNodes{fcnn}{{{,,}, {,,,}, {,,,}, {,,}}}
  \denselyConnectNodes{fcnn}{{3, 4, 4, 3}}
\path (fcnn-1-1) -- (fcnn-2-1) node[midway, right=1ex] (W1) {$W_1$};
  \path (fcnn-2-1) -- (fcnn-3-1) node[midway, right=1ex] (W2) {$W_2$};
  \path (fcnn-3-1) -- (fcnn-4-1) node[midway, right=1ex] (V) {$V$};
% MADE net
  \begin{scope}[xshift=93mm]
    \drawNodes{made}{{{3,1,2}, {2,1,2,2}, {1,2,2,1}, {3,1,2}}}
    \connectSomeNodes{made}{{
          {{}, {1,2,3,4}, {1,3,4}},
          {{2,3}, {1,2,3,4}, {2,3}, {2,3}},
          {{1,3}, {1}, {1}, {1,3}},
        }}
  \end{scope}
% Input + output labels
  \foreach \idx in {1,2,3} {
      \node[below=0 of fcnn-1-\idx] {$x_\idx$};
      \node[above=0 of fcnn-4-\idx] {$\hat x_\idx$};
      \node[below=0 of made-1-\idx] {$x_\idx$};
    }
% MADE output labels
  \node[xshift=2.5ex, above=0 of made-4-1] {$p(x_3|x_2)$};
  \node[above=0 of made-4-2] {$p(x_2)$};
  \node[xshift=-4ex, above=0 of made-4-3] {$p(x_1|x_2,x_3)$};
% Bottom legend
  \node[legend, below=of fcnn-1-2] (encoder) {autoencoder};
  \node[legend, below=of made-1-2] (made) {MADE};
  \node[legend, right=27mm of encoder] (masks) {masks};
  \node[legend, yshift=-1pt] (masks) at ($(encoder)!0.55!(masks)$) {\texttimes};
  \node[legend, yshift=-1pt] (masks) at ($(masks)!0.65!(made)$) {$\longrightarrow$};
% Mask matrices
  \begin{scope}[shift={(35mm,49mm)}, scale=0.4]
    \draw (0,0) grid (4,3);
    \node at (-1.8,1.5) {$M_V =$};
    \fill[black] (0,1) rectangle ++(4,1);
    \fill[black] (1,0) rectangle ++(2,1);
\begin{scope}[yshift=-5cm]
  \draw (0,0) grid (4,4);
  \node at (-1.8,2) {$M_{W_2} =$};
  \fill[black] (0,0) rectangle ++(1,1);
  \fill[black] (0,3) rectangle ++(1,1);
  \fill[black] (2,0) rectangle ++(2,1);
  \fill[black] (2,3) rectangle ++(2,1);
\end{scope}

\begin{scope}[yshift=-10cm]
  \draw (0,0) grid (3,4);
  \node at (-1.8,2) {$M_{W_1} =$};
  \fill[black] (0,0) rectangle ++(1,4);
  \fill[black] (2,2) rectangle ++(1,1);
\end{scope}


\end{scope}
\end{tikzpicture}
\end{document}

AndréC · Answer 2 · 2020-08-15T08:54:42.987

I adapted the code from Jang Soo Kim's plane partition. By creating two commands, one called mask which creates a mask line by line from bottom to top.

\documentclass[]{article}
\usepackage{tikz}
\newcounter{x}
\newcounter{y}
\newcommand\square[3]{
  \fill[fill=#1, draw=black] (#2,#3) rectangle(#2+1,#3+1);
}
\newcommand\mask[1]{
 \setcounter{y}{-1}
  \foreach \a in {#1} {
        \addtocounter{y}{1}
        \setcounter{x}{-1}
        \foreach \b in \a {
            \addtocounter{x}{1}
            \ifnum \b>0
            \square{black}{\value{x}}{\value{y}}
             \else   
             \square{white}{\value{x}}{\value{y}}
             \fi
    }
  }
}
\begin{document}
\tikzset{x={(.5,0)},y={(0,.5)}}
    \begin{tikzpicture}
    \begin{scope}[local bounding box=figW1]
    \mask{{1,0,0},{1,0,0},{1,0,1},{1,0,0}}
    \node[anchor=west,font=\bf] at (figW1.east){$=M^{W^1}$};
    \end{scope}
\begin{scope}[yshift=3cm,local bounding box=figW2]
\mask{{1,0,1,1},{0,0,0,0},{0,0,0,0},{1,0,1,1}}
\node[anchor=west,font=\bf] at (figW2.east){$=M^{W^2}$};
\end{scope}
\begin{scope}[yshift=6cm,local bounding box=figV]
\mask{{0,1,1,0},{1,1,1,1},{0,0,0,0}}
\node[anchor=west,font=\bf] at (figV.east){$=M^{V}$};
\end{scope}
\end{tikzpicture}



\end{document}

TikZ image of Masked Autoencoder for Distribution Estimation (MADE)

2 Answers2