If I understand correctly, I think your problem comes from the fact that a \matrix is just a special type of \node, and like all nodes it has a (by default non-zero) inner sep. You can see that if you add draw to the options of mat11, so that you have
\matrix[mymat, draw, anchor=west,row 2/.style={nodes={draw,fill=gray!30}}]
at (1.65,-5.5)
(mat11)
{
& & & \\
t_1 & t_2 & t_3 & t_4\\
};
which gives this:

So the lines you've drawn goes to the corners of the outer border, not the corners of the nodes.
I can think of two ways around this:
Draw the lines to mat11-2-1.north west and mat11-2-4.north east instead of mat11.north west/mat11.north east. This way the lines point to the nodes inside the matrix, instead of the matrix itself.
\draw[green, dashed](node1.south west) -- (mat11-2-1.north west);
\draw[green, dashed](node1.south east) -- (mat11-2-4.north east);

Set the inner sep of mat11 to 0, but then you have to set the inner sep of the nodes back to 0.333em, which is the default value:
\matrix[mymat, inner sep=0, anchor=west,row 2/.style={nodes={draw,fill=gray!30,inner sep=0.333em}}]
at (1.65,-5.5)
(mat11)
{
& & & \\
t_1 & t_2 & t_3 & t_4\\
};

Complete code with both methods:
\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{matrix,fit,positioning,arrows}
\tikzset{
mymat/.style={
matrix of math nodes,
text height=2.5ex,
text depth=0.75ex,
text width=3.25ex,
align=center,
column sep=-\pgflinewidth,
},
mymats/.style={
mymat,
nodes={draw,fill=#1}
}
}
\begin{document}
\begin{tikzpicture}[>=latex]
\matrix[mymat,anchor=west,row 2/.style={nodes=draw}]
at (0,0)
(mat1)
{
& \\
N1 \\
};
\matrix[mymat,right=of mat1,row 2/.style={nodes={draw,fill=gray!30}}]
(mat2)
{
& \\
t_2 & t_4 \\
};
\matrix[mymat,anchor=west,row 2/.style={nodes=draw}]
at (0,-1)
(mat3)
{
& \\
N2 \\
};
\matrix[mymat,right=of mat3,row 2/.style={nodes={draw,fill=gray!30}}]
(mat4)
{
& & \\
t_1 & t_3 & t_4 \\
};
\matrix[mymat,anchor=west,row 2/.style={nodes=draw}]
at (0,-2)
(mat5)
{
& \\
N3 \\
};
\matrix[mymat,right=of mat5,row 2/.style={nodes={draw,fill=gray!30}}]
(mat6)
{
& &\\
t_1 & t_2 & t_3 \\
};
\matrix[mymat,anchor=west,row 2/.style={nodes=draw}]
at (0,-3)
(mat7)
{
& \\
N4 \\
};
\matrix[mymat,right=of mat7,row 2/.style={nodes={draw,fill=gray!30}}]
(mat8)
{
& \\
t_3 & t_4 \\
};
\matrix[mymat,anchor=west,row 2/.style={nodes=draw}]
at (0,-4)
(mat9)
{
& \\
N5 \\
};
\matrix[mymat,right=of mat9,row 2/.style={nodes={draw,fill=gray!30}}]
(mat10)
{
& \\
t_1 & t_3 \\
};
\matrix[mymat, inner sep=0, anchor=west,row 2/.style={nodes={draw,fill=gray!30,inner sep=0.333em}}]
at (1.65,-5.5)
(mat11)
{
& & & \\
t_1 & t_2 & t_3 & t_4\\
};
\node[ draw, red, dashed, line width=0.7pt, fit=(mat2)(mat4)(mat6)(mat8)
(mat10)](node1){};
\begin{scope}[shorten <= -2pt]
\draw[*->]
(mat1-2-1.east) -- (mat2-2-1.west);
\draw[*->]
(mat3-2-1.east) -- (mat4-2-1.west);
\draw[*->]
(mat5-2-1.east) -- (mat6-2-1.west);
\draw[*->]
(mat7-2-1.east) -- (mat8-2-1.west);
\draw[*->]
(mat9-2-1.east) -- (mat10-2-1.west);
\end{scope}
\draw[green, dashed](node1.south west) -- (mat11.north west);
\draw[green, dashed](node1.south east) -- (mat11.north east);
\end{tikzpicture}
\bigskip
\begin{tikzpicture}[>=latex]
\matrix[mymat,anchor=west,row 2/.style={nodes=draw}]
at (0,0)
(mat1)
{
& \\
N1 \\
};
\matrix[mymat,right=of mat1,row 2/.style={nodes={draw,fill=gray!30}}]
(mat2)
{
& \\
t_2 & t_4 \\
};
\matrix[mymat,anchor=west,row 2/.style={nodes=draw}]
at (0,-1)
(mat3)
{
& \\
N2 \\
};
\matrix[mymat,right=of mat3,row 2/.style={nodes={draw,fill=gray!30}}]
(mat4)
{
& & \\
t_1 & t_3 & t_4 \\
};
\matrix[mymat,anchor=west,row 2/.style={nodes=draw}]
at (0,-2)
(mat5)
{
& \\
N3 \\
};
\matrix[mymat,right=of mat5,row 2/.style={nodes={draw,fill=gray!30}}]
(mat6)
{
& &\\
t_1 & t_2 & t_3 \\
};
\matrix[mymat,anchor=west,row 2/.style={nodes=draw}]
at (0,-3)
(mat7)
{
& \\
N4 \\
};
\matrix[mymat,right=of mat7,row 2/.style={nodes={draw,fill=gray!30}}]
(mat8)
{
& \\
t_3 & t_4 \\
};
\matrix[mymat,anchor=west,row 2/.style={nodes=draw}]
at (0,-4)
(mat9)
{
& \\
N5 \\
};
\matrix[mymat,right=of mat9,row 2/.style={nodes={draw,fill=gray!30}}]
(mat10)
{
& \\
t_1 & t_3 \\
};
\matrix[mymat, anchor=west,row 2/.style={nodes={draw,fill=gray!30}}]
at (1.65,-5.5)
(mat11)
{
& & & \\
t_1 & t_2 & t_3 & t_4\\
};
\node[ draw, red, dashed, line width=0.7pt, fit=(mat2)(mat4)(mat6)(mat8)
(mat10)](node1){};
\begin{scope}[shorten <= -2pt]
\draw[*->]
(mat1-2-1.east) -- (mat2-2-1.west);
\draw[*->]
(mat3-2-1.east) -- (mat4-2-1.west);
\draw[*->]
(mat5-2-1.east) -- (mat6-2-1.west);
\draw[*->]
(mat7-2-1.east) -- (mat8-2-1.west);
\draw[*->]
(mat9-2-1.east) -- (mat10-2-1.west);
\end{scope}
\draw[green, dashed](node1.south west) -- (mat11-2-1.north west);
\draw[green, dashed](node1.south east) -- (mat11-2-4.north east);
\end{tikzpicture}
\end{document}
Addendum
As mentioned in a comment, I don't think it's necessary to use a \matrix for the N nodes on the left. I don't understand the purpose of the empty first row in your matrices either, seems you're adding a lot of unnecessary code. Here is one alternative method for making your diagram. I make use of the chains library to create the N nodes on the left (could also have used a \matrix, actually). Everything is placed relative to other stuff, so no explicit coordinates are used. To draw the arrows, I used a loop.
I used the arrows.meta library instead of arrows, as TikZ considers the latter to be deprecated in favor of the former (see the manual for version 3.0.1a page 512).
\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{matrix,fit,positioning,chains,arrows.meta}
\tikzset{
nodestyle/.style={
text height=2.5ex,
text depth=0.75ex,
text width=3.25ex,
align=center,
draw
},
mymat/.style={
matrix of math nodes,
nodes={
nodestyle,
fill=gray!30,
},
column sep=-\pgflinewidth,
}
}
\begin{document}
\begin{tikzpicture}[node distance=1.2cm]
\begin{scope}[
start chain=N going below,
every node/.style={
nodestyle,
on chain
},
node distance=2mm
]
\node {$N1$};
\node {$N2$};
\node {$N3$};
\node {$N4$};
\node {$N5$};
\end{scope}
\matrix [mymat,right=of N-1, name=m1] {t_2 & t_4 \\ };
\matrix [mymat,right=of N-2, name=m2] {t_1 & t_3 & t_4 \\ };
\matrix [mymat,right=of N-3, name=m3] {t_1 & t_2 & t_3 \\ };
\matrix [mymat,right=of N-4, name=m4] {t_3 & t_4 \\ };
\matrix [mymat,right=of N-5, name=m5] {t_1 & t_3 \\ };
\node [draw, red, dashed, fit=(m1)(m2)(m3)(m4)(m5), name=M] {};
\foreach \i in {1,...,5}
\draw [Circle-Latex,shorten <=-2pt] (N-\i) -- (m\i-1-1);
\matrix [mymat,below=5mm of M, name=m11] {t_1 & t_2 & t_3 & t_4 \\ };
\draw[green, dashed](M.south west) -- (m11-1-1.north west);
\draw[green, dashed](M.south east) -- (m11-1-4.north east);
\end{tikzpicture}
\end{document}
fitandmatrix, as well as a\documentclass. And of course, it's not necessary to loadtikztwice. You're actually loading it three times, aspgfplotsloadstikzas well. – Torbjørn T. May 02 '18 at 10:19