pgfplots: Reshape a MATRIX of data before conditional 3D surface plotting

Question

As far as I know, for plotting a surface using pgfplots, the data should be organized as follows:

x1 y1 z11
x1 y2 z12
x1 y3 z13
.  .   .
.  .   .
.  .   .
xn ym znm

However, usually, I get the data to be plotted in the following matrix shape

   y1  y2  y3   ...      ym
x1 z11 z12 z13  ...     z1m
x2 z21 z22 z23           .
x3 z31 z32 z33           .
.   .            .       .
.   .              .
.   .                .
xn zn1       ...        znm

So, is there any way to reshape or use the data as it is for plotting without having to manually organize it the way pgfplots expects?

---

This is an exemplary sample of data set to be conditionally surface plotted the same way done in the answer to my previous question.

     0.8    0.78    0.76    0.74    0.72
3.1 12.17   11.5    100     100     100
3.3 12.17   11.5    11.5    100     100
3.5 12.17   11.5    11.5    11.5    100
3.7 12.17   11.5    11.5    11.5    11.5
3.9 12.17   11.5    11.5    11.5    11.5
4.1 12.17   11.5    11.5    11.5    11.5

---

Edit 1

As suggested by Christian Feuersänger, I made a script by MATLAB as follows to reshape the data, but I don't have any idea how to make its corresponding lua syntax.

Data=[
12.17   11.5    100     100     100;
12.17   11.5    11.5    100     100;
12.17   11.5    11.5    11.5    100;
12.17   11.5    11.5    11.5    11.5;
12.17   11.5    11.5    11.5    11.5;
12.17   11.5    11.5    11.5    11.5;
];

xvalues=3.1:0.2:4.1;
yvalues=0.8:-0.02:0.72;

ReshapedData=zeros(3,numel(Data)); % x y z

fileID = fopen('file.txt','w');% file to save data

for nx=1:length(xvalues)
    for my=1:length(yvalues)
        colno=my+(nx-1)*length(yvalues);
        ReshapedData(1,colno)=xvalues(nx);
        ReshapedData(2,colno)=yvalues(my);
        ReshapedData(3,colno)=Data(nx,my);
        fprintf( fileID,'%.1f\t%.2f\t%.2f\r\n',ReshapedData(:,colno) );
    end
        fprintf( fileID,'\r\n');
end
fclose(fileID);

Answer 1

As pointed out in some comments, this is possible with lua.

My script assumes data ordered as follows, whitespace-delimited only. This is basically as asked in the question. The indentation in the first line is optional.

   y1  y2  
x1 z11 z12
x2 z21 z22 
x3 z31 z32

The script reads the data, stores it in tables and writes the tables in an output file such that the data is reformated in the desired way.

x1 y1 z11 
x1 y2 z12 
x2 y1 z21 
x2 y2 z22 
x3 y1 z31 
x3 y2 z32 

A minimal example code, where the input file with your example data and the lua script are included and written into files with the help of filecontents, reads as follows.

\documentclass{article}
\usepackage{pgfplots,filecontents,luacode}

\begin{filecontents*}{mat-in.txt}
     0.8    0.78    0.76    0.74    0.72
3.1 12.17   11.5    100     100     100
3.3 12.17   11.5    11.5    100     100
3.5 12.17   11.5    11.5    11.5    100
3.7 12.17   11.5    11.5    11.5    11.5
3.9 12.17   11.5    11.5    11.5    11.5
4.1 12.17   11.5    11.5    11.5    11.5
\end{filecontents*}

\begin{filecontents}{mat-rewrite.lua}
\begin{luacode}

function rewrite_mat(infilename, outfilename)
  infile  = io.open( infilename, "r")
  outfile = io.open(outfilename, "w")

  ynum = 0
  xnum = -1
  yvals={}
  xvals={}
  zvals={}

  for line in infile:lines() do
    t={}                               -- make a table from the line (whitespace-delimited)
    length=1
    for v in string.gmatch(line, "%S+") do
      t[length]=v
      length = length + 1
    end 

    xnum = xnum + 1                    -- count number of x values 

    if xnum < 1 then                   -- first line: count number of y values
      ynum  = length -1
      yvals = t
    else
      xvals[xnum]=t[1]                 -- all other lines: store x and z values
      for y=1,ynum do
        index = ynum*(xnum-1)+y
        zvals[index]=t[y+1]
      end
    end
  end

  infile:close()

  for x = 1,xnum do
    for y = 1,ynum do
      outfile:write(xvals[x] .. " " .. yvals[y] .. " " .. zvals[(x-1)*ynum+y] .. " " .. "\string\n")
    end
  end

  outfile:close()

end  
\end{luacode}
\end{filecontents}

% load function
\input{mat-rewrite.lua}

% run function mat-in.txt for structure, mat-in2 for matrix with values
\directlua{rewrite_mat("mat-in.txt","out.txt")}


\begin{document}
\begin{tikzpicture}
\begin{axis}[mesh/ordering=y varies, unbounded coords=jump,]
\addplot3[surf,mesh/rows=5] 
  table[
        x index =0,
        y index =1,
        % use an expression to state that z values of 100 should be ignored
        z expr={\thisrowno{2} == 100 ? NaN : \thisrowno{2}},
       ] {out.txt};
\end{axis}
\end{tikzpicture}
\end{document}

I have reused the code from the previous question/answer to drop the z=100 values. The output plot of this example looks not very spectacular... But it demonstrates the conversion from the matrix data to a vector and subsequent plotting.