Example of NetChain running in MXNet cause Error

Question

Cross post here

Problem remains in Mathematica 11.2

First defining the net

SeedRandom[1234];
net = NetInitialize@NetChain[{5, 3}, "Input" -> 2];
{net[{1, 2}], net[{0, 0}], net[{0.1, 0.3}]}

{{1.27735, -1.21455, -1.02647}, {0., 0., 0.}, {0.141054, -0.145882, -0.099945}}

Exporting it to net of MXNet.

Export["simple model-symbol.json", net, "MXNet"]

The JSON file(It defines the net structure) is

From the code, we can see the name of the parameter is the same between params and JSON file.

import mxnet as mx
mx.nd.load('simple model-0000.params')

It gives an error ValueError: need more than 1 value to unpack when importing the net.

import mxnet as mx
sym, arg_params, aux_params = mx.model.load_checkpoint('simple model', 0)

Complete code to make prediction.(data_names and label_names can be shown in JSON file.)

import mxnet as mx
import numpy as np
sym, arg_params, aux_params = mx.model.load_checkpoint('simple model', 0)
mod = mx.mod.Module(symbol=sym,data_names=['Input'],label_names=['2'])
mod.bind(for_training=False, data_shapes=[('Input', (1,2))])
mod.set_params(arg_params, aux_params)

input_data=np.array([[1,2]])
array = mx.nd.array(input_data)

from collections import namedtuple
Batch = namedtuple('Batch', ['Input'])
mod.forward(Batch([array]))

prob = mod.get_outputs()[0].asnumpy()
prob = np.squeeze(prob)
print prob

Version:

Mathematica "11.2.0 for Microsoft Windows (64-bit) (September 11, 2017)"

MXNet 0.11.1 (the latest version 20170905_mxnet_x64_vc14_cpu.7z)

How to fix it?

PS: Thank you @Sebastian, but still have a problem when using RNN.

SeedRandom[1234];
net = NetInitialize@NetChain[{10, Ramp, ReshapeLayer[{5, 2}], 
                    LongShortTermMemoryLayer[5], 3}, "Input" -> 2]
net[{1, 2}]
Export["example.json", net, "MXNet"];

In MXNet, when executing this code e = sym.bind(mx.cpu(), nd), it gives the following errors.

raise ValueError('key %s is missing in %s' % (name, arg_key))

ValueError: key 4.State is missing in args

Answer 1

I will make a complete example of making predictions with exported nets in MXNet (Python binding). First,

SeedRandom[1234]; 
net = NetChain[{LinearLayer[5], LinearLayer[3]}, "Input" -> 2]; 
net = NetInitialize[net]; 
net[{1, 2}]

{1.27735, -1.21455, -1.02647}

Exporting:

Export["example.json", net, "MXNet"]

creates two files, "example.json" and "example.params". Now lets run this in MXNet. The first thing to note: this is designed to be used by the lowest-level Symbol and NDArray API, NOT higher-level APIs like Gluon or Model. These higher-level APIs are often Python specific: using the lowest level means that this will work on every language binding MXNet supports.

The first step is to import the symbol (stored as a json file) and NDArray:

import mxnet as mx
import numpy as np
sym = mx.symbol.load('example.json') 
nd = mx.nd.load('example.params')

If you tried to bind now via e = sym.bind(mx.cpu(), nd), it will fail, as you have not specified an input NDArray, which will tell MXNet what shape the executor will be. Rather, see what the different inputs are to the symbol:

sym.list_arguments()

['Input', '1.Weights', '1.Biases', '2.Weights', '2.Biases']

In MXNet, weights and biases are normal input arguments. Looking at the keys in our NDArray list:

list(nd)

['2.Weights', '2.Biases', '1.Biases', '1.Weights']

we see it does not contain an array for "Input". Lets add it:

inputND = mx.nd.array(np.array([[1,2]]));
nd["Input"] = inputND

Why did I use [[1,2]] and not [1, 2]? The answer is that MXNet always expects a batch dimension. So this example is using a batch size of 1. Now bind and do a forward pass:

e = sym.bind(mx.cpu(), nd)
out = e.forward()

This returns a list of NDArrays:

[
[[ 1.27735424 -1.21454573 -1.02646875]]
<NDArray 1x3 @cpu(0)>]

Which can be converted to a numpy array via out[0].asnumpy(). Further predictions can be made by directly setting the "Input" NDArray:

inputND[:] = np.array([[3,4]])
a = e.forward()

returning:

[
 [[ 3.56568646 -3.15509319 -3.13344383]]
 <NDArray 1x3 @cpu(0)>]

Answer 2

I have the right way to do that, after hours of work! The right way, is to modify the exported file from Mathematica to have them compliant with the standard MXNet.

The params file obtained by Mathematica is wrong as the other user said. But the problem is due to a wrong in the keys of the dictionary.

If you try to create a simple network in

import mxnet as mx
import numpy as np
from collections import namedtuple

#Create a graph
invar = mx.symbol.Variable('data')
net = mx.symbol.FullyConnected(data = invar, name = 'fc1', num_hidden = 2)

#Generate a module for the inference
mod = mx.mod.Module(net)

#Choose dimentions of the network
mod.bind(data_shapes=[('data',(1,5))])

#Weights initialization
mod.init_params()

#Define the input
Batch = namedtuple('Batch', ['data'])
invect = [mx.nd.ones((1,5))]

#Try forward function
mod.forward(Batch(invect))

#Export the network
mod.save_checkpoint('export',0)

#Load the network
sym = mx.symbol.load('export-symbol.json')
params = mx.nd.load('export-0000.params')

#Show the structure
sym.list_arguments()
# ['data', 'fc1_weight', 'fc1_bias']

Using this code is clear that dictionary have this structures:

{'arg:fc1_weight': .....,'arg:.... 

Instead the file created by Mathematica misses of the string "arg" at the front of each key of the dictionary. In the end the right way to export the netwotk in Mathematica is as follows:

(*Change the export directory*)
SetDirectory[NotebookDirectory[]];
(*Creation of a simple graph*)
data = Flatten@
   Table[{x, y} -> {x*y}, {x, -1, 1, .005}, {y, -1, 1, .005}];
net = NetChain[{4, Tanh, 1}, "Input" -> 2];
net = NetTrain[net, data, BatchSize -> 1024]
(*Loading the needed packages*)
<< MXNetLink`;
<< NeuralNetworks`;
<< GeneralUtilities`;
(*The network name*)
networkStr = { "MathematicaNet"};
(*Is the number of training epochs, it is possible to load the model \
at different training epochs number*)
numberOfEpochs = 0;

(*THE FIX*)
FixMXNnet[str_] := StringJoin[{"arg:", str}]

(*Export ".json" file*)

Export[StringJoin[{networkStr[[1]], "-symbol.json"}], 
 ToMXJSON[net][[1]], "String"]

(*Export ".params" file*)

plan = ToNetPlan[net];
NDArrayExport[
 StringJoin[{networkStr[[1]], "-", 
   StringPadLeft[ToString[numberOfEpochs], 4, "0"], ".params"}], 
 NDArrayCreate /@ KeyMap[FixMXNnet, plan["WeightArrays"]]]

With this fix the files exported by Mathematica are compliant with the standard. In Python you can load the network using:

import mxnet as mx
mx.model.load_checkpoint('MathematicaNet',0)

Answer 3

After some try and error.

I found there are at least two methods to load a mathematica-exported mxnet-model to predict and get the result.

I think some is bug of MXNet and some is bug of Mathematica.

One method is

_sym= mx.symbol.load(file_sym)
_nd= mx.nd.load(file_nd)

Second method is

   mod = mx.mod.Module(symbol=sym, context=ctx,data_names=['Input'],label_names=None)

1.

In your LSTM example I'm successful in method1

if __name__=='__main__':
ctx=mx.cpu()
from myPath import *
file_sym=os.path.join(root_dir, 'model-wolfram/wolfram.lstm-symbol.json')
file_nd=os.path.join(root_dir, 'model-wolfram/wolfram.lstm-0000.params')

_sym= mx.symbol.load(file_sym)
_nd= mx.nd.load(file_nd)

_nd['4.State']=mx.nd.array([[0,0,0,0,0]])
_nd['4.CellState']=mx.nd.array([[0,0,0,0,0]])

input_data=np.array([[1,2]])

array = mx.nd.array(input_data)

_nd["Input"] = array
_e = _sym.bind(ctx, _nd)

_out = _e.forward()
prob = _out[0].asnumpy()
prob = np.squeeze(prob)
print 'prob', prob

2. Question about load_checkpoint problem

`It gives an error ValueError: need more than 1 value to unpack when importing the net.

import mxnet as mx sym, arg_params, aux_params = mx.model.load_checkpoint('simple model', 0)`

I've posted an issue,

you can modify the 'model.py' file

symbol = sym.load('%s-symbol.json' % prefix)
save_dict = nd.load('%s-%04d.params' % (prefix, epoch))
arg_params = {}
aux_params = {}
for k, v in save_dict.items():
   if ':' in k: 
      tp,name = k.split(':', 1)
            if tp == 'arg':
                arg_params[name] = v
            if tp == 'aux':
                aux_params[name] = v
      else:
          arg_params[k]=v

Now some simple models generated by Mathematica can be imported into MXnet and get some useful result, but if aux_states are too many, preprocess in too complex, we should solve many problems. And I've found many other cases, I'll organize later.