Hessian of a loss function for a neural netwrok

Asked Jul 02 '19 at 21:08

Active Jul 03 '19 at 22:22

Viewed 256 times

Is a way to calculate the Hessian of the loss function in Mathematica. It's easy to define a loss function for neural nets but I couldn't figure out how to calculate the Hessian(more precisely I wanted to get access to eigenspectrum of Hessian). There are scientific papers, where they calculate Hessian of a loss using python(autograd (auto differentiation) algorithm).

The idea here to train a neural net for few epochs (say on a simple fully connected net or simplified LeNet) and then calculate the Hessian of the loss function.

Addendum: Here's one such paper: https://arxiv.org/pdf/1611.07476.pdf

edited Jul 03 '19 at 17:40

asked Jul 02 '19 at 21:08

sra

Could you share DOI of the paper you mentioned for reference? – Xminer Jul 02 '19 at 21:52
1

@Xminer Paper link added. Thanks for the suggestion. – sra Jul 02 '19 at 22:38
1

AFAIK Wolfram's NN framework works on top of MXNet library, so until it fully supports 2nd order differentiation it's not possible. Relevant issue. – swish Jul 03 '19 at 15:40
@swish This is quite frustrating then. Mathematica for theoretical ML research seems quite restricting. – sra Jul 03 '19 at 18:15
@psimeson because we can get 1st gradient,How about an approximation of Hessian? – Xminer Jul 03 '19 at 20:05
How can you access the loss function (say after training for about 10 training rounds) in such a way that you can differentiate it? – sra Jul 03 '19 at 20:22
1

@psimeson connect the net and loss function,train it with NetTrain and then get 1st Gradient with NetGradient[{position of the layer,"Weights"}] – Xminer Jul 03 '19 at 21:21
Hessian support is awkward across all major frameworks, I've made a wrapper library to make it easier to compute in PyTorch -- https://github.com/cybertronai/autograd-lib – Yaroslav Bulatov Oct 05 '21 at 21:12

Hessian of a loss function for a neural netwrok

0 Answers0

Linked