How can I add noise to a neural network input?

Question

A common "trick" in neural network training is to train an autoencoder using a "degraded" sample as input and the original sample as target output. The layer type DropoutLayer[] is made for this purpose. But I would like to play with different types of noise, e.g. Gaussian noise, salt&pepper noise, to train a denoising autoencoder.

I can think of a few ways to implement this, but I'm not happy with any of them:

I can add noise offline and pass the noisy and original samples to NetTrain as input and target, respectively. But then exactly the same noise trained in every round, which probably leads to overfitting.
I can pass noisy samples to NetTrain as above, but only train a few rounds, then restart with different noisy samples. (i.e. call NetTrain in a loop). But that seems to break a lot of the internal learning-rate heuristics in NetTrain (e.g. the momentum term is reset in every cycle)
I can link together a constant input, a several dropout layers and a TotalLayer to get a poor man's Gaussian noise approximation, (each DropoutLayer produces a Bernoulli random variable per pixel, and the sum of a number of Bernoulli random variables is approximately Gaussian). This is my best idea so far, but I need a lot of DropoutLayers to get close to a continuous distribution, and it's not very efficient to include an input that's always constant

May be you can use "$AugmentationFunction" -> f for image NetEncoder... you can check the answers from this link as an idea but I don't think it's exactly what you need... http://mathematica.stackexchange.com/questions/126120/training-data-generator — s.s.o, Dec 08 '16 at 21:17
@s.s.o: If this is called repeatedly for every round, it would be perfect, but I'll have to come up with some way to test this... Thanks for the link, that question is very similar and it's good to hear this will be addressed in 11.1 — Niki Estner, Dec 09 '16 at 07:28
If I understand correctly you want every round different noise... In fact NN can accept file names (in order to keep the original images) you can add the argumentation function to this file name get random noised images and feed it to NN. Similar to Alexey Golyshev multi round training with Do[ ... {numberOfRounds}] you can do multi stage training. Not best solution but may help. — s.s.o, Dec 09 '16 at 12:04
Has anyone figured out the most efficient way to do this? I tried using the new generator function in 11.1, which allows NetTrain to create each batch of training data according to any given function. However, this slows down the training by an order of magnitude. — dan7geo, Apr 03 '17 at 10:17

score 11 · Answer 1 · answered Dec 08 '16 at 18:39

11

For what it's worth, this is my "add binomial random variate layer":

Needs["NeuralNetworks`"] (* for ScalarTimesLayer, ScalarPlusLayer *)
With[{σ = 0.1, n = 5},
 addNoiseLayer = NetGraph[
   Flatten@{
     TotalLayer[], ScalarTimesLayer[σ*1./5], 
     ScalarPlusLayer[-1*σ], TotalLayer[],
     Table[DropoutLayer[], {n}]
     },
   Flatten@{
     1 -> 2 -> 3 -> 4,
     NetPort["Input"] -> 4,

     Table[NetPort["const"] -> 4 + i -> 1, {i, n}]
     }, "Input" -> inputDim, "const" -> inputDim]]

answered Dec 08 '16 at 18:39

Niki Estner

36,101
3
92
152

mma says "ScalarTimesLayer[0.02] is not a net function.",and environment is ""11.1.0 for Microsoft Windows (64-bit) (March 13, 2017)"" – partida Aug 31 '17 at 03:02
@partida: not tested, but I think you can use ElementwiseLayer[#*0.02 &] in version 11 – Niki Estner Aug 31 '17 at 06:34
yes,they do the same thing,thanks – partida Aug 31 '17 at 07:20

score 7 · Accepted Answer · answered Sep 11 '18 at 01:43

7

Use NeuralNetworks`NoiseLayer

<<NeuralNetworks`
noisy = NetGraph[{
    "noise" -> NoiseLayer[{3, 90, 160}, "Distribution" -> NormalDistribution[0, 0.1]],
    "add" -> ThreadingLayer[Plus]
},
    {{"noise", NetPort["Input"]} -> "add"},
    "Input" -> NetEncoder[{"Image", {160, 90}}],
    "Output" -> NetDecoder["Image"]
]

You can find more undocumented definitions with

NeuralNetworks`$LayerData["Noise"]//Dataset

answered Sep 11 '18 at 01:43

Aster

3,836
1
18
44

Hi, it seems like NoiseLayer is outdated. Do you know how to add the random noise for Mathematica v 11.3? – Maria Sargsyan Jul 14 '20 at 09:55

score 5 · Answer 3 · answered Jul 14 '20 at 10:51

In Mathematica 12.1, DropoutLayer exposes his mask, so it's possible to train using a random binary mask, with DropoutLayer["OutputPorts" -> "BinaryMask"] (or DropoutLayer["OutputPorts" -> {"Output", "BinaryMask"}])

In Mathematica 12.2, it will be possible to simulate many types of noises (including once where you give the parameters -like mean and variance- through outputs of other subparts of a NetGraph). The novelty is already documented in the current 12.2 builds under the name of RandomArrayLayer.

score 2 · Answer 4 · answered Jul 14 '20 at 11:13

You can also do this by using a generator function for generating network inputs during NetTrain. Here's a minimal (if somewhat nonsensical) example:

net = NetInitialize @ NetGraph[
   <|
    "chain" -> NetChain[{5, Ramp, 1}],
    "plus" -> ThreadingLayer[Plus]
    |>,
   {
    {NetPort["Input"], NetPort["Noise"]} -> "plus" -> "chain"
    },
   "Input" -> {1}
];
data = List /@ N[Range[10]];
genFun = Function[
   With[{dat = RandomChoice[data, #BatchSize]},
    <|
     "Input" -> dat,
     "Output" -> dat,
     "Noise" -> RandomVariate[NormalDistribution[], {#BatchSize, 1}]
     |>
    ]
];

Test the generator:

genFun[<|"BatchSize" -> 5|>]

<|"Input" -> {{9.}, {3.}, {10.}, {5.}, {5.}}, "Output" -> {{9.}, {3.}, {10.}, {5.}, {5.}}, "Noise" -> {{0.605894}, {-1.57657}, {0.586584}, {-1.8153}, {0.443682}}|>

Train using the generator:

NetTrain[net, genFun]

Each round, genFun will be evaluated to create a new batch of training examples, so each round you'll get fresh random numbers.

How can I add noise to a neural network input?

4 Answers4