12

I want to try Advanced Activations Layers in Mathematica,but not found.

So I try to use ElementwiseLayer to implement it.

Thank you @nikie,LeakyReLU,ELU,ThresholdedReLU can be written like this.

LeakyReLU : ElementwiseLayer[Ramp[#] - Ramp[-#]*0.3 &]

ELU : ElementwiseLayer[Ramp[#] - Ramp[-#]/#*(Exp[#] - 1) &]

ThresholdedReLU : ElementwiseLayer[Ramp[# - 1]/(# - 1)*Ramp[#] &]

PReLU has a learned parameter alpha，but I don't know how to train the net ...

graph = NetGraph[{ConstantArrayLayer["Array" -> ConstantArray[0.3, 5]], ThreadingLayer[Ramp[#] - Ramp[-#]*#2 &]}, {{NetPort["Input"], 1} -> 2}]
graph[{-1, -0.5, 0, 0.5, 1}](*{-0.3, -0.15, 0., 0.5, 1.}*)

enter image description here

Is there any more simple method to make Advanced Activations Layers?

Application: this post used leayReLU[alpha_] := ElementwiseLayer[Ramp[#] - alpha*Ramp[-#] &]

partida
  • 6,816
  • 22
  • 48
  • Setting Attribute to Listable seems to be the problem. You can try with Function directly: g = Function[x, Piecewise[{{0.3*x, x < 0}, {x, x > 0}}], Listable] – Anjan Kumar May 22 '17 at 04:53
  • @AnjanKumar Thank you,I edit my question. – partida May 22 '17 at 05:12
  • Regarding the learned parameter of PReLU, I think you need to use ConstantArrayLayer for learned constants, and (I guess) ThreadingLayer to combine input from the constant and the "data" input layers – Niki Estner May 22 '17 at 07:09

3 Answers3

8

The documentation of ElementwiseLayer explicitly lists which functions are allowed, and UnitStep is not in this list, I believe that's why the function fails.

Simple workaround: Use a combination of functions from that list, like Ramp:

f = Ramp[#] - Ramp[-#]*0.3 &;

l = ElementwiseLayer[f]
l[{-1, -0.5, 0, 0.5, 1}]

{-0.3, -0.15, 0., 0.5, 1.}

Niki Estner
  • 36,101
  • 3
  • 92
  • 152
4

This is how to constract a PReLU

data = Thread[RandomReal[1, {100, 2}] -> RandomReal[1, {100, 3}]];
net = NetGraph[{5, ConstantArrayLayer["Output" -> 1], 
         ReplicateLayer[5], FlattenLayer[], 
         ThreadingLayer[Ramp[#1] - #2*Ramp[-#1] &], 3}, 
         {NetPort["Input"] -> 1 -> 5 -> 6, 2 -> 3 -> 4 -> 5}]

enter image description here

NetTrain[net, data, MaxTrainingRounds -> 100, BatchSize -> 32];
(*well done*)

In 11.2,mma provide more Advanced Activations Layers

enter image description here

partida
  • 6,816
  • 22
  • 48
2

In 2020, the function ParametricRampLayer was introduced to implement leaky ReLU layers. The slope can be either hard-coded or learned.

Michael Poulshock
  • 156
  • 4