ElementwiseLayer with Piecewise?

Question

I need a specific custom activation function; how can I implement it with Piecewise?

This is what I've tried (using the recommendation from this question):

p = Function[x, Piecewise[{{Exp[x] - (2 + E)/(2 E), x < -1}, {x/2, x < 0}, {x, 
 x > 0}}], Listable]
ElementwiseLayer[p]

Is there a systematic workaround to handle the conversion from an aribitrary Piecewise function into an ElementwiseLayer?

Possible duplicate of How to make Advanced Activations Layers in Mathematica? — Niki Estner, Mar 19 '18 at 19:51
@NikiEstner that question is similar but doesn't actually address the general case of any Piecewise condition. — user5601, Mar 19 '18 at 20:28

score 3 · Accepted Answer · answered Mar 19 '18 at 20:07

Let me give a general method for re-expressing a Piecewise[] expression. This hinges on two things. First:

FullSimplify[1 - Sign[Ramp[-x]] == UnitStep[x], x ∈ Reals]
   True

The second part is that in principle, any piecewise expression can be re-expressed in terms of UnitStep[]; in particular, there is the undocumented function Simplify`PWToUnitStep[] (see here) for performing the conversion.

With these two considerations (and using a less trivial example):

Simplify`PWToUnitStep[Piecewise[{{2 - #, # <= 0}, {# + 1, # > 0}}] &[x]] /.
UnitStep -> (1 - Sign[Ramp[-#]] &)
   (2 - x) (1 - Sign[Ramp[x]]) + (1 + x) Sign[Ramp[x]]

and now one can do ElementwiseLayer[Function[x, (2 - x) (1 - Sign[Ramp[x]]) + (1 + x) Sign[Ramp[x]]]].

thanks for a simple generalizable explanation! – user5601 Mar 20 '18 at 14:55 — user5601, Mar 20 '18 at 14:55

score 0 · Answer 2 · edited Jun 16 '20 at 09:23

From the documentation of ElementwiseLayer:

The function f can be any one of the following: Ramp, LogisticSigmoid, Tanh, ArcTan, ArcTanh, Sin, Sinh, ArcSin, ArcSinh, Cos, Cosh, ArcCos, ArcCosh, Log, Exp, Sqrt, Abs, Gamma, LogGamma.

In general, f can be any object that when applied to a single argument gives any combination of Ramp, LogisticSigmoid, etc., together with Plus, Subtract, Times, Divide, Power, Min, Max, Clip, and numbers.

So you'll need to use something like ElementwiseLayer[Ramp[#] - Ramp[-#] &]

ElementwiseLayer with Piecewise?

2 Answers2