Python: Underflow vs. exp of large negative numbers

Question

Suppose I have values of log(P(x_i)), i.e. log-probabilities to events x_i. The probabilities are very small, so that these log-values are of the order of -1e3.

I want to compute an expectation value. In order to evaluate the corresponding sum, I need the probabilities P(x_i) themselves so I tried to call numpy.exp. This however, returns only 0 as the precision of basic floats in Python is not high enough to resolve a number like exp(-1000).

What is the typical way to circumvent this issue?

Federico Poloni · Accepted Answer · 2022-02-16T08:56:30.413

8

If your final result is of the order of magnitude of exp(-1000) $\approx 5 \cdot 10^{-435}$, then you are out of luck; no matter how you compute it, it will always underflow. There is simply no representable floating-point binary64 number that small. The slower Bigfloats are the only way out.

Some of the issues with large/small intermediate values, however, can be solved with a few manipulations:

gathering and simplifying certain constants (the so-called log-sum-exp trick).
skillful use of the library functions log1p and expm1.

edited Feb 16 '22 at 08:56

answered Feb 07 '22 at 22:09

Federico Poloni

11,344
1
31
59

Thank you for the hints! I will try to set up Bigfloat then. – reloh100 Feb 08 '22 at 22:57
1

Have you tried the log-sum-exp trick before abandoning the idea? It seems tailored to what you need. – Federico Poloni Feb 09 '22 at 07:31

Python: Underflow vs. exp of large negative numbers

1 Answers1