0

Say I have a set of numerous probabilities given by their logarithm : $\{\ln p_i, 1 \leq i \leq N\}$.

I want to compute $\sum p_i$, if possible without exponentiating $\ln p_i$, since some of those probabilities are really small and I would suffer a dramatic loss of precision by doing so.

Do you know of any clever trick ?

Edit 14/06

  • I compute the probabilities along a probability tree whose depth can go s high as $D = 10 000$. This tree is typically sparse, but I don't have a better upper bound than $N = 2^D$

  • Some of these probabilities get very low (e.g. $10^{-200}$). I work in python. I haven't witnessed firsthand a precision loss, but I suspected it might happen and decided to reach out to more kowledgeable people.

  • My current implementation keeps only the 1000 highest log probabilities, exponentiates and sums them.

  • Computing the log of the sum is perfect for my application, since I have a natural baseline for the log probs. I'd gladly mark this as an approved answer.

Alex
  • 1
  • 1
  • I don't think there should be loss of precision in this computation. Do you mean you have trouble with underflow (probabilities smaller than $10^{-308}$ that get approximated to 0 in IEEE)? Otherwise, can you provide an example? – Federico Poloni Jun 12 '20 at 14:31
  • How many terms do you want to add up? – Wolfgang Bangerth Jun 12 '20 at 15:01
  • 2
    If you are satisfied with computing the log of the sum of the probabilities, the “log sum exp” trick is commonly applied to this in statistics and machine learning settings. Of course, you could compute the log of the sum and then exponentiate, but this approach should give some under/overflow protection. This Wikipedia article gives the trick, based on making this computation in terms of the largest probability instead of in absolute terms, https://en.m.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations . I’m unsure of other approaches though. – cdipaolo Jun 12 '20 at 16:19

0 Answers0