0

I am reading the article of Deriving probability distributions using the Principle of Maximum Entropy

and I am trying to derive some of the equations in it automatically using Mathematica.

1. Derivation of maximum entropy probability distribution with no other constraints (uniform distribution)

First, we solve for the case where the only constraint is that the distribution is a pdf, which we will see is the uniform distribution. To maximize entropy, we want to minimize the following function: $$ J(p)=\int_a^b p(x) \ln p(x) d x-\lambda_0\left(\int_a^b p(x) d x-1\right) $$ . Taking the derivative with respect ot $p(x)$ and setting to zero, $$ \frac{\delta J}{\delta p(x)}=1+\ln p(x)-\lambda_0=0 $$

Can I derive the second equation automatically using Mathematica?

if just $ J(p)=\int_a^b p(x) \ln p(x) dx$, I can do

Needs["VariationalMethods`"]
VariationalD[p[x] Log[p[x]], p[x], x]

the output is 1 + Log[p[x]] as expected.

But here $ J(p)=\int_a^b p(x) \ln p(x) d x-\lambda_0\left(\int_a^b p(x) d x-1\right) $, how can I do that?

benjaminchanming
  • 143
  • 5
  • 1
    Why not VariationalD[p[x] Log[p[x]] - λ0 ( p[x] - 1), p[x], x]? What's the issue? – bmf Jan 12 '23 at 11:40
  • @bmf: This returns the input in 13.2 on Windows 10. Did you try your suggestion before having posted it? – user64494 Jan 12 '23 at 11:52
  • 1
    @user64494 that's weird. on my laptop it returns 1 - λ0 + Log[p[x]] which is the expected answer from the OP – bmf Jan 12 '23 at 11:55
  • 1
    @bmf's code works as expected also in 12.3.1 (Windows) and 13.2.0 (Wolfram Cloud). – Domen Jan 12 '23 at 11:56
  • @bmf Thanks, I am not sure Inegrate[p[x] Log[p[x]] - \[Lambda]0 (p[x] - 1)] === Inegrate[p[x] Log[p[x]]] - \[Lambda]0*(Inegrate[p[x]] - 1), the output is False – benjaminchanming Jan 12 '23 at 12:00
  • @benjaminchanming firstly Integrate and also with respect to what? And also, this last comment contradicts what you wrote in the OP in my opinion – bmf Jan 12 '23 at 12:02
  • @bmf: Now it produces 1-\[Lambda]0+Log[p[x]] and a warning. – user64494 Jan 12 '23 at 12:04
  • @bmf By the way, Can I use Mathematica to automatically transform J(p) into the form Integrate[f[x]], to get the expression of f, so that I can pass f into the first parameter of VariationalD[f,u[x],x] ? – benjaminchanming Jan 12 '23 at 12:04
  • @user64494 with 13.2.0 on macOS I have no warnings; see – bmf Jan 12 '23 at 12:07
  • @bmf Thanks, I changed it into Integrate[p[x] Log[p[x]] - \[Lambda]0 (p[x] - 1), x] === Integrate[p[x] Log[p[x]], x] - \[Lambda]0*(Integrate[p[x], x] - 1), but it still return False. And why the comment contradicts what I wrote in the OP? – benjaminchanming Jan 12 '23 at 12:14
  • 1
    @benjaminchanming I mean that you wrote that if $J$ has the simple form you know how to do the variational derivative in Mathematica. For the complicated form I am unsure why you got stuck, but I showed you that you can write the same command effectively and get the result that you want for $\tfrac{\delta J}{\delta \rho}$ and now you say you are unsure about the result, which is the one you wanted. Unless, I am missing something – bmf Jan 12 '23 at 12:18
  • @bmf Thanks! Thanks! Sorry I am not good at calculus. So that I am not sure that the $\int_a^b$ can be factored out to transform

    $$ \int_a^b p(x) \ln p(x) d x-\lambda_0\left(\int_a^b p(x) d x-1\right) $$

    into $$ \int_a^b (p(x) \ln p(x)-\lambda_0\left( p(x)-1\right))dx $$

    By which rule in calculus can we do the factoring out $\int_a^b $ transformation?

     – benjaminchanming Jan 12 '23 at 12:32
  • 1
    @benjaminchanming, linearity is one of the basic properties of integral, and the equality holds (if both of the integrals exist). However, for ill-behaved (!!) functions, the equality might not hold, and that is why Mathematica returns False. – Domen Jan 12 '23 at 12:42
  • @Domen Thanks! Now I understand. By the way, Is there a built-in function in Mathematica that can do the factor-out transformation automatically for me? – benjaminchanming Jan 12 '23 at 12:48
  • @benjaminchanming you can write a rule for yourself and apply the rule if you want :) – bmf Jan 13 '23 at 01:42
  • @bmf Thanks! I will try it. – benjaminchanming Jan 15 '23 at 12:45

0 Answers0