Continuous functions satisfying $f(f(x))=x$, for all $x \in \mathbb{R}$, and $\int_{-x}^{0} f(t)dt - \int_{0}^{x^2}f(t)dt=x^3$ for $x>0$

Question

Find all continuous functions $f: \mathbb{R} \to \mathbb{R}$ satisfying:

I. For all $x\in \mathbb{R}$, $f(f(x))=x$

II. For all $x>0$, $\displaystyle\int_{-x}^{0} f(t)dt - \displaystyle\int_{0}^{x^2}f(t)dt=x^3$

So far I got that $f$ is strictly decreasing. Also I differentiated both sides of the equation in II and got the equation $f(-x)-2xf(x^2)=3x^2$ for all $x>0$, but I'm stuck here. How can I solve this problem?

Edit: I found an answer, but not a proof:

$x>0$ : $f(x)=-\sqrt{x}$, $x<0$ : $f(x)=x^2$

Answer 1

This is not an answer, but is too long for a comment. This is as far as I got.

Write the anti-derivative of $F$ as $f$.

Then by (2), for any $x>0$: $$x^3 = F(0) - F(-x) - F(x^2) + F(0) = 2F(0) - F(-x) - F(x^2).$$

Taking a derivative: $$3x^2 = f(-x) - 2xf(x^2) \iff f(-x) = 3x( x + \frac{2}{3}f(x^2)).$$

So if we know $f(t)$ for all $t>0$, we know $f(-t)$ for all $t>0$ also. Also note that $f(0) = f(-0) = 2 \cdot 0 ( 0 + \frac{2}{3}f(0^2)) = 0$, so $f(0)= 0$.

Now we consider property (1), which in other words says $f$ is a continuous involution.

Now note that $f(f(x)) =x $ implies that it must be surjective and injective, so it is a bijection, (surjective as for any $x$, $f(f(x)) =x $, so $f$ maps onto $\mathbb{R}$ and injective as $f(x) = f(y)$ implies $x=f(f(x)) = f(f(y)) = y$). Then continuous bijections are strictly monotonic.

Suppose $f$ is increasing. For any $x$ there are three cases, $f(x) < x, f(x) = x, f(x)>x$. In case (1), we then have $x=f(f(x)) <x$ a contradiction, similarly in case (3). Hence $f(x) = x$ for all $x$ if $f$ increasing. It is is easy to see this $f$ doesn't work.

So we know that $f$ is strictly decreasing.

Then $f(0) =0$, so $f(-x) >0$ for all $x>0$. Thus $3x(x+\frac{2}{3}f(x^2))>0$, so $f(x^2) > -\frac{3}{2}x$ for $x>0$, or $f(x) > - \frac{3}{2}\sqrt{x}$ for $x>0$.

We also have $x>0$ implies $f(x)<0$, so for any $x >0$, we have $- \frac{3}{2} \sqrt{x} < f(x) < 0$. Thus also for $x>0$, $f(-x) = 3x(x+ \frac{2}{3}f(x^2)) < 3x^2$.

Note then that $f|_{[0,\infty)}$ is a bijection onto $(-\infty,0]$. So for any $x>0$, $f(-x) = f^{-1}(-x)$.

Thus $$\int_{-x}^0 f(t) dt = \int_{-x} f^{-1}(t) dt = (tf^{-1}(t) - F(f^{-1}(t)))|_{-x}^0 = - F(f^{-1}(0)) - (-x f^{-1}(-x) - F(f^{-1}(-x))) = xf^{-1}(-x) + F(f^{-1}(-x)),$$ where $F$ is the antiderivative of $F$ (WLOG suppose $F(0) =0$), and we use this.

Thus $$\int_{-x}^0 f(t) dt - \int_0^{x^2} f(t) dt = xf^{-1}(-x) + F(f^{-1}(-x)) - F(x^2) = x^3.$$

A summary:

For any $f$ satisfying (1) and (2), let $F$ be it's antiderivative s.t. $F(0) =0 $.

Then:

(1) $f(0)=0$.

(2) $f$ is a strictly decreasing bijection.

(3) $f(-x) = 3x (x+\frac{2}{3}f(x^2))$

(4) For all $x>0$: $$- \frac{3}{2} \sqrt{x} < f(x) < 0< f(-x) <3x^2$$

(5) $$xf^{-1}(-x) + F(f^{-1}(-x)) - F(x^2) = x^3$$