TL;DR I'm not sure if either of my attempts are correct, will update later.
Attempt 1:
First SE post I've written so I apologize in advance if anything is out of order. The error messages from the OP coming from NIntegrate could be because NIntegrate usually has a hard time dealing with symbolic limits of integration, so NIntegrate was being called before a value was plugged into $r$.
I was able to get a solution to the problem in MM version 12.2 with:
fun[rho_, r_] := Times[r, Integrate[Times[rho, p, p], {p, 0, r}]];
sol = NDSolve[{R'[r] == D[fun[R[r], r], r], R[0] == 1}, R, {r, 0, 1}][[1,1,2]];
Plot[sol[r], {r, 0, 1}]
gives a plot that looks like:
I would note that the solution seems to be faster without using NumericQ on the function, but depending on your application, you may or may not wish to keep it there.
I was able to get two different analytical solutions, the first being:
a1 = DSolve[{R'[r] == D[r * Integrate[q * q * R[q], {q, 0, r}], r], R[0] == p}, R[r], r]
and a more refined version over the unit interval:
a2 = DSolve[{R'[r] == D[r * Integrate[q * q * R[q], {q, 0, r}], r], R[0] == p}, R[r], {r, 0, 1}]
yielding the expressions:
\begin{equation}
R\left( r\right)=\frac{1}{4\sqrt{2}} e^{\frac{r^{4}}{4}}p\left[ r\;\Gamma\left( -\frac{1}{4}\right) -\left( r^{4}\right)^{\frac{1}{4}} \Gamma\left( -\frac{1}{4}, \frac{r^{4}}{4}\right) \right]
\end{equation}
where $\Gamma\left(\cdot\right)$ and $\Gamma\left(\cdot,\cdot\right)$ are intended to mean the gamma and upper-incomplete gamma functions respectively, and on the unit interval:
\begin{equation}
R\left(r\right)=\frac{1}{4}e^{\frac{r^{4}}{4}}p\left[E_{5/4}\left(\frac{r^{4}}{4}\right) + 2\sqrt{2}\;r\;\Gamma\left(\frac{3}{4}\right)\right]
\end{equation}
where $E_{n}$ is the exponential integral function.
To be true(er) to the original problem though, I was able to use:
mfun[rho_?NumericQ, r_?NumericQ] := r NIntegrate[rho r1^2, {r1,0,r}];
msol = NDSolve[{R'[r] == D[mfun[R[r], r], r], R[0] == 1}, R, {r,0,1}][[1,1,2]]
which when plotted gives the same visual as shown above.
funa[rho_, r_] = r Integrate[rho r1^2, {r1, 0, r}]; DSolve[{R'[r] == D[funa[R[r], r], r], R[0] == 1}, R, r], but this isn't suitable for the real case, right? – xzczd May 16 '21 at 15:22Derivativecan handle things likeDerivative[1, 0][fun][1., 1.]to some degree, related: https://mathematica.stackexchange.com/q/196998/1871 ) mattiav27, what version are you in? – xzczd May 16 '21 at 15:27fun[R[r], r], doesNIntegrateunderstand thatR[r]must be evaluated at the pointr1inside the integral? – mattiav27 May 16 '21 at 15:30r NIntegrate[R[r1] r1^2, {r1, 0, r}]? If so, your code is wrong. – xzczd May 16 '21 at 15:32