I know the substitution but how should I continue? 
$$\int_0^1 x\sqrt{\frac{1-\sqrt{x}}{1+\sqrt{x}}}dx$$
HINT:
For $x\in [0,1]$, $\theta\in [0,\pi/2]$. Then, we can write
$$\begin{align} \sqrt{\frac{1-\sqrt x}{1+\sqrt x}}&=\sqrt{\frac{1-\cos(\theta)}{1+\cos(\theta)}}\\\\&=\sqrt{\frac{(1-\cos(\theta))^2}{1-\cos^2(\theta)}}\\\\&=\frac{1-\cos(\theta)}{\sin(\theta)} \end{align}$$
$$\int_0^1 x\sqrt{\frac{1-\sqrt{x}}{1+\sqrt{x}}}dx$$
Let $\sqrt{x} = \cos\theta,\theta \in (0,\pi/2)$
$$\int_0^1 x\sqrt{\frac{1-\sqrt{x}}{1+\sqrt{x}}}dx$$
$$\int_{0}^{\pi/2} \cos^2x \frac{1-\cos x}{\sin x}2\cos x\sin xdx$$
$$2\int_{0}^{\pi/2} \cos^3x -\cos^4 xdx$$ We could use the following formula to calculate the final result: $$\int\cos^n x \ dx = \frac{1}n \cos^{n-1}x \sin x + \frac{n-1}{n}\int\cos^{n-2} x \ dx$$
Here is a less brute-force way to get to the desired subsitution
$$\int_0^1 x\sqrt{\frac{1-\sqrt{x}}{1+\sqrt{x}}}dx\stackrel{x \to x^2}{=} 2\int_0^1 x^3\sqrt{\frac{1-x}{1+x}}dx$$
The substitution $u = \frac{1-x}{1+x}$ is now obvious, yielding
$$-2\int_0^1 \left(\frac{1-u}{1+u}\right)^3u^{-1/2}\left(\frac{1-u}{(u+1)^2} - \frac{1}{u+1}\right)du$$
$$-4\int_0^1 \frac{(u-1)^3 \sqrt u}{(u + 1)^5}du$$
From here we have obvious partial fraction decomposition, which could also clearly be solved with a simple tangent substitution. A bit tedius (subsitute $v=u+1$ to avoid clutter if desired) but should yield the desired answer quite cleanly.
An alternative way - by setting $x=u^2$, then $u=\frac{1-v}{1+v}$ and $v=z^2$, we are left with:
$$ \int_{0}^{1}2u^3\sqrt{\frac{1-u}{1+u}}\,du = 4\int_{0}^{1}\frac{(1-v)^3}{(1+v)^5}\sqrt{v}\,dv =8\int_{0}^{1}\frac{z^2(1-z^2)^3}{(1+z^2)^5}\,dz$$ Then by applying integration by parts we get $$ \int_{0}^{1}\frac{1-9 z^2+15 z^4-7 z^6}{(1+z^2)^4}\,dz=-7\,I_1+36\,I_2-60\,I_3+32\,I_4 $$ with $I_k=\int_{0}^{1}\frac{dz}{(1+z^2)^k}.$ These integrals can be computed through differentiation under the integral sign, since for any $a>0$ we have $$ \int_{0}^{1}\frac{a\,dz}{a+z^2}=\frac{\text{arccot}\sqrt{a}}{\sqrt{a}} $$ hence $I_1=\frac{\pi}{4}, I_2=\frac{1}{4}+\frac{\pi}{8}, I_3=\frac{1}{4}+\frac{3\pi}{32}, I_4=\frac{11}{48}+\frac{5\pi}{64}$. Summarizing,
$$ \int_{0}^{1}x\sqrt{\frac{1-\sqrt x}{1+\sqrt x}}\,dx = \color{red}{\frac{4}{3}-\frac{3\pi}{8}}.$$
Note that if you set $\sqrt{x} = \cos\theta$, then $$\frac{1-\cos\theta}{1+\cos\theta} = \tan\frac{\theta}{2}.$$