This is a partial answer
showing some some ideas to attack the integral.
It was started some days ago...
But well, time during xmas an new year is very volatile,
here is all that could be typed in the direction of a solution.
It is posted hoping that somebody may complement and finish.
Some "idea of calculus" stays in foreground in the presentation, so simple, stupid relations might be written explicitly
(and more complex ones not...).
Some of the relations were checked numerically, code can be provided if wished.
To start, i will use a substitution similar to the one of the OP:
$$
t = \sqrt{\frac x4} + \sqrt{\frac x4+1} \ .
$$
So under the first logarithm in the integral we have $1/t$.
Then
$$
\frac{t^2-1}t
=
\frac 1t
\left(\frac x4 + 2\sqrt{\frac x4\left(\frac x4+1\right)} +\frac x4 +1-1\right)
=
\frac1t
\cdot 2\left(\frac x4 + \sqrt{\frac x4\left(\frac x4+1\right)}\right)
=2\sqrt{\frac x4}=\sqrt x
\ .
$$
So we can express $x$ in terms of $t$.
For $x=0$ we get $t=1$, for $x=1$ we get $t=\varphi:=\frac 12(\sqrt5+1)$, the golden number.
The given integral, $J$ in my notation, becomes:
$$
\begin{aligned}
J
&=\int_0^1
\log ^2\left(\sqrt{\frac{x}{4}+1} - \sqrt{\frac{x}{4}}\right) \;\log ^3x\; \frac{1+9x}{1+x} \;
\frac{dx}x
\\
&=\int_1^\varphi
\log^2 t\cdot\log^3\left(\ \frac{(t^2-1)^2}{t^2}\ \right)
\cdot
\left(9-
\frac{8t^2}{t^4 - t^2 + 1}
\right)
\cdot
\frac{t^2+1}{t^2-1}
\cdot\frac{2\; dt}t
\\
&\qquad\text{ use $s=t^2$ to reduce polynomial degrees in subexpressions}
\\
&=\frac 14\int_1^{\varphi^2=\varphi+1}
\log^2 s\cdot\log^3\left(\ \frac{(s-1)^2}s\ \right)
\cdot
\left(9-
\frac{8s}{s^2 - s + 1}
\right)
\cdot
\frac{s+1}{s-1}
\cdot\frac{ds}s
\\
&=-\frac 14\int_{1/\varphi^2=2-\varphi}^1
\log^2 s\cdot\log^3\left(\ \frac{(s-1)^2}s\ \right)
\cdot
\left(9-
\frac{8s}{s^2 - s + 1}
\right)
\cdot
\frac{s+1}{s-1}
\cdot\frac{ds}s
\\
&=\color{blue}{-\frac 14\int_{1/\varphi^2=2-\varphi}^1
\log^2 s\cdot\log^3\left(\ \frac{(s-1)^2}s\ \right)
\cdot
\left(
\frac 2{s - 1} - \frac 9s + \frac {8(2s-1)}{s^2 - s + 1}
\right)\;
ds\ .\qquad(*)}
\end{aligned}
$$
At the step moving $[1,\varphi^2]=[1,\varphi+1]$ to $[\varphi^{-2},1]=[2-\varphi,1]$ we have substituted $s\to 1/s$,
which makes sense in case of higher logarithms showing up later - it is easier to avoid monodromy and
let arguments live in the convergence domain of the defining series.
The full partial fraction decomposition of the rational fraction part involved above in the last integral is:
$$
\frac 2{s - 1} - \frac 9s + \frac {8(2s-1)}{s^2 - s + 1} =
\frac 2{s - 1} - \frac 9s + \frac 8{s-a} + \frac 8{s-b}\ .
$$
Here, $a=\frac 12(1+i\sqrt3)$ is a primitive root of order six of unity, and $b=\bar a=1/a=a^5$ is the complex conjugate.
We try to compute $\color{blue}{(*)}$.
Mixed powers of logarithms are usually a problem, in best case,
the appearance of one logarithmic factor (at some power)
would be helpful.
For instance, i tried a long time to make the following work. Let us denote by $A,B,C$ the involved quantities:
$$
A=\log s\ ,\qquad B=\log|s-1|=\frac 12\log(\ (s-1)^2\ )\ ,\qquad C=2B-A=\log\frac{(s-1)^2}s\ .
$$
In $\color{blue}{(*)}$ there is a factor $A^2C^3=A^2(2B-A)^3$. Denote by $J(P)$ the integral obtained
formally from the one in $\color{blue}{(*)}$ by replacing $A^2C^3$ by $P$,
but always keeping $
\displaystyle\color{blue}{
\left(
\frac 2{s - 1} - \frac 9s + \frac {8(2s-1)}{s^2 - s + 1}
\right)}
$. By linearity in $P$ for $J(P)$ we can write a relation between integrals
$$
\color{brown}{J(\ (A+C)^5\ )}
-
\color{brown}{J(\ (A-C)^5\ )}
=2\left(\qquad \color{brown}{J(C^5)} + 10\color{red}{\underbrace{J(A^2C^3)}_{=J}}
+ 5 \color{magenta}{J(A^4C)}\qquad \right)\ .
$$
It comes from
$(A+C)^5-(A-C)^5=2(C^5+10A^2C^3+5A^4C)$.
It turns out that polylog formulas can be written to evaluate the $J(P^5)$-terms like
$\color{brown}{J((A\pm C)^5)}$ and $\color{brown}{J(C^5)}$.
However the mixed $J(A^4C)$ makes problems of different complexity.
I had to give up this path, and try to make a "polarization (of order five)" work...
The brown $J$-integrals are computed below. The pink one makes problems.
We can however split:
$$
\color{magenta}{J(A^4C)}
=
J(A^4(2B-A))=2J(A^4B)-J(A^5)\ .
$$
The integral $J(A^5)$ can be computed, we will start with it.
However, i am stuck with $J(A^4B)$. The unexploited direction
is the one concerning substitutions that "move (and split)"
the interval of integration.
So, in other words, we write the "extracted polynomial" $A^2C^3$ as a linear combination
of suitable polynomials in $A,B,C$ which are pure fifth powers (like $A^5$, $C^5$, $B^5$).
And the pure 5.th powers usually lead to computable integrals.
Unfortunately, i could not make this work. But - as a sample - the idea is that a relation like
$$
\tag{$\dagger$}
0=45\color{red}{\underbrace{A^2(2B-A)^3}_{\text{wanted}}}
-81\underbrace{B^5}_{\text{known}}
-81\underbrace{(B-A)^5}_{\text{known}}
+\underbrace{(B+A)^5}_{\text{"almost known"}}
+5\underbrace{(2B-A)^5}_{\text{known}}
+\underbrace{(B-2A)^5}_{\text{???}}
$$
together with the computation of the corresponding $J$-integrals
(all but the needed one)
is already a computation of $J=J(A^2C^3)=J(A^2(2B-A)^3)$.
It is also possible that some "mixed" (non-fifth-power) $J$-term can be computed,
and can aid, but i could not isolate one.
The value for $J(B^5)$ is already explicitly known in terms of polylogarithms,
through the primitive $S$ above. We have only to substitute $(1-s)\to s$
to move $B=\log|1-s|$ to $\log s$,
a linear substitution, obtain $\log^5 s$ times some rational fraction,
use partial fraction decomposion in linear terms over $\Bbb C$, then use $F$.
(Monodromy problems may show up, to make this point exact a numerical check is enough.)
For $J((B-A)^5$ we need to substitute again the argument of $\log$
inside $B-A=\log|s-1|-\log s$, so $t=\frac {s-1}s$ makes the progress.
For $J((2B-A)^5$ we need to substitute again the argument of $\log$
inside $2B-A=\log\frac{(s-1)^2}s$, so $t=\frac {(s-1)^2}s$ should be the try.
It turns out that $dt$ can be indeed isolated times a function of $t$,
so the substitution works!
For $J((B+A)^5)$ the corresponding $\log$ part is
$\log(s(s-1))$, and the substitution
$t=s(s-1)$ works with the fraction $\frac{(2s-1)\; ds}{s^2-s+1}=\frac{dt}{t+1}$,
however some more effort is needed to express
the parts in $\frac{ds}s$ and $\frac{ds}{s-1}$ that appear in the rational part
of $J((B+A)^5)$. Their sum is $\frac{ds}s+\frac{ds}{s-1}=\frac{(2s-1)\; ds}{s(s-1)}=\frac{dt}t$,
so one term can be used against the other one.
I could not make $(\dagger)$ work, nor could i found a similar
relation of degree five ("polarization") in the polynomial ring $\Bbb Q[A,B]$
expressing in a convenient manner $A^2C^3=A^2(2B-A)^3$.
To see what may more or less algorithmically work, let us compute such integrals.
First of all, recall some relations. The following primitives will be useful. They involve the logarithm $\log$, the dilogarithm
$\operatorname{Li}_2$, and more general polylogarithms $\operatorname{Li}_k$, $k=2,3,4,5,6$. Here
$\operatorname{Li}_k(z)=\sum z^n/n^k$.
It is useful to use also $\operatorname{Li}_1$ instead of a $\log$-expression.
Collected relations follow now.
$$
\begin{aligned}
\int\frac{\log^5 s}{s}\; ds
&=
\frac 16\log^6 s\ ,
\\
F(s)=F_A(s) := \int \frac{\log^5 s}{s - A}\; ds
&=
- \log^5 s\operatorname{Li}_1\left(\frac sA\right)
+ 5\log^4 s\operatorname{Li}_2\left(\frac sA\right)
- 20\log^3 s\operatorname{Li}_3\left(\frac sA\right)
+ 60\log^2 s\operatorname{Li}_4\left(\frac sA\right)
\\
&\qquad
- 120\log s\operatorname{Li}_5\left(\frac sA\right)
+ 120 \operatorname{Li}_6\left(\frac sA\right)
\\
&
= \sum_{0\le k\le 5}(-1)^k\frac {5!}{k!}\log^k s\operatorname{Li}_{6-k}\left(\frac sA\right)\ .
\\
\operatorname{Li}_k\left(\frac sa\right)
+ \operatorname{Li}_k\left(\frac sb\right)
&=-\operatorname{Li}_k(-s) +\frac 1{3^{k-1}} \operatorname{Li}_k(-s^3)\ , \qquad k\ge 1\ .
\\
\operatorname{Li}_k\left(\frac s{a-1}\right)
+ \operatorname{Li}_k\left(\frac s{b-1}\right)
&=-\operatorname{Li}_k(s) +\frac 1{3^{k-1}} \operatorname{Li}_k(s^3)\ , \qquad k\ge 1\ .
\end{aligned}
$$
The last two relations use the fact that $a,b$ and $(a-1), (b-1)$ are conjugated pairs of
primitive roots of unity of order $6$, respectively $3$.
Some values of the dilogarithm in powers of $\varphi$ are known, for instance:
$$
\begin{aligned}
\operatorname{Li}_2(\varphi^{-1}) &= \frac 1{10}\pi^2 -\log^2\varphi \ ,\\
\operatorname{Li}_2(-\varphi ) &= -\frac 1{10}\pi^2 -\frac 12\log^2\varphi \ ,\\
\operatorname{Li}_2(\varphi^{-2}) &= \frac 1{15}\pi^2 -\log^2\varphi \ ,\\
\operatorname{Li}_2(-\varphi^{-1}) &= -\frac 1{15}\pi^2 +\frac 12\log^2\varphi \ ,
\end{aligned}
$$
but i am not aware of (useful) higher polylogarithmic relations related to evaluation in $\varphi$-powers.
The integral $J(A^5)$ is possibly useful, let us compute it.
$$
\begin{aligned}
J(A^5)
&=-\frac 14\int_{2-\varphi}^1
\log^5 s
\cdot
\left(
\frac 2{s - 1} + \frac 8{s-a} + \frac 8{s-b} - \frac 9s
\right)
\; ds
\\
&=
\frac 24
\left[ \sum_{0\le k\le 5}(-1)^k\frac {5!}{k!}\log^k s\operatorname{Li}_{6-k}\left(\frac s1\right)\right]_{s=1}^{s=2-\phi}
\\
&\qquad\qquad
+\frac 84
\left[ \sum_{0\le k\le 5}(-1)^k\frac {5!}{k!}\log^k s\operatorname{Li}_{6-k}\left(\frac sa\right)\right]_{s=1}^{s=2-\phi}
\\
&\qquad\qquad
+
\frac 84
\left[ \sum_{0\le k\le 5}(-1)^k\frac {5!}{k!}\log^k s\operatorname{Li}_{6-k}\left(\frac sb\right)\right]_{s=1}^{s=2-\phi}
\\
&\qquad\qquad\qquad\qquad
-
\frac 94 \left[ \frac 16\log^6 \right]_{s=1}^{s=2-\phi}
\\
&\qquad\text{ and can be computed explicitly. For instance, for the pieces:}
\\
\int_{2-\varphi}^1
\log^5 s
\cdot
\frac {ds}{s - 1}
&=
\frac {44}3\log^6\varphi -4\log^4\varphi \cdot \pi^2+32\log^3\varphi\cdot \zeta(3)-\frac 2{63}\pi^6\\
&\qquad\qquad
+60\log^2\varphi \operatorname{Li}_4(2-\varphi)
+60\log \varphi \operatorname{Li}_5(2-\varphi)
+30 \operatorname{Li}_4(2-\varphi)
&\qquad\text{ and for the piece in $\frac 1{s-a} + \frac 1{s-b}=\frac{2s-1}{s^2 -s +1}$}
\\
&\qquad\text{ one can use either the relation between polylogarithms in $s/a$, $s/b$; $-s$, and $-s^3$ (monodromy problems)}
\\
&\qquad\text{ or alternatively use $\frac 1{s-a} + \frac 1{s-b}=\frac{2s-1}{s^2 -s +1}$ and }
\\
\int_{2-\varphi}^1
\log^5 s
\cdot
\frac {(2s-1)\;ds}{s^2 - s + 1}
&=
\int_{2-\varphi}^1
\log^5 s
\cdot
\left(\frac{3s^2}{s^3+1} -\frac1{s+1}\right)\; ds
\\
&=
\frac 1{3^5}\int_{(2-\varphi)^{1/3}}^1
\log^5 s
\cdot
\frac {ds}{s+1}
-
\int_{2-\varphi}^1
\log^5 s
\cdot
\frac {ds}{s+1}
\ ,
\end{aligned}
$$
and again we have in the last line an explicit relation using $F(s)$ (and thus polylogarithms) computed
in $2-\varphi$ and its third root.
This computation is a model of what can be done.
The integral $\color{brown}{J(C^5)}$ is in fact divergent, we take it from $2-\varphi$ to $1_-$ which
is $1-\varepsilon$, for some $\varepsilon>0$.
$$
\begin{aligned}
\color{brown}{J(C^5)}
&=-\frac 14\int_{2-\varphi}^{1_-=1-\varepsilon}
\log^5 \frac{(s-1)^2}s
\cdot
\left(9-
\frac{8s}{s^2 - s + 1}
\right)
\cdot
\frac{(s+1)(s-1)}{(s-1)^2/s}
\cdot\frac{ds}{s^2}
\\
&\qquad\qquad\text{ Substitution $u=(s-1)^2/s=s-2+\frac 1s$, then $du=\left(1-\frac1{s^2}\right)\,ds=-\frac{s^2-1}{s^2}\; ds$ }
\\
&=
\frac 14\int_{0_+}^1
\log^5 u
\cdot
\left(9-
\frac{8}{u+1}
\right)
\cdot \frac {du}u
=\frac 14\int_{0_+}^1
\log^5 u
\cdot
\left( \frac{8}{u+1} +\frac 1u\right)
\; du
\\
&
=
\frac 14\int_{0_+}^1
\log^5 u
\cdot
\frac{8\; du}{u+1}
+
\frac 14\int_{0_+=\varepsilon^2/(1-\varepsilon)}^1
\log^5 u
\cdot\frac {du}u
\\
&=-\frac {31}{126}\pi^6 -\frac 1{24}\log^6\frac{\varepsilon^2}{1-\varepsilon}+O(\varepsilon)\ .
\end{aligned}
$$
(Because of the logarithmic singularity, the use of
related series is more complicated, we use related integrals.)
The integral $\color{brown}{J(\ (C+A)^5\ )}=32J(B^5)$. The equality
holds since $C+A=\log\frac{(s-1)^2s}+\log s=\log(\ (s-1)^2\ )=2B$. The integral is divergent, too,
we take it from $2-\varphi$ to $1_-$ with the same conventions as above.
$$
\begin{aligned}
\color{brown}{J(\ (C+A)^5\ )} %=32J(B^5)
&=-\frac {32}4\int_{2-\varphi}^{1_-=1-\varepsilon}
\log^5 |s-1|
\cdot
\left(
\frac 2{s - 1} - \frac 9s + \frac 8{s-a} + \frac 8{s-b}
\right)
\cdot ds
\\
&=-\frac {32}4\int_{0_+=\varepsilon}^{\varphi-1}
\log^5 s
\cdot
\left(
-\frac 2s + \frac 9{s - 1} - \frac 8{s-(a-1)} - \frac 8{s-(b-1)}
\right)
\cdot ds
\ ,
\end{aligned}
$$
and all parts can be computed (after splitting the parenthesis). Note that
$\varphi-1=\frac 12(\sqrt 5-1)=\varphi^{-1}$.
For example, using $F_1(\varphi-1)$:
$$
\int_0^{\varphi-1=1/varphi}
\log^5 s\frac{ds}{s-1}
=
2\log^6\varphi
+
5\log^4\varphi\cdot\operatorname{Li}_2(\varphi-1)
+
20\log^3\varphi\cdot\operatorname{Li}_3(\varphi-1)
+
60\log^2\varphi\cdot\operatorname{Li}_4(\varphi-1)
+
60\log\varphi\cdot\operatorname{Li}_5(\varphi-1)
+
120\operatorname{Li}_6(\varphi-1)
\ .
$$
The other two terms involving $\frac 8{s-(a-1)}$ and $\frac 8{s-(b-1)}$
add to $F_{a-1}(\varphi-1)+F_{b-1}(\varphi-1)$,
and we obtain sums of polylogarithms of the shape
$
\displaystyle
\operatorname{Li}_k\left(\frac{\varphi-1}{a-1}\right) +
\operatorname{Li}_k\left(\frac{\varphi-1}{a-1}\right) =
-
\operatorname{Li}_k(\varphi {-1})
+
\frac 1{3^{k-1}}
\operatorname{Li}_k(\varphi^{-3})$,
times $\log\varphi$ powers.
The singular term is $\frac 1s\log^5s$, is easily integrated to $\frac 16\log^6s$,
delivers a $\log^6\varphi$ finite part, and the control on the $\log^6\varepsilon$ singularity.
The integral $\color{brown}{J(\ (C-A)^5\ )}=32J(\ (B-A)^5)$ is again divergent,
same conventions as above apply also below.
$$
\begin{aligned}
\color{brown}{J(\ (C-A)^5\ )}
&=-\frac 14\int_{2-\varphi}^{1_-=1-\varepsilon}
\log^5 \left(\frac{(s-1)^2}s\cdot\frac1s\right)
\cdot
\left(
\frac 2{s - 1} - \frac 9s + \frac {8(2s-1)}{s^2 -s +1}
\right)
\cdot ds
\\
&=-\frac 14\cdot 32
\int_{2-\varphi}^{1_-}
\log^5 \left|\frac{s-1}s\right|
\cdot
\left(
\frac 2{s - 1} - \frac 9s + \frac {8(2s-1)}{s^2 -s +1}
\right)
\cdot ds
\\
&\qquad\text{ and with $t=(1-s)/s=1/s-1$, $dt=-ds/s^2$}
\\
&= -\frac 14\cdot 32
\int_{0_+=\varepsilon/(1-\varepsilon)}^{\varphi}
\log^5 t
\cdot
\left(
\frac 2t -\frac 9{t-1} + \frac {8(t-2)}{t^2-t+1}
\right)
\cdot dt
\ ,
\end{aligned}
$$
and all parts obtained after
expanding the parenthesis can be computed. (Note that
$\varphi-1=\frac 12(\sqrt 5-1)=\varphi^{-1}$.)
For example, using $F_1(\varphi-1)$:
$$
\int_0^{\varphi-1=1/varphi}
\log^5 s\frac{ds}{s-1}
=
2\log^6\varphi
+
5\log^4\varphi\cdot\operatorname{Li}_2(\varphi-1)
+
20\log^3\varphi\cdot\operatorname{Li}_3(\varphi-1)
+
60\log^2\varphi\cdot\operatorname{Li}_4(\varphi-1)
+
60\log\varphi\cdot\operatorname{Li}_5(\varphi-1)
+
120\operatorname{Li}_6(\varphi-1)
\ .
$$
The other two terms involving $\frac 8{s-(a-1)}$ and $\frac 8{s-(b-1)}$
add to $F_{a-1}(\varphi-1)+F_{b-1}(\varphi-1)$,
and we obtain sums of polylogarithms of the shape
$
\displaystyle
\operatorname{Li}_k\left(\frac{\varphi-1}{a-1}\right) +
\operatorname{Li}_k\left(\frac{\varphi-1}{a-1}\right) =
-
\operatorname{Li}_k(\varphi {-1}) =
+
\frac 1{3^{k-1}}
\operatorname{Li}_k(\varphi^{-3})$,
times $\log\varphi$ powers.
The singular term is $\frac 1s\log^5s$, is easily integrated to $\frac 16\log^6s$,
delivers a $\log^6\varphi$ finite part, and the control on the $\log^6\varepsilon$ singularity.
Some final words. The relation to be shown is "clean", it is for this reason
possible that a short proof exists. To find it, the search may be hairy, but once
having one path it is easy to refine it, or to elaborate a quicker on hiding the same essence.
Above, some things worked in the direction of some in between progress, but a verdict about progress
can be given only when having a solution. Maybe a combination of integration techniques with
series computations (values of generalized hypergeometric functions) can go deeper,
but again the problem is that the integration is done on intervals with boundary in numbers like
$\varphi^k$, $k=-2,1,2$, and simple substitutions tend to move the boundaries without seeing them
back again. I have to stop here... and come back when i have a solution.