First off, Cayley's resolvent is meant for irreducible quintics, which the given example is not. If the quintic is reducible, then it can always be solved by radicals through the solution of the lower-degree factors and yet Cayley's resolvent may fail to give a rational root. For instance, if we blindly apply Cayley's resolvent to the equation
$x^5-x^3+x-1=0,$
we ultimately fail to find a rational root to the resolvent; yet this quintic is solvable by factoring it as $(x-1)(x^4+x^3+1)$ and applying the usual radical-based algorithm to the quartic factor.
When applying Cayley's resolvent to an irreducible quintic equation, you do not need to fully solve the resolvent sextic. You need only to check for a rational root. If none exists, you're done with the method; the quintic is not solvable. If the rational root does exist, then the quintic will be solvable and you're good if that is all you need to know. However, going on to actually get the quintic roots requires putting the resolvent root and the depressed coefficients into a quite complicated formula.
It is good practice to find the constant term in the cubic polynomial $P$ first, because this could be zero. If so, then the full sextic resolvent also has a zero constant term and we can immediately identify zero as the required rational root (also the formula for the quintic roots themselves becomes simplified).
See this answer for an application to the depressed quintic equation $x^5+20x^3+20x^2+30x+10=0$.