The example in the question only has the trivial solution c1==c2==0 because the assumed form of V doesn't actually correspond to an integral of the motion.
To illustrate the method for finding the constants in general, let's pick a different example where at least the conserved quantity has a form similar to V with non-trivial coefficients.
I'll be using the Euler equations for the angular velocity of a freely spinning object in the body frame, where the two known conserved quantities (energy and magnitude of the angular momentum) have exactly the quadratic form given in the question. The parameters are the principal moments of inertia, i1, i2, i3. They take the place of a, b, d. The equations of motion are again similar to the ones in the question, except that there's no linear term (the original problem is actually of the Lotka-Volterra type).
After defining the derivatives and the quadratic form V, I use SolveAlways to determine the parameters. Everything that doesn't appear in the second argument of SolveAlways is considered to be a constant parameter. This is an important ingredient in finding the solution.
x'[t_] := (i2 - i3)/i1 y[t] z[t];
y'[t_] := (i3 - i1)/i2 x[t] z[t]; z'[t_] := (i1 - i2)/i3 x[t] y[t]
V[x_, y_, z_] := c1 x^2 + c2 y^2 + c3 z^2
sols =
SolveAlways[0 == D[V[x[t], y[t], z[t]], t], {x[t], y[t], z[t]}]
{{i1 -> 0, i3 -> 0}, {i1 -> 0, i3 -> i2}, {i2 -> 0,
i1 -> 0}, {i2 -> 0, i3 -> 0}, {i2 -> 0, i3 -> i1}, {i2 -> i1,
i3 -> 0}, {c2 -> 0, i2 -> 0}, {i2 -> i1, i3 -> i1}, {c1 -> 0,
i1 -> 0}, {c2 -> c1, i2 -> i1},
{c3 -> (i3 (c2 i1^2 - c1 i2^2 - c2 i1 i3 + c1 i2 i3))/(
i1 (i1 - i2) i2)}}
Inspecting the result, only the last case is of interest. The others are possible ways of getting a conserved quantities for special choices of the principal moments. The last answer is the most general.
Therefore, define V with the last choice of parameters as our conserved quantity:
Clear[c1, c2, i1, i2, i3, constOfMotion];
constOfMotion[c1_, c2_] = Simplify[V[x[t], y[t], z[t]] /. Last[sols]]
c1 x[t]^2 + c2 y[t]^2 + (
i3 (c2 i1 (i1 - i3) + c1 i2 (-i2 + i3)) z[t]^2)/(i1 (i1 - i2) i2)
This is the general answer for the rigid body problem. It contains two free parameters c1 and c2 because no matter what principal moments you choose there are always two constants of the motion, $I_1 x^2 + I_2 y^2 + I_3 z^2$ and $I_1^2 x^2 + I_2^2 y^2 + I_3^2 z^2$. The two parameters c1, c2 encode the weights with which these two are added to form another constant of motion.
Set(=) withSetDelayed(:=) andEqual(==). Equations are defined usingEqual, functions are defined with aBlank(_) pattern as an argument andSetorSetDelayed, i.ef[x_]:=1+x^2. [Edit] your question to fix that first. – rhermans Jul 12 '18 at 17:42x1supposed to bex1? Also, A trivial solution isc1 == c2== 0. – Jens Jul 12 '18 at 17:47x1in the definition ofV... – Jens Jul 12 '18 at 17:56c1 == c2 ==0is valid, but I am looking for a method to solve forc1andc2for other functionsVthat might be a lot more complicated. – rpa Jul 12 '18 at 18:46