The weights for mixing have to be nonnegative and to sum to 1. The colors live in a three-dimensional space. So what you are looking for are barycentric coordinates in a three-dimensional simplicial complex. In general, there is no simple formula, but one can write down an algorithm to determine the mixing rates - provided that mixing is possible. Mixing is possible if and only if the color to mix lies in the convex hull of the existent colors.
Let's generate a list of random colors and
SeedRandom[666];
colors = RandomInteger[{0, 255}, {20, 3}];
colormesh = DelaunayMesh[colors];
memberQ = RegionMember[colormesh];
Now, for a given color, say
color = RandomInteger[{0, 255}, {3}];
we obtain a list of four colors and four mixing weights as follows:
color = RandomInteger[{0, 255}, {3}];
If[! memberQ[color],
Print["Warning: Color is not mixable. Computing the mixing for the \
closest mixable color instead."];
];
cell = Region`Mesh`MeshNearestCellIndex[colormesh, color];
cellcoordinates = MeshPrimitives[colormesh, cell][[1]];
result = With[{
x = LinearSolve[
Transpose[Rest[cellcoordinates] - ConstantArray[cellcoordinates[[1]], cell[[1]]]],
nearest[color] - cellcoordinates[[1]]
]
},
Association[
"MixingColorIndices" -> MeshCells[colormesh, cell][[1]],
"MixingColors" -> cellcoordinates,
"MixingWeights" -> Prepend[x, 1 - Total[x]]
]
]
<|
"MixingColorIndices" -> {19, 4, 3, 9},
"MixingColors" -> {{8., 163., 61.}, {104., 116., 69.}, {73., 246., 29.}, {35., 222., 114.}},
"MixingWeights" -> {0.098337, 0.0389252, 0.788665, 0.074073}
|>
Test mixing:
Norm[result["MixingWeights"].result["MixingColors"] - color]
0.
Great, the error is zero. Actually, there should only be a nonnegligible error, if the color color is not mixable; the small program above would warn you when this is the case.
The association result contains also the field "MixingColorIndices". It is the list of integer indices of the mixing colors within the list colors. This might make it easier to find the shelf where the colors are to be looked up physically.