I have a deck of $25$ cards, $10$ of them are identical to each other, and the other $15$ are made of $5$ sets of $3$ identical cards for visualisation:
$10$ black, $3$ green, $3$ red, $3$ yellow, $3$ blue, $3$ pink
If I draw $10$ cards randomly, how many different hands can I draw?
I tried using my combinatorics knowledge but it wasn't enough, the answer itself is less important to me but the actual way is what I'm after here.
Use the generating function $$(1+x+x^2+x^3)^5(1+x+x^2+x^3+x^4+x^5+x^6+x^7+x^8+x^9+x^{10})$$ and look at the co-efficient of the $x^{10}$ term. This generating function could be written as $$\left(\dfrac{1-x^4}{1-x}\right)^5 \dfrac{1-x^{11}}{1-x}$$
I get $1 + 5 + 15 + 35 + 65 + 101 + 135 + 155 + 155 + 135 + 101 = 903$ distinct hands though these are not equally like.
Let $x_b$ be the number of black cards, $x_{\color{blue}{b}}$ be the number of blue cards, $x_g$ be the number of green cards, $x_r$ the number of red cards, $x_p$ be the number of pink cards, and $x_y$ be the number of yellow cards. Then the number of hands with ten cards is the number of solutions of the equation $$x_b + x_{\color{blue}{b}} + x_g + x_p + x_r + x_y = 10 \tag{1}$$ in the nonnegative integers subject to the restrictions that \begin{align*} x_b & \leq 10\\ x_{\color{blue}{b}}, x_g, x_p, x_r, x_y & \leq 3 \end{align*} Since we only have ten cards in our hand, the condition $x_b \leq 10$ cannot be violated. However, up to two of the remaining five conditions can be violated. It is not possible to violate three of them since $3 \cdot 4 = 12 > 10$.
If there were no restrictions, a particular solution of equation 1 would correspond to the placement of five addition signs in a row of ten ones. For instance, $$1 1 1 1 + 1 1 + 1 + + 1 1 1 +$$ corresponds to the solution $x_b = 4$, $x_{\color{blue}{b}} = 2$, $x_g = 1$, $x_p = 0$, $x_r = 3$, and $x_y = 0$. The number of such solutions is the number of ways we can insert five addition signs in a row of ten ones, which is $$\binom{10 + 5}{5} = \binom{15}{5}$$ since we must choose which five of the fifteen positions (for ten ones and five addition signs) will be filled with addition signs.
From these, we must exclude those cases in which one or more of the restrictions are violated.
Suppose $x_{\color{blue}{b}} > 3$. Then $x_{\color{blue}{b}} \geq 4$. Let $y_{\color{\blue}{b}} = x_b - 4$. Then $y_{\color{blue}{b}}$ is a nonnegative integer. Substituting $y_{\color{blue}{b}} + 4$ for $x_b$ in equation $1$ yields \begin{align*} x_b + y_{\color{blue}{b}} + 4 + x_g + x_r + x_p + x_y & = 10\\ x_b + y_{\color{blue}{b}} + x_g + x_r + x_p + x_y & = 6 \tag{2} \end{align*} Equation 2 is an equation in the nonnegative integers with $$\binom{6 + 5}{5} = \binom{11}{5}$$ solutions. By symmetry, there are equal number of solutions in which one of the other four restrictions is violated. Hence, there are $$\binom{5}{1}\binom{11}{5}$$ cases in which one of the restrictions is violated.
However, if we subtract $\binom{5}{1}\binom{11}{5}$ from $\binom{15}{5}$, we will have counted those cases in which two of the conditions are violated twice, once for each way we counted one of the colors as violating the restriction. Since we only want to subtract them once, we must these cases back.
Suppose $x_{\color{blue}{b}} > 3$ and $x_g > 3$. Let $y_{\color{blue}{b}} = x_{\color{blue}{b}} - 4$ and $y_g = x_g - 4$. Then $y_{\color{blue}{b}}$ and $y_g$ are nonnegative integers. Substituting $y_{\color{blue}{b}} + 4$ for $x_{\color{blue}{b}}$ and $y_g + 4$ for $x_g$ in equation 1 yields \begin{align*} x_b + y_{\color{blue}{b}} + 4 + y_g + 4 + x_p + x_r + x_y = 10\\ x_b + y_{\color{blue}{b}} + y_g + x_p + x_r + x_y & = 2 \tag{3} \end{align*} Equation 3 is an equation in the nonnegative integers with $$\binom{2 + 5}{5} = \binom{7}{5}$$ solutions. By symmetry, each of the $\binom{5}{2}$ cases in which two of the five restrictions are violated results in an equal number of cases. Hence, there are $$\binom{5}{2}\binom{7}{5}$$ cases in which two of the restrictions are violated.
By the Inclusion-Exclusion Principle, the number of ten card hands that can be formed is $$\binom{15}{5} - \binom{5}{1}\binom{11}{5} + \binom{5}{2}\binom{7}{5}$$
Yikes ... the only way I can think of is just brute force, where I have 11 major cases (from 10 to 0 black cards), and then subcases for each:
$8A$. Same color for $2$ remaining cards ($5$ options)
$8B$. Different colors for those $2$ cards (so pick $2$ out of $5$ colors: ${5 \choose 2}=10$.
So total of $15$ options with $8$ black cards
$7A$: all $3$ remaining cards are same color ($5$ options)
$7B$: $2$ out of remaining $3$ have same color, last one is different: $5*4=20$ options ($5$ options for the color of the $2$ cards, and $4$ choices for the last card)
$7C$: all $3$ remaining cards are different color: ${5 \choose 3}=10$ options
Total for case $7$: $35$ options
$6A$: $3$ same color, $1$ different: $20$ options
$6B$: $2$ same color + $2$ same color: $10$ options
$6C$: $2+1+1$: $5*{4 \choose 2}=5*6=30$ options
$6D$: $1+1+1+1$: ${5 \choose 4}=5$ options
Total of $65$ options for case $6$
$5A$: $3+2:20$
$5B$: $3+1+1:30$
$5C$: $2+2+1:30$
$5D$: $2+1+1+1:5*{4\choose3}=5*4=20$
$5E$: $1+1+1+1+1$: $1$ option
Total for $5$: $101$
etc ... ( I think you will now have seen how to handle the different cases)
EDIT
Alright .. I hate leaving answer incomplete ...
$4A$: $3+3:10$
$4B$: $3+2+1: 5*4*3=60$
$4C$: $3+1+1+1: 5*4=20$
$4D$: $2+2+2: 10$
$4E$: $2+2+1+1: 10*3=30$
$4F$: $2+1+1+1+1: 5$
Total: $135$
$3A$: $3+3+1: 10*3=30$
$3B$: $3+2+2: 5*6=30$
$3C$: $3+2+1+1: 5*4*3=60$
$3D$: $3+1+1+1+1: 5$
$3E$: $2+2+2+1: 10*2=20$
$3F$: $2+2+1+1+1:10$
Total: $155$
$2A$: $3+3+2: 10*3=30$
$2B$: $3+3+1+1: 10*3=30$
$2C$: $3+2+2+1: 5*6*2=60$
$2D$: $3+2+1+1+1: 5*4=20$
$2E$: $2+2+2+2: 5$
$2F$: $2+2+2+1+1:10$
Total: $155$
$1A$: $3+3+3: 10$
$1B$: $3+3+2+1: 10*3*2=60$
$1C$: $3+3+1+1+1: 10$
$1D$: $3+2+2+2: 5*4=20$
$1E$: $3+2+2+1+1: 5*6=30$
$1F$: $2+2+2+2+1: 5$
Total: $135$
$0A$: $3+3+3+1: 10*2=20$
$0B$: $3+3+2+2: 10*3=30$
$0C$: $3+3+2+1+1: 10*3=30$
$0D$: $3+2+2+2+1: 5*4=20$
$0E$: $2+2+2+2+2: 1$
Total: $101$
Total total: $1+5+15+35+65+101+135+155+155+135+101=\boxed{903}$ possible hands
