Compute following: $$ \sum_m\binom{n}{m}\sum_k\frac{\binom{a+bk}{m}\binom{k-n-1}{k}}{a+bk+1} $$ Only consider real numbers a, b such that the denominators are never 0.
Now I simplify it into $$ -\frac{1}{n}\sum_k\binom{n}{k}\binom{-n}{a+bk+1}(-1)^{a+bk+k} $$ I have problem with this question in which I can't eliminate coefficient b.
But I can't find any formula to get answer.Please help me!
Here is a slightly different variation of the theme. It is convenient to use the coefficient of operator $[z^n]$ to denote the coefficient of $z^n$ of a series. This way we can write e.g. \begin{align*} [z^k](1+z)^n=\binom{n}{k} \end{align*}
We obtain \begin{align*} \sum_{m=0}^{n}&\binom{n}{m}\sum_{k=0}^{n}\frac{1}{a+bk+1}\binom{a+bk}{m}\binom{k-n-1}{k}\tag{1}\\ &=\sum_{k=0}^{n}\frac{(-1)^k}{a+bk+1}\binom{n}{k}\sum_{m=0}^{n}\binom{n}{m}\binom{a+bk}{m}\tag{2}\\ &=\sum_{k=0}^{n}\frac{(-1)^k}{a+bk+1}\binom{n}{k}\binom{a+bk+n}{n}\tag{3}\\ &=\frac{1}{n}\sum_{k=0}^{\infty}(-1)^k\binom{n}{k}\binom{a+bk+n}{n-1}\tag{4}\\ &=\frac{1}{n}\sum_{k=0}^{\infty}(-1)^k[z^{k}](1+z)^{n}[u^{n-1}](1+u)^{a+bk+n}\tag{5}\\ &=\frac{1}{n}[u^{n-1}](1+u)^{a+n}\sum_{k=0}^\infty(-1)^k(1+u)^{bk}[z^k](1+z)^n\tag{6}\\ &=\frac{1}{n}[u^{n-1}](1+u)^{a+n}(1-(1+u)^b)^n\tag{7}\\ &=\frac{(-1)^n}{n}[u^{n-1}](1+u)^{a+n}\left(\sum_{j=1}^\infty\binom{b}{j}u^j\right)^n\tag{8}\\ &=0 \end{align*}
Comment:
In (1) we write lower and upper limits of the sum.
In (2) we exchange the sums, do small rearrangements and use the identity \begin{align*} \binom{k-n-1}{k}=(-1)^k\binom{n}{k} \end{align*}
In (3) we apply Vandermonde's identity. With $q:= a+bk$ we get \begin{align*} \sum_{m=0}^{n}\binom{n}{m}\binom{q}{m}=\sum_{m=0}^{n}\binom{q}{m}\binom{n}{n-m}=\binom{q+n}{n} \end{align*}
In (4) we use the identity \begin{align*} \frac{q}{p-q+1}\binom{p}{q}=\binom{p}{q-1}\\ \end{align*} and we also change the uppper limit of the series to $\infty$ without changing anything since we add only zeros.
In (5) we apply the coefficient of operator twice
In (6) we use the linearity of the coefficient of operator and do some rearrangements
In (7) we apply the substitution rule of the coefficient of operator \begin{align*} A(u)=\sum_{k=0}^\infty a_ku^k=\sum_{k=0}^\infty u^k[z^k]A(z) \end{align*}
In (8) we use the binomial series expansion and observe the smallest power of $u$ is $n$ so that the coefficient $[u^{n-1}]$ is zero.
Suppose we seek to evaluate
$$\sum_{m=0}^n {n\choose m} \sum_{k=0}^{n+1} \frac{1}{a+bk+1} {a+bk\choose m} {k-n-1\choose k}.$$
Now we have
$${a+bk\choose m} = \sum_{q=0}^m (-1)^{m-q} {a+bk+1\choose q} \\ = (-1)^m + \sum_{q=1}^m (-1)^{m-q} {a+bk+1\choose q}$$
and hence
$$\frac{1}{a+bk+1} {a+bk\choose m} = \frac{(-1)^m}{a+bk+1} + \sum_{q=1}^m \frac{1}{q} (-1)^{m-q} {a+bk\choose q-1}.$$
Now from the first component we get in the main sum
$$\sum_{m=0}^n {n\choose m} \sum_{k=0}^{n+1} \frac{(-1)^m}{a+bk+1} {k-n-1\choose k} \\ = \sum_{k=0}^{n+1} \frac{1}{a+bk+1} {k-n-1\choose k} \sum_{m=0}^n {n\choose m} (-1)^m = 0.$$
We are thus left with the following sum:
$$\sum_{k=0}^{n+1} {k-n-1\choose k} \sum_{m=0}^n {n\choose m} \sum_{q=1}^m \frac{1}{q} (-1)^{m-q} {a+bk\choose q-1}.$$
Working with the inner sum we obtain
$$\sum_{m=1}^n {n\choose m} \sum_{q=1}^m \frac{1}{q} (-1)^{m-q} {a+bk\choose q-1} \\ = \sum_{q=1}^n \frac{(-1)^q}{q} {a+bk\choose q-1} \sum_{m=q}^n {n\choose m} (-1)^m \\ = \sum_{q=1}^n {n-1\choose q-1} \frac{1}{q} {a+bk\choose q-1} \\ = \frac{1}{n} \sum_{q=1}^n {n\choose q} {a+bk\choose q-1}.$$
Now put
$${a+bk\choose q-1} = {a+bk\choose a+bk-q+1} = \frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{1}{z^{a+bk-q+2}} (1+z)^{a+bk} \; dz$$
to get
$$\frac{1}{n} \frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{1}{z^{a+bk+2}} (1+z)^{a+bk} \sum_{q=1}^n {n\choose q} z^q \; dz \\ = \frac{1}{n} \frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{1}{z^{a+bk+2}} (1+z)^{a+bk} (-1 + (1+z)^n)\; dz $$
The inner constant term does not contribute and we are left with
$$\frac{1}{n} \frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{(1+z)^{a+bk+n}}{z^{a+bk+2}} \; dz = \frac{1}{n}{a+bk+n\choose a+bk+1} = \frac{1}{n}{a+bk+n\choose n-1}.$$
Returning to the main sum we thus have
$$\frac{1}{n} \sum_{k=0}^{n+1} {k-n-1\choose k} {a+bk+n\choose n-1} \\ = \frac{1}{n} \sum_{k=0}^{n+1} {-k\choose n+1-k} {a+b(n+1)+n-bk\choose n-1}.$$
Note that $${-k\choose n+1-k} = \frac{1}{(n+1-k)!} \prod_{q=0}^{n-k} (-k-q) = \frac{(-1)^{n-k+1}}{(n+1-k)!} \prod_{q=0}^{n-k} (k+q) \\ = \frac{(-1)^{n-k+1}}{(n+1-k)!} \frac{n!}{(k-1)!} = (-1)^{n-k+1} {n\choose k-1}.$$
This means for the main sum
$$\frac{(-1)^{n+1}}{n} \sum_{k=1}^{n+1} {n\choose k-1} (-1)^k {a+b(n+1)+n-bk\choose n-1} \\ = \frac{(-1)^{n}}{n} \sum_{k=0}^{n} {n\choose k} (-1)^k {a+bn+n-bk\choose n-1}.$$
Introduce $${a+bn+n-bk\choose n-1} = \frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{1}{z^n} (1+z)^{a+bn+n-bk} \; dz$$
We get for the sum
$$\frac{(-1)^{n}}{n} \frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{1}{z^n} (1+z)^{a+bn+n} \sum_{k=0}^n {n\choose k} (-1)^k \frac{1}{(1+z)^{bk}} \; dz \\ = \frac{(-1)^{n}}{n} \frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{1}{z^n} (1+z)^{a+bn+n} \left(1-\frac{1}{(1+z)^b}\right)^n \; dz \\ = \frac{(-1)^{n}}{n} \frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{1}{z^n} (1+z)^{a+bn+n} \frac{((1+z)^b-1)^n}{(1+z)^{bn}} \; dz \\ = \frac{(-1)^{n}}{n} \frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{1}{z^n} (1+z)^{a+n} ((1+z)^b-1)^n\; dz.$$
This is $$\frac{(-1)^{n}}{n} [z^{n-1}] (1+z)^{a+n} ((1+z)^b-1)^n.$$
Note however that $$((1+z)^b-1)^n = \left({b\choose 1}z + {b\choose 2}z^2 + \cdots\right)^n = b^n z^n + \cdots$$
so there is no coefficient on $[z^{n-1}]$ because the powered term starts at $z^n.$ Therefore the end result of the whole calculation is
$$\Large\color{#0A0}{0}.$$
Remark. We have made several uses of
$${n\choose m} = \sum_{q=0}^m (-1)^{m-q} {n+1\choose q}.$$
If this is not considered obvious we can prove it with the integral
$${n+1\choose q} = \frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{1}{z^{q+1}} (1+z)^{n+1} \;dz$$
to get $$\frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{1}{z} (1+z)^{n+1} \sum_{q=0}^m (-1)^{m-q} \frac{1}{z^q}\;dz \\ = \frac{1}{2\pi i} \int_{|z|=\epsilon} \frac{(-1)^m}{z} (1+z)^{n+1} \frac{1-(-1/z)^{m+1}}{1+1/z} \\ = \frac{1}{2\pi i} \int_{|z|=\epsilon} (-1)^m (1+z)^{n+1} \frac{1-(-1/z)^{m+1}}{1+z} \; dz \\ = \frac{1}{2\pi i} \int_{|z|=\epsilon} (-1)^m (1+z)^{n} (1-(-1/z)^{m+1}) \; dz \\ = - (-1)^m \times (-1)^{m+1} {n\choose m} = {n\choose m}.$$
