If $\frac{a^{2}-1}{b+1}+\frac{b^{2}-1}{a+1}$ is an integer, prove that also $\frac{a^{2}-1}{b+1}$ and $\frac{b^{2}-1}{a+1}$ are integers.
By doing the math, I get $\frac{(a-1)(a+1)^{2}+(b-1)(b+1)^{2}}{(a+1)(b+1)}$ is an integer which means that $(a+1)(b+1)$ divides $(a-1)(a+1)^{2}+(b-1)(b+1)^{2}$.
but I don't know how to continue.
By the way, this is not homework or anything. I just found it in a book.
The problem states that, for some natural numbers $a$ and $b$, there's an integer $i$ where
$$\frac{a^2 - 1}{b + 1} + \frac{b^2 - 1}{a + 1} = i \tag{1}\label{eq1A}$$
If the first term is an integer, say $\frac{a^2 - 1}{b + 1} = j$, then $\frac{b^2 - 1}{a + 1} = i - j$ is also an integer. Doing the same thing with the second term instead shows that if either term is an integer, the other one is as well.
Assume neither left side term in \eqref{eq1A} is an integer. As you did, combining the terms using a common denominator gives
$$\frac{(a - 1)(a + 1)^{2} + (b - 1)(b + 1)^{2}}{(a + 1)(b + 1)} = i \tag{2}\label{eq2A}$$
Since $a + 1 \mid (a - 1)(a + 1)^{2}$, this means $a + 1 \mid (b - 1)(b + 1)^{2} = (b + 1)(b^2 - 1)$. However, since it's assumed $a + 1 \not\mid b^2 - 1$, this means there's a prime $p$ where, using the $p$-adic order function, we have
$$\nu_{p}(a + 1) \gt \nu_{p}(b^2 - 1) \tag{3}\label{eq3A}$$
but
$$\nu_{p}(a + 1) \le \nu_{p}((b + 1)(b^2 - 1)) \tag{4}\label{eq4A}$$
This means $p \mid b + 1$. Define
$$\nu_{p}(a + 1) = k_1 \tag{5}\label{eq5A}$$
$$\nu_{p}(b + 1) = k_2 \tag{6}\label{eq6A}$$
If $p \neq 2$, then $p \not\mid b - 1$ and $p \not\mid a - 1$. Thus, \eqref{eq3A} gives
$$k_1 \gt k_2 \tag{7}\label{eq7A}$$
Since $\nu_{p}((a - 1)(a + 1)^2) = 2k_1$ and $\nu_{p}((b - 1)(b + 1)^2) = 2k_2$, this means with the numerator of \eqref{eq2A},
$$\nu_{p}((a - 1)(a + 1)^{2} + (b - 1)(b + 1)^{2}) = 2k_2 \tag{8}\label{eq8A}$$
However, with the denominator,
$$\nu_{p}((a + 1)(b + 1)) = k_1 + k_2 \gt 2k_2 \tag{9}\label{eq9A}$$
Thus, the fraction can't be an integer. Since it is an integer, this means the assumption $p \neq 2$ must be incorrect, i.e., $p = 2$. Thus, $a$ and $b$ are odd, with $p \mid a - 1$ and $p \mid b - 1$. Define
$$\nu_{p}(a - 1) = m_1 \tag{10}\label{eq10A}$$
$$\nu_{p}(b - 1) = m_2 \tag{11}\label{eq11A}$$
Using \eqref{eq6A} and \eqref{eq11A} in \eqref{eq3A} gives
$$k_1 \gt k_2 + m_2 \tag{12}\label{eq12A}$$
Note the same arguments as above could have been used to check $b + 1$ not dividing into $a^2 - 1$, with it also reaching the conclusion the prime involved must be $2$. Thus, this would give \eqref{eq12A} but with the $1$ and $2$ indices switched around, i.e.,
$$k_2 \gt k_1 + m_1 \tag{13}\label{eq13A}$$
Combining \eqref{eq12A} and \eqref{eq13A} gives
$$k_1 \gt k_2 + m_2 \gt (k_1 + m_1) + m_2 \implies 0 \gt m_1 + m_2 \tag{14}\label{eq14A}$$
This is not possible since $m_1$ and $m_2$ must be positive integers. Thus, the original assumption of the two left terms in \eqref{eq1A} not being integers must be incorrect, i.e., they are both integers.
Call the fractions $r,s.\,$ Note $\,rs = (a\!-\!1)(b\!-\!1) := m\in \Bbb Z,\,$ so if $\,r\!+\!s = n\in\Bbb Z\,$ then $\,r,s\,$ are roots of $\, (x-r)(x-s)\, =\ \color{#c00}1\cdot x^2 - n\, x + m\,$ so $\,r,s\in \Bbb Z\,$ by Rational Root Test (i.e. if $\,c/d\,$ is a reduced rational root then $\,d\mid\color{#c00}1,\,$ so $\,d\!=\!1,\,$ so $\,c/d\in\Bbb Z)$.
Generally this shows $\,\bbox[5px,border:1px solid #c00]{\text{two rationals with integer sum & product must be integers}}$