Nick Mackinnon gives what appears to be the story in an article that appeared in the Mathematical Gazette in March 1992 (Vol. 76, No. 475), entitled "Newton's Teaser." He writes that Newton conjured up the poser, in fits and starts, in response to Leibniz's series
$$
1 - \frac13 + \frac15 - \frac17 + \cdots = \frac\pi4
$$
The other answers here are not off the mark, really. Newton was able to evaluate—determine areas for—among other things, expressions of the form
$$
\int \frac{dx^{\eta-1}}{e+fx^\eta+gx^{2\eta}}
$$
(Actually, he used $z$ instead of $x$, but I'll use the more usual $x$, because that's what Mackinnon does in most of his exposition other than direct cites of Newton.) He pointed out that by letting $\eta = 1$, $e = g = 1$, and $f = 0$, then the result can be used to evaluate Leibniz's series, and he further suggested that setting $\eta = 1$, $e = g = 1$, and $f^2 = 2eg$ (i.e., $f = \sqrt2$) enables the evaluation of the series in question:
$$
1 + \frac13 - \frac15 - \frac17 + \frac19 + \frac{1}{11} - \cdots
$$
Following Newton's suggestion, we have
\begin{align}
\int_{x=-1}^1 \frac{dx}{1+\sqrt2x+x^2}
& = \int_{x=-1}^1 \frac{dx}{\left(x+\frac{1}{\sqrt2}\right)^2
+\left(\frac{1}{\sqrt2}\right)^2} \\
& = \left. \sqrt2 \arctan \left(
\frac{x+\frac{1}{\sqrt2}}{\sqrt2}\right) \right]_{x=-1}^1 \\
& = \sqrt2 \left[ \arctan \left( \frac12 + \frac{1}{\sqrt2} \right)
- \arctan \left( \frac12 - \frac{1}{\sqrt2} \right) \right] \\
& = \sqrt2 \left( \frac{3\pi}{8} + \frac\pi8 \right) \\
& = \frac{\pi}{\sqrt2}
\end{align}
Newton apparently recorded in his worksheets the factorization
$$
1+x^4 = (1+\sqrt2x+x^2)(1-\sqrt2x+x^2)
$$
so evidently he means to evaluate the integral alternatively as
\begin{align}
\require{cancel}
\int_{x=-1}^1 \frac{dx}{1+\sqrt2x+x^2}
& = \int_{x=-1}^1 \frac{1-\sqrt2x+x^2}{1+x^4} \, dx \\
& = \int_{x=-1}^1 \frac{1+x^2}{1+x^4} \, dx
- \cancel{\int_{x=-1}^1 \frac{\sqrt2x}{1+x^4} \, dx}
\qquad \text{because $\sqrt2x$ is odd} \\
& = 2\int_{x=0}^1 \frac{1+x^2}{1+x^4} \, dx \qquad \text{because this is even}
\end{align}
Combining these gives us
\begin{align}
\frac{\pi}{2\sqrt2}
& = \int_{x=0}^1 \frac{1+x^2}{1+x^4} \, dx \\
& = \int_{x=0}^1 \frac{dx}{1+x^4} + \int_{x=0}^1 \frac{x^2\,dx}{1+x^4} \\
& = \int_{x=0}^1 1-x^4+x^8-x^{12}+\cdots \, dx
+ \int_{x=0}^1 x^2-x^6+x^{10}-x^{14}+\cdots \, dx \\
& = \left. x-\frac{x^5}{5}+\frac{x^9}{9}
-\frac{x^{13}}{13}+\cdots \right]_{x=0}^1
+ \left. \frac{x^3}{3}-\frac{x^7}{7}+\frac{x^{11}}{11}
-\frac{x^{15}}{15}+\cdots \right]_{x=0}^1 \\
& = 1+\frac13-\frac15-\frac17+\frac19+\frac{1}{11}-\frac{1}{13}-\frac{1}{15}
+ \cdots
\end{align}
Mackinnon adduces some circumstantial evidence that strongly suggests Leibniz never cracked Newton's little chestnut.