This theorem is Proposition LXXI, Theorem XXXI in the Principia. To warm up, consider the more straightforward proof of the preceding theorem, that there's no inverse-square force inside of a spherical shell:

The crux of the argument is that the triangles HPI and LPK are similar. The mass enclosed in the small-but-near patch of sphere HI goes like the square of the distance HP, while the mass enclosed in the large-but-far patch of sphere KL, with the same solid angle, goes like the square of the distance KP. This mass ratio cancels out the distance-squared ratio governing the strength of the force, and so the net force from those two patches vanishes.
For a point mass outside a shell,
Newton's approach
is essentially the same as the modern approach:

One integral is removed because we're considering a thin spherical shell rather than a solid sphere. The second integral, "as the semi-circle AKB revolves about the diameter AB," trivially turns Newton's infinitesimal arcs HI and KL into annuli.
The third integral is over all the annuli in the sphere, over $0\leq\phi\leq\tau/2$ or over $R-r\leq s\leq R+r$. This one is a little bit hairy, even with the advantage of modern notation.
Newton's clever trick is to consider the relationship between the force due to the smaller, nearer annulus HI and the larger, farther annulus KL defined by the same viewing angle (in modern notation, $d\theta$).
If I understand correctly
he argues again, based on lots of similar triangles with infinitesimal angles, that the
smaller-but-nearer annulus and the larger-but-farther annulus exert the same force at P.
Furthermore, he shows that the force doesn't depend on the distance PF, and thus doesn't depend on the radius of the sphere; the only parameter left is the distance PS (squared) between the particle and the sphere's center. Since the argument doesn't depend on the angle HPS, it's true for all the annuli, and the theorem is proved.