In fact, it is not convergent: the horizontal component of the gravitational force can depend arbitrarily on parts of the sheet which are far away, if you are sufficiently malicious in your choice of surfaces which exhaust the infinite plane.
To make this explicit, let's suppose we are suspended at the cartesian point $(0,0,D)$, and place some mass of surface density $\sigma$ on the plane $z=0$, within a distance $\rho_1$ of the origin on the half-plane $y \leq 0$, and within a distance $\rho_2$ of the origin on the half-plane $y \geq 0$. We assume that $D>0$ and $\rho_2\geq \rho_1$, and look at what happens to the horizontal component of force when $\rho_1,\rho_2 \gg D$.
The first thing to notice is that the entire disc of radius $\rho_1$ is massive. So, by symmetry, its contribution to the horizontal force vanishes. That is, we need only consider the contribution of the "half-annulus"
$$
\{p: \rho_1 \leq |p| \leq \rho_2, y \geq 0\}
$$
Also by symmetry, the $x$ component of force vanishes, so we only need to compute its $y$ component. We work in polar coordinates. A small patch of mass at coordinates $(r,\theta)$ with area $r \, dr \, d\theta$ is at distance $\sqrt{r^2+D^2}$ from us, and has mass $\sigma r \, dr\, d\theta$. So the overall magnitude of the gravitational force vector will be
$$
|F|=\frac{G \sigma r \, dr\, d\theta}{r^2+D^2}
$$
The fraction of this which is directed horizontally is $\frac{r}{\sqrt{r^2+D^2}}$; the fraction of that which is directed in the $y$ direction is $\sin \theta$. So our small patch of mass contributes
$$
G \sigma \frac{r^2 \sin \theta \, dr\, d\theta}{(r^2+D^2)^{3/2}}
$$
to the $y$-component of the force. The entire $y$-component of the force is
then given by integrating over the half-annulus:
$$
F_y=G\sigma \int_{0}^{\pi} \int_{\rho_1}^{\rho_2} \frac{r^2 \sin \theta \, dr\, d\theta}{(r^2+D^2)^{3/2}}=2G\sigma\int_{\rho_1}^{\rho_2} \frac{r^2 \, dr}{(r^2+D^2)^{3/2}}
$$
If $\rho_1,\rho_2 \gg D$, then it follows that
$$
F_y \approx 2G\sigma \int_{\rho_1}^{\rho_2} \frac{dr}{r}
=2G\sigma \log\left(\frac{\rho_2}{\rho_1}\right)
$$
which could be arbitrarily large regardless of the size of $\rho_1$, unless we have some further bound on $\rho_2$.
How good a bound on $\rho_2$ do we need in order for the standard result to be a reasonable approximation? Say we want the horizontal component of force to be less than $\epsilon$ times as large as the vertical component. By the standard argument, the vertical component will be well-approximated by $2\pi G\sigma$, so we want
$$
2G\sigma \log\left(\frac{\rho_2}{\rho_1}\right) < \epsilon (2\pi G\sigma)
$$
from which it follows that
$$
\frac{\rho_2}{\rho_1} < e^{\pi \epsilon} \tag{*}
$$
For example, if we want to guarantee the horizontal component to be a full order of magnitude smaller than the vertical component, we take $\epsilon = 0.1$ and so must have $\frac{\rho_2}{\rho_1} < e^{0.1\pi} \approx 1.37$.
In principle this only applies to our specific example. But in fact our example is the worst-case scenario among all mass configurations where the nearest edge of the mass is at horizontal distance $\rho_1$ from us, and the entire mass is within horizontal distance $\rho_2$ of us. (Of all such configurations, it involves the largest possible contribution in the $y$ direction without any unnecessary cancellations.) So $(*)$ can be used as a general rule of thumb in all such situations.
