Is the ultimate reason that the neutron star loses angular momentum that the magnetic-dipole radiation is not spherically symmetric and photons have momentum?
Just for the loss of angular momentum, particles emitted from the neutron star surface are not necessary, right?
Radiation carries a momentum per unit volume given by $$ \vec{p} = \epsilon_0 \vec{E} \times \vec{B}$$
It therefore carries an angular momentum of $$ \vec{L} = \vec{r}\times \vec{p} = \epsilon_0 \vec{r} \times (\vec{E}\times \vec{B})$$
For plane-polarised transverse electromagnetic waves propagating in the $\hat{r}$ direction, then $\vec{L}=0$. However, if the axis of a magnetic dipole is inclined to the rotation axis then the radiation is not a single pure transverse wave; the dipole moment is the sum of two oscillating magnetic dipole moments that are perpendicular and oscillate $\pi/2$ out of phase. These produce two transverse waves, with arbitrary amplitudes, perpendicular polarisations and a $\pi/2$ phase shift between them - a.k.a. elliptically polarised light. Elliptically polarised light carries angular momentum.