Going down this group the proton affinity decreases, so basicity decreases:
$$\mathrm{PA}(\ce{X})=\mathrm{D}(\ce{X+-H})-\mathrm{IE}(\ce{X})$$
where $\mathrm{D}$ is the bond disassociation energy, and $\mathrm{IE}$ is the ionization energy. Down a column this expression is dominated by the $\mathrm{D}(\ce{X^+-H})$ term, and since $\ce{N}$ has the highest bond strength with hydrogen, it is the most basic.
The structural (or hybridization) way to look at it is that the lone pair in $\ce{NH3}$ is very, very different than the lone pair of a similar compound of $\ce{P}$, $\ce{As}$, etc. $\ce{NH3}$ is almost perfectly $\mathrm{sp^3}$ hybridized, but the others have little to no hybridization, so the lone pair is actually held in a orbital that has mainly $\mathrm{s}$ character. This makes the lone pair far less directional (the charge is distributed over a larger area), making it less reactive and harder to form bonds, so less basic.
Finally, $\ce{N}$ is the only one on the series where the HOMO (which contains the lone pair) is concentrated on the $\ce{N}$. For the rest of the series, the HOMO is very delocalized and significant electron density is held on the $\ce{H}$ atoms, so it will not react well as a base as well.