Without any restrictions, there are $\dbinom{16}{4}$ ways
$16$ ways where all four people are neighbors
$16\dbinom{11}{1}$ ways where exactly three people are neighbors
Now for the case when two people are neighbors, no. of ways would be
$$16\dbinom{12}{2}-\frac{16*11}{2}$$
$16\dbinom{12}{2}$ might look as the number of ways in which $2$ people are also sitting together. But we are counting twice the cases in which $2$ people are neighbors and the other $2$ are also sitting together.
So, first we have to count the number of cases in which $2$ pairs of people are sitting together.
There would be $16$ ways of selecting $2$ people sitting together.
Now for the other two we are left with $12$ people to choose from. There would be $11$ ways to select other $2$ neighboring people. Right now we have $16*11$ but we double counted each pair of neighbors. Ask me in the comments if you didn't get this part.
So, it would be $16*11/2$.
Thus the answer is
$$\dbinom{16}{4}-16-\left(16\dbinom{12}{2}-\frac{16*11}{2}\right)-16\dbinom{11}{1}=660$$