There is almost certainly a better way of doing this, probably through limits.conf but I don't know it so here's a dirty hack.
This command will kill all processes owned by the user terdon that are using more than 3G of RAM:
ps -u terdon -o vsize= -o pid= |
while read mem pid; do [ $mem -gt 3145728 ] && kill $pid;
done
The ps -u terdon -o vsize= -o pid= command will print the PID and memory used (1024-byte units) by each of the processes owned by terdon. Obviously, you should adapt this to use your friend's user name. The output of the ps command is then passed through a while loop that reads the memory usage and pid and then kills the PID if it is using more than 3145728K of memory.
If you now add this as a crontab run by root, it will be run every minute and kill those processes that are using more RAM than you want to allow. So, add this line to /etc/crontab:
* * * * * root ps -u terdon -o vsize= -o pid= | while read mem pid; do [ $mem -gt 3145728 ] && kill $pid; done
As I said, this is inelegant and just an ugly hack but it might be enough.
ulimit. There is already a post about that here on this site. – Hennes Apr 09 '14 at 18:05