The following command will list all packages which have ever been removed (or purged), as far back as apt's history allows:
zgrep -E '^(Remove:|Purge)' /var/log/apt/history.log*
This does not distinguish automatic removals from explicit removals, but with a little work you should be able to reconstruct that information.
If you always use the command line, then
zgrep -E ^Commandline /var/log/apt/history.log*
will show you all the commands you issued, which will include explicit removals. You could filter it to only list remove or purge, but that won't include other forms of removals.
If you remove packages using dpkg, that won't appear in apt's logs; you'd need to look at /var/log/dpkg.log and search for remove or purge.
Basically you can try searching various logs of apt and see if there is anything you can fit into some pattern.
– MatthewRock Aug 25 '15 at 10:42