I have VirtualBox running a windows 7 VM on it and the performance is really choppy, likely because of the 3D acceleration.
However I read through VirtualBox's security page, and it states: "Enabling 3D graphics via the Guest Additions exposes the host to additional security risks".
Now I am wondering how likely it is for malware to exploit this security hole? and if it's something I should even be worrying about?
'How likely' is kind of a hard question to answer. But I am going to try and analyze the risk to give you a feel for the 'likelihood'.
The main problem here is that your host machine (clean) is giving to the guest (potentially infected) unsupervised, direct access to the hardware drivers, in this case OpenGL and Direct3d. This access is only as safe as the drivers the guest can access.
Unfortunately, those drivers are not always secure and bugs in drivers are found very often. This should probably help us estimate the 'likelihood' of such malware.
To complete your understanding, consider the market for virtual machine escaping malware. At the last Pwn2Own competition, VM escape exploits fetched $105,000 to the winners. And that's for a reason. Escaping from a VM means you can compromise the underlying host, which defeats the whole point of virtualization (at least for security reasons).
So I'd say it is very likely malware will target drivers that reside in the host.