That could perhaps be the distortion you ran into. I looked at your model and you shouldn't need to use shape keys or rotate the wings in edit mode. It comes with an armature that you can use to move and animate it. It looks a little complex, so take some time to play around with it, and do some reading on armatures if you need to.
Edit: I think your idea about vertices not being selected is most likely the answer. Editing for shapekeys is basically the same as ordinary editing, so I don't think the shapekeys themselves would cause that problem. (Artist put lock on hind wings-see last paragraph below.)
In general, though, shapekeys are not used for large rotations because they don't work well for that. The individual vertices move to their new positions by the shortest route, and the mesh can become quite distorted in the middle positions. I know - I tried to shapekey fingers once.
If you need something to rotate in a specific way and don't want to use an armature (I still haven't made peace with armatures) you can set the origin to the spot where you want it to pivot and then use rotation to move it. You can even set rotation locks so it will only move in one or two axes (click the locks in the Transform/Rotation menu in 3D view - if you can't see it try pressing N).
Change pivot or local origin of an object
In trying to pivot the wings just now, it seems the artist has tethered the bottom of the hind wing to the body - not sure how he did that, it would be interesting to find out. That could be the distortion you ran into. Apparently this is how the butterfly actually works, but you have to rotate it on 2 axes to get it to look right. You can read about it in the comments here: http://blog.hawkbats.com/2011/03/07/3d-commission-models-and-art/