I think I accidentally pressed something to make it like this:

See how the XYZ arrows are at the 3D cursor instead of those vertexes I've selected?
How do I fix that? Whenever I select on a vertex or something, those blue, red, green arrows would always be where I selected, now it just keeps going to the 3D cursor.

.(period), which sets the pivot point to the 3D cursor. You can press,(comma) to set it to bounding box, andCtrl+,to set it to median (the default). – gandalf3 Apr 15 '14 at 19:10