The ions in the salt bridge never leave the salt bridge.
That's not true. The ions literally have to move to balance the charge of the gain / loss of the electrons at the electrodes. (You even allude to this in your question.) One of the ways that a battery can run out is that the bridge is used up. You could views this by using a deeply colored ions (e.g. MnO4-) to see the migration.
While the salt bridge is in place, there is no build up of charge. Electrons moving through the wires are offset by the ions moving through the salt bridge and into the respective half-cell. Thus, if you remove the salt bridge, there is no net charge that will cause anything to reverse; there is no 'force' that will cause ions to revert.
If you want to reverse the reaction (and charge the battery), you have to force electricity back in the other direction, using more voltage than what you got out of the battery. This doesn't work for all batteries. (I.e. this is why only some batteries are rechargeable). As you force electricity backwards through a battery, any movement of ions from the salt bridge means that the half-cell might now be contaminated with chemicals other than the ones you want in your half-reaction (unless you designed your salt bridge just right). Thus, the "right" ions may not go back to being uncharged (or vice versa).