This is because pole angles aren't always created properly in blender. There is a setting in the IK constraint that allows setting this pole angle manually, specifically for cases such as this. You can find that here:

This effect is common when building IK chains. You can see that with even a very simple chain here:

Realign the chain with the pole angle, like this:

In your case, it looks like your pole angle will be 90 or -90 degrees. There are numerous ways to get the correct angle for this setting, but the easiest is during chain creation. I always make my IK chains coplanar and always on the cardinal axes (global X, Y, Z) from an orthographic view. This keeps every bone on the same plane with proper roll angles and the pole angle will always be some multiple of 90 degrees.
There are more technical ways to find the pole angle, for example this script, but these are more targeted at technical artists.
You can find more information on this answer.