One way would be to give the angle with the horizontal, instead of providing the 5mm length.

The problem in this case is that the angle in most cases will not be a round number. However, you might be able to use a function in the dimension (e.g. atan(3/5)).
UPDATE: to explain why its fully constrained
One thing to remember, is that parametric modelers try to solve the constraints. So they try to convert them in mathematical formulas. So in this particular scenario, one of the points that define the vertical blue line is part of the circle, therefore it needs to satisfy an equation of the following form:
$$(x-x_o)^2+ (y-y_o)^2 = r^2$$
where:
- $(x_0, y_0)$ the coordinates of the center
- $r$ the radius
The reason why you have the problem is that at some point you have an equation of the form $x^2 = 25$. For that equation there are two solutions $x_1 = -5$, and $x_2 =5$, which are both equally valid.
If you notice with respect to the center of the circle you could say that the first solution you are presenting is for the negative solution ($x=-5$), while the second solution is $x=+5$.
The solution which will be picked is based on the implementation of the algorithm, but usually, it does a rounding to the closest value.