Set the handle type to Aligned
I should have read the question better, re subdividing, to have a new point remain on the circle to change the handle type.

After adding the Bézier circle, select all in edit mode and Control Points → Set Handle Type → Aligned (Or only the two handles being sub'd.) The default Bézier curve has handle type 'AUTO' (look for a link to the handle type.)
After which the result of subdividing produces handles on the tangent.

The active object is the default Bézier circle section subdivide, over the result above.
For a circle, after adding a point as above, can then "point slide" it in its two original handle quadrants by rotating about the circle center pivot point and adjust the handle points to make a sine, cosine ratio.
The left-to-right handle length will be constant. The alignment after rotate will be correct. To keep as a circle is a matter of sliding handles. For example, it would be all horizontal right handle if rotated back to the top, and all vertical left handle clockwise to right original point...
It is somewhat reminiscent of the position in a slot of
How to do Trammel Of Archimedes Animation?
About to add the handle fix to How to rotate bezier control point (viewport not updating) ...
However, for the answer to the question marked as a dupe has gone above and beyond - What is the fastest way to add vertices to a curve at a specific point?