A comment to this question sparked my interest. If we start a rocket from the surface at the equator, it has some angular momentum with regard to Earth. This angular momentum is definitely much too small to reach enough speed for lower Earth orbit or even geostationary orbit. However, sufficiently large orbits have arbitrarily low angular velocities. So, if the rocket did not turn and kept straight ahead¹, would it be possible to reach a stable (circular or elliptical) orbit?
1) The notion of "straight ahead" could mean either "keep pointing towards the same star" or "keep pointing the exhaust towards Earth's centre". An analysis of both possibilities would be welcome.