Whatever point the limit is they were at it by the mid 1940s, and went beyond it with lighter, more powerful jets. Make no mistake about it though, these aircraft had to flown extremely
carefully at low speeds lest the engine/propeller torque become too strong to control. Jets greatly lessened these issues and offered even higher speeds.
But you may find more happiness in discovering how little power you need to fly. Try building gliders and then incrementally add power. You will see there is no point in going anywhere near the "limit" in recreational flying. You may have an appetite for higher performance for aerobatics or greater speed, but it comes at a cost in weight, fuel consumption, complexity, and risk.
It takes 4x more thrust for a plane to go from 100 to 200mph, 9x more for 300mph, and 16x more for 400mph. What are your goals? This is a ladder that can only be safely climbed from the bottom to the top. But throwing a huge engine on an airframe, as fantastic a garage project as it may seem, would probably be making life a little shorter.
All that said, assuming CG is worked out with counter weights, new engine mount, moving wing, etc., your airframe stress limits are already there in your POH as neg and pos Gs, and Vne. A larger engine makes it easier to reach Vne, but you can do it with your original in a dive. But with your heavier re-make, stall and maneuvering speeds will have to be carefully checked, as well as rudder authority in low speed/high power (TakeOff/GoAround) situations. Your new plane will have much better climb performance.
So, this may work out after all by taking a 170/172 engine and putting it on the 150. I would start by doing some research (so lucky to have the internet). The Screamin' Sasquatch might give you some ideas.