Lets assume a normal person who does not have any technique applies chi
This is a poor assumption. I am not aware of any martial system that believes you can "apply chi" to fighting without first having learned fighting techniques.
There are three stages of practice common to at least to xingyi and bagua. The first stage, ming jing, requires you to learn in an obvious, easily observable way. After you learn this, the next stage is an jing, where you learn to "hide" force, applying the same principles as the obvious version in smaller space. The one-inch punch is an example of this.
Breaking the bottom brick in a stack of bricks is still a matter of physics. You have to apply force differently from a sledgehammer; you can't just swing hard with all of your mass. You can, however, alter your application of force over time. Think of the difference between how a whip and a sledgehammer apply force.
Don't equate parlor tricks with fighting capability. Being able to break bricks with your hands or chest does not necessarily mean that you can fight an opponent who moves and does not attack you like a brick.
Even if it's all just physics, you still have to learn to skillfully apply force from a human body. The traditional roadmap for learning this involves "qi feeling", which is rather controversial on this site. How do you know whether an exercise is doing anything? Partly because you feel something different, either during or after the exercise. This among other things, is traditionally described in terms of qi.
I would start to characterize the fighting aspect of taiji, xingyi, and bagua in this way: it is more important to have complete control of your body and especially the nervous system than it is to move massive weights. You do get old people who are "weaker" who can best young people who can move heavier weights. But these people do not think of themselves as the ones who are weaker; they have spent a long time developing parts of the body you probably have not considered. These people also still think of their younger, more muscular selves as better fighters. Does also being able to move massive weights help? Yes.