Here is the shape I want to achieve, sharp square hole in the cylinder:
I did inset, extrude and beveled edges to keep square form.
The problem is when I am applying subsurf modifier, it distorts hole edges with weird overhangs:
Here is my blender file:
And here is modifier settings:
The issue is cause by the beveling of the edges that you did. When you applied the bevel operation, a small triangle was created in each corner.
Triangles are notorious for having issues like this when being subdivided. In this case, it may be difficult (although not impossible) to remove the triangle completely though. If you restructure the mesh so that the triangle is located on a section that isn't a corner the smoothing will look much nicer. Here is a way you could do that.
Here is a comparison between the corners when subdivided:
As said by others, those corner triangles (probably bevel applied) are involved in this (edge split is not),
I tred adding some more loopcut, eg. like
with subsurf:
in object mode:
compared to your squared hole:
under another angle
here is visible the full cylinder, without subsurf
with subsurf (note that I also removed sharp edges but added loops at top and bottom)
and in object mode (here I disabled edge split: top/bottom loops gives the edge more definition even with the subsurf)
here a test file
