It mostly depends on the type of garage door (composite slider, thin aluminium, or solid wood flip up etc... ) but generally the automated motors connected to your door are extremely tough at preventing a forced opening (i.e by hand)
The problem comes where most garage openers have a quick release coupler which is made from plastic (i.e it can get brittle). I have seen it that my door can release the coupler with enough force from the outside. So you would need to assess your garage door first.
The other issue is most garage doors can be set to back up in the event of a collision, so if there is something in the way, you may drive off and have it open up again without you noticing.
Another issue is signal mirroring on your remote, Depending on the remote, some older types can be copied fairly easily (this is extremely rare)
With that said.. (in the crime ridden area where I live) many hi-jackings occur when you are getting out of the car, If someone does attempt to follow you into your house, its always a good idea to have a slam-lock style gate from the garage to the house so that you can escape into the safety of the house.