Studying billiards is a difficult problem in general, even in pretty simple cases it has plenty of interesting properties.
I would like to understand what can be applications (mathematically or in concrete life or other sciences) of billiards. Are they used to model physical problems for instance?
I can only think of physics particles bumping into a compact box, is it relevant?
Absolutely they are used to model physical problems. They are almost a go-to model for any first-pass-investigation of classical (and now in some cases quantum) phenomena. The first reason for their popularity is that one can do away with complicated evolution laws and instead alter dynamical behaviours by changing the shape of the table. And there is indeed a wealth of different behaviours to be observed, many starring as the main actors in dynamical systems. The second reason is that despite simple evolution rules, they are not entirely trivial for the most part. It is not as though you can pick an arbitrary table geometry and produce detailed results about any old observable. This leaves plenty of room for theoretical physicists to focus on their physics, without the need to explain in-depth mathematical results which don't exist yet.
The opening chapter of Boltzmann's lectures on gas theory is a "mechanical analogy" based on hard spheres:
The simplest such picture is one in which the molecules are completely elastic, negligibly deformable spheres, and the wall of the container is a completely smooth and elastic surface. However, when it is convenient we can assume a different law of force. Such a law, provided it is agreement with the general mechanical principles, will be no more and no less justified than the original assumption of elastic spheres.
In less particular circumstances, there is often an equivalent representation for hard sphere models as regular billiards. In this direction is has to be said that statistical physics has benefited a great deal from billiard models.
No one is going to argue that perfectly elastic reflections or walls with infinite potential are real. But it is often easier to discover and develop new phenomena with pen, paper and computer than to conduct experiments. Once these phenomena can be established in real systems, the two realms (ideally) feed off each other. As for 1-1 applications (as opposed to models), I don't think so. Although it wouldn't surprise me if there were very accurate measurements; maybe optical wave guides or Knudsen environments. There are also a number of modifications which can be made to make things more 'realistic': external fields, non-specular reflections, time delayed reflections, ray splitting, finite sized (not necessarily spherical) billiard balls, ....
