Since $\frac{1}{2}\hbar \omega$ is the zero point energy of the ground state of the harmonic oscillator, then there is no way to extract this energy.
Therefore, in what way is this value different from zero? Is not it just about where we decide to choose the reference point?
You are correct in that for any given harmonic oscillator we can define the zero of the energy so that the ground state has zero energy. However, there are two things to point out.
Coming from a classical perspective, it's still a curiosity: The harmonic potential itself, $\frac{1}{2} m \omega^2 x^2$ has $0$ as its minimum, and the ground state energy lies exactly $\frac{1}{2}\hbar \omega$ above that minimum. So, even if we shift the energy around to have the ground-state energy be $0$, then the potential minimum is at $-\frac{1}{2}\hbar \omega$, i.e. there still is a finite difference between the energy of the ground state and the minimum of the potential.
Different oscillators have different $\omega$ and thus different zero point energy, so we can't choose a reference point where they all have zero ground state energy. Or maybe in your system, $\omega$ isn't even a constant but depends on other parameters of your system. This is the handwaving explanation behind the Casimir effect: Bringing two metal plates closer together reduces the zero point energy and, as a consequence, those two metal plates experience an attractive force.
