There's several things involved in your question.
The first is what would realistically happen if you set up this system and you managed to measure its temperature. Here there's no easy answer: you have to consider in detail how you set up the system and how it will interact with its surroundings. Simple idealizations like "isolated" aren't adequate here because the temperature you can measure in empty space is mostly due to the cosmic microwave background. This means that, for example, if you use box made of matter, microwaves would impinge on the outside and transfer heat so that in practice it's impossible to place a truly isolated box in empty space.
If you did manage to do it through magical unphysical means and you also made sure to remove all the radiation inside the box, then the temperature inside it would be zero due to the absence of any excited degrees of freedom. There would be quantum fluctuations, yes, such as the Casimir effect, but no thermal fluctuations.
The second one is whether or not it's possible to define temperature for a system with no matter in it. And the answer is yes. In quantum field theory we can tune the temperature of a system completely independently of the density. To make a system at finite temperature, we consider a theory in a Euclidean spacetime (that is, the distances inside it are given by $ds^2 = dx^2 + dy^2 + dz^2 + d\tau^2$, where $\tau$ is the "Euclidean time" rather than the usual $ds^2 = dt^2 - dx^2 - dy^2 - dz^2$) and make the time direction compact, that is, we make it finite and impose boundary conditions. Spacetime then becomes like an infinite cylinder whose circumference is $1 \over T$ in natural units.
To make a system have a finite density, we can tune the chemical potential. You can think of the chemical potential as a "bias" that allows you to tune the average particle number per unit volume, much like the temperature allows you to tune the average energy per unit volume. The way to actually implement this in field theories is a bit technical so I'll leave it out, but the gist of it is that it's perfectly reasonable to leave the chemical potential at zero and still have a system e.g. a gas at a finite temperature.
The interpretation here, considering QED at finite temperature would be that the average electron number is zero because you have a gas with equal numbers of electrons and positrons flying around, plus photons and whatever messy interactions they may have.
Perhaps this isn't exactly what you wanted, but to define a finite temperature you really need excitable degrees of freedom. At best, you have this zen like idea where equal numbers of electrons and positrons means the average electron number is zero, but this isn't to say that the box is filled with nothing. There's stuff in it.
Disclaimer: this is the particle physics perspective. Other branches of physics may have subtly different approaches to things which may lead to confusion. Just be aware.