Maybe there is a smarter solution, the only one I can think of is the following: create a new View Layer. In it you disable all collections so that there is nothing visible to render. Under View Layer Properties > Passes > Light > Other you enable Environment.

Let's say you now have one called ViewLayer with the render scene including the holdout object and ViewLayer_001 where all collections are disabled.
In the Compositor you need two Render Layers, one that is set to ViewLayer and the other is for the empty ViewLayer_001. After rendering you take an Alpha Over node and plug the Env output of ViewLayer_001 in the first input and the Image output of ViewLayer in the second input. This way you can combine the foreground with the background.
Unfortunately you have to use two layers. I thought I remembered it differently, like you could use the Env output direct with a single layer, but the background is black where the objects are so that doesn't work. Maybe someone has a better solution.
