Resolution of the mesh vs Resolution of the simulation
There are two possible mode to take advance of the Ocean modifier.
As stated by the Blender Manual's page:
Generate
Creates a tiled mesh grid that exactly corresponds with the resolution of the simulation data
or
Displace
Uses the existing geometry rather than replacing it. Vertices are displaced along the local Z-axis.
Basically, with Generated option enable a new mesh is generated. It resolution is the same of the simulation, and what it is inheriting from the starting objects are just transformation properties (location, rotation, scale, keyframes... and things like that). The geometry of the object doesn't matter

Switching to Displace option, means you are asking Blender to move the vertices of the mesh on the local Z axis (so keep an eye on the mesh rotation) according to a simulation.
If your starting geometry is made up of a single plane, increasing the resolution of the simulation won't result in a more detailed output, as it can be express only with already existing vertices only (4).

You'll have to increase the mesh-to-be-displaced too. You can use, for example, a subdivision surface modifier stacked above the Ocean modifier:
