In regards to actively trying to control your eyes while they are closed, like attempting to do visualization like NLP, I believe that you choose what to focus on and where to focus at, exactly the same as my "pitch black" analogy whereby if you look into, say, a pitch black room, then although you know it's only 15 feet towards the wall, your vision does not.
Therefore, if you believe that it extends for millions of miles, then you can choose quite easily to focus into the darkness that way.
In regards to if you close your eyes and just kind of forget about them without going to sleep...like "resting your eyes", then I believe Chuck in another answer answered this best in his bottom two paragraphs. Quick link: https://cogsci.stackexchange.com/a/4467/3433
Now in regards to during REM sleep, this gets tricky.
I did not think I'd find a study that answers your exact question about "how is focal point determined during REM sleep?". I'll just start with that study first, then throw in a few others that are related to something called the "scanning hypothesis" which will help to unravel some of your other questions more specifically if they're about REM sleep specifically.
Let's start off with a definition of the "scanning hypothesis":
According to psychologydictionary.com,
The scanning hypothesis is a concept hypothesizing that rapid eye
movements identified in the course of the REM stage of sleep correlate
to subjective gaze adjustments of the dreamer casting their gaze about
in the dream with fixations in particular places.
At first I was worried that all the studies would just focus on this "scanning hypothesis," then I came across this gem:
Exp Brain Res. 1997 Oct;117(1):153-60. Binocular eye movements not
coordinated during REM sleep. Zhou W, King WM.
Department of Neurology, University of Mississippi Medical Center,
Jackson 39216, USA. wuz@vor.umsmed.edu
Abstract
Rapid eye movements (REMs) are a defining characteristic of REM sleep
during which vivid dreams occur. It has been suggested that REMs may
be binocularly coordinated and related to "watching" dream images. For
the first time, binocular eye movements were recorded during natural
REM sleep in monkeys to test the conjugate nature of the oculomotor
system and the "scanning hypothesis" of REMs during sleep. During REM
sleep, the lines of sight of the two eyes are frequently misaligned up
to 30 degrees horizontally and/or vertically. Since the lines of sight
usually don't intersect, there is no fixation point. Instead, each eye
is aimed at a different part of the visual field during REM sleep.
Furthermore, REMs are not usually conjugate, but are disjunctive or
even monocular in horizontal or vertical directions. These data argue
against the idea that REMs actually "track" dream images, unless each
eye is watching its own dream! Binocular misalignment and disjunctive
(even monocular) REMs during sleep suggest that separate left eye and
right eye pathways generate saccades in each eye and control the
position of each eye. Binocular coordination cannot be the passive
result of anatomical connectivity as has been argued previously, but
instead must result from a high-level process associated with the
awake state that coordinates activity in left-eye and right-eye
pathways. Hering's law of equal innervation is not consistent with
these data. PMID: 9386014 [PubMed - indexed for MEDLINE]
Apparently also, part of the "scanning hypothesis" is that your eyes are trying to track down something...in other words fixate upon it...which would cause them to focus. This results from this study say no to that.
So, as far as...
"By this I mean are the muscles moving in accordance with focusing.
How is the focal point determined?"
...during REM sleep, this study says that there is no fixation upon one thing and thus there is no focal point. So, no, according to this study the muscles are not moving in accordance with focusing.
As a precursor to this study, a definition of "rapid eye movement disorder" is good to know. So here is wikipedia's definition:
Rapid eye movement disorder is a sleep disorder (more specifically a parasomnia) that
involves abnormal behaviour during the sleep phase with rapid eye movement
(REM) sleep. It was first described in 1986. The major and arguably
only abnormal feature of RBD is loss of muscle atonia (paralysis)
during otherwise intact REM sleep. This is the stage of sleep in which
most vivid dreaming occurs. The loss of motor inhibition leads to a
wide spectrum of behavioural release during sleep. This extends from
simple limb twitches to more complex integrated movement, in which
sufferers appear to be unconsciously acting out their dreams. These
behaviours can be violent in nature and in some cases will result in
injury to either the patient or their bed partner.
It's important to note the difference between the first study which was done in 1997 and this one done in 2010. Perhaps the overall definition of the "scanning hypothesis" changed, or there is slight subjectivity to it.
For instance, in the first study the author says,
"It has been suggested that REMs may
be binocularly coordinated and related to "watching" dream images. For
the first time, binocular eye movements were recorded during natural
REM sleep in monkeys to test the conjugate nature of the oculomotor
system and the "scanning hypothesis" of REMs during sleep."
So he's trying to test the "scanning hypothesis" by looking to see if the monkeys actually do focus on something by looking to see if the eyes are working together (ie. trying to triangulate distance and thus focus) are actually working together.
Now these French people (they always seem to have creative ideas actually) admit that the "scanning hypothesis" has been testing in this way...as in...by monitoring the muscles in the eyes. So they're taking a different approach and (cleverly might I add) exploiting people who essentially have a disorder that is very similar to sleepwalking...but they lay in bed...THEN they are mixing in control patients who are normal so that they can compare the two and use that mixture to extrapolate it to the normal population as well & noone can criticize them for only using patients with that 1st specific disorder.
They're looking to see how their eyes align/what direction they move to when they are acting out certain motions.
Brain. 2010 Jun;133(Pt 6):1737-46. doi: 10.1093/brain/awq110. Epub
2010 May 16.
Do the eyes scan dream images during rapid eye movement sleep? Evidence from the rapid eye movement sleep behaviour disorder model.
Leclair-Visonneau L, Oudiette D, Gaymard B, Leu-Semenescu S, Arnulf I.
Source Sleep Disorder Unit, Pitié-Salpêtrière Hospital, Assistance
Publique - Hôpitaux de Paris, 75013 Paris, France.
Erratum in Brain. 2011 Jul;134(Pt 7):1286.
Abstract
Rapid eye movements and complex visual dreams are salient
features of human rapid eye movement sleep. However, it remains to be
elucidated whether the eyes scan dream images, despite studies that
have retrospectively compared the direction of rapid eye movements to
the dream recall recorded after having awakened the sleeper. We used
the model of rapid eye movement sleep behaviour disorder (when
patients enact their dreams by persistence of muscle tone) to
determine directly whether the eyes move in the same directions as the
head and limbs. In 56 patients with rapid eye movement sleep behaviour
disorder and 17 healthy matched controls, the eye movements were
monitored by electrooculography in four (right, left, up and down)
directions, calibrated with a target and synchronized with video and
sleep monitoring. The rapid eye movement sleep behaviour
disorder-associated behaviours occurred 2.1 times more frequently
during rapid eye movement sleep with than without rapid eye movements,
and more often during or after rapid eye movements than before. Rapid
eye movement density, index and complexity were similar in patients
with rapid eye movement sleep behaviour disorder and controls. When
rapid eye movements accompanied goal-oriented motor behaviour during
rapid eye movement sleep behaviour disorder (e.g. grabbing a fictive
object, hand greetings, climbing a ladder), which happened in 19
sequences, 82% were directed towards the action of the patient (same
plane and direction) (Taal Note: Note what they just said in parenthesis). When restricted to the determinant rapid eye
movements, the concordance increased to 90%. Rapid eye movements were
absent in 38-42% of behaviours. This directional coherence between
limbs, head and eye movements during rapid eye movement sleep
behaviour disorder suggests that, when present, rapid eye movements
imitate the scanning of the dream scene. Since the rapid eye movements
are similar in subjects with and without rapid eye movement sleep
behaviour disorder, this concordance can be extended to normal rapid
eye movement sleep.
So,
1) Although behavioral movements and what the eyes are doing do not always seem to correlate:
"Rapid eye movements were absent in 38-42% of behaviours."
When they do correlate at least during goal-oriented motor behavior, they do "imitate the 'scanning' of the dream scene." And that seems to happen about 82-90% of the time when they do this sort of behavior.
The math confuses me there as well though in contrast to how 38-42% of movements were absent of eye moments as well, but I believe the discretion is due to using the control vs. disordered group and lumping different group(s) into different calculated percentages.
Now also note above that when the goal-oriented behavior occurred and the eyes reflected this, they did so in the "same plane and direction." This was peculiar as the word "plane" could mean horizontal or vertical plane I believe...it could also mean focal plane. I wish they distinguished that better - I'm going to hypothesize it was just either the vertical or horizontal plane so that
In essence, a large part of the time people are acting out their behaviors when asleep the eyes correspond highly to the direction, but only to one plane, which means there wasn't really focal point - as you'd need to correspond to both the horizontal and vertical planes then.