No, white matter structure is not fixed at a very early age. Although a person is born with most of the neurons they will ever have, those neurons must form connections between each other, via axons and synapses, and many of the axons become myelinated during this period. Although the number of neurons remains relatively constant, these processes lead to a substantial increase in overall brain volume. After a few years, there is then a period of synaptic pruning (https://en.wikipedia.org/wiki/Synaptic_pruning), in which many of those connections are rationalised. These processes explain many of the characteristics of human development.
For example, mammalian species can be characterised as precocial or altricial. A precocial species is one in which much of the brain's development has been completed in utero. This allows for a relatively sophisticated behavioural repertoire to be immediately available (think precocial = 'precocious'). For example, a foal is born and almost immediately is able to walk, follow its mother, and so on. Meanwhile, an altricial species like a human is born in a relatively helpless state, incapable of independent locomotion and with even very limited perceptual processes. It is the development of axonal and white matter pathways, with associated synapses, that allow learning and development of new skills in the post-natal period.
During this process, the brain is extremely flexible. For example, a baby can initially distinguish between the phonemes of any language. As its white matter connections develop in response to a lot of exposure to the language(s) in its home environment, the brain becomes more specialised. Eventually, past the pre-teen period, it becomes much harder to perceive and articulate subtle phoneme distinctions from other languages, and the acquisition of new grammars shifts from effortless and rapid, to slow and requiring formalised instruction rather than being spontaneous.
This shift to relative inflexibility reflects the second phase of white matter development (i.e. large-scale synaptic pruning). That is, in the first phase, there is widespread growth of connections between neurons via axons and synapses, along with myelination. In the second phase, the flexibility of this system is traded for the efficiency and focus of a brain that has learned what specific abilities it needs to have. This continues up until about age 10, at which stage 50% of the synapses present at age 2 have been lost (https://link.springer.com/referenceworkentry/10.1007%2F978-0-387-79061-9_2856). Learning is then no longer so rapid, but with specialisation and development, an adult is able to focus attention explicitly and maintain discipline towards achieving goals, where a child might get more easily distracted and seek novelty.
Synaptic formation and loss then continues throughout life, as the underpinning of learning processes, but no longer on the mass structural scale seen in earlier development.
