MoCA is designed to work over the 75Ω coaxial TV antenna / cable TV (henceforth "CATV") / satellite TV cable you already have in your home. Topologically, it's expected to be a tree made by a single splitter or a small hierarchy of splitters.
In general, something like quad-shielded RG-6 is going to work better than RG-59. In fact, RG-59 should be avoided because its attenuation is too high.
The fewer splitters your MoCA signals have to to go through, the better.
The MoCA specs allow for MoCA devices to use different frequency bands in order to avoid the frequency bands used by the CATV or satellite TV signals that might also be on the same coax in some homes. In general, CATV is expected to use frequencies from 5MHz up to 850 MHz or 1GHz, and MoCA is designed to use frequencies above that, up to 1.6GHz, and perhaps up to 2GHz in the future.
Different cable designs can attenuate different frequencies to different degrees, so it would be helpful if you knew how much your coax cables attenuate the signal at frequencies between 1GHz and 2GHz. Coax cable attenuation ratings are often given in decibels per 100 feet.
It is helpful to make sure your splitters are rated for 5MHz (or below) to 2GHz (or above). If your splitters don't say a rating or say a rating that doesn't go up to 2GHz, that doesn't mean they don't pass those higher frequencies very well, it just means no one designed or tested them to do that, so you don't know. Maybe they do, maybe they don't. You could test it yourself if you were a EE or RF nerd with some lab equipment.
TV signal boosters (amplifiers) often only amplify frequencies traditionally used by CATV channels (i.e. less than 850MHz or 1GHz), and may filter out frequencies above that. So TV signal amplifiers on your in-home coax wiring can be a MoCA killer. If you really need a CATV signal booster, you may be able to find a MoCA friendly one nowadays.
It's important to put a low-pass filter at the "Point Of Entry" (POE) where the CATV line comes into your home, before the first splitter. These "MoCA POE" filters allow the lower frequency (DC to 1GHz) CATV signals to pass into your home, but block higher frequencies (1GHz and above as used by MoCA), keeping your neighbors MoCA from interfering with yours, and vice-versa. AND as a HUGE BONUS, rather than just absorbing the frequencies they filter, they actually reflect your own MoCA signals back into the rest of your home's wiring, which helps the signal cross splitters and generally improves signal strength. So go buy and install a MoCA POE filter if you haven't already.
Typically coax splitters allow lots of signal to pass from the "in" to the "outs" and vice-versa, but they often greatly attenuate (isolate) the signal between the "outs". This can be a problem if your MoCA devices are on separate "outs" of a splitter. If you go to RF equipment suppliers, you may be able to get splitters with low isolation between the outs.
MoCA devices often have a diagnostics screen where they show you what signal strength they're seeing from the other MoCA devices on the network. It might be interesting to know what signal your devices are seeing for each other, and see if that's what you'd expect given what you know of your cable ratings and lengths, splitter ratings, and topology. Ideally you want the received signal strength to be -40dBm or better to get the top rates. If the low rate you're getting is only due to too much attenuation, it suggests your received signal strength is -55dBm or worse.
The MoCA Alliance (the industry consortium that specifies, certifies, and markets MoCA) has a MoCA installation guide with a lot more information here (warning, direct PDF link): http://www.mocalliance.org/technology/Final_Best-Practices-for-Installation-of-MoCA_170516rev01.pdf
