40

I am having some trouble with NFS, and I'd like to try using just plain old TCP.

I have no idea where to begin, though.

Hardware-wise, I am using an ethernet crossover cable to network two netbooks.

To network them, I type

$ sudo ifconfig eth0 192.168.1.1 up && ping -c 10 -s 10 192.168.1.2 && sudo /etc/init.d/nfs-kernel-server start

on the first netbook and

$ sudo ifconfig eth0 192.168.1.2 up
$ ping -c 10 -s 10 192.168.1.1
$ mount /mnt/network1

on the second

where /mnt/network1 is specified in /etc/fstab as

192.168.1.1:/home /mnt/network1 nfs noauto,user,exec,soft,nfsvers=2 0 0

as well as in /etc/exports (using the syntax of that file), on the first netbook.

The above works fine, but the files and directories are huge. The files average about half a gigabyte a piece, and the directories are all between 15 and 50 gigabytes.

I'm using rsync to transfer them, and the command (on 192.168.1.2) is

$ rsync -avxS /mnt/network1 ~/somedir

I'm not sure if there's a way to tweak my NFS settings to handle huge files better, but I'd like to see if running an rsync daemon over plain old TCP works better than rsync over NFS.

So, to reiterate, how do I set up a similar network with TCP?

UPDATE:

So, after a good at few hours of attempting to pull myself out of the morass of my own ignorance (or, as I like to think of it, to pull myself up by my own bootstraps) I came up with some useful facts.

But first of all, what led me on this rabbit trail instead of simply accepting the current best answer was this: nc is an unbelievably cool program that resolutely fails to work for me. I've tried the netcat-openbsd and netcat-traditional packages with no luck whatsoever.

The error I get on the receiving machine (192.168.1.2) is:

me@netbook:~$ nc -q 1 -l -p 32934 | tar xv
Can't grab 0.0.0.0:32934 with bind
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

route gives:

me@netbook:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         dir-615         0.0.0.0         UG    0      0        0 wlan0
link-local      *               255.255.0.0     U     1000   0        0 eth0
192.168.0.0     *               255.255.255.0   U     2      0        0 wlan0
192.168.1.0     *               255.255.255.0   U     0      0        0 eth0

But, here's the good news: having the static IP addresses set in /etc/network/interfaces, which I started doing while trying to get nc working, fixed all my NFS problems and rekindled my love for NFS.

The exact configuration I used (with 192.168.1.1 for the first netbook, of course) was:

auto eth0
iface eth0 inet static
address 192.168.1.2
netmask 255.255.255.0

With those settings, the two netbooks will be able to ping each other directly after being booted up, without even an ifup.

Anyway, I'd still really like to see nc in action, so I'm hoping someone help me debug this process.

ixtmixilix
  • 13,230
  • If both directories are local you're better off just using plain old /bin/cp or not use NFS at all – Karlson Sep 17 '12 at 12:46
  • NFSv2 is pretty old. What OS are you using? – Nils Sep 17 '12 at 15:20
  • the latest Debian and the latest Ubuntu, respectively. i got all of those commands (including nfsvers=2) from this tutorial (http://michaelminn.com/linux/home_network/) – ixtmixilix Sep 17 '12 at 16:43
  • 1
    Runing rsync against a file accessed over NFS means that the entire contents of the file needs to copied over the network at least once. You don't need a daemon to invoke a client/server rsync - just run it over ssh. (it's theoretically possible to invoke the remote end over telnet / rsh - but rather silly to run such a service in practice - ssh doesn't add a lot of overhead). – symcbean Sep 17 '12 at 12:55
  • 7
    actually, ssh adds a pretty large amount of overhead, crypto is not cheap. Over normal Internet speeds, it doesn't matter, but over a LAN (or direct cross-connect, in this case) you may notice. Over gigabit, except on the very fastest machines (or ones with AES-NI instructions, if SSH uses those) I'm pretty sure it'll be noticeable. – derobert Sep 17 '12 at 16:49
  • Have you had any luck maxing out a 10 gigabit link without going to something like RDMA? It looks like the main Linux networking bottleneck is the speed of a single TCP socket (300MB/sec or so is the best I've seen). Sadly, NFS doesn't do multiple sockets like Solaris ( please please fix this some day ). –  Apr 24 '15 at 22:34
  • 'nc -q 1 -l ...' according to the man page (and help output), nc does not support a -q option. – David Balažic Jan 19 '16 at 13:17
  • strace the nc call. That'll help you find the precise error message that happens when it goes to bind the LISTEN socket. You may also check your SELinux audit logs, in case something in there is blocking the LISTEN socket from being opened. – Michael Mol May 25 '17 at 15:48
  • See also https://unix.stackexchange.com/questions/227951/what-is-the-fastest-way-to-send-massive-amounts-of-data-between-two-computers – rogerdpack Sep 24 '18 at 19:39

3 Answers3

66

The quick way

The quickest way to transfer files over a LAN is likely not rsync, unless there are few changes. rsync spends a fair bit of time doing checksums, calculating differences, etc. If you know that you're going to be transferring most of the data anyway, just do something like this (note: there are multiple implementations of netcat; check the manual for the correct options. In particular, yours might not want the -p):

user@dest:/target$ nc -q 1 -l -p 1234 | tar xv

user@source:/source$ tar cv . | nc -q 1 dest-ip 1234

That uses netcat (nc) to send tar over a raw TCP connection on port 1234. There is no encryption, authenticity checking, etc, so its very fast. If your cross-connect is running at gigabit or less, you'll peg the network; if its more, you'll peg the disk (unless you have a storage array, or fast disk). The v flags to tar make it print file names as it goes (verbose mode). With large files, that's practically no overhead. If you were doing tons of small files, you'd turn that off. Also, you can insert something like pv into the pipeline to get a progress indicator:

user@dest:/target$ nc -q 1 -l -p 1234 | pv -pterb -s 100G | tar xv

You can of course insert other things too, like gzip -1 (and add the z flag on the receiving end—the z flag on the sending end would use a higher compression level than 1, unless you set the GZIP environment variable, of course). Though gzip will probably actually be slower, unless your data really compresses.

If you really need rsync

If you're really only transferring a small portion of the data that has changed, rsync may be faster. You may also want to look at the -W/--whole-file option, as with a really fast network (like a cross-connect) that can be faster.

The easiest way to run rsync is over ssh. You'll want to experiment with ssh ciphers to see which is fastest, it'll be either AES, ChaCha20, or Blowfish (though there are some security concerns with Blowfish's 64-bit block size), depending on if your chip has Intel's AES-NI instructions (and your OpenSSL uses them). On a new enough ssh, rsync-over-ssh looks like this:

user@source:~$ rsync -e 'ssh -c aes128-gcm@openssh.com' -avP /source/ user@dest-ip:/target

For older ssh/sshd, try aes128-ctr or aes128-cbc in place of aes128-gcm@openssh.com.

ChaCha20 would be chacha20-poly1305@openssh.com (also needs a new enough ssh/sshd) and Blowfish would be blowfish-cbc. OpenSSH does not allow running without a cipher. You can of course use whichever rsync options you like in place of -avP. And of course you can go the other direction, and run the rsync from the destination machine (pull) instead of the source machine (push).

Making rsync faster

If you run an rsync daemon, you can get rid of the crypto overhead. First, you'd create a daemon configuration file (/etc/rsyncd.conf), for example on the source machine (read the rsyncd.conf manpage for details):

[big-archive]
    path = /source
    read only = yes
    uid = someuser
    gid = somegroup

Then, on the destination machine, you'd run:

user@dest:~$ rsync -avP source-ip::big-archive/ /target

You can do this the other way around too (but of course you'll need to set read only to no). There are options for authentication, etc., check the manpage for details.

derobert
  • 109,670
  • 2
    This is an excellent answer. The other one is great too. Is there no accepted answer just because the asker can't choose between them? – sudo Jan 14 '16 at 00:47
  • How robust is the netcat approach? If the network drops packets, it seems like it'll lose random parts of the files. – sudo Jan 15 '16 at 23:33
  • 1
    @sudo it's using TCP, which will retransmit as needed. So it should be fine against packet loss, random corruption (to the extent TCP and Ethernet checksums catch it), etc. Of course, it's not secure against attack like tunneling over ssh. – derobert Jan 15 '16 at 23:36
  • @sudo yep. Though if you're running over a network that loses packets for reasons unrelated to congestion, you might be able to speed it up by changing your TCP congestion control algorithm or using some reliable UDP implementation (basically TCP in userspace). – derobert Jan 15 '16 at 23:47
  • Thanks. I'm not sure how reliable my connection is, so I'm going to tar the file beforehand on the sending end and not open the tar on the receiving end, then I can compare the MD5 checksums. I definitely drop packets when pinging, but that doesn't use TCP, so TCP might be OK. – sudo Jan 15 '16 at 23:52
  • 1
    @sudo you can do it all at once, insert some tee commands into the pipe on both sides to compute the checksums. – derobert Jan 15 '16 at 23:54
  • I don't know what version of netcat @derobert is using, but on Redhat 6 it says the combination of -l and -p is illegal. I ended up using the commands target: "nc -l 1234 | tar xv" and source: "tar cv . | nc dest-ip 1234" – Connor Jan 13 '17 at 02:36
  • @ConnorMcKay There are two (at least) independent implementations of netcat, I should probably mention the options are a little different. [BTW: despite what the netcat-openbsd manpage says, it's perfectly happy with -l -p PORT]. Also, for the record, it's RHEL 6, RedHat 6 is much, much, older. I remember it because it was one of the first distros I used professionally. – derobert Jan 13 '17 at 02:48
  • This might be a stupid question but I'm new to linux: Where is the file specified in the nc command? Or does it just take everything in the current directory?! – TheStoryCoder Mar 23 '18 at 08:27
  • 1
    @TheStoryCoder The dot in the tar part is telling it to do current directory. That's not actually part of the nc command, tar is being used to create a tar archive, which is being piped in to netcat (and on the other side, netcat is being piped in to tar to extract the archive). I'm afraid a comment isn't really sufficient to explain pipes, but hopefully that's enough to get you started... – derobert Mar 23 '18 at 08:36
  • 1
    @derobert ESXi 5.1 that I'm trying this on doesn't support -q for nc. Does that matter? I'm not getting any data transferred. Source starts listing two file entries (. and then a real file) and then quits after a while. – TheStoryCoder Mar 23 '18 at 15:53
  • 1
    @TheStoryCoder There are different implementations of netcat which unfortunately take different options. -q 1 tells it to exit 1 second after tar finishes (otherwise, it'd stay running until the remote netcat disconnects). So you should be able to leave that out (or check the docs of your netcat for an equivalent), though maybe you'll have to control-D or control-C one sides when its done. I think the way to specify ports (-p X) might also be different, that'd need to be changed or it won't work. – derobert Mar 23 '18 at 16:05
  • 1
    -c on GNU netcat does the same thing as -q 1 FWIW – Wes Dec 03 '20 at 17:31
  • Your nc option appears to me to be about the same speed as rsync --each are ~400 KiB/sec (~0.4 MiB/sec). I'm not sure why they are so slow. It's over a local network with the computers side-by-side. Maybe it's because the source computer is super old (low-end laptop from 2015) and has a slow wifi adapter. Except that doesn't make sense because I'm using an external 5 GHz USB adapter that is pretty fast (600 Mbps max I think). My local wifi network is a high-end Orbi mesh router system at 1 Gbps. – Gabriel Staples Aug 26 '21 at 04:02
  • @GabrielStaples something is wrong there, that's unacceptably slow for a computer from 2005, much less 2015. You could try iperf3 to look for network issues, iostat for disk I/O issues, etc. – derobert Aug 26 '21 at 05:43
22

How? Or TL;DR

The fastest method I've found is a combination of tar, mbuffer and ssh.

E.g.:

tar zcf - bigfile.m4p | mbuffer -s 1K -m 512 | ssh otherhost "tar zxf -"

Using this I've achieved sustained local network transfers over 950 Mb/s on 1Gb links. Replace the paths in each tar command to be appropriate for what you're transferring.

Why? mbuffer!

The biggest bottleneck in transferring large files over a network is, by far, disk I/O. The answer to that is mbuffer or buffer. They are largely similar but mbuffer has some advantages. The default buffer size is 2MB for mbuffer and 1MB for buffer. Larger buffers are more likely to never be empty. Choosing a block size which is the lowest common multiple of the native block size on both the target and destination filesystem will give the best performance.

Buffering is the thing that makes all the difference! Use it if you have it! If you don't have it, get it! Using (m}?buffer plus anything is better than anything by itself. it is almost literally a panacea for slow network file transfers.

If you're transferring multiple files use tar to "lump" them together into a single data stream. If it's a single file you can use cat or I/O redirection. The overhead of tar vs. cat is statistically insignificant so I always use tar (or zfs -send where I can) unless it's already a tarball. Neither of these is guaranteed to give you metadata (and in particular cat will not). If you want metadata, I'll leave that as an exercise for you.

Finally, using ssh for a transport mechanism is both secure and carries very little overhead. Again, the overhead of ssh vs. nc is statistically insignificant.

Edit

Ok, people, this answer is ten years old. Times have changed.

  • This question is specifically about LANs, not between continents.
  • mbuffered ssh vs mbuffered nc is statistically insignificant. Ssh alone vs nc alone is a different matter. Nc in tcp vs udp mode is a different matter.
  • The best general answer for high speed file transfer, at the time of this update (2022-01), is still scp -c aes128-gcm@openssh.com (as mentioned by @derobert in the comments).
bahamat
  • 39,666
  • 4
  • 75
  • 104
  • 5
    openssl speed on an i7-3770 gives ~126–146 MB/sec for blowfish CBC and ~138–157 MB/sec for AES CBC (this chip has AES-NI instructions). Then ~200–300 MB/sec for sha256. So it can just barely push 1 gigabit. With OpenSSH 6.1+, you could use AES GCM, which it can do at blinding rates (370–1320 MB/sec, depending on message size). So I think its only true that OpenSSH has little overhead if you are running 6.1+ on a chip with AES-NI and using AES-GCM. – derobert Nov 12 '13 at 15:27
  • 1
    Ugh, I changed that to 6.1+ instead of 6.2+ at the last minute, having quickly re-checked. Of course, that was a mistake, it is changes since 6.1. So OpenSSH 6.2+ is the correct version. And it won't let me edit the comment any more now. Comments older than 5 minutes must remain incorrect. Of course, if less than OpenSSH 6.4, see http://www.openssh.com/txt/gcmrekey.adv as without a patch, there was an exploitable flaw in OpenSSH's AES-GCM implementation. – derobert Nov 12 '13 at 15:34
  • The overhead for ssh (or rsync over ssh) is very, VERY important. I have a NAS that uses an Intel Atom CPU. The SSH encryption ABSOLUTELY TANKS the transfer speed. I get consistently < 400 Mbit/sec for RSA, manually overriding it to RC4 gets me ~600 Mbits/sec, and if I use rsync as a daemon, it runs at the link native speed (> 900 MBit/sec, on a gigabit connection). – Fake Name Oct 24 '14 at 08:20
  • While it's true that for many situations, the transport is not critical, it is absolutely important to consider it, particularly if you're not running on extremely high-end hardware. In my case, the Atom (it's a D525, dual core 1.8 Ghz) makes for a completely fine NAS, with plenty of speed for SMB, but encryption absolutely kills it. – Fake Name Oct 24 '14 at 08:22
  • 2
    I get an fatal error due to the parametrization of mbuffer: 'mbuffer: fatal: total memory must be larger than block size \n Terminated' . To correct, I suspect it should read something like 'mbuffer -s 1K -m 512M' with the final 'M' standing for MByte (source: man mbuffer) – Peter Lustig Apr 19 '15 at 06:52
  • I've tried using Cyberduck for transferring large files over SSH, and it's slow and usually fails randomly (over wifi). – sudo Jan 15 '16 at 23:46
  • Basically the reason why mbuffer is needed is because zfs send and receive themselves don't buffer up data, and can be blocked from sending/receiving when they are doing other things. Adding mbuffer to the 2 sides of sending and receiving allows one to keep sending/receiving even when the ZFS tools are blocked. But of course your buffer size is still limited by your available RAM. I got this from: http://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/ – CMCDragonkai Feb 03 '16 at 05:24
  • But mbuffer doesn't seem to be needed anymore due: https://github.com/zfsonlinux/zfs/issues/1161 – CMCDragonkai Feb 03 '16 at 05:28
  • this answer has good faith and transfering with gz + mbuffer is a great idea, but the statement that ssh/scp carries "very little overhead" is very false. It might work well for you in local lan, but doesn't work well at all in cross continent connections. – Rafael Sanches Sep 08 '18 at 00:12
  • "Finally, using ssh for a transport mechanism is both secure and carries very little overhead. " LOL – poige Jan 08 '22 at 18:05
3

You don't even need to use TCP. AoE is an ATA implementation over Ethernet, being layer 2 it is a lower-overhead approach with no knowledge of the TCP/IP stack. It will provide you the fastest possible transfer with the least overhead.***

https://en.wikipedia.org/wiki/ATA_over_Ethernet

***if the network is the bottleneck be sure you are sending compressed data.

William Deans
  • 152
  • 3
  • 1
    Wow that is hard core! :) Wonder if there's any benchmarks... – rogerdpack Sep 24 '18 at 15:57
  • wow i forgot about this, I'm glad i stumbled across this post. I'm gonna set it up and see how fast i can get across my 10gb links – Tim Sep 02 '21 at 08:34
  • 1
    No much explaining in Wikipedia page, specially on software side, here is Arch wiki which did bit more details https://wiki.archlinux.org/title/ATA_over_Ethernet – Salem F Jan 17 '22 at 01:30