2

Just a simple question. Say I have a bunch of *.txt files on a server somewhere and I'm using the standard scp command to download *.txt to my computer. However, say one of the files, call it "bad.txt" is extremely large and happens to be a file I don't need for this specific occasion.

Downloading this file would slow down everything else, so I'd like to not download this file. Is it possible to somehow skip the file from being downloaded as all of the *.txt files are being downloaded? I hope I'm phrasing this question properly.

Amit
  • 441
  • 1
  • 6
  • 14

1 Answers1

1

If you use rsync instead of scp you can use --exclude=bad.txt. i.e.

rsync -av --exclude=bad.txt remote.host:/files/*.txt /local/folder

Alternatively you can change your scp command to pick up the other files. i.e. if you have text1 - test9 and you don't want test9 try just using test[1-8].txt. Although the rsync method should work better.

Justin
  • 1,161
  • 6
  • 9
  • 2
    Thanks for the answer @Justin. Is there any way to do it in real time? Suppose I don't know the exact name of the file, but I do know it has a txt extension. So I just grab all *.txt files. But then as it's downloading all of them, I realize it's downloading one that is extremely large and I don't need -- any chance I can skip it on the fly? – Amit Jan 21 '13 at 04:43
  • No, not that I'm aware of. However that rsync command will resume where it was up to (-a is the archive option; or -rplTGO). So you could start the rsync, when you see it has hit a file you don't want, hit ctrl+c, add the exclusion and re-run it to resume. You may want to add --progress to the rsync in that case as the -v will just show the current file name as it is being copied. (in case you are trying to pick up on the bad file by it's size instead of the name).

    Alternatively, if it's just large files. rsync supports --max-size=SIZE i.e. --max-size=5M or --max-size=2G

     – Justin Jan 21 '13 at 08:15
  • That's actually really helpful. Looks like I'm going to make the switch from the scp to the rsync command, as the -max-size could be just what I need! – Amit Jan 22 '13 at 02:11
  • 1
    Just an FYI; rsync requires the rsync daemon to be running on the system. This is usually not an issue as something like 99% of distros will have it enabled by default. something like gentoo/slax might cause some issues if it hasn't been enabled however. – Justin Jan 22 '13 at 05:33
  • Thanks! I use OS X, and I believe it's enabled, and if it's not, I must've done something to make rsync work, as I've used it before :) Very helpful though, appreciate it. – Amit Jan 23 '13 at 14:26
  • Another detail about rsync, if the destination already includes the file and it has not change, then no transfer happens (just a few handshakes to determine whether the copy is required.) – Alexis Wilke Jun 25 '17 at 19:21