1

@slhck provided a useful answer to this similar question.

Downloading multiple files with wget and handling parameters

This code was provided to read urls from a text file filelist, then run a loop to download each.

    #!/usr/bin/env bash 
    while read line 
      do   
        wget -c --load-cookies cookies.txt $line -O ${line##*/} 
    done < filelist

 

As I'm not familiar with shell scripts, I am wondering if you could download multiple xml files by:

  • creating a simple comma seperated text file filelist
  • read from:

filelist

renamedfile.xml, url-01.php
renamedFileDifferent.xml, url-02.php
specificFileRename.xml, "url-03"
newFilename.xml, "url-04"

  • read through each line
  • split the line into newfile, remoteFile

  • And run:  

    wget -O newfile remoteFile >/dev/null 2>&1

 

Is it possible to write this in a shell script? Can you provide an example?

Community
  • 1
rrrfusco
  • 145

1 Answers1

1

To get the text from before the comma you use ${line/,*}.

(What this actually does is replace ",*" or all the text from the comma onwards with empty string - leaving only the part of the string before the comma)

To get the text from after the comma you use ${line/*,}.

So the full command would be:

while read line
  do
    wget -O ${line/,*} ${line/*,} >/dev/null 2>&1 
done < filelist

or, on one line:

while read line; do echo wget -O ${line/,*} ${line/*,} >/dev/null 2>&1; done < filelist

In Windows (assuming you've installed wget, from http://gnuwin32.sourceforge.net/packages/wget.htm, and set your path correctly), it would be:

for /f "delims=, tokens=1*" %a in (filelist) do wget -O %a %b
Luke
  • 1,155
  • 1
  • 9
  • 18
  • What directory is a good place to put the text file, and shell file typically? This will be run as a cron job. – rrrfusco Sep 25 '12 at 00:59
  • Wherever you want to run it from (and save the files to). eg. if you put filelist and test.sh in sub directory of your home, and then run it from there, that's where it will save the files. – Luke Sep 25 '12 at 01:03
  • I'm not sure it will run as expected if run as a cron job. You will want to specify absolute paths in the filelist (~\downloaddir\renamedfile.xml) and probably in the crontab as well (46 8 * * cd ~\downloaddir; test.sh or maybe just 46 8 * * ~\downloaddir\test.sh). Refer http://stackoverflow.com/questions/13204/why-doesnt-my-cron-job-work-properly and http://www.unix.com/shell-programming-scripting/129329-crontab-working-directory.html – Luke Sep 25 '12 at 11:09
  • another option is to harcode the paths into the script, eg.

    wget -O ~/downloaddir/${line/,*} ${line/*,} >/dev/null 2>&1 and

    done < ~/downloaddir/filelist

     – Luke Sep 25 '12 at 11:12