1

Calculating checksums for large files, ie tens to hundreds of gigabytes is a lengthy process.

Are there file downloading programs that calculate checksums during the download process? I think some web servers have the ability to include checksum information in the response headers, but can the downloading process verify the information itself, rather than wait for the download to complete?

aria has some partial capability with some kind of files, but are there any with full capability built in?

vfclists
  • 115
  • 4
  • 1
    If the headers were spoofed it would defeat the entire purpose of the checksum. Sadly, I think you need the whole file to be sure that the checksum "checks" out. Even aria, using the .metalink framework, does the "automatic checksum verification" on finished downloads, although it does it automatically. – INV3NT3D Sep 26 '16 at 12:29
  • I am not so concerned about the accuracy of the checksum. I am more concerned about the time it takes to compute it, as there is bound to be some spare CPU capacity during the downloading process and there is no need to read the whole file again to calculate its checksum. – vfclists Sep 26 '16 at 12:59
  • 1
    If you aren't worried about the accuracy, then why are you calculating it at all? – Douglas Leeder Sep 26 '16 at 13:03
  • @DouglasLeeder The checksum will always be calculated locally. Doing it in parallel with the downloading process is concerning here. Whether calculated in parallel or not the file will always be rejected if the checksum matches. With a large file on a slow link I'd rather have the checksum available as soon as the downloaded finishes, rather than wait for a long time for the file to finish then having to nice the checksum process to avoid too much CPU on a large file. – vfclists Sep 26 '16 at 18:59
  • If the file is 10GB and you're downloading with 1Megabyte/sek, you need something like 3 hours for downloading and 2 minutes for calculating the checksum. I don't know if this is really worth the hassle. However, when download speed and drive speed are similar, it does make a difference (see my answer) – Lukas Sep 26 '16 at 19:12

2 Answers2

3

A very similar question has already been answered on Unix & Linux Stackexchange.

wget -O - http://example.com/file | tee file | md5sum > file.md5

Just replace the md5sum with sha1sum and it should work. Note that the proposed solution works only with *nix operating systems - getting it to work within Windows might be a challenge (but the Bash Subsystem for Windows or Cygwin might work for you).

Edit:

As I was asked in the comments how this works, I'll explain it here:

wget -O - http://example.com/file downloads the file (in sequential order) and outputs its content on the console. The pipe | allows me to forward this data to the next program: tee file stores the output of wget into the file file and forwards the data to the last program in the chain. md5sum > file.md5 calculates the md5-hash and stores it in file.md5.

Performance: (YMMV!)

  • Downloading a big file (4GB) from a local server takes roughly 80 seconds
  • Calculating the hash of the file (when stored locally) takes 70 seconds
  • Doing both simultaneously takes again 80 seconds, resulting in a speed boost of almost 50%.
Community
  • 1
Lukas
  • 3,188
  • 1
  • 16
  • 20
  • I'm curious as to how this works. Is the file downloaded in sequential order, then the md5 is calculated through the output stream as the file comes through? So that way, exactly as vfclists hoped, the checksum is calculated "line by line" while the file comes through, rather than all at the end? Does the file even have to be downloaded in the proper order for this to work? Sorry for my limited knowledge of *nix command-lines. – INV3NT3D Sep 26 '16 at 15:45
  • 1
    @INV3NT3D I have edited my answer to explain this with more details. Feel free to ask again if it is still not clear ;-) – Lukas Sep 26 '16 at 16:25
  • 1
    Nailed the explanation, thank you @Lukas . If I could up-vote your answer twice, I'd still be looking for a way to give you a third up-vote. – INV3NT3D Sep 26 '16 at 16:31
  • If there was an http/browser/ftp protocol which created checksums of portions of the file and enabled the file to be downloaded in parts with each part checksummed individually that would be nice, like the BitTorrent method. Other than the BitTorent method is there another such scheme, or under development? – vfclists Oct 02 '16 at 18:46
  • I don't know. I suggest you to take a look at rsync and check whether it meets your needs (it's a little bit different, though.) – Lukas Oct 02 '16 at 20:02
3

As I understand your question the software should verify the downloading file piece by piece and ensure that each piece is valid. This method of download verification is implemented in BitTorrent protocol. As stated in the Wikipedia page:

The file being distributed is divided into segments called pieces.
...
Each piece is protected by a cryptographic hash contained in the torrent descriptor.

Torrents are public by default. If you want private transfer between two nodes you should use private syncing clients like Resilio Sync (formerly BitTorrent Sync) and Syncthing.

Koorosh Pasokhi
  • 1,107
  • 1
  • 9
  • 11