gzip - redirection or piping?

Question

I will be backing up a large (750GB) disk to an external USB disk using dd.
Should I be using redirection or piping? Which is more efficient? Or is there a difference?

Also, what is the best block size? USB is likely to be the bottleneck here.

dd if=/dev/sda bs=1M | gzip -c > /mnt/sdb1/backups/disk.img.gz

gzip -dc /mnt/sdb1/backups/disk.img.gz | dd of=/dev/sda bs=1M

vs

dd if=/dev/sda bs=1M | gzip -c | dd of=/mnt/sdb1/backups/disk.img.gz

dd if=/mnt/sdb1/backups/disk.img.gz | gzip -dc | dd of=/dev/sda bs=1M

Thanks.

I would try to backup file systems, not raw disk images. And to tune the parameters, you'll have to benchmark. — , Mar 12 '12 at 06:20
dd is archaic (any other tool thankfully does away with bs= and count=, like ddrescue), and in your case also pretty much pointless. Just pipe it into gzip -c </dev/sda >disk.img.gz & gzip -cd disk.img.gz >/dev/sda. — , Mar 12 '12 at 11:12
Or do it with rsync. Just construct the command with the parameters, and it will do you everything in the most efficient way it thinks. — vakufo, Mar 12 '12 at 11:53
In the second command of the second listing, what is the dd if=.../disk.img.gz | part supposed to do? As far as I understand, gzip -dc (=zcat) doesn't care for its stdin when there's a file argument. — sr_, Mar 12 '12 at 13:57
I don't think Linux can reliably reproduce Windows installations by simply copying the file structure. Hence, the need for bit-wise copying like dd. ddrescue looks more robust then plain dd. Would you use GNU ddrescue or dd_rescue? What is a good Live CD to download? — mcu, Mar 12 '12 at 20:45

score 11 · Accepted Answer · edited Aug 09 '18 at 11:55

11

You don't need to use dd or piping at all.

</dev/sda gzip >/mnt/sdb1/backups/disk.img.gz
</mnt/sdb1/backups/disk.img.gz gunzip >/dev/sda

I once made a benchmark and found using dd slower than cat for a straight copy between different disks. I would expect the pipe to make any solution involving dd even slower in this case.

edited Aug 09 '18 at 11:55

Stéphane Chazelas

544,893

answered Mar 13 '12 at 08:38

Gilles 'SO- stop being evil'

829,060

That is an example of using redirection. – psusi Mar 13 '12 at 17:42
@psusi Gah! I meant to write “dd or piping”. Thanks. – Gilles 'SO- stop being evil' Mar 13 '12 at 17:47
Is there a way to combine pv in these commands? – Yaron Feb 19 '18 at 21:24
2

@Yaron Sure: replace gzip by pv | gzip and gzip -d by gzip -d | gv – Gilles 'SO- stop being evil' Feb 19 '18 at 21:58
So, just to be sure, using redirection on an unmounted device, like </dev/sda or </dev/sda1 will read raw disk data, and not the file system, is it correct? I need this to work with a Windows partition. – mcu Nov 11 '20 at 13:31
1

@mcu Right: /dev/sda is the whole disk, and /dev/sda1 is one whole partition. This just reads the data on the disk/partition without caring if that data may have some meaning as a filesystem. – Gilles 'SO- stop being evil' Nov 11 '20 at 19:02
Would this terminate the copying process somewhat gracefully if the user presses Ctrl+C? – mcu Nov 11 '20 at 19:07
1

@mcu It'll interrupt the copy in progress, leaving a partial image file if interrupted during the first command and a half-written (unmountable) disk if interrupted during the second command. A truly graceful interruption that restores the destination disk to its original content if interrupted is not really doable. – Gilles 'SO- stop being evil' Nov 11 '20 at 21:18
How might we adjust the command containing the pipe to add sudo so that it can run as superuser (needed to access the raw disk)? </dev/nvme2n1 pv | gzip >disk.img – Steven Lu Dec 21 '22 at 07:34
Update: Seems like sudo bash -c "</dev/nvme2n1 pv | gzip >disk.img" is working for me – Steven Lu Dec 21 '22 at 07:37

score 3 · Answer 2 · edited May 23 '17 at 12:39

3

Piping involves one more process and one more user-land copy, so it should be more efficient to use redirection.

But I guess that on nowadays hardware & software caching system, it should not make any real difference. Maybe you can have better results using bs=4k or bs=64k, since it's the pipe's limit under linux. See this question for more detail about different bs parameters.

edited May 23 '17 at 12:39

Community

1

answered Mar 12 '12 at 12:39

Coren

5,010

score 2 · Answer 3 · answered Aug 09 '18 at 11:30

2

Simpler:

gzip -c /dev/sda > disk.img.gz
gzip -dc /mnt/sdb1/backups/disk.img.gz > /dev/sda

answered Aug 09 '18 at 11:30

Mike Redrobe

231

gzip - redirection or piping?

3 Answers3