12

I'm about to compress a large directory and I want to know how large, exactly, the resulting file will be.

I've tried using du:

$ tar -cv dir | du -h -
du: cannot access '-': No such file or directory

Then I tried using the file version of '-':

$ tar -cv dir | du -h /dev/stdin
1.0K

I'm certain this number isn't accurate. How can I get the size of stdin?

Gilles 'SO- stop being evil'
  • 829,060
strugee
  • 14,951

5 Answers5

15

tl;dr: tar -cv dir | wc -c - | cut -d' ' -f 1 | awk '{print $1/1000"K"}'

du doesn't actually count the size of the file itself. It just asks the kernel to query the filesystem, which already keeps track of file size. This is why it's so fast. Because of that, and the fact that you're counting a stream, not a file, du doesn't work. My guess is that 1.0K is a hardcoded size for /dev/std* in the kernel.

The solution is to use wc -c, which counts bytes itself instead of querying the kernel:

$ tar -cv dir | wc -c

If you want output similar to du -h:

$ tar -cv dir | wc -c | awk '{print $1/1000"K"}'

The awk turns the number into a human-readable result.

strugee
  • 14,951
9

I'd suggest:

tar cf - dir | wc -c

A simple c (no leading - is required) is used to create a tar archive, f specifies an output file and - denotes that it be stdout. (Note that if you want just the size and there are many files beneath dir you may rather omit tar's v for performance reasons.)

Janis
  • 14,222
  • @mikeserv; I seem to recall that I worked with tars in the past where specifying a tar-file (thus f and -) were required. - Just looked it up; without f - the tar assumed /etc/mt0 as default. – Janis Jun 01 '15 at 03:00
  • 1
    I've looked it up in a book that I used as a manual at that time, and I think was based on SysV R4. Few folks certainly recall what /etc/mt0 actually means - "magnetic tape" ;-) I'd be interested how Solaris' tar behaves (because Solaris is one of the contemporary OSes that are known to have still really old stuff in /bin). – Janis Jun 01 '15 at 04:32
  • @mikeserv; PS: The book mentions AT&T's "UNIX Programmers Manual Volumes 1, 2A, 2B" as source (but no manual date or UNIX release version, though; but must have been from the early 1980's, 1983, or so). – Janis Jun 01 '15 at 04:41
  • Have you ever seen this? Unrelated - but I just found it today, and thought you might like it. – mikeserv Jun 01 '15 at 05:10
  • I don't quite understand how this answer is different from mine. is it the presence of the -f flag to tar? – strugee Jun 01 '15 at 18:20
  • @strugee; It was a bit more different before you edited yours just a few minutes ago. The current differences are, 1) unnecessary use of v, unnecessary use of - (in front of c, which is of course just a detail), possible problem with some versions of tar that expect -f - if stdin is concerned. – Janis Jun 01 '15 at 18:49
  • OK, that makes sense. – strugee Jun 02 '15 at 14:52
  • Best answer, easy to remember! – rkok Aug 06 '21 at 04:48
9

With GNU tar you can just do:

tar --totals -c . >/dev/null

...which will render output like...

Total bytes written: 5990400 (5.8MiB, 5.5GiB/s)

...on stderr. Similarly, with any tar (or stream) you can use dd to deliver a report on byte counts. This may or may not be preferable to wc, but dd defaults to a block-size of 512 bytes - which is identical to tar's block-size. If your system's PIPE_BUF is large enough, you can even expand dd's block-size to match tar's record size - which is 20 blocks, or 10240 bytes. Like this:

tar -c . | dd bs=bx20 >/dev/null
585+0 records in
585+0 records out
5990400 bytes (6.0 MB) copied, 0.0085661 s, 699 MB/s

This may or may not offer a more performant solution than wc.

In both the dd and tar use-cases you needn't actually dispose of the stream, though. I redirect to /dev/null above - but I could have as easily redirected to some file and still received the report on its size at the time it was written.

Community
  • 1
mikeserv
  • 58,310
  • In case the file gets written to disk there would be no use to separately determine the size since that information is stored with the file. (+1) for the expected performance gain of dd (compared to wc). – Janis Jun 01 '15 at 04:53
  • 1
    @Janis - possibly true in the simplest case - but imagine rather that dd's output is passed on - to a compressor, say - and for whatever reason you find it desirable to know both the raw size of the archive and the compressed one. Also useful is to get an instant report on the record counts - tar is not just an archive, but a stream format. It can be used in ways other than just saving to a group of files to some other file. It is often useful for blocking a stream before modifying it. At each of those record boundaries is a whole block of NULs. – mikeserv Jun 01 '15 at 05:03
6

I would go for @strugee's answer but I'll use the numfmt tool which is designed for memory unit conversion and supports many options:

tar cf - dir | wc -c | numfmt --to=si

--to=si will display the output in SI units (e.g: 4M means 4000000 bytes). If you prefer so-called "binary" units, pass --iec-i (e.g: 4Mi means 4194304 bytes).

Stéphane Chazelas
  • 544,893
Mustapha-Belkacim
  • 161
  • 1
  • 3
2

The wording of your question lends itself to the tar ... | wc -c answers above. I originally read your question with a silent assumption that you wanted the size to be reported while it was creating the tar file (perhaps tar's output was then being piped over a network link?).

In which case, I'd suggest pv -- pipe viewer. I've seen reference to it but have not yet had a chance to play with it.

References

cuonglm
  • 153,898
Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255