4

I have a folder(which contains a lot of sub-folders and files) on a machine, I used 

   du -m

and it shows the disk usage of all sub-folders and files, anyway, the overall disk usage is 78M

I used scp -r to copy the folder into another machine, this time, du -m get the overall disk usage: 12M, very different.

Why does this happen?

I'm afraid some of the files or sub-folders are not copied fully, so are there any other ways to check the total number of bytes?

Anthon
  • 79,293
misteryes
  • 1,333
  • 1
    Were there errors during the copy? Are all the files present in the copy? Do they have the same size? What was the exact scp command line: were you just copying the directory? – Gilles 'SO- stop being evil' May 07 '13 at 01:06
  • there is no errors. I think all files are present in the copy. They don't have the same size, but they have the same content. I used scp -r – misteryes May 07 '13 at 08:25
  • Do you really have all the files (compare the output of find | sort)? If not, what can you spot about the ones that are missing? If you do, do all the files have the same size (compare the output of du -ak | sort -k2)? – Gilles 'SO- stop being evil' May 07 '13 at 08:28
  • I used du -k|wc -l, the number of files/subfolders is the same. But for the same file, the size is different, e.g, for a file on one machine it is 4K with du -k but 16 bytes with du -b, on another machine it is 0K with du -k and 16 bytes with du -b – misteryes May 07 '13 at 09:30
  • What are the operating systems (Linux/OSX/Solaris/FreeBSD/Windows/…) and filesystems (ext4/btrfs/hpfs+/ffs/ntfs/…) on both sides? Are these a few big files or are there a lot of small files — how many files are there in total? – Gilles 'SO- stop being evil' May 07 '13 at 10:04
  • they are all linux. The big-sized folder is in a ufs system while the small-sized folder is in a nfs system. There are a lot of small files and many sub-folders within sub-folders. 1 to 2 thousand files. – misteryes May 07 '13 at 11:15

2 Answers2

3

Several possibilities.

a) scp may not have copied hidden files/directories; hard to tell w/o knowing your data, seeing how you invoked scp.

b) another scenario will be that the filesystems use different block-sizes, and du doesn't add up the files byte sizes, it measures occupied disk space.

tink
  • 6,765
  • so how to get the file byte sizes? – misteryes May 06 '13 at 21:39
  • and how to use scp so that even hidden files can be copied, but anyway, I don't think there are hidden files because the folder is full of data files which are produced by some of my programs – misteryes May 06 '13 at 21:40
  • Have you considered using rsync? As for the how .. show us how you did it, and describe the directory layout. – tink May 06 '13 at 21:49
  • @misteryes scp doesn't care about dot files ("hidden files") so don't worry. Just the shell does when you invoke scp: scp -r *.foo ... – Hauke Laging May 06 '13 at 22:08
  • what is Just the shell does when you invoke scp: scp -r *.foo ...? – misteryes May 06 '13 at 22:14
  • @misteryes You should use the @user comment feature so that those you address get informed. The shell expands *.foo to a list of files. scp doesn't see the *. In the default configuration the shell would remove a file name like .file.foo from this list because of the leading ".". You can check that with echo *.foo; echo .*.foo. – Hauke Laging May 06 '13 at 22:21
  • @misteryes, use du -b to get the apparent size rather than the actual used space. – psusi May 06 '13 at 23:42
2

Another possibility, which is remote in my judgement: your original directory and the directories residing inside it have had a lot of files added and deleted. In most filesystems, directories only grow in size, never shrink. The missing space could constitute unused slots in directories.