7

I have a drive containing 122,865 image files that are about 82.1 GB in total. However they use 171 GB of space on disk. The average file size is about 717KB. Since I intend to continue adding more files of this size I would like to reformat my drive with a new file system or alter the existing one to better accommodate these small files.

The drive is currently running ext4 with the defaults provided by ArchLinux's mkfs.ext4. I am not the slightest bit certain on where to begin with solving this, can anyone point me in the right direction or provide some resources for me to work with?

phuclv
  • 27,773

4 Answers4

7

The smaller the block size (1024 bytes, p.e.), the better for efficient disk usage, in case there's a lot of small files on that partition. Try to reformat that partition with the smallest block size:

mkfs.ext4 -b 1024 /dev/your_partition
phuclv
  • 27,773
Roberto Paz
  • 242
  • 2
    We're in this use case (IMAP server), and for a long time each new file system (pr customer) has been created with /sbin/mke2fs -t ext4 -b 1024 -i 1024 -m 0 -I 128 -L <LV-Label> /dev/<path>. Been trying to read up and see if there is new advice - but it seems to hold - still after 8 years... – sastorsl May 27 '20 at 09:51
1

Some time ago it was very common to use ReiserFS for that type of cases. I have no experience with it but I know that :)

1

In fact bigger block size is better for large items like multimedia files because it reduces the amount of metadata necessary for file management, resulting in less seeks required to read/write a file. Many cameras even format the SD card to a larger block size if you use the in-camera format feature. See

Reducing block size to 1024 bytes increases the metadata size significantly, and it'll perform far worse if your ext4 doesn't have extents enabled. Having a larger number of blocks also increases the level of indirection in case of ext2/3/4

So the solution is to increase the block size to at least 64 KB and use some filesystem that supports block suballocation to store the last odd-sized block of multiple files into a single block, saving memory. Some examples of such filesystems: ReiserFS, Reiser4, Btrfs and ZFS. They're optimized for a large number of files and is much better than ext4 in such situations. For more information read What filesystem can I use for a large amount of small-sized data?

phuclv
  • 27,773
1

If you are not welded to using ext4, other file systems may handle such cases better.

Reiser4 file system has a feature known as "tail packing" where multiple file tails that don't require a full block can be jointly stored in a block. This can significantly improve space usage with many files. Reiser4 also supports compression.

ZFS uses variable block sizes, from a single sector (note: many modern drives use 4KB sectors!) up to 128KB (or even higher in some cases). It also supports compression. The embedded_data feature can save further space by storing blocks that compress down to 112 bytes or less into the block pointer page itself. These features coupled with 512 byte sectors, this can potentially net you very significant space savings for your use case.

Gordan Bobić
  • 3,430