0

A program of mine is crunching numbers and depending on results creates a directory which stores all the data. Last night it failed. Mkdir no longer worked.

After checking things out I saw that my data directory has close to 36,000 sub directories underneath.

Deleting some of them helped fight the cause.

With that I ask you: is there a magic number of subdirectories that may exist underneath a parent directory? Does it have to do with the amount of data stored? (I don't think so, but still would like a confirmation)

In the above scenario, can I possibly reduce number of directories, maybe gzip some .. Will that solve the problem?

Generally under what circumstance will mkdir fail?

Thank you

Zypher
  • 37,551
JAM
  • 113

3 Answers3

3

Depends on which file system you're using. Reiserfs (my favorite) can handle up to 2^31 files per dir (that's 2 billion), with a max of 2^32 (4 billion) files on the filesys total. It can handle up to 64000 subdirs in a dir

Ext2 (and hence also ext3) has a limit of 32000 subdirs per dir. The max number of files per dir is theoretically unlimited (actually around 130 trillion), but performance becomes terrible with above 10-15 thousand files. The max number of total files on the filesys is limited by the number of inodes you have. With a 1 gig filesystem and a 4k block/inode ratio (the default), you have around 260000 inodes, and that's also the max number of files you can have.

How many files in a directory is too many? (Downloading data from net)

Community
  • 1
Rajat
  • 3,349
  • Just to clarify, how can i tell if my system is EXT2 or not? – JAM Aug 03 '10 at 13:41
  • I believe fdisk -l will print that out. – Luke has no name Aug 03 '10 at 14:14
  • 1
    [root@rajat Rajat]# parted -l Model: ATA HDS728080PLA380 (scsi) Disk /dev/sda: 82.3GB Sector size (logical/physical): 512B/512B Partition Table: msdos

    Number Start End Size Type File system Flags 1 1049kB 525MB 524MB primary ext4 boot 2 525MB 4820MB 4295MB primary linux-swap(v1) 3 4820MB 82.3GB 77.5GB primary ext4

     – Rajat Aug 03 '10 at 14:36
2

It depends on which filesystem you use. For ext3, there is a limit on 31998 subdirectories per directory. On ext3 it's also possible to run out of inodes before running out of free space if you have lots of very small files (the ratio of blocks per inode can be set a fs creation time).

More modern filesystems typically have higher limits, or are limited only by the available disk space.

That being said, that many directories or files is typically an indication that you're doing something wrong. Consider storing data in some binary scientific format such as netcdf, hdf5, or a database.

janneb
  • 3,851
0

You have an ext4 file system. The max sub directories with ext4 is 64,000.

EDIT: It can technically go higher using the "dir_nlink" feature of ext4.

Jacob K
  • 139