Command to Delete Millions of Files From Directory

Question

I have directory which contains millions of files of small size (output from a big data program). I need to delete those directories but when I type the standard rm * I get:

zsh: sure you want to delete more than 100 files in data/output [yn]? y
zsh: argument list too long: rm

The files all have the same prefix with a unique number appended after like this

data-12343
data-12344
... etc

So I can't even use regular expressions to trash the files in a piecemeal way. Looking for advice and tips on how to do this efficiently and in an automated way.

Thanks.

score 4 · Accepted Answer · answered Apr 28 '19 at 12:53

4

Use xargs to run your rm command. It will re-run rm, using up the maximum number of arguments each time, until it has completed your original intention of *.

In the words of the man page:

The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits. Some commands can usefully be executed in parallel too; see the -P option.

answered Apr 28 '19 at 12:53

Jeff

920

1

Great. I just did ls | xargs rm and it did the trick! – Apr 28 '19 at 12:55
3

@M.K. although that worked in this case, you shouldn't rely on the output of ls for things like this: a better option would be printf './%s\0' * | xargs -0 rm. Please refer to Why you shouldn't parse the output of ls(1) – steeldriver Apr 28 '19 at 14:59
2

I'm glad it worked for you. Similar to @steeldriver's comment, another option would be to use find and pipe the results to xargs like so: find . -type f -print0 | xargs -0 rm – Jeff Apr 28 '19 at 15:34
@steeldriver if rm * failed why would printf ... * work? – Chris Davies Apr 28 '19 at 16:06
@roaima I believe it's because (at least in zsh, which the OP is using, and bash) printf is a shell builtin and so not subject to the ARG_MAX limit – steeldriver Apr 28 '19 at 16:32

Command to Delete Millions of Files From Directory

1 Answers1

Linked