14

Context for the question: According to POSIX specs, ARG_MAX is maximum length of command-line arguments to exec() family of functions. Which lead me to believe that's actual number of arguments, however that clearly didn't work:

$ ulimit -s
8192
$ touch {1..18000}.jpg
$ rm *.jpg
$

Clearly, this works fine, despite being in length over 8192 items. According to D.W.'s answer, the 8192 is supposedly size in kB. So clearly the previous assumption was wrong.

This is where the actual question comes in: How do I figure out the amount of items that actually will get above 8192 kB limit ? In other words, what sort of computation I have to perform to ensure that *.jpg type of glob will result into Argument list too long error ?

Please note, this isn't a duplicate of What defines the maximum size of single command argument. I know about getconf ARG_MAX and ulimit -s values, that's not my question. I need to know how to generate enough arguments in size that will be above the limit. In other words, I need to find a way to get the error, not avoid it.

Sergiy Kolodyazhnyy
  • 16,527
  • What is the result of getconf ARG_MAX for you? – jesse_b Jun 23 '18 at 18:03
  • @Jesse_b 2097152 . And I'm on Linux, btw – Sergiy Kolodyazhnyy Jun 23 '18 at 18:06
  • Are you trying to manually reproduce xargs? – Jeff Schaller Jun 23 '18 at 18:23
  • 1
    @JeffSchaller No, I wanted to generate enough files to play around with various methods of handling Argument list too long error. So far I've figured out ulimit -s gives stack space, ARG_MAX is subset to that. I see so far that getconf ARG_MAX gives basically 2M by default for argument size, 8M for stack space. So basically all I need to know is how to create enough filenames to go beyond 2M limit, so that when I do *.jpg in any command that involves exec() family of function would throw that error. – Sergiy Kolodyazhnyy Jun 23 '18 at 18:33
  • I think I figured out what needs to be done. I'll post answer once I confirm it works – Sergiy Kolodyazhnyy Jun 23 '18 at 18:40
  • 1
    Looks useful: https://unix.stackexchange.com/a/110301/117549 – Jeff Schaller Jun 23 '18 at 19:11
  • @JeffSchaller Yep, and I commented on Stephane's answer. I think that's almost what I need, but he didn't include calculations on how he got specific number that ends the sequence – Sergiy Kolodyazhnyy Jun 23 '18 at 19:50
  • you can guess at what the limit is with a minimized environment, e.g. https://github.com/thrig/scripts/blob/master/asm/Darwin/x86_64/t/echo.t#L59 – thrig Jun 23 '18 at 21:53
  • ulimit -s reports the stack size, which is unrelated to the size of the 'argument + environment' limit. – Jonathan Leffler Jun 24 '18 at 18:59
  • @JonathanLeffler Then how do you explain that when ulimit -s value changes, getconf ARG_MAX value also changes proportionally ? – Sergiy Kolodyazhnyy Jun 24 '18 at 19:07
  • I don't. I merely observe that on some systems (specifically macOS 10.13.5), using ulimit -a reports one line saying stack size (kbytes, -s) 8192, so on this system, ulimit -s directly affects the process stack size. Referring to POSIX is no help; its ulimit command only affects file sizes. I don't know why Linux changes ARG_MAX when the stack size changes. On a Mac, both before and after ulimit -s 2048, getconf ARG_MAX still reports 262,144. […continued…] – Jonathan Leffler Jun 24 '18 at 19:19
  • […continuation…] POSIX <limits.h> says: {ARG_MAX} Maximum length of argument to the exec functions including environment data. Minimum Acceptable Value: {_POSIX_ARG_MAX} POSIX execve() says: The number of bytes available for the new process' combined argument and environment lists is {ARG_MAX}. It is implementation-defined whether null terminators, pointers, and/or any alignment bytes are included in this total. – Jonathan Leffler Jun 24 '18 at 19:19

4 Answers4

10

Using getconf ARG_MAX to generate a long list of x and calling an external utility with that as its argument would generate an "Argument list too long" error:

$ /bin/echo $( perl -e 'print "x" x $ARGV[0]' "$(getconf ARG_MAX)" )
/bin/sh: /bin/echo: Argument list too long

The environment and the length of the string /bin/echo will be included in what makes the error occur, so we can try to find the biggest possible number by subtracting these:

$ env
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/X11R6/bin:/usr/local/bin

(I started this shell with env -i sh, so there's only the PATH variable in the environment)

$ /bin/echo $( perl -e 'print "x" x ($ARGV[0] - length($ENV{"PATH"}) - length("/bin/echo"))' "$(getconf ARG_MAX)" )
sh: /bin/echo: Argument list too long

Still too long. By how much?

i=0
while ! /bin/echo $( perl -e 'print "x" x ($ARGV[0] - length($ENV{"PATH"}) - length("/bin/echo") - $ARGV[1])' "$(getconf ARG_MAX)" "$i" )
do
    i=$(( i + 1 ))
done

This loop exits for i=8.

So there's four bytes that I can't immediately account for (four of the eight must be for the name of the PATH environment variable). These are the null terminators for the four strings PATH, the value of PATH, /bin/echo and the long string of x characters.

Note that each argument is null terminated, so the more arguments you have to the command, the shorter the combined length of them can be.

Also, just to show the effect of a big environment:

$ export BIG=$( perl -e 'print "x" x $ARGV[0]' "$( getconf ARG_MAX )" )

$ /bin/echo hello
sh: /bin/echo: Argument list too long

$ /bin/echo
sh: /bin/echo: Argument list too long
Kusalananda
  • 333,661
  • So ARG_MAX is actually a character count for the whole command line, or is it rather a buffer size? How does character encoding play into this? – dessert Jun 24 '18 at 08:21
  • 2
    @dessert ARG_MAX is the number of bytes that the combined length of the environment and the command argument list, including the command name and any null terminators on each argument (and on environment variable names/values), can be. So multi-byte characters would take more space. It may be a kernel or library buffer size. – Kusalananda Jun 24 '18 at 08:28
  • @dessert Unquoted whitespace between the command's arguments would not count, so it's not the length of the command line. – Kusalananda Jun 24 '18 at 08:29
  • That’s true, but if command path and environment are included it’s not just the list of arguments to that command (but to the shell I assume?), that’s what I meant. – dessert Jun 24 '18 at 08:44
  • @dessert Correct (although I don't think the arguments to the shell matter). See the end of my answer. – Kusalananda Jun 24 '18 at 08:45
  • This is pretty much what I needed. Good answer as always. – Sergiy Kolodyazhnyy Jun 24 '18 at 19:07
  • If you play with that on Linux, remember that there's the 128 kB limit for the length of a single argument. That's usually much lower than the limit for the total length. Since that Perl generates a single string (without whitespace), it'll hit that limit. – ilkkachu Jun 24 '18 at 19:25
  • @ilkkachu Oh? I did not know this. Is that really a limit on the size of a string, or is it actually PATH_MAX? – Kusalananda Jun 24 '18 at 20:05
  • 2
    @Kusalananda, I've understood it's the limit on the argument string size, it's discussed in the linked question. E.g. this would trigger it: /bin/echo $( perl -e 'print "x" x $ARGV[0]' 131072 ) > /dev/null (one less works). I think PATH_MAX is usually 4096 (again, on Linux), and I don't think it's related. – ilkkachu Jun 24 '18 at 20:14
  • 1
    @ilkkachu Thanks for pointing this out. That limit does not seem to be an issue on OpenBSD, but I'll find a Linux system somewhere and may update the answer when I have time. – Kusalananda Jun 24 '18 at 20:23
  • Sorry, i hope i'm not too late. I actually have mixed results, and i'm confused. Running which echo returns /bin/echo. And so , i ran echo \perl -e 'print "x" x 1000000'`` I got no error. But when i ran /bin/echo \perl -e 'print "x" x 1000000'``, i got the /bin/echo: Argument list too long. error. Why is that so? – lionel319 Nov 23 '18 at 03:14
  • @lionel319 which explicitly looks for external commands. If which says that echo is found at /bin/echo, this may still mean that your shell has a built-in echo that will take precedence over the external one. Test with type echo, or with command -v echo (does not show a path if it's a built-in). – Kusalananda Nov 23 '18 at 06:42
8

On many Linux distributions, you can find out what's the current value of ARG_MAX by running

getconf ARG_MAX

A simple way to generate the message "Argument list too long" is to actually supply a long argument list; for example, on my system

/bin/echo "$(find / -xdev 2> /dev/null)"

works, but

/bin/echo "$(find / -xdev 2> /dev/null)" "$(find / -xdev 2> /dev/null)"

produces

bash: /bin/echo: Argument list too long

For an in-depth discussion, see "Argument list too long error for rm, cp, mv commands" on Stack Overflow.

P.S. The length in question is the actual amount of memory needed to store the arguments and environment. It is not related to the number of arguments.

AlexP
  • 10,455
  • It's ever so slightly related to the number of arguments since each argument would need a null terminator. So having helloworld would use 11 bytes, while hello and world would use 12. – Kusalananda Jun 24 '18 at 08:36
6

Instead of listing all the files on the system, which takes forever and doesn't work on sparse systems, the command

/bin/echo {1..200000}

is a much faster (163ms) way to generate the error. For me ARG_MAX is 2097152 (2,097,152 on the order of 2 million) but the command still errors.

If your shell doesn't have {start..end}, you can use the somewhat slower (but still faster than listing files)

/bin/echo $(seq 1 200000)

If either command doesn't work, just increase the end value until it does, until bash runs out of memory, or until seq can't count any higher.

cat
  • 3,468
  • 4
  • 23
  • 51
6

To make it easier start a new shell (exit when done) and set the limit to a lower value (100kbytes):

$ bash
$ ulimit -s 100

With the example you used, the error could be triggered:

$ touch {1..100000}.jpg
bash: /usr/bin/touch: Argument list too long

But better to use something like echo. Understand that a builtin might not trigger an error, this command should work correctly:

$ echo {000001..100000}6789

Also note that the amount of bytes is well over the limit set above (1.1Mbyte):

$ echo {000001..100000}6789 | wc -c
1100000

But this command won't work:

$ /bin/echo {000001..100000}6789
bash: /bin/echo: Argument list too long

The limit is set by the size of the argument list in bytes added to the environment size of the shell.