[Toybox] [landley/toybox] xargs: exec ... Argument list too long (#40)

[enh]

On Wed, Aug 10, 2016 at 11:28 AM, Rob Landley <rob at landley.net> wrote:
> On 08/10/2016 07:44 AM, Matthias Urhahn wrote:
>> On some Android devices (N5 at 6.0) using a chain of |find .... | xargs
>> .... stat ....|, yields the error |Argument list too long|.
>>
>> Some googlfu shows that this is likely related to the argument exceeding
>> the kernels ARG_MAX value.
>> Androids ARG_MAX according to this 131072
>> <https://github.com/android/platform_bionic/blob/master/libc/kernel/uapi/linux/limits.h>.
>
> The Linux kernel removed that limit in 2007
> (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b6a2fea39318
> went into the 2.6.22 release).
>
> That said, let's cd to a directory with a lot of files in it:
>
>   $ find . | wc
>     68971   69024 3159459
>   $ find . | toybox xargs echo
>   xargs: exec echo: Argument list too long
>
> And I'm seeing a limit here too. Hmmm...
>
>> Manually limiting the xargs argument length using |-s bytes| fixes this
>> for me.
>
> The kernel should not be enforcing this? Maybe something else is? There
> isn't an obvious ulimit for this... Ah, in include/uapi/linux/limits.h:
>
>   #define ARG_MAX    131072 /* # bytes of args + environ for exec() */
>
> And that's used in fs/exec.c:
>
>   /*
>    * We've historically supported up to 32 pages (ARG_MAX)
>    * of argument strings even with small stacks
>    */
>   if (size <= ARG_MAX)
>     return page;
>
>   /*
>    * Limit to 1/4-th the stack size for the argv+env strings.
>    * This ensures that:
>    *  - the remaining binfmt code will not run out of stack space,
>    *  - the program will have a reasonable amount of stack left
>    *    to work from.
>    */
>   rlim = current->signal->rlim;
>   if (size > ACCESS_ONCE(rlim[RLIMIT_STACK].rlim_cur) / 4) {
>     put_page(page);
>     return NULL;
>   }
>
> Ok, so that's still there as a MINIMUM, and the maximum is 1/4 the
> current stack size ulimit. According to ulimit -s that defaults to 8192
> kilobytes (at least on xubuntu 14.04), so 1/4 of 8 megs is 2 megs. And...
>
>   $ find . | head -n 40000 | wc
>     40000   40002 1853620
>   $ find . | head -n 40000 | toybox xargs echo | wc
>         1   40002 1853620
>
> Yup, the _actual_ default limit is way more than 131072, and that
> default is arbitrarily adjustable even for normal users:
>
>   $ ulimit -Hs
>   unlimited
>   $ ulimit -s 9999999
>   $ find . | toybox xargs echo | wc
>         1   69023 3159436
>
> (Huh, what happens if I set the stack size limit to _less_ than 131072
> and then try to exec stuff? Not trying it on my work netbook just now,
> thanks...)
>
>> I've looked at the code for xarg in toybox but could not figure out yet
>> how the max argument length is determined.
>
> It didn't enforce a default maximum length limit because as far as I
> knew the kernel hadn't required one for 9 years. I worked out the math
> for enforcing such a limit back when I first did xargs:
>
>   http://landley.net/notes-2011.html#17-12-2011
>
> So it's not hard to add a default, but what the default should be isn't
> obvious. Back when I noticed the kernel had changed its logic I
> _checked_ that I could feed it more than 131072 bytes, and I could, so I
> removed the limit. (Not just from xargs, but from find in commit
> aa784b09a9fb and possibly elsewhere.)
>
>> Busybox doesn't seem to run into this issue using the same command on
>> the same device. Busybox seems to have some sanity checks
>> <https://git.busybox.net/busybox/tree/findutils/xargs.c#n529>, adjusting
>> the argument length if necessary.
>
> The sanity checks are there (-s is implemented), toybox just doesn't
> have a default value if you don't specify one.
>
>> Could there be an issue with how toybox's |xarg| determines the default
>> argument length on Android?
>
> See above.
>
> There's a sysconf(_SC_ARG_MAX), which does appear to be calculating this
> properly (at least built against glibc, musl is giving me 131072 and I
> still haven't got a host bionic toolchain because building bionic
> outside of AOSP was nontrivial when I looked at it).

yeah, bionic currently just returns ARG_MAX. looks lik glibc divides
the stack by 4 as you describe above.

running

#include <unistd.h>
int main() {
 return sysconf(_SC_ARG_MAX)/1024;
}

under strace gets me

getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
exit_group(2048)                        = ?

> Oh well, use the sysconf and then file a bug against musl. (Elliott will
> probably notice this, and if not he'll notice the commit.)

yeah, this is as good a bionic bug reporting forum as any :-)

> Thanks for the heads up,
>
> Rob
> _______________________________________________
> Toybox mailing list
> Toybox at lists.landley.net
> http://lists.landley.net/listinfo.cgi/toybox-landley.net



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.