[Toybox] mkroot works again, pending release.

[Patrick Oppenlander]

On Wed, Oct 21, 2020 at 1:38 PM Rob Landley <rob at landley.net> wrote:
>
>
>
> On 10/20/20 7:14 PM, Patrick Oppenlander wrote:
> > On Wed, Oct 21, 2020 at 11:02 AM Rob Landley <rob at landley.net> wrote:
> >>
> >>
> >>
> >> On 10/20/20 5:23 PM, Patrick Oppenlander wrote:
> >>> On Tue, Oct 20, 2020 at 10:58 AM Rob Landley <rob at landley.net> wrote:
> >>>>
> >>>> I got toysh implementing all the bits that mkroot's init script needs, so I
> >>>> expect to cut a release in the next couple days. There's still a zillion
> >>>> dangling threads but I should be able to park it for now.
> >>>>
> >>>> If anybody wants to holler about something I should fix now, this would be a
> >>>> good time. I'm back in tokyo working on $DAYJOB again, and am likely to down
> >>>> tools on toybox after the release for at least a couple months.
> >>>>
> >>>> I note that except for a couple large holes (lack of $((math)) and function()
> >>>> support, haven't finished trap/signal/jobs, needs command line editing and
> >>>> history) toysh is... sort of working now? Several known bugs (test suite
> >>>> failures), but it's actually starting to be actually run real scripts. Needs
> >>>> auditing for memory leaks, and the largest script I've thrown at it _is_ the
> >>>> mkroot init, but... it's advanced from "don't bother" to "object of curiosity".
> >>>>
> >>>> Right now, useful bug reports would come in the form of "thing I could add to
> >>>> tests/sh.test and get to later", and keep in mind I've got...
> >>>>
> >>>>   $ wc sh.tests
> >>>>   845  3791 20673 sh.tests
> >>>>
> >>>> One or two of those already, not yet properly formatted into runnable regression
> >>>> tests. (And one of the current sh.test entries HANGS because a
> >>>> subshell-plus-redirection test fails to hand off a filehandle properly and we
> >>>> wait forever reading from nothing, so I need to teach tests to time out...)
> >>>>
> >>>> Rob
> >>>
> >>> Hi Rob,
> >>>
> >>> first, thanks for all the hard work!
> >>>
> >>> I had a go at using toysh instead of busybox ash and immediately ran into this:
> >>>
> >>> % ./toybox sh
> >>> $ for i in $(./toybox); do echo $i; done
> >>> $
> >>
> >> It ran for me?
> >>
> >>   $ ./toybox sh -c 'for i in $(./toybox); do echo $i; done' | wc
> >>     211     211    1355
> >>
> >>> That seems to start an interactive subshell somehow (takes two ^D's to
> >>> quit) instead of looping.
> >>
> >> Hmmm, how do I reproduce this...
> >>
> >> Rob
> >
> > Something strange is happening.
> >
> > % ./toybox sh -c 'for i in $(./toybox); do echo $i; done' | wc
> >       0       0       0
> >
> > Also, it takes a long time.
> >
> > % time ./toybox sh -c 'for i in $(./toybox); do echo $i; done'
> > ./toybox sh -c 'for i in $(./toybox); do echo $i; done'  0.00s user
> > 0.00s system 0% cpu 1.387 total
> >
> > Here's a what I'm doing on my end:
> >
> > % git describe --tags
> > 0.8.3-197-g9c8b4051
> >
> > % make allyesconfig
>
> Um. I haven't tested that in a couple months. (I should add a
> "CONFIG_BREAK_THE_BUILD" to that so people don't just blindly allyesconfig but
> have to switch something OFF to make it work.) Let's see...
>
> .....toys/lsb/md5sum.c:86:25: fatal error: openssl/md5.h: No such file or directory
>
> Doesn't compile for me, of course. I don't have the openssl libraries
> installed... The "ar.c" in my pending is half-finished, so is hd.c, mkdosfs,
> strace, I have a typo in my readelf...
>
> Ok, I have repeated your hang.
>
> Ah, of course, you enabled TOYBOX_FORCE_NOMMU which is using the "vfork, exec
> yourself, and send state data through a pipe" path because nommu systems haven't
> got fork(). I haven't tested that path since June, it's conceptually all there
> but hasn't been regression tested and has like 3 pending TODO items.
>
> The problem here is it's sending the whole -c line and not the contents of the
> $(./toybox) to the child process, so it's  looping forking off another child
> process lots of times. :)
>
> So yeah, allyesconfig throws #warnings during the build for a reason. I forgot
> people test like that.

Well that was a lucky coincidence then.

I initially observed the issue on my ARMv7-M NOMMU target.
"allyesconfig" was the closest hammer to get a toybox build with sh on
my host, and just happened to also show the same behaviour.

Thanks for looking into it.

Patrick