[Toybox] toysh - Bash interoperability, clang+musl build, testing (for corner cases)?

[Rob Landley]

On 12/25/20 3:45 AM, Thorben Brandt wrote:
> Hi there!
> 
> This is my first post to this list (any list, in fact) so please bear with
> me if I'm doing something wrong. I've been following the Toybox project for
> a while now after stumbling into a recording of one of Rob's talks on Youtube.
> When digging down into things recently, I was particularly delighted to see
> that toysh aims to adhere to the (quote unquote) bash standard, it was quite
> fun to follow through the bunch of threads regarding that subject.

Alas, it's only maybe 3/4 finished. But I have some time off for the holidays
and I'm poking at it.

> Here's some links to (in my eyes) particularly interesting threads on the
> list archive, as I'm not sure if slapping them in here as a bunch of quotes
> would be the right approach.
> 
> If anyone has a link to a paper titled something along the lines of "How to
> use a Mailing List as a 90s kid", I'll happily study it!
> 
> [ toysh vs dash ]
> 
> http://lists.landley.net/pipermail/toybox-landley.net/2020-October/012087.html
> 
>> I'm aiming to replace bash. I don't have a "remove any feature that's not in
>> posix" flag yet.
> 
> See above, this is what piqued my interest to the point of eventually signing
> upfor the list and browsing through it for the better part of an afternoon.

I should _add_ a "posix" flag at some point, but it's not that big of a stretch
given the current TT.options stuff.

> [ Would someone please explain what bash is doing here? ]
> 
> http://lists.landley.net/pipermail/toybox-landley.net/2020-June/011788.html
> 
>> I looked at containers for self-contained build and test environments about
>> three years ago, but the lack of a controlling terminal made it not useful.
>> Has that changed since? I haven't really looked.
> 
> Docker has supported allocation of pseudo-ttys and stdin streams for a while
> now [1] (am I missing something?), and systemd-nspawn [2] is intriguing but I
> haven't poked at it yet. Coming from a kind-of-sort-of background in working
> with containerization concepts (let's be real though, who doesn't these days)

I had a contract with parallels in 2011 porting container infrastructure from
openvz to lxc and the vanilla kernel, but there was some cultural conflicts
between me and one of the Russian developers and it didn't quite last a full 6
months. (Plus they gave me a list of 6 things I could work on and I said
"anything except NFS" so they assigned me to work exclusively on NFS.)

http://landley.net/lxc/

I probably wrote that because of https://landley.livejournal.com/55534.html

Anyway, I want to create a "contain" command for toybox that launches a simple
container configured entirely from the command line, but it's not in
https://landley.net/toybox/roadmap.html so it goes post-1.0. I've poked various
people and tried to keep up over the years, but it's a moving target and I
haven't got the bandwidth to focus on it just now...

> this is something I'd totally love to look into and help out with. Problems
> with interactive TTYs in containerized environments pretty much always stem
> from running something that doesn't gracefully handle signals as PID 1. But
> that's about all I know when it comes to the nitty-gritty of shells so I'd
> love to help and in return learn in the process!

I did oneit.c long ago because getting PID 1 to gracefully handle signals turned
out to be a whole _thing_...

> [ mkroot works again, pending release. ]
> 
> http://lists.landley.net/pipermail/toybox-landley.net/2020-October/012073.html
> 
>> I note that except for a couple large holes (lack of $((math)) and function()
>> support, haven't finished trap/signal/jobs, needs command line editing and
>> history) toysh is... sort of working now? Several known bugs (test suite
>> failures), but it's actually starting to be actually run real scripts. Needs
>> auditing for memory leaks, and the largest script I've thrown at it _is_ the
>> mkroot init, but... it's advanced from "don't bother" to "object of curiosity".
> 
> I suspect that hasn't changed since, alas the prospect of building a statically
> linked toysh and trying to daily it on "my actual computer" to in return bombard
> you with bug reports isn't quite viable as of yet, is it?

A few days ago E5ten told me he tried to run ldd and it exploded. I forgot that
ldd was a shell script, so I tried running it. I've fixed 3 bugs so far and have
2 more open in tabs. (This thing actually uses complex case statements in loops,
something my shell scripts have never done...)

> On that note, it's been an awful while (about 10 years) since I've had any
> meaningful exposure to C. I tried to work my way through the cross compilation
> notes and what makes the Android (clang + Bionic) and musl-cross-make (GCC + musl)
> processes work, but couldn't quite apply that to making the (new?) clang + musl on
> x86_64 combo spin (it eventually compiled but immediately crashed iirc, need to
> look into it again).

I've been meaning to combine clang and musl, but the llvm build prerequisites
are horrific because it's written in C++ and requires the 2023 standard to
compile, which the committee should finish arguing over sometime in the next 5
years...

(Me, I mostly stick with c99. I haven't even found anything in C11 yet that I
really care about, I don't use threading heavily enough.)

If you want a clang musl toolchain, I'd ask the musl maintainer (Rich Felker).

> If anyone has played around with that particular setup, let me know!

http://ellcc.org/ had one working long ago, but he got a job at Johnson Controls
in 2018 and hasn't really resurfaced since. (Plus his build wasn't "extract
these vanilla packages and apply these patches", it was "I've checked all the
source into my own one big giant source control tree and applied my own patches
there" which is not good project hygiene.)

> TL;DR: I'd like to help - to the extent of my abilities - with making the shell
> (more of) a thing in some way if I may, any starting pointers/feedback on my above
> interpretations of what's been going on would be greatly appreciated.
> I hope y'all are having nice holidays.

Real world tests are always nice. Give me a reproducible failing test and I'll
prioritize fixing it.

(That said, "echo $((1+2)) doesn't work yet" and function() isn't implemented
are known missing features, as is signal handling and job control...

Rob