[Toybox] Out of tree build support

Thu Dec 21 09:16:06 PST 2017

On 12/19/2017 06:25 PM, Patrick Oppenlander wrote:
> On 20/12/17 09:24, Rob Landley wrote:
> 
>> Really what I need to do is come up with a way to build toybox sed and
>> toysh standalone from a canned script with no config step, and then
>> build the rest with that.
> 
> Makes sense.
> 
>> Speaking of which, did I ever point you at the cp -s trick?
>>
>>    $ cp -rs "$PWD"/toybox walrus
>>    $ cd walrus
>>    $ make distclean && make defconfig && make
>>
>> (I taught toybox to work with relative rather than absolute paths. The
>> gnu/dammit version complains about relative paths, and yes that means
>> even "cp -rs toybox walrus" won't work unless you're using toybox cp.
>> It's because they don't bother to calculate the dynamic ../ stack.)
> 
> That is a neat trick. First time I've seen it, thanks!

https://xkcd.com/1053/

(I keep thinking the fact I've repeated something 30 times over the
years means people are sick of it, when most of them have never heard of
it. One of the reasons I'm so interested in computer history is
preserving what used to be common knowledge, because
https://web.archive.org/web/20120111055334/http://wrttn.in/04af1a having
to come from archive.org sadly _isn't_ ironic, it's harmonic. It
_reinforces_ the darn point.)

>> One of my todo items is writing a new kconfig to replace the kernel
>> plumbing I borrowed way back when. (Which is from 2.6.12.)
> 
> Even before that the issue is that there's no way to build kconfig from
> a script, so the *config targets depend on make.

There's no way to build the _current_ kconfig plumbing from a script.
But if I write new infrastructure...

(Alas it's another thing that's beyond the threshold of what I can sit
down and do in a couple hours from a standing start, which would be
negatively impacted by interruption and forced context switch...)

>>> I've pushed the current state of affairs to
>>> https://github.com/pattop/toybox if you're interested in taking a look.
>>
>> Hmmmm... I see what you're doing, but you're adding complexity to the
>> Makefile and I want to strip it out and move it into the scripts.
> 
> I think the changes to the top level Makefile aren't too bad. They
> really just boil down to teaching make where to find sources & scripts
> by sprinkling in a few $(SRCDIR)s and don't change any of the logic at all.

Complexity often accumulates in small amounts. I'd prefer to solve this
by _removing_ stuff from Makefile instead of adding it.

Looking at the design, I suspect I should rename "scripts" to "make" and
have a subdirectory under there for files that aren't a build target
(make/include or something).

Then the top level Makefile should be a wrapper that _just_ calls
scripts under "make". For the moment kconfig/Makefile can stay crazy
because that whole mess is its own can of worms.

But I need to close tabs before opening a new one just now...

> kconfig/Makefile probably needs to be replaced by a script so the
> hackery in there is fairly moot. Would you be interested in a script
> that replaces this?

Somebody wrote some plumbing to (partially) replace it in awk once, but
I just couldn't maintain it. :(

> Are you happy with the direction of the changes to the scripts in general?

Sort of?

Make provides an expected command line API. The classic build is
"./configure ; make ; make install" and long ago I ranted (in two parts)
about how that's annoyingly obsolete at:

http://lists.landley.net/pipermail/aboriginal-landley.net/2011-June/000859.html

http://lists.landley.net/pipermail/aboriginal-landley.net/2011-June/000860.html

But we haven't standardized on something new and better yet (the way cvs
gave way to git). Instead we've got fragmentation: the gnu guys did
their own "automake" syntax which is almost sane but nobody else picked
it up because it had gnu all over it and those were the guys who did
"info".  Android decided to use something called ninja while
simultaneously switching to an llvm toolchain that wants to build with
cmake... Despite the many downsides of make, it has the advantages that
it's posix and still near-ubiquitous.

Getting back to configure/make/install: the Linux kernel swapped out
configure for "make defconfig" and friends, and as with Linux switching
to git lots of people are familiar with that now. It was copied by
busybox and uclibc, spread from there to buildroot (which started life
as uClibc's test harness), and now openembedded and so on have taken it
from there...

That said, we shouldn't be tied to a given implementation of either. An
implementation is not a standard, so you can't _have_ a standard until
there are two compatible implementations. (This was official IETF policy
at one point, dunno if it still is.)

So I want to write my own mostly-bash-compatible shell, I want to write
my own mostly-gmake-compatible make, and I want to write my own kconfig
plumbing.

Long ago I tried to push some of my kconfig changes from busybox
upstream into linux-kernel, this would have been maybe 2007? And it got
derailed into Matt Mackall and such bikeshedding about how if kconfig is
going to become generally useful for other projects it needs to become
its own separate package, and this blue sky plan was used as an excuse
not to take my patches unless I volunteered to do an order of magnitude
more work, and as usual I went "nuts to your white mice" and went back
to what I was doing because I was _already_ three interrupts deep from
what I was _trying_ to work on...

Anyway, the point is if I write a new menuconfig implementation that's
BSD licensed, there's a nonzero chance other projects might pick it up.
I only personally use menuconfig and the command line ones, so I'll
leave the qt/gtk arguments to somebody else (see "delegate to nobody" in
my standard Linux gui rant).

But again: too many open tabs just now.

> It might be worth considering passing $(MAKECMDGOALS) to make.sh (or a
> higher level driver script) at some point.

Then the layers have an unbounded API between them.

> It could then call single.sh
> or test.sh and eliminate the requirement for .singlemake.

You can already call "scripts/single.sh command", singlemake just
provides make targets for each command and its test_command. Having
make.sh be a dispatcher to other scripts on top of what it already does
is not an improvement.

> You could then
> call make.sh from a catch all rule in the top level makefile and really
> turn the makefile into a trivial wrapper.

Not quite the direction I want to go in, thanks.

>>> The symlinks in kconfig are a bit nasty (they really should go in the
>>> build directory to avoid polluting the source dir, but as they're always
>>> the same...). I got this working with vpaths earlier but wasn't too
>>> thrilled with the result.
> 
> What's the reason for the symlinks in kconfig anyway? I had a quick look
> through the history and couldn't find anything obvious.

Look at linux/scripts/kconfig in the kernel source. I made minimal
changes because at the time I cared about tracking upstream (and then
never did), and because I want to throw out that whole directory and
replace it with a fresh implementation in C.

I note that menuconfig, vi, and screen have similar plumbing. I
prototyped that plumbing a bit with hexedit and top (the
interestingtimes.c stuff which is my not-curses), but the guy part is
only half of it, the reading/writing/dependency resolution part is the
other half.

Which HAS BACKSTORY. (As everything I do these days seems to.)
Specifically, I tried to push some simple miniconfig support upstream
into the kernel in 2005:

  https://lwn.net/Articles/160497/
  https://lwn.net/Articles/161086/

Which turned into some dude named Roman Zippel bikeshedding at me to try
to get me to go solve some unrelated problem for him and do 10x as much
work as I'd done and he'd hold my patch hostage until then, so I went
away and tried again a year later:

  http://lkml.iu.edu/hypermail/linux/kernel/0607.0/1805.html

And it was still bikeshedding, so I went off and did my own support for it:

  http://landley.net/aboriginal/FAQ.html#dev_miniconfig
  https://github.com/landley/aboriginal/blob/master/more/miniconfig.sh

And pretty much ignored the kernel guys. Meanwhile, the kernel grew some
crazy new stuff:

https://github.com/landley/aboriginal/blob/master/sources/patches/linux-deeplystupid.patch
https://github.com/landley/aboriginal/blob/master/sources/baseconfig-linux#L62

So I dug into the kconfig source a bit to see if a proper "make
miniconfig" mode would be easy to do (spoiler: nope), and it's been
"write a new one" ever since.

But I used some of that experience writing
https://github.com/landley/toybox/blob/master/scripts/config2help.c and
every time I poke at that file I get distracted by "I should write a
full kconfig replacement and make this a function of it", which is a
problem significantly larger in scope than what it does...

>>> CC'ing you as I don't seem to be getting list emails at the moment.
>>
>> I seem to be, but I think I've been cc'd on everything the past couple
>> days anyway?
> 
> I'm not receiving emails for this thread. Maybe the mailing list is
> smart enough to know you're CC'ing me?

Ah, you're using gmail. This is a known perennial bug in gmail:

  https://landley.net/notes-2011.html#05-04-2011

Which the gmail developers insist is not a bug, it's a feature:

  https://landley.net/notes-2011.html#30-04-2011

But it's a feature that can't be worked around if you're using
Thunderbird due to conflicting hardwired assumptions:

  https://landley.net/notes-2012.html#15-10-2012

And so on, and so forth. (Everything is backstory today. I should step
away from the computer.)

Rob