[Toybox] Toybox - size, what's in it and command overlays!

[Rob Landley]

On 05/15/14 23:01, David Seikel wrote:
> On Fri, 16 May 2014 13:15:19 +1000 scsijon <scsijon at lamiaworks.com.au>
> wrote:
> 
>> Considering what is now within toybox and the increase and spread of 
>> command content range it has and is being added.
>>
>> I was wondering if it's time to think about splitting some of it's 
>> content out into a number of smaller group command overlay sets with
>> the primary command set being able to load and unload the others on a
>> need or as-used basis.

The kernel's ELF launcher mmaps the underlying file, jumps to its entry
point specified in the ELF header, and demand faults in pages as the
processor tries to access them. (In the case of a dynamically linked
executable, the kernel actually mmaps the shared library loader and then
_it_ mmaps the main program, reads the table of libraries it addresses
and mmaps those files too, and then reads tables in the files to find
pointers in one that need to point into the other.)

I don't know what you mean by "overlay" here, the last time I
encountered a program using overlays was DOS because the operating
system there didn't have a MMU so it couldn't swap in/out at the page
level. Not hugely interested in reimplementing things the operating
system (or C library) should be doing for us.

>> I think it would have to be considered soon / now, otherwise like 
>> busybox has become, it would become unreasonable to do because of the 
>> rewrite and redesign work required.
>>
>> If anyone needs / wants more expansion. I'll add on request. I will
>> add that i'm not a programmer, so what you get will need to be
>> tempered by that.

I admit I don't understand exactly what's being requrested here.

> One of the design goals is to have just one file to upload to give you
> a known set of useful commands.  If you statically link toybox (I think
> that's the default) then that's what you get, a single file with no
> dependencies (other than a reasonable kernel).

Yup. Static linking isn't the default, but it's fairly easy to do:

  CFLAGS=--static make

Another design goal is that adding a command means adding a single .c
file that you drop into any of the directories under "toys" and if it's
got all the data it needs, ala:

  http://landley.net/toybox/code.html#adding

It should just work. Adding a new directory under there's pretty easy
too, just add the directory and put a README file in it with the first
line being how you want the directory described in menuconfig.

Having multiple directories under toys is largely cosmetic, but it makes
dealing with large numbers of commands a bit easier. I'm thinking of
adding "login" and "network" directories so "other" doesn't get too big...

> It's also possible (or at least is planned) to compile individual
> commands.  Nothing stops you from compiling several sets of commands,
> each under a different name.  There is minimal overhead on both of
> these options, so I don't see what the gain is in doing what you
> suggest.

It's there now: scripts/single.sh $COMMANDNAME

I need to upgrade that so it can build OLDTOY() as well as NEWTOY().
(And be less confused by commands with multiple NEWTOY() in them.) But
it can already build the majority of the commands standalone.

> Unlike busybox, I would think that Rob can keep a tight lid on the
> spread of commands being added to toybox.  Another of his design goals.

Goal, yes. Clear bright dividing line, no.

There's a largeish "requests" section at the end of the roadmap where
people with different real world use cases than mine want things I don't
personally care about. Never undertood the point of "bootchartd" for
example, but the CELF guys offered a bounty to add it to busybox a few
years back, so it's obviously important to somebody:

http://elinux.org/Add_bootchart_boot_logger_functionality_to_busybox

I'm reluctant to declare other people's use cases invalid just because
it's not what I personally need. There's a certain amount of "driven by
users" going on here. But I am trying to keep it down to a dull roar. :)

Something I'm personaly wrestling with is whether or not I need to build
https support into toybox. 5 years ago the answer was clearly "no, use
an external stunnel", but these days you can't wget from all sorts of
places without https anymore (github, kernel.org, twitter...) and that's
only going to increase in future. Having https support is more or less a
requirement for doing http these days.

But if I open that can of worms, my personal netbook becomes a target of
the increasingly insane out of control NSA here in the US. Then again
I'm sitting here with a band-aid over the camera because I assume they
already ARE doing that, so I can't exactly call that a deterrent. (If
you assume the worst about the NSA, you'll still undershoot. Yes, the
endless outsourcing budget means there are something like a million
snowden-style conractors on the payroll to the point where they could
dedicate me a full-time babysitter, and opening that can of worms makes
a large headache for myself preventing Junior Sysadmin Man from
interfering with the normal operation of my laptop with Clumsy Bad VNC.
But again: I assume they're doing that _anyway_, and random black hats
deleting or corrupting code on my laptop behind my back isn't that much
different than a hard drive crash. This is what multiple backups and
distributed public development where everybody has copies of the repo
are for...)

*shrug* Probably a post-1.0 thing. (The real problem with https is it
needs a pile of root keys to authenticate site certificates against. We
don't require external config files, and I wouldn't want to build a
fixed set of those in...)

Rob

 1400236285.0