[Toybox] [PATCH] hostname: fix behavior when in jail.

[Rob Landley]

On 1/25/19 5:06 PM, enh wrote:
> On Sun, Jan 20, 2019 at 10:06 PM Rob Landley <rob at landley.net> wrote:
>>
>> On 1/20/19 4:20 PM, enh wrote:
>>> the link in the roadmap (the checked-in copy; the web site is too old)
>>> is, for better or worse, a link to the current status:
>>> https://android.googlesource.com/platform/build/soong/+/master/ui/build/paths/config.go
>>
>> It looks like the remaining commands are:
>>
>>   awk bash bc bzip2 cp date dd diff egrep find fuser getopt git grep gzip
>>   hexdump hostname jar java javap lsof m4 openssl patch pstree python python2.7
>>   python3 realpath rsync sed sh tar timeout tr unzip xz zip zipinfo
>>
>> Triage:
>>
>> All this is out of scope for toybox:
>>
>>   openssl jar java javap python python2.7 python3
> 
> yeah, i'm pretty sure we already use prebuilts for the JDK and there
> _are_ prebuilts for python, so presumably we use those too. dunno
> about openssl, but obviously not your problem.
> 
>> The ones that should work already (or are probably trivially fixable, send test
>> cases) are:
>>
>>   cp date egrep find grep hostname patch realpath sed timeout tr
> 
> yeah, cp had the symlink issue (i just switched us back over so
> hopefully it sticks this time), date has all the weird shit that i
> need to wade through at some point,

Stuff like "yesterday" and "today" has to be special cased, but wouldn't be hard...

> find is at the bottom of my list
> because it was the first thing i had trouble with when i
> _accidentally_ tried to use toybox to build AOSP,

Testcase(s)?

> grep i haven't
> looked at yet because i feel weird saying "it's not ready to replace
> BSD grep on the system but, hey, who cares about the build?",

I added --color and fixed all the pendin tests, and just fixed Ed Maste's issue
yeserday. What's left?

> hostname
> just went back in with my DNS/jail fixes, patch looks like a pain,

Eh, not that much of a pain. I'm trying to cut a release this weekend and I'm
almost done with sntp, but lemme see if I can find time for patch.c fuzz (and
maybe to get the mkfs.vfat stuff I did over the holidays to a good stopping
point)...

If not, I can presumably do it right after the release. :)

> realpath i have notes saying "-s and --relative-to" but haven't looked
> at writing the patch yet,

Not in the ubuntu 14.04 man page. I need to get that devuan install finished and
move to a newer dev environment, I just refuse to use systemd. Let's see...

  http://man7.org/linux/man-pages/man1/realpath.1.html

What does "strip symlinks" _mean_? Just... remove the .. and . entries and
prepend $PWD to the start but never recognize anything as a symlink? Hmmm... ok,
xabspath() needs to be a wrapper around xabspath_from() that takes a char *cwd
argument, and "exact" needs to turn into flag bits the way DIRTREE_* and
MKPATHAT_* did.

#define ABSPATH_NOFILE 1 // current exact >0
#define ABSPATH_NODIR  2 // current exact <0
#define ABSPATH_NOLINK 4

*shrug* Seems doable...

> the sed with the -i and -z fixes seems to be
> failing obscurely and the build server isn't giving me enough^Wany
> information about what's wrong so that'll take more local prodding,

Another thing I poked at a few days ago was getting my old record-commands
plumbing moved into either toybox or mkroot. See "command logging wrapper" in
the old aboriginal linux FAQ:

  http://landley.net/aboriginal/FAQ.html#debug_logging

Which was these two at runtime:


https://github.com/landley/aboriginal/blob/master/sources/root-filesystem/sbin/record-commands

  https://github.com/landley/aboriginal/blob/master/sources/toys/wrappy.c

And this at build time:

  https://github.com/landley/aboriginal/blob/master/more/record-commands.sh

https://github.com/landley/aboriginal/blob/master/sources/utility_functions.sh#L114

Anyway, needed a bit more cleanup than the 15 minutes I thought (wrappy.c uses a
64k static buffer with no bounds checking, not adding a toybox command like
that), but it's a useful thing. You can see each command line, and from time to
time I've had something like "tee" in it that captured input and output so you
could see not just what previous command got run but what data it read and wrote
so you can see where the new implementation differs...

If you're wondering how I got this far...

> timeout needs a bunch of missing options, and my notes claim tr is
> still in pending, and my checkout seems to agree.
> 
>> These are in pending and complete-ish but I need to go over them. (They probably
>> work but are all in pending for a _reason_):
>>
>>   bc dd diff lsof tar xz
> 
> yeah, i don't want to have to deal with the inevitable fallout if you
> rewrite these.

I'm not currently planning a complete flag-day rewrite of any of them. They have
existing users, so need incremental cleanups.

lsof is more than half cleaned up already, most of my todo items there are
missing features (it doesn't do +D) and potentially more code sharing.

dd I have a fairly good theoretical handle on (I've done 3 abandoned cleanup
passes on it already, grrr) and the stuff that's been blocking me there is
mostly stuff you don't care about (ala making sure the input and output
granularity of transaction attempts match what was asked for, which nobody who
isn't running "strace" is likely to even notice as anything but performance; you
have to disable the nagle algorithm for it to affect network packetization, for
example).

xz has to go from "known working" to "smaller, streamlined, but doing the same
thing" in tiny steps because I am NOT reverse engineering that compression
algorithm the way I did bunzip2. But I've done that sort of cleanup before.
(Also there's like 3 different variants here, xz and lzma and 7zip, and the
algorithm has per-cpu compression types which seem like they're likely to grow
new stuff for riscy and so on...? I have reading to do there, the historical
public domain snapshot in toybox is several years old now, at what point do
version upgrades get forced by data format compatibility issues?)

The same "known working" transform mostly goes for bc: somebody already put a
lot of effort into that, I'm not starting over. Although really I'm only doing
bc because Peter Anvin broke the kernel with it after I finally talked the other
kernel devs out of his previous breakage (perl).

I kinda wanted diff to use patience diff, but it doesn't and I might give up on
that goal.

Tar I need more regression tests for. I know the kernel guys went with cpio
because gnu tar is a horrible next of corner cases, but I don't have an
exhaustive _list_ of those corner cases and posix is useless here.

> plus xz we actually have "real" xz in the tree, so i assume we'll just
> have a prebuilt of that.

Quite likely. I want a decompressor for mkroot so it can build source tarballs
in common formats, but that doesn't mean you'll use it any more than you'd use
the non-zlib gzip code.

>> These are todo list items I haven't properly started yet but more or less know
>> how to do. (I could say more about any of these if asked.)
>>
>>   awk bash/sh getopt git m4 pstree rsync unzip zip zipinfo
> 
> awk we have one-true-awk (though i'm trying to work out why i breaks
> the build),

I used to send issues to the busybox awk maintainer on a regular-ish basis when
I found something broken in aboriginal linux. :)

  http://lists.busybox.net/pipermail/busybox/2005-October/050760.html

Did that for years, most of them eventually got ironed out but it took a while:


http://lists.landley.net/pipermail/aboriginal-landley.net/2014-November/002384.html

The reason I haven't opened that can of worms yet is I have some idea of the
scope of the problem and want to be able to do it like sed (sit down and let it
eat 4 months of my life) rather than like dd (do some work, get called away
until I forget where I left off and what exactly I was trying to accomplish with
this set of changes, scrap it and try again, rinse repeat).

> m4 we have the GNU m4 because apparently the user(s) make
> heavy use of some kind of huge m4 standard library. the zip stuff
> we'll probably use libziparchive-based tools to match the platform.

Indeed. m4 isn't as bad as awk but it's got complications. That said, gnu
packages are gpl and the FSF broke it in 2007, and I'm definitely trying to
complete the circle without that...

But you can definitely get to a hermetic build faster with supplementary tools.
I'm trying to do a self-contained (auditable) base system with a minimum number
of packages, but that's not one of _your_ immediate goals.

I'd like to prioritize low-hanging fruit on my todo list that's useful to you,
but if you're going to be deploying zlib and boringssl's hash functions at
runtime because of optimization, it makes sense for you to build with them too.
(In that case, "simple" means one common implementation for all your use cases.)

I need to fill in those gaps for _my_ use case of "providing a package with no
external dependencies that provides the command line build functionality the
toolchain, libc, and kernel don't", but I don't expect you to ever use 100% of
those implementations in AOSP. (Except maybe as some sort of test mode.)

Rob