[Aboriginal] dev-environment.sh depends on gnu coreutils

[Rob Landley]

On 03/11/14 03:29, John Spencer wrote:
> Rob Landley wrote:
>> On 03/02/14 13:57, John Spencer wrote:
>>> Rob Landley wrote:
>>>> On 02/28/14 09:02, John Spencer wrote:
>>>>> on my busybox install (sabotage linux):
>>>>>
>>>>> ~/qemu/system-image-sh4 $ HDBMEGS=2048 ./dev-environment.sh
>>>>> ./dev-environment.sh: line 44: truncate: command not found
>>>> It's been in toybox since 2011:
>>> so what ? it's by no measure a *standard* command at all.
>>
>> Please point me to the posix entries for chroot, eject, ifconfig,
>> insmod, losetup, netcat, swapon, which, whoami, yes, install, killall,
>> reboot... heck even _tar_ isn't posix (they went with "pax" and an ever
>> so slightly different archive format).
> 
> doesnt change the fact that tar is available everywhere, same as dd.

And so, these days, is truncate. That's sort of my point.

>> Widespread deployment of this command predates qemu's 1.0 release.
> 
> it's not about that some platforms have it, it's about that some
> platforms have it *not*.

Remember in 2010 how Ubuntu LTS didn't install "patch" by default? Or
the way Fedora didn't have a _toolchain_ unless you did a magic yum
incantation with a space in the name to install a dev envrionment or
some such (I kept having to look that up...)

Obviously, those aren't "real" commands then...

>>>> It used to be using that:
>>>>
>>>> http://landley.net/hg/aboriginal/rev/1630
>>> reasoning for replacing a standard command with a non-standard one:
>>> "It's been long enough since "truncate" was introduced"
>>
>> No, the reasoning is that dd can never create a fully sparse file, and
>> the truncate syntax for doing so is simple and straightfoward where the
>> dd syntax is anything but.
> 
> oh ? do you have more detail on that ?

Do I need to provide it? It's how the system calls work; dd does a seek
and a write. You can't create a file with zero allocations by writing
data to it, but writing 0 bytes is a NOP that won't extend the file
(unless the kernel's changed recently).

If your file _does_ have a dangling allocation at the end, and then the
disk is used for something else, the result tends to be a fragmented
file. It's not the end of the world, but it's untidy.

> i guess it would have been helpful to have this reasoning/information in
> the commit message then.

Sorry, I thought you knew. I've been stomping around in this area for
many years now. Here's from a decade ago:

http://lkml.iu.edu//hypermail/linux/kernel/0312.1/1195.html

>> Also that it had been in coreutils for 5 years and in BSD before that.
>>
>>> huh ? you're meaning introduced to toybox ? so do you want to force
>>> anyone that wants to use aboriginal to install toybox just to be able to
>>> use some shell scripts ? come on, this is silly.
>>
>> No, if that was the case I would have made Aboriginal use the "touch -l"
>> extension added to do this back in 2007:
>>
>> http://landley.net/hg/toybox/rev/162
>>
>> (Which was ripped out in favor of implementing the much more standard
>> truncate command.)
> 
> 
> following your logic, it's reasonable for the kernel developers to use a
> bit of perl here, a bit of bc there, and some python over here.
> because they all have been on ubuntu since 5 years.

You honestly don't see a difference between perl and truncate.

Truncate is a command I could (and _did) write a new implementation of
from scratch in under an hour. The Perl developers rather famously
failed to create a new implementation of their own project from scratch
(the parrot project, to unify some of its infrastructure with python)
over the course of something like 7 years.

Everything there is to know about truncate fits on one screen.
Everything there is to know about perl (especially given HPA's code was
pulling from cpan) does not exist in any single human's head.

These cases are the same to you? Really? You can't distinguish between
"must use microsoft word back when nobody's ever cloned it and we just
have the one magic implementation" and "I didn't like the one the system
came with so I wrote my own this morning"?

>>> my interest to compile alpha toybox is *zero*, and i dont plan to
>>> replace busybox with it until it's seen at least a decade of thorough
>>> testing in embedded products all over the world.
>>> so at the current development speed this is 20 years in the future.
>>
>> So you won't start using toybox for a decade after its 1.0 release. The
>> qemu 1.0 release was December 1, 2011. So presumably you'll start using
>> qemu in 8 years?
>>
>> No, that doesn't make sense. we're talking about the qemu wrapper
>> script, all it does is launch qemu. You must mean from the START of the
>> project. (In which case you've only got 2 more years to wait until it's
>> a decade for toybox.)
> 
> no, it doesnt just launch qemu, it also prepares your work environment -
> and that part is broken for busybox users.

It prepares a work environment _in_qemu_.

How does that invalidate the observation that the purpose of the script
is to launch qemu, and everything else it does is in service to
launching qemu? It can't NOT launch qemu and do anything useful, if qemu
isn't on the system this script is a brick.

And if you same objections were applied _to_qemu_, then qemu itself
wouldn't come close to passing by your criteria. Which makes your entire
line of argument kind of silly.

>> Let's see, the first commit in the qemu repository was February 18,
>> 2003, so congratulations on your one year anniversary of using qemu! I'm
>> sorry that my qemu wrapper script was only of use to you for part of
>> that time. (We _are_ still discussing the contents of the qemu launcher
>> script, right?)
> 
> you're mixing apples and oranges. qemu is not comparable to
> toybox/busybox: the former is a single application, the latter are
> collections of almost any standard utility. and i rather have these well
> tested.

So, you honestly don't know how to build+install a single command in
busybox without replacing the whole of busybox, even though busybox.net
explains how to do that in its FAQ?

  http://busybox.net/FAQ.html#backporting

That technique works fine for toybox too (make allnoconfig, make, mv
toybox truncate), but you can also do:

  tar xvjf toybox.tar.bz2
  cd toybox
  scripts/single.sh truncate
  ./truncate --help

I'm guessing that it's not that you don't know this, but that you don't
care.

>> Leaving aside your complete dismissal of my current main project as a
>> negotiating tactic... ("That new thing you're spending all your time on?
>> I will _never_ care about it. Now do what I told you to do on this other
>> thing you're only really still putting any time into as a test harness
>> for that new thing.")
> 
> i have no influence on what projects you hack on.

Given your discussion style here, I agree with that statement.

> i like and use your aboriginal project, but toybox is currently a
> building lot, incomplete, and lacking sufficient testing.

I'm honestly not trying to force anybody to use toybox. I'm also not
saying toybox originated this command. I'm saying it implemented this
command _years_ after coreutils did which was _years_ after freebsd did.

Busybox's failure to keep up is not my problem. Poke them. Or write a
standalone one in 15 lines of C.

Or, here's a fun one, google for "truncate.c" (with quotes so it
actually looks for that) and on the first page of hits find:

  http://src.gnu-darwin.org/src/usr.bin/truncate/truncate.c

> so i see no reason apart from GPL hate (which i dont feel) to leave
> busybox (which is almost feature-complete, well-tested, and has almost
> all commands i need) behind and switch to a software in pre alpha shape.

Obviously since I'm using bash you had to switch from busybox to
coreutils, because you can't use a different shell implementation when
busybox already has a shell implementation. It's utterly impossible to
disable a busybox command, or install things that aren't busybox on to
your system, or ask the busybox developers to add a trivial system call
wrapper command...

I'm really not following your argument here.

> this includes playing pre-alpha-tester for toybox via aboriginal.
> imo you shouldnt mix these two projects together.

Your opinion has been noted. Thank you for telling me what I should do
with my hobby programming time. At great length.

>> You _are_ aware that projects like busybox can build individual
>> commands, right? So you can just add the _one_ command without affecting
>> the rest? (Even if you don't want to use scripts/single.sh you can just
>> make allnoconfig, switch on truncate, make toybox, mv toybox truncate,
>> just like you would with busybox?)
> 
> the point here is that adding yet another command will just make my
> busybox binary bigger, and result in one more symlink in my already
> fullstuffed /bin. and having to add a new command just to make a single
> app that uses it happy kinda sucks.

$ ls -l /usr/local/bin/qemu-system-arm
-rwxr-xr-x 1 root root 5832320 Jul 8 2013 /usr/local/bin/qemu-system-arm

(Checking... yes, that's stripped, and not dynamically linked.)

So, a little under 6 megabytes, as opposed to:

$ wget http://src.gnu-darwin.org/src/usr.bin/truncate/truncate.c
100%[======================================>] 4,903  --.-K/s in 0.1s
$ gcc truncate.c -s -Os
$ ls -l a.out
-rwxrwxr-x 1 landley landley 10392 Mar 16 15:44 a.out

10k for the standalone BSD version, which built out of the box first try
for me. Riiight... I'm assuming the problem is a shortage of inodes
maybe? Have you tried installing busybox with hardlinks instead of
symlinks? (Each symlink takes an inode, each hardlink does not.)

Again, this script _launches_qemu_. That's what it's for. It does so
with some rather elaborate setup, but running qemu is the purpose of the
script, and you can't run the script in a way that _doesn't_ launch
qemu. If you want to use something other than qemu, you have to modify
or provide your own script.

And you're complaining about the addition of 10 kilobytes to an overhead
of 5.8 megabytes, so an expansion of 0.000086%.

>>> it's bad enough that you still stick with GNU bash scripts despite your
>>> GNU hate.
>>
>> I intend to implement a reasonable bash replacement shell in toybox.
>>
>> That said, a todo item near the top of my aboriginal todo list is seeing
>> how much of my scripts will run with busybox ash. (Last time I tried:
>> not much.)
>>
>> Alas, given that I'm mostly using aboriginal as a toybox test bed these
>> days and it gets about the fourth slice of my hobby programming time...
>> (There's a ton of stuff i want to do. And with my current dayjob, 3-4
>> hours of commuting time to and from work each day.)
> 
> i think others use aboriginal for other purposes, which do not involve
> testing toybox. and in my case, i would prefer if busybox was installed
> instead of toybox, so stuff (including date) would just work.

You noticed the "BUSYBOX=1" config option in Aboriginal's config, right?

QEMU is not in busybox. This is a wrapper script to launch QEMU.
Building QEMU these days requires all sorts of libraries, a fairly
elaborate build environment. That was the context within which I wrote
the script. the build system itself is recapitulating phylogeny and
carefully tracking the dependencies necessary to recreate itself from
first principles. But the emulator is a giant deus ex machina hairball
of extraneous crap I can't finitely define, and you get it from "elsewhere".

The emulator is optional. You can also run on real hardware. The qemu
launch scripts don't do that. They provide a template you can use to
write your own script to do stuff on real hardware, but in their current
form they do not. The one line patch to replace truncate with dd is
noise compared to the changes necessary to format an sd card or flash
through openocd or drive and expect to get u-boot to tftp fetch and boot
an image.

That said, I _do_ provide an implementation of the truncate command as
used by those scripts. Busybox doesn't, but aboriginal's default
configuration does. (I don't feel obligated to do so at the design
level, because it _is_ a qemu wrapper script and qemu is out of scope,
thus the script to drive qemu is out of scope.) But your refusal to use
the code I provided, or install any of the multiple alternatives I've
suggested, or patch it out locally, really isn't my problem.

> anyway, it was not my intention to turn this into a toybox rant.

*shrug*

>>>> Feel free to locally revert that patch, but the proper fix is either
>>>> for
>>>> busybox to implement truncate or for me to finish toybox. :)
>>> no, the proper fix is to use a portable command, i.e. revert the patch
>>> "globally" in aboriginal.
>>
>> Because me pointing out the patch you can revert locally is _unclean_,
>> and you're fighting on behalf of all those other silent people who can't
>> be bothered.
> 
> yeah, i joined your fight to get the perl dependencies out of the kernel
> build process, and now i fight to get the coreutils/toybox dependencies
> out of the aboriginal usage process.

And you don't see any difference between them.

Did you notice that I'm not pushing my bc removal patch upstream?
Introducing that was a nasty thing for Peter Anvin to do, it broke Linux
>From Scratch and buildroot and openembedded and so on.

But: it's a posix command, so the correct thing for me to do is
implement bc in toybox. (Which is horrible because it's an arbitrary
precision calculator, which is why I've been reading the libtommath code.)

The fact it was not humanly possible for me to do that with perl is why
I lobbied to remove the dependency on perl. There _is_ no specification
for perl. There is just a single implementation that grew by accretion
over multiple decades. (And on top of that the code peter stuck in the
kernel sucked in additional code from the CPAN repository for any
nondefault configuration, so it required extensions _beyond_ the base
perl to function.)

>> You won't use coreutils either, presumably until 4 more years have
>> passed and the truncate command has been around for your 10 year metric.
>>
>> You could always install that freebsd one, you only have to wait 2 more
>> years for that one to be "ripe".
>>
>> I assume asking the busybox developers to add a truncate command would
>> still saddle you with a ten year waiting period...
>>
>> You could write your own truncate, since it's a trivial wrapper around a
>> single syscall. The toybox one is 26 lines, half of which is the
>> menuconfig entry. Almost all of the actual work is command line option
>> parsing.
> 
> again, i just don't want to add yet another command just to make a
> single program happy.

Then patch it out.

> it doesnt matter how much code that program is, it
> matters that it sits in my /bin unused for 99.99999999999% of the time,
> so it just shouldnt be there to begin with.

By that logic you shouldn't have a "reboot" command either.

>> But then you'd have to wait 10 years.
> 
> hm ?

Well if I write code and you won't use it for ten years, even if you
inspect it, then presumably if _you_ write code it has to age for ten
years before it's worth using.

Strange that this prohibition does not apply to qemu for some reason,
and in fact half the point of aboriginal is to let you use new versions
of the kernel and busybox and such...

Ah, there's your solution! Use the previous release of aboriginal, and
never upgrade again, because no _new_ code can be worth using until it's
been around for 10 years, and thus a project whose releases are driven
by quarterly kernel releases of new kernel code can't possibly be of
interest to you. I'm constantly introducing new package versions that
destabilize stuff, that's the main reason for new releases, and
obviously that's a bad thing for you.

Besides, I'm currently working on a rewrite of ccwrap to let me switch
the whole thing from uClibc to musl libc, and you'd have utterly hated that.

Rob

 1395004821.0