[mkroot] Contribution

[Rob Landley]

On 03/06/2018 07:50 AM, Ruhollah Majdoddin wrote:
> 
> 2018-03-05 17:36 GMT+01:00 Rob Landley <rob at landley.net <mailto:rob at landley.net>>:
> 
>     On 03/02/2018 04:56 AM, Ruhollah Majdoddin wrote:
>     > Hello,
>     > I have spent the past month over mkroot (and native build with QEMU),
> 
>     What do you think of it so far? (It's a newish project I haven't gotten too much
>     feedback on yet...)
> 
> I started in 2015 with Aboriginal Linux. I was searching for such a thing to
> avoid Yocto for arm embedded devices.
> I got a linux and compiled Lighttpd and White-Rabbit PTP natively. My chef was
> satisfied.
> 
> Last month, in my new job, I tried it again. Your tutorial presentation was very
> informative (I wish you could do more like that).
> There was two issues.
> 1- The company runs a lot of software on their embedded device. So it seemed
> that we needed a Linux-Distribution to manage package dependencies.
> 2- Mkroot still does not provide a native build system.
> 
> Therefore I tactically decided to use Debain Arm ports with QEMU.
> I got a running Debian-Linux on QEMU, with that I used Gnu toolchaing to compil
> the kernel from Boundary-Devices for the I.MX6 chip and run Debian with it (I
> had to append the defconfig of mkroot when compiling the kernel, to be able to
> run the image on the QEMU with Arm virt machine).
> Finally I compiled and installed some modules from the company, I got the
> system-image and installed it on the embedded device. But the chef and other
> colleagues were not very excited (they feel comfortable with Yocto/OE) and I got
> fired.
> I can say that your presentation and mkroot, served as a guide and test bench,
> to say the least.
>  
> 
>     > now I have
>     > free time for one or two months. Where can I contribute?
>     > If toybox has more priority now (I see more activity there)  please tell me
>     > where can I contribute there (I have browsed through the roadmap and other
>     > documentation).
> 
>     Mkroot needs a lot of work. I haven't got the native toolchain working under it
>     yet (partly for design reasons, there's no non-gpl make, and should ninja or
>     cmake be in there too?), the old build control image infrastructure at
>     https://landley.net/aboriginal/control-images
>     <https://landley.net/aboriginal/control-images> needs to be redone in the new
>     context, the old Linux From Scratch build image there needs to be recreated
>     using current LFS... But I'm not sure any of that's a good starting place for a
>     new developer?
> 
> 
>     Reproducing the existing build with other toolchains might be nice too (android
>     NDK, bootlin's toolchains, the crosstool images on kernel.org
>     <http://kernel.org>, https://ellcc.org
>     and so on), but none of them provide native compilers, but they might start and
>     it would be very nice to have an llvm build baseline.
> 
> This is interesting, it seems crosstool images are derived from bootlin.

Crosstool-ng is a longstanding project, although it looks like
http://crosstool-ng.org redirects to github now. They're build seemed hugely
overcomplicated to me last time I looked at it, but people have gotten it to work.

> Is Compert-C also an option? http://compcert.inria.fr/compcert-C.html
> Although it is just a (formally verified) compiler, no assember and no linker.

Personally I consider formal verification a downside (it means you're dealing
with math people who think that helps), but sure?

There's a bunch of compilers out there, from the amsterdam compiler kit to
https://github.com/drh/lcc and http://open-watcom.github.io/open-watcom/ and
https://developer.amd.com/x86-open64-compiler-suite/ and http://pcc.ludd.ltu.se/
and https://github.com/libfirm/cparser and so on.

However, there are four compilers I'm aware of that have built a bootable linux
kernel, and of those https://bellard.org/tcc/tccboot.html last did so in 2004
and
https://software.intel.com/sites/default/files/article/146679/linuxkernelbuildwhitepaper.pdf
is a proprietary intel thing.

That leaves gcc and http://llvm.linuxfoundation.org.

And the interesting part about llvm is that Apple sponsored its development (I
think it was some grad student project, but when gcc when gplv3 apple froze
xcode on the last gplv2 versions of gcc and binutils for 5 years while they
threw money at a replacement, and wrote their own samba from scratch and so on,
ala http://meta.ath0.com/2012/02/05/apples-great-gpl-purge/). Along the way
Google picked it up and has been building Android with it (including the kernel)
for years now. The two compilers were available alongside each other for a
while, but android deprected gcc in 2015:
https://news.ycombinator.com/item?id=10772826

You'll notice the toolchains I pointed at are all gcc or llvm based. (More or
less "toolchain distros" of those two base packages.)

It's a bit like "I've gotten linux and freebsd to work" and you going "how about
irix?" Um... technically yes? Good luck with it? I think you'll have a hard time.

>     For toybox I want to use mkroot as a way to run tests requiring root access and
>     a known build environment, but again that's not necessarily a good starting
>     point for new developers. (How you do proper "top" tests is a hard problem.)
>     Fixing existing test failures might be, though? (Or at least diagnosing them, I
>     haven't had time to even sort which test scripts pass and which fail, and which
>     ones are actually providing reasonable test coverage and which aren't yet. It's
>     on the todo list...)
> 
> This is also interesting. I think you mean the scripts under toybox/tests.

Yeah, "make tests" and "make test_sed" and so on. It's all calls to
scripts/test.sh (which calls scripts/runtest.sh) to call those scripts.

The most interesting controlknobx are TEST_HOST=1 and VERBOSE=fail, ala:

VERBOSE=fail TEST_HOST=1 make tests

Which when I tried it just went:

FAIL: blkid file
echo -ne '' | blkid temp.img
--- expected	2018-03-06 11:27:47.092149228 -0600
+++ actual	2018-03-06 11:27:47.100149229 -0600
@@ -1 +1 @@
-temp.img: TYPE="squashfs"
+temp.img: TYPE="squashfs"
make: *** [tests] Error 1


Because the host version of blkid outputs a trailing space and mine doesn't.
(Technically that's a bug in the host blkid, but sometimes I need to filter the
output so they match. That said, trailing whitespace doesn't come up much
because it generally IS a bug and if _mine_ was doing it I'd wanna know...)

The toybox test suite is an ENORMOUS time sink, and part of it's figuring out if
you're testing the right stuff (what tests are missing, is this test
meaningful...) And I have lots of stuff that needs root access to test, and more
that needs a known environment (ifconfig against which interface names, is the
dummy driver loaded, etc?) which is why I'm setting up a VM for it.

>     Something I've hit that might be worth a look is actually a qemu issue. I test
>     new mkroot builds with something like:
> 
>     for i in $(./cross.sh); do (sleep 5; echo date; sleep 1; echo ifconfig; sleep 1;
>     echo tail /dev/?da; sleep 1; echo exit; sleep 2) | ./run.sh qemu-$i |
>     smokelog-$i.txt; done
> 
>     But when I do that, qemu often _hangs_ with current -git. Something about qemu's
>     stdio console handling is wonky when dealing with pipes instead of pty
>     instances, and I'm not sure what.
> 
> 
> The Qemu problem is what I have been working on the last two days, my laptop is
> not fast, it got 9 hours to build musl. Actually, I could not reproduce the
> problem, it worked for me (but see below).
> 
> My system: x86_64 Sony Vaio Laptop
> $ uname -a
> Linux ruhollah 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3+deb9u1 (2017-12-23) x86_64
> GNU/Linux
> 
> I cloned QEMU from git, compiled the Master branch with GCC (Does the toolchain
> matter?)
> 
> ~/qemu/build/x86_64-softmmu$ ./qemu-system-x86_64 --version
> QEMU emulator version 2.11.50 (v2.11.0-1911-g136c67e078-dirty)
> 
> I built mkroot for x86_64 with GCC from debian,   and mkroot for armvl with
> musl, and did similar tests with similar results.
> 
> ~/mkroot/output/armv7l$ (sleep 7; echo ls; sleep 1; echo date; sleep 1; echo
> tail '/dev/pts' ; sleep 1; echo ifconfig; sleep 1; echo exit; sleep 1) | 
> ./qemu-armv7l.sh
> .
> .
> .
> This architecture does not have kernel memory protection.
> random: fast init done
> Type exit when done.
> / # ^[[50;5Rls
> bin dev etc home init lib mnt proc root sbin sys tmp usr var
> / # ^[[50;5Rdate
> Tue Mar  6 11:11:23 UTC 2018
> / # ^[[50;5Rtail /dev/pts
> / # ^[[50;5Rifconfig
> eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56  Driver virtio_net
>           inet addr:10.0.2.15  Bcast:10.255.255.255  Mask:255.0.0.0
>           inet6 addr: fec0::5054:ff:fe12:3456/64 Scope: Site
>           inet6 addr: fe80::5054:ff:fe12:3456/64 Scope: Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:1 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:134 TX bytes:512
> 
> / # ^[[50;5Rexit
> reboot: Restarting system
> ~/mkroot/output/armv7l$ ;5R;5R;5R;5R;5R
> 
> As you see it does not hang, but there is always a "^[[50;5R" on the QEMU
> prompt.

That's busybox attempting to probe the terminal size using ANSI sequences, and
technically it's my fault. :)

http://lists.busybox.net/pipermail/busybox/2009-May/069314.html

It can't tell _not_ to do that because inside the vm /dev/console looks like a
tty. (It should figure out it doesn't get a response and remember it, but nobody
implemented that. It's on my todo list for toysh but I need to clear space to
work on that which means shoveling out the backlog of so much _else_ first...)

> Afterwards I got some ";5R" on the prompt and I have problems with my
> bash settings (I‌ use Konsole): a long line does not  wrap and overruns itself.

If the ansi probe makes it to your terminal, your terminal will respond
basically pasting an ansi escape sequence in as if you'd typed it. But since
qemu's input isn't coming from the keyboard it gets buffered until something
listens to the keyboard again...

(The probe is "query cursor position". You can query terminal size by saving the
cursor position, moving to the bottom right corner, querying position, and
returning to saved position.)

See https://github.com/landley/toybox/blob/master/lib/interestingtimes.c#L49 for
sending the probe and
https://github.com/landley/toybox/blob/master/lib/interestingtimes.c#L164

(Of course you can't tell the difference between the return string from querying
the screen size and querying the cursor position, so you have to remember what
probe you sent...)

> If I redirect the output of QEMU to a file,
> 
> ~/mkroot/output/armv7l$./qemu-armv7l.sh > log.txt
> ^C
> $vi log.txt
> 
> then I‌ see other characters at prompt. I‌f I do it with mkroot for x86_64 I get
> ^M (carriage return) on end of lines.
> It seems to be a problem with stty (and $PS1 ?) settings. Any suggestions?

Nah, that's not the problem I'm seeing. But I'm only seeing it on one machine
and not the other, something's different about build environment. I'll try to
track it down here...

You could try getting the microblaze target to work? Something's wrong with the
ELF parser, the kernel refuses to launch init. Not quite sure why, I was
planning on drilling down through the kernel's load_elf_binary()
(https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/binfmt_elf.c#n685)
to see if I can figure it out...

That might be throwing you in the deep end a bit. :)

Rob