<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 4, 2021 at 5:33 AM Rob Landley <<a href="mailto:rob@landley.net">rob@landley.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 9/2/21 7:12 PM, enh wrote:<br>
> On Thu, Sep 2, 2021 at 4:37 PM Rob Landley <<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a><br>
> > to me, if the open() succeeded but the write() failed --- we're in weird<br>
> > uncharted "shouldn't happen" territory, and probably want to know?<br>
> <br>
> Hmmm, I thought the open would succeed and the write fail, but if it's reliable<br>
> that the open is what fails when selinux is disabled...<br>
> <br>
> (note that i haven't tested that!)<br>
<br>
And I can't because I don't have selinux test environment coverage, or the<br>
stomach to read through enough of the terrible selinux code in the kernel.<br>
<br>
This is not an area I have sufficient existing expertise to feel confident<br>
about, and unlike sed/bzip/bash/awk/bc I'm not planning on doing a multi-month<br>
deep dive to BUILD said expertise. (Because I still think NSA selinux was a<br>
terrible idea and cannot generate the necessary enthusiasm.)<br>
<br>
> There's your problem: never -Werror. <br>
> <br>
> if you don't -Werror, you inevitably check in code with warnings, spamming the<br>
> build and making it easier for more warnings to creep in.<br>
<br>
And then you remove the warnings again? Toybox doesn't have -Werror...<br>
<br>
Way back when the linux kernel was full of warnings because they got lost in the<br>
noise of the always verbose output, and I made a blueberry.py script that would<br>
filter the output into something intelligible<br>
(<a href="https://static.lwn.net/2002/0117/bigpage.php3#:~:text=Blueberry" rel="noreferrer" target="_blank">https://static.lwn.net/2002/0117/bigpage.php3#:~:text=Blueberry</a>), and the<br>
kernel guys did their usual response to a "don't ask questions post errors" and<br>
implemented the functionality in a much less terrible way inside (what became)<br>
kbuild, and once you could SEE the warnings they removed them from the build.<br>
<br>
I'm not sure "the linux kernel is so much smaller and simpler than our package,<br>
it just doesn't have the scalability problems we do" is a compelling argument<br>
for most packages?<br></blockquote><div><br></div><div>Android tries to avoid falling into the trap of "oh, you didn't have $X enabled in that corner? we just assumed because it's on everywhere else!".</div><div><br></div><div><a href="https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.15-Werror">https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.15-Werror</a><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> Compiler version updates always hallucinate<br>
> new weirdness. If nothing else you're making it harder to build old versions in<br>
> new build environments for regression testing comparisons.<br>
> <br>
> with hermetic builds you're using the right compiler for that version anyway.<br>
> (forget features: when 100M lines of code you have, reliant on compiler bugs you<br>
> will be.)<br>
<br>
As with "reproducible builds", that's true if you accept it as a premise and a<br>
lot of tedious work to fix when you don't. :P<br>
<br>
This is why I wanted to split the build into layers:<br>
<br>
1) Base OS layer with libc+toybox plus native compiler (and maybe kernel):<br>
minimum needed to rebuild itself under itself in a tiny VM/container accessed<br>
via serial console or similar. (Which makes it self-regression-testing to an<br>
extent.)<br>
<br>
2) Natively building the generic non-gui infrastructure layer: git/repo if it<br>
isn't in the base, probably ninja and cmake go here, more languages (perl,<br>
python, java, lua..), more libraries (ssl, ncurses, libxml, libusb...), better<br>
access mechanism (ssh or adb), whatever daemons define your environment (httpd,<br>
bind, cups... I know android has its own suite of these starting with a custom<br>
init and a security management thingy but don't have a relationship diagram in<br>
front of me...)<br>
<br>
3) Building the GUI/desktop can of worms, UNDER stage 2 above.<br>
<br>
Back under aboriginal linux I had layer 1 working pretty well and had a proof of<br>
concept linux from scratch layer 2, and a friend built a gentoo layer 2 but it<br>
was hack city. I did a couple years of research on building a red hat layer 2, a<br>
debian layer 2, and a suse layer 2, but all four of those distros were<br>
incestuous tangled crap. (Debian was the least bad, which is not an endorsement.)<br>
<br>
One of the advantages of a layered approach is you can stop and snapshot. Build<br>
stage 2 on top of a known stage 1, build stage 3 on top of a known stage 2.<br>
<br>
Shipping your own base OS layer (to build stage 2 on top of a known stage 1)<br>
_is_ a hermetic build. Rebuilding stage 3 on top of a known stage 2 should be<br>
similarly orthogonal. (Or at least versionable.)<br>
<br>
And I still think that at some point the NDK build should be dogfooded as the<br>
AOSP toolchain build. (Maybe with a config switch or extra build stage<br>
overlaying libraries NDK doesn't ship. But then "adding libraries after the fact<br>
to an existing filesystem" is a normal part of the stage 2 build above? :)<br>
<br>
I don't have my head properly wrapped around how selinux annotation works into<br>
this, but I do know there are tools to annotate a filesystem after the fact<br>
because I keep bumping into references to them. And I suspect there are more<br>
granular approaches but splitting off those 3 layers seems like the obvious start...<br>
<br>
> > and any<br>
> > given warning's false positive rate can be highly dependent on the specific<br>
> > codebase, and i don't get the feeling compiler writers think about scaling to<br>
> > "can i rebuild all of Debian?" or even just "can i rebuild all of Android?".)<br>
> <br>
> Their userbase kinda does.<br>
> <br>
> it's too late by then :-(<br>
<br>
Alas, they're using the Microsoft approach: your users are unpaid beta testers,<br>
you ship perpetually broken crap and course correct on the fly as the bug<br>
reports come in, which never quite keeps up with new breakage.<br>
<br>
> Those are all "cannot continue to parse the input file format", not "output had<br>
> a problem"...<br>
> <br>
> ah, so you're talking about the perror_msg() versus perror_exit() distinction.<br>
> still a _failure_ but not an immediate one. (though as someone who always uses<br>
> 'v' with tar, i'd never notice the failure interactively unless it was in the<br>
> last 40 lines!)<br>
<br>
Hence the exit code so at least your script notices. </blockquote><div><br></div><div>assuming you've told bash to exit on error, which most scripts don't. i really wish that was the default...</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Also, my debian host tar says:<br>
<br>
tar: Exiting with failure status due to previous errors<br>
<br>
Which is a sadly verbose message but we should say _something_. Hmmm...<br>
<br>
> > So I _think_ your patch changes "tar x --selinux" behavior to exit<br>
> instead of<br>
> > spamming warnings when run on a system without sexlinux support.<br>
> ><br>
> > seems like a feature to me? don't say --selinux if you don't actually care?<br>
> <br>
> Which would mean --selinux is special and behaves differently from the rest<br>
> of tar.<br>
> <br>
> i was going to say "but if you got the wrong selinux labels, what you wrote out<br>
> probably won't work" ... but then realized that's true of chmod() too. i'm not<br>
> going to be able to run that binary if its 'x' bit isn't set, any more than if<br>
> it doesn't have the right selinux label.<br>
<br>
I just added a "tar: had errors" message at the end, ala "oy, check the backscroll".<br></blockquote><div><br></div><div>yeah, that's a good idea. (and the debian message does sound vaguely familiar now you mention it... though this conversation reminded me that it's been a long time since i last ran out of space, and even longer since i last had any other kind of error from tar!)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Rob<br>
</blockquote></div></div>