<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 4, 2022 at 3:14 AM Rob Landley <<a href="mailto:rob@landley.net">rob@landley.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 6/27/22 14:40, enh wrote:<br>
> On Sat, Jun 25, 2022 at 3:45 PM Rob Landley <<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a><br>
> <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a>>> wrote:<br>
> <br>
> On 6/24/22 19:35, enh wrote:<br>
> > On Thu, Jun 23, 2022 at 11:48 PM Rob Landley <<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a><br>
> <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a>><br>
> > <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a> <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a>>>> wrote:<br>
> ><br>
> > On 6/22/22 20:02, enh wrote:<br>
> > > On Wed, Jun 22, 2022 at 1:52 PM Rob Landley <<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a><br>
> <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a>><br>
> > <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a> <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a>>><br>
> > > <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a> <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a>><br>
> <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a> <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a>>>>> wrote:<br>
> ><br>
> > The problem with the mac tar test is even though it's easy enough to<br>
> find what<br>
> > /etc/passwd calls UID 0:<br>
> ><br>
> > ROOT="$(sed -n '/[^:]*:[^:]*:0:/s/:.*//p' /etc/passwd)"<br>
> ><br>
> > That doesn't change the fact it'll be putting a different string into the<br>
> > tarball, with different sha1sums. Um. (I was using "root" as the one known<br>
> > constant account that didn't vary across distros. Possibly I need a<br>
> way to tell<br>
> > it to use an alternate /etc/passwd file to lookup usernames. This is<br>
> why I've<br>
> > been poking at mkroot, but making that work on a mac is just... ow.)<br>
> <br>
> FYI, I committed your patch shortly after sending that message.<br>
> <br>
> thanks. interestingly, i realized that i think we also wouldn't get a red cross<br>
> in the github ui if we broke the _linux_ tests?<br>
<br>
Hmmm... looks like I broke VERBOSE=all's exit code when I moved the tests into a<br>
subshell (commit e00b4c26553b) and de474ba03950 wasn't a full enough fix...<br>
<br>
Try now?<br></blockquote><div><br></div><div>looks plausible: github's saying that linux passed (<a href="https://github.com/landley/toybox/runs/7178532773?check_suite_focus=true">https://github.com/landley/toybox/runs/7178532773?check_suite_focus=true</a>) but macos failed (<a href="https://github.com/landley/toybox/runs/7178532509?check_suite_focus=true">https://github.com/landley/toybox/runs/7178532509?check_suite_focus=true</a>).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> it's only a build failure that<br>
> counts? not obvious to me from the .yaml syntax why that is/what we could do<br>
> about it.<br>
<br>
Wasn't the yaml, it was my fault.<br>
<br>
(I semi-regularly do "make distclean defconfig toybox tests" but that stops at<br>
first failure with a visible result, so I hadn't noticed the return code...)<br>
<br>
That said, I'm not sure the MAC tests failing is really something we're ready to<br>
call a failure yet?<br></blockquote><div><br></div><div>i look at the red/green status specifically for this purpose. well, not specifically for the _mac_, but to see if something's going to be broken the next time i take an update, and get it fixed upstream first.</div><div><br></div><div>and the good news is that it's _only_ tar that's failing.</div><div><br></div><div>(btw, one thing i did with the Android toybox test runner was collect the _names_ of the failing tests. that's a handy thing to have at the bottom of the output, but perhaps more handy for me when i have to answer the question "is this bug a dupe of the existing one i'm already looking at, or do i have two problems?".)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> I can't immediately think of a better short-term fix, with the possible<br>
> exception of tagging tests as "known to fail on macos because macos is buggy".<br>
> (And "we extended a zero length file three times with truncate() and along the<br>
> way it allocated a megabyte of storage to store LITERALLY NO DATA" sounds like a<br>
> bug to me. I am neither interested in fixing nor reporting MacOS bugs because<br>
> they're 100% proprietary with 0% open source input, and they ain't paying me to<br>
> make them richer thanks. For the same reason, I don't want to put a lot more<br>
> cycles into _thinking_ about macos either.)<br>
> <br>
> The mkroot stuff is all about "I can mount ext2 or tmpfs to run this test on and<br>
> have exactly known behavior". I understand "somebody ran the test on xfs and it<br>
> behaved differently than any other filesystem so far", but I think this is a bug<br>
> in the VFS layer in a test environment I haven't got. When porting tests into<br>
> mkroot, I'd presumably do some annotation for "this test runs in the<br>
> known/mkroot environment" anyway, and logically I'd tag the ones that have known<br>
> problems outside that environment, whatever those problems may be...<br>
> <br>
> or we could have a more specific fs-specific "skipnot", since "what fs is this?"<br>
> seems to be one of the most common problems.<br>
<br>
Is there a portable way to determine filesystem type, though? df . doesn't say,<br>
I have to look in /proc/mounts and I doubt mac has that?<br>
<br>
$ grep -w "^$(df . | tail -n 1 | toybox cut -DF 1)" /proc/mounts | toybox cut -DF 3<br>
<br>
(I THINK that if the device has a space in it df will output the escaped<br>
version, which should match for grep... But again: Linux.)<br></blockquote><div><br></div><div>on the mac it looks like mount is your best bet:</div><div><br></div><div>~$ mount<br>/dev/disk3s1s1 on / (apfs, sealed, local, read-only, journaled)<br></div><div>...</div><div><br></div><div>but maybe just "it's a mac, it's probably using apple's default fs" is sufficient?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> Another TODO item is packing up debootstrap and alpine root filesystems to test<br>
> under mkroot as more rigorous "TEST_HOST=1" runs. With the kernel<br>
> config/version, mountpoint selection, and qemu board emulation parameters.<br>
> Presumably running my init script instead of theirs to do the setup and start<br>
> the test, but using their $PATH of binaries (gnu/fsf and busybox, respectively).<br>
> But that's after I get the base mkroot testing well...<br>
> <br>
> > note that it's /etc/*group* that's weird, not *passwd*. uid 0 is root, but<br>
> group<br>
> > 0 is wheel. (i think that's true of all bsds?)<br>
<br>
I suspect the easy solution is "skipnot grep -qw root /etc/group".<br>
<br>
Because it's not gonna make the same tarball otherwise, and this is testing data<br>
fetched out of /etc/passwd. (We have --owner=NAME:ID tests: this isn't that.)<br></blockquote><div><br></div><div>wouldn't it make more sense to use one of the low-numbered groups that's the same on both though, rather than skip?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> I'm not spotting any negative gids in /etc/group on devuan. And I think that<br>
> violates posix?<br>
> <br>
> <a href="https://pubs.opengroup.org/onlinepubs/9699919799/functions/chown.html" rel="noreferrer" target="_blank">https://pubs.opengroup.org/onlinepubs/9699919799/functions/chown.html</a><br>
> <br>
> chown(-1) means "don't change". So you can't set it to -1 through the posix<br>
> specified API.<br>
> <br>
> they're just taking advantage of a scanf("%u") somewhere else. the interesting<br>
> part is that this means those _aren't_ actually the 64Ki and 64Ki-1 i was expecting:<br>
<br>
Wait, scanf("%u") will accept an input starting with a minus sign?<br>
<br>
Yes, posix says so, and the boilerplate says they got that from C99. They defer<br>
to strtoul() which says "If the subject sequence begins with a <hyphen-minus>,<br>
the value resulting from the conversion shall be negated" but nothing about what<br>
strtoul() should do about that...<br>
<br>
(I'm interested because the C++ loons who hijacked gcc development declared<br>
signed integer wrapping to be "undefined behavior", a thing I noticed a few<br>
years back when the compiler "optimize out" code that did that with a constant<br>
at compile time. I had to typecast it to unsigned and then back again to do the<br>
math...)<br>
<br>
> ~$ id nobody<br>
> uid=4294967294(nobody) gid=4294967294(nobody)<br>
> groups=4294967294(nobody),12(everyone),61(localaccounts),100(_lpoperator)<br>
> <br>
> but the small positive ones look okay?<br>
<br>
Linux uses 65534 for nobody, and it's the OTHER magic UID in Linux hardwired<br>
into the kernel. Well, these days they give you a gratuitous sysctl to change<br>
it, but I've never seen it used. And they don't have an equivalent sysctl to<br>
move root off of UID 0.<br>
<br>
include/linux/highuid.h:#define DEFAULT_OVERFLOWUID 65534<br>
<br>
It's kind of like the new ping API needing you to<br>
/proc/sys/net/ipv4/ping_group_range before it can be used. Linux kernel<br>
development collapsed into a pile of bureaucracy some years back. Kind of sad.<br>
<br>
They just broke my "build with gelf" fix for a third time:<br>
<br>
/home/landley/toybox/cleanser/root/build/x86_64-tmp/linux/tools/objtool/include/objtool/elf.h:10:10:<br>
fatal error: gelf.h: No such file or directory<br>
<br>
No other architecture requires you to install a magic extra elf package on your<br>
host, but the x86 maintainers are insane:<br>
<br>
<a href="https://lore.kernel.org/lkml/20211024192742.uo62mbqb6hmhafjs@treble/" rel="noreferrer" target="_blank">https://lore.kernel.org/lkml/20211024192742.uo62mbqb6hmhafjs@treble/</a><br>
<br>
(No other target needs this. I'm building kernels to run in a dedicated<br>
environment. I don't need spectre/meltdown mitigation, and Linux has THREE<br>
COPIES of elf plumbing in the tree already before this! Pulling in an external<br>
package to do ELF stuff is just SAD here...)<br>
<br>
> $ rm -f empty; for i in 1k 1m 1g; do truncate -s $i empty; stat -c %b empty;<br>
> done; ls -l empty<br>
> 0<br>
> 0<br>
> 0<br>
> -rw-r--r-- 1 landley landley 1073741824 Jun 25 17:15 empty<br>
> <br>
> I'm guessing it's not gonna say 0.<br>
> <br>
> ~/toybox$ rm -f empty; for i in 1k 1m 1g; do ./toybox truncate -s $i empty;<br>
> ./toybox stat -c %b empty; done; ls -l empty<br>
> 0<br>
> 2048<br>
> 2048<br>
> -rw-r--r-- 1 enh primarygroup 1073741824 Jun 27 12:38 empty<br>
> ~/toybox$<br>
<br>
So "truncate -s 1k empty; truncate -s 1m empty" allocates a megabyte of disk space.<br>
<br>
The second truncate makes the file NOT SPARSE.<br>
<br>
> the weird part for me was that it wasn't obvious _what_ the non-zero number was<br>
> going to be. <br>
<br>
The bug seems to be if you extend a sparse file the result is not sparse. I'm<br>
guessing if you go:<br>
<br>
truncate -s 1k empty; truncate -s 2m empty; stat -c %b empty<br>
<br>
You'll get 4096. (Because 2048*512=1m and that's what we asked it to expand the<br>
sparse file to.)<br></blockquote><div><br></div><div>/tmp$ ~/toybox/toybox truncate -s 1k empty; ~/toybox/toybox truncate -s 2m empty; ~/toybox/toybox stat -c %b empty<br>4096<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> > They swapped to zsh three years ago:<br>
> ><br>
> > <br>
> <a href="https://www.theverge.com/2019/6/4/18651872/apple-macos-catalina-zsh-bash-shell-replacement-features" rel="noreferrer" target="_blank">https://www.theverge.com/2019/6/4/18651872/apple-macos-catalina-zsh-bash-shell-replacement-features</a><br>
> ><br>
> ><br>
> > oh, yeah, good point. my reaction to that was similar to your reaction to<br>
> dash.<br>
> > _i'll_ be using bash on macos until they remove it.<br>
> <br>
> All my stuff says #!/bin/bash at the top, but I dunno how github is running<br>
> what...<br>
> <br>
> just `VERBOSE=all make tests`<br>
<br>
Which might be fixed now, for a definition of "fixed" that means "notices it's<br>
failing".<br>
<br>
Lateral progress!<br>
<br>
> > I'm tempted to borrow my wife's mac for a bit, but I have no idea how<br>
> to set up<br>
> > a development environment on a mac. The first google hit is<br>
> > <a href="https://sourabhbajaj.com/mac-setup/Xcode/" rel="noreferrer" target="_blank">https://sourabhbajaj.com/mac-setup/Xcode/</a><br>
> <<a href="https://sourabhbajaj.com/mac-setup/Xcode/" rel="noreferrer" target="_blank">https://sourabhbajaj.com/mac-setup/Xcode/</a>> which looks... more elaborate than I<br>
> > want to do on a borrowed machine.<br>
> ><br>
> > iirc it's a bit simpler than that (if you don't have some company policy that<br>
> > says you can only install binaries from their servers) --- you just run "make"<br>
> > and it pops up a window saying "you want to install all that shit?" and<br>
> you say<br>
> > "it's a unix system; i know this", bish bash bosh, job done.<br>
> <br>
> Except I can't easily _undo_ it afterwards and don't want to eat I dunno how<br>
> many gigs on my wife's machine with Apple's soldered-in ssd.<br>
> <br>
> yeah, it's not small.<br>
<br>
It's a pity there isn't a mac dev environment I can ssh into. I can can think of<br>
three Linux ones off the top of my head that are still up which I probably have<br>
credentials for. (Make that five. Probably more if I thought about it.) But then<br>
Fabrice Bellard's jslinux run a Linux vm in a web page, so "lemme try this out<br>
really quick" was never a high bar for Linux.<br>
<br>
Mac, not so much...<br>
<br>
Rob<br>
</blockquote></div></div>