<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 19, 2021 at 11:56 PM Rob Landley <<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 10/19/21 10:57 AM, enh wrote:<br>
> are you sure? remember that if a test is currently checked in [and Android uses<br>
> the toy; my test runner reports but ignores failures for tests where readlink<br>
> says it's not actually toybox], that means that the tests pass on Android.<br>
> <br>
> there's even a CTS test:<br>
> ```<br>
> TEST(wchar, wcwidth_non_spacing_and_enclosing_marks_and_format) {<br>
> if (!have_dl()) return;<br>
> <br>
> EXPECT_EQ(0, wcwidth(0x0300)); // Combining grave.<br>
> EXPECT_EQ(0, wcwidth(0x20dd)); // Combining enclosing circle.<br>
> EXPECT_EQ(0, wcwidth(0x00ad)); // Soft hyphen (SHY).<br>
> EXPECT_EQ(0, wcwidth(0x200b)); // Zero width space.<br>
> }<br>
> ```<br>
> <br>
> my guess is that you're using a statically-linked binary?<br>
<br>
Yup. I can't test dynamic bionic on devuan, bionic isn't installed in /lib.<br></blockquote><div><br></div><div>if you don't mind creating a /system directory (or symlink) in /, you can just run like that. there's a script in AOSP bionic that sets up all the links you need to run the bionic tests on the host.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
(And when I _tried_ to create a chroot as a normal user, bionic segfaulted<br>
before calling main() because /dev/null wasn't there. Did that ever get fixed?<br>
Hard to run an init linked against bionic when that's the case. </blockquote><div><br></div><div>have you tried? i'm pretty sure there's a special case for pid 1, for exactly this reason :-)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The kernel guys<br>
never merged my <a href="https://lkml.org/lkml/2017/9/13/651" rel="noreferrer" target="_blank">https://lkml.org/lkml/2017/9/13/651</a> stuff. I gave 'em three<br>
strikes and moved on. I have a todo item to teach cpio to hallucinate /dev nodes...)<br>
<br>
> bionic doesn't have a<br>
> "static libdl", so when it tries to dlopen() icu4c to handle an i18n question,<br>
<br>
... why would it need to do that?<br></blockquote><div><br></div><div>because the last thing the world needs is _multiple_ copies of all the i18n data. icu is the source of truth for that on Android. (we've had bugs in the past in this area that made us clean up our act.)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> that'll fail and in most cases bionic will fall back to "what do i know about<br>
> ASCII?" but otherwise report failure. (that's what the first line of the test is<br>
> checking too --- "if we're the static version of the tests, skip this test<br>
> because this isn't available".)<br>
<br>
It's not skipping them for me, but I'm still using android-ndk-r21d so maybe the<br>
skipping logic is getting either NDK weirdness or old version weirdness?<br>
<br>
But back to "why does this need to to dlopen() anything, I that that was a<br>
uniquely glibc failing?"<br>
<br>
Sigh. I wrote my own utf8 conversion stuff, I _can_ do my own width detection<br>
logic if libc isn't reliable.<br>
<br>
Hmmm, according to <a href="https://twitter.com/doctorow/status/1449072198069587969" rel="noreferrer" target="_blank">https://twitter.com/doctorow/status/1449072198069587969</a> the<br>
unicode version of rot13 is rot8000, which implies ballpark of 16k glyphs. Seems<br>
tractable. Let's see, wikipedia[citation needed] says 144k code points (of a<br>
million and change possible).<br>
<br>
I remember Rich being quite proud of his compression scheme for unicode<br>
evaluation, where's that...<br>
<br>
int wcwidth(wchar_t wc)<br>
{<br>
if (wc < 0xffU)<br>
return (wc+1 & 0x7f) >= 0x21 ? 1 : wc ? -1 : 0;<br>
if ((wc & 0xfffeffffU) < 0xfffe) {<br>
if ((table[table[wc>>8]*32+((wc&255)>>3)]>>(wc&7))&1)<br>
return 0;<br>
if ((wtable[wtable[wc>>8]*32+((wc&255)>>3)]>>(wc&7))&1)<br>
return 2;<br>
return 1;<br>
}<br>
if ((wc & 0xfffe) == 0xfffe)<br>
return -1;<br>
if (wc-0x20000U < 0x20000)<br>
return 2;<br>
if (wc == 0xe0001 || wc-0xe0020U < 0x5f || wc-0xe0100 < 0xef)<br>
return 0;<br>
return 1;<br>
}<br>
<br>
Oh goddess, what is this doing... The first 256 entries are peeled off by the<br>
first if() statement and taken care of by two ? : that honestly might as well<br>
just pull that later return 0 clause up to the top and if (!wc || ...) it all in<br>
one gulp. (Maybe he's micro-optimizing for performance?)<br>
<br>
Then the if (wc < 0xfffe) stanza is a bit silly because we have a HARD LIMIT of<br>
0x10ffff entries in unicode so there should be an if (wc>0x10ffff) return -1 and<br>
then this should probably be & 0x1effff...<br>
<br>
The if (table[]) and if (wtable[]) bits are identical and initially confusing<br>
because of the if (table[table[]) construct, but since we already peeled off the<br>
first 256 wc values that table[wc>>8] part will never fetch entry zero, and<br>
nothing in the first 512 entries (of _either_ table) is <16, and 16*32 is 512 so<br>
the start of the table seems to be _just_ indexes to later in the table. (And if<br>
you feed it values >1ffff the guarding if() around it drops through, so this<br>
looks like it's actually two disjoint tables stored together.)<br>
<br>
And then it's just cleanups for specific cases. 90% of the cleverness here is in<br>
the two tables. table[] is initialized from #include "nonspacing.h" which is a<br>
big array of numbers, and wtable[] is similarly initialized from #include<br>
"wide.h". Which are both PAINFUL.<br>
<br>
Those tables are just under 2k each, and could SERIOUSLY be compressed. For<br>
example, the first 512 bytes of the first table are all "16" except for a sparse<br>
set of sequential values running from 18 to 59, so it could be initialized from<br>
a bitmask. Let's see, quick and dirty bash to generate said bitmask from the table:<br>
<br>
X=0; Y=0; for i in $(cat src/ctype/nonspacing.h | tr , ' ' | xargs); do<br>
[ $i -ne 16 ] && let 'Y|=(1<<(31&X))'; [ $((31&++X)) -eq 0 ] &&<br>
printf '%x\n' $Y && Y=0; [ $X == 512 ] && break<br>
done<br>
<br>
And then the C would be something like (untested):<br>
<br>
unsigned bits[] = {0x3f89fff8, 0x13001, 0, 0, 0, 0xf40, 0, 0xc8000000,<br>
0x430402, 0, 0, 0x8000, 0, 0, 0x60000, 0}, i, j;<br>
<br>
memset(table, 16, 512);<br>
for (i = 0, j = 18; i<512; i++) if (bits[i/32]&(1<<(i&31)) table[i] = j++;<br>
<br>
And then you go:<br>
<br>
memset(table+512, 0, sizeof(table)-512);<br>
memset(table+512+32, 255, 46);<br>
<br>
And the rest of it could be filled by traversing another sparse bit array I<br>
can't be bothered to scrape together right now but it would have... um... (for i<br>
in $(cat src/ctype/nonspacing.h | tr , ' ' | tail -n +24); do [ $i -ne 0 ] &&<br>
echo $i; done | xargs | wc -w)<br>
<br>
It would have 249 entries, and the whole thing looks like a dozen lines of C to<br>
init each table and then the function to use the table, putting it within range<br>
of merging if Rich is ok with me using his tables under 0bsd. And then I<br>
wouldn't _have_ to depend on libc to do this.<br>
<br>
Not sure if I _should_, but I _can_. (It was nice to leave this to libc. Then it<br>
wasn't my problem to update it every time Microsoft wrote another check to the<br>
unicode committee. Both glibc and musl can do this when statically linked. Sigh.)<br></blockquote><div><br></div><div>i'll only send a patch so i can disable this again on Android even if you _do_ do it, so i don't think it's a great use of your limited time :-)</div><div><br></div><div>hang on, let me look out the symlink script... here you go:</div><div><br></div><div><a href="https://android.googlesource.com/platform/bionic/+/master/build/run-on-host.sh">https://android.googlesource.com/platform/bionic/+/master/build/run-on-host.sh</a><br></div><div><br></div><div>that should set you up with everything you need (and both 32- and 64-bit too, if you're chasing something 32-bit-only).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Rob<br>
<br>
P.S. Rich's other table has some 17s mixed in the 16s which... I think it moves<br>
in runs of 8? Very small bitmap if so? It would be so much easier to work out<br>
the alignment if he'd wordwrapped his tables to a consistent number of entries<br>
per line, but no. Eh, runs of 4, 54 bits total. Plus two isolated weirdos.) And<br>
most of the nonzero values in the latter part are 255, so traverse a bitmap of<br>
THOSE and there's not much left to initialize afterwards. Yeah, a dozen lines<br>
per table of initialization is looking doable, within range of sticking it in<br>
portability.c...<br>
</blockquote></div></div>