<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 5, 2021 at 10:53 PM Rob Landley <<a href="mailto:rob@landley.net">rob@landley.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 1/5/21 7:17 PM, enh wrote:<br>
> On Sat, Dec 26, 2020 at 11:09 AM Rob Landley <<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a><br>
> <mailto:<a href="mailto:rob@landley.net" target="_blank">rob@landley.net</a>>> wrote:<br>
> <br>
> On 12/5/20 1:32 PM, enh wrote:<br>
> > (you'd expect to see '東해 물과 백두산이' if this actually worked: change from<br>
> > the hangeul for "east" to the hanja for "east" but leave everything else<br>
> alone.)<br>
> ><br>
> > xxd confirms it's just screwing up bytes:<br>
> <br>
> Oh I know the others are all doing bytes, but that doesn't make it the right<br>
> thing to do. :)<br>
> <br>
> > ~$ echo '동해 물과 백두산이' | xxd<br>
> > 00000000: eb8f 99ed 95b4 20eb acbc eab3 bc20 ebb0 ...... ...... ..<br>
> > 00000010: b1eb 9190 ec82 b0ec 9db4 0a ...........<br>
> > ~$ echo '동해 물과 백두산이' | tr '동' '東' | xxd<br>
> > 00000000: e69d b1ed 95b4 20e6 acbc eab3 bc20 e6b0 ...... ...... ..<br>
> > 00000010: b1e6 9190 ec82 b0ec 9db4 0a ...........<br>
> ><br>
> > 동 is 0xeb 0x8f 0x99 and the equivalent Chinese character 東 is 0xe6 0x9d<br>
> 0xb1,<br>
> > and you can see from those hex dumps that what tr did was replace 0xeb with<br>
> > 0xe6, 0x8f with 0x9d, and 0x99 with 0xb1. this did the right thing by accident<br>
> > for 동 but mangled other characters that contained any of those bytes.<br>
> ><br>
> > and although philosophically i'm usually on board with your "all times are<br>
> ISO,<br>
> > all text is UTF8", i'm really not sure it makes much sense to even *try* to<br>
> > support this in tr. why? because i think it opens the i18n/l11n can of worms<br>
> > again. if you think about non-binary uses of tr, they're often stuff like<br>
> > "convert to all caps", but are we going to get that right for Turkish/Azeri<br>
> > dotted/dotless 'i's, Greek final/non-final sigma, etc? are we going to<br>
> have tr's<br>
> > behavior then depend on your locale?<br>
> <br>
> I've been handing that off to libc, which I'm aware bionic does not currently<br>
> do,<br>
> <br>
> to be clear, *no* C library can get all of those examples right because the C<br>
> APIs are broken as designed. you genuinely need something like icu4c for this.<br>
> that's *why* bionic only supports the C locale.<br>
<br>
Sadly, this is one of the things I've wanted to "study until it makes sense and<br>
I can understand what the right thing to do is", and it's one where staring at<br>
it has made it worse.<br>
<br>
Yeah combining characters are out of scope for tr but I thought "map one unicode<br>
point to another unicode point" was a reasonable goal for tr? You're making a<br>
fairly strong case that it isn't. (I knew the unicode consortium was insane, but<br>
I'd assumed a base layer of competency that had gotten crapped upon by microsoft<br>
and IBM joining the committee, which I might be able to dig down to reach.<br>
Sounds like it's turtles all the way...)<br></blockquote><div><br></div><div>to be fair to them, this one's not their fault --- this is those annoying humans. (rule #1 of i18n --- for any "obviously universally true" thing you can think of, there's a language for which it isn't true.)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> but, yeah, in retrospect i shouldn't have led with an example that libc *can*<br>
> get right and followed up with one it can't. the other way round would<br>
> definitely have been better :-)<br>
> <br>
> but i can explain why i did that: we've had a ton of trouble with the<br>
> dotted/dotless 'i' in Java because you really have to think about whether you<br>
> want that behavior. and it's often quite non-obvious because it doesn't just<br>
> depend on your locale, it depends on your *input*. if i'm a Turk changing the<br>
> case of some Turkish text, yes, i want that locale-specific behavior. if i'm a<br>
> Turk who's just trying to build AOSP, i don't, because it's going to corrupt ASCII.<br>
<br>
Sigh. I think I need to bow to your expertise here. You at least have real-world<br>
experience with the mess via the bionic and dalvik library bug reports.<br>
<br>
Is this _just_ a problem with case mapping? (Can we say [:lower:] and [:upper:]<br>
are the relevant 26 ascii characters and _otherwise_ accept a unicode point<br>
input at each end as one character?<br></blockquote><div><br></div><div>depends what you mean...</div><div><br></div><div>yes, "case mapping is broken". for example, take "Κολο<span style="background-color:rgb(255,255,0)">ΣΣ</span>ὸ<span style="background-color:rgb(255,255,0)">Σ</span>" (where the initial kappa and the three sigmas are all uppercase, and i've highlighted the letters to pay attention to). lowercased that becomes "κολο<span style="background-color:rgb(255,255,0)">σσ</span>ὸ<span style="background-color:rgb(255,255,0)">ς</span>" because the lowercase of "Σ" is *either* "σ" *or* "ς" depending on where in the word you are. this is why useful APIs for upper/lower case take a *string* rather than a character.</div><div><br></div><div>tr's byte-by-byte/character-by-character model is just not a good fit for language.</div><div><br></div><div>but if you mean "could a tr implementation correctly replace 동 with 東?" (as in the earlier Korean national anthem example), yes, it could. i'm just not convinced that's a useful tool and thus worth the complexity/performance hit of doing it "properly".</div><div><br></div><div>(if you're _looking_ for something to do along these lines, i'd suggest -e ENCODING for strings(1) instead. i've almost implemented it myself on a couple of occasions, but since one was `-e S` and the other was `-e b` [and i've never had a use for l/B/L] it hasn't hit my own "useful enough" threshold yet.)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> my claim here is that "most Turks^wpeople using tr(1) are programmers trying to<br>
> build something" :-)<br>
> <br>
> you changed the main.c intro code to not pass along environmental utf8-ness,<br>
> I changed it back and have another pending commit from you (which still isn't<br>
> going to make a difference on bionic yet anyway... :)<br>
> <br>
> that's unrelated. that's some weird macOS-only thing about what they call<br>
> their locales.<br>
<br>
Your updated one is applied.<br></blockquote><div><br></div><div>(thanks! after the chmod.test fix, AOSP is back up to date again for the first time in longer than i'd realized.)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> > are we going to deal with combining<br>
> > characters too, or do i have to specify all the ways you can write "ö" to get<br>
> > "Freude, schöner Götterfunken" right (because without a hex dump, neither you<br>
> > nor i know whether those two 'ö's were encoded the same way)?<br>
> <br>
> No, I was just going to substitute individual utf8 sequences. There comes a<br>
> point where you switch to "sed"...<br>
> <br>
> so what's the point? you can't get this right, and no-one cares anyway, but it<br>
> makes things slower and more complicated because... ? who are you, and what have<br>
> you done to the real rob? :-)<br>
<br>
Buried in shell logic, giving everything else combined like 1/4 voltage. (And<br>
shell has to care about this crap too because bash has ${PATH^^*} and friends.<br>
Wildcard matching has [:upper:] and variables have declare -u.)<br>
<br>
Did you know you can have exported local variables? I'm not sure who to argue<br>
with about that, but I expect it would require time travel to the mid-1980's. (I<br>
figured out how to make it work, I just object to NEEDING to at a conceptual level.)<br>
<br>
> > amusingly, the Plan9 man page only gave one example of using tr(1), and it was<br>
> > converting ASCII upper/lower. so i don't think they had any _use_ for it<br>
> either,<br>
> > they just wrote everything in terms of runes.<br>
> <br>
> They were english-only white guys in the early 90's. Good intentions only go so<br>
> far if it never leaves the lab and hits real world data.<br>
> <br>
> ...which is why the C API is broken for this, and -- unless you're going to<br>
> accept icu4c as a dependency -- these two well-intentioned white guys in the<br>
> early 2020's aren't going to fix it either.<br>
> <br>
> (personally i'd be fine with the icu4c dependency, though i still don't think<br>
> it's *useful* for tr(1).)<br>
<br>
If bash didn't have case mapping I'd say this whole can of worms is out of<br>
scope, but unfortunately bash has several instances of case mapping.<br>
<br>
You're right mistranslating upper/lower case is a failure, but not being able to<br>
tr hiragana to katakana seems like a failure too? (But then it still can't<br>
handle romanji because that's a 2 char to 1 char mapping. Hmmm...)<br>
<br>
I'm trying to figure out where the 80/20 line is. It's not hard to parse unicode<br>
points, and for 1/1 mapping lists it's reasonably straightforward what to do<br>
with them. It's this [:banana:] nonsense where it wants to map between leitmotif<br>
and color palette that's confusing: the relationship is not FIXED. (And why does<br>
it even HAVE [:print:]?)<br>
<br>
Part of the problem is I'm not a heavy tr user. It always struck me as one of<br>
those obsolete tools like "ed" left over from the daisy-wheel typesetting days<br>
("man" is built on a stack of like 6 of them), and now that I'm trying to figure<br>
out what it's FOR... I find understanding this tool's use case profile highly<br>
non-obvious.<br></blockquote><div><br></div><div>that was the case i was trying to make: i'm pretty sure that in 2020 it's "for" replacing control characters/punctuation with other control characters/punctuation. "-" -> "_" or " " -> "\n" or whatever.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> Doesn't look too expensive? In musl time() is a clock_gettime(CLOCK_REALTIME)<br>
> wrapper which lives in the vdso. In bionic you're doing<br>
> reinterpret_cast<decltype(&time)>(__libc_globals->vdso[VDSO_TIME].fn); before<br>
> making a call to gettimeofday() which ALSO lives in the vdso...?<br>
> <br>
> the gettimeofday() is the fallback for kernels whose VDSO doesn't support it.<br>
> (things are especially complicated if you're a 32-bit process on aarch64, for<br>
> example. it's all fixed as of 5.x but we have something like a 7 year rule too.)<br>
<br>
I.E. on older systems it would work but be slow. Which as failure modes go...<br></blockquote><div><br></div><div>exactly. (though to be clear "older" in this context is pretty damn recent!)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
My point is that performance optimization is a can of worms that would take my<br>
full attention to do right, and I'm not there yet. In the meantime partial<br>
optimizations that change the output of watch and tee and such rub my nose in<br>
the incompleteness of my test suite which can't really see terminal output yet<br>
because I haven't done proper pty emulators. But then I haven't taught "watch"<br>
to parse ascii color change sequences and jumps yet (and I first wrote code to<br>
parse the full DOS ansi.sys sequence set and write directly into VGA text mode<br>
memory at 0xa800000 or whatever it was in 1991).<br>
<br>
> My original <a href="https://landley.net/toybox/design.html#goals" rel="noreferrer" target="_blank">https://landley.net/toybox/design.html#goals</a> ordering of "simple,<br>
> small, fast, and full-featured" has shuffled around a bit already. Several<br>
> commands have gotten a LOT more full featured, and CONFIG_SORT_SMALL went away a<br>
> while ago. Needs of the users and all that. I still want simple first, but<br>
> "simplest implementation of..." has "of" doing some heavy lifting these days.<br>
> <br>
> doing tr(1) *right* (while a laudable goal) is so hard (or at least "pulls in a<br>
> very large dependency") that i don't think it makes sense until/unless someone<br>
> actually needs it. and the fact that no other tr(1) does it right suggests<br>
> no-one does.<br>
<br>
That's hard to argue against.<br>
<br>
I'm still not quite sure "handles basic 1-1 unicode point mapping, doesn't care<br>
what each unicode point IS so treats combining characters as a series of unicode<br>
points, none of the [:splat:] macros expand to unicode characters" isn't a<br>
reasonable place to draw the line. But then I don't really know what combining<br>
characters DO and why they exist. "This can't work for everything, can it work<br>
for enough to be useful"... sigh. I dunno.<br>
<br>
That's why it's still in pending. "Only does ascii" wasn't so much a choice as a<br>
historical accident held in place by inertia. Accepting that as "what tr should<br>
do" seems sad, but I'm not qualified to improve upon it.<br></blockquote><div><br></div><div>so "good enough for now, wait for counterexample"?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> (yes, i know that kind of logic can be easily abused, but i think<br>
> it's reasonable here --- it's not like GNU is averse to adding<br>
> features/reimplementing libc within their own tools, and they do support Unicode<br>
> just about everywhere else.) and even if you don't buy that, remember that<br>
> "fixed" is complicated here (well, in Turkey and Azerbaijan anyway).<br>
<br>
If you want to explicitly list the set of lowercase characters that turn into<br>
each corresponding uppercase character, with your entire alphabet spelled out in<br>
the from and to sections of tr, it SEEMS like tr should be able to handle that?<br></blockquote><div><br></div><div>(the point of my Greek example is that you *can't* do that for all languages, but...)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
There should _be_ a command line utility that can do that. Is it lex, maybe?<br></blockquote><div><br></div><div>...yeah, lex could do the broken thing.</div><div><br></div><div>lex could also do the "really correct" thing: lex (to break into words) calling icu4c on each word (to convert case).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Anyway, not gonna get to promoting tr this week. I have opened that can of worms<br>
that is shell functions, and am scheduled to fly back to Tokyo on the 20th<br>
(covid willing) and should probably cut a toybox release before getting on the<br>
plane. I've got a bug heap to fix before then with date -I and so on in it...<br>
<br>
> > Rob<br>
> <br>
> Rob<br>
<br>
Rob<br>
</blockquote></div></div>