[Toybox] ls (/-b/-q) (/on a tty)

Mon Dec 11 05:42:43 PST 2017

On 12/10/2017 01:25 PM, enh wrote:
> On Fri, Dec 8, 2017 at 6:12 PM, Rob Landley <rob at landley.net> wrote:
>> I had my reasons for changing it (the default output should be an
>> unambiguous representation that doesn't cause collisions), but can
>> put it back if you guys have something real it's breaking or feel
>> strongly enough about matching ubuntu.
> 
> it was reported to us as a bug, and certainly "doesn't behave like the
> desktop" is a reasonable working definition of "bug" unless the
> desktop is obviously wrong.

Indeed.

You've heard my rant about open source development model being
fundamentally unable to deal with aesthetic issues (it was in the 2013
talk), and selecting defaults is the primary example of that. There
isn't always a right answer somebody's just gotta _decide_...

>>> the attached patch includes tests and a fix, though it's not obvious
>>> from the git history why you've gone back and forth here over time,
>>
>> I try to either blog or have a mailing list message about the logic of
>> that sort of thing, which was it this time... Looks like:
>>
>> http://lists.landley.net/pipermail/toybox-landley.net/2017-April/008937.html
>> http://lists.landley.net/pipermail/toybox-landley.net/2017-April/008938.html
> 
> a code comment and/or a checkin comment are more useful though. tests
> better still.

I haven't figured out how to properly test that yet, and the other two
are a balancing act between tl;dr and useful. In this case yeah a
checkin comment with why would probably have helped, but that's
hindsight telling us that. :)

(I often "git annotate COMMIT^1" my way back to when a change happened
and then check the mailing list web archives for relevant discussion.
That's why http://landley.net/kdocs/fullhist/ and
http://lkml.iu.edu/hypermail/linux/kernel/ plus
https://www.kclug.org/old_archives/linux-activists/ are so vital to my
kernel development process. And yes I've got the missing months in mbox
format somewhere, I think I asked on the list and Alan Cox sent them to me?)

> (although this is another case where i don't know how to capture the
> "when on a tty" part of "-q does the wrong thing for ' ' when on a
> tty" without proper unit tests.)

I need to take https://github.com/landley/mkroot and do an automated
thing to run the test suite under it is root, then write tests that
assume _that_ environment. Clean, minimal, known, self-contained, and
(in theory) portably runnable from a commit hook. (But right now my
mkroot cursor is on "why isn't arm64 giving me the virtual block device
and ethernet under the 4.14 kernel". Alas toybox isn't the _only_
project of mine starving for cycles. Working on it...)

I also need to make a pty wrapper that's implements the master side and
runs the rest of the command line under the slave side and is basically
its own terminal emulator so it can read terminal status and x/y
coordinates and so on. But I haven't done that yet because that
plumbing's needed for "screen" so I might as well do both at once, but
the screen command needs plumbing I have to do for vi/less so it's after
that in the todo heap...

>> I try to have ubuntu and toybox both pass my tests, hard to do
>> that when you're testing a place they diverge.
> 
> in the test runner i use for android, i check whether something is a
> symlink to toybox or not. i only use it to print a "note: this isn't
> the toybox tool", but you could use it to explicitly express the
> different expectations, and make it clear when it's deliberate.

Except I run "make test_blah" more often than testing the whole thing,
and that does the standalone build so it's not a symlink. Hmmm, but what
I _can_ do is run command --version and see if the first word of the
result is "toybox"... (So TEST_HOST=1 because TOYBOX_ONLY=1 maybe? Ok,
todo list item added.)

>> And "default output
>> should provide an unambiguous representation of filenames with no
>> collisions" is hard to encapsulate in a test.
> 
> even just including representative examples (like ' ' and ^G) helps,
> and gives you a handy place to link to any mailing list discussion.

I need to spend _months_fluffing out the test suite. It's one of my big
"around 1.0" timesinks, although I'm not sure if it's before or after...

> anyway, yeah, i don't know... if we hadn't already shipped this in O
> i'd be more strongly against. as it is, everyone's expectations are
> now broken under some circumstances anyway. and there's definitely
> something to be said for "default output should provide an unambiguous
> representation of filenames with no collisions", though that ought to
> be documented, especially because it's a deviation from tradition.

Indeed.

> the -q ' ' behavior seems more questionable, but if your actual
> argument is "-q is useless", why not remove it?

It's in posix. (And it's not useless, just less useful. It wildcards out
nonstandard characters, but that opens up the poosibility of collisions
if you feed the output to anything to work on the files.)

Parsing it back in usefully (so one\ two is seen as a single filename)
is still an unsolved problem for bash, but this might just be a bash bug:

  $ for i in one\ two; do echo $i; done
  one two
  $ touch "and space"
  $ for i in $(ls -b and*); do echo $i; done
  and\
  \
  space

What's different about the two contexts there? I'd have to read the
posix shell spec a lot more closely to confirm they never thought of
this and make no mention of how to handle it properly...

> current debian busybox ls doesn't seem to have it,

Busybox not having soemthing is as often an oversight as anything else.
(I left that project before getting around to serious standards
compliance scrutiny on more than a few of the commands, and nobody since
seems to have really cared.)

In this case, A) it hasn't got -b either, B) -q is a posix option,
C) the default behavior of busybox _is_ -q whether the option's there or
not.

$ cd sub &&  busybox ls
l????e????t???? ????t????h????e???? ????r????i????t????u????a????l????
????c????o????m????m????e????n????c????e????
name1
name2
name3
name4
name5
name6
name7
name8
name9
one?two
$ busybox | head -n 1
BusyBox v1.27.2 (2017-09-07 15:58:50 CDT) multi-call binary.

> and having poked at it i'm really
> struggling to see when anyone would _want_ it.

Posix! And it's trivial to implement so removing it doesn't save us much:

$ grep FLAG_q toys/*/ls.c
  if (toys.optflags&FLAG_q) *buf = '?';

> and then your choice of
> -b over -q becomes "obvious". and the documentation becomes just
> another line in your usual "deviations from posix" section (that's at
> least readable to folks working _on_ toybox).

If nothing else I can certainly add a comment to that effect. This
command doesn't have a proper "deviations from posix" section and
probably should, but I wrote it before I was doing those and I haven't
gone back and done a nitpicky "vs posix spec" review yet. (-Z is another
one.)

I would like there to BE a spec, but posix is... Sigh. I'm waiting for
someone specific (http://landley.net/notes-2016.html#11-03-2016) to die
(or simply acknowledge
https://www.hpcwire.com/2017/09/07/oracle-layoffs-reportedly-hit-sparc-solaris-hard/
but I think dying comes first) before going back to the posix committee
and going "the 90's are over, here's a trail of bread crumbs to this
century..."

I don't have a "deviations from LSB" section because that sadly hasn't
been a functional specification since the linux foundation ate the Free
Standards Group. (https://lwn.net/Articles/658809/ was very much a
trailing indicator. As Eben Moglen explained at
https://www.softwarefreedom.org/blog/2016/apr/11/lf/ the linux
foundation is _not_ acting in any sort of public interest or public
good, it's a trade association where dues-paying members band together
to achieve common business purposes. It pretends to represent open
source the exact same way AOL used to pretend to represent the internet,
because that convinces people to give it money. The Linux Foundation is
legally no different from Microsoft's old Software Publisher's
Association, I.E. the "don't copy that floppy" people:
http://www.siia.net/About/About-SIIA )

Seriously, if I won a $100 million powerball and could hire a small
staff I'd have some college students come up with a new standard
(document, not legislate. What are things CURRENTLY doing, and an
implementation is NOT A STANDARD, it's not standardized until two
independent implementations are doing it, and yes the BSD linux
emulation layer counts for linux /proc and friends, and yes making puppy
eyes at a second implementation to ADD something so you can document it
is only very mildly cheating)...

Plus it'd be nice to have somewhere to PUT stuff like the command list
in https://landley.net/notes-2008.html#04-07-2008 and the material I
tried to cover in https://www.youtube.com/watch?v=Sk9TatW9ino (but was
sadly so jetlagged I was borderline incoherent and got through maybe 1/3
of it)...

Ahem. </tangent>

Rob