[Toybox] [PATCH] Implement mv -n / cp -n (no clobber).

[Rob Landley]

On 04/15/2016 02:40 PM, Andy Chu wrote:
> On Fri, Apr 15, 2016 at 10:16 AM, enh <enh at google.com> wrote:
>> things to avoid/reasons why i'd happily get rid of mksh in Android:
> 
> OK interesting -- can you say more about the use cases of shell and
> toybox in Android?
> 
> For example, why does Android need toybox expr?


A) It's in posix,

B) It shares infrastructure with the shell $((math)) functions,

C) It's used rather a lot in autoconf and shell scripts. Let's see,the
first package that always broke in my linux from scratch builds when I
was making busybox work with that was binutils...

$ grep -wr expr binutils | wc -l
27429

$ grep -w expr binutils/configure | head
if expr a : '\(a\)' >/dev/null 2>&1; then
  as_expr=expr
  as_lineno_3=`(expr $as_lineno_1 + 1) 2>/dev/null`
  as_lineno_3=`(expr $as_lineno_1 + 1) 2>/dev/null`
if expr a : '\(a\)' >/dev/null 2>&1; then
  as_expr=expr
  ac_optarg=`expr "x$ac_option" : 'x[^=]*=\(.*\)'`
    ac_feature=`expr "x$ac_option" : 'x-*disable-\(.*\)'`
    expr "x$ac_feature" : ".*[^-_$as_cr_alnum]" >/dev/null &&
    ac_feature=`expr "x$ac_option" : 'x-*enable-\([^=]*\)'`


$ grep -w expr binutils/*.sh |head
	  extracted_serial=`expr $extracted_serial + 1`
		  match_pattern_regex=`expr "$deplibs_check_method" : "$1 \(.*\)"`
		major=`expr $current - $age`
	    current=`expr $number_major + $number_minor`
	    current=`expr $number_major + $number_minor - 1`
	  major=.`expr $current - $age`
	  minor_current=`expr $current + 1`
	  major=`expr $current - $age + 1`
	    iface=`expr $revision - $loop`
	    loop=`expr $loop - 1`

> Does some shell script use it?

In the current Linux kernel build? (Well, monday's git, anyway?) Yes:

$ grep -w expr Makefile scripts/decodecode
Makefile:	expr $(VERSION) \* 65536 + 0$(PATCHLEVEL) \* 256 +
0$(SUBLEVEL)); \
scripts/decodecode:width=`expr index "$code" ' '`
scripts/decodecode:marker=`expr index "$code" "\<"`
scripts/decodecode:	marker=`expr index "$code" "\("`

> I only know a little about Android (from building AOSP
> 18 months ago), but my impression was that the use of shell scripts
> was limited, and they were all "in-tree" so you can patch them.

If you want to turn Android into a build environment that has at least
the _option_ to build existing packages, you need expr, yes.

> So if you had some upstream packages that used shell for an init
> script or something, you could patch it to use $(()) in mksh instead
> of expr, and avoid expr.

You could, yes. But toybox's shell needs a $(()) implementation and it
might as well share code with expr.

> Or maybe it's just not worth the effort to
> do that, and it's easier to have expr on there.  (I thought I was the
> only one who used expr :) )

It's in posix. It's used by shell scripts and makefiles in the wild.
Somebody other than me submitted an implementation to toybox.

> Who uses the shell?  I would have guessed it's for the boot process
> mainly?

Didn't he just said that a shell problem broke his build?

Doesn't adb launch a shell?

> I don't think people are installing compilers on Android
> phones and building GNU packages, but I could be mistaken.  And I
> would think the main use cases right now for toybox are on the device,
> not for platform and NDK builds, no?

Toybox is not at 1.0. Toybox is not ready to carry the weight of a
self-hosting build environment by itself yet. That's why it's not at 1.0.

> But maybe it is useful for build
> with toybox to make the build hermetic.  I know all the compilers are
> vendored in.
> 
> I imagine the big users now are not end users or app developers, but
> platform developers (e.g. Samsung engineers) who need to tweak things
> when bringing up the OS on new hardware.  I'm able to run toybox on my
> Nexus 5x via ConnectBot, though honestly I'm not sure what to do with
> it since it's limited to an individual app UID  :)

On my nexus 5 I installed "terminal" out of the app store (still dunno
why that's not there by default since it's in the AOSP build) and got a
shell prompt with toybox available.

Aboriginal Linux never requires root to build. In theory AOSP shouldn't
either. Instead I provide ownership and permission overrides to the
initramfs, squashfs, or ext2 filesystem image generators when packaging
up the directory. In theory whatever image type android's making could
similarly do that. An awful lot of the requirements for that went away
when devtmpfs happened, because now you don't need to mknodin the new
filesystem.

Speaking of which, I have a pending todo item to make
CONFIG_DEVTMPFS_MOUNT apply to initmpfs. I should finish and submit that...



> I think it would be cool to have my shell on Android, and it's written
> in Google-style C++ like Ninja like I said, with unit tests, no
> globals and (tasteful) dependency injection and everything.  It's only
> 2 weeks of work and 2800 lines of code now, so that's a long ways off
> :)

Toybox needs an integrated shell and isn't going to import c++ code, but
if android winds up using your shell instead of that one it's Elliott's
call, not mine.

Writing a shell isn't actually _hard_, what it is is _elaborate_.
There's a lot of surface area to cover, and I want to do it _right_
which means not stopping and restarting 30 times while interrupted by
something else. (Once I open this can of worms it's going to eat 6
months of my life.)

That said I may need to start on this for $DAYJOB soon, since the only
nommu shell we've got right now is busybox hush and they don't want to
ship gpl code in userspace either, especially from a project with a
history of litigation. (Can't say I blame 'em...)

> But I'm curious about the use cases.  I think I can probably get some
> other shell users at Google to contribute, because I've done a lot of
> the hard parts in a clean way.  Lexing and parsing shell are
> particularly hard IMO; compare with mksh's yylex() and associated
> macros in lex.c, which is about 1000 lines in a single function, and
> it only does part of the job my lexer does.

*shrug* Toybox will eventually have a shell anyway, but if you want a
different one written in a different language, have fun?

> thanks,
> Andy

Rob

 1460752514.0