[Toybox] [PATCH] Use the correct types for recvfrom.

[Rob Landley]

On 03/01/2016 09:18 PM, enh wrote:
> No worries. Is it easier to keep track of things if I use github pull
> requests?

Not really. That's not the problem.

Right now I'm trying to figure out how to redo lib/dirtree.c so that rm
works with infinite depth. (I have a test in the test suite, but haven't
checked it in yet.) Posix requires infinite recursion depth out of rm,
but if dirtree closes the parent's filehandle and tries to reacquire it
later (via openat("..") and drilling down again from the top if that
doesn't match up, and keeping symlink parent filehandles open because
reacquiring those is nuts at the best of times), then you have to do
breadth first search because dirclose() loses your place and order isn't
guaranteed, so I need a mode that eats filehandles and a mode that eats
memory.

I haven't written the dirtree code but I've written the start o the test
case for it:

+# Create insanely long dir beyond PATH_MAX, them rm -rf it
+
+// Create directory string 1024 entries deep (and half of PATH_MAX),
+// then use it to create an 8192 entry directory chain. Note that this
+// "mv a chain under another chain" technique means you can't even enforce
+// a $PATH length limit with a mkdir check, the limit can be violated
+// afterwards, so rm -r _must_ be able to clean up.)
+//
+// This one is _excessively_ long to also violate filehandle limits,
+// so naieve dirtree openat() implementation keeping filehandle to each
+// parent directory would _also_ exhaust limits (ulimit -Hn = 4096).
+// (But hopefully not so long we run out of inodes creating it.)
+X='a/'
+for i in 1 2 3 4 5; do X=$X$X$X$X; done
+for i in 1 2 3 4 5 6 7 8
+do
+  mkdir -p "$TOPDIR/$i/$X" &&
+  mv "$TOPDIR/$i ." &&
+  cd "$i/$X" &&
+  continue
+  i=
+done
+if [ ! -z "$i" ]
+then
+  break 2>/dev/null
+  exit 1
+fi

I'm trying to get NFS working with toybox mount. (You'll notice the
recent "mount doesn't pass through -o leftover string data" fix.) I
documented a simple NFS test environment years ago, ala
http://landley.livejournal.com/49382.html (in qemu you mount 10.0.2.2
rather than 127.0.0.1) but busybox was passing through the old binary
config blob format instead of the string config format, and there's no
reason the string version SHOULDN'T work but I haven't made it do so yet
so I'm sticking printk()s in the kernel I'm running under qemu to see why.

This week I fixed the bzcat integer overflow John Regehr reported, and
the base64 wrap failure, dealt with Mike's heads on on glibc breaking
the makedev includes, fixed three different build environment breaks
(prlimit probe and MS_RELATIME define for uClibc, finit_module probe for
ubuntu 12.04, ) and redid the test suite so it's consistently printing
the command name at the start of each test and then factored that out so
the test infrastructure was doing it.

The need for file to detect cpio files got me looking at cpio again,
remember my pending "add xattr support to cpio/initramfs" half-finished
patches, and I also noticed that it wasn't checking the cpio magic
(fixed that, it accidentally got checked in with the makedev header
stuff, although the major()/minor() rename in lsof didn't because of
pending lsof cleanup I didn't want to check in yet).

My pending cleanup of lsof needs to address the fact it takes 18 seconds
to produce its first line of output when run with no arguments on my
netbook (the ubuntu version takes 0.7). That's a largeish thing. Plus
this looks like it chould share the ps /proc parsing infrastructure,
which needs to be genericized into /lib. (I did some but not all of that
refactoring work in the last interminable round of ps meddling; the
common code can't access TT.* or FLAGS_* because that's all
command-specific. Doing that would ALSO let me break out ps/pkill/top
into separate command files.)

I was using the bash 2.05b limitations (toybox not rebuilding under it
because jobs -p wasn't implemented yet) as an excuse to reopen the toysh
can of worms, which I'm tempted to continue because digging into it has
reminded me that the research I did circa 2006 is full of things I no
longer remember. (Plus the busybox ash/hush split is sad, and aboriginal
linux needing to use two shells simultaneously goes against the entire
point of the project. So I need a proper bash replacement that works on
nommu, and that means I write one.) But for big things like this I need
to devote large blocks of time, because chipping away at them produces
no progress. (I spend all my time figuring out where I left off and why.)

I have a build break in my local sed.c to remind me to add the
"sed,+NUMBER" range extension feature and "s///NUMBER" extensions.

I have another build break in netcat to remind me to:

A) finish factoring out the xconnect() and xpoll() stuff into lib/net.c,
taking into account the ipv6 name lookup stuff ala:

  egrep "(->|[.])ai_" toys/*/*.c

Not to mention also converting:

  grep gethostby toys/*/*.c

B) Add the UDP support for:


http://sarah.thesharps.us/2009/02/22/debugging-with-printks-over-netconsole/

C) Figure out whether I should merge this with tcpsvd.c and/or telnet.c,
or if factoring out the common code into lib/ is enough. (Also, does the
tail -f code work into that merge or stay separate? There's a whole
infrastructure design rat's nest I need to do a lot of pacing and
staring off into space about there. It's on the todo list.)

I have xdaemonize_nofork() in lib/portability.c to remind me to do a
second pass of nommu support over everything. (That's ANOTHER thing to
fix in netcat.c.)

scripts/install.sh has a one line patch not checked in to remind me of
THIS issue I need to deal with:

+  # todo: install? --remove-destination? (Will install stomp symlinks?)
   [ "$1" == "--force" ] && DO_FORCE="-f"

Remember the help.c rewrite to more intelligently shuffle and recombine
help text? That's pending too.

In tests/expr.test I have:

  +# expr +1
  +# expr 2+1
  +# expr 2 + 1
  +# expr 2 +1
  +# expr X * 2
  +# expr X + 2

Which is a year-old reminder of
http://landley.net/notes-2015.html#30-01-2015 and that's just a side
issue, the real reason expr needs a rewrite is priority grouping ala
http://landley.net/notes-2014.html#02-12-2014

In tests/printf.test i have:

+# The posix spec explicitly specifies inconsistent behavior,
+# so treating the \0066 in %b like the \0066 not in %b is wrong
+# because posix.
+testing "printf posix inconsistency" "$PRINTF '\\0066-%b' '\\0066'" \
+  "\x066-6" "" ""

Which isn't checked in yet so "git diff" shows it and thus I remember
it. (I hope to get to the point where just HAVING a failing test in the
test suite reminds me of an issue to fix, but I tried to do a triage
pass on the test suite last month to split out "contributed test needs
to go in 'pending' even though I don't have a category for that because
it's not remotely testing the right things", from "tests are valid but
incomplete" from "there is no test for this command at all" from "these
tests need to run as root in a controlled environment and the last time
I sat down to make an aboriginal linux test environment run under qemu I
tried to test 'ps' and couldn't figure out how to get output that wasn't
hugely subject to kernel version skew then got distracted" from "I went
through every #(%(&# line of the posix spec for this command and I've
covered every corner case and this one's DONE"...

(Sheesh, the test suite could eat multiple months all on its own...)

I have another note in tests/test.test that "test" is a shell builtin so
this test suite never actually tested our version, and that I need to
add a way to to detect this. (VERBOSE=false to symlink the binary to
/bin/false maybe? No, VERBOSE=hello to substitue the "hello world"
command out of toys/examples which is _guaranteed_ to produce the wrong
output, if not the wrong exit code... :)

I was partway through doing vi and less (completing the
lib/interestingtimes.c and lib/linestack.c stuff), and I really need to
get back to that before the "I forgot the details of how shells should
work, need to reread everything again" problem sets in on that. (I was
halfway through a dd.c rewrite once, tabled because the comma_iterate()
infrastructure wasn't there yet and I knew I'd need common code for
mount -o and ps -p and so on. That infrastructure's still not exactly
DONE, but I've forgotten the details about dd and have to relearn them
again at this point anyway.)

The recent wget submission from Lipi Lee (the second patch to which
didn't apply, last hunk failed and I'm not sure why, but it brought up
that my patch.c not only doesn't handle the git rename and permissions
extensions which I need to add, but it doesn't even handle that "\ no
newline at end of file" message if it comes before the last line of the
diff. I note the failure was "git am" refusing to accept it, neither did
ubuntu's patch, it was broken for some reason anyway, I just started
debugging it in mine because I can get better debug output from my code
with CONFIG_DEBUG and -x. Maybe I should make -x a default option, "why
this patch didn't apply", but the output's way too verbose...)

Oh, speaking of patch, it's the main user of the "data is char * instead
of void *" feature of double_list, and I keep meaning to go look if
double_list would more naturally be something else, and if so how to
modify patch.c to deal with it. Lots of OTHER things cast away the char
* when void * wouldn't need a cast...

Anyway, the problem with wget itself (modulo cleanup and adding support
for cookies and redirects and logins and a progress indicator and making
sure it handles all the headers from
https://en.wikipedia.org/wiki/List_of_HTTP_header_fields and...)

The REAL problem is I need to make it understand https:// and shell out
to that openssh command line utility Isaac Dunham pointed me at last
year that's better than stunnel. AND I need to make it handle ftp:// and
combine that with the eventual ftpget/ftpput/ftpd commands (which is why
i wasn't opening this can of worms yet, I eventually want an httpd that
can do enough CGI support to run https://github.com/symisc/PH7)

Also, Rich Felker recently fixed strace for nommu, and I have that
bookmarked:

  https://sourceforge.net/p/strace/mailman/message/34893907/

Along with:


https://blog.nelhage.com/2010/08/write-yourself-an-strace-in-70-lines-of-code/

Given that I worked to port strace to hexagon in 2010 I'm reasonably
familiar with the underlying mechanisms and could probably implement a
really basic one for toybox in about a week, if I had a spare week.

It's on the todo list. So is collating the unescape logic between
printf.c, echo.c, sed.c, and sh.c and maybe adding a new wrapper layer
to handle hex and/or octal escapes. (How that works in with the
printf.test above, i don't know yet.)

Another nommu issue is that "did exec work or not" is difficult to
determine, you have to pass the failure back through pipe[3]

On, I need to fix xopen() to pass the "did it exec" failure back through
pipe[3] to make vfork() be able to detect inability to exec ala:

  http://lkml.iu.edu/hypermail/linux/kernel/0607.1/0837.html

I still haven't gotten all the singleconfig commands working, for
example "make ftpput" fails because the command is an OLDTOY() alias of
ftpget and the singleconfig infrastructure isn't smart enough to work
out what to do to build it. The config is wrong, the help text logic is
wrong (unless I already fixed the help text logic part, I'd have to
check. I was working on it and got distracted by emergency du jour.)

I need to figure out if readfile() should trim the \n and if so
audit/change all the callers (and I thought I'd allowed a chomp()
variant into lib but couldn't find it when I went to look.)

I'd like to figure out why date is failing with weird error messages:

  $ sudo ./toybox date -D %s 1453236324
  date: bad date '1453236324'; Tue February 53 23:63:00 CST 2024 !=
  Wed Mar 26 01:03:00 CDT 2025

Ubuntu's wc -mc shows both, it would be nice if ours could.

This is not a complete list. This is me glancing around at my todo heaps
and remembering a thing or going "why have I got that browser tab open",
"what's in THIS directory's todo.txt... ah 'collate todo.txt files out
of these other three directories', of course..." I have TODO: comments
at the top of a bunch of toybox files I haven't looked at in ages.
That's not even counting tackling the rest of the pending directory or
lib/pending.h, this is just the top couple layers of my todo list. The
stuff I've been working on _recently_.

And all that's just toybox. I need to do an Aboriginal Linux release
soon, so I'm trying to figure out how to fix my old version of binutils,
which the 4.4 kernel broke on arm:

  http://landley.net/notes-2016.html#13-01-2016

And on mips:

  http://landley.net/notes-2016.html#15-02-2016

(And on superh but Rich is doing a workaround for that. The other two
I've root caused but not fixed yet. The core issue is nobody but me
regression tests on the last GPLv2 release of binutils anymore, so I
have to patch either their projects or binutils, and the patches to
binutils are wandering into "add entire new features" territory.)

I'm trying to finish the Linux From Scratch build control image upgrade
from LFS 6.8 to LFS 7.8 (a multi-year gap and more or less a complete
redo, which is where the perl sed fix came from).

Luckily the recent fix for the chromeos guys also fixed the "toybox
doesn't build under bash 2.05b in aboriginal) problem. The other uClibc
regressions I checked in fixes for recently.

I'm trying to convert the j-core mercurial repository (which has a
half-dozen subrepos) to a single unified git repository. I figured out
that the previous blocker was that git "rename" patches weren't getting
their paths rewritten consistently and git was dying with an error
message so misleading it didn't even blame the right _line_. (Found it
out by removing chunks of patch at the end until I found the specific
line that made the error manifest.)

My ELC talk was accepted (well, one of them), but it's really a talk
that my boss Jeff Dionne knows 10x more about, but he has even less time
than I do, so we somehow have to coordinate on putting together a slide
deck and co-presenting by April 4.

I've been invited back to talk at Flourish, I have a half-dozen
potential talks I could give there and need to select/prepare a subset
for them. That's April 1 and 2, I plan to fly up to Chicago on the 31st,
be there April 1 and 2, fly to San Diego on the 3rd, be at ELC, probably
be there for a while longer due to $DAYJOB, and then fly back to Austin.
(There's presumably a trip to Japan at some point but that's sometime
after this batch of travel.)

I'm trying to put together an open hardware microconference at plumber's
(http://wiki.linuxplumbersconf.org/2016:open_hardware) which involves
asking people if they _would_ want to go (and possibly speak), assuming
it makes and without explicit promise of travel budget because we dunno
who will get what yet.

I need to finish migrating nommu.org/jcore to j-core.org, which has
turned into a complete rewrite of the site. I need to install the VHDL
development tools from a different tarball onto a different machine
(updating the install instructions) and then build the current version
from the converted git repository to post current binaries into a
downloads directory that does not yet exist.

Oh, and I need to write a VHDL tutorial, which would involve me learning
VHDL first.

I had a lot more todo items written down on a piece of paper, but I lost it.

> Also, did you say a few months back that you'd started on 'ioctl'? If
> so, do you want to check that in? If not, correct my misapprehension
> and I'll do it at some point. (For that and prctl I'm also slowed
> down by the fact that there's much to dislike about the existing
> commands...)

Oh right, that. I should go do that.

Rob

P.S. The email notification for
https://github.com/landley/toybox/pull/26.patch came in since I hit
"reply" to this message.

P.P.S. Sorry I haven't set up a proper Android test environment.
Dismantling AOSP is something I really look forward to, and do NOT have
the desk space for at present...

P.P.P.S. Same for Tizen and chromeos, and I need to resubmit the toybox
addition patch to buildroot...

P.P.P.P.S. Oh just hit send already.

 1457130494.0