[Toybox] [PATCH] Use the correct types for recvfrom.

[enh]

On Fri, Mar 4, 2016 at 2:28 PM, Rob Landley <rob at landley.net> wrote:
> On 03/01/2016 09:18 PM, enh wrote:
>> No worries. Is it easier to keep track of things if I use github pull
>> requests?
>
> Not really. That's not the problem.
>
> Right now I'm trying to figure out how to redo lib/dirtree.c so that rm
> works with infinite depth. (I have a test in the test suite, but haven't
> checked it in yet.) Posix requires infinite recursion depth out of rm,
> but if dirtree closes the parent's filehandle and tries to reacquire it
> later (via openat("..") and drilling down again from the top if that
> doesn't match up, and keeping symlink parent filehandles open because
> reacquiring those is nuts at the best of times), then you have to do
> breadth first search because dirclose() loses your place and order isn't
> guaranteed, so I need a mode that eats filehandles and a mode that eats
> memory.
>
> I haven't written the dirtree code but I've written the start o the test
> case for it:
>
> +# Create insanely long dir beyond PATH_MAX, them rm -rf it
> +
> +// Create directory string 1024 entries deep (and half of PATH_MAX),
> +// then use it to create an 8192 entry directory chain. Note that this
> +// "mv a chain under another chain" technique means you can't even enforce
> +// a $PATH length limit with a mkdir check, the limit can be violated
> +// afterwards, so rm -r _must_ be able to clean up.)
> +//
> +// This one is _excessively_ long to also violate filehandle limits,
> +// so naieve dirtree openat() implementation keeping filehandle to each
> +// parent directory would _also_ exhaust limits (ulimit -Hn = 4096).
> +// (But hopefully not so long we run out of inodes creating it.)
> +X='a/'
> +for i in 1 2 3 4 5; do X=$X$X$X$X; done
> +for i in 1 2 3 4 5 6 7 8
> +do
> +  mkdir -p "$TOPDIR/$i/$X" &&
> +  mv "$TOPDIR/$i ." &&
> +  cd "$i/$X" &&
> +  continue
> +  i=
> +done
> +if [ ! -z "$i" ]
> +then
> +  break 2>/dev/null
> +  exit 1
> +fi
>
> I'm trying to get NFS working with toybox mount. (You'll notice the
> recent "mount doesn't pass through -o leftover string data" fix.) I
> documented a simple NFS test environment years ago, ala
> http://landley.livejournal.com/49382.html (in qemu you mount 10.0.2.2
> rather than 127.0.0.1) but busybox was passing through the old binary
> config blob format instead of the string config format, and there's no
> reason the string version SHOULDN'T work but I haven't made it do so yet
> so I'm sticking printk()s in the kernel I'm running under qemu to see why.
>
> This week I fixed the bzcat integer overflow John Regehr reported, and
> the base64 wrap failure, dealt with Mike's heads on on glibc breaking
> the makedev includes, fixed three different build environment breaks
> (prlimit probe and MS_RELATIME define for uClibc, finit_module probe for
> ubuntu 12.04, ) and redid the test suite so it's consistently printing
> the command name at the start of each test and then factored that out so
> the test infrastructure was doing it.
>
> The need for file to detect cpio files got me looking at cpio again,
> remember my pending "add xattr support to cpio/initramfs" half-finished
> patches, and I also noticed that it wasn't checking the cpio magic
> (fixed that, it accidentally got checked in with the makedev header
> stuff, although the major()/minor() rename in lsof didn't because of
> pending lsof cleanup I didn't want to check in yet).
>
> My pending cleanup of lsof needs to address the fact it takes 18 seconds
> to produce its first line of output when run with no arguments on my
> netbook (the ubuntu version takes 0.7). That's a largeish thing.

do you know why? for me on my desktop, toybox lsof is 10x faster than
the GNU one (2s vs 18s). (that's time to complete. but the initial
pause is longer for the GNU one too.)

> Plus
> this looks like it chould share the ps /proc parsing infrastructure,
> which needs to be genericized into /lib. (I did some but not all of that
> refactoring work in the last interminable round of ps meddling; the
> common code can't access TT.* or FLAGS_* because that's all
> command-specific. Doing that would ALSO let me break out ps/pkill/top
> into separate command files.)
>
> I was using the bash 2.05b limitations (toybox not rebuilding under it
> because jobs -p wasn't implemented yet) as an excuse to reopen the toysh
> can of worms, which I'm tempted to continue because digging into it has
> reminded me that the research I did circa 2006 is full of things I no
> longer remember. (Plus the busybox ash/hush split is sad, and aboriginal
> linux needing to use two shells simultaneously goes against the entire
> point of the project. So I need a proper bash replacement that works on
> nommu, and that means I write one.) But for big things like this I need
> to devote large blocks of time, because chipping away at them produces
> no progress. (I spend all my time figuring out where I left off and why.)
>
> I have a build break in my local sed.c to remind me to add the
> "sed,+NUMBER" range extension feature and "s///NUMBER" extensions.
>
> I have another build break in netcat to remind me to:
>
> A) finish factoring out the xconnect() and xpoll() stuff into lib/net.c,
> taking into account the ipv6 name lookup stuff ala:
>
>   egrep "(->|[.])ai_" toys/*/*.c
>
> Not to mention also converting:
>
>   grep gethostby toys/*/*.c
>
> B) Add the UDP support for:
>
>
> http://sarah.thesharps.us/2009/02/22/debugging-with-printks-over-netconsole/
>
> C) Figure out whether I should merge this with tcpsvd.c and/or telnet.c,
> or if factoring out the common code into lib/ is enough. (Also, does the
> tail -f code work into that merge or stay separate? There's a whole
> infrastructure design rat's nest I need to do a lot of pacing and
> staring off into space about there. It's on the todo list.)
>
> I have xdaemonize_nofork() in lib/portability.c to remind me to do a
> second pass of nommu support over everything. (That's ANOTHER thing to
> fix in netcat.c.)
>
> scripts/install.sh has a one line patch not checked in to remind me of
> THIS issue I need to deal with:
>
> +  # todo: install? --remove-destination? (Will install stomp symlinks?)
>    [ "$1" == "--force" ] && DO_FORCE="-f"
>
> Remember the help.c rewrite to more intelligently shuffle and recombine
> help text? That's pending too.
>
> In tests/expr.test I have:
>
>   +# expr +1
>   +# expr 2+1
>   +# expr 2 + 1
>   +# expr 2 +1
>   +# expr X * 2
>   +# expr X + 2
>
> Which is a year-old reminder of
> http://landley.net/notes-2015.html#30-01-2015 and that's just a side
> issue, the real reason expr needs a rewrite is priority grouping ala
> http://landley.net/notes-2014.html#02-12-2014
>
> In tests/printf.test i have:
>
> +# The posix spec explicitly specifies inconsistent behavior,
> +# so treating the \0066 in %b like the \0066 not in %b is wrong
> +# because posix.
> +testing "printf posix inconsistency" "$PRINTF '\\0066-%b' '\\0066'" \
> +  "\x066-6" "" ""
>
> Which isn't checked in yet so "git diff" shows it and thus I remember
> it. (I hope to get to the point where just HAVING a failing test in the
> test suite reminds me of an issue to fix, but I tried to do a triage
> pass on the test suite last month to split out "contributed test needs
> to go in 'pending' even though I don't have a category for that because
> it's not remotely testing the right things", from "tests are valid but
> incomplete" from "there is no test for this command at all" from "these
> tests need to run as root in a controlled environment and the last time
> I sat down to make an aboriginal linux test environment run under qemu I
> tried to test 'ps' and couldn't figure out how to get output that wasn't
> hugely subject to kernel version skew then got distracted" from "I went
> through every #(%(&# line of the posix spec for this command and I've
> covered every corner case and this one's DONE"...
>
> (Sheesh, the test suite could eat multiple months all on its own...)
>
> I have another note in tests/test.test that "test" is a shell builtin so
> this test suite never actually tested our version, and that I need to
> add a way to to detect this. (VERBOSE=false to symlink the binary to
> /bin/false maybe? No, VERBOSE=hello to substitue the "hello world"
> command out of toys/examples which is _guaranteed_ to produce the wrong
> output, if not the wrong exit code... :)
>
> I was partway through doing vi and less (completing the
> lib/interestingtimes.c and lib/linestack.c stuff), and I really need to
> get back to that before the "I forgot the details of how shells should
> work, need to reread everything again" problem sets in on that. (I was
> halfway through a dd.c rewrite once, tabled because the comma_iterate()
> infrastructure wasn't there yet and I knew I'd need common code for
> mount -o and ps -p and so on. That infrastructure's still not exactly
> DONE, but I've forgotten the details about dd and have to relearn them
> again at this point anyway.)
>
> The recent wget submission from Lipi Lee (the second patch to which
> didn't apply, last hunk failed and I'm not sure why, but it brought up
> that my patch.c not only doesn't handle the git rename and permissions
> extensions which I need to add, but it doesn't even handle that "\ no
> newline at end of file" message if it comes before the last line of the
> diff. I note the failure was "git am" refusing to accept it, neither did
> ubuntu's patch, it was broken for some reason anyway, I just started
> debugging it in mine because I can get better debug output from my code
> with CONFIG_DEBUG and -x. Maybe I should make -x a default option, "why
> this patch didn't apply", but the output's way too verbose...)
>
> Oh, speaking of patch, it's the main user of the "data is char * instead
> of void *" feature of double_list, and I keep meaning to go look if
> double_list would more naturally be something else, and if so how to
> modify patch.c to deal with it. Lots of OTHER things cast away the char
> * when void * wouldn't need a cast...
>
> Anyway, the problem with wget itself (modulo cleanup and adding support
> for cookies and redirects and logins and a progress indicator and making
> sure it handles all the headers from
> https://en.wikipedia.org/wiki/List_of_HTTP_header_fields and...)
>
> The REAL problem is I need to make it understand https:// and shell out
> to that openssh command line utility Isaac Dunham pointed me at last
> year that's better than stunnel. AND I need to make it handle ftp:// and
> combine that with the eventual ftpget/ftpput/ftpd commands (which is why
> i wasn't opening this can of worms yet, I eventually want an httpd that
> can do enough CGI support to run https://github.com/symisc/PH7)
>
> Also, Rich Felker recently fixed strace for nommu, and I have that
> bookmarked:
>
>   https://sourceforge.net/p/strace/mailman/message/34893907/
>
> Along with:
>
>
> https://blog.nelhage.com/2010/08/write-yourself-an-strace-in-70-lines-of-code/
>
> Given that I worked to port strace to hexagon in 2010 I'm reasonably
> familiar with the underlying mechanisms and could probably implement a
> really basic one for toybox in about a week, if I had a spare week.
>
> It's on the todo list. So is collating the unescape logic between
> printf.c, echo.c, sed.c, and sh.c and maybe adding a new wrapper layer
> to handle hex and/or octal escapes. (How that works in with the
> printf.test above, i don't know yet.)
>
> Another nommu issue is that "did exec work or not" is difficult to
> determine, you have to pass the failure back through pipe[3]
>
> On, I need to fix xopen() to pass the "did it exec" failure back through
> pipe[3] to make vfork() be able to detect inability to exec ala:
>
>   http://lkml.iu.edu/hypermail/linux/kernel/0607.1/0837.html
>
> I still haven't gotten all the singleconfig commands working, for
> example "make ftpput" fails because the command is an OLDTOY() alias of
> ftpget and the singleconfig infrastructure isn't smart enough to work
> out what to do to build it. The config is wrong, the help text logic is
> wrong (unless I already fixed the help text logic part, I'd have to
> check. I was working on it and got distracted by emergency du jour.)
>
> I need to figure out if readfile() should trim the \n and if so
> audit/change all the callers (and I thought I'd allowed a chomp()
> variant into lib but couldn't find it when I went to look.)
>
> I'd like to figure out why date is failing with weird error messages:
>
>   $ sudo ./toybox date -D %s 1453236324
>   date: bad date '1453236324'; Tue February 53 23:63:00 CST 2024 !=
>   Wed Mar 26 01:03:00 CDT 2025
>
> Ubuntu's wc -mc shows both, it would be nice if ours could.
>
> This is not a complete list. This is me glancing around at my todo heaps
> and remembering a thing or going "why have I got that browser tab open",
> "what's in THIS directory's todo.txt... ah 'collate todo.txt files out
> of these other three directories', of course..." I have TODO: comments
> at the top of a bunch of toybox files I haven't looked at in ages.
> That's not even counting tackling the rest of the pending directory or
> lib/pending.h, this is just the top couple layers of my todo list. The
> stuff I've been working on _recently_.
>
> And all that's just toybox. I need to do an Aboriginal Linux release
> soon, so I'm trying to figure out how to fix my old version of binutils,
> which the 4.4 kernel broke on arm:
>
>   http://landley.net/notes-2016.html#13-01-2016
>
> And on mips:
>
>   http://landley.net/notes-2016.html#15-02-2016
>
> (And on superh but Rich is doing a workaround for that. The other two
> I've root caused but not fixed yet. The core issue is nobody but me
> regression tests on the last GPLv2 release of binutils anymore, so I
> have to patch either their projects or binutils, and the patches to
> binutils are wandering into "add entire new features" territory.)
>
> I'm trying to finish the Linux From Scratch build control image upgrade
> from LFS 6.8 to LFS 7.8 (a multi-year gap and more or less a complete
> redo, which is where the perl sed fix came from).
>
> Luckily the recent fix for the chromeos guys also fixed the "toybox
> doesn't build under bash 2.05b in aboriginal) problem. The other uClibc
> regressions I checked in fixes for recently.
>
> I'm trying to convert the j-core mercurial repository (which has a
> half-dozen subrepos) to a single unified git repository. I figured out
> that the previous blocker was that git "rename" patches weren't getting
> their paths rewritten consistently and git was dying with an error
> message so misleading it didn't even blame the right _line_. (Found it
> out by removing chunks of patch at the end until I found the specific
> line that made the error manifest.)
>
> My ELC talk was accepted (well, one of them), but it's really a talk
> that my boss Jeff Dionne knows 10x more about, but he has even less time
> than I do, so we somehow have to coordinate on putting together a slide
> deck and co-presenting by April 4.
>
> I've been invited back to talk at Flourish, I have a half-dozen
> potential talks I could give there and need to select/prepare a subset
> for them. That's April 1 and 2, I plan to fly up to Chicago on the 31st,
> be there April 1 and 2, fly to San Diego on the 3rd, be at ELC, probably
> be there for a while longer due to $DAYJOB, and then fly back to Austin.
> (There's presumably a trip to Japan at some point but that's sometime
> after this batch of travel.)
>
> I'm trying to put together an open hardware microconference at plumber's
> (http://wiki.linuxplumbersconf.org/2016:open_hardware) which involves
> asking people if they _would_ want to go (and possibly speak), assuming
> it makes and without explicit promise of travel budget because we dunno
> who will get what yet.
>
> I need to finish migrating nommu.org/jcore to j-core.org, which has
> turned into a complete rewrite of the site. I need to install the VHDL
> development tools from a different tarball onto a different machine
> (updating the install instructions) and then build the current version
> from the converted git repository to post current binaries into a
> downloads directory that does not yet exist.
>
> Oh, and I need to write a VHDL tutorial, which would involve me learning
> VHDL first.
>
> I had a lot more todo items written down on a piece of paper, but I lost it.
>
>> Also, did you say a few months back that you'd started on 'ioctl'? If
>> so, do you want to check that in? If not, correct my misapprehension
>> and I'll do it at some point. (For that and prctl I'm also slowed
>> down by the fact that there's much to dislike about the existing
>> commands...)
>
> Oh right, that. I should go do that.

ping. others can't start fixing it until something's checked in.

(i think it's fine to have a directory even more broken than pending/
if that makes you more comfortable.)

> Rob
>
> P.S. The email notification for
> https://github.com/landley/toybox/pull/26.patch came in since I hit
> "reply" to this message.
>
> P.P.S. Sorry I haven't set up a proper Android test environment.
> Dismantling AOSP is something I really look forward to, and do NOT have
> the desk space for at present...
>
> P.P.P.S. Same for Tizen and chromeos, and I need to resubmit the toybox
> addition patch to buildroot...
>
> P.P.P.P.S. Oh just hit send already.



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.

 1458358263.0