[Toybox] awk seen in the wild

[enh]

looks like i never sent my previous patch to the list, and looks like i
lost it on disk too. (it's probably still *somewhere*, but at some point
it's cheaper to stop looking and just do it again...)

attached is a rewrite that shows the basic idea, but misses the step of
refactoring your portable implementation to look like the libcrypto one so
we can remove the 10 lines of duplication. (the majority of the lines added
to this patch is just the same help text duplicated over and over.)

this actually makes toybox faster than the version in ubuntu 14.04.


[PATCH] Add support for libcrypto for MD5/SHA.

Orders of magnitude faster (for architectures where OpenSSL/BoringSSL
has optimized assembler).

Also adds sha224sum, sha256sum, sha384sum, and sha512sum for folks
building with libcrypto.

The fallback portable C implementations could easily be refactored
to be API-compatible, but I don't know whether they'd stay here or
move to lib/ so I've left that part alone for now.
---
 Config.in         |   6 +++
 scripts/make.sh   |   2 +-
 toys/lsb/md5sum.c | 133
++++++++++++++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 135 insertions(+), 6 deletions(-)


On Mon, Jul 11, 2016 at 2:21 PM, Rob Landley <rob at landley.net> wrote:

> On 07/11/2016 03:10 PM, enh wrote:
> > On Mon, Jul 11, 2016 at 12:45 PM, Rob Landley <rob at landley.net> wrote:
> >> On 07/11/2016 11:46 AM, enh wrote:
> >>> On Sun, Jul 10, 2016 at 10:28 AM, Rob Landley <rob at landley.net> wrote:
> >>> another plug for supporting a libcrypto dependency: the *sum utilities
> >>> are orders of magnitude faster with libcrypto, the SSL support in
> >>> things like netcat/wget would be something i could actually use on
> >>> Android (there's no way i'll be able to ship an alternative SSL
> >>> implementation) and you'd get arbitrary precision integers too.
> >>
> >> What I want to do is take the approach Isaac Dunham suggested, of using
> >> "openssl s_client -quiet -connect" as an alternative to netcat. So
> >> toybox wget should call out to that to get https support, and that would
> >> be provided by something external.
> >>
> >> Lipi Lee did a first pass at this already, which I didn't immediately
> >> apply because for some reason the patch he sent me didn't apply to the
> >> wget he sent me (I don't think I'd modified it). When I tested the wget
> >> it was corrupting the files it downloaded (outputting numbers in the
> >> middle of the data), and that sent me down the road of rewriting the
> >> thing...
> >>
> >> I'm balancing some competing design goals here: 'self-contained' vs
> >> 'people use this and need speed out of some tightly optimized
> >> algorithms'. The way busybox dealt with this was by having multiple
> >> implementations (CONFIG_MD5_SMALL has 4 settings), which I very much
> >> don't want to do...
> >>
> >> The problem with having an internal an external implementation is the
> >> internal one gets much less testing that way. I suspect the right answer
> >> is to just lump it and have the actual unrolled fast version in toybox,
> >> because the simple one isn't good enough for the userbase. That said, a
> >> lot of these external libraries have assembly optimized versions for
> >> various platforms, and I KNOW I'm not going there...
> >>
> >> Hmmm. Which lib is "libcrypto", by the way?
> >
> > any flavor of openssl/libressl/boringssl. they all share the same API
> > for this subset.
> >
> > boringssl -- which Google uses -- is basically just openssl cut down
> > to "what you actually need in the modern world" anyway. boringssl has
> > an equivalent that would make the s_client approach work, so that's
> > fine by me for netcat/wget.
> >
> >>> i don't think you're likely to go this route, but i do like to keep
> >>> bringing it up so the idea of being API-compatible enough that it's
> >>> possible to use toybox with either your backend or *ssl is in the back
> >>> of your mind...
> >>>
> >>> (no one's complained about the slow *sum commands yet, but if you're
> >>> interested i'm happy to send a patch.)
> >>
> >> People have sent patches to speed up md5sum and sha1sum and it boils
> >> down to lots of loop unrolling that makes the algorithm harder to
> >> understand. It was back around here:
> >>
> http://lists.landley.net/pipermail/toybox-landley.net/2014-May/006638.html
> >>
> >> I applied the first few, but the code got very large and very unreadable
> >> and I kept hoping there was a way the compiler's darn optimizer could do
> >> that for me. I should go back and look at those patches again, but it's
> >> competing with 60 other todo items...
> >
> > oh, no, my patch just left your portable-but-slow implementations in
> > place and called *ssl if configured that way. so it's basically one or
> > two extra lines per toy (an if and a function call).
>
> Ok, I'll bite. What does your patch look like?
>
> (If I can shove it in portability.c, retain the option of _not_ doing it
> via menuconfig, and probably only support static linking of this extra
> library... I'm still mildly concerned about the built-in version getting
> less testing, but hashes aren't likely to fail in a _subtle_ manner...)
>
> Rob
>



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20160713/29600d35/attachment-0003.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Add-support-for-libcrypto-for-MD5-SHA.patch
Type: text/x-patch
Size: 6630 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20160713/29600d35/attachment-0004.bin>