[Toybox] awk seen in the wild

[enh]

On Mon, Jul 11, 2016 at 12:45 PM, Rob Landley <rob at landley.net> wrote:
> On 07/11/2016 11:46 AM, enh wrote:
>> On Sun, Jul 10, 2016 at 10:28 AM, Rob Landley <rob at landley.net> wrote:
>> another plug for supporting a libcrypto dependency: the *sum utilities
>> are orders of magnitude faster with libcrypto, the SSL support in
>> things like netcat/wget would be something i could actually use on
>> Android (there's no way i'll be able to ship an alternative SSL
>> implementation) and you'd get arbitrary precision integers too.
>
> What I want to do is take the approach Isaac Dunham suggested, of using
> "openssl s_client -quiet -connect" as an alternative to netcat. So
> toybox wget should call out to that to get https support, and that would
> be provided by something external.
>
> Lipi Lee did a first pass at this already, which I didn't immediately
> apply because for some reason the patch he sent me didn't apply to the
> wget he sent me (I don't think I'd modified it). When I tested the wget
> it was corrupting the files it downloaded (outputting numbers in the
> middle of the data), and that sent me down the road of rewriting the
> thing...
>
> I'm balancing some competing design goals here: 'self-contained' vs
> 'people use this and need speed out of some tightly optimized
> algorithms'. The way busybox dealt with this was by having multiple
> implementations (CONFIG_MD5_SMALL has 4 settings), which I very much
> don't want to do...
>
> The problem with having an internal an external implementation is the
> internal one gets much less testing that way. I suspect the right answer
> is to just lump it and have the actual unrolled fast version in toybox,
> because the simple one isn't good enough for the userbase. That said, a
> lot of these external libraries have assembly optimized versions for
> various platforms, and I KNOW I'm not going there...
>
> Hmmm. Which lib is "libcrypto", by the way?

any flavor of openssl/libressl/boringssl. they all share the same API
for this subset.

boringssl -- which Google uses -- is basically just openssl cut down
to "what you actually need in the modern world" anyway. boringssl has
an equivalent that would make the s_client approach work, so that's
fine by me for netcat/wget.

>> i don't think you're likely to go this route, but i do like to keep
>> bringing it up so the idea of being API-compatible enough that it's
>> possible to use toybox with either your backend or *ssl is in the back
>> of your mind...
>>
>> (no one's complained about the slow *sum commands yet, but if you're
>> interested i'm happy to send a patch.)
>
> People have sent patches to speed up md5sum and sha1sum and it boils
> down to lots of loop unrolling that makes the algorithm harder to
> understand. It was back around here:
> http://lists.landley.net/pipermail/toybox-landley.net/2014-May/006638.html
>
> I applied the first few, but the code got very large and very unreadable
> and I kept hoping there was a way the compiler's darn optimizer could do
> that for me. I should go back and look at those patches again, but it's
> competing with 60 other todo items...

oh, no, my patch just left your portable-but-slow implementations in
place and called *ssl if configured that way. so it's basically one or
two extra lines per toy (an if and a function call).

> According to http://valerieaurora.org/hash.html both md5sum and sha1sum
> are semi-obsolete. I need to do sha256 and sha3 and so on, and adding

well, i did sha256, sha384, and sha512 too. no reason not to when
they're all there for free.

plus being able to say "if you care about performance, link against
*ssl and get hand-optimized assembler for your specific target
platform" gets all the loop unrollers off your lawn forever :-)

> those is a higher priority todo item for me than speeding up the old
> ones. But despite being obsolete as _cryopto_, rsync moved from md4sum
> to md5sum a couple years back, and git is based on sha1, and neither is
> moving off those because you wrap the transport in https and sign the
> commits if you care about security...
>
> My deflate implementation is also half the speed it should be, although
> the first pass was focusing on correctness rather than any kind of
> optimization. (In theory zlib started life as a public domain
> implementation, but that version seems to have fallen off the net and
> doesn't have a lot of modern optimizations anyway.)
>
> One thing I've been meaning to do with deflate/inflate is add the SMP
> mode (where it outputs zero byte packets at each dictionary reset so you
> can scan ahead for them and do blocks in parallel).
>
> So many todo items...
>
> Rob



-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.