[Toybox] awk seen in the wild

[Rob Landley]

On 07/11/2016 03:10 PM, enh wrote:
> On Mon, Jul 11, 2016 at 12:45 PM, Rob Landley <rob at landley.net> wrote:
>> On 07/11/2016 11:46 AM, enh wrote:
>>> On Sun, Jul 10, 2016 at 10:28 AM, Rob Landley <rob at landley.net> wrote:
>>> another plug for supporting a libcrypto dependency: the *sum utilities
>>> are orders of magnitude faster with libcrypto, the SSL support in
>>> things like netcat/wget would be something i could actually use on
>>> Android (there's no way i'll be able to ship an alternative SSL
>>> implementation) and you'd get arbitrary precision integers too.
>>
>> What I want to do is take the approach Isaac Dunham suggested, of using
>> "openssl s_client -quiet -connect" as an alternative to netcat. So
>> toybox wget should call out to that to get https support, and that would
>> be provided by something external.
>>
>> Lipi Lee did a first pass at this already, which I didn't immediately
>> apply because for some reason the patch he sent me didn't apply to the
>> wget he sent me (I don't think I'd modified it). When I tested the wget
>> it was corrupting the files it downloaded (outputting numbers in the
>> middle of the data), and that sent me down the road of rewriting the
>> thing...
>>
>> I'm balancing some competing design goals here: 'self-contained' vs
>> 'people use this and need speed out of some tightly optimized
>> algorithms'. The way busybox dealt with this was by having multiple
>> implementations (CONFIG_MD5_SMALL has 4 settings), which I very much
>> don't want to do...
>>
>> The problem with having an internal an external implementation is the
>> internal one gets much less testing that way. I suspect the right answer
>> is to just lump it and have the actual unrolled fast version in toybox,
>> because the simple one isn't good enough for the userbase. That said, a
>> lot of these external libraries have assembly optimized versions for
>> various platforms, and I KNOW I'm not going there...
>>
>> Hmmm. Which lib is "libcrypto", by the way?
> 
> any flavor of openssl/libressl/boringssl. they all share the same API
> for this subset.
> 
> boringssl -- which Google uses -- is basically just openssl cut down
> to "what you actually need in the modern world" anyway. boringssl has
> an equivalent that would make the s_client approach work, so that's
> fine by me for netcat/wget.
> 
>>> i don't think you're likely to go this route, but i do like to keep
>>> bringing it up so the idea of being API-compatible enough that it's
>>> possible to use toybox with either your backend or *ssl is in the back
>>> of your mind...
>>>
>>> (no one's complained about the slow *sum commands yet, but if you're
>>> interested i'm happy to send a patch.)
>>
>> People have sent patches to speed up md5sum and sha1sum and it boils
>> down to lots of loop unrolling that makes the algorithm harder to
>> understand. It was back around here:
>> http://lists.landley.net/pipermail/toybox-landley.net/2014-May/006638.html
>>
>> I applied the first few, but the code got very large and very unreadable
>> and I kept hoping there was a way the compiler's darn optimizer could do
>> that for me. I should go back and look at those patches again, but it's
>> competing with 60 other todo items...
> 
> oh, no, my patch just left your portable-but-slow implementations in
> place and called *ssl if configured that way. so it's basically one or
> two extra lines per toy (an if and a function call).

Ok, I'll bite. What does your patch look like?

(If I can shove it in portability.c, retain the option of _not_ doing it
via menuconfig, and probably only support static linking of this extra
library... I'm still mildly concerned about the built-in version getting
less testing, but hashes aren't likely to fail in a _subtle_ manner...)

Rob