[Toybox] Working on a [currently lame] downloader

[Isaac Dunham]

On Tue, Jul 14, 2015 at 02:17:08PM -0500, Rob Landley wrote:
> On 07/14/2015 02:25 AM, Isaac Dunham wrote:
> > On Tue, Jul 14, 2015 at 12:21:28AM -0500, Rob Landley wrote:
> >> On 07/13/2015 01:18 AM, Isaac Dunham wrote:
> >>> Hello,
> >>> I've been working on an HTTP(S) downloader (what wget does, but currently
> >>> completely incompatible with everything) for toybox.
> >>> Currently it works to some degree, so I thought I'd mention that it's
> >>> in progress, ask for a general idea of what's desired, and give people
> >>> an idea of how completely lame it is right now and how I'm doing it.
> >>>
> >>> I presume that the agenda for toybox is implementing some subset of wget
> >>> in a compatible manner; is "what busybox wget supports + SSL" a rough
> >>> approximation of the desired functionality?
> >>
> >> Yup.
> >>
> >>> I mentioned that it's HTTP(S); it fetches files over SSL without
> >>> implementing SSL. I cheated on networking: it calls netcat or 
> >>> openssl s_client -quiet -connect.
> >>
> >> Huh, I didn't know about that. Cool.
> > 
> > I actually found out about that from the busybox popmaildir help.
> > 
> >>> It uses an approach roughly similar to xpopen_both(), except that
> >>> it uses socketpair() instead of pipe(); it should be possible to switch
> >>> to xpopen_both(), which would probably fix a few of the bugs.
> >>> (I'd not realized that there was an xpopen_both until just now.)
> >>> This strategy is probably the main part that will actually be useful.
> >>
> >> Ok. (I note there was no patch attached to this...)
> >  
> > Yes, I figured the code as it stands is close to useless; it's more
> > of a proof-of-concept.
> 
> So should I wait for a patch form you, or just start poking a this
> myself? (The lack of a strategy for https was the blocker, the fact
> openssl has a builtin stunnel makes life much easier...)

If it's high enough on your todo list, feel free to try for yourself;
I'm expecting to spend a few days working on a package request for
Alpine Linux, and you're more familiar with HTTP than I am.
The extent of my knowledge is GET, 200 OK vs 3xx and 4xx, and the Host:
and Content-Length: fields. I also know that there is Content-Encoding:
chunked, but have no idea how to decipher it.
Also, I'm probably not going to write very good code for dealing with
the reads...

Thanks,
Isaac Dunham