[Toybox] Working on a [currently lame] downloader

[Rob Landley]

On 07/14/2015 03:32 PM, Isaac Dunham wrote:
> On Tue, Jul 14, 2015 at 02:17:08PM -0500, Rob Landley wrote:
>> On 07/14/2015 02:25 AM, Isaac Dunham wrote:
>>> On Tue, Jul 14, 2015 at 12:21:28AM -0500, Rob Landley wrote:
>>>> On 07/13/2015 01:18 AM, Isaac Dunham wrote:
>>>>> Hello,
>>>>> I've been working on an HTTP(S) downloader (what wget does, but currently
>>>>> completely incompatible with everything) for toybox.
>>>>> Currently it works to some degree, so I thought I'd mention that it's
>>>>> in progress, ask for a general idea of what's desired, and give people
>>>>> an idea of how completely lame it is right now and how I'm doing it.
>>>>>
>>>>> I presume that the agenda for toybox is implementing some subset of wget
>>>>> in a compatible manner; is "what busybox wget supports + SSL" a rough
>>>>> approximation of the desired functionality?
>>>>
>>>> Yup.
>>>>
>>>>> I mentioned that it's HTTP(S); it fetches files over SSL without
>>>>> implementing SSL. I cheated on networking: it calls netcat or 
>>>>> openssl s_client -quiet -connect.
>>>>
>>>> Huh, I didn't know about that. Cool.
>>>
>>> I actually found out about that from the busybox popmaildir help.
>>>
>>>>> It uses an approach roughly similar to xpopen_both(), except that
>>>>> it uses socketpair() instead of pipe(); it should be possible to switch
>>>>> to xpopen_both(), which would probably fix a few of the bugs.
>>>>> (I'd not realized that there was an xpopen_both until just now.)
>>>>> This strategy is probably the main part that will actually be useful.
>>>>
>>>> Ok. (I note there was no patch attached to this...)
>>>  
>>> Yes, I figured the code as it stands is close to useless; it's more
>>> of a proof-of-concept.
>>
>> So should I wait for a patch form you, or just start poking a this
>> myself? (The lack of a strategy for https was the blocker, the fact
>> openssl has a builtin stunnel makes life much easier...)
> 
> If it's high enough on your todo list, feel free to try for yourself;
> I'm expecting to spend a few days working on a package request for
> Alpine Linux, and you're more familiar with HTTP than I am.
> The extent of my knowledge is GET, 200 OK vs 3xx and 4xx, and the Host:
> and Content-Length: fields. I also know that there is Content-Encoding:
> chunked, but have no idea how to decipher it.
> Also, I'm probably not going to write very good code for dealing with
> the reads...

Following redirects, limiting the number of follows when doing so to
avoid getting caught in circular redirects...

Yeah, there's details.

Rob