[Toybox] wget-axel

[Rob Landley]

On 10/13/14 13:32, stephen Turner wrote:
> Do you or will you have a feature within toybox to do a torrent like
> download using wget or similar command line tool?

I could add a bittorrent client, sure. But probably not until after 1.0.

I haven't done wget yet because about half the web has already switched
over to requiring https, so you these days you can't download from
twitter or github or kernel.org without some form of https support. (I'd
been planning to use "stunnel" but nobody's providing a reasonable
embedded implementation of it. I've pestered dropbear about it for
years, but no...)

> I found a program
> that looks very appealing and i'm hoping its functionality will make it
> into your application at some point.
>
> It splits the data for a single file/location and apparently also
> accepts multiple locations/mirrors.
> 
> http://axel.alioth.debian.org/
> 
> https://docs.oseems.com/general/operatingsystem/linux/command-line-download-accelerator

Oh this old gem.

Way back in the http 1.0 days, the mosaic web browser would open
multiple simultaneous connections to download web page elements faster.
Their stated rationale was each connection had setup/teardown costs, but
the actual problem was that busy websites would apportion the bandwidth
between each connection, and if you did multiple connections in parallel
you'd get multiple shares of the bandwidth. This turned into an arms
race between web browsers for how _many_ parallel connections you could
do, and once they were well into double digits it led to websites
putting in firewall rules to block browsers that did this. (The solution
was http 1.1 and pipelining. Penalizing sites for abusive connections
scaled back when masquerading when mainstream, but for a while there it
was common to account bandwidth per IP instead of per-connection in
fancy load balancers and such.)

Most of the references to this have fallen off the web, but here's
something from that era:

http://chimera.labs.oreilly.com/books/1230000000545/ch11.html

The technique you're using does not actually speed up downloads, since a
single connection can go somewhere around 100 megabytes/second on a
gigabit lan. This does not make the server, the client, or the routers
in between go faster. What that technique _does_ is get multiple shares
of bandwidth from busy websites, at the expense of other users, by
essentially "voting twice".

The only actual benefit from it is your "apparently also" (which does
not sound like the primary use when phrased like that). If this
technique were to become common again, webmins would probably add
firewall rules to treat it as DOS and block your IP from accessing the
site at all for minutes at a time. (Some of the oldtimers still hold a
grudge. And it's trivial to detect.)

That said, an _actual_ bittorrent client would be a reasonable thing to
add, sure.

> Thanks,
> Stephen

Rob

 1413227509.0