From mi@aldan.algebra.com  Mon Dec 31 18:28:09 2007
Return-Path: <mi@aldan.algebra.com>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id CBBDD16A417
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Dec 2007 18:28:09 +0000 (UTC)
	(envelope-from mi@aldan.algebra.com)
Received: from aldan.algebra.com (aldan.algebra.com [216.254.65.224])
	by mx1.freebsd.org (Postfix) with ESMTP id 8414113C458
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Dec 2007 18:28:09 +0000 (UTC)
	(envelope-from mi@aldan.algebra.com)
Received: from aldan.algebra.com (localhost [127.0.0.1])
	by aldan.algebra.com (8.14.1/8.14.1) with ESMTP id lBVI04Il004918
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Dec 2007 13:00:04 -0500 (EST)
	(envelope-from mi@aldan.algebra.com)
Received: (from mi@localhost)
	by aldan.algebra.com (8.14.1/8.14.1/Submit) id lBVI046T004917;
	Mon, 31 Dec 2007 13:00:04 -0500 (EST)
	(envelope-from mi)
Message-Id: <200712311800.lBVI046T004917@aldan.algebra.com>
Date: Mon, 31 Dec 2007 13:00:04 -0500 (EST)
From: "Mikhail T." <mi@aldan.algebra.com>
To: FreeBSD-gnats-submit@freebsd.org
Subject: [wish] parallel download from multiple servers by fetch
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         119196
>Category:       bin
>Synopsis:       fetch(1): [request] parallel download from multiple servers by fetch
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    des
>State:          suspended
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Mon Dec 31 18:30:02 UTC 2007
>Closed-Date:    
>Last-Modified:  Thu May 01 06:31:31 UTC 2008
>Originator:     Mikhail T.
>Release:        FreeBSD 6.3-PRERELEASE amd64
>Organization:
Virtual Estates, Inc.
>Environment:
	FreeBSD

>Description:
	In most cases, when fetch(1) is invoked (or libfetch called
	into), it is to either download sources for a port, or
	binaries for a package.

	In all of those cases multiple servers hosting the same files
	are known to the caller. Unfortunately, the caller must use
	the servers /in sequence/, because fetch does not allow
	downloading different ranges of the same file from different
	sources /in parallel/.

	This is, of course, suboptimal as some servers (such as
	ftp.freebsd.org) are hit by much higher loads. The users
	may also have a (much) wider download bandwidth, than an
	upload bandwidth, that a particular server has (or would
	allow a single client to use).

	It should be possible to implement parallel downloads (over
	different protocols -- http, ftp, etc.). The implementation
	should also be adaptive: if the download from one of the
	servers finishes faster, the program should proceed to
	automatically fetch a not-yet-downloaded sub-range of the
	range, that is being downloaded from another server.

	Various "p2p" downloading programs (torrent, gnuttella)
	implement this feature already -- out of necessity. We
	should do it, because we can.

	No, I don't have a patch. Not yet, anyway... Somebody looking
	for a cool project is most welcome to this idea.

>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->des 
Responsible-Changed-By: kris 
Responsible-Changed-When: Tue Jan 1 11:16:16 UTC 2008 
Responsible-Changed-Why:  
Assign to fetch maintainer 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119196 

From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des@des.no>
To: "Mikhail T." <mi@aldan.algebra.com>
Cc: FreeBSD-gnats-submit@FreeBSD.org
Subject: Re: bin/119196: [wish] parallel download from multiple servers by fetch
Date: Wed, 02 Jan 2008 19:28:47 +0100

 Unfortunately, this is not possible within the framework of libfetch as
 it stands today.  Libfetch was designed and implemented with very
 specific requirements, namely that the API should provide a drop-in
 replacement for fopen(3) (fetchGetURL(3), which pkg_add(1) uses) and
 that it should cache server connections between invocations.  As a
 consequence, libfetch must store a lot of state in global variables, and
 is not parallelizable.
 
 DES
 -- 
 Dag-Erling Sm=C3=B8rgrav - des@des.no

From: Mikhail Teterin <mi+mill@aldan.algebra.com>
To: Dag-Erling =?utf-8?q?Sm=C3=B8rgrav?= <des@des.no>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: bin/119196: [wish] parallel download from multiple servers by fetch
Date: Wed, 2 Jan 2008 13:41:55 -0500

 =D1=81=D0=B5=D1=80=D0=B5=D0=B4=D0=B0 02 =D1=81=D1=96=D1=87=D0=B5=D0=BD=D1=
 =8C 2008 01:28 =D0=BF=D0=BE, Dag-Erling Sm=C3=B8rgrav =D0=92=D0=B8 =D0=BD=
 =D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB=D0=B8:
 > Unfortunately, this is not possible within the framework of libfetch as
 > it stands today.  Libfetch was designed and implemented with very
 > specific requirements, namely that the API should provide a drop-in
 > replacement for fopen(3) (fetchGetURL(3), which pkg_add(1) uses) and
 > that it should cache server connections between invocations.  As a
 > consequence, libfetch must store a lot of state in global variables, and
 > is not parallelizable.
 
 Well, you wrote it -- can you try to reduce the number of places, where the
 global variables are used (implicitly) in favor of an explicitly passed 
 context-pointer or some such?
 
 No necessarily to the full extent of implementing the wish, but to simply make 
 some other would-be implementer's job easier...
 
 Or should we take libcurl as the basis instead? I don't know their API, but
 the license claims to be MIT/X-derivative:
 
 http://64.233.169.104/search?q=cache:GGdoH55jw08J:curl.haxx.se/docs/copyright.html
 
 so it may be Ok to include it into our tree some day...
 
  -mi
State-Changed-From-To: open->suspended 
State-Changed-By: linimon 
State-Changed-When: Sat Jan 26 05:15:47 UTC 2008 
State-Changed-Why:  
It does not sound as though this is being actively worked on. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119196 
>Unformatted:
