Subj : Re: Threading in JavaScript
To   : comp.lang.javascript,netscape.public.mozilla.jseng
From : Stephen <ssansom@austin.rr.com>
Date : Mon Feb 03 2003 03:14 am

Gordan wrote:
> Stephen wrote:
> 
> 
snip some...

>>>I am already aware of the XMLHTTP method for downloading things.
>>>Unfortunately, it only allows the download of one file at a time, which
>>>is why I was looking for a different solution that allows me to download
>>>multiple files simultaneously.
>>>
>>
>>Will these (possibly as many as 100) files come from the same server or
>>from multiple different servers? Do you have control over all/any of
>>these servers' configs?
> 
> 
> No, no control over any server should be assumed. In theory, it could be any 
> internet URL.
> 
> 
>>How do you envision "download[ing] multiple files simultaneously"
>>happening across TCP/IP using HTTP? What networking issues have you
>>considered in this context?
> 
> 
> I'm not sure what you mean by this.

Well, this may depend on whether your concept is to obtain these data 
files from a single server or from a multitude of different servers. But ...

I am thinking that once the requests go out the wire you're at the mercy 
of the network. It could be that all the fancy multi-threading in the 
world (assuming this were possible) would come to naught because of 
network considerations:

--overhead for multiple tcp connections rather than one persistent 
connection
--timeout limits placed by server configs on the time allowed for any 
single persistent connection
--limitations server configs might place on the number of simultaneous 
connections allowed from a single client (thus my question above about 
whether you control the servers and thus the servers' configs)
--network congestion that could increase latency because of a large 
number of requests/responses --especially responses coming back to the 
single point of origin
--user's bandwidth

it could be these and related issues would completely negate any 
advantage you obtain (or believe you might obtain) by doing 
"simultaneous downloads"

So ... do you have a vision of what happens on the network and on the 
server if, say, 100 requests from the same client hit the server roughly 
simultaneously? Simply meaning, if the network considerations of a large 
number of "simultaneous" downloads might overwhelm any advantage, then 
the cost outweighs benefit, and we've probably used up more discussion 
time than the issue is worth. But that's an empirical determination yet 
to be made...



  What I would like to do is fork a
> separate connection to each server and go a GET request. 

Is this really necessary? I understand the browser to work something 
like the following:

A stream of text arrives at the browser. This stream of text is an HTML 
page. The browser parses this incoming stream. During this parse it may 
encounter URLs that cause it to issue additional requests for those 
resources. Some of these might be <img src=> or <script src=> or <link 
rel=stylesheet href=> or <iframe src=> or basically anything that has a 
URL that the browser must obtain in order to render or otherwise process 
the page.

The browser will issue these additional requests as it encounters the 
URLs while parsing the input stream; it then continues parsing the 
input. If it encounters another URL it needs, the browser makes this 
next request, and so forth.

This is a broad-stroke scenario, but I believe it is substantially 
accurate. If so, where is the need for multi-threading?

(I'm dealing only with the downloading issue here...)

Then either wait
> for all of them to finish (not very efficient), or start processing for 
> output as each separate download finishes (more desireable, but again, this 
> gets academic without proper threading).
> 

I believe it's been previously mentioned that if the data come from 
different servers from the main page, there could be security issues re 
"processing" this data.

> While there isn't a "hard" limit on the number of pages that may be 
> downloaded at the same time, I do not want to build in a specific limit. 
> Ultimately, it will be limited by the bandwidth and other resources of the 
> usre's machine. 100 files is a very extreme case. In reality, it is not 
> likely to exceed 10, probably closer to 5 in the average case. 

It certainly seems that if the typical number of files is 5 or 10, and 
this happens, say 99+% of the time(?), then the amount of effort you're 
putting into this simultaneous-downloads/multi-threading issue is really 
excessive. Just do things the normal way. (Unless, are these data files 
  Really Huge?) In the one case out of (100?)(1000?)(10,000?) where you 
need 100 files -- well, is it worth all this effort for that small a 
number of cases?


New consideration:
The following may not be very fruitful, but might be worth some 
research... (not a javascript solution, though; applies to downloading 
only)... (And, this deals with obtaining files from the same server.)

There is a feature of HTTP 1.1 that is not, AFAIK, supported at all in 
browsers or HTTPRequest-type approaches, but which you might be able to 
implement in an applet or activex object. This feature is "pipelining" 
in which several requests can be -- well, pipelined -- within the same 
HTTP request.

The normal way is a single "request line" followed by (in http 1.1) 1 or 
more headers. In pipelining, one can place several request lines, in 
sequence, before the headers. See RFC 2616, section 8.1.1 
(Connections/Persistent Connections/Purpose).

Example: instead of ("ordinary" case)

GET /path/to/myfile.html HTTP/1.1
Host: www.somehost.com

you have (pipelining)

GET /path/to/myfile.html HTTP/1.1
GET /path/to/myNextFile.html HTTP/1.1
GET /path to/yetAnotherFile.html HTTP/1.1
Host: www.somehost.com


I also do not know how well supported this is by Servers :-).

Regards,
Stephen

.