lnet, best algorithm to send files over a network using TLTCP and maximize speed lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- RickD · 2016-12-29 18:16 Hello all, I build a basic framework with lnet to send files over a network using TLTCP. The whole thing works great, however i only get speeds up to 8% of the maximum network capacity. Right now i use a pingpong system like this: client   : request packet server : send packet client   : process packet and request new one server : send new packet I fixed part of the problem by using bigger packets (started with 200 bytes, now use 1250 bytes) which lowers the number of requests. And increases the transfer speed. For some reason around 1250 bytes seems the maximum size of a packet (i guess lnet has an upper limit for packet size), so i can't improve much more on that. My next idea is to use OnCanSend and just rapid fire the packets without requests to remove most of the communication latency (does not seem very elegant though). However before i try that i want to check here what the best way to tackle this problem is. I'm sure a lot of other people have worked on this and there is no need to reinvent the wheel. So any ideas, tips and tricks on this are much appreciated! Regards, Rick ---------------------------------------------------------------------- RickD · 2017-01-07 14:17 An update: I rebuild the project with OnCanSend to just fire packets when possible, but most packets seem to be lost in the void of my home network or are received damaged. Also no error event seems to be triggered. Not really sure what goes wrong. As far as i know tcp/ip should take care of the syncing and make sure the packets are received in order and not damaged (in theory). ---------------------------------------------------------------------- balazsszekely · 2017-01-07 22:05 QuoteFor some reason around 1250 bytes seems the maximum size of a packet (i guess lnet has an upper limit for packet size), so i can't improve much more on that. This is the limitation of the network. You can try something like this: 1. client   : request file 2. server : send filesize 3. server : send file in chunks Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i const  BufferSize = 1024;var  Fs: TFileStream;  TotalCount: Integer;  Buffer: Pointer;  BytesToRead: Integer;  begin  Fs := TFileStream.Create('pathtoyourfile', fmOpenRead or fmshareDenyWrite);  try    TotalCount := 0;    GetMem(Buffer, BufferSize);    repeat      if TotalCount + BufferSize > Fs.Size then        BytesToRead := FS.Size - TotalCount      else        BytesToRead := BufferSize;      Fs.Read(Buffer^, BytesToRead);      FCon.Send(Buffer, BufferSize);    TotalCount := TotalCount + BytesToRead;    until TotalCount >= FS.Size;    FreeMem(Buffer);  finally    Fs.Free;  end;                end; 4. client : receive file size 5. client : receive file in loop(almost same code as above), you should read until the streamsize becomes equal to the filesize received previously(step 4) ---------------------------------------------------------------------- RickD · 2017-01-09 07:54 Quote from: GetMem on January 07, 2017, 11:05:25 pmQuoteFor some reason around 1250 bytes seems the maximum size of a packet (i guess lnet has an upper limit for packet size), so i can't improve much more on that. This is the limitation of the network. You can try something like this: 1. client   : request file 2. server : send filesize 3. server : send file in chunks Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i const  BufferSize = 1024; [snip]     Fs.Free;  end;                  end; 4. client : receive file size 5. client : receive file in loop(almost same code as above), you should read until the streamsize becomes equal to the filesize received previously(step 4) I used a similar, but event based method (like the example on the lnet site, which is supposed to work correctly): Note that the block_size:=200 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TLTCPTest.OnCanSendL(aSocket: TLSocket);varSent,i: Integer; begin  if (global_packet_index>0) then begin        repeat                begin                try                BlockRead(F, packet_buffer[5] , block_size, i);                except end;                inc(global_packet_index);                if (i ',sent);              end;        until ( (sent=0) or (global_packet_index=0) ); end; end;  The result : no error events are fired and , sent is always > 0. However the packets are not received correctly even though OnCanSend should handle the network syncing. Seems to me that everything is correct, but the system just can't handle the speed without any extra syncing. Also note that if i activate the //writeln('---> ',sent); line, more packets are received correctly , which indicates the extra writeln pause makes things better. But if anybody sees any mistakes in my method, please let me know .. ---------------------------------------------------------------------- balazsszekely · 2017-01-09 13:58 QuoteThe result : no error events are fired and , sent is always > 0. However the packets are not received correctly even though OnCanSend should handle the network syncing. Seems to me that everything is correct, but the system just can't handle the speed without any extra syncing. Also note that if i activate the //writeln('---> ',sent); line, more packets are received correctly , which indicates the extra writeln pause makes things better. But if anybody sees any mistakes in my method, please let me know ..There is not enough code to see the mistake(if any). I will make a small demo application today or tomorrow. ---------------------------------------------------------------------- balazsszekely · 2017-01-09 22:15 Ok. I made a very basic demo. Just change the parameters for Client/FTCPClient.GetFile. You need to provide remote/local files. https://drive.google.com/open?id=0B9Me_c5onmWoeHg2M2pldW5wZ2s PS: I like Indy/Synapse much better then lnet. ---------------------------------------------------------------------- RickD · 2017-01-10 15:24 Quote from: GetMem on January 09, 2017, 11:15:29 pmOk. I made a very basic demo. Just change the parameters for Client/FTCPClient.GetFile. You need to provide remote/local files. https://drive.google.com/open?id=0B9Me_c5onmWoeHg2M2pldW5wZ2s PS: I like Indy/Synapse much better then lnet. Thanks for posting this. I will look into it later this week. ---------------------------------------------------------------------- Awesome Programmer · 2017-04-18 14:53 Quote from: GetMem on January 09, 2017, 11:15:29 pmOk. I made a very basic demo. Just change the parameters for Client/FTCPClient.GetFile. You need to provide remote/local files. https://drive.google.com/open?id=0B9Me_c5onmWoeHg2M2pldW5wZ2s PS: I like Indy/Synapse much better then lnet. I have looked at your demo for LNET, but I am not understanding your uclient receive procedure or event. Your code doesn't take into account that the file might be received in chunks not in one BIG CHUNK. You just read once for a file and then execute rest of the code. Maybe that's why you made your Buffersize big 1048578 in size. Maybe that is big enough of a size for most of the files in the world. However, you may not receive or read file in one big chunk... If you could clarify your code, that would so helpful.