[SOLVED] Indy 10 TCP ReadLn is very slow with large data?
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
aidv · 2015-10-04 15:20

I send a 3 Megabyte chunk from my client to my server, and when I call
ReadLn from within IdTCPServerExecute
it can take 9 seconds or more to read the data.

How can I speed it up?

This is my code:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i ... var
time: int64; ... procedure THoster.IdTCPServerExecute(AContext:
TIdContext);  var s: ansistring;begin  if
AContext.Connection.IOHandler.InputBuffer.Size = 0 then exit;  
acontext.connection.iohandler.MaxLineLength:= 20000000; //setting
MaxLinelength to 20 MB   time := GetTickCount64;  s :=
AContext.Connection.IOHandler.ReadLn('<EOP>',
IndyTextEncoding_Default, IndyTextEncoding_Default);  log('ReadLn
time: ' + inttostr(GetTickCount64 - time));   //Do processing hereend;

----------------------------------------------------------------------
Laksen · 2015-10-04 15:59

Are you sending text specifically or is it binary data?

Using line buffering is always slower than sending binary data with a
known length since the protocol will need to scan for a terminator,
and do buffering manually.

An example of how you would send binary data (not sure how the Indy
framework does this):
Send length as 8 byte integer
Send N bytes of data

Client will then read 8 byte data, and then read N bytes of data.
There doesn't have to be any buffering and both sides can be done with
two calls to the OS.

----------------------------------------------------------------------
aidv · 2015-10-04 16:08

I send binary, but the packet itself contains text which I need to
read, I guess I need to optimize my protocol a little bit.

I was mostly checking if could be a faster way of doing it.

Thanks for the input.

----------------------------------------------------------------------
aidv · 2015-10-04 16:35

I found the problem.

I am using ansistrings a lot, and using Copy() didn't work with
ansistrings for me so I created my own procedure which
copies parts of an ansistring.

The code looks like this:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
trimFromTo(var aData: ansistring; aFrom, aTo: int64);  var s:
ansistring;begin  for i := aFrom to aTo do s := s + aData[i];  aData
:= s;  s := '';end;
When calling this many times the time added up.

I have now optimized it, so it's super fast.

The new code looks like this:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
trimFromTo(var aData: ansistring; aFrom, aTo: int64);  var s:
ansistring;begin  setLength(s, aTo - aFrom + 1);  Move(aData[aFrom],
s[1], aTo - aFrom + 1);  setLength(aData, length(s));  aData := s;end;
  

Here are my benchmark results!
CodeTimeOld30478 millisecondsNew80 millisecondsResult:381x faster
!!!!!

----------------------------------------------------------------------
balazsszekely · 2015-10-04 16:35

Never send text that large + you need some kind of protocol between
the client and server.

1. Server side
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TfMain.TIdTCPServer1Execute(AContext: TIdContext);var  Command:
String;   Size: Int64;  MS: TMemoryStream;begin   Command :=
AContext.Connection.Socket.ReadLn; //read command  if Command =
'LARGE_TEXT' then //text incomming  begin    MS :=
TMemoryStream.Create;    try      Size :=
AContext.Connection.Socket.ReadInt64;     
AContext.Connection.Socket.ReadStream(MS, Size, False);     
MS.Position := 0;          //Start processing text here, but don't
forget at this time you're inside a thread, so never update visual
controls directly...use TIdSync or TIdNotify.       //The main
difference: TIdSync is synchronous, TIdNotify on the other hand is
asynchronous.      //Memo1.Lines.LoadFromStream(MS); //<---this is
fundamentally wrong, check your log function    finally     MS.Free; 
 end;  end;end; 
2. Client side
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i var  MS:
TMemoryStream;begin  //save txt, picture, etc to memory stream  MS :=
TMemoryStream.Create;  try    Memo1.Line.SaveToStream(MS);   
MS.Position := 0;      with IdTCPClient1.IOHandler do    begin     
 WriteLn('LARGE_TEXT');       Write(MS.Size);              Write(MS,
0);   end;  finally    MS.Free;  end;end;

----------------------------------------------------------------------
aidv · 2015-10-04 17:13

@GetMem I have my own protocol and my server is non-visual. Everything
is non-visual.

My protocol is "dynamic" (if that's even a thing) so I can send
packets of different lengths, mostly around 3 MB.

A packet contains a command, parameters and data. Parameters and data
are optional.

----------------------------------------------------------------------
balazsszekely · 2015-10-04 17:23

Quote@aidv
 have my own protocol and my server is non-visual. Everything is non-
visual.
My protocol is "dynamic" (if that's even a thing) so I can send
packets of different lengths, mostly around 3 MB.
A packet contains a command, parameters and data. Parameters and data
are optional.
Ok. None of that was obvious from the description and the code you
provided, but I'm glad it's working.

----------------------------------------------------------------------
aidv · 2015-10-04 17:58

@GetMem: Sorry about that. I thought I'd keep it simple and just ask
for a specific thing. Thanks for the input, I appreciate it.