lNet chunked replies issue lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- InRong · 2013-05-31 10:46 Hi,  I am having some strange issues with a large chunked reply. The reply comes in 8192byte HTTP chunks. The "input" trigger fires initially at a much smaller lever (the first one is always slightly under 800 bytes, which I assume is related to the ethernet packet size. The trigger then returns the rest of the 8192 bytes, then the next is 8192, which is all fine. However, the next one is smaller than the first, then the remainder of the 8192, then 8192... All fine so far, and for smaller replies, there is no problem. However after 144 triggers of OnInput, this all stops. The list below shows the sizes of data to the OnInput trigger (code below, but this is fairly standard).  I am using the standard TLHTTPClient from 0.6.5 but with an added ClearExtraHeaders procedure. I can't use the IDE version as I need to compile it in OSX, and the Carbon eventers are not planned to 0.7 Code: [Select]function TForm1.HTTPInput(ASocket: TLHTTPClientSocket;   ABuffer: pchar; ASize: integer): integer; var   oldLength: Integer;   ThisString: String;   lastByte, lastByte2:Char;   testpchar: pChar;   buffSize: Integer; begin   inc(TotalInputs);   inc(ThisInputs);   oldLength := Length(HTTPBuffer);   buffSize:=Length(ABuffer);   setlength(HTTPBuffer,oldLength + ASize);   move(ABuffer^,HTTPBuffer[oldLength + 1], ASize);   Result := aSize; // tell the http buffer we read it all   //If (aSize = 0) then Self.HTTPDoneInput(ASocket);   Self.Memo1.Append('Chunk size was : '+IntToStr(aSize)+'  Buffer Size is : '+IntToStr(Length(aBuffer)));   //Self.Memo1.Append(LeftStr(ABuffer, ASize)); end; Chunk size was : 813  Buffer Size is : 813 Chunk size was : 7379  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8996 Chunk size was : 796  Buffer Size is : 796 Chunk size was : 7396  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8979 Chunk size was : 779  Buffer Size is : 779 Chunk size was : 7413  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8962 Chunk size was : 762  Buffer Size is : 762 Chunk size was : 7430  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8945 Chunk size was : 745  Buffer Size is : 745 Chunk size was : 7447  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8928 Chunk size was : 728  Buffer Size is : 728 Chunk size was : 7464  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8911 Chunk size was : 711  Buffer Size is : 711 Chunk size was : 7481  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8894 Chunk size was : 694  Buffer Size is : 694 Chunk size was : 7498  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8877 Chunk size was : 677  Buffer Size is : 677 Chunk size was : 7515  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8860 Chunk size was : 660  Buffer Size is : 660 Chunk size was : 7532  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8843 Chunk size was : 643  Buffer Size is : 643 Chunk size was : 7549  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8826 Chunk size was : 626  Buffer Size is : 626 Chunk size was : 7566  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8809 Chunk size was : 609  Buffer Size is : 609 Chunk size was : 7583  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8792 Chunk size was : 592  Buffer Size is : 592 Chunk size was : 7600  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8775 Chunk size was : 575  Buffer Size is : 575 Chunk size was : 7617  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8758 Chunk size was : 558  Buffer Size is : 558 Chunk size was : 7634  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8741 Chunk size was : 541  Buffer Size is : 541 Chunk size was : 7651  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8724 Chunk size was : 524  Buffer Size is : 524 Chunk size was : 7668  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8707 Chunk size was : 507  Buffer Size is : 507 Chunk size was : 7685  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8690 Chunk size was : 490  Buffer Size is : 490 Chunk size was : 7702  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8673 Chunk size was : 473  Buffer Size is : 473 Chunk size was : 7719  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8656 Chunk size was : 456  Buffer Size is : 456 Chunk size was : 7736  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8639 Chunk size was : 439  Buffer Size is : 439 Chunk size was : 7753  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8622 Chunk size was : 422  Buffer Size is : 422 Chunk size was : 7770  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8605 Chunk size was : 405  Buffer Size is : 405 Chunk size was : 7787  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8588 Chunk size was : 388  Buffer Size is : 388 Chunk size was : 7804  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8571 Chunk size was : 371  Buffer Size is : 371 Chunk size was : 7821  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8554 Chunk size was : 354  Buffer Size is : 354 Chunk size was : 7838  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8537 Chunk size was : 337  Buffer Size is : 337 Chunk size was : 7855  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8520 Chunk size was : 320  Buffer Size is : 320 Chunk size was : 7872  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8503 Chunk size was : 303  Buffer Size is : 303 Chunk size was : 7889  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8486 Chunk size was : 286  Buffer Size is : 286 Chunk size was : 7906  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8469 Chunk size was : 269  Buffer Size is : 269 Chunk size was : 7923  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8452 Chunk size was : 252  Buffer Size is : 252 Chunk size was : 7940  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8435 Chunk size was : 235  Buffer Size is : 235 Chunk size was : 7957  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8418 Chunk size was : 218  Buffer Size is : 218 Chunk size was : 7974  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8401 Chunk size was : 201  Buffer Size is : 201 Chunk size was : 7991  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8384 Chunk size was : 184  Buffer Size is : 184 Chunk size was : 8008  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8367 Chunk size was : 167  Buffer Size is : 167 Chunk size was : 8025  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8350 Chunk size was : 150  Buffer Size is : 150 Chunk size was : 8042  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8333 Chunk size was : 133  Buffer Size is : 133 Chunk size was : 8059  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8316 Chunk size was : 116  Buffer Size is : 116 Chunk size was : 8076  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8299 Chunk size was : 99  Buffer Size is : 99 Chunk size was : 8093  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8282 Chunk size was : 82  Buffer Size is : 82 Chunk size was : 8110  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8265 Chunk size was : 65  Buffer Size is : 65 Chunk size was : 8127  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8248 Chunk size was : 48  Buffer Size is : 48 Chunk size was : 8144  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8231 Chunk size was : 31  Buffer Size is : 31 Chunk size was : 8161  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8214 Chunk size was : 14  Buffer Size is : 14 Chunk size was : 8178  Buffer Size is : 16383 Chunk size was : 8192  Buffer Size is : 8197  As this shows, there were still 5bytes in the lNet buffer, but the onInput never triggers again. I have checked, and the data flows to the machine, but never make it to my code. At this point, it is around half of what is sent, and I can see the whole thing in wireshark.  The server is a commercial one, so I cannot make any changes to that, and the data is someones phone book, so I cannot post that, so I will try to replicate the behavior of the server wih my own code if I need to. TIA ---------------------------------------------------------------------- InRong · 2013-05-31 15:21 Not sure if it is polite to reply to your own posts, but here goes anyway.....   I am not sure if it is a workaround or a solution, but I have changed the following constants in lhttp   RequestBufferSize = 1024;   DataBufferSize = 16*1024; to   RequestBufferSize = 4*1024;   DataBufferSize = 32*1024;  The RequestBufferSize on it's own solved it, but I thought the relative sizes were too close then. Taking RequestBufferSize to 16k made things look much more stable on the size of the "chunks" being given to the OnInput trigger.  Looking at this more, I can see the logic in the size of the buffer vs the size of the data chunks causing issues, but I have been looking at this for days, and only see it after asking everyone else.... ---------------------------------------------------------------------- BigChimp · 2013-05-31 15:50 Better answer your own post with good info than not doing it, I'd say. Thanks for the info and glad it works! ---------------------------------------------------------------------- Almindor · 2013-07-21 23:20 Thanks for the info. I'll change the constants to your values to make http a bit more consistent. Is there a way I could reproduce this somehow? There's definitely an underlying problem that needs solving, probably on the TCP level.