Post a huge file lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- nagylzs · 2019-10-27 16:31 I'm trying to post a huge file to a server. The file is so big that I cannot load it into memory, and I have to use a progress bar to show how the upload goes. First I was using TFPHTTPClient.StreamFormPost method, because I tought that StreamFormPost will certainly not load the whole file into memory before posting it. But it seems that I was wrong. :-( As it turns out, it does load the whole file into memory here: https://github.com/alrieckert/freepascal/blob/master/packages/fcl- web/src/base/fphttpclient.pp#L2090 This makes the StreamFormPost method useless (not much different from FileFormPost). I wonder if there is a way to truly stream files into a multipart/form-data request withouth loading them to memory? I have done this in many languages, but I don't know how to do it with Lazarus. ---------------------------------------------------------------------- nagylzs · 2019-10-27 16:46 I have also tried TIdHttp with TIdMultiPartFormDataStream and it seems to be working fine. Except that I can't intercept the progress of the file upload. The addFile method of  TIdMultiPartFormDataStream does not accept a custom stream. It only accepts a filename. There is no Indy interceptor that I could use to be notified by the upload progress. In reality, it does create a TFileStream: https://github.com/acras/components/blob/master/thirdparty/Indy9/IdMul tipartFormData.pas#L210 So in theory, I could use a different stream that is tracing the read progress. So it might be the solution... but is it the right way to do it? ---------------------------------------------------------------------- nagylzs · 2019-10-27 17:27 Finally, I could do it. I'm posting the code here, maybe it will help somebody else. There are two custom classes. One allows uploading a file with a custom Stream: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit CustomMultiPartDataStream; {$mode objfpc}{$H+} interface uses  Classes, SysUtils, IdMultipartFormData, IdGlobal, IdGlobalProtocols; type   { TCustomMultipartDataStream }   TCustomMultipartDataStream = class(TIdMultiPartFormDataStream)  public    function AddFileStream(const AFieldName, AFileName: string;      const ACustomFileStream: TIdReadFileExclusiveStream;      const AContentType: string = ''): TIdFormDataField;  end; implementation { TCustomMultipartDataStream } function TCustomMultipartDataStream.AddFileStream(const AFieldName, AFileName: string;  const ACustomFileStream: TIdReadFileExclusiveStream;  const AContentType: string = ''): TIdFormDataField;var  LItem: TIdFormDataField;begin  LItem := FFields.Add;  LItem.FieldName := AFieldName;  LItem.FileName := ExtractFileName(AFileName);  LItem.FieldStream := ACustomFileStream;  if AContentType <> '' then  begin    LItem.ContentType := AContentType;  end  else  begin    LItem.ContentType := GetMIMETypeFromFile(AFileName);  end;  LItem.ContentTransfer := sContentTransferBinary;   Result := LItem;end; end.  The other one is a custom stream that has event handlers for data read: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit ProgressFileStream; {$mode objfpc}{$H+} interface uses  Classes, SysUtils, Types, IdGlobal; type  TProgressFileStream = class;  TProgressFileStreamEvent = procedure (Sender:TProgressFileStream; Size: longint) of object;   { TProgressFileStream }   TProgressFileStream = class(TIdReadFileExclusiveStream)  private    FOnReadProgress : TProgressFileStreamEvent;    FOnWriteProgress : TProgressFileStreamEvent;  public    function Read(var Buffer; Count: Longint): Longint; override;    function Write(const Buffer; Count: Longint): Longint; override;     property OnReadProgress : TProgressFileStreamEvent             read FOnReadProgress write FOnReadProgress;    property OnWriteProgress : TProgressFileStreamEvent             read FOnWriteProgress write FOnWriteProgress;  end; implementation { TProgressFileStream }  function TProgressFileStream.Read(var Buffer; Count: Longint): Longint;begin  Result:=inherited Read(Buffer, Count);  if Assigned(FOnReadProgress) then FOnReadProgress(self, Result);end; function TProgressFileStream.Write(const Buffer; Count: Longint): Longint;begin  Result:=inherited Write(Buffer, Count);  if Assigned(FOnWriteProgress) then FOnWriteProgress(self, Result);end; end.   And this is how to use it: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   src := TProgressFileStream.Create(eFileName.Text);  try    src.OnReadProgress := @OnReadProgress; // You can add your code here for displaying upload progress    response := TStringStream.Create('');    try      try        form := TCustomMultipartDataStream.Create;        try        form.AddFileStream('file', eFileName.Text, src);          IdHTTP.Post('http://localhost:8888/upload', form, response);        finally          form.Free;        end;      except        on e: EAbort do        begin          ShowMessage('User aborted upload.');      end;      end;    finally      response.Free;    end;   finally  src.Free;  end;  If you call Abort from your OnReadProgress event handler then it is propagated to the except clause around IdHTTP.Post. ---------------------------------------------------------------------- winni · 2019-10-27 17:41 Hi! Forget the whole idea: Nearly all internet providers have an upper limit for mails between 10 MB and 25 MB. Winni ---------------------------------------------------------------------- lucamar · 2019-10-27 18:04 Quote from: nagylzs on October 27, 2019, 06:27:57 pmI'm posting the code here, maybe it will help somebody else. Please, see this (in the wiki) to know how to post code correctly (and why you should follow that advice): Forum: Use code tags. TIA! ---------------------------------------------------------------------- nagylzs · 2019-10-27 18:47 Changed my post to code tags. ---------------------------------------------------------------------- nagylzs · 2019-10-27 18:48 Quote from: winni on October 27, 2019, 06:41:38 pmForget the whole idea: Nearly all internet providers have an upper limit for mails between 10 MB and 25 MB. This question has absolutely nothing to do with SMTP or sending e-mails. I think you misunderstood something. ---------------------------------------------------------------------- nagylzs · 2019-11-01 20:13 Update: my code for showing the upload progress works, but Indy itself is broken in a different way. The current latest version cannot use anything above TLS 1.0. That is not something that you would want to use. Update2: I was wrong, Indy can use TLS 1.2, see here: https://forum.la zarus.freepascal.org/index.php/topic,47221.msg338040.html#msg338040