[SOLVED] Speed-up masive file writing. lazarus / 62-General ---------------------------------------------------------------------- miki · 2016-08-22 22:27 Hi everybody  I'm making a program with some test and PoC, in both FreePascal [3.0] and Java [1.8]. I have a function to write integer arrays into text files (each line is a number). The Java code: Code: Javascript  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     public void arrayToFile(int[] array, String fileName) throws IOException    {        BufferedWriter writer = new BufferedWriter(new FileWriter(new File(fileName)));        for (int number: array)        {            writer.write(Integer.toString(number));            writer.newLine();      }        writer.close();    }  This code takes 700~800ms to save 10,000,000 integers, creating a 110MB file. When writing this in FreePascal, I first came to this solution Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure arrayToFile(numbers: Array of Integer; fileName: String);var    list: TStrings;    i, len: Integer;begin    list := TStringList.Create;    len := Length(numbers) - 1;    for i := 0 to len do    begin        list.Add(IntToStr(numbers[i]));    end;    list.SaveToFile(FileName);    list.Free;end;  It took 2.2~2.8 seconds. Setting list.Capacity or list.Begin/EndUpdate had no measurable effect. Searching for something better, I tried this approach: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure arrayToFile(numbers: Array of Integer; fileName: String);var    data: TMemoryStream;    i, len: Integer;    Str: String;begin    data := TMemoryStream.Create;    len := Length(numbers) - 1;    for i := 0 to len do    begin        Str := IntToStr(numbers[i]) + lineEnding;      data.write(str[1], Length(str));    end;    data.Position := 0;    data.SaveToFile(fileName);    data.Free;    done;end;  I removed the sizeOf(char) part, because I expect this to be always 1, right? This was better, 1.8~2.0 seconds. Still slower than Java. Searching further, I also found this other thing: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure arrayToFile(numbers: Array of Integer; fileName: String);var    data: TMemoryStream;    i, len: Integer;    str, t: String;begin    data := TMemoryStream.Create;    len := Length(numbers) - 1;    for i := 0 to len do    begin        System.Str(numbers[i], t);        Str := t + lineEnding;        data.write(str[1], Length(str));    end;    data.Position := 0;    data.SaveToFile(fileName);    data.Free;end;  Faster!!! 1.0~1.2 seconds. Still slower than Java, but very close. The question: any ideas on how to speed up this process? To make it at least as fast as Java, problably reaching my HDD bottleneck. Regards  PS - extra details: Compiler options: -O4 -XX -CX -Xs JVM options: -server -XX:CompileThreshold=2 -XX:+AggressiveOpts -XX:+UseFastAccessorMethods ---------------------------------------------------------------------- Phil · 2016-08-22 22:31 Try doing a WriteLn to the file. You'll need AssignFile on a TextFile, then Rewrite, then the WriteLn calls, finally CloseFile. See FPC docs. ---------------------------------------------------------------------- miki · 2016-08-22 22:58 Hi Phil, thanks for your fast answer. Today it's about speed  Quote from: Phil on August 23, 2016, 12:31:46 amYou'll need AssignFile on a TextFile, then Rewrite, then the WriteLn calls, finally CloseFile. See FPC docs. Don't need to see any docs, I learnt to use those functions when I was 14 (maaaany years ago). For some reason, I excepted them to be slower. But I was wrong: 900ms 620ms Anyways, I got the answer for my own question. Just removing the string concatenation, things go much faster. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure arrayToFile(numbers: Array of Integer; fileName: String);var    data: TMemoryStream;    i, len, lendl: Integer;    str, lend: String;begin  data := TMemoryStream.Create;    len := Length(numbers) - 1;    lend := lineEnding;    lendl := length(str);    for i := 0 to len do    begin        System.Str(numbers[i], str);        data.write(Str[1], Length(Str));        data.write(lend[1], lendl);    end;    data.Position := 0;    data.SaveToFile(fileName);    data.Free;end;  Time: 660~680ms. I'm happy for today Of course, the WriteLn approach has a great pro: smaller memory footprint. But 100MB aren't that much in modern computers. Regards. EDIT: running this in a SSD device, the saving time is even a bit faster, about 600ms. So hey, the code is faster than the HDD. ---------------------------------------------------------------------- johnsson · 2016-08-23 00:59 Can I suggest a modification? I' m realy sorry, I don't see the string conversion, so here is the correct version. Code: [Select]procedure arrayToFile(var numbers: Array of Integer; fileName: String); var   Data: TMemoryStream;   I: Integer;   P, S: Pointer;   V: string; begin   Data := TMemoryStream.Create;   Data.SetSize(15 * Length(numbers));   P := Data.Memory;   S := P;   for I := 0 to High(numbers) do   begin     System.Str(numbers[i], V);     Move(V[1],P^,Length(V));     P += Length(V);     Move(LineEnding,P^,2);     P += 2;   end;   Data.SetSize(P - S);   Data.SaveToFile(fileName);   Data.Free; end; Here I got a 1400ms execution time, but I' am running this in a laptop with rly poor CPU. The major cost is the conversion procedure. ---------------------------------------------------------------------- Laksen · 2016-08-23 01:29 Try this. Much simpler and much faster Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i rocedure arrayToFile3(const numbers: Array of longint; const fileName: String);var  i: longint;  buf: array[0..65535] of char;  f: Text;begin  AssignFile(f, fileName);  Rewrite(f);   SetTextBuf(f,buf[0],sizeof(buf));   for i := 0 to high(numbers) do    writeln(f, numbers[i]);   CloseFile(f);end; ---------------------------------------------------------------------- Phil · 2016-08-23 01:36 Quote from: Laksen on August 23, 2016, 03:29:45 amTry this. Much simpler and much faster Yes, that's the way I would write it. Question: Does SetTextBuf make much of a difference in performance? That was a common trick years ago, but I'm wondering if modern OS buffering reduces its effect. ---------------------------------------------------------------------- miki · 2016-08-23 18:42 Ok, people, sumary (all test in SSD drive): My last approach: 580~610ms Phil's approach: 620~630ms johnsson's approach: 460~480ms (the result is wrong) Lanksen's approach: 435ms (Phil's + SetTextBuf) Notes: Yes, it seems SetTextBuf makes a difference. I don't like the remarks in docs, but it's the fastest! Phil's approach: faster now? yes, I made a profiling error yesterday, sorry for that. johnsson's: what is wrong? The line "Move(LineEnding...)" didn't compile for some reason. I guess we are using different OS, and this constant may be a string for you and a char for me. I made a little change in the code to make it work, but introduced some mistake, because the final file is a bit bigger and has broken line breaks [get the joke? broken breaks]. Anyways, very fast, too. Lots of thanks to all of you. Regards. ---------------------------------------------------------------------- Phil · 2016-08-23 18:56 Quote from: miki on August 23, 2016, 08:42:59 pmYes, it seems SetTextBuf makes a difference. I don't like the remarks in docs, but it's the fastest! I would say the remarks are just common sense; Laksen's code is safe. Good to know that SetTextBuf is still useful. ---------------------------------------------------------------------- marcov · 2016-08-23 19:19 Default the buffer is 128 bytes for historical reasons. So yes it saves something. Setting it above 32kb is never noticable, and typically 8kb is already enough. ---------------------------------------------------------------------- User137 · 2016-08-24 02:41 Is there a typo in the docs? http://www.freepascal.org/docs- html/rtl/system/settextbuf.html QuoteThe maximum size of the newly assigned buffer is 65355 bytes.Should be 65535? (2^16 - 1) ---------------------------------------------------------------------- johnsson · 2016-08-24 03:58 Quote from: miki on August 23, 2016, 08:42:59 pmOk, people, sumary (all test in SSD drive): My last approach: 580~610ms Phil's approach: 620~630ms johnsson's approach: 460~480ms (the result is wrong) Lanksen's approach: 435ms (Phil's + SetTextBuf) Notes: Yes, it seems SetTextBuf makes a difference. I don't like the remarks in docs, but it's the fastest! Phil's approach: faster now? yes, I made a profiling error yesterday, sorry for that. johnsson's: what is wrong? The line "Move(LineEnding...)" didn't compile for some reason. I guess we are using different OS, and this constant may be a string for you and a char for me. I made a little change in the code to make it work, but introduced some mistake, because the final file is a bit bigger and has broken line breaks [get the joke? broken breaks]. Anyways, very fast, too. Lots of thanks to all of you. Regards. I' m working with Win 10 64bits and Lazarus 1.6, the LineEnding is just a constant #13#10 in the System unit. Btw here the SetBufText approach take more time to execute than my approach, also, I don't got any compiler error and the final file is correct. Maybe a few diferences between SO and my PC specs cause this. PC Spec Laptop Asus S400CA Core I3 1.5ghz (2 Cores + 2 HT) 6GB DDR3 1600mhz SSD 120gb PNY Anyaway the most important is the best time execution front the java version. Free Pascal Rocks  A last comment, there a few routines optimization to convert Int type to string using assembly and SIMD commands, probably this will result in a better time execution. ---------------------------------------------------------------------- marcov · 2016-08-24 07:52 Quote from: User137 on August 24, 2016, 04:41:33 amIs there a typo in the docs? http://www.freepascal.org/docs- html/rtl/system/settextbuf.html QuoteThe maximum size of the newly assigned buffer is 65355 bytes.Should be 65535? (2^16 - 1) Maybe not. If a 16-bits value is used to hold buffersize. My guess however is that that is old TP leftover, and FPC accepts larger ones. Delphi allows buffers of MBs and larger (not that it matters much) ---------------------------------------------------------------------- miki · 2016-08-24 19:28 Quote from: johnsson on August 24, 2016, 05:58:55 amI' m working with Win 10 64bits and Lazarus 1.6, the LineEnding is just a constant #13#10 in the System unit. Btw here the SetBufText approach take more time to execute than my approach, also, I don't got any compiler error and the final file is correct. Maybe a few diferences between SO and my PC specs cause this. PC Spec Laptop Asus S400CA Core I3 1.5ghz (2 Cores + 2 HT) 6GB DDR3 1600mhz SSD 120gb PNY Similar computer here, but desktop, not laptop. Using Manjaro Linux 64bits. According to documentation, LineEnding is system dependent. In my case (Linux), it's just a #10, so it may be seen as char, while your Windows' #13#10 is a string. Anyways, the wrong file I was getting was my fault, wrong adaption of your code. I realized later. Here the correct one: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure .arrayToFile(numbers: Array of Integer; fileName: String);var    Data: TMemoryStream;    I, Ll: Integer;    P, S: Pointer;    V, L: string;begin    Data := TMemoryStream.Create;    L := LineEnding; // <-- if it's a char, now it's a string    Ll:= Length(LineEnding); // <-- 1 on *NIX, 2 on windows    Data.SetSize((11 + Ll) * Length(numbers)); //<-- as numbers are signed 32bits integers, they will never be longer than 11digits + LineEnding    P := Data.Memory;  S := P;    for I := 0 to High(numbers) do    begin        System.Str(numbers[i], V);        Move(V[1], P^, Length(V));        P += Length(V);        Move(L, P^, Ll);        P += Ll; // <--- my error was here, leaving your "2"; that's the wrong thing with magic numbers :)    end;    Data.SetSize(P - S);    Data.SaveToFile(fileName);    Data.Free;end;  FreePascal faster than Java? Well, I expected so, but it seems I have to do some tricks to achieve that! I did the reverse-way function (fileToArray), and to get a fast result, I had to write my own StrToInt. Special one, because it reads. First attempt, with TStringList + StrToInt loop, 2.8s Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function fileToArray(fileName: String): TIntArray;var    list: TStrings;    i, len: Integer;begin    list := TStringList.Create;    list.LoadFromFile(fileName);    len := list.Count - 1;    setLength(result, len + 1);    for i := 0 to len do    begin        result[i] := StrToInt(list[i]);    end;    list.Free;end;  Second attempt, using AssignFile, ReadLn, ... 1.2~1.6s. The magic number makes a lot!!! big one is faster. SetTextBuf is relevant, but not that much. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function fileToArray(fileName: String): TIntArray;var    F: TextFile;    i, bufs, s: Integer;    buf: array[0..65535] of char;begin    AssignFile(F, fileName);    SetTextBuf(F, buf[0], sizeof(buf));    Reset(F);    i := 0; bufs := 1; s := bufs*10000000;    SetLength(result, s);    while not eof(F) do    begin        ReadLn(F, result[i]);        Inc(i);        if i = s then        begin          Inc(bufs);            s := bufs*10000000;            SetLength(result, s);        end;    end;    CloseFile(F);    setLength(result, i);end;  And the winner, using a stream, fastest but bigger, algo more memory footprint, 350ms!!!!!! Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function fileToArray(fileName: String): TIntArray;var    fs: TFileStream;    i, bufs, s: Integer;    n, num, sign: Integer;    ss: string; c:char;    skip: boolean;begin    fs := TFileStream.Create(fileName, fmOpenRead);    setLength(ss, fs.size);    fs.read(ss[1], fs.size);  fs.free;        i := 0; bufs := 1; s := bufs*10000000;    SetLength(result, s);        num := 0; sign := 1; skip := true;    for n := 1 to high(ss) do    begin        c := ss[n];        if c = '-' then        begin            skip := false;            sign := -1;      end        else if c in ['0'..'9'] then        begin            skip := false;            num := num*10 + (ord(c) - 48);        end      else if not skip then        begin            skip := true;          result[i] := num * sign;            sign := 1; num := 0;          Inc(i);            if i = s then            begin                Inc(bufs);                s := bufs*10000000;                SetLength(result, s);            end;        end;    end;    setLength(result, i);end;  The Java version is not that fast, but still competes and much cleaner. It takes about 900ms. Code: Javascript  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     public int[] fileToArray(String fileName) throws IOException    {        try (BufferedReader br = new BufferedReader(new FileReader(fileName))) {          int[] res = br.lines()                     .mapToInt(Integer::parseInt)                     .toArray();           return res;        }    }  Comparisons are not fair, anyways. Java hasn't the compatibility with old code and Delphi and other stuff, and FreePascal hasn't the Java budget. What do you say? Regards. ---------------------------------------------------------------------- marcov · 2016-08-25 09:18 Quote from: miki on August 24, 2016, 09:28:41 pm Comparisons are not fair, anyways. Java hasn't the compatibility with old code and Delphi and other stuff, and FreePascal hasn't the Java budget. What do you say? And it is just one terribly small piece, aka microbenchmarking. Make an application that actually does something. A task that maps (pun intended) nearly wholly onto a library or language feature will look shorter using that. ---------------------------------------------------------------------- Leledumbo · 2016-08-25 13:25 Quote from: miki on August 24, 2016, 09:28:41 pmThe Java version is not that fast, but still competes and much cleaner. It takes about 900ms. It's not difficult to make Pascal version that looks like Java one. TReadBufStream is practically BufferedReader counterpart, so is TFileStream for FileReader. Their interface is different, though. TReadBufStream has no lines() method, but extending the class to create such method is no difficult (just use array of String or TStrings for return value) with the help of StreamIO unit, just ReadLn until EOF. Again, mapToInt() can be implemented using type helper (for array of String) or extending TString(List) with such a method. No need for toArray() as it's better for the method to directly return an array (of Integer) instead of another stream (of int). Wanna try creating and benchmarking that version?