TMemoryStream file format lazarus / 62-General ---------------------------------------------------------------------- jcmontherock · 2020-06-08 15:18 Hello, I have the following code: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i TParmData = record    FormRect: TRect;    sName :   AnsiString;    age  :    Integer;    male :    Boolean;  end;  ....  ClearStream:  TMemoryStream;  ParmData:     TParmData;  slServers:    TStringList;  ....  ParmData.FormRect := Bounds(Form1.Left, Form1.Top, Form1.Width, Form1.Height);  ParmData.sName := 'ééààèèüüüüü abcdefghijklmnopqrstuvwxy 40........50' +                    'Turner ==^<=abcdefghijklmnopqrstuvwxy 40........50' +                    'Turner ==^<=abcdefghijklmnopqrstuvwxy 40........50';  ParmData.age  := 45;  ParmData.male := False;    Writing file: ..............  ClearStream.Write(ParmData, SizeOf(ParmData));  ClearStream.WriteAnsiString(slServers.DelimitedText);  ClearStream.Position := 0;  ClearStream.SaveToFile(sFilename);   Reading file: ............  if FileExists(sFilename) then begin    ClearStream.Clear;    ClearStream.LoadFromFile(sFilename);    ClearStream.Position := 0;    ClearStream.Read(ParmData, SizeOf(ParmData));    slServers.DelimitedText := ClearStream.ReadAnsiString;  ...   Aftert reading disk file, every data are OK, and I get:   ------------------------------------------------------   sName length = 161                      OK. Increase of size was done by Utf-8 encoding.   SizeOf(ParmData) = 32                   ???????   Length(slServers.DelimitedText) = 147   OK. Same remark.   Disk Record = 179 (0xB3) without BOM    ???????   Encoding record: utf-8, W10. Disk record contains the name (150 char or 161 bytes), a text (143 char or 147 bytes) from TStrings, a TRect (16), an Integer (4) and a boolean (1). Total = 161 + 147 + 16 + 4 + 1 = 329 bytes ?????? Record size:                     179 bytes I don't understand these numbers. I search to find in which format, disk record was written. I did'nt find it. Has somebody an idea ? ---------------------------------------------------------------------- marcov · 2020-06-08 15:26 ansistring is a reference type. IOW the record only stores a pointer. ---------------------------------------------------------------------- jcmontherock · 2020-06-08 16:14 Ok, I read the file in a second run... It's not a pointer in the file. ---------------------------------------------------------------------- wp · 2020-06-08 16:49 Quote from: jcmontherock on June 08, 2020, 06:14:33 pmOk, I read the file in a second run... It's not a pointer in the file. No, it is. A string IS a pointer. Your record contains only the pointer to the string characters - the character array is stored somewhere else. Either you use string[255] (or similar) which holds the characters within the record or you read/write the string characters separately from the record: Not tested: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i TParmData = record    FormRect: TRect;    age  :    Integer;    male :    Boolean;    sName :   AnsiString;      // <---- Move the string to the end of the record  end;var  ParmData: TParmData;  sNameLen: Integer;...  // Writing  ClearStream.Write(ParmData, SizeOf(TParmData) - SizeOf(AnsiString));  // Write the record, but without the string (which is a pointer!), ...  sNameLen := Length(ParmData.sName);                                  ClearStream.Write(sNameLen, SizeOf(sNameLen));                        // ... write the string length ...  if sNameLen > 0 then    ClearStream.Write(ParmData.sName[1], sNameLen);                     // ... and the string data separately...  // Reading  ClearStream.Read(ParmData, SizeOf(TParmData) - SizeOf(AnsiString));   // do not read the string here, but...  ClearStream.Read(sNameLen, SizeOf(sNameLen));                         // ... read the string length  SetLength(ParmData.sName, sNameLen);                                // ... allocate memory for the string  if sNameLen > 0 then    ClearStream.Read(ParamData.sName[1], sNameLen);                     //... and read the string characters separately  Alternatively to explicitly writing/reading the stringlength and string characters you could also use the Write/ReadAnsiString method of the stream, as you already do with the slServers. ---------------------------------------------------------------------- jamie · 2020-06-08 23:20 Looking at the original code he posted it does look like the poster is well aware of the pointer workings of the string.  In short he is making a variable length record but this is going to be hard to index of course..  He should decide on the maximum size the string will ever be and replace it with a ShortString with the max size indicated..  sName:ShortString[20];  That will ensure there is at least 20 bytes with a length marker at the start..  with that, simply write the complete record out and don't worry about the string..  when reading the record just read the complete record in.  the correction of putting that AsnsiString at the end helps but it really does not solve the issue at hand overall.. ---------------------------------------------------------------------- jcmontherock · 2020-06-09 08:25 As I said, it works perfectly. It could be used for saving parms of an application. My question was about the file format. It seems to be compressed because total length is less than the sum of each element. Except for the AnsiSAtring text and the BOM, all the file record text is unreadable. ---------------------------------------------------------------------- wp · 2020-06-09 08:40 Quote from: jcmontherock on June 09, 2020, 10:25:06 amAs I said, it works perfectly. It could be used for saving parms of an application. My question was about the file format. It seems to be compressed because total length is less than the sum of each element. Except for the AnsiSAtring text and the BOM, all the file record text is unreadable. "It works perfectly" and "the file record text is unreadable" are contradicting statements. I do not understand what you are saying. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i type  TParmData = record    FormRect: TRect;    sName :   AnsiString;    age  :    Integer;    male :    Boolean;  end; Calculation of the SizeOf(ParmData): - FormRect TRect is 4 integers --> 16 bytes - sName: is a pointer --> 4 bytes (or 8 of you have a 64-bit binary) - no matter how long the string is because the characters are not stored within the record! - age: integer --> 4 bytes - male: boolean: 1 byte. Since the record is not packed it will be expanded to 4 bytes --> total: 16 + 4 + 4 + 4 = 28 for 32 bit, or 16 + 8 + 4 + 4 = 32 for 64 bit. As already mentioned the record saved by your code does NOT contain the string characters but the value of the pointer to the characters and thus is useless. ---------------------------------------------------------------------- rvk · 2020-06-09 08:54 Quote from: jcmontherock on June 09, 2020, 10:25:06 amAs I said, it works perfectly. It could be used for saving parms of an application. It might work perfectly if you read the record back in the same session as you wrote the file. Because the string-pointer might still be valid. But if you end your program and read it back after restart of your program, you'll see that your string isn't there (what the others tried to explain). ---------------------------------------------------------------------- jcmontherock · 2020-06-09 09:37 It works. file is written at form close and read at show form event (2 runs). You are right: I am surprise that it works fine. ---------------------------------------------------------------------- rvk · 2020-06-09 09:42 Quote from: jcmontherock on June 09, 2020, 11:37:54 amIt works. file is written at form close and read at show form event. Yes, in that case it might work. But try to close your program, start it again and read the file. You'll see the string is garbage. When you don't close your program, you also won't need to write the config to disk. It is still in memory. So why are you using a file??? If you need the config after you restart the program, you'll need to listen what the others are telling you. You only save the pointer to the string. After restart, that pointer isn't valid anymore and has no information. ---------------------------------------------------------------------- wp · 2020-06-09 09:44 Quote from: rvk on June 09, 2020, 11:42:01 amQuote from: jcmontherock on June 09, 2020, 11:37:54 amIt works. file is written at form close and read at show form event. Yes, in that case it might work. But try to close your program, start it again and read the file. You'll see the string is garbage. Maybe one step further: close the IDE and run the program directly from the OS. ---------------------------------------------------------------------- jamie · 2020-06-09 11:08 Guys look closer. What he is doing is writing a variable sized data field. It starts with fixed known types and is trailed with a string that ends with a bull char. If he was to write a series of them to file you still can read them back following the bull market ect. This has been done many times although u really need to know your code to understand it. That is the author should fully understand it. Ps. Bull = null . Stupid auto correct. ---------------------------------------------------------------------- rvk · 2020-06-09 11:21 Quote from: jamie on June 09, 2020, 01:08:34 pmIt starts with fixed known types and is trailed with a string that ends with a null char. I don't see a null character written anywhere in that code. WriteAnsiString for the TStringList.Text has a prefix of the length. So, that way, there could be multiple records written without any problem, Anyways... I think you are missing the point about the record-type which is saved. It has an AnsiString (which is normally a pointer to a string). You can't save the record like that because only the pointer gets saved. Not the actual characters. Doing the calculations... QuoteDisk record contains the name (150 char or 161 bytes), a text (143 char or 147 bytes) from TStrings, a TRect (16), an Integer (4) and a boolean (1). Total = 161 + 147 + 16 + 4 + 1 = 329 bytes ?????? Record size:                     179 bytes I don't understand these numbers. I search to find in which format, disk record was written.Actually. a TRect (16), a sName (pointer = 4), age (4) and male (4 because of non-packed) = 28 , on a 32 byte boundary = 32 Adding the TStringList.Text (147, which is done correctly) is a total of 179 bytes. That's what written to disk. But the actual characters of sName are NOT written. But the sName problem needs to be fixed (which is already discussed). ---------------------------------------------------------------------- jamie · 2020-06-09 11:25 The null is already at the end of string. Any ways this looks like a school project and should be done via text file instead. ---------------------------------------------------------------------- rvk · 2020-06-09 11:29 Quote from: jamie on June 09, 2020, 01:25:48 pmThe null is already at the end of string. Yes, the null is at the end of the string but is not written to disk. Look at the implementation of Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Procedure TStream.WriteAnsiString (const S : String);Var L : Longint;begin  L:=Length(S);  WriteBuffer (L,SizeOf(L));  WriteBuffer (Pointer(S)^,L);end; So if the string is 100 characters... (without the #0) exactly 100 characters are written. But in front of those 100 characters the length is written so TStream.ReadAnsiString can read them back. So, although there is no null written behind the string, the string can still be read because it is prefixed by the length !!! (But that's not the original problem of the poster.) ---------------------------------------------------------------------- jamie · 2020-06-09 11:29 If this is not school project then a short string should used write the record once as a whole. ---------------------------------------------------------------------- wp · 2020-06-09 11:30 Again (the last time): the string characters are NOT embedded in the record, the record contains only a pointer to the characters somewhere in memory. What you describe is an array of chars with terminating null. This is not a dynamic Pascal string as used here. ---------------------------------------------------------------------- jcmontherock · 2020-06-09 14:01 I give you the project. ---------------------------------------------------------------------- rvk · 2020-06-09 14:06 Quote from: jcmontherock on June 09, 2020, 04:01:10 pmI give you the project. And what is the problem? Did you read the other posts? If you use AnsiString then your sName is NOT saved to the file. We even showed you the calculation for it. The solution was also given (use string[255] or ShortString, which isn't a pointer but a stringarray which will be saved). ---------------------------------------------------------------------- wp · 2020-06-09 15:28 OK, maybe you do not understand what rvk and I were trying to explain. Therefore I fixed your demo to work. I marked all lines changed by my initials. The code stores only the non-string elements of the record - this is easy when the string is moved to the end of the record definition, and the size of the string (the size of a pointer) is subtracted from the size of the record. Then the string is saved separately by calling ClearString.WriteAnsiString. Reading is similar. Alternatively you could declare the string sName as String[255] because this kind of string is NO pointer and has its characters inside the record. But because now even for an empty string 255 bytes are occupied in the file I prefer the other solution. ---------------------------------------------------------------------- jcmontherock · 2020-06-09 15:42 OK, thenks a lot to everybody...