Treating UTF-8 BOM in TCSVDataSet lazarus / 16-Databases ---------------------------------------------------------------------- egsuh · 2025-11-18 07:35 I think this is not applicable only to TCSVDataSet. Anyway TCSVDataSet.LoadFromCSVFile causes problems when the CSV file is encoded as UTF-8 with BOM.  I can change the encoding using TextEditor, etc., but isn't it possible to rewrite the procedure itself as remove BOM characters? ---------------------------------------------------------------------- dbannon · 2025-11-18 07:45 Is the problem the UTF8 characters in the dataset or just the BOM itself ? My app reads and writes some xml files, I had a problem (cannot remember details) but ended up removing the BOM, it is "optional" and my LCL code is quite happy to read UTF8 without it. Davo ---------------------------------------------------------------------- egsuh · 2025-11-18 08:06 QuoteIs the problem the UTF8 characters in the dataset or just the BOM itself ? Just the BOM itself. The first field name is not recognized correctly. When I remove BOM programmatically, then there are no problems. ---------------------------------------------------------------------- Thausand · 2025-11-18 08:28 May be try work round for test: - have TStringList.Create - StringList.LoadFromFile('CSV') - have TMemoryStream.Create - StringList.WriteBOM = false - StringList.SaveToStream(MemoryStream) - StringList.Free - MemoryStream.Position = 0 - TCSVDataset.LoadFromStream(MemoryStream) - MemoryStream.Free ---------------------------------------------------------------------- H₂SO₄ · 2025-11-18 08:31 Where do these CSVs come from? UTF-8 with BOM is Microsoft nonsense and shouldn't even exist in the first place - the root cause are the programs that output such rubbish. Quote from: egsuh on November 18, 2025, 08:35:40 am[...] but isn't it possible to rewrite the procedure itself as remove BOM characters? You could extend  TCustomCSVDataset  and override some of the protected virtual methods... ---------------------------------------------------------------------- egsuh · 2025-11-18 08:39 QuoteWhere do these CSVs come from? Saving Excel file to .csv.  LibreOffice does not add BOMs. ---------------------------------------------------------------------- Thausand · 2025-11-18 08:43 Quote from: egsuh on November 18, 2025, 09:39:11 amSaving Excel file to .csv.  LibreOffice does not add BOMs. I no have windows and no have excel then no can test self: https://usercomp.com/news/1464236/save-csv-utf-8-without-bom ---------------------------------------------------------------------- BrunoK · 2025-11-18 11:14 I dont know the current status of default windows encoding, but the BOM specifies for UTF-8 and UTF-16, at least if it is UTF-8, or UTF-16 little_endian/big_endian. That can be detected via program if there are many #0 in the stream and for endianness wether the #0 are at odd/even position in the stream. Works in general. Apparently, on mac (according to Wiki), a BOM must be written when not saved in UTF-8 or so I understood, correctly or wrongly... ---------------------------------------------------------------------- LeP · 2025-11-18 11:46 Quote from: Khrys on November 18, 2025, 09:31:43 am ............... UTF-8 with BOM is Microsoft nonsense and shouldn't even exist in the first place - the root cause are the programs that output such rubbish. I don't think a BOM is a nonsense ... it's like other kind of files that may have different types of content. Thinks about graphic files (jpeg, png, bmp) or audio/video files that have a header (FOURCC) at the begin of the file. BOM needs to identify correctly the "type of characters" (... ANSI, UTF-8, UTF-16, UTF32, LE/BE) are inside a TEXT files, and I think that should be imperative. A BOM is needed in Windows if you want use a text file coded in UTF-8 with Excel (for example). But, find if a text files has a BOM is simple. May be all component that use TEXT files should check for a BOM. ---------------------------------------------------------------------- egsuh · 2025-11-19 03:40 QuoteMay be all component that use TEXT files should check for a BOM. This is what I'm concerned. Removing BOM characters is simple. ---------------------------------------------------------------------- H₂SO₄ · 2025-11-19 06:03 Quote from: LeP on November 18, 2025, 12:46:59 pmQuote from: Khrys on November 18, 2025, 09:31:43 am ............... UTF-8 with BOM is Microsoft nonsense and shouldn't even exist in the first place - the root cause are the programs that output such rubbish. I don't think a BOM is a nonsense ... it's like other kind of files that may have different types of content. Thinks about graphic files (jpeg, png, bmp) or audio/video files that have a header (FOURCC) at the begin of the file. BOM needs to identify correctly the "type of characters" (... ANSI, UTF-8, UTF-16, UTF32, LE/BE) are inside a TEXT files, and I think that should be imperative. That's what a magic number is for, not a byte order mark (which by itself is nonsense in UTF-8 since there is no endianness to be concerned about). And the BOM fails to do even that (file type identification) - quoting the post I originally linked: Quote from: jpsecher on stackoverflow.comIt is not sufficient because an arbitrary byte sequence can happen to start with the exact sequence that constitutes the BOM.It is not necessary because you can just read the bytes as if they were UTF-8; if that succeeds, it is, by definition, valid UTF-8. While I do think that  TCSVDataSet  should be able to handle this, it's a pity that Microsoft forced yet another plague onto us programmers. ---------------------------------------------------------------------- LeP · 2025-11-19 08:01 Quote from: Khrys on November 19, 2025, 07:03:05 am.... That's what a magic number is for, not a byte order mark (which by itself is nonsense in UTF-8 since there is no endianness to be concerned about). And the BOM fails to do even that (file type identification) - quoting the post I originally linked: No, this is not only for byte order (LE / BE), but for type of (UTF-8, UFT-16. UTF-32) too. That is, UTF-8 is one of the code, NOT THE ONLY CODE. Quote from: Khrys on November 19, 2025, 07:03:05 amQuote from: jpsecher on stackoverflow.comIt is not sufficient because an arbitrary byte sequence can happen to start with the exact sequence that constitutes the BOM.It is not necessary because you can just read the bytes as if they were UTF-8; if that succeeds, it is, by definition, valid UTF-8.While I do think that  TCSVDataSet  should be able to handle this, it's a pity that Microsoft forced yet another plague onto us programmers. Speaking about chars, of course in a text files should be only text chars .....  .... but since Unicode now is not true and in a text file now can be present any "byte". This is a good reasons to ALWAYS use a BOM !!! And don't blame Microsoft, blame who thought about "TCSVDataSet" .... ---------------------------------------------------------------------- dbannon · 2025-11-19 22:35 wikipedia describes a cvs file -  (CSV) is a plain text data format for storing ...simplicity of use and human readability. The BOM is 'invisible', does not belong in a plain text file. Cannot seen with the, e.g. cat or type command, so is hardly human readable. Davo ---------------------------------------------------------------------- cdbc · 2025-11-20 06:18 Hi Back in the day, I used to use the following to load textfiles into stringlists:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  bcUTF8bom: string = #$ef#$bb#$bf;  ... procedure bcLoadFromFileUTF8(aStrings: TStrings; const aFilename: string);var  Fs: TFileStream;  Buf: string;  Cp: word;begin  Fs:= TFileStream.Create(aFilename,fmOpenRead or fmShareDenyNone);  try    SetLength({%H-}Buf,3);    Fs.Read(Buf[1],3);    if Buf = bcUTF8bom then begin      aStrings.LoadFromStream(Fs);    end else begin      Fs.Seek(0,soFromBeginning);      aStrings.LoadFromStream(Fs);      Cp:= bcGuessEncoding(aStrings.Text); // similar to the 1 in LazUTF8.pas      aStrings.Text:= bcEncodeUTF8(aStrings.Text,Cp); // similar to LConvEncoding    end;  finally Fs.Free; end;end; { bcLoadFromFileUTF8 } procedure bcSaveToFileUTF8(aStrings: TStrings; const aFilename: string);var  Fs: TFileStream;begin  Fs:= TFileStream.Create(aFilename,fmCreate or fmOpenWrite);  try    Fs.Write(bcUTF8bom[1],length(bcUTF8bom));    aStrings.SaveToStream(Fs); // only utf8 encoding from app  finally Fs.Free; end;end; { bcSaveToFileUTF8 } Maybe you can tweak it to your use-case... Regards Benny