How should I manipulate UTF8 strings lazarus / 62-General ---------------------------------------------------------------------- hakelm · 2017-12-20 15:35 I want to manipulate strings containing two byte UTF8 character symbols and I am at a loss on how to proceed. Some of my problems are illustrated in the little program below. The writeln statement produces no visible output. Running hexdump on the two files produced produces: hexdump utfbug0.txt 0000000 b6c3 b6c3 b6c3                        hexdump utfbug1.txt 0000000 b6b6 b6c3 b6c3  so sr[1]:=s[2]; doesn't work as expected and neither can I do s[1]:='ö'; which produces a compiler error. So it seems that accessing a string symbol with a statement like s[n] only accesses the nth byte. What have I misunderstood and what can I do to access and manipulate individual character symbols? Thanks in advance for any help H uses   {$IFDEF UNIX}{$IFDEF UseCThreads}   cthreads,   {$ENDIF}{$ENDIF}   Classes,sysutils   { you can add units after this }; procedure savestring(fname, s: string); var str:tfilestream; p:pbyte; begin   str:=tfilestream.Create(fname,fmcreate);   p:=@s[1];   while p^<>0 do begin     str.Write(p^,1);     inc(p)   end;   str.Free; end; var s,sn,sr:string; n:integer;  p:pbyte; begin   s:='äöä';   writeln(s[1]);   sr:=s;   sr[1]:=s[2];   savestring('utfbug0.txt',s);   savestring('utfbug1.txt',sr); end. ---------------------------------------------------------------------- howardpc · 2017-12-20 16:19 Your utf8 "characters" need to be strings (they can be up to 4 bytes in length). Something like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; {$ifdef windows}  {$apptype console}{$endif}{$Mode objfpc}{$H+} uses  LazUTF8; var  s: String = 'äöä';  tmp: String;  pc: PChar;  len, start: Integer;  total: Integer = 0; begin  pc:=PChar(s);  start:=1;  repeat    len:=UTF8CharacterLengthFast(pc);    Inc(total, len);     tmp:=Copy(s, start, len);    WriteLn(tmp);     Inc(pc, len);    Inc(start, len);  until total >= Length(s);  WriteLn('program ended, press [Enter] to exit');  ReadLn;end. ---------------------------------------------------------------------- m.abudrais · 2017-12-20 16:36 UTF8 strings and characters may be useful. ---------------------------------------------------------------------- JuhaManninen · 2017-12-20 16:48 Yes, and don't forget unit LazUnicode. Instead of code from howardpc you can do: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1;{$Mode objfpc}{$H+}uses LazUnicode; var  S: String = 'äöå';  tmp: String;begin  for tmp in S do    WriteLn(tmp);end. Remember, the project must have LazUtils as dependency. Otherwise it cannot access neither LazUTF8 nor LazUnicode. BTW, UTF8CharacterLengthFast is renamed in trunk to UTF8CodepointSizeFast. A "character" is a fuzzy term in Unicode and causes confusion. Quoteso sr[1]:=s[2]; doesn't work as expectedActually it works as expected. It copies one byte (CodeUnit in Unicode) which is still often useful. See the link given by m.abudrais for examples. ---------------------------------------------------------------------- hakelm · 2017-12-20 20:42 Thank you both of you. I had hoped that the intricacies of UTF8 had been wrapped up in the traditional string routines. Replacement, insertion, deletion etc. of character symbols now is a bit more complicated and perhaps error prone. An object that realises these tasks could be of benefit to irregular programmers like me. Is there such an animal or is there any advice on best string programming  around? H ---------------------------------------------------------------------- JuhaManninen · 2017-12-21 09:28 Quote from: hakelm on December 20, 2017, 09:42:08 pmI had hoped that the intricacies of UTF8 had been wrapped up in the traditional string routines. I don't know what "traditional" means in this context but all intricacies of UTF-8 have indeed been wrapped into functions in unit LazUTF8. It is not really very intricate. Only CodePoints are encoded. The real complexity of Unicode is elsewhere. QuoteReplacement, insertion, deletion etc. of character symbols now is a bit more complicated and perhaps error prone. What do you mean by "character"? The wiki link given to you shows that CodeUnit resolution is still very useful with Unicode. QuoteAn object that realises these tasks could be of benefit to irregular programmers like me. Is there such an animal or is there any advice on best string programming  around? When working with Unicode you must understand at least its basics. There is no shortcut.