HTML to UTF8 edcoding lazarus / 21-General ---------------------------------------------------------------------- dbannon · 2021-01-15 06:21 I am being fed some text that has things like (but not limited to) Ώ that I want to convert to nice LCL friendly UTF8 and have it displayed as Omega. The Ώ is an HTML entity, there are thousands of them !  I have tried calls like  HTTPEncode(St), UnescapeHTML(St), UTF8Decode(St), XMLValueToStr(St) and the best they can handle are the five 'basic' ones, <,  >,  ;,  & and  " all of which are pretty easy conversions to UTF8 ! Am I building a huge lookup table here or can someone suggest a library function that that I can use ? Davo ---------------------------------------------------------------------- circular · 2021-01-15 06:37 Those codes corresponds to Unicode. See: https://en.wikipedia.org/wiki /Character_encodings_in_HTML#Character_references So basically, you need to parse the text, look for '&#', decode the number and then make the UTF8 string from the code point (function LazUtf8.UnicodeToUTF8). Note that the final ';' is optional and that there can be any number of digits. So '!!' means '!!' ---------------------------------------------------------------------- dbannon · 2021-01-15 06:52 Ahh, thats a much better approach Circular, I will give it a try ! > can be any number of digits. Oh, that could be fun testing for .... Anyway, still sounds a lot better than a huge look up table, thank you ! Davo ---------------------------------------------------------------------- circular · 2021-01-15 06:59 You're welcome ---------------------------------------------------------------------- paweld · 2021-01-15 08:21 function ResolveHTMLEntityReference in HTMLDefs unit ---------------------------------------------------------------------- wp · 2021-01-15 09:06 Quote from: paweld on January 15, 2021, 09:21:25 amfunction ResolveHTMLEntityReference in HTMLDefs unit Unfortunately this operates on WideStrings/WideChars, like other html/xml tools of the FCL. This means unnecessary string conversions for UTF8 used by Lazarus. I wonder why the FPC teams focusses here so much on widestrings. ---------------------------------------------------------------------- dbannon · 2021-01-15 10:45 This works for me - Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function TForm1.RemoveHTMLNumericCode(var St : string) : boolean;var    Target : integer = 1;    Buff : string = '';begin    repeat        Target := Pos('&#', St, Target);               // that constant is a & and a #, the & is being escaped by renderer ....        if Target = 0 then exit(False);                 // None left, lets get out of here.      if Target + 3 > St.length then exit(false);     // No room ....      inc(Target, 2);        if (St[Target] in ['0'..'9']) then begin      // Looks like we have one !            while St[Target] in ['0'..'9'] do begin                Buff := Buff + St[Target];                delete(St, Target, 1);                if Target > St.Length then break;            end;            if St[Target] = ';' then delete(St, Target, 1);            Dec(Target, 2);                             // Back to start of Entity            delete(St, Target, 2);            insert(UnicodeToUTF8(Buff.ToInteger), St, Target);            exit(True);        end;                                            // Oh, well, how sad, try again ?   until Target > St.length;end;                            St := 'some text & and Ώ and β';    Memo1.append('===== Starting with [' + St + ']');    while RemoveHTMLNumericCode(St) do;    Memo1.Append(St);       Thanks Folks ! Davo ---------------------------------------------------------------------- marcov · 2021-01-15 11:13 Quote from: wp on January 15, 2021, 10:06:45 amQuote from: paweld on January 15, 2021, 09:21:25 amfunction ResolveHTMLEntityReference in HTMLDefs unit Unfortunately this operates on WideStrings/WideChars, like other html/xml tools of the FCL. This means unnecessary string conversions for UTF8 used by Lazarus. I wonder why the FPC teams focusses here so much on widestrings. I wonder why Lazarus uses utf8 when FPC set up unicode Delphi compatible using Widestrings from the beginning :-) ---------------------------------------------------------------------- BeniBela · 2021-01-15 11:20 I have one here: https://github.com/benibela/bbutils/blob/master/bbutils.pas#L5352 Quite faster and handles corner cases ---------------------------------------------------------------------- PascalDragon · 2021-01-15 12:05 Quote from: wp on January 15, 2021, 10:06:45 amQuote from: paweld on January 15, 2021, 09:21:25 amfunction ResolveHTMLEntityReference in HTMLDefs unit Unfortunately this operates on WideStrings/WideChars, like other html/xml tools of the FCL. This means unnecessary string conversions for UTF8 used by Lazarus. I wonder why the FPC teams focusses here so much on widestrings. Because this code predates both the introduction of codepage aware AnsiString as well as the reference counted UnicodeString.