HTML to UTF8 edcoding
lazarus / 21-General
----------------------------------------------------------------------
dbannon · 2021-01-15 06:21
I am being fed some text that has things like (but not limited to)
Ώ that I want to convert to nice LCL friendly UTF8 and have it
displayed as Omega.
The Ώ is an HTML entity, there are thousands of them !
I have tried calls like HTTPEncode(St), UnescapeHTML(St),
UTF8Decode(St), XMLValueToStr(St) and the best they can handle are the
five 'basic' ones, <, >, ;, & and " all of which are pretty easy
conversions to UTF8 !
Am I building a huge lookup table here or can someone suggest a
library function that that I can use ?
Davo
----------------------------------------------------------------------
circular · 2021-01-15 06:37
Those codes corresponds to Unicode. See: https://en.wikipedia.org/wiki
/Character_encodings_in_HTML#Character_references
So basically, you need to parse the text, look for '', decode the
number and then make the UTF8 string from the code point (function
LazUtf8.UnicodeToUTF8).
Note that the final ';' is optional and that there can be any number
of digits. So '!!' means '!!'
----------------------------------------------------------------------
dbannon · 2021-01-15 06:52
Ahh, thats a much better approach Circular, I will give it a try !
> can be any number of digits.
Oh, that could be fun testing for ....
Anyway, still sounds a lot better than a huge look up table, thank you
!
Davo
----------------------------------------------------------------------
circular · 2021-01-15 06:59
You're welcome
----------------------------------------------------------------------
paweld · 2021-01-15 08:21
function ResolveHTMLEntityReference in HTMLDefs unit
----------------------------------------------------------------------
wp · 2021-01-15 09:06
Quote from: paweld on January 15, 2021, 09:21:25 amfunction
ResolveHTMLEntityReference in HTMLDefs unit
Unfortunately this operates on WideStrings/WideChars, like other
html/xml tools of the FCL. This means unnecessary string conversions
for UTF8 used by Lazarus. I wonder why the FPC teams focusses here so
much on widestrings.
----------------------------------------------------------------------
dbannon · 2021-01-15 10:45
This works for me -
Code: Pascal [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TForm1.RemoveHTMLNumericCode(var St : string) : boolean;var Target
: integer = 1; Buff : string = '';begin repeat Target :=
Pos('', St, Target); // that constant is a & and a #,
the & is being escaped by renderer .... if Target = 0 then
exit(False); // None left, lets get out of here.
if Target + 3 > St.length then exit(false); // No room ....
inc(Target, 2); if (St[Target] in ['0'..'9']) then begin
// Looks like we have one ! while St[Target] in ['0'..'9']
do begin Buff := Buff + St[Target];
delete(St, Target, 1); if Target > St.Length then
break; end; if St[Target] = ';' then delete(St,
Target, 1); Dec(Target, 2); //
Back to start of Entity delete(St, Target, 2);
insert(UnicodeToUTF8(Buff.ToInteger), St, Target);
exit(True); end; //
Oh, well, how sad, try again ? until Target > St.length;end;
St := 'some text & and Ώ and β';
Memo1.append('===== Starting with [' + St + ']'); while
RemoveHTMLNumericCode(St) do; Memo1.Append(St);
Thanks Folks !
Davo
----------------------------------------------------------------------
marcov · 2021-01-15 11:13
Quote from: wp on January 15, 2021, 10:06:45 amQuote from: paweld on
January 15, 2021, 09:21:25 amfunction ResolveHTMLEntityReference in
HTMLDefs unit
Unfortunately this operates on WideStrings/WideChars, like other
html/xml tools of the FCL. This means unnecessary string conversions
for UTF8 used by Lazarus. I wonder why the FPC teams focusses here so
much on widestrings.
I wonder why Lazarus uses utf8 when FPC set up unicode Delphi
compatible using Widestrings from the beginning :-)
----------------------------------------------------------------------
BeniBela · 2021-01-15 11:20
I have one here:
https://github.com/benibela/bbutils/blob/master/bbutils.pas#L5352
Quite faster and handles corner cases
----------------------------------------------------------------------
PascalDragon · 2021-01-15 12:05
Quote from: wp on January 15, 2021, 10:06:45 amQuote from: paweld on
January 15, 2021, 09:21:25 amfunction ResolveHTMLEntityReference in
HTMLDefs unit
Unfortunately this operates on WideStrings/WideChars, like other
html/xml tools of the FCL. This means unnecessary string conversions
for UTF8 used by Lazarus. I wonder why the FPC teams focusses here so
much on widestrings.
Because this code predates both the introduction of codepage aware
AnsiString as well as the reference counted UnicodeString.