How to determine Unicode character type (letter, punctuation, symbol,
etc.)
lazarus / 21-General

----------------------------------------------------------------------
Manlio · 2022-05-27 08:10

I need to parse Unicode strings with text in different languages, and
I need to know, for every character (code point) that I parse, whether
it is a letter, a digit, a symbol, or punctuation.

I tried to look into FPC unicode-related units but I didn't find
anything.

Can anyone kindly point me into the right direction?

Thank you!

----------------------------------------------------------------------
AlexTP · 2022-05-27 08:49

This gets category of a widechar.
One of UGC_xxx.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses 
Classes, SysUtils,  fpwidestring,  StrUtils,  unicodedata; function
IsUnicodeWordChar(AChar: WideChar): boolean;var  NType: byte;begin  if
AChar='_' then    Exit(true);   if Ord(AChar) >= LOW_SURROGATE_BEGIN
then    Exit(False);   NType := GetProps(Ord(AChar))^.Category; 
Result := (NType <= UGC_OtherNumber);end; function GetCateg(c: word):
byte;begin  Result:= GetProps(c)^.Category;end;

----------------------------------------------------------------------
paweld · 2022-05-27 08:57

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses 
LazUTF8, Character;procedure TForm1.Button1Click(Sender: TObject);var 
s, r: String;  i: Integer;begin  s := 'Test 12,3 gęŚlą jaŹń 〇⌀→Ⓣ■•'; 
for i := 1 to UTF8Length(s) do  begin    r := '';    if IsLetter(s, i)
then    begin      if IsLower(s, i) then        r := ' > Lower
letter'      else        r := ' > Upper letter'    end    else if
IsNumber(s, i) then      r := ' > Number'    else if IsPunctuation(s,
i) then      r := ' > Punctation'    else if IsSeparator(s, i) then  
  r := ' > Separator'    else if IsSymbol(s, i) then     r := ' >
Symbol'    else      r := ' > ???';    Memo1.Lines.Add(UTF8Copy(s, i,
1) + r);  end;end;

----------------------------------------------------------------------
Manlio · 2022-05-27 10:13

Thank you both for the great working code!

For everyone else interested, the "Character" unit is where the magic
happens.