SynEdit question lazarus / 31-LCL ---------------------------------------------------------------------- typo · 2014-07-13 16:10 Currently SynAnySyn property editor for keywords converts letters to uppercase but not the diacritically signaled letters. They remain lowercase if they are so and the editor does not highlight a lowercase diacritically signaled letter if it is not explicitally set on keywords list. My goal is to make an algorithm editor and so I need to use diacritically signaled letters in keywords. How to make SynAnySyn handle properly these letters? http://bugs.freepascal.org/view.php?id=26477 Thanks. ---------------------------------------------------------------------- Edson · 2014-07-14 17:23 I haven't tested your patches, but I've seen that the SynAnySyn, like all the Highlighters of Lazarus, are basically "Byte oriented". None of them is a UTF-8 highlighter.  They use the first char for filter an identifier, It means that are not designed for filter correctly an UTF-8 keyword. I think for "correctly" working with diacritically signaled letters, they have to be redesigned from the root. Moreover, SynAnySyn is more a a demo than a production HL. I wouldn't do a serious job with it. Maybe Martin have another opinion. ---------------------------------------------------------------------- typo · 2014-07-14 17:37 Quote from: Edson on July 14, 2014, 07:23:58 pmNone of them is a UTF-8 highlighter.  They use the first char for filter an identifier, My solution was to allow MakeIdentTable procedure to recognize the first UTF-8 char, if any, and then run through the UTF-8 word, recognizing also #195 (which precedes any diacritically signaled letter) in MakeMethodTables procedure. In IsKeyWord function a little modification to allow comparison in UTF-8 uppercase which, anyway, is not relevant to the UTF-8 recognition of the keyword (this only allows user to not be forced to set UTF-8 chars twice, one as uppercase and a second time as lowercase). So the comparison remains byte oriented, I don't compare UTF-8 chars. This solution was tested also for chinese characters and works fine. ---------------------------------------------------------------------- marcov · 2014-07-15 08:52 (Afaik the direction most utf8 language parsers follow is to change from defining which characters are allowed as first or follow characters as identifiers to change to which ones AREN'T allowed, since that is the smaller set) ---------------------------------------------------------------------- typo · 2014-07-15 10:45 Yes, I think this is a good way for this highlighter too, since identifiers become to be in an untreatable amount. I will try it. ---------------------------------------------------------------------- typo · 2014-07-15 11:54 I tried this: Code: [Select]procedure MakeIdentTable; var   I: Char;   //idents:string;   NotIdents :string; begin   NotIdents := '"!@#$%&*+=[]{}|\<,>.:;/§¬¢£³²¹°ªº';   //idents:='_0123456789áàãâäéèêëéïóòõôöúüçabcdefghijklmnopqrstuvwxyzÁ ÀÃÂÄÉÈÊËÉÏÓÒÕÔÖÚÜÇABCDEFGHIJKLMNOPQRSTUVWXYZ-?!';   for I := #0 to #255 do   begin     //if pos(i,idents)>0 then identifiers[i]:=true     if Pos(i, NotIdents) < 1 then Identifiers[i] := True     else identifiers[i]:=false; //    case I in ['_', '0'..'9', 'a'..'z', 'A'..'Z','-','?','!'] of true: Identifiers[I] := True; //    else Identifiers[I] := False; //    end;   end; end; but it does not work, maybe I am missing something. ---------------------------------------------------------------------- marcov · 2014-07-15 14:08 Are all values in that string < #128 ? ---------------------------------------------------------------------- typo · 2014-07-15 15:34 No. The signaled lettrers are greather than it. I would like to include chinese and hindi chars if possible. ---------------------------------------------------------------------- typo · 2014-07-15 15:50 I don't know about chinese and hindi chars. For instance, all 'A' are in the range #195+#128 / #195+#133. Portuguese signaled chars are all in the range #128 /  #191 (preceded by #195). ---------------------------------------------------------------------- marcov · 2014-07-15 16:42 Quote from: marcov on July 15, 2014, 04:08:33 pmAre all values in that string < #128 ? Those are probably multibyte, and that's why your test fails. ---------------------------------------------------------------------- typo · 2014-07-15 17:02 Yes. For now it works for portuguese with IDENTS string. But next languages will need to manage it by themselves.