Pascal Security lazarus / 62-General ---------------------------------------------------------------------- Grahame Grieve · 2021-11-01 19:48 https://www.schneier.com/blog/archives/2021/11/hiding-vulnerabilities- in-source-code.html - is FPC across these issues? ---------------------------------------------------------------------- AlexTP · 2021-11-01 19:57 > uses Unicode directionality override characters I have read about this 5 years ago on russian Habr.com. And made a special handling of these chars in CudaText - it shows these chars specially. ---------------------------------------------------------------------- MarkMLl · 2021-11-01 20:39 Quote from: Grahame Grieve on November 01, 2021, 08:48:11 pmhttps://www.schneier.com/blog/archives/2021/11/hiding- vulnerabilities-in-source-code.html - is FPC across these issues? Since you've not summarised that: some languages accept Unicode for function names and for things that might be used in the context of a reserved word. Does FPC? https://www.freepascal.org/docs- html/current/ref/refse4.html#x15-140001.4 provides no useful information other than that the first character of a function (etc.) name must be an underscore or a letter. MarkMLl ---------------------------------------------------------------------- AlexTP · 2021-11-01 21:00 But the wiki tells is https://wiki.freepascal.org/Identifiers ---------------------------------------------------------------------- howardpc · 2021-11-01 21:04 Quote from: MarkMLl on November 01, 2021, 09:39:56 pmhttps://www.freepascal.org/docs- html/current/ref/refse4.html#x15-140001.4 provides no useful information other than that the first character of a function (etc.) name must be an underscore or a letter. That page makes clear that only letters, digits and _ are valid characters in identifiers and reserved words; digits may not be the first character and & is accepted only as an initial identifier character. In other words FPC accepts only a fairly restricted ANSI 7-bit alphabet for identifiers (and likewise for operators and separators). Recent Delphis, on the other hand, accept other characters in identifiers, I believe. ---------------------------------------------------------------------- dbannon · 2021-11-01 23:18 Easily tested. On the other hand, testing against ALL utf8 characters might be a bit harder .... EDIT: No, its not about 'funny' characters in identifier names : QuoteBy injecting Unicode Bidi override characters into comments and strings, an adversary can pro- duce syntactically-valid source code in most modern languages for which the display order of characters presents logic that diverges from the real logic. That could be a bit more interesting ..... ---------------------------------------------------------------------- Grahame Grieve · 2021-11-02 00:27 Since this affects any parsed text at all, it's not just compilers. Does anyone have a routine at hand to say whether a unicode character point is one of the offending characters? ---------------------------------------------------------------------- dbannon · 2021-11-02 04:14 Quote from: Grahame Grieve on November 02, 2021, 01:27:39 amSince this affects any parsed text at all, it's not just compilers. Does anyone have a routine at hand to say whether a unicode character point is one of the offending characters? Or, perhaps can tell the differences between the "offending characters" being used legitimately or not ? Davo ---------------------------------------------------------------------- AlexTP · 2021-11-02 05:55 >Does anyone have a routine at hand to say whether a unicode character point is one of the offending characters? Simple case-block with the codes: (U+202A, U+202B, U+202C, U+202D, U+202E, U+2066, U+2067, U+2068, U+2069, U+061C, U+200E и U+200F) ---------------------------------------------------------------------- dbannon · 2021-11-02 06:19 The paper highlights two (main) exploits - Invisible characters in Identifiers - fpc spits them out (I think). Bidi override characters used in comments (or string literals?) to terminate the comment early, leaving content that appears as a comment (or a string literal) as compile-able code.  Unless we make a decision to never allow such (unicode) characters anywhere in a source file, not much can be done. But the paper does mention programmer's editors showing up characters, maybe (as Alextp is clearly thinking) just giving a reviewer a 'heads up' every time a file is loaded into an editor that it does contain one of the risky characters ?  This does not stop the problem but provides an easy audit tool. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i showmessage('warning line 34, pos 52 contains U+202B Right-to-Left Embedding character'); I guess it will annoy hell out of a programmer who does have a legitimate reason for such content, even more than the "inline is not inlined" thingo but it could be turned of if developing your own code, only turned on when reviewing someone else's ?  An editor option, not a project one. I think it would be a nice feather in Lazarus's hat if it included such an audit tool. And maybe fpc could issue a warning if it sees such a character too ? As alextp noted, the characters them selves would be easy enough to search for, each individually. Use a bit of time but that would be a OK at load time only I guess. Davo ---------------------------------------------------------------------- AlexTP · 2021-11-02 06:53 Audit tool, another dialog in the IDE? Better not, just add to SynEdit the special highlight of those chars, that's it. (Optional, or not, I don't care) ---------------------------------------------------------------------- Grahame Grieve · 2021-11-02 07:29 Well, here's some source material to help things along. The idea is that you either ban bidi characters altogether (using hasUnicodeBiDiChars) or you at least ensure that whitespace and string and comments are well formed unicode (using checkUnicodeWellFormed)  I think it's solid, though I'll post updates as I hear about issues (I have a java port of the same code): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit UnicodeUtilities; interface uses  Classes, SysUtils, Contnrs; const  LRE = #$202a;  RLE = #$202b;  PDF = #$202c;  LRO = #$202d;  RLO = #$202e;  LRI = #$2066;  RLI = #$2067;  FSI = #$2068;  PDI = #$2069;  LRM = #$200E;  RLM = #$200F;  ALM = #$061C;  PARA = #10;   ALL_BIDI_CHARS : TUCharArray = (LRE, RLE, PDF, LRO, RLO, LRI, RLI, FSI, PDI, LRM, RLM, ALM, PARA);  CONTROL_CHARS_1 : TUCharArray = (LRE, RLE, LRO, RLO, LRM, RLM, ALM);  CONTROL_CHARS_2 : TUCharArray = (LRI, RLI, FSI); type   { TUnicodeUtilities }   TUnicodeUtilities = class (TObject)  public    class function hasUnicodeBiDiChars(src : String) : boolean;    class function describe(c : UnicodeChar) : String;    class function replaceBiDiChars(src : String) : String;     // returns '' if src is well formed, or a description of a structure problem with bi-directional characters    class function checkUnicodeWellFormed(src : String) : String;  private    FList : TObjectList;    constructor Create;    function checkWellFormed(src : String) : String;    procedure popJustOne(chars : TUCharArray);    procedure popOneAndOthers(chars, others : TUCharArray);    function summary : String;  public    destructor Destroy; override;  end;  implementation type function unicodeChars(s : String) : TUCharArray;var  i, c, l, cl : integer;  ch : UnicodeChar;  p: PChar;begin  l := length(s);  SetLength(result, l); // maximum possible length  i := 0;  c := 1;  p := @s[1];  while l > 0 do  begin    ch := UnicodeChar(UTF8CodepointToUnicode(p, cl));    result[i] := ch;    inc(i);    dec(l, cl);    inc(p, cl);  end;  SetLength(result, i);end;  function InSet(c : UnicodeChar; arr : TUCharArray) : boolean;var  t : UnicodeChar;begin  result := false;  for t in arr do    if t = c then      exit(true);end;    { TStateStack }   TStateStack = class  private    c : UnicodeChar;    i : integer;    constructor create(aC: UnicodeChar; aI : integer);  end; { TStateStack } constructor TStateStack.create(aC: UnicodeChar; aI: integer);begin  inherited Create;  c := aC;  i := aI;end;  { TUnicodeUtilities } class function TUnicodeUtilities.hasUnicodeBiDiChars(src: String): boolean;var  c : UnicodeChar;begin  result := false;  for c in unicodeChars(src) do    if inSet(c, ALL_BIDI_CHARS) then      exit(true);end; class function TUnicodeUtilities.describe(c: UnicodeChar): String;begin  case c of    LRE : result := 'LRE';    RLE : result := 'RLE';    PDF : result := 'PDF';    LRO : result := 'LRO';    RLO : result := 'RLO';    LRI : result := 'LRI';    RLI : result := 'RLI';    FSI : result := 'FSI';    PDI : result := 'PDI';  LRM : result := 'LRM';    RLM : result := 'RLM';    ALM : result := 'ALM';    PARA : result := 'PARA';  else    result := c;  end;end; class function TUnicodeUtilities.replaceBiDiChars(src: String): String;var  b : TStringBuilder;  c : UnicodeChar;begin  b := TStringBuilder.create;  try    for c in unicodeChars(src) do      if inSet(c, ALL_BIDI_CHARS) then        b.append('|'+describe(c)+'|')    else        b.append(TEncoding.UTF8.getString(TEncoding.UTF8.GetBytes(c)));    result := b.toString();  finally    b.free;  end;end; class function TUnicodeUtilities.checkUnicodeWellFormed(src: String): String;var  this : TUnicodeUtilities;begin  this := TUnicodeUtilities.create;  try    result := this.checkWellFormed(src);  finally    this.free;  end;end; constructor TUnicodeUtilities.Create;begin  inherited Create;  FList := TObjectList.create;  FList.OwnsObjects := true;end; function TUnicodeUtilities.checkWellFormed(src: String): String;var  i : integer;  c : UnicodeChar;begin  i := 0;  for c in unicodeChars(src) do  begin    inc(i);    if inSet(c, ALL_BIDI_CHARS) then    begin      case c of        PARA:          FList.clear();      LRO, RLO:          FList.add(TStateStack.create(c, i));        PDF:          popJustOne(CONTROL_CHARS_1);        LRI, RLI, FSI:          FList.add(TStateStack.create(c, i));        PDI:          popOneAndOthers(CONTROL_CHARS_2, CONTROL_CHARS_1);        LRM, RLM, ALM:          FList.add(TStateStack.create(c, i));      end;    end;  end;  if (FList.Count = 0) then    result := ''  else    result := summary;end; procedure TUnicodeUtilities.popJustOne(chars: TUCharArray);begin  if (FList.count > 0) and InSet(TStateStack(Flist.Last).c, chars) then    FList.Delete(FList.count-1);end; procedure TUnicodeUtilities.popOneAndOthers(chars, others: TUCharArray);var  found, done : boolean;  i : integer;begin  found := false;  for i := 0 to FList.count - 1 do  begin    if InSet(TStateStack(Flist[i]).c, chars) then    begin      found := true;      break;    end;  end;   if (found) then  begin    while (FList.count > 0) and (InSet(TStateStack(Flist.Last).c, chars) or InSet(TStateStack(Flist.Last).c, others)) do    begin      done := InSet(TStateStack(Flist.Last).c, chars);      FList.Delete(FList.count-1);      if (done) then        break;    end;  end;end; function TUnicodeUtilities.summary: String;begin  result := 'Unicode Character '+describe(TStateStack(Flist.Last).c)+' at index '+inttostr(TStateStack(Flist.Last).i)+' has no terminating match';end; destructor TUnicodeUtilities.Destroy;begin  FList.Free;  inherited Destroy;end; end. ---------------------------------------------------------------------- MarkMLl · 2021-11-02 08:25 Quote from: howardpc on November 01, 2021, 10:04:24 pmIn other words FPC accepts only a fairly restricted ANSI 7-bit alphabet for identifiers (and likewise for operators and separators). Where is that documented?And to what extent is it enforced? I don't see anything like that in either the formal documentation or in the wiki page (pointing out firmly that the wiki is not formal documentation). MarkMLl ---------------------------------------------------------------------- AlexTP · 2021-11-02 08:32 >Where is that documented?And to what extent is it enforced? https://wiki.freepascal.org/Identifiers It tells exactly that, ie what chars are allowed in Ids. ---------------------------------------------------------------------- Bart · 2021-11-02 08:42 Quote from: MarkMLl on November 02, 2021, 09:25:20 amI don't see anything like that in either the formal documentation or in the wiki page (pointing out firmly that the wiki is not formal documentation). Official documentation: https://www.freepascal.org/docs- html/current/ref/refse4.html#x15-140001.4. Bart ---------------------------------------------------------------------- MarkMLl · 2021-11-02 08:46 Quote from: Bart on November 02, 2021, 09:42:47 amOfficial documentation: https://www.freepascal.org/docs- html/current/ref/refse4.html#x15-140001.4. Which is the page I cited near the start of the thread, and says nothing about what characters are acceptable in positions after the first. MarkMLl ---------------------------------------------------------------------- PascalDragon · 2021-11-02 13:40 Quote from: MarkMLl on November 02, 2021, 09:46:33 amQuote from: Bart on November 02, 2021, 09:42:47 amOfficial documentation: https://www.freepascal.org/docs- html/current/ref/refse4.html#x15-140001.4. Which is the page I cited near the start of the thread, and says nothing about what characters are acceptable in positions after the first. FPC does not yet support characters that are neither digits (0-9) nor characters (a-z and A-Z) nor an underscore in identifiers. The wording of the documentation could be cleared up however. ---------------------------------------------------------------------- MarkMLl · 2021-11-02 13:47 Quote from: PascalDragon on November 02, 2021, 02:40:39 pmFPC does not yet support characters that are neither digits (0-9) nor characters (a-z and A-Z) nor an underscore in identifiers. The wording of the documentation could be cleared up however. Thanks for that. There's one place on the wiki page that refers to "English alphabet", but otherwise I think that people have seen "limits identifiers to 127 characters" in the context of their length and misinterpreted it to mean ANSI (or "strict ASCII", or whatever ones chosen term) i.e. below 0x80. MarkMLl ---------------------------------------------------------------------- Warfley · 2021-11-02 22:13 It's funny but there are a lot of situations in which having bad accessibility can increase the security, not only w.r.t. programming language only allowing english characters. For example on android, if you disallow screen and touch recordings in your app, disability tools don't work anymore, but this can substantially increase the security of your app, e.g. if it requires you entering a ping in your banking app So at last we can claim that the bad accessibility is a security feature of the language ---------------------------------------------------------------------- Grahame Grieve · 2021-11-03 08:43 Well, I wrote that code above into my ci-build script so that it checks for these unicode characters. But it would be better for FPC to check constants and strings as well. I'll live in hope ;-) ---------------------------------------------------------------------- MarkMLl · 2021-11-03 08:54 Quote from: Grahame Grieve on November 03, 2021, 09:43:23 amWell, I wrote that code above into my ci-build script so that it checks for these unicode characters. But it would be better for FPC to check constants and strings as well. I'll live in hope ;-) Note PascalDragon's comment: QuoteFPC does not yet support characters that are neither digits (0-9) nor characters (a-z and A-Z) nor an underscore in identifiers. The wording of the documentation could be cleared up however. I think it's reasonable to assume that the compiler will explicitly reject things like the bidi overrides now that their abuse is a known issue. My own personal feeling is that sticking to the basic English character set is entirely reasonable, but that needs to be explicitly documented. MarkMLl ---------------------------------------------------------------------- Grahame Grieve · 2021-11-04 09:19 I certainly hope that the FPC compiler will be updated. In the meantime, I won't get any new compiler features for a while unless patches are issued on old versions/branches. So hence I put it in my ci-build script ---------------------------------------------------------------------- dbannon · 2021-11-10 07:44 Note that Debian's lintian scans for these scary characters.  And find lots off them, mostly in .po files and .xml files https://lintian.debian.org/tags/unicode-trojan davo ---------------------------------------------------------------------- Warfley · 2021-11-10 18:46 The thing about these characters is, it's not that they are bad or malicious characters, but are normal characters that are *required* for some languages. So if an arabic programmer wants to write programs for an arabic audience, they will use arabic text in strings, and possibly also comments. And arabic text simply requires these characters. Filtering these characters would result in all programs containing some amount of arabic text or comments triggering these filters, which could pose some usability issues when people get warnings that their pretty normal programs all look suspect to the compiler. I think that just the fact that pascal does not allow non english characters in identifiers is already an accessibility issue on it's own, as especially programming beginners will usually use their own language for identifiers, which they simply can't with pascal, sealing the language off for like half the planets population. But then also having warnings (or even worse errors) thrown every time you write a program in your language will just shy away potential pascal developers. It's easy to say that we should just use the latin alphabet, I mean for me as a german there are literally only 4 characters ä, ö, ü and ß that can easiely be replaced by ae oe ue and ss, but for some other people learning a programming language should not require learning a new natural language ---------------------------------------------------------------------- MarkMLl · 2021-11-10 19:11 Quote from: Warfley on November 10, 2021, 07:46:29 pmThe thing about these characters is, it's not that they are bad or malicious characters, but are normal characters that are *required* for some languages. And- if I'm correct- the .po files contain language-specific ("internationalised") strings etc. QuoteI think that just the fact that pascal does not allow non english characters in identifiers is already an accessibility issue on it's own, as especially programming beginners will usually use their own language for identifiers, which they simply can't with pascal, sealing the language off for like half the planets population. I've given this some thought in the past in the context of some of my own R&D activities, and broadly agree with you. /However/, if Modern Pascal and Lazarus/FPC are to continue to hew to the principles established by Wirth, I think it's fair to point out that despite being Swiss he was entirely happy to design and document the language assuming the English alphabet and made no provision for either non- English identifiers or localised reserved words. QuoteIt's easy to say that we should just use the latin alphabet, I mean for me as a german there are literally only 4 characters ä, ö, ü and ß that can easiely be replaced by ae oe ue and ss, but for some other people learning a programming language should not require learning a new natural language Quite frankly I find ß in particular pleasantly quaint, even if it tends to look somewhat jarring in most typefaces. MarkMLl ---------------------------------------------------------------------- Warfley · 2021-11-10 20:28 Quote from: MarkMLl on November 10, 2021, 08:11:46 pmAnd- if I'm correct- the .po files contain language-specific ("internationalised") strings etc. Sure, if you enable internationalization/i18n, but if you develop an application as a local only for other locals, using i18n just adds additional work and debugging overhead for you, personally I write most of my applications in english and don't care about i18n because it is simply a lot of extra work which I don't think is justified for my 1 man projects. And even without strings, there are still comments, I regularly see non english comments, in a korean project I've seen all the comments where in korean. Generally speaking people just like to use their native language. And the question isn't even if thats a good or a bad idea, but if people like using their native language and pascal makes it harder or doesn't allow them to, well then the result will be that simply in those countries, people will not use pascal but rather languages like C++ or C# that are fully unicode. And personally I like seeing when people start using pascal, I'd love to see it being used around the world. Especially here in europe it is generally considered a beginners language often taught at schools. This is of course completely off the table in a lot of other countries, because when you want to teach it to children that possible aren't that fluent in english, they will probably choose a programming language where they can name their identifiers in their own language. QuoteI've given this some thought in the past in the context of some of my own R&D activities, and broadly agree with you. /However/, if Modern Pascal and Lazarus/FPC are to continue to hew to the principles established by Wirth, I think it's fair to point out that despite being Swiss he was entirely happy to design and document the language assuming the English alphabet and made no provision for either non- English identifiers or localised reserved words.I think it's always easy to say as someone who is comes from a general european background. Sure  Wirth was swiss and depending on where he was from he either spoke italian, german or french (I think it was german swiss but I am not sure right now). All of these languages just have a few chars differently, not a whole different alphabeth, with even different syntactic rules (as arabic for example does not have vocals but vocals come from the context of which the consonants are written). Also it must be noted that for someone who is native in one european language (maybe except hungarian) has a much easier time learning another like english than someone who comes from a completely different linguistic background ---------------------------------------------------------------------- MarkMLl · 2021-11-10 20:55 Quote from: Warfley on November 10, 2021, 09:28:45 pmQuote from: MarkMLl on November 10, 2021, 08:11:46 pmAnd- if I'm correct- the .po files contain language-specific ("internationalised") strings etc. Sure, if you enable internationalization/i18n, The reason I was pointing that out is that it very likely explains why checking software was finding "exotic" Unicode in certain types of file: basically, there was no nefarious intent and the results were false positives. QuoteI think it's always easy to say as someone who is comes from a general european background. Sure  Wirth was swiss and depending on where he was from he either spoke italian, german or french (I think it was german swiss but I am not sure right now). Zurich born and bred. I checked, because I was prepared to make disparaging comments about the chauvinism of certain ethnicities who /demand/ that their orthographic foibles be accommodated whatever the cost. MarkMLl ---------------------------------------------------------------------- Warfley · 2021-11-10 21:07 Quote from: MarkMLl on November 10, 2021, 09:55:01 pmThe reason I was pointing that out is that it very likely explains why checking software was finding "exotic" Unicode in certain types of file: basically, there was no nefarious intent and the results were false positives. Ah ok you where referencing the post earlier. QuoteZurich born and bred. I checked, because I was prepared to make disparaging comments about the chauvinism of certain ethnicities who /demand/ that their orthographic foibles be accommodated whatever the cost. MarkMLl Well I don't think you will necessarily get that here, because the people here already use pascal and are therefore proabably fine with the decision. But there are a lot of people who will never learn pascal simply because it's for them much more inconvinient. As I stated in my earlier post, here in europe Pascal is often used in education to teach programming, and personally I think if there is one thing where Pascal really shines it is teaching (because it covers pretty much all areas of basic programming, from fully procedural console applications to OOP gui applications, with high level classes but also low level pointer logic etc.) but as programming is tought in school at ages where the children will probably not be so comfortable with going fully english yet, it will probably not be an option. I remember back at my school when we learned Pascal, we were also using german words for the identifiers. This works because german is very closely related to english, and also, germany is on good terms with the anglo saxon countries. But in other countries with other languages, and maybe even political tensions such that using english would be considered unpatriotic or something similar, this rules out pascal as a learning language. ---------------------------------------------------------------------- MarkMLl · 2021-11-11 08:18 Quote from: Warfley on November 10, 2021, 10:07:15 pmmaybe even political tensions such that using english would be considered unpatriotic or something similar, this rules out pascal as a learning language. Yes, there was a BASIC interpreter at one point with all the keywords translated to Welsh. But I pity anybody condemned to writing a lexer (or for that matter a filesystem) using variable-length UTF8 characters or an orthography where the direction can (and does) change... which gets us back on topic :-) MarkMLl ---------------------------------------------------------------------- PascalDragon · 2021-11-11 08:26 Quote from: Warfley on November 10, 2021, 07:46:29 pmI think that just the fact that pascal does not allow non english characters in identifiers is already an accessibility issue on it's own, as especially programming beginners will usually use their own language for identifiers, which they simply can't with pascal, sealing the language off for like half the planets population. Sooner or later we'll probably implement support for Unicode identifiers, cause Delphi allows them as well. ---------------------------------------------------------------------- winni · 2021-11-11 09:37 Hi! The feature of Unicode Identiftiers is again a mistake. It will prohibit the international exchange of source code. I am not able to translate identifiers - as shown in the link - from Persian, Malayalam or Sinhala to ASCII. Instead of simplyfing things it seems to get worse. Winni ---------------------------------------------------------------------- PierceNg · 2021-11-11 12:24 Quote from: Warfley on November 10, 2021, 09:28:45 pmSure, if you enable internationalization/i18n, but if you develop an application as a local only for other locals, using i18n just adds additional work and debugging overhead for you, personally I write most of my applications in english and don't care about i18n because it is simply a lot of extra work which I don't think is justified for my 1 man projects. And even without strings, there are still comments, I regularly see non english comments, in a korean project I've seen all the comments where in korean. Generally speaking people just like to use their native language. SQL too. Code: SQL  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i % sqlite3 /tmp/ml.db SQLite version 3.27.2 2019-02-25 16:06:06Enter ".help" FOR usage hints.sqlite> .header ONsqlite> .schema CREATE TABLE 表一 (键一 INTEGER PRIMARY KEY, 列二 text DEFAULT '中文');sqlite> SELECT * FROM 表一;键一|列二1|中文2|值二sqlite> In Japan, I watched some Open University classes on TV where they taught PostgreSQL, and similar to my Chinese example above, table names, column names etc were all in Japanese. ---------------------------------------------------------------------- MarkMLl · 2021-11-11 12:35 Quote from: PierceNg on November 11, 2021, 01:24:25 pmIn Japan, I watched some Open University classes on TV where they taught PostgreSQL, and similar to my Chinese example above, table names, column names etc were all in Japanese. ...where I don't think that (modern) Chinese or Japanese have directionality issues. Also there's a distinction to be made between keywords, table/column identifiers and identifiers which identify something executable... which gets "interesting" in the case of e.g. Pascal where a variable (which could fairly be expected to be named using a localised language) points to a function/procedure (which might arguably be subject to more restrictive naming, for reasons discussed at the start of the thread). MarkMLl ---------------------------------------------------------------------- PascalDragon · 2021-11-11 12:44 Quote from: winni on November 11, 2021, 10:37:20 amIt will prohibit the international exchange of source code. I am not able to translate identifiers - as shown in the link - from Persian, Malayalam or Sinhala to ASCII. Not every source code needs to be internationally exchanged however. A small company in Malasya developing local software might prefer to use native identifers, because a) the (native) developers can maybe express themselves better in their native language and b) the terms of the problem domain can be used (e.g. if local terms for some processes need to be used that would be hard to translate). Also people like to program in their native language. There are many programs out there that are written with German identifiers (I just saw one in one of the other threads). People that use the Latin alphabet can do that, but everyone else is precluded from that with FPC. So simply for this inclusivity FPC should support this as well, it's only fair. ---------------------------------------------------------------------- marcov · 2021-11-11 13:06 Quote from: winni on November 11, 2021, 10:37:20 amHi! The feature of Unicode Identiftiers is again a mistake. It will prohibit the international exchange of source code. I am not able to translate identifiers - as shown in the link - from Persian, Malayalam or Sinhala to ASCII. And you will be able to translate those identifiers if they are somehow superimposed onto the Latin alphabet? Keep in mind that just limiting to ASCII doesn't mean it will be English, languages like Turkish and Malay/Indonesian use Latin alphabet too. In short, I think unicode identifiers are inevitable, and as Pascaldragon says, Delphi changed a decade ago. But it is a major undertaking, and half solutions are not really desirable. Either do it right, or wait. ---------------------------------------------------------------------- kupferstecher · 2021-11-11 13:22 Quote from: MarkMLl on November 11, 2021, 01:35:35 pm...where I don't think that (modern) Chinese or Japanese have directionality issues. You mean right-to-left writing? There are rare occasions for that in Chinese, see attached picture and compare the characters on the left and right side of the aircrafts, both are written from front to back of the aircraft instead of left to right. (Picture source: https://aircraftphotos.top/2020/07/06/%E4%B8%AD%E5%9B%BD%E4%B8%9C%E6%9 6%B9%E8%88%AA%E7%A9%BA%E4%BA%91%E5%8D%97%E5%85%AC%E5%8F%B8-b737-700-b- 5826/ https://aircraftphotos.top/2020/08/03/%e4%b8%9c%e6%96%b9%e8%88%aa%e7%a 9%ba737-700-b-5276-%e5%ad%94%e9%9b%80%e6%b6%82%e8%a3%85/) ---------------------------------------------------------------------- MarkMLl · 2021-11-11 13:58 Quote from: kupferstecher on November 11, 2021, 02:22:00 pmQuote from: MarkMLl on November 11, 2021, 01:35:35 pm...where I don't think that (modern) Chinese or Japanese have directionality issues. You mean right-to-left writing? There are rare occasions for that in Chinese, see attached picture and compare the characters on the left and right side of the aircrafts, both are written from front to back of the aircraft instead of left to right. Yes, but that would imply direction-changing escapes at the start of a literal or (perhaps) an identifier, not embedded in it and definitely not with multiple embedded escapes. There's already a rule that identifiers have to start in a certain way that differs from what might follow (i.e. the acceptability of digits) and that could potentially be extended to when escapes were permitted. Should two identifiers which comprised the same character sequence but rendered in opposing directions be equivalent or distinct? Should a reserved word rendered in the unconventional direction be considered equivalent? MarkMLl ---------------------------------------------------------------------- winni · 2021-11-11 16:06 Hi! In the early Delphi days I was happy to find some solutions in a russian Delphi forum. I did not understand a word of the communication, but the code was (mostly) written in some kind of "Pascal English". If the code was cyrillic it would't have helped me. Winni ---------------------------------------------------------------------- kupferstecher · 2021-11-11 16:45 Quote from: MarkMLl on November 11, 2021, 02:58:45 pmShould two identifiers which comprised the same character sequence but rendered in opposing directions be equivalent or distinct?Good question. One might search for a term (in the source code) and expect it to be typed as seen or he might search a term where he doesn't even know it is present in reversed form. In the second case the equivalent representation could be helpful. Anyways i don't think that the reversed direction is required on source level. I just wanted to show an example of an unexpected right-to-left occurance. ---------------------------------------------------------------------- Warfley · 2021-11-11 22:12 Quote from: MarkMLl on November 11, 2021, 02:58:45 pmYes, but that would imply direction-changing escapes at the start of a literal or (perhaps) an identifier, not embedded in it and definitely not with multiple embedded escapes. There's already a rule that identifiers have to start in a certain way that differs from what might follow (i.e. the acceptability of digits) and that could potentially be extended to when escapes were permitted. Well, but the normal syntax constructs are left to right, arabic text is right to left, so if you have an arabic identifier it must always start with a direction switch, similar it always has to end with one to start the parameter list. Also it is common for such languages to embedd english words in the latin alphabet into their sentences. And identifiers usually are not words but phrases, a function like "PerfomRESTRequest" could be written as the arabic phrase for perfom request and the latin word REST, so in such a case you need at least 3 direction changes, possibly 4. Honestly I see no way around to, if langauges like arabic should be supported, to allow for arbitrary direction changes. Quote from: MarkMLl on November 11, 2021, 02:58:45 pmShould two identifiers which comprised the same character sequence but rendered in opposing directions be equivalent or distinct? Should a reserved word rendered in the unconventional direction be considered equivalent? MarkMLl Well pretty simple, an identifier is referenced by the characters it is made of, not how the characters are displayed. Simply for the fact that it would be completetly impractical to compare renderings, you would have to render every identifier and compare the bitmaps or something similar. Then the question, what rendering engine should be used for this? The thing is, yes unicode is complicated and no one likes internationalization. But it's the only viable solution if you don't want to exclude over half the planet. Quote from: winni on November 11, 2021, 05:06:20 pmHi! In the early Delphi days I was happy to find some solutions in a russian Delphi forum. I did not understand a word of the communication, but the code was (mostly) written in some kind of "Pascal English". If the code was cyrillic it would't have helped me. Winni Forcing people to use a language you understand helps you to understand their code... What a revelation. But you know this goes the other way around, while this makes it easier for you to understand the code, for the russians, the simple fact of learning Pascal requires them to learn english, which makes it much harder than if they could use russian and look at other russian code. I wrote my first programs (Delphi) at the age of 7-8, where I did not understand a word english. I really started learning programming at the age of 12/13, where I knew a little bit english but wasn't that confident with it. If I couldn't have used german identifier names, I would simply not have learned Pascal. Restricting the identifiers to the english alphabeth only will simply result in people from other countries not using the language, but rather languages that will support their alphabet. And having less overall pascal programmers won't help you finding more code online either. ---------------------------------------------------------------------- MarkMLl · 2021-11-12 07:48 Quote from: Warfley on November 11, 2021, 11:12:59 pmWell, but the normal syntax constructs are left to right, arabic text is right to left, so if you have an arabic identifier it must always start with a direction switch, Yes, START WITH is the point I'm trying to emphasise. That doesn't mean that such things remain legal after the initial direction indicator. QuoteWell pretty simple, an identifier is referenced by the characters it is made of, not how the characters are displayed. I'm unhappy with all possibilities. It is not desirable for somebody to be able to define iP as 3.0 locally. MarkMLl ---------------------------------------------------------------------- kupferstecher · 2021-11-12 10:07 Perhaps some arabic speaker could shed light on how they would see a useful way of using right-to-left in source code. I can't really imagine it to be readable to have right-to-left identifiers in source code that is generally left-to-right. ---------------------------------------------------------------------- PascalDragon · 2021-11-12 14:37 Quote from: MarkMLl on November 11, 2021, 02:58:45 pmYes, but that would imply direction-changing escapes at the start of a literal or (perhaps) an identifier, not embedded in it and definitely not with multiple embedded escapes. There's already a rule that identifiers have to start in a certain way that differs from what might follow (i.e. the acceptability of digits) and that could potentially be extended to when escapes were permitted. Should two identifiers which comprised the same character sequence but rendered in opposing directions be equivalent or distinct? Should a reserved word rendered in the unconventional direction be considered equivalent? Please take a look at the Unicoode Identifier and Pattern Syntax which is what we'd follow. Quote from: winni on November 11, 2021, 05:06:20 pmIn the early Delphi days I was happy to find some solutions in a russian Delphi forum. I did not understand a word of the communication, but the code was (mostly) written in some kind of "Pascal English". If the code was cyrillic it would't have helped me. And if the code would have been in Latinized Russian it wouldn't have helped you either. Same that a Pascal program written with German identifiers won't help an Arabian or a Chinese (as long as they are don't able to understand German). ---------------------------------------------------------------------- Warfley · 2021-11-12 15:05 Quote from: MarkMLl on November 12, 2021, 08:48:06 amYes, START WITH is the point I'm trying to emphasise. That doesn't mean that such things remain legal after the initial direction indicator. But also that is not quite right. As I said, latin charset names are kept in latin charset. So consider these three very normal function identifier names (I use english terms because I don't speak arabic but it explains the concept) PerformRESTRequest,  QueryRESTAPI, RESTQueryGET, RESTRequest One or more names (here in red) can occur at any place. They can be at the start, so the identifier would not start with a right-to-left char. They can be at the end so an identifier might not end with a left-to-right char. They can be enclosed by natural language or can encompass it. Basically a left-to-right and right-to-left can occur at any place. The only rule is that if a right-to-left one occurs, there must always be a left-to-right at some point afterwards. But thats pretty much it. Quote from: MarkMLl on November 12, 2021, 08:48:06 amI'm unhappy with all possibilities. It is not desirable for somebody to be able to define iP as 3.0 locally. MarkMLl Sure internationalization is not great, but it's the best possible solution. But your example isn't actually that bad. Sure it will look weird like but all the code support (like ctrl+click/go to definition) will not consider this whenever you write pi. I mean we aren't exactly inventing something new here. Pretty much all of the most popular languages (C++, Java, Python, C#, etc.) use unicode identifiers for years, in some cases for more than a decade, and it works fine. In the very most cases it will change absolutely nothing for most of the programmers here. I've been using C++ and Java for a long time now, and I must say, I never had any problems related to unicode in the code. ---------------------------------------------------------------------- SymbolicFrank · 2021-11-12 15:27 At some point, Microsoft decided it would be a good idea to translate their programming languages. The result is mostly unreadable, even for the native speakers. You would have to redesign the whole language and syntax, simply translating the words doesn't work. And a hodgepodge of English programming syntax and native identifiers isn't better. Even more so: you should never put the content of strings directly in the code, especially if you want it to work in different languages. Ergo: keep the programming language, syntax and identifiers in English. That automatically prevents chars that look the same but aren't as it isn't easy to translate those in Unicode. In other words: you might think you're typing the exact same word, but it isn't. And the only way to figure out how a certain glyph is translated is with a hex-editor or by copy-pasting it in a program that tells you. ---------------------------------------------------------------------- MarkMLl · 2021-11-12 15:34 Quote from: PascalDragon on November 12, 2021, 03:37:45 pmPlease take a look at the Unicoode Identifier and Pattern Syntax which is what we'd follow. ...which appears to say that direction specifications should not appear in identifiers: "Certain Unicode characters are known as Default_Ignorable_Code_Points. These include ... bidirectional ordering control, ... "Variation selectors, in particular, including standardized variants and sequences from the Ideographic Variation Database, are not included in the default identifier syntax. These are subject to the same considerations as for other Default_Ignorable_Code_Points listed above. Because variation selectors request a difference in display but do not guarantee it, they do not work well in general-purpose identifiers." MarkMLl ---------------------------------------------------------------------- SymbolicFrank · 2021-11-13 08:40 If you were to translate the programming language to, for example, Chinese or Korean, not only the direction is a problem. They basically have two languages, one consisting of ~50+ individual glyphs you can combine to form words, somewhat like Latin. But, those glyphs get combined into more complex glyphs. And the second language only exists of thousands of more complex, combined glyphs. Which are of course hard to type on a keyboard. So, first you should make a canvas on which you can do calligraphy: laying out glyphs in a human-readable format. From painting them yourself to building them by combining other glyphs. And then you have the computer process that as a programming syntax? How? And if you think that is far fetched: the Unicode committee shot itself in the foot by more or less demanding that even for European languages that have a lot of non-Latin and/or combined glyphs. That's also why they are so hard to sort. One of the first graphical handhelds, the Palm, already had a way to input glyphs by drawing on the screen with a stylus. Only, it translated those swipes to Latin characters. What you should do is something like: