LCLIntf.ExtTextOut and Dx offsets for accent-chars lazarus / 31-LCL ---------------------------------------------------------------------- AlexTP · 2025-08-22 18:05 In CudaText, I want to solve problems with ExtTextOut rendering of accent-chars (U+0301, for example, is the accent-char used for Ru text). I made primitive demo to mimic CudaText. Attached. Currently: accent-chars are rendered in separate cells, and with 'circles' before the accent - on Linux GTK2 - screenshot attached. I need: render accents like in TextOut call, ie accent located over the lefter char, and no 'circle'. ---------------------------------------------------------------------- AlexTP · 2025-08-22 18:33 What I tried: set to 0 the Dx-item for accent chars (see the demo's code). It helped to hide accent additional cell, but 'circle' is still rendered (over the normal char). And line-figure of accent is painted over the wrong letter then. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   for i:= 1 to Length(sw) do  begin    if Ord(Sw[i])=$301 then      Dx[i-1]:= 0 // <---------- it don't help    else      Dx[i-1]:= CharW* 5 div 4; //for test, I need customized char-widths  end; ---------------------------------------------------------------------- DomingoGP · 2025-08-23 09:44 After many trial and error I thing I got it  . For each [grapheme] the dx array must have an entry for each code point (32 bits) The first entries for each grapheme must be 0 and the last entry the desired char width. example: Бы́ло    graphemes       [Б]          [ы́]         [л]     [о] utf32        [ 00411 ] [ 0044b 00301 ] [0043b] [0043E] Dx                15         0    15       15      15 Dirty working code (with fixed equal width for each char) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     for i:= 1 to Length(sw) do  begin    //    ccw:= 20;// CharW* 5 div 4;    ccw:= CharW;    if (Ord(Sw[i])=$301) and (i>1) then    begin      Dx[i-2]:= 0;      //ccw:=CharW*2;    end;    Dx[i-1]:= ccw; //CharW* 5 div 4; //for test, I need customized char-widths  end;  I have attached one project for testing. I would dare to recommend You my port of libgrapheme for Pascal to segment utf8 text into graphemes. https://github.com/DomingoGP/libgrapheme_pascal/tree/pascal/pascal_src since your aproach will not work with composed emojis with many codepoints. PS: the images are taken from my windows 7 that not render well some emojis. Regards Domingo. ---------------------------------------------------------------------- AlexTP · 2025-08-23 10:13 Thanks. This method works on Win, but Linux GTK2 app shows this - accent char overfills the letter char. Attached. Maybe for GTK2/Qt the only working method is splitting the string to parts containing one letter+one accent, and rendering of accented parts with TextOut. ---------------------------------------------------------------------- Martin_fr · 2025-08-23 10:30 SynEdit does the 0 width for combining codepoints... (and you can force the IDE source editor to use it, by setting "extra char width" in the options) Note: The below info is older / not recently reviewed. The entire concept of the "DX" array is not natively supported on GTK2 (not sure you may refer to QT / not tested QT). So using it, will have side effect. E.g. when outputting Arabic, which "connects" graphemes throughout the word. So most letters look different inside a word, than they do when printed stand alone. But under gtk, the "DX" code prints each letter on its own. I haven't looked at how the accents are handled by that code... Maybe that can be improved... Maybe there is a bug, that it prints the 2 separately (but I have not checked). ---------------------------------------------------------------------- DomingoGP · 2025-08-23 13:18 Lately, I've been playing around with the SynEdit code to improve emoji support, and I think the result is not bad. It improves cursor behavior, which now moves from grapheme to grapheme. The backspace deletes entire graphemes. For right-to-left texts, it uses the old code, and for the rest, it draws the graphemes one by one and works on GTK2 as you can see in the attached image. Arabic text only looks good with certain fonts. The attached project is a git repository with a commit containing the old code and a coomit whti the changes, so they can be easily viewed with GIT GUI or the github browser, for example. You can compile it without having to mess up lazarus at all. It has also a folder with the unmodified synedit to esily compare old and new behavior. The disadvantage (apart from the bugs I may introduce) is that it depends on libgrapheme. Maybe Martin would like to take a look and tell us what he thinks. Since the code is so big and the forum don't allow to attach I have created a repo in Github. https://github.com/DomingoGP/TestSyneditEmojis The forum software gives error with the link,  you must edit the link and remove to get  access.  Attached is an image running on Windows 7 and Linux Mint GTK2. ---------------------------------------------------------------------- Martin_fr · 2025-08-23 14:20 Quote from: DomingoGP on August 23, 2025, 03:18:10 pmLately, I've been playing around with the SynEdit code to improve emoji support, and I think the result is not bad. I am trying to figure the exact list of changes (other than the one visual).... Maybe you can attach a text file with the emojis that cause issues? And also, if the smileys - always cause issues - only if "DX" is used/forced DX (ETO) is used if either - the font is not strictly monospaced - extra char spacing is set For me emojis seem to work fine (using DejaVu / not sure if Windows uses font substitution). CHANGE 1) One thing I can see, is that you draw the emojis bigger than the space in the grid. Which also means they get cut off in height. I don't know if Unicode defines them as full or half-width? QuoteIt improves cursor behavior, which now moves from grapheme to grapheme. The backspace deletes entire graphemes. That isn't a display issue? Well at least the backspace isn't. And it does for me already with existing code (hence, if you attach a text with the emojis that don't). CHANGE 2) ??? (works for me already) "delete entire" => that sounds like SynEdit does ignore some of the "combining codepoints"?. Mind, they are currently - hardcoded (so its possible some are missing) - hardcoded in a totally wrong place.... But cleaning that up, will take some work (and doesn't affect functionality) Btw, I read somewhere that glyphs made from several codepoints, may allow (as in an implementation may offer) placing the caret mid-glyph, and removing individual codepoints.... (SynEdit doesn't) QuoteFor right-to-left texts, it uses the old code, and for the rest, it draws the graphemes one by one and works on GTK2 as you can see in the attached image. Currently I am not planing to immediately replace ExtTextOut. But on the long run, I want to look at that (to make it work, and work with all the rest, may need some bigger refactor). Btw, have you tested your code with word wrap? QuoteArabic text only looks good with certain fonts.I can't judge that, since I can't read it... I have it installed on my Windows system (and it took me weeks to figure out what ctrl+shift now does, and how to switch it back). On GTK (and don't know on others) it looks bad with DX => then yes, QuoteThe attached project is a git repository with a commit containing the old code and a coomit whti the changes, so they can be easily viewed with GIT GUI or the github browser, for example. You can compile it without having to mess up lazarus at all. QuoteThe disadvantage (apart from the bugs I may introduce) is that it depends on libgrapheme. Not sure if/when I get to this. ---------------------------------------------------------------------- Martin_fr · 2025-08-23 14:25 Ok I found the comment QuoteCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     //domingo:  emojis are FullWidth char So then that list may need to be updated. I need to get around to write a tool, that takes the relevant files from the Unicode definition, and creates the Pascal snippets. (so it can be updated more easily). But also, that code / that unit => it is totally wrong. As in: its logically/design-wise in the wrong place. (My bad) ---------------------------------------------------------------------- Martin_fr · 2025-08-23 14:57 About the hardcode Full/Half width, the hardcoded RTL/LTR, and partly the combining info... For now they need to stay hardcoded. All the code that accesses this, will need to be checked for minimizing its needs to access this. Working on text with very long lines (line len > 100K, maybe > 1M) can cause slow down. (Make sure to test without extra markup) When and if that is optimized, then the full/half width (and RTL) info need to move into the text-drawing code. (Yet they will still be needed outside). And when they have been moved, then there can be alternative variants of getting that info.... But to get there is still a long way. Also some Unicode info may be of interest to code outside the editor. And other apps... I haven't investigated which interfaces there are to get such info (OS, common libs, regular updated hardcoded units, ...) But that is a different story. ---------------------------------------------------------------------- DomingoGP · 2025-08-23 16:05 before all code changes there is a //domingo comment. QuoteI am trying to figure the exact list of changes (other than the one visual).... Maybe you can attach a text file with the emojis that cause issues? And also, if the smileys - always cause issues - only if "DX" is used/forced DX (ETO) is used if either - the font is not strictly monospaced - extra char spacing is set Basically all Emoji with ZWJ Sequence separator like composed emojis ***👍️*👍🏻*👍🏼*👍🏽*👍🏾*👍🏿** 👨🏽‍👩🏽‍👦🏽‍👦🏽 with monospaced and variable font pitch. https://emojipedia.org/emoji-zwj-sequence Actually the cursor moves between the parts of the emoji and delete and backspace only removes one part. In my opinion a black hand emoji is a character as a whole, not a hand and a color rectangle we must delete all. try to select some of these or maybe set the overwrite cursor and desplace over it only select a half of gliph on a monospaced font on variable font works well. 🐴🐎🐫🐑🐘🐍🐦🐤🐔🐧🐛🐙... Quote "delete entire" => that sounds like SynEdit does ignore some of the "combining codepoints"?. It happens with the composed emojis and in strings normalized in the NFD form when pressing backaspace deletes the accent and not the whole character as I expected and also don't render the accents on some fonts.. NFD áéíóú cañon ÚÀÇçÑ ÀÈÌÒÙ àèìòù äëïö <-copy this line in the ide and try backspace  QuoteBtw, have you tested your code with word wrap?No, but with code folding works well. I know it's not 100% perfect, but since I'm not sure  it will be mixed in the near future, I'm not going to spend much more time on it.  I think incremental changes are best. Anyway, I think it improves the current situation even without touching the rendering, just to calculate the length of the graphemes for cursor movement and text selection. Anyway, thanks for the time you've spent on it. ---------------------------------------------------------------------- Martin_fr · 2025-08-23 20:34 Quote from: DomingoGP on August 23, 2025, 06:05:44 pmBasically all Emoji with ZWJ Sequence separator like composed emojis ***👍️*👍🏻*👍🏼*👍🏽*👍🏾*👍🏿** 👨🏽‍👩🏽‍👦🏽‍👦🏽 Ah, so ZWJ needs to be added as "combining"-ish and also include the next char. Then the editor would see it as one char. Well, at least when it is between relevant emoji codepoints. It needs to be investigated what happens when it is between other codepoints. They should be joined, but will they be combined into something that is one char width? (full/half depending on what the input was)? On gtk2, output with DX will likely need to be fixed for that. And the full width list will also need an update. ---------------------------------------------------------------------- Martin_fr · 2025-09-04 13:58 Quote from: DomingoGP on August 23, 2025, 06:05:44 pmIt happens with the composed emojis and in strings normalized in the NFD form when pressing backaspace deletes the accent and not the whole character as I expected and also don't render the accents on some fonts.. NFD áéíóú cañon ÚÀÇçÑ ÀÈÌÒÙ àèìòù äëïö <-copy this line in the ide and try backspace  And that seems to have been intention Revision: d61a0ebb2e66e0c09154d975c93dc6891a24881f Author: martin Date: 09/12/2012 04:31:37 Message: SynEdit: ecDelete use codepoints instead of chars git-svn-id: trunk@39483 - Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i             // delete char              LogCounter := LogCaretXY.X;            if FCaret.BytePosOffset = 0 then begin                LogCaretXY.X := FTheLinesView.LogicPosAddChars(Temp, LogCaretXY.X, -1, [lpStopAtCodePoint]);                LogCounter := LogCounter - LogCaretXY.X;              end  "lpStopAtCodePoint" => is making that decision. Not sure why, or if it really is that good a decision... - Selecting the char works on the entire char - being in front of it, and pressing "del" (del next char) deletes the entire char. ---------------------------------------------------------------------- DomingoGP · 2025-09-04 19:21 Quote from: Martin_frNot sure why, or if it really is that good a decision... For sure it was for compatibility with other editors. I have done a little research and these are my findings. Code: [Select] SynEdit left and right keys      move by graphemes backspace                delete codepoints del                      delete graphemes keyboard selection       select graphemes          mouse selection          select graphemes  Notepad left and right keys      move by graphemes backspace                delete codepoints del                      delete graphemes keyboard selection       select graphemes mouse selection          select graphemes         Libreoffice writer left and right keys      move by graphemes backspace                delete codepoints    del                      delete graphemes keyboard selection       select graphemes mouse selection          select graphemes Visual studio code left and right keys      move by codepoints backspace                delete codepoints del                      delete codepoints keyboard selection       select codepoints           // NOT INTUITIVE mouse selection          select graphemes Notepad++ left and right keys      move by codepoints backspace                delete codepoints del                      delete codepoints keyboard selection       select codepoints mouse selection          select graphemes It therefore seems that there is unanimity that the backspace key deletes codepoints, even though it breaks the symmetry with the del key, and I personally believe that deleting graphemes is more intuitive. There is not as much uniformity with the other options, but I believe that selecting and moving by graphemes is the best option. In any case, these are rare use cases in a programming editor since the text generaly use the NFC normalization, and as it is, it works quite well.                        In the repository with my sample project, I have made a few more commits that correct the calculation of DX values for graphemes that have multiple codepoints by setting a zero for each codepoint except the last one, where I set the width. There is also a micro-optimization by changing the calls to ExtUtf8Out (which only wraps ExtTextOut) to ExtTextOut. ---------------------------------------------------------------------- Martin_fr · 2025-09-04 20:01 I am thinking about adding an ecDeleteLastCodePoint (and replace it by default for ecDeleteLastChar, then change ecDeleteLastChar). Then both would be available. Afaik, the WinScribe API (or something like it) can return "mid char/graphem" caret positions, so half steps can be visualized. Btw/Similarly, a text caret can have a left/right affinity marker (a little flag/triangle) to indicate its position when between LTR and RTL text. Imagen the uppercase was RTL:   abcXYZdef if the caret is after "c" then it is in FRONT of "Z", and once it goes past "X" it is visually again at the same location as the "after c" (only SynEdit will not display it there, as it only supports one side of each char) I have updated the EastAsian Width table. This is until width can be measured.  As the Unicode EAW property is not exact.... (and there are ambiguous ones). But right now, the code is not ready for doing measuring.... I have also updated the list of combining and added the emoji modifiers (skin tone). But not the ZWJ sequences. And I applied a fix to the gtk2 output, os the modifiers work. (and also the ä with combining / if you look at your example all the accents/dots were one char to the right). Technically ZWJ sequences should now work, **IF** the ZWJ *and* the following joined codepoint were set to CharWidth=0 (e.g. like combining). And that is without the external lib you use.... But I am not sure, when I may continue on that. (That wasn't my current plan, got other stuff on my list) I haven't checked the updates in your repo. (just reading your post now) Quotet the calculation of DX values for graphemes that have multiple codepoints by setting a zero for each codepoint except the last one, where I set the width. Shouldn't the width be on the first? That is what SynEdit would expect. Of course, the gtk2 drawing code is (was) broken for that. But I just fixed that. -- EDIT: Also, Win32 wants the width on the first ---------------------------------------------------------------------- DomingoGP · 2025-09-04 21:03 Quote from: Martin_fr on September 04, 2025, 10:01:55 pmI am thinking about adding an ecDeleteLastCodePoint (and replace it by default for ecDeleteLastChar, then change ecDeleteLastChar). Then both would be available. Good idea!. Quote from: Martin_fr on September 04, 2025, 10:01:55 pm And that is without the external lib you use.... I'm glad you can do it without external dependencies. Although libgrapheme is a proven library, used for example by Skia, and covers all grapheme combinations. Quote from: Martin_fr on September 04, 2025, 10:01:55 pmShouldn't the width be on the first? That is what SynEdit would expect. Of course, the gtk2 drawing code is (was) broken for that. But I just fixed that. -- EDIT: Also, Win32 wants the width on the first I tested basically on windows 7 and works well with the width at the end, but on windows 11 (in VM) works well in both cases as you can see in the attached image for the text (🤦🏼‍♂️a🤦🏼‍♂️a) You can test this with the program in my repo:  ExtTextOut/Project1. Quote from: Martin_fr on September 04, 2025, 10:01:55 pmBut I am not sure, when I may continue on that. (That wasn't my current plan, got other stuff on my list) That's fine by me, I don't really use emojis in my code, and I don't think it's a priority for anyone, although it would be nice to improve emoji support. Thanks . Domingo. ---------------------------------------------------------------------- Martin_fr · 2025-09-04 21:18 Quote from: DomingoGP on September 04, 2025, 11:03:05 pmI'm glad you can do it without external dependencies.Well, I don't know if ExtTextOut will do all combinations, and that may depend on the font actually having the emoji (the base ones), and IIRC pango combining them... At some point, there should be alternative "TextDrawers". Then they could be chosen, by just adding them. But that needs massive refactoring of the SynEdit... QuoteThat's fine by me, I don't really use emojis in my code, and I don't think it's a priority for anyone, although it would be nice to improve emoji support. The below work now in native SynEdit, with/without DX (except for the first thumb up / no idea why that does not / not yet debugged) (Windows keeps them black and white but that is the same in Notepad++) Quote*👍️*👍🏻*👍🏼*👍🏽*👍🏾*👍🏿*