How ANSI character map should behave on multibyte code pages lazarus / 6-IDE-CodeTools ---------------------------------------------------------------------- Sam Bliss · 2009-07-30 08:16 I am using Lazarus 0.9.27 r20911 FPC 2.2.5 i386-win32-win32/win64 The language of my computer is Chinese. When I use ANSI charactor map and put the mouse on #128..#255, the dialog shows. Quote                  Lazarus ============================================ Access voilation. Press OK to ignore and risk data corruption. Press Cancel to kill the program. ============================================                  OK CancelGet screen shots and more information see here. ---------------------------------------------------------------------- skalogryz · 2009-07-30 08:31 as Theo, already noted. May be, it's enough to fix the Lazarus, so no access violation is risen. And the half of ANSI table (128..255) should be left blank. Because there're no 128..255 characters in windows Chinese code page. ---------------------------------------------------------------------- theo · 2009-07-30 09:04 Yes, simply apply Patch #1 against IDE crash. Ansi tab is sort of obsolete for UTF-8 systems anyway. Lookup your code points in the unicode tab, in your case "Latin-1 Supplement" ---------------------------------------------------------------------- Sam Bliss · 2009-07-30 09:07 Re: skalogryyz Yes, I made a patch, but it is not perfect. See http://bugs.freepascal.org/view.php?id=14189. ---------------------------------------------------------------------- Sam Bliss · 2009-07-30 09:13 Re: theo Yes, but sometimes we need to search a character in ANSI tab, it's fast and easy. ---------------------------------------------------------------------- Sam Bliss · 2009-07-30 09:19 Re: theo You replied that issue, have you seen note 0029399 in that issue? It means the #5 patch is broken. ---------------------------------------------------------------------- theo · 2009-07-30 09:26 Quote from: Sam Bliss on July 30, 2009, 11:19:03 amIt means the #5 patch is broken. I meant: apply charactermapdlg.pas.diff.fixcrash.zip and leave the rest. ---------------------------------------------------------------------- Sam Bliss · 2009-07-31 09:15 Re: theo Quote from: charactermapdlg.pas.diff.fixcrash.zip by theoCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i CharOrd := Ord(UTF8ToAnsi(StringGrid1.Cells[Col, Row])[1]);Character #128..#255 need 2 bytes in UTF-8. We need to convert UTF-8 to Unicode, then convert to ANSI, or Col*16+Row-17. Please test it by yourself, is it OK? ---------------------------------------------------------------------- theo · 2009-07-31 09:59 Quote from: Sam Bliss on July 31, 2009, 11:15:57 amCharacter #128..#255 need 2 bytes in UTF-8. All I can say is, that the ANSI Tab works as expected on my Windows box (ANSI, German). It shows "?" on Linux for chars > 127 however. That's why we have the unicode tab. I don't have the right to apply patches to the LCL/IDE anyway. So let some core-developer (Paul Ishenin?) decide what to do. ---------------------------------------------------------------------- skalogryz · 2009-07-31 10:20 Quote from: theo on July 31, 2009, 11:59:07 amAll I can say is, that the ANSI Tab works as expected on my Windows box (ANSI, German). It shows "?" on Linux for chars > 127 however. That's why we have the unicode tab. I don't have the right to apply patches to the LCL/IDE anyway. So let some core-developer (Paul Ishenin?) decide what to do. what's German Windows code page? shouldn't it be ANSI, so special German characters are shown? ---------------------------------------------------------------------- theo · 2009-07-31 10:25 Quote from: skalogryyz on July 31, 2009, 12:20:23 pmshouldn't it be ANSI, so special German characters are shown? It does on windows, not on Linux (UTF-8) of course. ---------------------------------------------------------------------- Sam Bliss · 2009-08-01 12:36 Germany systems use these charsets: QuoteCP437 (United States) CP1141 (IBM EBCDIC - Germany(20273 + Euro)) CP1250 (Eastern European) CP20273 (IBM EBCDIC Germany) CP20106 (IA5 Germany) CP20127 (US-ASCII) CP28592 (ISO 8859-2 - Eastern European) CP65000 (UTF-7)* CP65001 (UTF-8)* ... ---------------------------------------------------------------------- Vincent Snijders · 2009-08-02 11:11 IMHO, the ansi character map is only valid for single byte code pages, or rather for characters that can be represented with one byte. I don't know mulitbyte code pages well enough, to say wat exactly should happen, but if say #130 is a valid (complete) character for your multibyte code page, it should be shown, if it is not a valid character, but just a partial character, a ? (or maybe empty) should be shown. ---------------------------------------------------------------------- skalogryz · 2009-08-03 16:21 Sam Bliss, the patch is applied. you should to use Unicode, to paste #128..#255 characters. Half of the ANSI table will be empty. ---------------------------------------------------------------------- Sam Bliss · 2009-08-04 07:23 I hope the ANSI charactor map will not be deleted in the future. Maybe Unicode charactor map is more important, but we also need ANSI charactor map. Swap Unicode charactor map to the first, and ANSI to the second one is better. ---------------------------------------------------------------------- Sam Bliss · 2009-08-04 07:35 Vincent Snijders, You know,  charactermapdlg.pas-5.zip can not work very well (see bug report note id #0029399 and screen shot charactermapdlg.pas-5.PNG). Quote from: Sam Bliss on July 30, 2009, 10:26:00 amFor charactermapdlg.pas-5, when I move my mouse on #$80..#$FE, it shows incorrect Decimal and Hex number! ASCII Dec Hex UTF-8 Display as (see the screen shot) 129   63  3F  C281  square 130   63  3F  C282  square 131   63  3F  C283  square ...   ... ... ...   ... 159   63  3F  C29F  quuare 160   63  3F  C2A0  space 161   63  3F  C2A1  letter i 162   51  33  C2A2  letter c with a line 163   52  34  C2A3  letter L with a line 164   161 A1  C2A4  small square 165   54  36  C2A5  letter Y with two lines 166   63  3F  C2A6  spilliter 167   161 A1  C2A7  two letter S's 168   161 A1  C2A8  two dots 169   63  3F  C2A9  letter C with circle 170   63  3F  C2AA  small number 2 171   63  3F  C2AB  two "less than" characters ...   ... ... ...   ...Snapshot here You said in the bug note: QuoteI committed the last patch. Thanks.I don't know what patch did you use, Maybe use charactermapdlg-4.zip is better. charactermapdlg.pas.diff.fixcrash.zip does not work, too. Quote from: Sam Bliss on July 31, 2009, 11:15:57 amRe: theo Quote from: charactermapdlg.pas.diff.fixcrash.zip by theoCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i CharOrd := Ord(UTF8ToAnsi(StringGrid1.Cells[Col, Row])[1]);Character #128..#255 need 2 bytes in UTF-8. We need to convert UTF-8 to Unicode, then convert to ANSI, or Col*16+Row-17. Please test it by yourself, is it OK? So, if you think, these patches are all not perfect, please tell us, let us write a new patch. In my opinion, use Patch 4 is great. ---------------------------------------------------------------------- theo · 2009-08-04 07:43 Quote from: Sam Bliss on August 04, 2009, 09:35:23 amcharactermapdlg.pas.diff.fixcrash.zip does not work, too. Is the IDE still crashing? ---------------------------------------------------------------------- Sam Bliss · 2009-08-04 08:02 Let me think, this patch is yours right? I said the thing wrong hereQuote from: Sam Bliss on August 04, 2009, 09:35:23 amQuote from: Sam Bliss on July 31, 2009, 11:15:57 amRe: theo Quote from: charactermapdlg.pas.diff.fixcrash.zip by theoCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i CharOrd := Ord(UTF8ToAnsi(StringGrid1.Cells[Col, Row])[1]);Character #128..#255 need 2 bytes in UTF-8. We need to convert UTF-8 to Unicode, then convert to ANSI, or Col*16+Row-17. Please test it by yourself, is it OK? ---------------------------------------------------------------------- Sam Bliss · 2009-08-04 08:05 Still now, there's only one patch works! No.4 Patch. ---------------------------------------------------------------------- Vincent Snijders · 2009-08-04 08:08 What is wrong there? What does AnsiToUTF8(#200) return for your codepage? Doesn't it return the empty string? ---------------------------------------------------------------------- Sam Bliss · 2009-08-04 08:27 Yes! ---------------------------------------------------------------------- Sam Bliss · 2009-08-04 08:29 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i UTF8Encode(WideChar(#200));This works. Look at the caption of two forms in screen shots. ---------------------------------------------------------------------- Vincent Snijders · 2009-08-04 08:33 I committed charactermapdlg.pas.diff.fixcrash.zip. I still don't understand what causes the crash then (You say it doesn't work). The lower half of the ansi table is empty and not used anymore for your code page. ---------------------------------------------------------------------- theo · 2009-08-04 08:34 @Sam Bliss: Your bug report was: QuoteWhen I use ANSI charactor map and put the mouse on 0000128..0000255, the dialog shows.                   Lazarus ============================================ Access voilation. Press OK to ignore and risk data corruption. Press Cancel to kill the program. ============================================ Does this still happen as of SVN version 21089    ? ---------------------------------------------------------------------- Sam Bliss · 2009-08-04 08:37 Sorry, I have not downloaded SVN version 21089. If you have, give me the oringal code of charactermapdlg.pas, I will test it. ---------------------------------------------------------------------- Vincent Snijders · 2009-08-04 08:42 You can get it from: http://svn.freepascal.org/cgi-bin/viewvc.cgi/trun k/ide/charactermapdlg.pas?revision=21089&root=lazarus ---------------------------------------------------------------------- Sam Bliss · 2009-08-04 08:43 I mean when you move you mouse on #$80..#$FE, the label under the string grid shows incorrect ASCII code. ---------------------------------------------------------------------- theo · 2009-08-04 08:44 But that was not your original bug report. You said it crashes. Does it still? ---------------------------------------------------------------------- Sam Bliss · 2009-08-04 08:50 I tried. See attached files. ---------------------------------------------------------------------- Vincent Snijders · 2009-08-04 08:51 What is wrong about that? What codepage do you use? ---------------------------------------------------------------------- Sam Bliss · 2009-08-04 08:54 cp936 (GB18030 Chinese Simplified) ---------------------------------------------------------------------- Sam Bliss · 2009-08-04 08:56 We must calculate these charactors from UCS-2 and convert to UTF-8, Convert ANSI to UTF-8 is not a good way (because the result is different if the codepage is different). ---------------------------------------------------------------------- theo · 2009-08-04 08:57 Quote from: Sam Bliss on August 04, 2009, 10:54:46 amcp936 (GB18030 Chinese Simplified) Looks correct: http://unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP936.TXT There's nothing usable from 0x81 to 0xff ---------------------------------------------------------------------- Vincent Snijders · 2009-08-04 09:00 Quote from: Sam Bliss on August 04, 2009, 10:56:36 amWe must calculate these charactors from UCS-2 and convert to UTF-8,That is what the unicode page does. QuoteConvert ANSI to UTF-8 is not a good way (because the result is different if the codepage is different). That is what the ansi tab does, which may not be what you need. In that case don't use it. ---------------------------------------------------------------------- Sam Bliss · 2009-08-07 09:02 And why the authors of Lazarus didn't add #0..#31 Because we don't need them? ---------------------------------------------------------------------- Vincent Snijders · 2009-08-07 09:11 I don't know, but I can think about some reasons: * they are usually hard to show. (how do you show #7)? * they have special meaning, for example #10 ---------------------------------------------------------------------- theo · 2009-08-07 09:13 Quote from: Sam Bliss on August 07, 2009, 11:02:20 amAnd why the authors of Lazarus didn't add #0..#31 Because we don't need them? These are control characters. Not printable: http://en.wikipedia.org/wiki/ASCII#ASCII_control_characters ---------------------------------------------------------------------- Sam Bliss · 2009-08-13 09:12 I downloaded the newest version Lazarus 0.9.27 r21212 FPC 2.3.1 i386-win32-win32/win64 The character map is also broken. ---------------------------------------------------------------------- Vincent Snijders · 2009-08-13 09:21 In what way is it broken? Please give a screenshot and tell us what you expect. Then we can confirm it or explain why your expectations are wrong. ---------------------------------------------------------------------- Sam Bliss · 2009-08-14 09:01 Screen shot here. ---------------------------------------------------------------------- Vincent Snijders · 2009-08-14 09:07 And what do you expect? #147 is not a valid ansichar in codepage 936, so it is correct. ---------------------------------------------------------------------- Sam Bliss · 2009-08-20 08:20 But we should shows "Latin-1 Supplement" characters in #127..#255. ---------------------------------------------------------------------- Vincent Snijders · 2009-08-20 08:32 Why should it show Latin-1 supplement charaters in your code page on the ansi tab. Latin-1 supplement characters are on the unicode code page. ---------------------------------------------------------------------- Sam Bliss · 2009-08-21 01:55 Oh, yes. And I suggest you add a search bar on the character map dialog. Because there are so many characters in Unicode Tab. We should find them quickly, right? Type the character, its Unicode or UTF-8 and its discribtion (Better support i18n).