[SOLVED] Unicodestring type and Array of Widechar type - pros and cons? lazarus / 21-General ---------------------------------------------------------------------- Gizmo · 2021-01-22 10:09 Hi For a Windows application that commonly requires UTF16, I've been using Unicodestring types, and also array of widechar types. But I have to be entirely honest - I am a bit confused as to what the differences are, and when one would use one over the other? I've read https://wiki.freepascal.org/Character_and_string_types#UnicodeString that states "UnicodeStrings are reference counted, null-terminated arrays, but they are implemented as arrays of WideChars" and for WideChar it states "A variable of type WideChar, also referred to as UnicodeChar, is exactly 2 bytes in size and usually contains one Unicode code point (normally a character) in UTF-16 encoding." So, if I have an array that is Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  strA : array [0..99] of widechar; // strA can not be any larger than 100 bytes, i.e. 50 characters  strB : unicodestring;                  // strB is unrestricted in size begin  strA := 'HellÖ';  strB := 'HellÖ';end;  So in this example, under what circumstances would you opt for UnicodeString over array of WideChar, and visa versa, aside from not wanting to limit the amount of data that can go into the string which of course a fixed length array does? ---------------------------------------------------------------------- marcov · 2021-01-22 10:42 In Delphi/FPC string operations work on string types, not on static array buffers. It is exactly the same as ansistrings vs static arrays of (ansi)char.  One is an automated type with many operations defined for it, the other is an array with some minor magic so that assignment to a literal works, and for other operations,  you either need to convert it to unicodestring or to roll your own routines. ---------------------------------------------------------------------- MarkMLl · 2021-01-22 11:07 I think the point that the wiki is trying to make is that anything based on WideChar has characters of uniform size (16-bit etc.). Normal strings default to UTF-8 encoding, where characters have non- uniform size hence in the general case it's not safe to step through using sequential index values. If I am wrong in that I'd appreciate being corrected, since this is a topic with which I'm not entirely happy: and I know I'm not alone in that. MarkMLl ---------------------------------------------------------------------- Zvoni · 2021-01-22 13:27 Code: [Select]strA : array [0..99] of widechar; // strA can not be any larger than 100 bytes, i.e. 50 charactersSorry, but this doesn't make sense to me. It's an Array of 100 Wide-Characters (WideChars), so the correct thing to say is: It's 200 Bytes (=100 WideChars) ---------------------------------------------------------------------- lucamar · 2021-01-22 14:15 Quote from: MarkMLl on January 22, 2021, 12:07:53 pmI think the point that the wiki is trying to make is that anything based on WideChar has characters of uniform size (16-bit etc.). Normal strings default to UTF-8 encoding, where characters have non- uniform size hence in the general case it's not safe to step through using sequential index values. If I am wrong in that I'd appreciate being corrected, since this is a topic with which I'm not entirely happy: and I know I'm not alone in that. WideString/UnicodeString don't have uniform-sized characters either, but 16-bit code-points coded in UTF16. For most normal (read: Western) scripts that means that yes, a character can be represented as a single code-point, but that isn't true for other languages or if you use surrogates, composites (e.g. to represent tilded characters), etc. So "it's not safe to step through using sequential index values" is valid also for Wide/UnicodeString, though perhaps a little less so. ---------------------------------------------------------------------- Remy Lebeau · 2021-01-23 00:49 Quote from: lucamar on January 22, 2021, 03:15:56 pmWideString/UnicodeString don't have uniform-sized characters either, but 16-bit code-points coded in UTF16. More accurately, they have 16-bit code-units encoded in UTF-16.  A code-point and a code-unit are two different things. Quote from: lucamar on January 22, 2021, 03:15:56 pmFor most normal (read: Western) scripts that means that yes, a character can be represented as a single code-point, but that isn't true for other languages or if you use surrogates, composites (e.g. to represent tilded characters), etc. Again, code-units, not code-points.  Code-points are the character values that Unicode defines.  Code-units are how those code-points are encoded in the various UTF-X encodings (where X is the bit size of each code-unit). ---------------------------------------------------------------------- lucamar · 2021-01-23 03:29 Quote from: Remy Lebeau on January 23, 2021, 01:49:02 amMore accurately, they have 16-bit code-units encoded in UTF-16.  A code- point and a code-unit are two different things. You're right, o.c.. Sorry, one gets confused oftentimes with Unicode concepts and terminology ---------------------------------------------------------------------- PascalDragon · 2021-01-23 10:16 Quote from: lucamar on January 22, 2021, 03:15:56 pmWideString/UnicodeString don't have uniform-sized characters either, but 16-bit code-points coded in UTF16. For most normal (read: Western) scripts that means that yes, a character can be represented as a single code-point, but that isn't true for other languages or if you use surrogates, composites (e.g. to represent tilded characters), etc. You don't even need to look at non-western scripts, just look at the widespread use of Emojis nowadays. ---------------------------------------------------------------------- MarkMLl · 2021-01-23 11:46 Quote from: PascalDragon on January 23, 2021, 11:16:47 amYou don't even need to look at non-western scripts, just look at the widespread use of Emojis nowadays. I always find myself wanting to pluralise those as "emojim" for some reason. You're right of course, I'm skewed there by the fact I'm rarely likely to import something that includes them... although I should be able to handle them if they appear in a comment field. I've found myself using combining characters for output, but fortunately not for input... my above comment should still apply :-/ MarkMLl ---------------------------------------------------------------------- lucamar · 2021-01-23 13:39 Quote from: MarkMLl on January 23, 2021, 12:46:09 pmI've found myself using combining characters for output, but fortunately not for input... my above comment should still apply :-/ The networking moto should apply here too; paraphrasing: be liberal in what you accept even if you're strict in what you produce. ---------------------------------------------------------------------- MarkMLl · 2021-01-23 13:58 Quote from: lucamar on January 23, 2021, 02:39:41 pmThe networking moto should apply here too; paraphrasing: be liberal in what you accept even if you're strict in what you produce. Yes, agreed. MarkMLl