[HN Gopher] How to Create a UTF-16 Surrogate Pair by Hand, with ...
       ___________________________________________________________________
        
       How to Create a UTF-16 Surrogate Pair by Hand, with Python
        
       Author : chubot
       Score  : 11 points
       Date   : 2023-06-17 16:26 UTC (6 hours ago)
        
 (HTM) web link (www.oilshell.org)
 (TXT) w3m dump (www.oilshell.org)
        
       | poorlyknit wrote:
       | The biggest takeaway/shock for me in this article is the fact
       | that JSON string literals can't contain escaped characters
       | outside the Basic Multilingual Plane (i.e. whose code points are
       | greater than U+FFFF).
       | 
       | Quoting RFC 8259:                  To escape an extended
       | character that is not in the Basic Multilingual        Plane, the
       | character is represented as a 12-character sequence,
       | encoding the UTF-16 surrogate pair.
       | 
       | So in order to encode U+1F914 THINKING FACE you can either do
       | {"text": "<thinking face>"}
       | 
       | (Emoji omitted bc of HN) or                 {"text":
       | "\ud83e\udd14"}
       | 
       | but not                 {"text": "\u1f914"}.
       | 
       | This seems to be a relic from ECMAScript which (iirc, only
       | skimmed it) assumes UCS-2/UTF-16. From that perspective it makes
       | a lot of sense but it steel feels a little icky to me having
       | surrogates referenced in a standard that is supposed to be UTF-8
       | only :)
        
         | chubot wrote:
         | Right exactly. The conventional ways of writing it would be
         | \U0001F914 -- must be exactly 8 digits              \u{1f914}
         | -- 1 to 6 digits
         | 
         | Almost all languages have these now, but JavaScript didn't when
         | JSON was invented.
         | 
         | The "J8 strings" extension mentioned technically only needs \a
         | syntax for bytes, which is \yff since \xff is (oddly) a synonym
         | for \u00ff and thus unsuitable.
         | 
         | But I also want to add \u{123456} because it allows people to
         | move away from the weird UTF-16 legacy in a UTF-8 format.
        
         | TRiG_Ireland wrote:
         | As JSON must be in Unicode (I think the latest RFC restricts it
         | to UTF-8), there's no actual _need_ to escape emoji. But if you
         | do need to (perhaps you're in an ASCII-only environment), you
         | have to do it that way, yes. Weird. (I asked once, on Stack
         | Overflow, why this was the case, and was merely told that it
         | was historical reasons, which matches what you surmised: it's a
         | relic.)
        
       ___________________________________________________________________
       (page generated 2023-06-17 23:02 UTC)