Subj : Re: not all is lost but far too much for far too long To : Ozz Nixon From : Rob Swindell Date : Wed Jul 03 2019 02:44 pm Re: Re: not all is lost but far too much for far too long By: Ozz Nixon to Maurice Kinal on Fri Jun 28 2019 09:23 pm > On 2019-06-28 02:01:09 +0000, Maurice Kinal -> Torsten Bamberg said: > > FTN Header versus actual message body conveying Unicode. > > When I telnet to a SQL server that speaks Unicode only, it always > returns the following characters (pascal): #239#187#191 Using telnet to connect to services that don't speak Telnet is generally a bad idea. Use netcat (nc) instead. > When I telnet to a web page that speaks Unicode, it too returns > #239#187#191 plus the etc. > > > So... would it not stand true that systems that are posting UTF8 do the > same introduction on the message body? Then authors *know* it > potentially has Unicode and leave it damn well alone, and also parse it > based upon UTF8 instead of 8bit char... It's an idea. But that's not how *other* charsets/encodings work and certainly not how MIME-encoded messages (e.g. email) works - header fields are used instead. > This is how I am coding things here, just based upon NexusSQL, > PremierSQL, MS SQL, Apache and Nexus Web Service. I do not have access > to my Oracle box nor the MySQL 5 server to see if they do the same > during the initial connection negotiation(s). > > A quick google: It's the utf8 byte order mark. That's actually a misnomer (there is not "byte order" in UTF-8). The actual unicode code point is Zero Width No-Break Space: https://www.compart.com/en/unicode/U+FEFF > Some editors save the > BOM inside the file (in order to be used as a header) which regularly > causes confusion because it is optional. > > So, if we wanted to help enforce at a reader (or even tosser level) how > to handle, I would offer this up as a required BOM to the message body > that is UTF8. And why is that better than a header field ("control paragraph" as defined in FTS-5003) which indicates UTF-8? .