[HN Gopher] JEP 400: UTF-8 by Default
___________________________________________________________________
JEP 400: UTF-8 by Default
Author : znpy
Score : 82 points
Date : 2023-07-26 16:06 UTC (6 hours ago)
(HTM) web link (openjdk.org)
(TXT) w3m dump (openjdk.org)
| bawolff wrote:
| Now do javascript ;)
|
| [I mean,make the strings use utf-8 instead of utf-16, which is a
| bit different than TFA]
| Pet_Ant wrote:
| If I had my druthers, String would be an interface with UTF-8,
| UTF-16, and UTF-32 (and UTF-7 on April Fool's Day) implementation
| classes. Then add Byte1, Byte2, and Byte4 as _unsigned_
| primitives. Maybe have a wrapper class CodePoint to allow
| abstracting over all of them.
| marginalia_nu wrote:
| Seems like it would make string serialization even more
| expensive than it already is (and it is).
| josho wrote:
| I agree but wonder if JVM optimizations could do away with the
| need to complicate the programmer experience.
| colejohnson66 wrote:
| Java can't even bother adding _unsigned_ integers or integer
| types smaller than 32-bits. How long have we been waiting for
| non-boxed primitives and runtime generics (Valhalla)? C# has
| had all (except generics) since day one. New "char" types
| aren't happening anywhere in the near future, unfortunately.
| marginalia_nu wrote:
| > integer types smaller than 32-bits
|
| ... other than short and byte, what are you missing? nibble?
|
| https://docs.oracle.com/javase/tutorial/java/nutsandbolts/da.
| ..
| Delk wrote:
| > or integer types smaller than 32-bits
|
| Java's got byte (8 bits) and short (16 bits)? (Still signed,
| though, of course.)
|
| Unless those are actually padded to 32 bits in the VM or
| something.
| efaref wrote:
| > Unless those are actually padded to 32 bits in the VM or
| something.
|
| They are.
| throwawaymobule wrote:
| Anyone interested in going whole-hog with this, forking Java, and
| making char utf8 too?
|
| I'm guessing it's not worth it, even aside from legal costs.
| kevin_thibedeau wrote:
| Already done: J++ begat C#.
| ElectricalUnion wrote:
| It is not worth it because:
|
| * If you're not really serious about using strings, why would
| you care?
|
| * If you're really serious about actually doing things with
| Strings, you are gonna need to reimplement International
| Components for Unicode (or something similar in scope), and you
| have a free, libre and already working one for the "UTF-16
| String Java" already. You don't have a working one for your
| custom fork with custom String handling.
| tialaramex wrote:
| > making char utf8 too?
|
| What would that even mean? A forked Java's _strings_ could
| insist their implementation is UTF-8 encoded bytes, but that 's
| strings, you're talking about char. Do you want char just to be
| a byte, like in C ? But Java already has a byte type.
| LukeShu wrote:
| In my forked Java, a char would be UTF-32, like in Go. (And
| String would be UTF-8 encoded bytes, as you say.)
| ElectricalUnion wrote:
| A 32 bit "wide char" is both very wasteful under normal use
| and mostly useless for grapheme cluster (what business
| people probably actually mean by "character") handling.
| kgeist wrote:
| In Go, strings are UTF8, but when you iterate over UTF8
| characters like this: for _, c := range
| str { }
|
| variable "c" is 32-bit. So it's not really wasteful, it
| fits in a register. It can be wasteful if you declare a
| slice of characters, like []rune, but I don't remember
| ever seeing it in practice.
| erik_seaberg wrote:
| Most of the things you could do with a single codepoint
| aren't valid unless you search for a group of
| unicode.IsMark(c) and process them together with the
| previous codepoint as a short string. Now I'm concerned
| that I don't see that in the standard library.
| aardvark179 wrote:
| In Java strings are stored in the heap either as compact
| strings for cases where everything fits in 8 bits, or as
| UTF-16. Iterating over them using chars will give you
| those 16 bit things, but there are other methods you can
| use to iterate over them as code points, just like you
| can in Go. The old char behaviour may not be what you
| really want, but it can't be changed without breaking
| existing code.
| Animats wrote:
| Nice.
|
| Better late than never.
| morelisp wrote:
| I guess I'll be happy to see this, but
|
| > Specify UTF-8 as the default charset without providing any
| means to change it -- The compatibility impact of this change
| would be too high.
|
| Nah, nobody is getting this correct right now. You should've
| swung for the fences.
| invalidname wrote:
| This is pretty old news by now... But yes, it was long overdue.
| vips7L wrote:
| Delivered almost 3 releases ago.
| cesarb wrote:
| If you follow the LTS releases, it will be delivered only on
| the next release, expected around two months from now.
| schemescape wrote:
| Am I reading correctly that characters are 16 bits? If so, is
| this just for the boundaries (e.g. file system, web)?
| za3faran wrote:
| In addition to what the other reply posted, Java has had
| compact strings for a while now[1], which uses 8-bit `byte`
| arrays for `String`s, and comes in handy for ISO-8859-1/Latin-1
| strings to use a single byte per character.
|
| [1] https://openjdk.org/jeps/254
| cesarb wrote:
| One annoyance with that approach is that a single non-latin-1
| codepoint is enough to double the size of the String; if it
| were an UTF-8 String, it would only add a couple of extra
| bytes.
| TacticalCoder wrote:
| Java predates Unicode 3.1 (I think I got the version correct:
| but basically when Java was created, Unicode had less than
| 65536 codepoints). So Java had 16 bits chars from the get go
| and to this day is still backward compatible with earlier Java
| code.
|
| You can also ask to get the "codepoint", which returns you an
| int.
|
| But, yup, the Java _char_ primitive is 16 bits.
|
| It's a SNAFU hard to understate. SNAFU doesn't even being to
| describe it. But to be honest Unicode in itself is probably the
| biggest SNAFU of our entire field.
| iforgotpassword wrote:
| It's the same with the windows API
| Quekid5 wrote:
| > But to be honest Unicode in itself is probably the biggest
| SNAFU of our entire field.
|
| Explain.
|
| EDIT: Oh! I construed this as a negative sentiment about
| Unicode, but perhaps it was just: "Yeah, it's complicated
| because language is complicated". Thanks, responders!
| EnergyAmy wrote:
| It was always going to be messy to create something that
| covered all of the writing systems in the world, but things
| like this didn't help, and is probably what they're
| referring to:
|
| https://en.wikipedia.org/wiki/Han_unification
| tialaramex wrote:
| I can't explain what the previous poster was thinking, but
| Unicode is frustrating to work with. The problem is, that
| frustration just replicates the reality of the human
| writing systems Unicode captures. Nobody invented weird
| nonsense to make your life harder _in Unicode_ , millions
| of humans developed a variety of weird nonsense to make
| their lives easier, over thousands of years and Unicode
| captures most of it.
|
| Case for example, why the heck would there be exactly two
| versions of A, one of which usually only appears at the
| start of words, and not always? Other people's squiggles
| don't do that, but notably several European writing systems
| (Latin and Cyrillic) do. People who use those systems think
| ABCDE and abcde are somehow "similar" even though they
| don't seem very similar at all. Then they're surprised
| other people don't do that.
| cryptonector wrote:
| Exactly, there's nothing really wrong with Unicode -- it's
| complicated, yes, but only because human scripts are.
| cryptonector wrote:
| Unicode is not a SNAFU. Though the UC has made mistakes here
| and there, the complexity of Unicode is only a mirror of the
| complexity of _human scripts_.
|
| Please stop taking dumps on Unicode. It does nothing for
| actually getting it implemented and implemented well.
| meepmorp wrote:
| Seriously, the zoo of text encodings that reigned before
| widespread Unicode uptake was a pain the ass once you had
| data that went beyond 7 bits, especially for non-Roman
| scripts.
| josefx wrote:
| > the complexity of Unicode is only a mirror of the
| complexity of human scripts.
|
| Oh please, Unicode didn't start out as an encoding covering
| human scripts. It started out as an encoding to end all
| existing encodings and that was a shit show with all kinds
| of weird and non printable symbols to begin with even
| before it started to add combining emojis into the mix.
___________________________________________________________________
(page generated 2023-07-26 23:01 UTC)