[HN Gopher] Java 18 is UTF-8 by Default
___________________________________________________________________
Java 18 is UTF-8 by Default
Author : ivanche
Score : 45 points
Date : 2022-04-14 07:28 UTC (15 hours ago)
(HTM) web link (thejavaguy.org)
(TXT) w3m dump (thejavaguy.org)
| mc4ndr3 wrote:
| Now your turn, Windows.
| jdjkfkf wrote:
| Windows is already Unicode and has been since Windows 2000. The
| internal representation however is UTF-16.
|
| This is in contrast to the article you are commenting on, where
| Java hasn't had a consistent default representation in Unicode
| until now.
| torginus wrote:
| Yeah but that's not the same. UTF-8 is vastly more efficient
| for european alphabets (and imo, ends up being a wash for
| non-european ones), while not pandering the seductive lie of
| UTF-16 that 1 character/rune == 2 bytes. It really shows up
| in memory usage of large pieces of text, speed of Regex etc.
|
| Unfortunately the NT codebase is old enough to have made the
| wrong choice regarding this.
|
| I wonder if such a a transition would be possible for .NET as
| well, because while technically you don't get access to the
| raw strings easily, there's tons of APIs that allow you to
| grab a pointer to the underlying memory in strings, which
| would immediately break if such a transition was made.
| pjmlp wrote:
| This is one of the scenarios where Java being more high
| level than .NET wins out.
|
| Additionally to what is being discussed, string compression
| and deduplication exists since several releases.
| WorldMaker wrote:
| .NET hasn't been so "low level" that it can't consider a
| different string implementation since "Core". There have
| been multiple explorations and discussions and
| considerations of moving .NET to using UTF-8 string
| representations internally (rather than UTF-16):
| https://github.com/dotnet/runtime/issues/6612
|
| It would potentially slow down platform invokes (C/C++
| DLLs) and COM calls, especially on Windows, but that's
| not a "low level" concern just as JNI's existence doesn't
| imply that Java is "low level".
| pjmlp wrote:
| People tend to forget CLR was developed to support all
| languages, including C and C++.
|
| As such, it is possible to access details in unsafe code
| via MSIL, than on the JVM side are not accessible.
|
| So naturally while nothing prevents them to eventually
| introduce such change, it might come with breaking
| changes to code that although legal, is playing with such
| low level details.
|
| As for the JNI, that is outside of what I am discussing
| of what is possible only with bytecodes.
| recursive wrote:
| It's certainly possible in .net, and has been seriously
| proposed.
|
| https://github.com/dotnet/runtime/issues/933
| pjmlp wrote:
| This is not the same as Java, because there the change is
| transparent, no need for additional types.
| torginus wrote:
| Yeah, but this is talking about introducing a separate,
| UTF-8 string type. .NET already has some support for
| UTF-8, for example ASP.NET and the new Json API uses
| UTF-8 directly, however the API is just gnarly.
| WorldMaker wrote:
| https://github.com/dotnet/runtime/issues/6612 is the more
| directly comparable proposal.
| torginus wrote:
| I'm not sure about compression, but deduplication has
| been in .NET since forever.
| recursive wrote:
| Are you suggesting that UTF-16 is not a Unicode encoding?
| pohl wrote:
| Both that -- and the opposite -- in different sentences.
| phonon wrote:
| Yeah? Then explain this (Windows 10, latest)
|
| https://imgur.com/gDGaQA2
| mananaysiempre wrote:
| Compatibility: http://archives.miloush.net/michkap/archive/
| 2005/09/17/46994.... (I'm not sure if the mention of it
| being implemented on the font level is up to date since
| locales have changed considerably on the way from the
| intensely buggy MUI packs on XP SP2 to seamless system-wide
| locale switching on 10.)
|
| Windows NT (the first implementation of Win32 to be
| released) was intended to be Unicode even before it was
| renamed from NT OS/2, and none of the native NT APIs accept
| strings in anything other than UTF-16.
| phonon wrote:
| It's US/English Windows, with a Japanese setting for all
| NON-unicode text. No reason whatsoever why that should
| affect how device manager renders code paths.
|
| https://imgur.com/teJx2m9
| lern_too_spel wrote:
| .NET StreamReader and StreamWriter already default to UTF-8.
| The only thing that is changing here is the default encoding
| when Java serializes a String to bytes and deserializes a
| String from bytes, which will both now match .NET. The .NET
| String constructor that takes a byte array still has the
| problem that Java has now with using the system default
| charset.
| rurban wrote:
| hopefully the POSIX libc's and compilers are next. runtime locale
| lookups are a nightmare and nobody uses them anymore.
| exabrial wrote:
| I'm curious if the internal representation will stay UTF-16?
| Someone smarter than I probably could say there's an advantage to
| using a fixed-length charset for internal purposes.
| pdpi wrote:
| UTF-16 isn't fixed length. It's sort of an extension of UCS-16,
| which _was_ fixed length, but couldn 't give you more than 64k
| codepoints. Internal representation is... complicated.
| java.lang.String uses UTF-16, but .class files use UTF-8.
| pjmlp wrote:
| And since Java 9 there is some compression going on.
| colejohnson66 wrote:
| Nitpick: UCS counts the _bytes_ , not the bits. So UTF-16,
| but limited to the BMP, is UCS-2, and UTF-32 is UCS-4.
| pdpi wrote:
| Oooops. You're absolutely right.
| golergka wrote:
| UTF16 is not fixed length, it has surrogate pairs.
| bjoli wrote:
| The java internal representation previously had a look up table
| for each Nth char (let's say 128, which is a nice round
| number), and if the indexes increased by 128 between every 128
| chars it simply did a linear lookup. For most text this means
| constant time access, instead of a linear search through the
| 128 chars.
|
| I don't think the benefits against a utf-8 system with a LUT
| would be very big, but as a retrofitted system (which it was)
| it is not bad. Pretty compact with mostly constant time access.
| pjmlp wrote:
| The internal representation isn't pure UTF-16 for quite some
| time now.
|
| https://openjdk.java.net/jeps/254
| alkonaut wrote:
| I love how they are ready to change behavior of existing code and
| require opting into a compat mode. I wish this way of fixing
| things properly while having _opt-in_ compat was more common. Too
| often we see aversion towards breaking changes for no obvious
| reason. If you don't want things to break then don't upgrade
| things.
| exabrial wrote:
| Agree. Java also has a great history of providing off-ramps for
| legacy code. When JDK modules came out, they allowed disabling
| of the system. This served an important purpose to allow older
| codebases to upgrade to newer JDK versions that had important
| security fixes while buying timing for those apps to age out of
| existence. Always a pleasure working with the jdk.
| Beltalowda wrote:
| I have a lot of little programs that are more or less finished:
| "does what it needs to do and there are no bugs". Some are
| public, some are not. I wrote them once - sometimes years ago -
| and they still work fine today.
|
| I really like it that code I write today still works in five or
| ten years without mucking about to "keep up to date". "Don't
| upgrade things" isn't as easy as you say: how do you get an old
| version of Ruby? How do you get an old browser? Do you even
| _want_ that?
|
| Opt-in should be easy and painless, probably even the default
| for _new_ projects (through some build file or whatnot; I 'm
| not very familiar with Java), but I think making existing
| projects work has a lot of value.
___________________________________________________________________
(page generated 2022-04-14 23:03 UTC)