https://github.com/AugustNagro/utf8.java Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Customer Stories + White papers, Ebooks, Webinars + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. {{ message }} AugustNagro / utf8.java Public * Notifications * Fork 6 * Star 50 Vectorized UTF-8 Validation for Java 50 stars 6 forks Activity Star Notifications * Code * Issues 2 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights AugustNagro/utf8.java This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/A] Use Git or checkout with SVN using the web URL. [gh repo clone August] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @AugustNagro AugustNagro Merge pull request #6 from LifeIsStrange/master ... 2619ea9 Nov 25, 2021 Merge pull request #6 from LifeIsStrange/master replace javadoc dead link 2619ea9 Git stats * 10 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time src/main Merge benchmarks into one class September 26, 2021 21:26 .gitignore update gitignore September 16, 2021 11:19 128-bit-jdk-comparison.png Static lookup tables with sealed interface, and updated benchmarks. September 13, 2021 18:55 256-bit-jdk-comparison.png Static lookup tables with sealed interface, and updated benchmarks. September 13, 2021 18:55 README.md replace dead link November 19, 2021 15:48 pom.xml Static lookup tables with sealed interface, and updated benchmarks. September 13, 2021 18:55 results-2020-12-20.png first March 1, 2021 21:14 results-2021-3-1.png first March 1, 2021 21:14 results-jdk16-2021-09-13.png Static lookup tables with sealed interface, and updated benchmarks. September 13, 2021 18:55 results-panama-latest-2021-09-13.png Static lookup tables with sealed interface, and updated benchmarks. September 13, 2021 18:55 View code utf8.java Verify Correctness Running Benchmarks Performance JDK 16 (16 March 2021) vs Latest Panama Build (13 September 2021) Conclusion README.md utf8.java Vectorized UTF-8 validation & benchmarks, written in Java. Based on the paper by John Keiser and Daniel Lemire, with minor modifications. Verify Correctness Make sure to have Java 17 or greater installed. Then execute: mvn compile assembly:single && \ java --enable-preview --add-modules jdk.incubator.vector \ -jar target/utf8.java-1.0-SNAPSHOT-jar-with-dependencies.jar [optional list of space-delineated file paths] With no arguments, this will run the UTF-8 validator on 4 source files: * twitter.json: 631_515 bytes, frequent multi-byte characters. Taken from here. * utf8-demo.txt: 13_459 bytes, many special / tricky utf8 characters. From w3.org. * utf8-demo-invalid.txt: same as utf8-demo.txt, but with one error. * 20k.txt: 3.8 MB, all ascii. Running Benchmarks mvn verify && java -jar target/benchmarks.jar The JMH benchmarks use the same 4 test files mentioned above, at 3 vector lengths: 128 bit, 256, and 512. Most likely your hardware does not support 512 bit vectors, so these benchmarks fallback to the slow array-based implementation.jdk_decode uses the JDK's new String(buf, UTF_8). This constructor produces a new String in addition to validation, but is good enough for a baseline. Performance Throughput for twitter.json: new String(buf, Utf8.validate(buf, new simdjson::validate_utf8 UTF_8) LookupTables256()) (str, len) .8 GB/sec 3.52 GB/sec 24 GB/sec * The JDK algorithm is very optimized, and uses intrinsics to check negatives (for the ASCII shortcut) and to elide array bound checks. * In the vectorized algorithm, 256 bit vectors currently perform best. We cannot go smaller than 128 bit, since nibbles (4 bits) are used to select from the lookup tables. JDK 16 (16 March 2021) vs Latest Panama Build (13 September 2021) 256-bit-jdk-comparison In 181 days, performance has increased up to 127 fold! 128-bit-jdk-comparison Conclusion * The Vector api is expressive and a pleasure to use. Performance is getting better. * Abstracting over ISA and even vector Shape is incredible for portability, given how fragmented vector instruction sets are. * The dissonance between Vector and ByteVector is a little annoying. * If I had benchmarked iteratively while developing, I could've discovered the causes of slowdown sooner. * I wish there was a (documented) debug/logging flag. * Vector::selectInto is awesome for lookup tables. * The project's JavaDoc is one of the best introductions to vectorization on the internet. * Debugging works great! I do wish we could make Vector::toString print hex instead of base 10 by default. * Would have been nice if performance was a success story, but failure is educational. * I think someone could implement simd-json in Java if they wanted to. Would it be fast? At least not for now. About Vectorized UTF-8 Validation for Java Resources Readme Activity Stars 50 stars Watchers 8 watching Forks 6 forks Report repository Releases No releases published Packages 0 No packages published Contributors 3 * @AugustNagro AugustNagro August Nagro * @amCap1712 amCap1712 Kartik Ohri * @LifeIsStrange LifeIsStrange Languages * Java 100.0% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.