https://github.com/alipay/fury Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} alipay / fury Public * Notifications * Fork 96 * Star 1.7k A blazing fast multi-language serialization framework powered by jit and zero-copy furyio.org License Apache-2.0 license 1.7k stars 96 forks Activity Star Notifications * Code * Issues 53 * Pull requests 4 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights alipay/fury This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 7 branches 14 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/a] Use Git or checkout with SVN using the web URL. [gh repo clone alipay] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @chaokunyang chaokunyang [Doc] Update JDK support doc (#992) ... ee756b1 Oct 8, 2023 [Doc] Update JDK support doc (#992) Update README.md ee756b1 Git stats * 518 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github [Java] add jdk 21 tests ci (#978) October 6, 2023 16:08 .vscode [Doc] Add c++ debugging doc (#448) June 5, 2023 23:57 bazel [Python] Support pyarrow6 bazel build (#859) August 7, 2023 20:12 ci [Java] fix jdk17/21 ci latest_jdk_tests skip (#989) October 8, 2023 14:26 docs Update java_object_graph_guide.md (#987) October 8, 2023 01:06 go [Doc] upgrade docs (#985) October 7, 2023 20:24 integration_tests bump version to 0.3.0-SNAPSHOT (#990) October 8, 2023 15:07 java bump version to 0.3.0-SNAPSHOT (#990) October 8, 2023 15:07 javascript bump version to 0.3.0-SNAPSHOT (#990) October 8, 2023 15:07 python bump version to 0.3.0-SNAPSHOT (#990) October 8, 2023 15:07 rust chore: apply license format (#783) July 31, 2023 17:46 src chore: apply license format (#783) July 31, 2023 17:46 .bazelrc [Python] Support pyarrow6 bazel build (#859) August 7, 2023 20:12 .flake8 chore: apply license format (#783) July 31, 2023 17:46 .gitignore Import js write performance (#569) July 9, 2023 14:55 BUILD [Python] Support arrow serialization in objgraph (#515) July 2, 2023 00:44 CODE_OF_CONDUCT.md [Doc] Create CODE_OF_CONDUCT.md (#761) July 25, 2023 21:22 CONTRIBUTING.md [Python] Support pyarrow6 bazel build (#859) August 7, 2023 20:12 LICENSE chore: apply license format (#783) July 31, 2023 17:46 README.md [Doc] Update JDK support doc (#992) October 8, 2023 22:57 SECURITY.md Update SECURITY.md (#818) August 1, 2023 16:06 WORKSPACE [C++] Add bazel C++ build support (#420) May 28, 2023 16:24 licenserc.toml [Go] add fury go util and ci tests (#950) October 5, 2023 19:45 View code [ ] Features Protocols Benchmarks Java Serialization Installation Java Python JavaScript Golang Quickstart Fury java object graph serialization Cross-language object graph serialization Row format Java Python Compatibility Schema Compatibility Binary Compatibility Security RoadMap How to Contribute Getting involved README.md [fury_github_banner] Build Status Slack Twitter Fury is a blazing fast multi-language serialization framework powered by jit(just-in-time compilation) and zero-copy, providing up to 170x performance and ultimate ease of use. https://furyio.org Features * Multiple languages: Java/Python/C++/Golang/Javascript. * Zero-copy: cross-language out-of-band serialization inspired by pickle5 and off-heap read/write. * High performance: A highly-extensible JIT framework to generate serializer code at runtime in an async multi-thread way to speed serialization, providing 20-170x speed up by: + reduce memory access by inline variable in generated code. + reduce virtual method invocation by inline call in generated code. + reduce conditional branching. + reduce hash lookup. * Multiple binary protocols: object graph, row format and so on. In addition to cross-language serialization, Fury also features at: * Drop-in replace Java serialization frameworks such as JDK/Kryo/ Hessian without modifying any code, but 100x faster. It can greatly improve the efficiency of high-performance RPC calls, data transfer and object persistence. * JDK serialization 100% compatible: JDK writeObject/readObject/ writeReplace/readResolve/readObjectNoData/Externalizable serialization API supported, JDK8~21 supported. * Supports shared and circular reference object serialization for golang. * Supports automatic object serialization for golang. Protocols Different scenarios have different serialization requirements. Fury designed and implemented multiple binary protocols for those requirements: * Cross-language object graph protocol: + Cross-language serialize any object automatically, no need for IDL definition, schema compilation and object to/from protocol conversion. + Support shared reference and circular reference, no duplicate data or recursion error. + Support object polymorphism. * Native java/python object graph protocol: highly-optimized based on type system of the language. * Row format protocol: a cache-friendly binary random access format, supports skipping serialization and partial serialization, and can convert to column-format automatically. New protocols can be added easily based on fury existing buffer, encoding, meta, codegen and other capabilities. All of those share same codebase, and the optimization for one protocol can be reused by another protocol. Benchmarks Different serialization frameworks are suitable for different scenarios, and benchmark results here are for reference only. If you need to benchmark for your specific scenario, make sure all serialization frameworks are appropriately configured for that scenario. Dynamic serialization frameworks support polymorphism and references, but they often come with a higher cost compared to static serialization frameworks, unless they utilize JIT techniques like Fury does. Because Fury generates code at runtime, it is recommended to warm up the system before collecting benchmark statistics. Java Serialization Title containing "compatible" represent schema compatible mode: support type forward/backward compatibility. Title without "compatible" represent schema consistent mode: class schema must be same between serialization and deserialization. Struct is a class with 100 primitive fields, MediaContent is a class from jvm-serializers, Sample is a class from kryo benchmark. [bench_serialize] [bench_serialize] [bench_serialize] [bench_serialize] [bench_deseriali] [bench_deseriali] [bench_deseriali] [bench_deseriali] See benchmarks for more benchmarks about type forward/backward compatibility, off-heap support, zero-copy serialization. Installation Java Nightly snapshot: sonatype https://s01.oss.sonatype.org/content/repositories/snapshots false true org.furyio fury-core 0.3.0-SNAPSHOT Release version: org.furyio fury-core 0.2.0 Python # Release version will be provided in the future. pip install pyfury --pre JavaScript npm install @furyjs/fury Golang Coming soon. Quickstart Here we give a quick start about how to use fury, see user guide for more details about java, cross language, and row format. Fury java object graph serialization If you don't have cross-language requirements, using this mode will have better performance. import io.fury.*; import java.util.*; public class Example { public static void main(String[] args) { SomeClass object = new SomeClass(); // Note that Fury instances should be reused between // multiple serializations of different objects. { Fury fury = Fury.builder().withLanguage(Language.JAVA) // Allow to deserialize objects unknown types, more flexible // but may be insecure if the classes contains malicious code. // .requireClassRegistration(false) .build(); // Registering types can reduce class name serialization overhead, but not mandatory. // If class registration enabled, all custom types must be registered. fury.register(SomeClass.class); byte[] bytes = fury.serialize(object); System.out.println(fury.deserialize(bytes)); } { ThreadSafeFury fury = Fury.builder().withLanguage(Language.JAVA) // Allow to deserialize objects unknown types, more flexible // but may be insecure if the classes contains malicious code. // .requireClassRegistration(false) .buildThreadSafeFury(); byte[] bytes = fury.serialize(object); System.out.println(fury.deserialize(bytes)); } { ThreadSafeFury fury = new ThreadLocalFury(classLoader -> { Fury f = Fury.builder().withLanguage(Language.JAVA) .withClassLoader(classLoader).build(); f.register(SomeClass.class); return f; }); byte[] bytes = fury.serialize(object); System.out.println(fury.deserialize(bytes)); } } } Cross-language object graph serialization Java import io.fury.*; import java.util.*; public class ReferenceExample { public static class SomeClass { SomeClass f1; Map f2; Map f3; } public static Object createObject() { SomeClass obj = new SomeClass(); obj.f1 = obj; obj.f2 = ofHashMap("k1", "v1", "k2", "v2"); obj.f3 = obj.f2; return obj; } // mvn exec:java -Dexec.mainClass="io.fury.examples.ReferenceExample" public static void main(String[] args) { Fury fury = Fury.builder().withLanguage(Language.XLANG) .withRefTracking(true).build(); fury.register(SomeClass.class, "example.SomeClass"); byte[] bytes = fury.serialize(createObject()); // bytes can be data serialized by other languages. System.out.println(fury.deserialize(bytes)); } } Python from typing import Dict import pyfury class SomeClass: f1: "SomeClass" f2: Dict[str, str] f3: Dict[str, str] fury = pyfury.Fury(ref_tracking=True) fury.register_class(SomeClass, "example.SomeClass") obj = SomeClass() obj.f2 = {"k1": "v1", "k2": "v2"} obj.f1, obj.f3 = obj, obj.f2 data = fury.serialize(obj) # bytes can be data serialized by other languages. print(fury.deserialize(data)) Golang package main import furygo "github.com/alipay/fury/fury/go/fury" import "fmt" func main() { type SomeClass struct { F1 *SomeClass F2 map[string]string F3 map[string]string } fury := furygo.NewFury(true) if err := fury.RegisterTagType("example.SomeClass", SomeClass{}); err != nil { panic(err) } value := &SomeClass{F2: map[string]string{"k1": "v1", "k2": "v2"}} value.F3 = value.F2 value.F1 = value bytes, err := fury.Marshal(value) if err != nil { } var newValue interface{} // bytes can be data serialized by other languages. if err := fury.Unmarshal(bytes, &newValue); err != nil { panic(err) } fmt.Println(newValue) } Row format Java public class Bar { String f1; List f2; } public class Foo { int f1; List f2; Map f3; List f4; } RowEncoder encoder = Encoders.bean(Foo.class); Foo foo = new Foo(); foo.f1 = 10; foo.f2 = IntStream.range(0, 1000000).boxed().collect(Collectors.toList()); foo.f3 = IntStream.range(0, 1000000).boxed().collect(Collectors.toMap(i -> "k"+i, i->i)); List bars = new ArrayList<>(1000000); for (int i = 0; i < 1000000; i++) { Bar bar = new Bar(); bar.f1 = "s"+i; bar.f2 = LongStream.range(0, 10).boxed().collect(Collectors.toList()); bars.add(bar); } foo.f4 = bars; // Can be zero-copy read by python BinaryRow binaryRow = encoder.toRow(foo); // can be data from python Foo newFoo = encoder.fromRow(binaryRow); // zero-copy read List f2 BinaryArray binaryArray2 = binaryRow.getArray(1); // zero-copy read List f4 BinaryArray binaryArray4 = binaryRow.getArray(3); // zero-copy read 11th element of `readList f4` BinaryRow barStruct = binaryArray4.getStruct(10); // zero-copy read 6th of f2 of 11th element of `readList f4` barStruct.getArray(1).getLong(5); RowEncoder barEncoder = Encoders.bean(Bar.class); // deserialize part of data. Bar newBar = barEncoder.fromRow(barStruct); Bar newBar2 = barEncoder.fromRow(binaryArray4.getStruct(20)); Python @dataclass class Bar: f1: str f2: List[pa.int64] @dataclass class Foo: f1: pa.int32 f2: List[pa.int32] f3: Dict[str, pa.int32] f4: List[Bar] encoder = pyfury.encoder(Foo) foo = Foo(f1=10, f2=list(range(1000_000)), f3={f"k{i}": i for i in range(1000_000)}, f4=[Bar(f1=f"s{i}", f2=list(range(10))) for i in range(1000_000)]) binary: bytes = encoder.to_row(foo).to_bytes() foo_row = pyfury.RowData(encoder.schema, binary) print(foo_row.f2[100000], foo_row.f4[100000].f1, foo_row.f4[200000].f2[5]) Compatibility Schema Compatibility Fury java object graph serialization support class schema forward/ backward compatibility. The serialization peer and deserialization peer can add/delete fields independently. We plan to add support cross-language serialization after meta compression are finished. Binary Compatibility We are still improving our protocols, binary compatibility are not ensured between fury major releases for now. it's ensured between minor version only. Please versioning your data by fury major version if you will upgrade fury in the future, see how to upgrade fury for further details. Binary compatibility will be ensured before fury 1.0. Security Static serialization are secure. But dynamic serialization such as fury java/python native serialization supports deserialize unregistered types, which provides more dynamics and flexibility, but also introduce security risks. For example, the deserialization may invoke init constructor or equals/hashCode method, if the method body contains malicious code, the system will be at risks. Fury provides a class registration option and enabled by default for such protocols, which allows only deserializing trusted registered types or built-in types. Do not disable class registration unless you can ensure your environment is secure. If this option is disabled, you are responsible for serialization security. You can configure io.fury.resolver.ClassChecker by ClassResolver#setClassChecker to control which classes are allowed for serialization. RoadMap * Meta compression, auto meta sharing and cross-language schema compatibility. * AOT Framework for c++/golang/rust to generate code statically. * C++/Rust object graph serialization support * Golang/Rust/NodeJS row format support * ProtoBuffer compatibility support * Protocols for features and knowledge graph serialization * Continuously improve our serialization infrastructure for any new protocols How to Contribute Please read the CONTRIBUTING guide. For ecosystem projects, please see https://github.com/fury-project Getting involved Estimated Platform Purpose Response Time GitHub Issues For reporting bugs and filing < 1 days feature requests. For collaborating with other Slack Fury users and latest < 2 days announcements about Fury. StackOverflow For asking questions about how < 2 days to use Fury. Zhihu Twitter Youtube Follow us for latest < 2 days announcements about Fury. WeChat Official Account WeChat Official Account Dingding (Wei Xin Gong Zhong Hao ) / Dingding Group < 2 days Group(Ding Ding Qun ) About A blazing fast multi-language serialization framework powered by jit and zero-copy furyio.org Topics javascript python java rust fast golang encoding serialization compression lightning cpp transfer zero-copy persistence marshalling jit rpc transport cross-language multiple-language Resources Readme License Apache-2.0 license Code of conduct Code of conduct Security policy Security policy Activity Stars 1.7k stars Watchers 25 watching Forks 96 forks Report repository Releases 5 v0.2.0 Latest Oct 8, 2023 + 4 releases Packages 0 No packages published Contributors 9 * @chaokunyang * @wangweipeng2 * @tisonkun * @hieu-ht * @pandalee99 * @rainsonGain * @leeco-cloud * @dependabot[bot] * @s31k31 Languages * Java 74.1% * Python 6.4% * C++ 4.6% * Cython 4.6% * TypeScript 4.1% * Go 3.4% * Other 2.8% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.