https://github.com/golang/go/discussions/63397 Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} golang / go Public * Notifications * Fork 17.2k * Star 115k * Code * Issues 5k+ * Pull requests 384 * Discussions * Actions * Projects 4 * Wiki * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Wiki * Security * Insights encoding/json/v2 #63397 dsnet announced in Discussions encoding/json/v2 #63397 @dsnet dsnet Oct 5, 2023 * 31 comments * 124 replies Return to top Discussion options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. [635] dsnet Oct 5, 2023 Collaborator - This is a discussion intended to lead to a formal proposal. This was written with input from @mvdan, @johanbrandhorst, @rogpeppe, @chrishines, @rsc. Background The widely-used "encoding/json" package is over a decade old and has served the Go community well. Over time, we have learned much about what works well and what does not. Its ability to marshal from and unmarshal into native Go types, the ability to customize the representation of struct fields using Go struct tags, and the ability of Go types to customize their own representation has proven to be highly flexible. However, many flaws and shortcomings have also been identified over time. Addressing each issue in isolation will likely lead to surprising behavior when non-orthogonal features interact poorly. This discussion aims to take a cohesive and comprehensive look at "json" and to propose a solution to support JSON in Go for the next decade and beyond. Improvements may be delivered either by adding new functionality to the existing "json" package and/or by introducing a new major version of the package. To guide this decision, let us evaluate the existing "json" package in the following categories: 1. Missing functionality 2. API deficiencies 3. Performance limitations 4. Behavioral flaws Missing functionality There are quite a number of open proposals, where the most prominent feature requests are: * The ability to specify custom formatting of time.Time (#21990) * The ability to omit specific Go values when marshaling (#11939, # 22480, #50480, #29310, #52803, #45669) * The ability to marshal nil Go slices and maps as empty JSON arrays and objects (#37711, #27589) * The ability to inline Go types without using Go embedding (#6213) Most feature requests could be added to the existing "json" package in a backwards compatible way. API deficiencies The API can be sharp or restrictive: 1. There is no easy way to correctly unmarshal from an io.Reader. Users often write json.NewDecoder(r).Decode(v), which is incorrect since it does not reject trailing junk at the end of the payload (#36225). 2. Options can be set on the Encoder and Decoder types, but cannot be used with the Marshal and Unmarshal functions. Similarly, types implementing the Marshaler and Unmarshaler interfaces cannot make use of the options. There is no way to plumb options down the call stack (#41144). 3. The functions Compact, Indent, and HTMLEscape write to a bytes.Buffer instead of something more flexible like a []byte or io.Writer. This limits the usability of those functions. These deficiencies could be fixed by introducing new API to the existing "json" package in a backwards compatible way at the cost of introducing multiple different ways of accomplishing the same tasks in the same package. Performance limitations The performance of the standard "json" package leaves much to be desired. Setting aside internal implementation details, there are externally visible APIs and behaviors that fundamentally limit performance: 1. MarshalJSON: The MarshalJSON interface method forces the implementation to allocate the returned []byte. Also, the semantics require that the "json" package parse the result both to verify that it is valid JSON and also to reformat it to match the specified indentation. 2. UnmarshalJSON: The UnmarshalJSON interface method requires that a complete JSON value be provided (without any trailing data). This forces the "json" package to parse the JSON value to be unmarshaled in its entirety to determine when it ends before it can call UnmarshalJSON. Afterwards, the UnmarshalJSON method itself must parse the provided JSON value again. If the UnmarshalJSON implementation recursively calls Unmarshal, this leads to quadratic behavior. As an example, this is the source of dramatic performance degradation when unmarshaling into spec.Swagger (kubernetes/kube-openapi#315). 3. Encoder.WriteToken: There is no streaming encoder API. A proposal has been accepted but not implemented (#40127). The proposed API symmetrically matches Decoder.Token, but suffers from the same performance problems (see next point). 4. Decoder.Token: The Token type is an interface, which can hold one of multiple types: Delim, bool, float64, Number, string, or nil. This unfortunately allocates frequently when boxing a number or string within the Token interface type (#40128). 5. Lack of streaming: Even though the Encoder.Encode and Decoder.Decode methods operate on an io.Writer or io.Reader, they buffer the entire JSON value in memory. This hurts performance since it requires a second pass through the JSON. In theory, only the largest JSON token (i.e., a JSON string or number) should ever need to be buffered (#33714, #7872, #11046). Limitations 1 and 2 can be resolved by defining new interface methods that operate on a streaming encoder or decoder. However, type-defined streaming methods are blocked on limitation 3 and 4, which requires having an efficient, streaming encoder and decoder API (#40127, #40128). Even if an efficient streaming API is provided, the "json" package itself would still be constrained by limitation 5, where it does not operate in a truly streaming manner under the hood (#33714, #7872, # 11046). The "json" package should operate in a truly streaming manner by default when writing to or reading from an io.Writer or io.Reader. Buffering the entire JSON value defeats the point of using an io.Reader or io.Writer. Use cases that want to avoid outputting JSON in the event of an error should call Marshal instead and only write the output if the error is nil. Unfortunately, the "json" package cannot switch to streaming by default since this would be a breaking behavioral change, suggesting that a v2 "json" package is needed to accomplish this goal. Behavioral flaws Various behavioral flaws have been identified with the "json" package: 1. Improper handling of JSON syntax: Over the years, JSON has seen increased amounts of standardization (RFC 4627, RFC 7159, RFC 7493, and RFC 8259) in order for JSON-based protocols to properly communicate. Generally speaking, the specifications have gotten more strict over time since loose guarantees lead to implementations disagreeing about the meaning of a particular JSON value. + The "json" package currently allows invalid UTF-8, while the latest internet standard (RFC 8259) requires valid UTF-8. The default behavior should at least be compliant with RFC 8259, which would require that the presence of invalid UTF-8 to be rejected as an error. + The "json" package currently allows for duplicate object member names. RFC 8259 specifies that duplicate object names result in unspecified behavior (e.g., an implementation may take the first value, last value, ignore it, reject it, etc.). Fundamentally, the presence of a duplicate object name results in a JSON value without any universally agreed upon semantic (#43664). This could be exploited by attackers in security applications and has been exploited in practice with severe consequences. The default behavior should err on the side of safety and reject duplicate names as recommended by RFC 7493. While the default behavior should be more strict, we should also provide an option for backwards compatibility to opt-in to the prior behavior of allowing invalid UTF-8 and/or allowing duplicate names. 2. Case-insensitive unmarshaling: When unmarshaling, JSON object names are paired with Go struct field names using a case-insensitive match (#14750). This is a surprising default, a potential security vulnerability, and a performance limitation. It may be a security vulnerability when an attacker provides an alternate encoding that a security tool does not know to check for. It is also a performance limitation since matching upon a case-insensitive name cannot be performed using a trivial Go map lookup. 3. Inconsistent calling of type-defined methods: Due to "json" and its use of Go reflection, the MarshalJSON and UnmarshalJSON methods cannot be called if the underlying value is not addressable (#22967, #27722, #33993, #55890). This is surprising to users when their declared MarshalJSON and UnmarshalJSON methods are not called when the underlying Go value was retrieved through a Go map, interface, or another non-addressable value. The "json" package should consistently and always call the user-defined methods regardless of addressability. As an implementation detail, non-addressable values can always be made addressable by temporarily boxing them on the heap. This could arguably be considered a bug and be fixed in the current "json" package. However, previous attempts at fixing this resulted in the changes being reverted because it broke too many targets implicitly depending on the inconsistent calling behavior. 4. Inconsistent merge semantics: When unmarshaling into a non-empty Go value, the behavior is inconsistent about whether it clears the target, resets but reuses the target memory, and/or whether it merges into the target (#27172, #31924, #26946). Most oddly, when unmarshaling into a non-nil Go slice, the unused elements between the length and capacity are merged into without being zeroed first (#21092). The merge semantics of "json" came about organically without much thought given to a systematic approach to merging, leading to fragmented and inconsistent behavior. 5. Inconsistent error values: There are three classes of errors that can occur when handling JSON: + Syntactic error: The input does not match the JSON grammar (e.g., an improperly escaped JSON string). + Semantic error: The input is valid JSON, but there is a type-mismatch between the JSON value and the Go value (e.g., unmarshaling a JSON bool into a Go struct). + I/O error: A failure occurred writing to or reading from an io.Writer or io.Reader. This class of errors never occurs when marshaling to or unmarshaling from a []byte. The "json" package is currently inconsistent about whether it returns structured or unstructured errors. It is currently impossible to reliably detect each class of error. These behavioral flaws of "json" cannot be changed without being a breaking change. Options could be added to specify different behavior, but that would be unfortunate since the desired behavior is not the default behavior. Changing the default behavior suggests the need for a v2 "json" package. Proposal The analysis above suggests that a new major version of the "json" package is necessary and worthwhile. In this section, we propose a rough draft of what a new major version could look like. Henceforth, we will refer to the existing "encoding/json" package as v1, and a hypothetical new major version as v2. This is a draft proposal as the proposed API and behavior is subject to change based on community discussion. Goals Let us define some goals for v2: * Mostly backwards compatible: If possible, v2 should aim to be mostly compatible with v1 in terms of both API and default behavior to ease migration. For example, the Marshal and Unmarshal functions are the most widely used declarations in v1. It is sensible for equivalent functionality in v2 to be named the same and have mostly the same signature. Behaviorally, we should aim for 95% to 99% backwards compatibility. We do not aim for 100% compatibility since we want the freedom to break certain behaviors that are now considered to have been a mistake. * More correct: JSON standardization has become increasingly more strict over time due to interoperability issues. The default serialization should prioritize correctness. * More performant: JSON serialization is widely used and performance gains translate to real-world resource savings. However, performance is secondary to correctness. For example, rejecting duplicate object names will hurt performance, but is the more correct behavior to have. * More flexible: We should aim to provide the most flexible features that address most usages. We do not want to overfit v2 to handle every possible use case. The provided features should be orthogonal in nature such that any combination of features results in as few surprising edge cases as possible. * Easy to use (hard to misuse): The API should aim to make the common case easy and the less common case at least possible. The API should avoid behavior that goes contrary to user expectation, which may result in subtle bugs. * Avoid unsafe: JSON serialization is used by many internet-facing Go services. It is paramount that untrusted JSON inputs cannot result in memory corruption. Consequently, standard library packages generally avoid the use of package "unsafe" even if it could provide a performance boost. We aim to preserve this property. + There are many community forks or reimplementations of v1 "json". While they provide impressive performance gains, they cannot be adopted into the standard library on the basis of their extensive use of package "unsafe". The 2021 Go Developer Survey shows that the assurance of reliability and security is a higher priority than CPU or memory performance. Overview JSON serialization can be broken down into two primary components: * syntactic functionality that is concerned with processing JSON based on its grammar, and * semantic functionality that determines the meaning of JSON values as Go values and vice-versa. We use the terms "encode" and "decode" to describe syntactic functionality and the terms "marshal" and "unmarshal" to describe semantic functionality. We aim to provide a clear distinction between functionality that is purely concerned with encoding versus that of marshaling. For example, it should be possible to encode a stream of JSON tokens without needing to marshal a concrete Go value representing them. Similarly, it should be possible to decode a stream of JSON tokens without needing to unmarshal them into a concrete Go value. In v2, we propose that there be two packages: "jsontext" and "json". The "jsontext" package is concerned with syntactic functionality, while the "json" package is concerned with semantic functionality. The "json" package will be implemented in terms of the "jsontext" package. In order for "json" to marshal from and unmarshal into arbitrary Go values, it must have a dependency on the "reflect" package. In contrast, the "jsontext" package will have a relatively light dependency tree and be suitable for applications (e.g., TinyGo, GopherJS, WASI, etc.) where binary bloat is a concern. block-diagram This diagram provides a high-level overview of the v2 API. Purple blocks represent types, while blue blocks represent functions or methods. The direction of the arrows represent the approximate flow of data. The bottom half (as implemented by the "jsontext" package) of the diagram contains functionality that is only concerned with syntax, while the upper half (as implemented by the "json" package) contains functionality that assigns semantic meaning to syntactic data handled by the bottom half. The "jsontext" package The jsontext package provides functionality to process JSON purely according to the grammar. This package will have a small dependency tree such that it results in minimal binary bloat. Most notably, it does not depend on Go reflection. Overview The basic API consists of the following: package jsontext // "encoding/json/jsontext" type Encoder struct { /* no exported fields */ } func NewEncoder(io.Writer, ...Options) *Encoder func (*Encoder) WriteToken(Token) error func (*Encoder) WriteValue(Value) error type Decoder struct { /* no exported fields */ } func NewDecoder(io.Reader, ...Options) *Decoder func (*Decoder) PeekKind() Kind func (*Decoder) ReadToken() (Token, error) func (*Decoder) ReadValue() (Value, error) func (*Decoder) SkipValue() error type Kind byte type Token struct { /* no exported fields */ } func (Token) Kind() Kind type Value []byte func (Value) Kind() Kind Values and Tokens The primary data types for interacting with JSON are Kind, Value, and Token. The Kind is an enumeration that describes the kind of a value or token. // Kind represents each possible JSON token kind with a single byte, // which is the first byte of that kind's grammar: // - 'n': null // - 'f': false // - 't': true // - '"': string // - '0': number // - '{': object start // - '}': object end // - '[': array start // - ']': array end type Kind byte func (k Kind) String() string A Value is the raw representation of a single JSON value, which can represent entire array or object values. It is analogous to the v1 RawMessage type. type Value []byte func (v Value) Clone() Value func (v Value) String() string func (v Value) IsValid() bool func (v *Value) Compact() error func (v *Value) Indent(prefix, indent string) error func (v *Value) Canonicalize() error func (v Value) MarshalJSON() ([]byte, error) func (v *Value) UnmarshalJSON(b []byte) error func (v Value) Kind() Kind // never ']' or '}' if valid The Compact and Indent methods operate similar to the v1 Compact and Indent function. The Canonicalize method canonicalizes the JSON value according to the JSON Canonicalization Scheme as defined in RFC 8785. A Token represents a lexical JSON token, which cannot represent entire array or object values. It is analogous to the v1 Token type, but is designed to be allocation-free by being an opaque struct type. type Token struct { /* no exported fields */ } var ( Null Token = rawToken("null") False Token = rawToken("false") True Token = rawToken("true") ObjectStart Token = rawToken("{") ObjectEnd Token = rawToken("}") ArrayStart Token = rawToken("[") ArrayEnd Token = rawToken("]") ) func Bool(b bool) Token func Int(n int64) Token func Uint(n uint64) Token func Float(n float64) Token func String(s string) Token func (t Token) Clone() Token func (t Token) Bool() bool func (t Token) Int() int64 func (t Token) Uint() uint64 func (t Token) Float() float64 func (t Token) String() string func (t Token) Kind() Kind Encoder and Decoder The Encoder and Decoder types provide the functionality for encoding to or decoding from an io.Writer or an io.Reader. An Encoder or Decoder can be constructed withNewEncoder or NewDecoder using default options. The Encoder is a streaming encoder from raw JSON tokens and values. It is used to write a stream of top-level JSON values, each terminated with a newline character. type Encoder struct { /* no exported fields */ } func (e *Encoder) Reset(w io.Writer, opts ...Options) // WriteToken writes the next token and advances the internal write offset. // The provided token must be consistent with the JSON grammar. func (e *Encoder) WriteToken(t Token) error // WriteValue writes the next raw value and advances the internal write offset. // The provided value must be consistent with the JSON grammar. func (e *Encoder) WriteValue(v Value) error // UnusedBuffer returns a zero-length buffer with a possible non-zero capacity. // This buffer is intended to be used to populate a Value // being passed to an immediately succeeding WriteValue call. // // Example usage: // // b := d.UnusedBuffer() // b = append(b, '"') // b = appendString(b, v) // append the string formatting of v // b = append(b, '"') // ... := d.WriteValue(b) func (e *Encoder) UnusedBuffer() []byte // OutputOffset returns the current output byte offset, which is the location // of the next byte immediately after the most recently written token or value. func (e *Encoder) OutputOffset() int64 The Decoder is a streaming decoder for raw JSON tokens and values. It is used to read a stream of top-level JSON values, each separated by optional whitespace characters. type Decoder struct { /* no exported fields */ } func (d *Decoder) Reset(r io.Reader, opts ...Options) // PeekKind returns the kind of the token that would be returned by ReadToken. // It does not advance the read offset. func (d *Decoder) PeekKind() Kind // ReadToken reads the next Token, advancing the read offset. // The returned token is only valid until the next Peek, Read, or Skip call. // It returns io.EOF if there are no more tokens. func (d *Decoder) ReadToken() (Token, error) // ReadValue returns the next raw JSON value, advancing the read offset. // The returned value is only valid until the next Peek, Read, or Skip call // and may not be mutated while the Decoder remains in use. // It returns io.EOF if there are no more values. func (d *Decoder) ReadValue() (Value, error) // SkipValue is equivalent to calling ReadValue and discarding the result except // that memory is not wasted trying to hold the entire value. func (d *Decoder) SkipValue() error // UnreadBuffer returns the data remaining in the unread buffer. // The returned buffer must not be mutated while Decoder continues to be used. // The buffer contents are valid until the next Peek, Read, or Skip call. func (d *Decoder) UnreadBuffer() []byte // InputOffset returns the current input byte offset, which is the location // of the next byte immediately after the most recently returned token or value. func (d *Decoder) InputOffset() int64 Some methods common to both Encoder and Decoder report information about the current automaton state. // StackDepth returns the depth of the state machine. // Each level on the stack represents a nested JSON object or array. // It is incremented whenever an ObjectStart or ArrayStart token is encountered // and decremented whenever an ObjectEnd or ArrayEnd token is encountered. // The depth is zero-indexed, where zero represents the top-level JSON value. func (e *Encoder) StackDepth() int func (d *Decoder) StackDepth() int // StackIndex returns information about the specified stack level. // It must be a number between 0 and StackDepth, inclusive. // For each level, it reports the kind: // // - 0 for a level of zero, // - '{' for a level representing a JSON object, and // - '[' for a level representing a JSON array. // // It also reports the length so far of that JSON object or array. // Each name and value in a JSON object is counted separately, // so the effective number of members would be half the length. // A complete JSON object must have an even length. func (e *Encoder) StackIndex(i int) (Kind, int) func (d *Decoder) StackIndex(i int) (Kind, int) // StackPointer returns a JSON Pointer (RFC 6901) to the most recently handled value. // Object names are only present if AllowDuplicateNames is false, otherwise // object members are represented using their index within the object. func (e *Encoder) StackPointer() string func (d *Decoder) StackPointer() string Options The behavior of Encoder and Decoder may be altered by passing options to NewEncoder and NewDecoder, which take in a variadic list of options. type Options = jsonopts.Options // AllowDuplicateNames specifies that JSON objects may contain // duplicate member names. func AllowDuplicateNames(v bool) Options // affects encode and decode // AllowInvalidUTF8 specifies that JSON strings may contain invalid UTF-8, // which will be mangled as the Unicode replacement character, U+FFFD. func AllowInvalidUTF8(v bool) Options // affects encode and decode // EscapeForHTML specifies that '<', '>', and '&' characters within JSON strings // should be escaped as a hexadecimal Unicode codepoint (e.g., \u003c) // so that the output is safe to embed within HTML. func EscapeForHTML(v bool) Options // affects encode only // EscapeForJS specifies that U+2028 and U+2029 characters within JSON strings // should be escaped as a hexadecimal Unicode codepoint (e.g., \u2028) // so that the output is valid to embed within JavaScript. // See RFC 8259, section 12. func EscapeForJS(v bool) Options // affects encode only // WithIndent specifies that the encoder should emit multiline output // where each element in a JSON object or array begins on a new, indented line // beginning with the indent prefix (see WithIndentPrefix) followed by // one or more copies of indent according to the nesting depth. func WithIndent(indent string) Options // affects encode only // WithIndentPrefix specifies that the encoder should emit multiline output // where each element in a JSON object or array begins on a new, indented line // beginning with the indent prefix followed by // one or more copies of indent (see WithIndent) according to the nesting depth. func WithIndentPrefix(prefix string) Options // affects encode only // Expand specifies that the JSON output should be expanded, where // every JSON object member or JSON array element appears on a new, indented line // according to the nesting depth. // If an indent is not already specified, then it defaults to using "\t". func Expand(v bool) Options // affects encode only The Options type is a type alias to an internal type that is an interface type with no exported methods. It is used simply as a marker type for options declared in the "json" and "jsontext" package. Latter option specified in the variadic list passed to NewEncoder and NewDecoder takes precedence over prior option values. For example, NewEncoder(AllowInvalidUTF8(false), AllowInvalidUTF8(true)) results in AllowInvalidUTF8(true) taking precedence. Options that do not affect the operation in question are ignored. For example, passing Expand to NewDecoder does nothing. The WithIndent and WithIndentPrefix flags configure the appearance of whitespace in the output. Their semantics are identical to the v1 Encoder.SetIndent method. Errors Errors due to non-compliance with the JSON grammar are reported as SyntacticError. type SyntacticError struct { // ByteOffset indicates that an error occurred after this byte offset. ByteOffset int64 // JSONPointer indicates that an error occurred within this JSON value // as indicated using the JSON Pointer notation (see RFC 6901). JSONPointer string // Err is the underlying error. Err error // always non-nil } func (e *SyntacticError) Error() string func (e *SyntacticError) Unwrap() error Errors due to I/O are returned as an opaque error that unwrap to the original error returned by the failing io.Reader.Read or io.Writer.Write call. The v2 "json" package The v2 "json" package provides functionality to marshal or unmarshal JSON data from or into Go value types. This package depends on "jsontext" to process JSON text and the "reflect" package to dynamically introspect Go values at runtime. Overview The basic API consists of the following: package json // "encoding/json/v2" func Marshal(in any, opts ...Options) (out []byte, err error) func MarshalWrite(out io.Writer, in any, opts ...Options) error func MarshalEncode(out *jsontext.Encoder, in any, opts ...Options) error func Unmarshal(in []byte, out any, opts ...Options) error func UnmarshalRead(in io.Reader, out any, opts ...Options) error func UnmarshalDecode(in *jsontext.Decoder, out any, opts ...Options) error The Marshal and Unmarshal functions mostly match the signature of the same functions in v1, however their behavior differs. The MarshalWrite and UnmarshalRead functions are equivalent functionality that operate on an io.Writer and io.Reader instead of []byte. The UnmarshalRead function consumes the entire input until io.EOF and reports an error if any invalid tokens appear after the end of the JSON value (#36225). The MarshalEncode and UnmarshalDecode functions are equivalent functionality that operate on an *jsontext.Encoder and *jsontext.Decoder instead of []byte. Default behavior The marshal and unmarshal logic in v2 is mostly identical to v1 with following changes: v1 v2 JSON object members are JSON object members are unmarshaled into a Go struct using unmarshaled into a Go struct using a case-insensitive name match. a case-sensitive name match. When marshaling a Go struct, a When marshaling a Go struct, a struct field marked as omitempty struct field marked as omitempty is omitted if the field value is is omitted if the field value an empty Go value, which is would encode as an empty JSON defined as false, 0, a nil value, which is defined as a JSON pointer, a nil interface value, null, or an empty JSON string, and any empty array, slice, map, object, or array (more discussion or string. ). The string option does affect Go The string option does not affect bools and strings. Go bools and strings. The string option does not The string option does recursively recursively affect sub-values of affect sub-values of the Go field the Go field value. value. The string option sometimes The string option never accepts a accepts a JSON null escaped within JSON null escaped within a JSON a JSON string. string. A nil Go slice is marshaled as a A nil Go slice is marshaled as an JSON null. empty JSON array (more discussion ). A nil Go map is marshaled as a A nil Go map is marshaled as an JSON null. empty JSON object (more discussion ). A Go array may be unmarshaled from A Go array must be unmarshaled a JSON array of any length. from a JSON array of the same length. A Go byte array is represented as A Go byte array is represented as a JSON array of JSON numbers. a JSON string containing the bytes in Base-64 encoding. MarshalJSON and UnmarshalJSON MarshalJSON and UnmarshalJSON methods declared on a pointer methods declared on a pointer receiver are inconsistently called receiver are consistently called. . A Go map is marshaled in a A Go map is marshaled in a deterministic order. non-deterministic order (more discussion). JSON strings are encoded with JSON strings are encoded without HTML-specific characters being any characters being escaped escaped. (unless necessary). When marshaling, invalid UTF-8 When marshaling, invalid UTF-8 within a Go string are silently within a Go string results in an replaced. error. When unmarshaling, invalid UTF-8 When unmarshaling, invalid UTF-8 within a JSON string are silently within a JSON string results in an replaced. error. When marshaling, an error does not When marshaling, an error does occur if the output JSON value occur if the output JSON value contains objects with duplicate contains objects with duplicate names. names. When unmarshaling, an error does When unmarshaling, an error does not occur if the input JSON value occur if the input JSON value contains objects with duplicate contains objects with duplicate names. names. Unmarshaling a JSON null into a Unmarshaling a JSON null into a non-empty Go value inconsistently non-empty Go value always clears clears the value or does nothing. the value. Unmarshaling a JSON value into a Unmarshaling a JSON value into a non-empty Go value always merges non-empty Go value follows if the input is an object, and inconsistent and bizarre behavior. otherwise replaces (more discussion). A time.Duration is represented as A time.Duration is represented as a JSON number containing the a JSON string containing the decimal number of nanoseconds. formatted duration (e.g., "1h2m3.456s"). Unmarshaling a JSON number into a Unmarshaling a JSON number into a Go float beyond its representation Go float beyond its representation results in an error. uses the closest representable value (e.g., +-math.MaxFloat). A Go struct with only unexported A Go struct with only unexported fields can be serialized. fields cannot be serialized. A Go struct that embeds an A Go struct that embeds an unexported struct type can unexported struct type cannot be sometimes be serialized. serialized. See here for details about every change. Every behavior change will be configurable through options, which will be a critical part of how we achieve v1-to-v2 interoperability. See here for more discussion. Struct tag options Similar to v1, v2 also supports customized representation of Go struct fields through the use of struct tags. As before, the json tag will be used. The following tag options are supported: * omitzero: When marshaling, the "omitzero" option specifies that the struct field should be omitted if the field value is zero, as determined by the "IsZero() bool" method, if present, otherwise based on whether the field is the zero Go value (per reflect.Value.IsZero). This option has no effect when unmarshaling. (example) + New in v2. The inability to omit an empty struct is a frequently cited issue in v1. This feature is intended to provide a general way to accomplish that goal (#11939, #22480 , #50480, #29310, #52803, #45669). * omitempty: When marshaling, the "omitempty" option specifies that the struct field should be omitted if the field value would have been encoded as a JSON null, empty string, empty object, or empty array. This option has no effect when unmarshaling. (example) + Changed in v2. In v1, the "omitempty" option was narrowly defined as only omitting a field if it is a Go false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string. In v2, it has been redefined in terms of the JSON type system, rather than the Go type system. They are practically equivalent except for Go bools and numbers, for which the "omitzero" option can be used instead (more discussion). * string: The "string" option specifies that StringifyNumbers be set when marshaling or unmarshaling a struct field value. This causes numeric types to be encoded as a JSON number within a JSON string, and to be decoded from either a JSON number or a JSON string containing a JSON number. This extra level of encoding is often necessary since many JSON parsers cannot precisely represent 64-bit integers. + Changed in v2. In v1, the "string" option applied to certain types where use of a JSON string did not make sense (e.g., a bool) and could not be applied recursively (e.g., a slice of integers). In v2, this feature only applies to numeric types and applies recursively. * nocase: When unmarshaling, the "nocase" option specifies that if the JSON object name does not exactly match the JSON name for any of the struct fields, then it attempts to match the struct field using a case-insensitive match that also ignores dashes and underscores. (example) + New in v2. Since v2 no longer performs a case-insensitive match of JSON object names, this option provides a means to opt-into the v1-like behavior. However, the case-insensitive match is altered relative to v1 in that it also ignores dashes and underscores. This makes the feature more broadly useful for JSON objects with different naming conventions to be unmarshaled. For example, "fooBar", "FOO_BAR", or "foo-bar" will all match with a field named "FooBar". * inline: The "inline" option specifies that the JSON object representation of this field is to be promoted as if it were specified in the parent struct. It is the JSON equivalent of Go struct embedding. A Go embedded field is implicitly inlined unless an explicit JSON name is specified. The inlined field must be a Go struct that does not implement Marshaler or Unmarshaler. Inlined fields of type jsontext.Value and map[string]T are called "inlined fallbacks", as they can represent all possible JSON object members not directly handled by the parent struct. Only one inlined fallback field may be specified in a struct, while many non-fallback fields may be specified. This option must not be specified with any other tag option. (example) + New in v2. Inlining is an explicit way to embed a JSON object within another JSON object without relying on Go struct embedding. The feature is capable of inlining Go maps and jsontext.Value (#6213). * unknown: The "unknown" option is a specialized variant of the inlined fallback to indicate that this Go struct field contains any number of "unknown" JSON object members. The field type must be a jsontext.Value, map[string]T. If DiscardUnknownMembers is specified when marshaling, the contents of this field are ignored. If RejectUnknownMembers is specified when unmarshaling, any unknown object members are rejected even if a field exists with the "unknown" option. This option must not be specified with any other tag option. (example) + New in v2. The "inline" feature technically provides a way to preserve unknown member (#22533). However, the "inline" feature alone does not semantically tell us whether this field is meant to store unknown members. The "unknown" option gives us this extra bit of information so that we can cooperate with options that affect unknown membership. * format: The "format" option specifies a format flag used to specialize the formatting of the field value. The option is a key-value pair specified as "format:value" where the value must be either a literal consisting of letters and numbers (e.g., "format:RFC3339") or a single-quoted string literal (e.g., "format:'2006-01-02'"). The interpretation of the format flag is determined by the struct field type. (example) + New in v2. The "format" option provides a general way to customize formatting of arbitrary types. + []byte and [N]byte types accept "format" values of either "base64", "base64url", "base32", "base32hex", "base16", or "hex", where it represents the binary bytes as a JSON string encoded using the specified format in RFC 4648. It may also be "array" to treat the slice or array as a JSON array of numbers. The "array" format exists for backwards compatibility since the default representation of an array of bytes now uses Base-64. + float32 and float64 types accept a "format" value of "nonfinite", where NaN and infinity are represented as JSON strings. + Slice types accept a "format" value of "emitnull" to marshal a nil slice as a JSON null instead of an empty JSON array. ( more discussion). + Map types accept a "format" value of "emitnull" to marshal a nil map as a JSON null instead of an empty JSON object. (more discussion). + The time.Time type accepts a "format" value which may either be a Go identifier for one of the format constants (e.g., "RFC3339") or the format string itself to use with time.Time.Format or time.Parse (#21990). It can also be "unix", "unixmilli", "unixmicro", or "unixnano" to be represented as a decimal number reporting the number of seconds (or milliseconds, etc.) since the Unix epoch. + The time.Duration type accepts a "format" value of "sec", "milli", "micro", or "nano" to represent it as the number of seconds (or milliseconds, etc.) formatted as a JSON number. This exists for backwards compatibility since the default representation now uses a string representation (e.g., "53.241s"). If the format is "base60", it is encoded as a JSON string using the "H:MM:SS.SSSSSSSSS" representation. The "omitzero" and "omitempty" options are similar. The former is defined in terms of the Go type system, while the latter in terms of the JSON type system. Consequently they behave differently in some circumstances. For example, only a nil slice or map is omitted under "omitzero", while an empty slice or map is omitted under "omitempty" regardless of nilness. The "omitzero" option is useful for types with a well-defined zero value (e.g., netip.Addr) or have an IsZero method (e.g., time.Time). Type-specified customization Go types may customize their own JSON representation by implementing certain interfaces that the "json" package knows to look for: type MarshalerV1 interface { MarshalJSON() ([]byte, error) } type MarshalerV2 interface { MarshalJSONV2(*jsontext.Encoder, Options) error } type UnmarshalerV1 interface { UnmarshalJSON([]byte) error } type UnmarshalerV2 interface { UnmarshalJSONV2(*jsontext.Decoder, Options) error } The v1 interfaces are supported in v2 to provide greater degrees of backward compatibility. If a type implements both v1 and v2 interfaces, the v2 variant takes precedence. The v2 interfaces operate in a purely streaming manner. This API can provide dramatic performance improvements. For example, switching from UnmarshalJSON to UnmarshalJSONV2 for spec.Swagger resulted in an ~40x performance improvement. Caller-specified customization In addition to Go types being able to specify their own JSON representation, the caller of the marshal or unmarshal functionality can also specify their own JSON representation for specific Go types (#5901). Caller-specified customization takes precedence over type-specified customization. // SkipFunc may be returned by MarshalFuncV2 and UnmarshalFuncV2 functions. // Any function that returns SkipFunc must not cause observable side effects // on the provided Encoder or Decoder. const SkipFunc = jsonError("skip function") // Marshalers holds a list of functions that may override the marshal behavior // of specific types. Populate WithMarshalers to use it. // A nil *Marshalers is equivalent to an empty list. type Marshalers struct { /* no exported fields */ } // NewMarshalers constructs a flattened list of marshal functions. // If multiple functions in the list are applicable for a value of a given type, // then those earlier in the list take precedence over those that come later. // If a function returns SkipFunc, then the next applicable function is called, // otherwise the default marshaling behavior is used. // // For example: // // m1 := NewMarshalers(f1, f2) // m2 := NewMarshalers(f0, m1, f3) // equivalent to m3 // m3 := NewMarshalers(f0, f1, f2, f3) // equivalent to m2 func NewMarshalers(ms ...*Marshalers) *Marshalers // MarshalFuncV1 constructs a type-specific marshaler that // specifies how to marshal values of type T. func MarshalFuncV1[T any](fn func(T) ([]byte, error)) *Marshalers // MarshalFuncV2 constructs a type-specific marshaler that // specifies how to marshal values of type T. // The function is always provided with a non-nil pointer value // if T is an interface or pointer type. func MarshalFuncV2[T any](fn func(*jsontext.Encoder, T, Options) error) *Marshalers // Unmarshalers holds a list of functions that may override the unmarshal behavior // of specific types. Populate WithUnmarshalers to use it. // A nil *Unmarshalers is equivalent to an empty list. type Unmarshalers struct { /* no exported fields */ } // NewUnmarshalers constructs a flattened list of unmarshal functions. // It operates in a similar manner as NewMarshalers. func NewUnmarshalers(us ...*Unmarshalers) *Unmarshalers // UnmarshalFuncV1 constructs a type-specific unmarshaler that // specifies how to unmarshal values of type T. func UnmarshalFuncV1[T any](fn func([]byte, T) error) *Unmarshalers // UnmarshalFuncV2 constructs a type-specific unmarshaler that // specifies how to unmarshal values of type T. // T must be an unnamed pointer or an interface type. // The function is always provided with a non-nil pointer value. func UnmarshalFuncV2[T any](fn func(*jsontext.Decoder, T, Options) error) *Unmarshalers The MarshalFuncV1 and UnmarshalFuncV1 functions can always be implemented in terms of the v2 variants, which calls into question their utility. There are several reasons for providing them: 1. To maintain symmetry and consistency with the method interfaces (which must provide both v1 and v2 variants). 2. To make it interoperate well with existing functionality that operate on the v1 signature. For example, to integrate the v2 "json" package with proper JSON serialization of protocol buffers, one could construct a type-specific marshaler using json.MarshalFuncV1(protojson.Marshal), where protojson.Marshal provides the JSON representation for all types that implement proto.Message (example). Caller-specified customization is a powerful feature. For example: * It can be used to marshal Go errors (example). * It can be used to preserve the raw representation of JSON numbers (example). Note that v2 does not have the v1 RawNumber type. * It can be used to preserve the input offset of JSON values for error reporting purposes (example). Options Options may be specified that configure how marshal and unmarshal operates: // Options configure Marshal, MarshalWrite, MarshalEncode, // Unmarshal, UnmarshalRead, and UnmarshalDecode with specific features. // Each function takes in a variadic list of options, where properties set // in latter options override the value of previously set properties. // // Options represent either a singular option or a set of options. // It can be functionally thought of as a Go map of option properties // (even though the underlying implementation avoids Go maps for performance). // // The constructors (e.g., Deterministic) return a singular option value: // opt := Deterministic(true) // which is analogous to creating a single entry map: // opt := Options{"Deterministic": true} // // JoinOptions composes multiple options values to together: // out := JoinOptions(opts...) // which is analogous to making a new map and copying the options over: // out := make(Options) // for _, m := range opts { // for k, v := range m { // out[k] = v // } // } // // GetOption looks up the value of options parameter: // v, ok := GetOption(opts, Deterministic) // which is analogous to a Go map lookup: // v, ok := opts["Deterministic"] // // There is a single Options type, which is used with both marshal and unmarshal. // Options that do not affect a particular operation are ignored. type Options = jsonopts.Options // StringifyNumbers specifies that numeric Go types should be marshaled as // a JSON string containing the equivalent JSON number value. // When unmarshaling, numeric Go types can be parsed from either a JSON number // or a JSON string containing the JSON number without any surrounding whitespace. func StringifyNumbers(v bool) Options // affects marshal and unmarshal // Deterministic specifies that the same input value will be serialized // as the exact same output bytes. Different processes of // the same program will serialize equal values to the same bytes, // but different versions of the same program are not guaranteed // to produce the exact same sequence of bytes. func Deterministic(v bool) Options // affects marshal only // FormatNilMapAsNull specifies that a nil Go map should marshal as a // JSON null instead of the default representation as an empty JSON object. func FormatNilMapAsNull(v bool) Options // affects marshal only // FormatNilSliceAsNull specifies that a nil Go slice should marshal as a // JSON null instead of the default representation as an empty JSON array // (or an empty JSON string in the case of ~[]byte). func FormatNilSliceAsNull(v bool) Options // affects marshal only // MatchCaseInsensitiveNames specifies that JSON object members are matched // against Go struct fields using a case-insensitive match of the name. func MatchCaseInsensitiveNames(v bool) Options // affects marshal and unmarshal // DiscardUnknownMembers specifies that marshaling should ignore any // JSON object members stored in Go struct fields dedicated to storing // unknown JSON object members. func DiscardUnknownMembers(v bool) Options // affects marshal only // RejectUnknownMembers specifies that unknown members should be rejected // when unmarshaling a JSON object, regardless of whether there is a field // to store unknown members. func RejectUnknownMembers(v bool) Options // affects unmarshal only // WithMarshalers specifies a list of type-specific marshalers to use, // which can be used to override the default marshal behavior // for values of particular types. func WithMarshalers(v *Marshalers) Options // affects marshal only // WithUnmarshalers specifies a list of type-specific unmarshalers to use, // which can be used to override the default unmarshal behavior // for values of particular types. func WithUnmarshalers(v *Unmarshalers) Options // affects unmarshal only // JoinOptions coalesces the provided list of options into a single Options. // Properties set in latter options override the value of previously set properties. func JoinOptions(srcs ...Options) Options // GetOption returns the value stored in opts with the provided constructor, // reporting whether the value is present. func GetOption[T any](opts Options, constructor func(T) Options) (T, bool) The Options type is a type alias to an internal type that is an interface type with no exported methods. It is used simply as a marker type for options declared in the "json" and "jsontext" package. This is exactly the same Options type as the one in the "jsontext" package. The same Options type is used for both Marshal and Unmarshal as some options affect both operations. The MarshalJSONV2, UnmarshalJSONV2, MarshalFuncV2, and UnmarshalFuncV2 methods and functions take in a singular Options value instead of a variadic list because the Options type can represent a set of options. The caller (which is the "json" package) can coalesce a list of options before calling the user-specified method or function. Being given a single Options value is more ergonomic for the user as there is only one options value to introspect with GetOption. While the JoinOptions constructor technically removes the need for NewEncoder, NewDecoder, Marshal, and Unmarshal from taking in a variadic list of options, it is more ergonomic for it to be variadic as the user can more readily specify a list of options without needing to call JoinOptions first. Errors Errors due to the inability to correlate JSON data with Go data are reported as SemanticError. type SemanticError struct { // ByteOffset indicates that an error occurred after this byte offset. ByteOffset int64 // JSONPointer indicates that an error occurred within this JSON value // as indicated using the JSON Pointer notation (see RFC 6901). JSONPointer string // JSONKind is the JSON kind that could not be handled. JSONKind Kind // may be zero if unknown // GoType is the Go type that could not be handled. GoType reflect.Type // may be nil if unknown // Err is the underlying error. Err error // may be nil } func (e *SemanticError) Error() string func (e *SemanticError) Unwrap() error Experimental implementation The draft proposal has been implemented by the github.com/ go-json-experiment/json module. Stability We have confidence in the correctness and performance of the module as it has been used internally at Tailscale in various production services. However, the module is an experiment and breaking changes are expected to occur based on feedback in this discussion, it should not be depended upon by publicly available code, otherwise we can run into situations where large programs fail to build. Consider the following situation: * Program P depends on modules A and B. * Module A depends on go-json-experiment/json@v0.5.0. * Module B depends on go-json-experiment/json@v0.8.0. * Let's suppose a breaking change occurs between v0.5.0 and v0.8.0. * MVS dictates that v0.8.0 be selected to build program P. * However, the use of v0.8.0 breaks module A since it is using the API for v0.5.0, which is not compatible. If open source code does use go-json-experiment, we recommend that use of it be guarded by a build tag or the entire module be forked and vendored as a dependency. Performance Due to a combination of both a more efficient implementation and also changes to the external API to better support performance, the experimental v2 implementation is generally as fast or slightly faster for marshaling and dramatically faster for unmarshaling. See the benchmarks for results. Beta Was this translation helpful? Give feedback. 48 You must be logged in to vote 111 2 39 [?] 46 31 2 All reactions * 111 * 2 * 39 * [?] 46 * 31 * 2 Replies: 31 comments * 124 replies * Oldest * Newest * Top Comment options * {{title}} Something went wrong. Quote reply [635] dsnet Oct 5, 2023 Collaborator Author - It is imperative that v1 and v2 interoperate well to provide a gradual migration from v1 to v2. Any code using v1 today must continue to function the same today and into the future. The key to v1-to-v2 interoperability lies in the API for composable options. Across the "jsontext" packages and v2 and v1 "json" packages, we have: package jsontext // "encoding/json/jsontext" type Options = jsonopts.Options func AllowDuplicateNames(v bool) Options // affects encode and decode func AllowInvalidUTF8(v bool) Options // affects encode and decode func EscapeForHTML(v bool) Options // affects encode only func EscapeForJS(v bool) Options // affects encode only func WithIndent(indent string) Options // affects encode only func WithIndentPrefix(prefix string) Options // affects encode only func Expand(v bool) Options // affects encode only package json // "encoding/json/v2" type Options = jsonopts.Options // DefaultOptionsV2 is the full set of all options that define v2 semantics. func DefaultOptionsV2() Options func StringifyNumbers(v bool) Options // affects marshal and unmarshal func Deterministic(v bool) Options // affects marshal only func FormatNilMapAsNull(v bool) Options // affects marshal only func FormatNilSliceAsNull(v bool) Options // affects marshal only func MatchCaseInsensitiveNames(v bool) Options // affects marshal and unmarshal func DiscardUnknownMembers(v bool) Options // affects marshal only func RejectUnknownMembers(v bool) Options // affects unmarshal only func WithMarshalers(v *Marshalers) Options // affects marshal only func WithUnmarshalers(v *Unmarshalers) Options // affects unmarshal only package json // "encoding/json" type Options = jsonopts.Options // DefaultOptionsV1 is the full set of all options that define v1 semantics. func DefaultOptionsV1() Options func FormatByteArrayAsArray(v bool) Options // affects marshal and unmarshal func FormatTimeDurationAsNanosecond(v bool) Options // affects marshal and unmarshal func IgnoreStructErrors(v bool) Options // affects marshal and unmarshal func MatchCaseSensitiveDelimiter(v bool) Options // affects marshal and unmarshal func MergeWithLegacySemantics(v bool) Options // affects unmarshal only func OmitEmptyWithLegacyDefinition(v bool) Options // affects marshal only func RejectFloatOverflow(v bool) Options // affects unmarshal only func ReportLegacyErrorValues(v bool) Options // affects marshal and unmarshal func SkipUnaddressableMethods(v bool) Options // affects marshal and unmarshal func StringifyWithLegacySemantics(v bool) Options // affects marshal and unmarshal func UnmarshalArrayFromAnyLength(v bool) Options // affects unmarshal only For brevity, we will use jsonv1 to refer to v1 "encoding/json" and jsonv2 to refer to "encoding/json/v2". There are several things to note: * The Options type across all three packages are identical, which implies that options declared in all three packages can be used with the jsonv2.Marshal and jsonv2.Unmarshal functions. * The jsonopts.Options type is declared in an internal package. It is an interface type without exported methods. It exists only to mark options declared in the v1 and v2 "json" package and the "jsontext" package. At present, we do not permit user-declared options, but that could be a future extension to the design. * The Options type represents either a singular option or a set of options. The jsonv1.DefaultOptionsV1 and jsonv2.DefaultOptionsV2 options represent the full set of all options that define default v1 or v2 behavior. The JoinOptions function combines multiple options together to form larger sets of options. * In the variadic listing of options that jsonv2.Marshal and jsonv2.Unmarshal accepts, latter options take precedence over prior options. * The exact list of options in "jsonv1" is still to-be-determined, but there will be an option for every behavior change relative to v2. Options that are reasonable to toggle will be declared in v2, while obscure behavior changes will have their options declared in the v1 package. Thus, these options can be composed together to obtain behavior that is identical to v1, identical to v2, or anywhere in between. For example: * jsonv1.Marshal(v) + uses default v1 semantics * jsonv2.Marshal(in, jsonv1.DefaultOptionsV1) + semantically equivalent to jsonv1.Marshal * jsonv2.Marshal(in, jsonv1.DefaultOptionsV1, jsontext.AllowDuplicateNames(false)) + uses mostly v1 semantics, but opts into one v2-specific behaviors * jsonv2.Marshal(in, jsonv1.FormatByteArrayAsArray(true), jsonv1.IgnoreStructErrors(true)) + uses mostly v2 semantics, but opts into two v1-specific behaviors * jsonv2.Marshal(v, ..., jsonv2.DefaultOptionsV2) + semantically equivalent to jsonv2.Marshal since jsonv2.DefaultOptionsV2 overrides any options specified earlier in the ... * jsonv2.Marshal(v) + uses default v2 semantics The implementation of jsonv1.Marshal will actually become jsonv2.Marshal(..., jsonv1.DefaultOptionsV1). This implies that v1 is entirely implemented in terms of v2. The implementation of v2 will work hard to ensure bug-for-bug compatibility for every v1 option. There are several advantages to implementing v1 in terms of v2: * It provides a gradual migration path from v1 to v2. * It reduces the maintenance burden of v1 as there is only one underlying implementation. * It reduces binary bloat as programs linking in both v1 and v2 only use one underlying implementation. * New v2 features that are backwards compatible with v1 (e.g., the omitzero, inline, or format tag options, types that implement jsonv2.MarshalerV2 or jsonv2.UnmarshalerV2, etc.) are immediately available in v1. In the long-term, we do not plan on ever deprecating the v1 "encoding /json" package. Rather, we will declare the high-level functions and types in that package and point users to the v2 equivalents. Deprecation of v1 functionality would not happen until at least two releases (i.e. 1 year) after v2 is available. Beta Was this translation helpful? Give feedback. 8 You must be logged in to vote 13 [?] 8 All reactions * 13 * [?] 8 0 replies Comment options * {{title}} Something went wrong. Quote reply [635] dsnet Oct 5, 2023 Collaborator Author - This is further discussion on the behavior of unmarshaling into a non-empty value. In v2, we aim to provide consistent merge semantics and recommend in UnmarshalerV1 and UnmarshalerV2 that unmarshalers implement some form of merge semantics. Merging as opposed to always clearing is a better default since "always clearing" can trivially be achieved by clearing the top-level Go value being unmarshaled into. On the other hand, merge semantics (if well defined) can be useful for unmarshaling from multiple JSON inputs into the same Go output value. There are many reasonable semantics for merging, but we should have a consistent approach to how inputs are merged. The merge semantics in v2 takes inspiration from JSON Merge Patch (RFC 7386). At a high level: * An input JSON null clears the target value (i.e., sets it to the zero Go value) * An input JSON object is merged into a target JSON object where input member values are recursively merged into target member values of the same name. * All other input JSON values replace the target JSON value. For examples of differences between v1 and v2, see this behavior difference test. Beta Was this translation helpful? Give feedback. 4 You must be logged in to vote All reactions 0 replies Comment options * {{title}} Something went wrong. Quote reply [635] dsnet Oct 5, 2023 Collaborator Author - This is further discussion on the marshaling order of Go map entries. The proposed v2 behavior is for Go maps to marshal in a non-deterministic order, matching the order (or lack thereof) provided by Go map iteration. Non-deterministic output can be made deterministic by setting the Deterministic option. In contrast, the v1 behavior is deterministic marshaling of maps. Non-deterministic marshaling is more performant since maps can be marshaled in a truly streaming manner. Any form of deterministic ordering would require sorting the map, which incurs O(n[?]log(n)) runtime and O(n) memory costs. Performance is important in RPC protocols where the ordering of JSON objects does not matter. However, non-deterministic ordering is detrimental to any use-case that assumes that the serialized output is stable, such as in tests, in the detection of changed configuration files, or in caching. There are several sources of instability in the JSON grammar: * representation of JSON numbers * representation of JSON strings * whitespace between JSON tokens * ordering of JSON object members Fortunately, RFC 8785 provides guidance for how JSON numbers and strings are to be formatted and v2 complies with that specification. The whitespace is non-existent by default or at least well-specified under WithIndent and WithIndentPrefix. Given that the other sources of instability have stable behavior, a reasonable case could be made to provide deterministic marshaling of Go maps by default. It comes down to a tradeoff between performance and convenience, with neither benefit clearly outweighing the other. --------------------------------------------------------------------- Vote on this comment for the default Go map ordering: * for non-deterministic ordering (better performance and the proposed v2 behavior) * for deterministic ordering (more convenient and the v1 behavior) Voting is no guarantee that the most popular behavior be adopted if compelling arguments for a given approach presents itself. Beta Was this translation helpful? Give feedback. 5 You must be logged in to vote 83 21 All reactions * 83 * 21 13 replies Show 8 previous replies @mvdan Comment options * {{title}} Something went wrong. Quote reply mvdan Oct 6, 2023 Collaborator - @husam-e we are talking about deterministic marshaling in general, including map, where the insertion order is not kept. So yes, for plain Go maps, that does mean sorting the keys. It's always possible to implement an ordered map and use it, and that seems separate from the deterministic option being set or not - presumably, an ordered map would always marshal its keys in the same order, so it would be entirely unaffected. Beta Was this translation helpful? Give feedback. 2 All reactions * 2 @husam-e Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. husam-e Oct 6, 2023 - Thanks for the quick clarification! In that case, non deterministic as the default for an unordered collection makes sense to me. If the collection is naturally ordered in some way (e.g. a slice) I would expect that to always be deterministic. So essentially matching the iteration behavior of the type, essentially. Supporting deterministic unmarshalling is also important, so if you can do that by unmarshalling Json to an ordered map, such that it maintains the order in the JSON, that would work. Helpful for reading then writing back to a config as mentioned prior, or testing. Beta Was this translation helpful? Give feedback. All reactions @Groxx Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. Groxx Oct 6, 2023 - To put in a reason for my : if you want canonical structure, Canonicalize() makes that clear and is much, much more likely to be stable. Otherwise JSON's spec is clear about order not being defined, and the variation across every language is so extreme that it's dangerous to implicitly expect an order. IMO it might even be worth explicitly randomizing it if not canonicalized, like map iteration, to help identify risky assumptions earlier. Beta Was this translation helpful? Give feedback. All reactions @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - @dsnet Any reason why this option is called Deterministic and not Sorted or something like that? Because to me it looks more future proof to call it Sorted as maybe in the future Golang will have insertion-order preserving maps (like Python got them a few years back) and then setting Deterministic to false might still be deterministic, but not sorted. Beta Was this translation helpful? Give feedback. All reactions @zamicol Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. zamicol Oct 6, 2023 - On a different topic from my first comment: Considering that the Go authors, Ken Thompson and Rob Pike, created UTF-8, and since Go and JSON are UTF-8 centric, I believe that JSONV2 should reject RFC 8785 and instead adhere to UTF-8. Contrary to RFC 8785, JSON's encoding is specified by JSON's RFC 8259, section 8.1. as UTF-8. Fortunately, a GitHub search reveals only four repositories mentioning RFC 8785, with one of them being the RFC's repository itself indicating that it is not a widely accepted specification. Unicode, UTF-8, and UTF-32 all share a common sorting order, while UTF-16 does not. UTF-16 misordering was an oversight corrected by UTF-8. RFC 8785's rationale for adopting UTF-16 was based on Java and .NET using UTF-16. However, Go and JSON are neither Java nor .NET. Therefore, I would recommend that JSONV2's Canonicalize method should order by UTF-8. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. [635] dsnet Oct 5, 2023 Collaborator Author - This is further discussion on how to marshal nil slices and maps. It is clear from #27589 and #37711 that users need the ability to control whether nil Go slices and maps marshal as either JSON null or empty JSON arrays or objects. However, what nil Go slices and maps should marshal as by default is less clear. The proposed default in v2 is to marshal nil Go slices and maps as empty JSON arrays and objects, while providing a format:emitnull option to opt into the alternative behavior on a per-field basis. We provide FormatNilSliceAsNull and FormatNilMapAsNull options to alter the behavior across the entire Marshal call. Regardless of the default, there will be an option to opt-into the alternate behavior. Benefit: * Avoids leaking quirks of the Go type system. JSON is a language-agnostic data interchange format. The fact that maps and slices are nil-able in Go is a semantic detail of the Go language. We should avoid leaking such details to the JSON representation. When JSON implementations leak language-specific details, it complicates transition to/from languages with different type systems. Detriment: * Cannot round-trip marshal/unmarshal slices or maps. Nil slices or maps marshal as empty JSON arrays and objects, which are subsequently unmarshaled as empty (but non-nil) slices. This means that the input and output values are not equal according to reflect.DeepEqual. However, exact equivalence of a marshal-to-unmarshal roundtrip is already not guaranteed in a number of situations such as marshaling Go struct values with non-zero unexported fields. Non-equivalence also means that in the case of maps, the input nil map would result in the allocation of a non-nil map in the output. However, unmarshal is already allocation heavy where this additional allocation may not be noticeable. --------------------------------------------------------------------- Vote on this comment for what the default nil Go slice or map representation should be: * for nil marshaling as an empty JSON object or array (the proposed v2 behavior) * for nil marshaling as JSON null (the v1 behavior) Voting is no guarantee that the most popular behavior be adopted if compelling arguments for a given approach presents itself. Beta Was this translation helpful? Give feedback. 3 You must be logged in to vote 60 6 All reactions * 60 * 6 12 replies Show 7 previous replies @ToadKing Comment options * {{title}} Something went wrong. Quote reply ToadKing Oct 6, 2023 - A big part of the previous talk about this (when it was proposed as a field tag) was that having to rely on custom types was too cumbersome for developers, especially when marshalling data returned by other packages or parts of the standard library. Beta Was this translation helpful? Give feedback. All reactions @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - @willfaught, by default the only way to get a JSON null is through a pointer or interface. Non-default ways are as follows: * Specifying FormatNilMapAsNull(true) or FormatNilMapAsNull(true) at the Marshal call * Specifying a format:emitnull tag option on Go struct field for a slice or map * Implementing a type-specified MarshalJSON or MarshalJSONV2 method that encodes a JSON null * Implementing a caller-specified MarshalFuncV1 or MarshalFuncV2 function that encodes a JSON null In general, we place control in both the hands of the authors of types and the callers of Marshal. Beta Was this translation helpful? Give feedback. All reactions @ToadKing Comment options * {{title}} Something went wrong. Quote reply ToadKing Oct 6, 2023 - Also, if you're marshalling a field whose type is an interface then you're kind of already saying that this field can be literally any type to begin with, so trying to adhere to a schema for that field seems kinda silly. Beta Was this translation helpful? Give feedback. All reactions @efd6 Comment options * {{title}} Something went wrong. Quote reply efd6 Oct 6, 2023 - It's also possible to just make the value be what you want to say. Beta Was this translation helpful? Give feedback. All reactions @doggedOwl Comment options * {{title}} Something went wrong. Quote reply doggedOwl Oct 6, 2023 - the v2 default behaviour is more frendly to consumers. Json gets consumed most of time by clients in other languages and mostly they expect an empty array especially most consumers in JS or python world where working directly on lists is a common idiom and they expect things like arrayItem.forEach to work by default or at least only having to check for the presence or not of the property (ommited or not) and not also it's nullness. Beta Was this translation helpful? Give feedback. 3 All reactions * 3 Comment options * {{title}} Something went wrong. Quote reply [635] dsnet Oct 5, 2023 Collaborator Author - This is further discussion on how to omit fields during marshal. Being able to control which fields are omitted is a reasonable feature. However, the challenge is providing the most flexible API for deciding when to omit a field. Broadly speaking, there are two dimensions that omission can be determined by: 1. It could be based on the Go type system, where certain Go values are defined as being omitted (e.g., zeros, nil, empty slices, etc.) or certain types that self-declare to be omitted (e.g., an IsZero or ShouldOmit method). 2. It could be based on the JSON type system, where Go field values that encode as certain JSON values are omitted (e.g., null, "", {}, etc.). Customized representation (e.g., via MarshalJSON) would affect whether the JSON value is to be omitted. Which approach is the best? Both have legitimate usages and neither covers all of the common use-cases by itself. For that reason, v2 proposes support for both omitzero and omitempty, where the former operates in terms of the Go type system, and the latter in terms of the JSON type system. Omission with omitzero The omitzero option omits the field if it is the Go zero value or possesses an IsZero() bool method that reports true. Properties of this approach: * The "Go zero value" is well-defined in the Go language. This is in contrast with the v1 definition of omitempty, which is narrowly defined as "false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string". Notably missing from the v1 definition are the zero value representation of structs and arrays. * The "Go zero value" allows us to omit the zero value of Go structs and arrays (#11939). Some types have well-defined zero values (e.g., netip.Addr). * The "Go zero value" allows us to distinguish between a nil versus empty (but non-nil) Go slices and maps, allowing us to omit the former, but not the latter (#22480). * Omitting the "Go zero value" has the nice property that marshaling a Go struct will emit only the non-zero fields such that unmarshaling the JSON output back into a Go struct will generally return the same result (per reflect.DeepEqual). * As an implementation detail, the reflect.Value.IsZero method provides first-class and optimized support for the "Go zero value". Alternative definitions that exclude unexported fields are both more complicated to explain and not as performant since we would have to implement our own recursive walk through a Go value using reflection. Attempts to exclude unexported fields or non-JSON serializable fields seem to actually want omission defined in terms of the JSON value (see below) and not the Go value. * Respecting an IsZero method allows types to provide their own custom definition of zero-ness. This works well with time.Time.IsZero which handles cases of zero time beyond just being time.Time{}. Omission with omitempty The omitempty option omits the field if the value would have been encoded as an empty JSON value, which we define as being a JSON null, "", {}, or []. Properties of this approach: * The application of this is recursively effective, allowing omitempty to handle cases that omitzero cannot. For example, consider the following: type Parent struct { Child Child `json:",omitempty"` } type Child struct { XXX string `json:"-"` Foo string `json:",omitzero"` Bar []int `json:",omitempty"` Baz json.RawMessge `json:",omitempty"` } json.Marshal(Parent{ Child: Child{ XXX: "ignored", // omitted since it is ignored by "-" option Foo: "", // omitted since it is the zero Go value Baz: []int{}, // omitted since it encodes as `[]` Raw: jsontext.Value("null"), // omitted since it encodes as `null` }, // omitted since it encodes as `{}` }) // thus, this just outputs `{}` * Omitting JSON [] or {} allows omitempty to omit empty Go slices and maps regardless of nil-ness similar to how the v1 operates today (albeit for different semantic reasons). * Whether a field value is to be omitted is dependent on any custom JSON serialization that would occur (e.g., via MarshalJSON). * We do not support omitting arbitrary JSON values as that would require buffering unbounded amounts of JSON text and making it difficult to provide true streaming support. For our definition, every empty JSON value has a limited width (i.e., at most 4 bytes). * Our definition of an empty JSON value does not include 0 since it is ill-defined whether -0, 0e123456, 0.000000, -0.000e+999 are included. Also, 0 is not what most users think of as being "empty". * As an implementation detail, it might seem slow to encode a JSON value only to discard it. In most cases, the internal implementation can short-circuit the actual encoding if it knows that it would have been an empty JSON value. Composability Both omitzero and omitempty are composable. You can specify both if you want, where the effect is the logical OR of the two. That is, if the Go struct field would be omitted under either omitzero or omitempty, then it is omitted. In many cases, the behavior of omitzero and omitempty will be identical. We recommend using omitzero when both have the same effect. Compatibility The omitzero option is new in v2 and can be introduced without compatibility problems. The omitempty option in v2 is redefined relative to v1, and this could cause compatibility issues. Despite the change in semantics, they generally produce the same result in most cases except the following: * Go bools and numeric kinds (e.g., int, float32, etc.) cannot be omitted with omitempty in v2 since the proposed definition of an empty JSON value does not include false or 0 (example). * Go pointers and interfaces may be omitted in v2 but not v1 if the underlying value encodes as an empty JSON value. For example, an empty string stored in an interface or a non-null pointer to an empty struct will operate differently under v2 semantics (example ). In both cases, the v1 semantics can be obtained by using the omitzero option. Thus, we propose the addition of omitzero to the v1 "json" package before the adoption of a v2 "json" package. This will provide time before the arrival of v2 where the author of Go types can migrate their type declarations to use omitzero instead of omitempty for Go bool and numeric kinds. To be clear, we are not proposing that we alter the behavior of omitempty in v1, but only providing a means for which the author of Go types can make their types compatible with both v1 and a future v2. Beta Was this translation helpful? Give feedback. 6 You must be logged in to vote 8 All reactions * 8 3 replies @diamondburned Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. diamondburned Oct 6, 2023 - Would it be helpful to have a way to express the three possible states of omitted, null and T as Go types? This would be especially useful when ingesting fields that convey different information depending on those states. For example, consider a hypothetical Omittable[T] type. We could then define a simple update type: type EventUpdate struct { // OtherID is the event A's other ID. // If omitted, then the field is unchanged. // If null, then the field has been reset to null. // If ID, then the field's value has been updated. OtherID json.Omittable[*ID] `json:"other_id"` } Users could then explicitly specify whether they want the OtherID field: // update event that makes no change to OtherID EventUpdate{} // an update event that sets OtherID to something else EventUpdate{OtherID: json.Present(ptr(123))} // an update event that resets OtherID to null EventUpdate{OtherID: json.Present(null)} or detect if a field is present or not: var update EventUpdate json.Unmarshal(b, &update) if !update.OtherID.IsZero() { // Add into a SQL UPDATE statement. q = append(q, "other_id = ?, ") v = append(v, update.OtherID.Value()) } Implementing an Omittable[T] type would be relatively easy with the v2 API: // Ommitable marks that a value of its type is either omitted (not in JSON at all) // or is present and is of type T. NOTE: the user MUST use this with `,omitzero`! type Omittable[T] struct { value T valid bool } // IsZero returns true if the field should be omitted. func (o Omittable[T]) IsZero() bool { return o.valid } // Value returns the present value. If none, then its zero value is returned. func (o Omittable[T]) Value() T { return o.value } // Present returns v wrapped in a new Omittable[T] that will not be omitted. func Present[T any](v T) Omittable[T] { return Omittable[T]{v, true} } One downside of this implementation is that the user still has to add ,omitzero manually. If the ShouldOmit() function is implemented (# 63397 (comment)), then this implementation could be slightly easier. I'm also not sure if json.Present is a good name. A better name should probably be used. The main question is: should this use case be covered by an additional type? Or should users write their own type boilerplate when they need to? Beta Was this translation helpful? Give feedback. 1 All reactions * 1 @mvdan Comment options * {{title}} Something went wrong. Quote reply mvdan Oct 6, 2023 Collaborator - I'd like to think about optional/omittable/nullable generic types more broadly across std, like #48702 - there is little here that would be JSON-specific. And if such a generic type were to appear later on, I imagine we could add support for it in encoding/json/v2 in a backwards compatible way. It's worth noting that #60370 was added, but I'd prefer to not simply follow that example and end up with very similar types across a handful of std packages. Beta Was this translation helpful? Give feedback. 2 All reactions * 2 @diamondburned Comment options * {{title}} Something went wrong. Quote reply diamondburned Oct 6, 2023 - I'd like to think about optional/omittable/nullable generic types more broadly across std I definitely agree! I think in an ideal world, this would be the best solution. I only suggested a type in encoding/json/v2 because I wasn't sure how practical it would be to add this into the stdlib, given that the proposal itself is for encoding/json/v2. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply [319] ianlancetaylor Oct 5, 2023 Maintainer - Thanks for the really excellent detailed proposal. While servers often use JSON as both input and output, there are a number of programs that generate JSON without reading it, and there are a number that read JSON without generating it. The JSON encoding and decoding functions seem largely distinct. Have you considered the possibility of separate encoding/json/encode and encoding/json/decode packages? They would perhaps both import from a shared encoding/json package for common information. Beta Was this translation helpful? Give feedback. 3 You must be logged in to vote 8 [?] 1 All reactions * 8 * [?] 1 3 replies @jub0bs Comment options * {{title}} Something went wrong. Quote reply jub0bs Oct 5, 2023 - This remark echoes @bradfitz's regrets about oversharing types between client and server in his design of net/http. Beta Was this translation helpful? Give feedback. [?] 4 All reactions * [?] 4 @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 5, 2023 Collaborator Author - It's unclear to me the benefit of this. Dead code elimination (DCE) has gotten much better in recent years such that the cost of importing a package (but not using it) has dropped significantly. Thus, splitting this out doesn't seem to benefit binary bloat. That said, the prototype v2 implementation generates the marshal/unmarshal functions together, so depending on only one would link in the implementation for both (today). But that's an implementation detail; we can modify the code to assist in DCE. The other benefit of splitting it out is to have a conceptual separation between marshal/unmarshal, but doing some brings a few detriments as well: * As it stands, having both in the "json" package makes the call site unambiguous. It's clear what json.Marshal and json.Unmarshal do. However, encode.Marshal and decode.Unmarshal is less clear unless we go with jsonencode.Marshal and jsondecode.Unmarshal, but that stutters quite a bit. + We could make it non-idiomatic and rely on the package name as a verb and have it be encode.JSON or decode.JSON, but we'd be making use of a potentially contended namespace for "encode" and "decode". How would we extend this same pattern to XML or YAML? It still bothers me that I can't tell whether rand.Read refers to "math/rand" or "crypto/rand". * json.Marshal is almost the entire opposite of json.Unmarshal. The "json" package provides a unified place to describe that mapping. I'd argue that there's closer correlation between Marshal and Unmarshal than there is between http.Client and http.Server. Beta Was this translation helpful? Give feedback. 20 All reactions * 20 @mvdan Comment options * {{title}} Something went wrong. Quote reply mvdan Oct 6, 2023 Collaborator - We could also write a regression test that would build a tiny Go program only using json.Marshal and then check that json.Unmarshal and the other decoding functions and types aren't included in the binary - and vice versa. This would keep us honest and ensure that we don't make any changes in the future that could tie the encoder and decoder implementations together. Beta Was this translation helpful? Give feedback. 4 All reactions * 4 Comment options * {{title}} Something went wrong. Quote reply [417] abhinav Oct 5, 2023 - Thanks, @dsnet. I've been looking forward to this discussion! One minor thing I want to bring up is the naming of the V2 MarshalJSON and UnmarshalJSON methods: type MarshalerV2 interface { MarshalJSONV2(*jsontext.Encoder, Options) error } type UnmarshalerV2 interface { UnmarshalJSONV2(*jsontext.Decoder, Options) error } I'd like to suggest methods without "V2" in their names. Two years down the line, if someone writes a type implementing only the V2 methods, I don't think "V2" should be present in their signatures. That's a small degree of noise that'll exist solely for historical reasons. I understand that being able to implement both V1 and V2 interfaces is a hard requirement so we cannot re-use the name MarshalJSON, but we can probably find something else. To start the conversation on the interface method names, how about EncodeJSON and DecodeJSON? type MarshalerV2 interface { EncodeJSON(*jsontext.Encoder, Options) error } type UnmarshalerV2 interface { DecodeJSON(*jsontext.Decoder, Options) error } Beta Was this translation helpful? Give feedback. 7 You must be logged in to vote 6 1 All reactions * 6 * 1 2 replies @zephyrtronium Comment options * {{title}} Something went wrong. Quote reply zephyrtronium Oct 5, 2023 - I'm also not a huge fan of the V2-named methods, but there is some advantage to the explicit indication that it's for encoding/json/v2. I can imagine having to look up frequently whether I want "MarshalJSON" or "EncodeJSON" for the new thing. Maybe the first argument being Encoder or Decoder will help with that, though. Beta Was this translation helpful? Give feedback. All reactions @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 5, 2023 Collaborator Author - It's fairly clear that we need a new method name as we don't want to reuse MarshalJSON and UnmarshalJSON. We need to have types that implement the v1 methods today be able to implement the v2 methods. For example, #63204 proposes adding Value.MarshalJSON today to fix a bug, but doing so will introduce a O(n^2) performance flaw. It must be possible to make slog.Value faster in the future by adding the v2 method. Regarding naming: * I propose we use the verbs "encode" and "decode" to refer to syntactic functionality dealing purely with the JSON grammar, while we use the verbs "marshal" and "unmarshal" to refer to semantic functionality that provides meaning to JSON data as Go data and vice-versa. Consequently, I don't think "encode" and "decode" are the right verbs to use since we're providing semantic meaning for a concrete Go type as JSON data. * In an earlier commit of the v2 prototype, we used the names MarshalNextJSON and UnmarshalNextJSON because the methods literally marshals or unmarshals the next JSON value in a stream (similar to tar.Reader.Next). However, some people were confused by the name and thought that "next" referred to "v2". Personally, I still like the naming with Next. Beta Was this translation helpful? Give feedback. 9 All reactions * 9 This comment has been hidden. Sign in to view @dsnet This comment has been hidden. Sign in to view Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. [523] jub0bs Oct 5, 2023 - @dsnet Are you planning to use some form of the pattern known as "functional options" for configuring Encoders and Decoders? If so, I have some thoughts: 1. AFAIK, the pattern is not yet used in the standard library. And, as much as I find the pattern useful in the right situation, it remains controversial in the community. Is there a risk of alienating the pattern's detractors? 2. Calling functional options on the hot path of the program can introduce performance issues. Granted, in most programs that encode/decode JSON, performance bottlenecks will lie elsewhere, but if one objective is to improve over v1's performance, this consideration should factor in the design. 3. NewDecoder and NewEncoder may never fail at this early stage of the design, but options added later may change that. In order to future-proof the API, I suggest designing them as fallible by making them also return an error: func NewDecoder(io.Reader, ...Options) (*Decoder, error) func NewEncoder(io.Reader, ...Options) (*Encoder, error) 4. Since multiple options can be specified in those functions' variadic parameter, shouldn't the type's name be singular (Option rather than Options)? 5. If some options only make sense for decoders, others only for encoders, and yet others for both, multiple options types may make sense. More of my thoughts about how to make most of the functional-options pattern are available online in video format. Interesting that we agree on declaring the option type as an opaque interface and that we both (ab)use type aliases! Beta Was this translation helpful? Give feedback. 10 You must be logged in to vote 3 1 All reactions * 3 * 1 11 replies Show 6 previous replies @bep Comment options * {{title}} Something went wrong. Quote reply bep Oct 6, 2023 - All in all a great proposal/spec, but I think you need to reconsider the options struct vs options funcs. It's a little ironic that the JSON options cannot be unmarshaled from ... JSON. As to migration, I don't see why I as a end user couldn't do: opts := jsonv1.DefaultOptionsV1 // AllowDuplicateNames = true opts.AllowDuplicateNames = false jsonv2.Marshal(in, opts) Having all the options documented in one place (the struct) is also much nicer than wandering around looking for options funcs in the GoDoc. Beta Was this translation helpful? Give feedback. All reactions @jub0bs Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. jub0bs Oct 6, 2023 - @bep It's a little ironic that the JSON options cannot be unmarshaled from ... JSON. I really don't think that the ability to marshal encoding/decoding options from JSON should factor in the design of v2's API. IMO, this is an orthogonal concern, and it doesn't actually conflict with the use of options. wandering around looking for options funcs in the GoDoc Discoverability of options in the documentation is a non-issue if those options are of a named type. More about that elsewhere. Beta Was this translation helpful? Give feedback. 2 All reactions * 2 @jub0bs Comment options * {{title}} Something went wrong. Quote reply jub0bs Oct 6, 2023 - @dsnet I'd prefer not make it worse with functional options. Could you clarify? Are those options not "functional options"? If not, how are they implemented? Could you show us how they work under the hood? Beta Was this translation helpful? Give feedback. All reactions @nemith Comment options * {{title}} Something went wrong. Quote reply nemith Oct 6, 2023 - There is another part to variadic optional arguments in the fact they are very hard to determine behavior. Often jumping through many different files, packages, temporary structs and default values to figure out what the actual intention of the option is doing. On the flip side a config struct is pretty easy and straight forward to see how it is used. I find the proposed variable options to be somewhat nicer to casually using them but more of a pain to read and extend (depending on how they are implemented). Beta Was this translation helpful? Give feedback. All reactions @jub0bs Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. jub0bs Oct 6, 2023 - @nemith There is another part to variadic optional arguments in the fact they are very hard to determine behavior. Often jumping through many different files, packages, temporary structs and default values to figure out what the actual intention of the option is doing. On the flip side a config struct is pretty easy and straight forward to see how it is used. Isn't good documentation the answer to this? To be fair to variadic options, I must point out that a config struct tells you nothing about how its fields are going to be used. The only advantage of a struct over variadic options is that multiple occurrences of the same field in a struct literal will make a compilation fail. By contrast, multiple calls to the same option are valid (as far as the compiler is concerned), but the semantics of those calls are up to the implementation, which may puzzle users. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply [674] josharian Oct 5, 2023 Collaborator - If we're thinking to the next decade, I'd like to make sure it will be easy to add support for HuJSON/JWCC. (I know Go isn't a trend-setter, but I'd even love for it to be available from the beginning.) Beta Was this translation helpful? Give feedback. 7 You must be logged in to vote 6 1 All reactions * 6 * 1 3 replies @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 5, 2023 Collaborator Author - HuJSON/JWCC would add a non-trivial amount of complexity to jsontext to represent comments and commas. According to the JSON grammar, commas are a token and would probably need a representation through the jsontext.Token type. Today, there is no such construct since JSON has the nice property that commas can be perfectly inferred when decoding and encoding since there is only one possible representation. Optional trailing commas complicates the decoder since we now need to decide whether to expose the fact that the next token was a comma or not. This is a solvable problem, but probably out-of-scope. My opinion is to keep jsontext as is and consider how to expand the API to support JWCC if/when the time is right. Beta Was this translation helpful? Give feedback. All reactions @nemith Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. nemith Oct 6, 2023 - There is also https://json5.org/ which is similar but goes further, so there are multiple competing "futures" Beta Was this translation helpful? Give feedback. 1 All reactions * 1 @ydnar Comment options * {{title}} Something went wrong. Quote reply ydnar Oct 6, 2023 - @dsnet what if jsonv2.UnmarshalDecode accepted an interface instead of a concrete jsontext.Decoder? Then others could implement a json5text package that handled the additional syntax, and plumb into the new json package without having to copy all of it? Beta Was this translation helpful? Give feedback. 1 All reactions * 1 Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. [674] josharian Oct 5, 2023 Collaborator - What do you think about having an inverted control API as well (i.e. instead of writing json, you get a reader)? Relevant historical discussion: https://gophers.slack.com/archives/ C0VP8EF3R/p1655155603882039, #51092 (comment). There's a general sense that the Right Fix is something general purpose involving io.Pipe, but error handling remains a unsolved issue. I'm bringing it up again here because generating JSON for use as an HTTP POST body is so common. Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote 2 All reactions * 2 2 replies @rogpeppe Comment options * {{title}} Something went wrong. Quote reply rogpeppe Oct 5, 2023 Collaborator - I wonder whether the coroutine proposal might end up helping in that respect. Beta Was this translation helpful? Give feedback. 2 All reactions * 2 @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 5, 2023 Collaborator Author - When I saw the coroutine package, I immediately started thinking about how it could solve the inverted io.Reader and io.Writer problem, but I haven't quite come to terms with exactly how. Since this problem exists outside of just "json", but also with gzip, base64, etc., it'd be great if we had a single solution for all packages. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply [674] josharian Oct 5, 2023 Collaborator - Dealing with union-typed JSON (e.g. a given field is either a string or a map[string]string or a map[string]SomeStruct, but nothing else) is a real pain point when working with APIs designed by people using loosely-typed languages. Might there be some first class support for union types? Beta Was this translation helpful? Give feedback. 9 You must be logged in to vote 10 [?] 1 All reactions * 10 * [?] 1 14 replies Show 9 previous replies @daenney Comment options * {{title}} Something went wrong. Quote reply daenney Oct 6, 2023 - Though I'm personally not a fan of the idea that a field can have multiple representations, it's a regular occurrence when dealing with W3 specs. W3 seems to primarily concern themselves on whether this can comfortable be expressed and worked with in JavaScript and similarly dynamically typed languages, to which the answer is almost always "yes". But it makes dealing with things like the W3 Web-of-Things spec, ActivityPub or anything that pulls in JSON-LD challenging in Go. It would be tremendously helpful to have the necessary building blocks to more easily deal with that situation than with encoding/ json. Beta Was this translation helpful? Give feedback. All reactions @mvdan Comment options * {{title}} Something went wrong. Quote reply mvdan Oct 6, 2023 Collaborator - I'd like us to avoid any form of user-facing global state in the new API - otherwise the "global registry" of types with custom encodings could easily grow out of control in a large enough codebase, and lead to either bad performance or confusing behavior. Look at net/http's global muxer or flag's global flagset as two examples, which are practically unusable for any medium sized Go project. The proposed API allows creating type-specific marshalers and unmarshalers. For example, see MarshalFuncV2 and UnmarshalFuncV2. You can then use that as an option via NewMarshalers and WithMarshalers. So this kind of per-type custom encoding and decoding is already supported. Then, instead of a library globally registering a custom type marshaler or unmarshaler, it would have to coordinate with the package that actually performs the JSON marshal or unmarshal to pass along the custom marshalers and unmarshalers to be used as options. Or perhaps export them as getters or global variables, and leave it to the importer to grab them. Beta Was this translation helpful? Give feedback. 2 All reactions * 2 @mvdan Comment options * {{title}} Something went wrong. Quote reply mvdan Oct 6, 2023 Collaborator - Worth noting that, as a library, you would typically implement methods like MarshalJSONV2. The type-specific marshalers and unmarshalers are meant to sit at a higher level, such as a Go HTTP server wanting to marshal all time.Time values in a peculiar way. It is still possible to combine custom marshalers and unmarshalers, but that should be done with some restraint. Beta Was this translation helpful? Give feedback. All reactions @diamondburned Comment options * {{title}} Something went wrong. Quote reply diamondburned Oct 6, 2023 - Then, instead of a library globally registering a custom type marshaler or unmarshaler, it would have to coordinate with the package that actually performs the JSON marshal or unmarshal to pass along the custom marshalers and unmarshalers to be used as options. Or perhaps export them as getters or global variables, and leave it to the importer to grab them. My main issue with this is the idea that libraries may have to manually manage their own list of "JSON dependencies": if a package requires any JSON types from another package, it too will have to maintain a list of marshalers that extend said package's. Otherwise, the user will have to do this job themselves, meaning they will have to go through all their indirect dependencies and figure out what's missing! For a language that already automates imports so much, it seems counterintuitive to have to remember what dependencies you have to track a list of JSON dependencies AND for the user to also remember to use it in their json calls. As far as I understand, whenever the user wants to use JSON types, they would also want to use said type's marshalers, if any. This is definitely the case for concrete types that can implement json.Marshaler and/or json.Unmarshaler. The main problem that my comment tries to solve is specifically for interface types. Worth noting that, as a library, you would typically implement methods like MarshalJSONV2. The type-specific marshalers and unmarshalers are meant to sit at a higher level, such as a Go HTTP server wanting to marshal all time.Time values in a peculiar way. Right, for concrete types, json.Marshaler and/or json.Unmarshaler should still be preferable. It's just not possible to do this with interface types :( Beta Was this translation helpful? Give feedback. All reactions @mvdan Comment options * {{title}} Something went wrong. Quote reply mvdan Oct 6, 2023 Collaborator - I think that's the underlying issue at hand: why do you imagine that libraries would need to declare custom marshalers/unmarshalers on interfaces? For an approximation of sum types, perhaps? Beta Was this translation helpful? Give feedback. 1 All reactions * 1 Comment options * {{title}} Something went wrong. Quote reply [261] soypat Oct 6, 2023 - Is it possible to specify a max stack/object depth and reject the JSON on hitting the limit? I don't see an Option that provides this functionality Beta Was this translation helpful? Give feedback. 3 You must be logged in to vote 1 All reactions * 1 1 reply @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - Probably not in the initial release, unless we get around to implementing it. We did set aside the possibility of adding jsontext.WithByteLimit and jsontext.WithDepthLimit options. Also, the proposal for at least something like jsontext.WithByteLimit has been accepted for v1. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply [102] ToadKing Oct 6, 2023 - UnmarshalRead consuming the entire input is a good change, but will there be a way exposed in the json package for getting the old behavior? Or will you have to roll your own using jsontext? Multiple JSON objects in a single stream are uncommon but maybe not uncommon enough to expose some way for users to do it explicitly. Perhaps as a jsonopts? Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote All reactions 1 reply @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - The equivalent behavior in v2 would be: json.UnmarshalDecode(jsontext.NewDecoder(r), &v) where you create a one-off jsontext.Decoder and only unmarshal the next JSON value in the stream and ignore the rest. It's a little bit more typing than: json.NewDecoder(r).Decode(&v) but I'd imagine that this is the far less common use-case. Beta Was this translation helpful? Give feedback. 3 All reactions * 3 Comment options * {{title}} Something went wrong. Quote reply [585] mitar Oct 6, 2023 - Thank you for a very detailed proposal. I read through it but maybe I missed it, but it seems to me that current proposal does not provide access to struct tags for MarshalJSONV2 and UnmarshalJSONV2? To me that was always the biggest limitation. That if I wanted to customize a bit how JSON was marshaled or unmarshaled (e.g., do some custom validation on values), all users of my struct would not be able to provide struct tags to fine control that custom implementation. I see that there is new format struct tag, which seems like something which will be passed to few standard structs, but I missed somewhere in Options where you access that format? Am I missing something? Or, in other words, I think Options should have a way to access json struct tag (or maybe even all struct tags, so that one can use additional struct tags, e.g., if json is missing but yaml exists, use yaml) associated with the value currently being marshaled or unmarshaled. Beta Was this translation helpful? Give feedback. 3 You must be logged in to vote 1 All reactions * 1 6 replies Show 1 previous reply @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - Yes, the question of recursion is also something I raised in general (not just for Options) in here. But the question here is I think broader, not just the question of format but in general of accessing to struct tags of the value being marshaled. Maybe the solution is to simply always provide all struct tags through Options and leave to custom marshal implementors to decide on how they want to utilize them. Beta Was this translation helpful? Give feedback. 1 All reactions * 1 @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - In turns out we had a more in-depth discussion about recursive format flags on 2021-04-09. Fundamentally, it's complicated and we weren't happy with what we could come up with at the time. Beta Was this translation helpful? Give feedback. 2 All reactions * 2 @nemith Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. nemith Oct 6, 2023 - This was a problem that we wanted to solve when dealing with generated structs in Thrift (specifically FBThrift). We had a number of endpoints that expected the numerical (value) representation of enum (implemented with const iotas) and a number of them that wanted a the enum/const name. example: type MyEnum int const ( MyEnum_VARIANT1 = iota MyEnum_VARIANT2 = 1 MyEnum_VARIANT3 = 2 ) Some times when encoding this i want the value to be a string of "VARIANT1" and sometimes i would want the value to be a numerical 1 (or even a string of "1"). Without duplicating every type and every structure that refers to that type There was no seemingly easily way to influence how these deeply nested structures would end up emitting these values when encoding with json since the format was tied to the type and couldn't be influenced at runtime/call site. It would be nice to be able to not modify a structure to influence the format options in some way. Not all structures are going to be in your control or flat enough to easily just embed and extend. Beta Was this translation helpful? Give feedback. All reactions @dsnet Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. dsnet Oct 6, 2023 Collaborator Author - @nemith We haven't implemented it in v2 "json" yet, but #56235 would solve the enum situation. For example, see #56235 (comment). Beta Was this translation helpful? Give feedback. 1 All reactions * 1 @nemith Comment options * {{title}} Something went wrong. Quote reply nemith Oct 6, 2023 - @dsnet I wasn't aware of that proposal and it seems very interesting but doesn't change the fact that I cannot choose what format to encode at runtime. Duplicating generated structs and embed "enum" types is still needed. How it seems that decoding seems like it could be more permissive. Thrift solved this by completely re-implemented the json encoding (which isn't bad cause it already knows about types, etc). Protobufs also has it's own JSON encoding so perhaps this isn't a big deal. Beta Was this translation helpful? Give feedback. All reactions 1 hidden item Load more... Comment options * {{title}} Something went wrong. Quote reply [585] mitar Oct 6, 2023 - I would suggest that the proposal is clarified how would one control the behavior if a slice is nested somewhere inside of a map[string] interface{} value? Would format apply recursively? Can even format be specified to control behavior of encoding inside a map? How can you provide multiple format values (to control format for dates and slices nested inside a map at the same time)? Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote All reactions 3 replies @dsnet Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. dsnet Oct 6, 2023 Collaborator Author - Heh, you're asking the right questions. No, format flags are not recursive. At present, we don't support plumbing format flags down composite Go values. We explored a syntax to allow for it, but it was getting complicated. We could add it in the future. If you're interested, we had a discussion about format flags and composite types on 2021-07-30. (Warning: it was a looong discussion) Beta Was this translation helpful? Give feedback. All reactions @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - I think struct tags should cover the common case and then custom unmarshal can cover other cases. But for this to work, everything which is possible with struct tags should be possible inside custom unmarshal. It should maybe even be documented, as a table, this feature in struct tags you can reimplement this and this way. (And then people can customize it, or use it as custom Marshaler when recursing.) Maybe one way to help here is to be able to specify a struct method for marshaling through a struct tag: struct Foo { Bar customStruct `json:",marshal:'barMarshal'" } func (f *Foo) barMarshal(*jsontext.Encoder, Options) error { ... } Beta Was this translation helpful? Give feedback. All reactions @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - Setting aside the inability of "reflect" to call unexported methods, dynamic method dispatch like what you're suggesting is probably a no-go. In order to implement that, we would need to rely on reflect.Type.MethodByName to dynamically lookup an exported name on an arbitrary type. Use of that functionality prevents dead code elimination, resulting in significant linker bloat, since the linker can't reason whether any arbitrary exported method might be used in Go reflection, so it links in everything. In a number of modules I maintained, there seems to be a concerted effort in recent years to remove all uses of reflect.Type.MethodByName. Right now, "json" doesn't depend on it, and I think we should aim to keep it that way. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply [585] mitar Oct 6, 2023 - How does this proposal address the common pattern with dynamic languages, where you have JSON like: { "type": "book" ... fields for the book } In v1, you have to double parse this JSON, once to extract type, and then again with the corresponding struct. Is there a way to make this more performant? Like not throw away the work done by the first parse? There is unknown struct tag. Is this possible if I set it on jsontext.Value field? How could then multiple unknown fields be stored in jsontext.Value? And I can then continue unmarshaling from jsontext.Value? Maybe I missed some example of this if this is already possible. Beta Was this translation helpful? Give feedback. 4 You must be logged in to vote All reactions 6 replies Show 1 previous reply @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - Unfortunately not. The v2 API aims to sit on-top of a streaming jsontext parser, which makes this complicated. Fundamentally, this form of conditional parsing is not guaranteed to be streaming based since it's valid JSON to receive: { ... fields for the book "type": "book" } where the "type" member comes afterwards. I had to deal with this problem when implementing "protojson" for google.protobuf.Any messages, which has a similar dynamic representation. I'm not sure there is an easy way to handle this. Implementation wise, you either: 1. Only accept JSON where the "type" member comes first, but that is a violation of how JSON works. 2. Write a double-pass parser that parses into a jsontext.Value, extracts the "type" member and then unmarshals the rest into a concrete Go value of the right type. 3. Write two implementations, one that optimizes for case 1, but otherwise falls back to case 2. That said, v2 does ease this situation slightly as you can unmarshal into a: type DynamicValue struct { Type string `json:"type"` Value jsontext.Value `json:",inline"` } where you can get the "type" member separate from the rest of the object payload. Note that the inline tag option allows you to collect up all the other members of the object. Beta Was this translation helpful? Give feedback. All reactions @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - @deefdragon, the use-case that @josharian posited in #63397 (comment) is slightly easier than what @mitar is asking for here since it doesn't require a double-parse of the JSON. For @josharian's use-case, we can discern from the JSON kind alone what concrete Go type to use. Of course, both @mitar's use-case and @josharian's use-case could benefit from actual sum types in the Go language. That won't automatically solve the problem, but will start to pave the way forward. Beta Was this translation helpful? Give feedback. All reactions @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - where you can get the "type" member separate from the rest of the object payload. Nice. I think this is good enough probably. So the contents of Value is then full original JSON object, without type field? Beta Was this translation helpful? Give feedback. All reactions @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - So the contents of Value is then full original JSON object, without type field? Yep. Beta Was this translation helpful? Give feedback. All reactions @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - Nice. But this is still then re-constructed buffer from tokens after input has already been tokenized? I was hoping that one of consequences of jsontext would be that one could retain tokens as-is and collected them for later parsing, so that at least repeated tokenization could be skipped? Maybe inline struct tag could also support []jsontext.Token as the type? So: type DynamicValue struct { Type string `json:"type"` Value []jsontext.Token `json:",inline"` } But not sure how would then one pass this to unmarshal? Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply [585] mitar Oct 6, 2023 - How does current MarshalJSONV2 proposal allow custom implementation to decide to omit the field? So if I have a struct like: struct { Values []Value } And Value implements MarshalJSONV2, how can it decide that some Value 's should not be encoded at all and simply skipped? Not calling anything on jsontext.Encoder does not seem to be enough because , from the slice should also be skipped. See more background on this issue in this (closed) issue: #50480 See also my comment there why omitzero is not enough here. Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote All reactions 6 replies Show 1 previous reply @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - See another example: type Foo struct { Field Field `json:"field"` } type Field struct { DataPublic bool Value interface{} } func (f Field) MarshalJSON() ([]byte, error) { if f.DataPublic { return json.Marshal(f.Value) } else { return []byte("null"), nil } } I do not think IsZero should be returning true for private data. So zero is just a special case of "skip this", but "skip this" is more general. Beta Was this translation helpful? Give feedback. All reactions @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - Regarding your first comment: No good answers in v2. It would be interesting to extend the concept of omitzero and omitempty to elements of a Go slice and entries of a Go map. Perhaps something like: struct { Values []Value `json:",format:{elem:omitzero}"` } Beta Was this translation helpful? Give feedback. All reactions @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - I'm not sure I follow your second comment. What are you trying to get omitted and under what condition? Beta Was this translation helpful? Give feedback. All reactions @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - Perhaps something like: I do not think using format helps where at all, if Value has custom MarshalJSONV2? So my question here is: can we make MarshalJSONV2 be able to skip marshaling even when it is non-zero. What are you trying to get omitted and under what condition? I think the example is clear? So I want to omit a field based on the value of another field. So it is not based on zero/non-zero, but based on DataPublic. The example returns []byte("null"), but I would prefer if the whole value is simply skipped. Maybe MarshalJSONV2 should support returning SkipFunc as well? Maybe it does already? What happens if Value's MarshalJSONV2 returns SkipFunc? Does it correctly return valid JSON array without that element? Or does it continue to search for the next marshaler function (e.g., in Options' Marshalers)? Beta Was this translation helpful? Give feedback. All reactions @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - In fact, I think there is a need for SkipValue (or OmitValue) sentinel error, similar to SkipFunc. So all marshaling functions should support both in my view. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply [585] mitar Oct 6, 2023 - During marshal, can I access current Marshalers through Options? It seems I can set them using WithMarshalers, and there is GetOption but I cannot inspect (or especially, I cannot prepend (to have higher precedence) a different custom marshaler to existing Marshalers value? Why I am asking this is because inside custom MarshalJSONV2 I might want to call marshal itself again, passing options through, but slightly modify them (to for example include my own marshaler to process some nested value in the map), but I would still want that caller could provide other marshalers. Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote All reactions 2 replies @dsnet Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. dsnet Oct 6, 2023 Collaborator Author - To override some Marshalers, would something like this work? func (v T) MarshalJSONV2(enc *jsontext.Encoder, opts Options) error { marshalers := json.GetOption(opts, json.WithMarshalers) marshalers = json.NewMarshalers(..., marshalers, ...) opts = json.JoinOptions(opts, json.WithMarshalers(marshalers)) ... } where in the pair of ... in the json.NewMarshalers constructur you can specify custom marshalers that take precedence before (or after) the previously set marshalers. Beta Was this translation helpful? Give feedback. All reactions @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - Oh, I missed the whole Marshalers manipulation functions in there. Great! Thanks. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. [184] andig Oct 6, 2023 - Unmarshaling a JSON number into a Go float beyond its representation uses the closest representable value (e.g., +-math.MaxFloat). I have not seen this mentioned above: imho it would be nice if Inf and NaN could be treated as invalid/missing data, i.e. be omitted or marshalled to json null. Otherwise, what would the closest value of NaN be? Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote All reactions 3 replies @mvdan Comment options * {{title}} Something went wrong. Quote reply mvdan Oct 6, 2023 Collaborator - Did you see this bit of the proposal? + float32 and float64 types accept a "format" value of "nonfinite", where NaN and infinity are represented as JSON strings. Those values result in marshal errors if the format option isn't set. Beta Was this translation helpful? Give feedback. All reactions @andig Comment options * {{title}} Something went wrong. Quote reply andig Oct 6, 2023 - It would be great if we could add an option to ignore these values (null). From my experience a different data type (float vs string) can cause issues. Beta Was this translation helpful? Give feedback. All reactions @mvdan Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. mvdan Oct 6, 2023 Collaborator - Is ignoring those values, as opposed to erroring on them, a common use case? The set of features covered in this API design is driven by the most popular feature requests and issues filed against v1, and this particular one doesn't ring a bell. Also note that you could implement a type marshaler via MarshalFuncV2 on float, having whatever custom behavior you like. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply [585] mitar Oct 6, 2023 - I suggest that all format handling logic be exported as set of functions so that you can manually do the same, e.g., MarshalDate(t time.Time, format string). Or, there should be a Format value to Options, so that you could set it to your format string and call it through json.Marshal (e.g., json.Marshal(t, json.SetFormat ("unixmilli")). In general I would suggest that anything you can control using struct tags should also be possible to implement in custom marshaler in an easy way (so that you do not have to re-implement custom date marshaling for example). Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote All reactions 3 replies @mvdan Comment options * {{title}} Something went wrong. Quote reply mvdan Oct 6, 2023 Collaborator - Note that format is typically for only a subset of the data to be marshaled, and they apply in specific ways to different Go types, so I don't think setting it at the very top level would generally be a good idea. This seems very much related to the other format thread about whether the option should apply recursively. Having format only apply directly to some struct fields is simple but a bit limiting. If we want something more powerful, it probably needs an entirely new design. Also note that you can always use reflect to create new struct types with whatever field tags you want. It's not particularly easy to do, but I also imagine that the vast majority of use cases will be covered by the proposed API design. Beta Was this translation helpful? Give feedback. All reactions @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - Sorry, I do not get how your comment here is related to my comment to which you are replying? Maybe you meant to wrote it elsewhere? I am asking here for exported functions corresponding to format values, so that in custom marshaling functions I can call into them if necessary. (That is also important because currently custom marshaling functions do not have access to struct tags, which I do hope is changed.) So if you want to do a similar thing in your custom marshaling function like the default implementation would do with the format option (but maybe do a bit more processing, like validation), then this should be easy to do and having a function to call is very easy. Beta Was this translation helpful? Give feedback. All reactions @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - Oh, you mean, that json.Marshal(t, json.SetFormat("unixmilli") sets it at top-level? Suureee. But that would be something you call inside another custom marshaling function. I completely agree it is not perfect, I would prefer just to have MarshalDate(t time.Time, format string) exported. But on the other hand, calling json.Marshal on a time value itself (and not having it be nested in some other struct) is also something which should be supported anyway. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. [880] narqo Oct 6, 2023 - JSON is, no doubt, among the most widely used formats in the present days. But have you considered exploring if and how the work could be beneficial for encoding/decoding Go types from/to other (not nessesarely text-based) data formats, e.g. csv, yaml, xml, msgpack, cbor, etc? I suppose, for Go ecosystem, outside the standard library, to stay cohesive, the std could encourage third-party library authors to use its "general primitives and patterns", while implementing the specifics of other data-formats [which aren't present in std]. Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote All reactions 2 replies @mitar Comment options * {{title}} Something went wrong. Quote reply mitar Oct 6, 2023 - I have few times had a need to start with a map[string]interface{} and wanting it to be converted to a struct in the same way as if I had first marshal the map to JSON and then unmarshal. Having some shared building blocks here could make this easier. In a way, everything is serialized bytes -> map -> struct conversion (with some potential optimizations to skip storing in a real map, and sometimes to keep it in a map). Beta Was this translation helpful? Give feedback. All reactions @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - To a degree, I think this question is out-of-scope for v2 "json". That said, I've filed a few proposal to improve interoperability with other formats: * proposal: encoding: add AppendText and AppendBinary #62384 aims to improve the performance of MarshalText * encoding: add ScalarMarshaler and ScalarUnmarshaler #56235 aims to provide first-class support for numbers and bools We don't support MarshalScaler yet in v2 "json". It's a matter of time. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply [486] pascaldekloe Oct 6, 2023 - Reader-writer gives terrible performance. Adding reusable buffers won't fix the issues introduced. The API should be fully append based I think. Also, func EscapeForJS(v bool) Options introduces a function for two constants. Go has powerful literal support: use config structs to match. Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote 2 All reactions * 2 18 replies Show 13 previous replies @pascaldekloe Comment options * {{title}} Something went wrong. Quote reply pascaldekloe Oct 6, 2023 - Yes, which is it: do you stream or do you append directly to bytes.Buffer. Can't have it both ways, unless you're replicating code, in which you again: should append directly in the API rather than hide in somewhere below. Beta Was this translation helpful? Give feedback. All reactions @mvdan Comment options * {{title}} Something went wrong. Quote reply mvdan Oct 6, 2023 Collaborator - Streaming when the writer is a *bytes.Buffer doesn't matter, which is why that type is treated in a special way: https://github.com/go-json-experiment/json/blob/dc36ffcf8533/jsontext /encode.go#L114-L116 So, yes, as far as I can tell, you can have it both ways. You can't append into a []byte without using a *bytes.Buffer in the middle, but that doesn't feel like such a big deal. Beta Was this translation helpful? Give feedback. All reactions @pascaldekloe Comment options * {{title}} Something went wrong. Quote reply pascaldekloe Oct 6, 2023 - This is replicating the code, once for bytes.Buffer, and once with a buffer wrapper/workaround for other writers. Both scenario are non streaming. They keep the bytes in memory like an Append function would, so then just do the append API and leave the Writing to the user. Beta Was this translation helpful? Give feedback. All reactions @mvdan Comment options * {{title}} Something went wrong. Quote reply mvdan Oct 6, 2023 Collaborator - I'm sorry, but saying that the API as currently implemented is "non streaming" is simply false. I can marshal a Go map with a million entries and it will work without holding the entire JSON value in memory at once. How would that work with an append-like API? My top level call would necessarily have to end up with a []byte holding the entire JSON value in memory. That is very different from how the implementation currently works with an io.Writer which is not a *bytes.Buffer, where streaming does happen. Many different Go APIs taking io.Writer do their own internal buffering, and this doesn't mean it's bad design for them to take a writer. The io.Writer contract doesn't require that you must write every single byte as soon as it is available. It does, however, allow you to provide output in chunks, without necessarily appending to a single buffer. You clearly hold the opinion that streaming to an io.Writer is pointless, which is one of the main design objectives of this proposal. The proposal also explains why that's a design requirement. I don't think there's anything else I can say on the matter. Beta Was this translation helpful? Give feedback. 4 All reactions * 4 @pascaldekloe Comment options * {{title}} Something went wrong. Quote reply pascaldekloe Oct 6, 2023 - Streaming to a writer is a step on top of the appends you use to construct JSON, even when all is hidden with tricks as you propose. Instead, make the core construction use append explicitly, and use a an explicit buffer level such as the ArrayWriter I suggested above. Code readers should see where the buffer/stream cut of is, rather than learning the internals by experience. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply [455] georgesolomos Oct 6, 2023 - This is excellent. The improvements are really solid and I'm happy to see they've been used in Tailscale internally. It's also written up very clearly, big kudos on your writing and layout. Beta Was this translation helpful? Give feedback. 4 You must be logged in to vote 4 [?] 1 All reactions * 4 * [?] 1 0 replies Comment options * {{title}} Something went wrong. Quote reply [213] mnashmi Oct 6, 2023 - Great news, would love to have the gjson and sjson packages too , getting or setting objects from a string path. Beta Was this translation helpful? Give feedback. 3 You must be logged in to vote [?] 1 All reactions * [?] 1 0 replies Comment options * {{title}} Something went wrong. Quote reply edited * {{editor}}'s edit {{actor}} deleted this content . {{editor}}'s edit Something went wrong. [523] jub0bs Oct 6, 2023 - @dsnet Latter option specified in the variadic list passed to NewEncoder and NewDecoder takes precedence over prior option values. For example, NewEncoder(AllowInvalidUTF8(false), AllowInvalidUTF8 (true)) results in AllowInvalidUTF8(true) taking precedence. Is there no alternative to this behaviour? IMO, dependency on option order makes calling code harder to read and reason about. Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote All reactions 0 replies Comment options * {{title}} Something went wrong. Quote reply [175] timbray Oct 6, 2023 - I have processed really really a lot of JSON in Go and this would address the pain points I encountered. Sign me up as a supporter. It's nice that it references RFC7493, I-JSON, which says "don't accept surrogates". But it turns out that Unicode contains other flavors of garbage: specifically "control codes" and "noncharacters". These things can't be displayed to humans and are commonly used in exploits. Many perfectly competent devs aren't well-informed about this junk. So I think you might benefit from a glance at https:// www.ietf.org/archive/id/draft-bray-unichars-06.html - an early-stage Internet Draft, but is getting intense discussion and I think something like it will end up being an RFC before too long. Thus, I suggest that an option be provided to describe a reader's character repertoire. You'd need a CharacterRepertoire type. That Internet-draft suggests three increasingly-prescriptive character repertoires you might want to support. Your existing AllowInvalidUTF8, which in Unicode jargon is "all the code points", could become another character repertoire. (But not a good one.) I assume you are aware that while it's perfectly possible to encode a surrogate into UTF-8 bit patterns and round-trip it, the Unicode standard asserts that that's not valid UTF-8, so I think that if you don't set AllowInvalidUTF8, strings containing such surrogates would be rejected? (Even though they're dead easy to create, especially in Java.) Beta Was this translation helpful? Give feedback. 4 You must be logged in to vote 1 [?] 1 All reactions * 1 * [?] 1 3 replies @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - Thanks for support! It's certainly a possible extension to jsontext to support a CharacterRepertoire type as long as the implementation doesn't result in significant binary bloat. The challenge with Unicode handling is that they often require large tables to be linked in. The CharacterRepertoire type could be an interface or function such that not using it results in pretty much no code bloat. I assume you are aware that while it's perfectly possible to encode a surrogate into UTF-8 bit patterns and round-trip it, the Unicode standard asserts that that's not valid UTF-8 The "json" package ultimately relies on utf8.Valid which does reject surrogate values encoded at UTF-8 bit patterns. Beta Was this translation helpful? Give feedback. All reactions @timbray Comment options * {{title}} Something went wrong. Quote reply timbray Oct 6, 2023 - The Internet-draft suggests 3 subsets (really, have a look, it explains Unicode Garbage (TM) and isn't long). I'll copy in the ABNF for the suggested subsets. If you think it's worth the effort, I'll try to whip up efficient validators. Does the repo include good sample data I could benchmark with? Unicode Scalars (== I-JSON) unicode-scalar = %x0-D7FF / %xE000-10FFFF ; exclude surrogates XML Characters (widely used since 1998, very few problems over the years) xml-character = %x9 / %xA / %xD / ; useful controls %x20-D7FF / ; exclude surrogates %xE000-FFFD/ ; exclude FFFE and FFFF nonchars %x100000-10FFFF Unicode Assignables (All the problematic garbage excluded, what people should mostly use) unicode-assignable = %x9 / %xA / %xD / ; useful controls %x20-7E / ; exclude C1 Controls and DEL %xA0-D7FF / ; exclude surrogates %xE000-FDCF ; exclude FDD0 nonchars %xFDF0-FFFD / ; exclude FFFE and FFFF nonchars %x10000-1FFFD / %x20000-2FFFD / ; (repeat per plane) %x30000-3FFFD / %x40000-4FFFD / %x50000-5FFFD / %x60000-6FFFD / %x70000-7FFFD / %x80000-8FFFD / %x90000-9FFFD / %xA0000-AFFFD / %xB0000-BFFFD / %xC0000-CFFFD / %xD0000-DFFFD / %xE0000-EFFFD / %xF0000-FFFFD / %x100000-10FFFD Beta Was this translation helpful? Give feedback. All reactions @timbray Comment options * {{title}} Something went wrong. Quote reply timbray Oct 6, 2023 - Huh, and you'd want to do the validating on the raw utf-8 bytes as you read them. Beta Was this translation helpful? Give feedback. All reactions Comment options * {{title}} Something went wrong. Quote reply [279] Thiht Oct 6, 2023 - Would you consider supporting a Decoder option to unmarshal JSON5 1 2 , or a subset of its features? I'm not sure all of it make sense for Go (especially +/-Infinity or NaN), but supporting trailing commas in objects/arrays and comments would go a long way Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote All reactions 1 reply @thepudds Comment options * {{title}} Something went wrong. Quote reply thepudds Oct 6, 2023 Collaborator - Hi @Thiht, see also the discussion at #63397 (comment). Beta Was this translation helpful? Give feedback. 1 All reactions * 1 Comment options * {{title}} Something went wrong. Quote reply [638] willfaught Oct 6, 2023 - The jsontext package provides functionality to process JSON purely according to the grammar. Why export this package instead of making it internal? package jsontext // "encoding/json/jsontext" jsontext will rarely be imported, so why not name it "encoding/json/ text"? We shouldn't name packages "foo/foobar/foobarbaz" just so someone doesn't have to occasionally write import foobarbaz "foo/bar/ baz". It's OK for package names to not be unique, as long as their paths are. The Options type is a type alias to an internal type that is an interface type with no exported methods. It is used simply as a marker type for options declared in the "json" and "jsontext" package. Why not put Options in json/v2? Seeing jsonopts.Options, I would think, "Where is jsonopts? What did I miss?" It's confusing. Options that do not affect the operation in question are ignored. For example, passing Expand to NewDecoder does nothing. Why not panic in that case? It seems to me that it's better to minimize the API surface area when compat is a concern. A Go byte array is represented as a JSON string containing the bytes in Base-64 encoding. This seems...random? Why make this the default? The old behavior of an array of numbers seems to me like a less surprising default. A time.Duration is represented as a JSON string containing the formatted duration (e.g., "1h2m3.456s"). Do microseconds use "us" or "ms"? I suggest "us". Unmarshaling a JSON number into a Go float beyond its representation uses the closest representable value (e.g., +-math.MaxFloat). This seems like it would hide an error. Why not make it cause an error by default, and have an option to do this? What do we do for integers that exceed the size of the variable? A Go struct with only unexported fields cannot be serialized. Why not just produce an empty object? Someone may be laying the groundwork to add exported fields later. A Go struct that embeds an unexported struct type cannot be serialized. Why? introsepct Misspelling. The MarshalJSONV2, UnmarshalJSONV2, MarshalFuncV2, and UnmarshalFuncV2 methods and functions take in a singular Options value instead of a variadic list because the Options type can represent a set of options. The caller (which is the "json" package) can coalesce a list of options before calling the user-specified method or function. Being given a single Options value is more ergonomic for the user as there is only one options value to introsepct with GetOption. I don't follow why some funcs take a variadic slice of Options, and others take only one Options. If Options can be merged, then shouldn't all funcs take one Options? The exact list of options in "jsonv1" is still to-be-determined, but there will be an option for every behavior change relative to v2. Options that are reasonable to toggle will be declared in v2, while obscure behavior changes will have their options declared in the v1 package. Why is DefaultOptionsV1 declared in jsonv1 instead of jsonv2? Where is it useful in jsonv1? If they are in separate packages, I see no reason for the V* suffix. The implementation of v2 will work hard to ensure bug-for-bug compatibility for every v1 option. The "work hard" softening of the wording makes it unclear whether this means it will be backward compatible in all ways if configured to be so. However, exact equivalence of marshal unmarshal unmarshal is already not guaranteed Not sure what this is saying. Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote All reactions 2 replies @ydnar Comment options * {{title}} Something went wrong. Quote reply ydnar Oct 6, 2023 - There are already consumers of the json.Decoder that don't use the rest of the package. The jsontext package just puts the lower level primitive in a separate package, so jsonv1, jsonv2, and third party packages can import it. Beta Was this translation helpful? Give feedback. All reactions @dsnet Comment options * {{title}} Something went wrong. Quote reply dsnet Oct 6, 2023 Collaborator Author - Thanks. Fixed some of the typos mentioned above. Why export this package instead of making it internal? The MarshalJSONV2 and UnmarshalJSONV2 methods reference jsontext.Encoder and jsontext.Decoder, so the "jsontext" package must be exported for custom implementations to exist. At Tailscale, we do quite a bit of raw JSON manipulation that makes use of just the "jsontext" package. jsontext will rarely be imported, so why not name it "encoding/ json/text"? We shouldn't name packages "foo/foobar/foobarbaz" just so someone doesn't have to occasionally write import foobarbaz "foo/bar/baz". It's OK for package names to not be unique, as long as their paths are. As mentioned above, I disagree that it will be "rarely" imported. It certainly won't be as popular as "json", but it has a lot of utility. It's called "jsontext" because it literally handles "JSON text", which is specifically called out as a term in RFC 8259, section 1.2. Why not put Options in json/v2? Seeing jsonopts.Options, I would think, "Where is jsonopts? What did I miss?" It's confusing. "jsonopts" is an internal package. It can't be in "json" since "jsontext" needs to reference it, but since "json" depends on "jsontext", we have a cyclic dependency. It can't be in "jsontext" since some of the options (e.g., WithMarshalers and WithUnmarshalers) depend transitively on Go reflection, and it is critical that "jsontext" has a lightweight dependency tree. The "jsonopts" package is a shared library that both "json" and "jsontext" can depend on, and we rely on some dependency injection to work around the "reflect" dependency issue. Why not panic in that case? It seems to me that it's better to minimize the API surface area when compat is a concern. This would require us also panicking on json.Marshal(..., json.DefaultOptionsV2) since DefaultOptionsV2 contains the set of all options (including those that are only for Unmarshal). This seems...random? Why make this the default? The old behavior of an array of numbers seems to me like a less surprising default. Almost every single [...]byte in Go is semantically a binary sequence of bytes, rather than a [...]uint8. Examples include encryption keys and hashes. Base64 encoding captures the intent of the binary encoding. Some use-cases would actually want this to use hexadecimal encoding, but we didn't consider it worth it to make it inconsistent with [] byte. We support a format flag that can opt into hexadecimal encoding or the v1 encoding as a JSON array. Do microseconds use "us" or "ms"? I suggest "us". It's "ms" as outputted by time.Duration.String. I'd argue that changing it be out-of-scope here. This seems like it would hide an error. Why not make it cause an error by default, and have an option to do this? What do we do for integers that exceed the size of the variable? Integers by definition have a discrete representation on the number line. Thus, we strictly reject integers that exceed the range. Floating-point number by nature are always lossy. You can have: * a JSON number that too small to represent and aliases to 0 * a JSON number with precision too great to represent in a float64, or * a JSON number that exceeds the finite representation of float64, but different JSON implementations may disagree on exactly where the cutoff is when it becomes invalid. Since floating-point numbers are already lossy, we stay consistent with that principle by using the closest finite representation. Also, if a piece of JSON text is valid, it is should always be possible to unmarshal into an any without error. Why not just produce an empty object? Someone may be laying the groundwork to add exported fields later. A very common pitfall of JSON in Go is that people accidentally marshal a Go struct with only unexported fields, not realizing that it doesn't do what they expect. This behavior is intended to fix that sharp edge. If you're just setting the groundwork, you could add a json:"-" tag to one of the unexported fields to explicitly signal that you thought about how JSON should interact with this type in the future. A Go struct that embeds an unexported struct type cannot be serialized. Why? See https://github.com/go-json-experiment/json/blob/ dc36ffcf853375022a55817ac699f310e38cca9b/diff_test.go#L1152-L1177 Long story short: Go reflection doesn't let this work reliably. It's buggy. It's better to consistently reject it than to inconsistently support it. I don't follow why some funcs take a variadic slice of Options, and others take only one Options. If Options can be merged, then shouldn't all funcs take one Options? Variadic arguments are more ergonomic to call. It would be cumbersome to always do: json.Marshal(v, json.JoinOptions(jsontext.AllowInvalidUTF8(true), jsontext.AllowDuplicateNames(true))) Also, there's a performance gain since json.Marshal can cache the composite options struct that's produced from internally calling json.JoinOptions. Why is DefaultOptionsV1 declared in jsonv1 instead of jsonv2? Where is it useful in jsonv1? If they are in separate packages, I see no reason for the V* suffix. DefaultOptionsV1 exists in jsonv1 to partition all the v1-specific functionality in v1. Technically, all the options declared in v1 could be in the v2 package; it's an issue of cleanliness. The V1 suffix in DefaultOptionsV1 is because both packages are named "json" and users may not consistently import v1 as "jsonv1" or v2 as "jsonv2". Thus, it is explicit from the call site what json.DefaultOptionsV2 refers to. The "work hard" softening of the wording makes it unclear whether this means it will be backward compatible in all ways if configured to be so. It's the best gaurantee that we can give. There are alot of really esoteric bugs in v1 that I can't describe in words, but I also know people depend on due to Hyrum's law. Some of these bugs also relies on bugs in Go reflection. It's possible that we may need to break some things, but hopefully not. We haven't finished the v1 compatibility layer yet. The hardest part of the v2 effort isn't actually the v2 implementation itself, it's actually ensuring v1 compatibility, and I assure you we take that seriously. Beta Was this translation helpful? Give feedback. 2 All reactions * 2 Comment options * {{title}} Something went wrong. Quote reply [305] gjvnq Oct 6, 2023 - The time.Duration type accepts a "format" value of "sec", "milli", "micro", or "nano" to represent it as the number of seconds (or milliseconds, etc.) formatted as a JSON number. This exists for backwards compatibility since the default representation now uses a string representation (e.g., "53.241s"). If the format is "base60", it is encoded as a JSON string using the "H:MM:SS.SSSSSSSSS" representation. Why not add an option for ISO 8601? This sounds like it could help a lot of people who deal with durations that excede a day. Eg: the string "P1Y2M3DT4H5M6S" would mean "1 year, 2 months, 3 days, 4 hours, 5 minutes, and 6 seconds", and the string "P1W" would mean one week. Just to be clear, I'm excluding the date range format (/ ) from my suggestion as having to calculate date differences to get the time duration feels like a bad return on investment as almost nobody seems to use it when sending time durations over JSON documents. Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote All reactions 0 replies Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment Category Discussions Labels None yet 37 participants @dsnet @timbray @ydnar @abhinav @willfaught @rogpeppe @josharian @smyrman @Groxx @narqo @mikeschinkel @andig @cespare @nemith @bep @daenney @mitar @ToadKing @zephyrtronium @husam-e @Thiht and others Add heading text Add bold text, Add italic text, Add a quote, Add code, Insert Link Link Text [ ] URL [ ] Add Add a link, Add a bulleted list, Add a numbered list, Add a task list, Directly mention a user or team Reference an issue or pull request Add heading text Add bold text, Add italic text, Add a bulleted list, Add a numbered list, Add a task list, 1 reacted with thumbs up emoji 1 reacted with thumbs down emoji 1 reacted with laugh emoji 1 reacted with hooray emoji 1 reacted with confused emoji [?] 1 reacted with heart emoji 1 reacted with rocket emoji 1 reacted with eyes emoji Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.