https://github.com/golang/go/discussions/54763 Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Changelog * Solutions + By Size + Enterprise + Teams + Compare all + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} golang / go Public * Notifications * Fork 15.4k * Star 104k * Code * Issues 5k+ * Pull requests 295 * Discussions * Actions * Projects 2 * Wiki * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Wiki * Security * Insights discussion: structured, leveled logging #54763 jba announced in Discussions discussion: structured, leveled logging #54763 @jba jba Aug 29, 2022 * 27 comments * 116 replies Return to top [184] jba Aug 29, 2022 Maintainer - This is a discussion that is intended to lead to a proposal. We would like to add structured logging with levels to the standard library. Structured logging is the ability to output logs with machine-readable structure, typically key-value pairs, in addition to a human-readable message. Structured logs can be parsed, filtered, searched and analyzed faster and more reliably than logs designed only for people to read. For many programs that aren't run directly by person, like servers, logging is the main way for developers to observe the detailed behavior of the system, and often the first place they go to debug it. Logs therefore tend to be voluminous, and the ability to search and filter them quickly is essential. In theory, one can produce structured logs with any logging package: log.Printf(`{"message": %q, "count": %d}`, msg, count) In practice, this is too tedious and error-prone, so structured logging packages provide an API for expressing key-value pairs. This draft proposal contains such an API. We also propose generalizing the logging "backend." The log package provides control only over the io.Writer that logs are written to. In the new package (tentative name: log/slog), every logger has a handler that can process a log event however it wishes. Although it is possible to have a structured logger with a fixed backend (for instance, zerolog outputs only JSON), having a flexible backend provides several benefits: programs can display the logs in a variety of formats, convert them to an RPC message for a network logging service, store them for later processing, and add to or modify the data. Lastly, we include levels in our design, in a way that accommodates both traditional named levels and logr-style verbosities. Our goals are: * Ease of use. A survey of the existing logging packages shows that programmers want an API that is light on the page and easy to understand. This proposal adopts the most popular way to express key-value pairs, alternating keys and values. * High performance. The API has been designed to minimize allocation and locking. It provides an alternative to alternating keys and values that is more cumbersome but faster (similar to Zap's Fields). * Integration with runtime tracing. The Go team is developing an improved runtime tracing system. Logs from this package will be incorporated seamlessly into those traces, giving developers the ability to correlate their program's actions with the behavior of the runtime. What Does Success Look Like? Go has many popular structured logging packages, all good at what they do. We do not expect developers to rewrite their existing third-party structured logging code en masse to use this new package. We expect existing logging packages to coexist with this one for the foreseeable future. We have tried to provide an API that is pleasant enough to prefer to existing packages in new code, if only to avoid a dependency. (Some developers may find the runtime tracing integration compelling.) We also expect newcomers to Go to become familiar with this package before learning third-party packages, so they will naturally prefer it. But more important than any traction gained by the "frontend" is the promise of a common "backend." An application with many dependencies may find that it has linked in many logging packages. If all of the packages support the handler interface we propose, then the application can create a single handler and install it once for each logging library to get consistent logging across all its dependencies. Since this happens in the application's main function, the benefits of a unified backend can be obtained with minimal code churn. We hope that this proposal's handlers will be implemented for all popular logging formats and network protocols, and that every common logging framework will provide a shim from their own backend to a handler. Then the Go logging community can work together to build high-quality backends that all can share. Prior Work The existing log package has been in the standard library since the release of Go 1 in March 2012. It provides formatted logging, but not structured logging or levels. Logrus, one of the first structured logging packages, showed how an API could add structure while preserving the formatted printing of the log package. It uses maps to hold key-value pairs, which is relatively inefficient. Zap grew out of Uber's frustration with the slow log times of their high-performance servers. It showed how a logger that avoided allocations could be very fast. zerolog reduced allocations even further, but at the cost of reducing the flexibility of the logging backend. All the above loggers include named levels along with key-value pairs. Logr and Google's own glog use integer verbosities instead of named levels, providing a more fine-grained approach to filtering high-detail logs. Other popular logging packages are Go-kit's log, HashiCorp's hclog, and klog. Overview of the Design Here is a short program that uses some of the new API: import "log/slog" func main() { slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr))) slog.Info("hello", "name", "Al") slog.Error("oops", net.ErrClosed, "status", 500) slog.LogAttrs(slog.ErrorLevel, "oops", slog.Int("status", 500), slog.Any("err", net.ErrClosed)) } It begins by setting the default logger to one that writes log records in an easy-to-read format similar to logfmt . (There is also a built-in handler for JSON.) The program then outputs three log messages augmented with key-value pairs. The first logs at the Info level, passing a single key-value pair along with the message. The second logs at the Error level, passing an error and a key-value pair. The third produces the same output as the second, but more efficiently. Functions like Any and Int construct slog.Attr values, which are key-value pairs that avoid memory allocation for some values. slog.Attr is modeled on zap.Field. The Design Interaction Between Existing and New Behavior The slog package works to ensure consistent output with the log package. Writing to slog's default logger without setting a handler will write structured text to log's default logger. Once a handler is set, as in the example above, the default log logger will send its text output to the structured handler. Handlers A slog.Handler describes the logging backend. It is defined as: type Handler interface { // Enabled reports whether this handler is accepting records. Enabled(Level) bool // Handle processes the Record. Handle(Record) error // With returns a new Handler whose attributes consist of // the receiver's attributes concatenated with the arguments. With(attrs []Attr) Handler } The main method is Handle. It accepts a slog.Record with the timestamp, message, level, caller source position, and key-value pairs of the log event. Each call to a Logger output method, like Info, Error or LogAttrs, creates a Record and invokes the Handle method. The Enabled method is an optimization that can save effort if the log event should be discarded. Enabled is called early, before any arguments are processed. The With method is called by Logger.With, discussed below. The slog package provides two handlers, one for simple textual output and one for JSON. They are described in more detail below. The Record Type The Record passed to a handler exports Time, Message and Level methods, as well as four methods for accessing the sequence of Attrs: * Attrs() []Attr returns a copy of the Attrs as a slice. * NumAttrs() int returns the number of Attrs. * Attr(int) Attr returns the i'th Attr. * SetAttrs([]Attr) replaces the sequence of Attrs with the given slice. This API allows an efficient implementation of the Attr sequence that avoids copying and minimizes allocation. SetAttrs supports "middleware" handlers that want to alter the Attrs, say by removing those that contain sensitive data. The Attr Type The Attr type efficiently represents a key-value pair. The key is a string. The value can be any type, but Attr improves on any by storing common types without allocating memory. In particular, integer types and strings, which account for the vast majority of values in log messages, do not require allocation. The default version of Attr uses package unsafe to store any value in three machine words. The version without unsafe requires five. There are convenience functions for constructing Attrs with various value types: * Int(k string, v int) Attr * Int64(k string, v int64) Attr * Uint64(k string, v uint64) Attr * Float64(k string, v float64) Attr * String(k, v string) Attr * Bool(k string, v bool) Attr * Duration(k string, v time.Duration) Attr * Time(k string, v time.Time) Attr * Any(k string, v any) Attr The last of these dispatches on the type of v, using a more efficient representation if Attr supports it and falling back to an any field in Attr if not. The Attr.Key method returns the key. Extracting values from an Attr is reminiscent of reflect.Value: there is a Kind method that returns an enum, and a variety of methods like Int64() int64 and Bool() bool that return the value or panic if it is the wrong kind. Attr also has an Equal method, and an AppendValue method that efficiently appends a string representation of the value to a []byte, in the manner of the strconv.AppendX functions. Loggers A Logger consists of a handler and a list of Attrs. There is a default logger with no attributes whose handler writes to the default log.Logger, as explained above. Create a Logger with New: func New(h Handler) *Logger To add attributes to a Logger, use With: l2 := l1.With("url", "http://example.com/") The arguments are interpreted as alternating string keys and and arbitrary values, which are converted to Attrs. Attrs can also be passed directly. Loggers are immutable, so this actually creates a new Logger with the additional attributes. To allow handlers to preprocess attributes, the new Logger's handler is obtained by calling Handler.With on the old one. You can obtain a logger's handler with Logger.Handler. The basic logging methods are func (*Logger) Log(level Level, message string, kvs ...any) which logs a message at the given level with a list of attributes that are interpreted just as in Logger.With, and the more efficient func (Logger) LogAttrs(level Level, message string, attrs ...Attr) These functions first call Handler.Enabled(level) to see if they should proceed. If so, they create a Record with the current time, the given level and message, and a list of attributes that consists of the receiver's attributes followed by the argument attributes. They then pass the Record to Handler.Handle. Each of these methods has an alternative form that takes a call depth, so other functions can wrap them and adjust the source line information. There are four convenience methods for common levels: func (*Logger) Info(message string, kvs ...any) func (*Logger) Warn(message string, kvs ...any) func (*Logger) Debug(message string, kvs ...any) func (*Logger) Error(message string, err error, kvs ...any) They all call Log with the appropriate level. Error first appends Any ("err", err) to the attributes. There are no convenience methods for LogAttrs. We expect that most programmers will use the more convenient API; those few who need the extra speed will have to type more, or provide wrapper functions. All the methods described in this section are also names of top-level functions that call the corresponding method on the default logger. Context Support Passing a logger in a context.Context is a common practice and a good way to include dynamically scoped information in log messages. For instance, you could construct a Logger with information from an http.Request and pass it through the code that handles the request by adding it to r.Context(). The slog package has two functions to support this pattern. One adds a Logger to a context: func NewContext(ctx context.Context, l *Logger) context.Context As an example, an HTTP server might want to create a new Logger for each request. The logger would contain request-wide attributes and be stored in the context for the request: func handle(w http.ResponseWriter, r *http.Request) { rlogger := slog.With( "method", r.Method, "url", r.URL, "traceID", getTraceID(r)) ctx := slog.NewContext(r.Context(), rlogger) // ... use ctx ... } To retrieve a Logger from a context, call FromContext: slog.FromContext(ctx).Info(...) FromContext returns the default logger if it can't find one in the context. Levels A level is a positive integer, where lower numbers designate more severe or important log events. The slog package provides names for common levels, with gaps between the assigned numbers to accommodate other level schemes. (For example, Google Cloud Platform supports a Notice level between Info and Warn.) Some logging packages like glog and Logr use verbosities instead, where a verbosity of 0 corresponds to the Info level and higher values represent less important messages. To use a verbosity of v with this design, pass slog.InfoLevel + v to Log or LogAttrs. Provided Handlers The slog package includes two handlers, which behave similarly except for their output format. TextHandler emits attributes as KEY=VALUE, and JSONHandler writes line-delimited JSON objects. Both can be configured with the same options: * The boolean AddSource option controls whether the file and line of the log call. It is false by default, because there is a small cost to extracting this information. * The LevelRef option, of type LevelRef, provides control over the maximum level that the handler will output. For example, setting a handler's LevelRef to Info will suppress output at Debug and higher levels. A LevelRef is a safely mutable pointer to a level, which makes it easy to dynamically and atomically change the logging level for an entire program. * To provide fine control over output, the ReplaceAttr option is a function that both accepts and returns an Attr. If present, it is called for every attribute in the log record, including the four built-in ones for time, message, level and (if AddSource is true) the source position. ReplaceAttr can be used to change the default keys of the built-in attributes, convert types (for example, to replace a time.Time with the integer seconds since the Unix epoch), sanitize personal information, or remove attributes from the output. Interoperating with Other Log Packages As stated earlier, we expect that this package will interoperate with other log packages. One way that could happen is for another package's frontend to send slog.Records to a slog.Handler. For instance, a logr.LogSink implementation could construct a Record from a message and list of keys and values, and pass it to a Handler. To facilitate that, slog provides a way to construct Records directly and add attributes to it: func NewRecord(t time.Time, level Level, msg string, calldepth int) Record func (*Record) AddAttr(Attr) Another way for two log packages to work together is for the other package to wrap its backend as a slog.Handler, so users could write code with the slog package's API but connect the results to an existing logr.LogSink, for example. This involves writing a slog.Handler that wraps the other logger's backend. Doing so doesn't seem to require any additional support from this package. Acknowledgements Ian Cottrell's ideas about high-performance observability, captured in the golang.org/x/exp/event package, informed a great deal of the design and implementation of this proposal. Seth Vargo's ideas on logging were a source of motivation and inspiration. His comments on an earlier draft helped improve the proposal. Michael Knyszek explained how logging could work with runtime tracing. Tim Hockin helped us understand logr's design choices, which led to significant improvements. Abhinav Gupta helped me understand Zap in depth, which informed the design. Russ Cox provided valuable feedback and helped shape the final design. Appendix: API package slog // import "golang.org/x/exp/slog" FUNCTIONS func Debug(msg string, args ...any) Debug calls Logger.Debug on the default logger. func Error(msg string, err error, args ...any) Error calls Logger.Error on the default logger. func Info(msg string, args ...any) Info calls Logger.Info on the default logger. func Log(level Level, msg string, args ...any) Log calls Logger.Log on the default logger. func LogAttrs(level Level, msg string, attrs ...Attr) LogAttrs calls Logger.LogAttrs on the default logger. func NewContext(ctx context.Context, l *Logger) context.Context NewContext returns a context that contains the given Logger. Use FromContext to retrieve the Logger. func SetDefault(l *Logger) SetDefault makes l the default Logger. After this call, output from the log package's default Logger (as with log.Print, etc.) will be logged at InfoLevel using l's Handler. func Warn(msg string, args ...any) Warn calls Logger.Warn on the default logger. TYPES type Attr struct { // Has unexported fields. } An Attr is a key-value pair. It can represent some small values without an allocation. The zero Attr has a key of "" and a value of nil. func Any(key string, value any) Attr Any returns an Attr for the supplied value. Any does not preserve the exact type of integral values. All signed integers are converted to int64 and all unsigned integers to uint64. Similarly, float32s are converted to float64. However, named types are preserved. So given type Int int the expression log.Any("k", Int(1)).Value() will return Int(1). func Bool(key string, value bool) Attr Bool returns an Attr for a bool. func Duration(key string, value time.Duration) Attr Duration returns an Attr for a time.Duration. func Float64(key string, value float64) Attr Float64 returns an Attr for a floating-point number. func Int(key string, value int) Attr Int converts an int to an int64 and returns an Attr with that value. func Int64(key string, value int64) Attr Int64 returns an Attr for an int64. func String(key, value string) Attr String returns a new Attr for a string. func Time(key string, value time.Time) Attr Time returns an Attr for a time.Time. func Uint64(key string, value uint64) Attr Uint64 returns an Attr for a uint64. func (a Attr) AppendValue(dst []byte) []byte AppendValue appends a text representation of the Attr's value to dst. The value is formatted as with fmt.Sprint. func (a Attr) Bool() bool Bool returns the Attr's value as a bool. It panics if the value is not a bool. func (a Attr) Duration() time.Duration Duration returns the Attr's value as a time.Duration. It panics if the value is not a time.Duration. func (a1 Attr) Equal(a2 Attr) bool Equal reports whether two Attrs have equal keys and values. func (a Attr) Float64() float64 Float64 returns the Attr's value as a float64. It panics if the value is not a float64. func (a Attr) Format(s fmt.State, verb rune) Format implements fmt.Formatter. It formats a Attr as "KEY=VALUE". func (a Attr) HasValue() bool HasValue returns true if the Attr has a value. func (a Attr) Int64() int64 Int64 returns the Attr's value as an int64. It panics if the value is not a signed integer. func (a Attr) Key() string Key returns the Attr's key. func (a Attr) Kind() Kind Kind returns the Attr's Kind. func (a Attr) String() string String returns Attr's value as a string, formatted like fmt.Sprint. Unlike the methods Int64, Float64, and so on, which panic if the Attr is of the wrong kind, String never panics. func (a Attr) Time() time.Time Time returns the Attr's value as a time.Time. It panics if the value is not a time.Duration. func (a Attr) Uint64() uint64 Uint64 returns the Attr's value as a uint64. It panics if the value is not an unsigned integer. func (a Attr) Value() any Value returns the Attr's value as an any. If the Attr does not have a value, it returns nil. func (a Attr) WithKey(key string) Attr WithKey returns an attr with the given key and the receiver's value. type Handler interface { // Enabled reports whether this handler is accepting records // at the given level. Enabled(Level) bool // Handle processes the Record. // Handle methods that produce output should observe the following rules: // - If r.Time() is the zero time, do not output it. // - If r.Level() is Level(0), do not output it. Handle(Record) error // With returns a new Handler whose attributes consist of // the receiver's attributes concatenated with the arguments. With(attrs []Attr) Handler } A Handler processes log records produced by Logger output. Any of the Handler's methods may be called concurrently with itself or with other methods. It is the responsibility of the Handler to manage this concurrency. type HandlerOptions struct { // Add a "source" attributes to the output whose value is of the form // "file:line". AddSource bool // Ignore records with levels above LevelRef.Level. // If nil, accept all levels. LevelRef *LevelRef // If set, ReplaceAttr is called on each attribute of the message, // and the returned value is used instead of the original. If the returned // key is empty, the attribute is omitted from the output. // // The built-in attributes with keys "time", "level", "source", and "msg" // are passed to this function first, except that time and level are omitted // if zero, and source is omitted if AddSource is false. ReplaceAttr func(a Attr) Attr } HandlerOptions are options for a TextHandler or JSONHandler. A zero HandlerOptions consists entirely of default values. func (opts HandlerOptions) NewJSONHandler(w io.Writer) *JSONHandler NewJSONHandler creates a JSONHandler with the given options that writes to w. func (opts HandlerOptions) NewTextHandler(w io.Writer) *TextHandler NewTextHandler creates a TextHandler with the given options that writes to w. type JSONHandler struct { // Has unexported fields. } JSONHandler is a Handler that writes Records to an io.Writer as line-delimited JSON objects. func NewJSONHandler(w io.Writer) *JSONHandler NewJSONHandler creates a JSONHandler that writes to w, using the default options. func (h JSONHandler) Enabled(l Level) bool Enabled reports whether l is less than or equal to the maximum level. func (h *JSONHandler) Handle(r Record) error Handle formats its argument Record as a JSON object on a single line. If the Record's time is zero, it is omitted. Otherwise, the key is "time" and the value is output in RFC3339 format with millisecond precision. If the Record's level is zero, it is omitted. Otherwise, the key is "level" and the value of Level.String is output. If the AddSource option is set and source information is available, the key is "source" and the value is output as "FILE:LINE". The message's key is "msg". To modify these or other attributes, or remove them from the output, use [HandlerOptions.ReplaceAttr]. Values are formatted as with encoding/json.Marshal. Each call to Handle results in a single, mutex-protected call to io.Writer.Write. func (h *JSONHandler) With(attrs []Attr) Handler With returns a new JSONHandler whose attributes consists of h's attributes followed by attrs. type Kind int Kind is the kind of an Attr's value. const ( AnyKind Kind = iota BoolKind DurationKind Float64Kind Int64Kind StringKind TimeKind Uint64Kind ) func (k Kind) String() string type Level int A Level is the importance or severity of a log event. The higher the level, the less important or severe the event. const ( ErrorLevel Level = 10 WarnLevel Level = 20 InfoLevel Level = 30 DebugLevel Level = 31 ) Names for common levels. func (l Level) String() string String returns a name for the level. If the level has a name, then that name in uppercase is returned. If the level is between named values, then an integer is appended to the uppercased name. Examples: WarnLevel.String() => "WARN" (WarnLevel-2).String() => "WARN-2" type LevelRef struct { // Has unexported fields. } A LevelRef is a reference to a level. LevelRefs are safe for use by multiple goroutines. Use NewLevelRef to create a LevelRef. If all the Handlers of a program use the same LevelRef, then a single Set on that LevelRef will change the level for all of them. func NewLevelRef(l Level) *LevelRef NewLevelRef creates a LevelRef initialized to the given Level. func (r *LevelRef) Level() Level Level returns the LevelRef's level. If LevelRef is nil, it returns the maximum level. func (r *LevelRef) Set(l Level) Set sets the LevelRef's level to l. type Logger struct { // Has unexported fields. } A Logger generates Records and passes them to a Handler. Loggers are immutable; to create a new one, call New or Logger.With. func Default() *Logger Default returns the default Logger. func FromContext(ctx context.Context) *Logger FromContext returns the Logger stored in ctx by NewContext, or the default Logger if there is none. func New(h Handler) *Logger New creates a new Logger with the given Handler. func With(attrs ...any) *Logger With calls Logger.With on the default logger. func (l *Logger) Debug(msg string, args ...any) Debug logs at DebugLevel. func (l *Logger) Enabled(level Level) bool Enabled reports whether l emits log records at level. func (l *Logger) Error(msg string, err error, args ...any) Error logs at ErrorLevel. If err is non-nil, Error appends Any("err", err) to the list of attributes. func (l *Logger) Handler() Handler Handler returns l's Handler. func (l *Logger) Info(msg string, args ...any) Info logs at InfoLevel. func (l *Logger) Log(level Level, msg string, args ...any) Log emits a log record with the current time and the given level and message. The Record's Attrs consist of the Logger's attributes followed by the Attrs specified by args. The attribute arguments are processed as follows: - If an argument is an Attr, it is used as is. - If an argument is a string and this is not the last argument, the following argument is treated as the value and the two are combined into an Attr. - Otherwise, the argument is treated as a value with key "!BADKEY". func (l *Logger) LogAttrs(level Level, msg string, attrs ...Attr) LogAttrs is a more efficient version of Logger.Log that accepts only Attrs. func (l *Logger) LogAttrsDepth(calldepth int, level Level, msg string, attrs ...Attr) LogAttrsDepth is like Logger.LogAttrs, but accepts a call depth argument which it interprets like Logger.LogDepth. func (l *Logger) LogDepth(calldepth int, level Level, msg string, args ...any) LogDepth is like Logger.Log, but accepts a call depth to adjust the file and line number in the log record. 0 refers to the caller of LogDepth; 1 refers to the caller's caller; and so on. func (l *Logger) Warn(msg string, args ...any) Warn logs at WarnLevel. func (l *Logger) With(attrs ...any) *Logger With returns a new Logger whose handler's attributes are a concatenation of l's attributes and the given arguments, converted to Attrs as in Logger.Log. type Record struct { // Has unexported fields. } A Record holds information about a log event. func NewRecord(t time.Time, level Level, msg string, calldepth int) Record NewRecord creates a new Record from the given arguments. Use Record.AddAttr to add attributes to the Record. If calldepth is greater than zero, Record.SourceLine will return the file and line number at that depth. NewRecord is intended for logging APIs that want to support a Handler as a backend. Most users won't need it. func (r *Record) AddAttr(a Attr) AddAttr appends a to the list of r's attributes. It does not check for duplicate keys. func (r *Record) Attr(i int) Attr Attr returns the i'th Attr in r. func (r *Record) Attrs() []Attr Attrs returns a copy of the sequence of Attrs in r. func (r *Record) Level() Level Level returns the level of the log event. func (r *Record) Message() string Message returns the log message. func (r *Record) NumAttrs() int NumAttrs returns the number of Attrs in r. func (r *Record) SourceLine() (file string, line int) SourceLine returns the file and line of the log event. If the Record was created without the necessary information, or if the location is unavailable, it returns ("", 0). func (r *Record) Time() time.Time Time returns the time of the log event. type TextHandler struct { // Has unexported fields. } TextHandler is a Handler that writes Records to an io.Writer as a sequence of key=value pairs separated by spaces and followed by a newline. func NewTextHandler(w io.Writer) *TextHandler NewTextHandler creates a TextHandler that writes to w, using the default options. func (h TextHandler) Enabled(l Level) bool Enabled reports whether l is less than or equal to the maximum level. func (h *TextHandler) Handle(r Record) error Handle formats its argument Record as a single line of space-separated key=value items. If the Record's time is zero, it is omitted. Otherwise, the key is "time" and the value is output in RFC3339 format with millisecond precision. If the Record's level is zero, it is omitted. Otherwise, the key is "level" and the value of Level.String is output. If the AddSource option is set and source information is available, the key is "source" and the value is output as FILE:LINE. The message's key "msg". To modify these or other attributes, or remove them from the output, use [HandlerOptions.ReplaceAttr]. Keys are written as unquoted strings. Values are written according to their type: - Strings are quoted if they contain Unicode space characters or are over 80 bytes long. - If a value implements [encoding.TextMarshaler], the result of MarshalText is used. - Otherwise, the result of fmt.Sprint is used. Each call to Handle results in a single, mutex-protected call to io.Writer.Write. func (h *TextHandler) With(attrs []Attr) Handler With returns a new TextHandler whose attributes consists of h's attributes followed by attrs. Beta Was this translation helpful? Give feedback. 111 You must be logged in to vote 6 1 6 [?] 131 4 29 Replies 27 comments * 116 replies Oldest Newest Top edited [7] evanphx Aug 29, 2022 - hclog developer here! Curious about seeing if the stdlib could mostly contain the interface surface area for different implementations to step into. slog.Logger could be something an implementation of that interface. Were that route taken, I'd propose the surface area that looks something like this: type Logger interface { Log(level int, msg string, args ...any) Debug(msg string, args ...any) Info(msg string, args ...any) Warn(msg string, args ...any) Error(msg string, args ...any) With(args ...any) Logger } var DefaultLogger Logger func Debug(msg string, args ...any) { DefaultLogger.Debug(msg, args...) } // Toplevel for Info, Warn, Error, and With func InContext(ctx context.Context, log Logger) context.Context { ... } func FromContext(ctx context.Context) Logger { ... } // returns DefaultLogger if none available Beta Was this translation helpful? Give feedback. 15 You must be logged in to vote 9 3 replies @akshayjshah edited akshayjshah Aug 30, 2022 - Curious about seeing if the stdlib could mostly contain the interface surface area for different implementations to step into. Is this suggesting that the slog package export the Logger interface you outline, which third-party packages may then implement ("step into")? If so, the interface you propose ends up being fairly slow because passing ints, strings, and other values as any often allocates. These allocations are problematic because they're nearly impossible to avoid. Even if debug-level logging is disabled, logger.Debug("some high-volume thing happened", "key1", someString, "key2", someOtherString) will make multiple allocations just to call the logger implementation, which then realizes that debug-level logs are disabled and does nothing. These unavoidable allocations make logging difficult on code paths that need to be fast. This is often just fine - most code doesn't need to be particularly high-performance. If we're going to put structured logging into the stdlib, though, the logger interface should probably expose enough internals to be useful to performance-sensitive applications. Beta Was this translation helpful? Give feedback. 9 @evanphx evanphx Aug 30, 2022 - That's a fair concern! Perhaps adding In(level int) bool to allow a user to guard the statements in high throughput areas is prudent here. It's been a little while since I benchmarked the allocations of string->any and int->any conversions, which are the 2 vast majority of argument cases. With the more complex APIs, the trade is carefully crafting them to not allocate and having them sometimes suffer ergonomically and decrease the developer usage. Beta Was this translation helpful? Give feedback. @evanphx evanphx Aug 30, 2022 - Looking at the benchmarking, the only allocation today is is creating the slice it appears, no individual allocs for the arguments. Beta Was this translation helpful? Give feedback. [138] vearutop Aug 30, 2022 - I have a use case where I would like to control logging level based on context (e.g. enabling Debug for requests with magical header). Seems I can do this by creating Logger instance using debug-enabled Handler and then making NewContext with it. However, this context instrumentation needs to happen before any other (for example context may already have a Logger.With(...)) to avoid loss of state. Maybe it would make sense to allow replacing Handler using WithHandler(h Handler) *Logger to make such manipulations more flexible. Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote 2 replies @jba jba Sep 3, 2022 Maintainer Author - Your WithHandler is the same as slog.New. To change the level of a Logger while preserving the attributes (which live in the Handler actually), you could use something like this LevelHandler, which wraps an existing Handler with a new Enabled method: type LevelHandler struct { level slog.Level handler slog.Handler } func NewLevelHandler(level slog.Level, h slog.Handler) *LevelHandler { return &LevelHandler{level, h} } func (h *LevelHandler) Enabled(level slog.Level) bool { return level <= h.level } func (h *LevelHandler) Handle(r slog.Record) error { return h.handler.Handle(r) } func (h *LevelHandler) With(attrs []slog.Attr) slog.Handler { return NewLevelHandler(h.level, h.handler.With(attrs)) } Then your code would look like logger2 := slog.New(NewLevelHandler(slog.DebugLevel, logger1.Handler())) Beta Was this translation helpful? Give feedback. 2 @jba jba Sep 3, 2022 Maintainer Author - The question is, is this common enough to warrant built-in support? Beta Was this translation helpful? Give feedback. 1 [138] vearutop Aug 30, 2022 - What would happen if kvs ...any arguments are not well-formed (odd number or key is not a string)? Beta Was this translation helpful? Give feedback. 3 You must be logged in to vote 20 replies @evanphx evanphx Aug 30, 2022 - Amusingly I completely forgot how we handled this in hclog! Unpaired values are handled by padding the list up with a MISSING_KEY value, effectively the same as !BADKEY. If a key value is not a string, we put it through fmt.Sprint to use the normal string conversion logic. So in fact hclog does not panic, rather it tries it's darnest to just make it work. I forgot mostly because the logic evolved over time and became more forgiving, likely due to the same concerns voiced here. Beta Was this translation helpful? Give feedback. 2 @smlx smlx Aug 31, 2022 - So in fact hclog does not panic, rather it tries it's darnest to just make it work. But that is not working. It's printing MISSING_KEY instead of the actual key that provides value in your logs. Beta Was this translation helpful? Give feedback. 1 @evanphx evanphx Aug 31, 2022 - Eh? Not sure what you mean? In this case, it still prints all the data, it just assumes the unmatched argument is a value. Beta Was this translation helpful? Give feedback. @thockin thockin Sep 11, 2022 - Logr also pads the variadic to an even number and also verifies that all keys are actually strings. Seems sufficient so far. Beta Was this translation helpful? Give feedback. @sethgrid sethgrid Sep 11, 2022 - my former employer's in-house logger's solution for that was to produce an error key -- and we alerted on all errors. This would catch any error as it went out. I like the MISSING_KEY solution better :) Beta Was this translation helpful? Give feedback. View more edited [638] willfaught Aug 30, 2022 - Looks promising. Some thoughts: But more important than any traction gained by the "frontend" is the promise of a common "backend." An application with many dependencies may find that it has linked in many logging packages. If all of the packages support the handler interface we propose, then the application can create a single handler and install it once for each logging library to get consistent logging across all its dependencies. Since this happens in the application's main function, the benefits of a unified backend can be obtained with minimal code churn. We hope that this proposal's handlers will be implemented for all popular logging formats and network protocols, and that every common logging framework will provide a shim from their own backend to a handler. Then the Go logging community can work together to build high-quality backends that all can share. Why is this being proposed now? Why not 5-8 years ago? This seems to tout handlers as an innovation over the state of the art. Do none of the existing solutions have a comparable handler design? Will this effort to establish a common interface for the community extend to other problem domains, like audio? The program then outputs three log messages augmented with key-value pairs. In my opinion, the loose ...any key/value pairs parameter design, like in go-kit, feels a little too "loose". I understand going with it, because I can't think of a better way to do it either, but it always felt like there was a better way out there somewhere. It's not worth getting hung up on, but I'd be very interested in any viable alternatives. What happens if a value is missing? The second logs at the Error level, passing an error and a key-value pair. What if there isn't a Go error value when logging an error condition? What value should be used? FromContext returns the default logger if it can't find one in the context. Could this hide configuration or setup mistakes, where there should have been a logger, but there wasn't? InfoLevel Level = 30 DebugLevel Level = 31 Why Is DebugLevel 31, when the other levels increment by 10? TextHandler emits attributes as KEY=VALUE What is the exact intended initial format, whether or not you want to document it? What is the order of the fields? Are the "built-in" fields first? Are the pairs delimited by a space? Are strings containing whitespace quoted? JSONHandler writes line-delimited JSON objects Is the JSON minimized? What is the order of the fields? Are the "built-in" fields first? ReplaceAttr can be used to change the default keys of the built-in attributes What are the default keys? Functions like Any and Int construct slog.Attr values slog.LogAttrs Nit: Consider using the names Field/Fields, like zap. "Attr" is a little more abstract than "field," and "WithFields" is a little clearer and reads a little better than "WithAttrs," in my humble opinion. Any(k string, v any) Attr Why not include variants for all built-in types, like Int8, Rune or Complex64? Would generics help here? The boolean AddSource option controls whether the file and line of the log call. Have you considered the potential demand for a stack trace option (as opposed to just the current file name and line)? func (l *Logger) With(attrs ...any) *Logger I don't see a Logger.WithAttrs variant. Is that because it wouldn't help avoid allocations? If so, why is that? Beta Was this translation helpful? Give feedback. 3 You must be logged in to vote 23 replies @willfaught willfaught Sep 7, 2022 - A common, popular logger design that has an error parameter That would be logr I don't think that design counts as common, as I showed with the design survey; only logr has this design. I don't think that design counts as popular, either, but I acknowledge that logr is one of the more notable log packages, albeit in the lowest "tier" of notable (i.e. less than logrus and zap; at the same level as go-kit). No, the main point is that it's more convenient. That is just a side benefit. No, I assumed that none of them were. That is what I meant when I said the percentages alone would be evidence for an error argument: because even if none were mistakes, programmers would have an easier time overall. OK, so this isn't about foot-guns, or about user-perceived convenience that we've measured, it's about observing that supplying error values correlates quite a bit with a particular log level, specializing part of the design around that particular use case, and assuming that users will find that more preferable. The correlation threshold for doing this is ~2/3 of the time; ~4/10 of the time is too low. Let's consider your zap numbers again. Your analysis takes into account the convenience that the error parameter would have had ~2/3 of the time when there was an error value, but it doesn't take into account the inconvenience it would have had ~1/3 of the time when there wasn't an error value. It also doesn't take into account the design downsides of having a different method signature from the other levels. While weighing those trade-offs does involve some subjectivity, I think the consensus of the community is against the design of logr, and against this design, in this particular respect. Beta Was this translation helpful? Give feedback. 2 @smlx smlx Sep 8, 2022 - It also doesn't take into account the design downsides of having a different method signature from the other levels. I also find this choice of inconsistency confusing. In my experience you often want to log errors at any log level, not just Error. This design also implies that if you have an error then you should be logging at Error, which is not correct. Beta Was this translation helpful? Give feedback. 4 @jba jba Sep 8, 2022 Maintainer Author - It's new and unfamiliar. There's evidence that it is a net benefit. (It makes some things worse, but more things better.) Let's try it. There will be plenty of time to kick the tires. Beta Was this translation helpful? Give feedback. 3 @hherman1 hherman1 Sep 11, 2022 - Putting a vote in for the convenience of the error argument. Not clear to me why the slight inconsistency with other log levels matters. Beta Was this translation helpful? Give feedback. @thockin thockin Sep 11, 2022 - As one of the logr authors, I can see both sides of this. My own argument AGAINST the logr style error arg is the need to give it a hard-coded "magic" key ("err"). I can buy the argument that fewer magic keys would be better. The counter argument is that there is value in a consistent name across all call-sites. logr was informed by kubernetes, which really wanted the consistency and really does have an error in almost all Error() calls Beta Was this translation helpful? Give feedback. View more edited [972] akshayjshah Aug 30, 2022 - I'm glad to see this discussion reignited! I'm one of the original zap authors, and may have a little additional context to share. Points below notwithstanding, this proposal seems carefully thought-through and researched. Many thanks to the authors for all the work they've put in Previous art Peter Bourgon and Chris Hines made a proposal with similar goals in 2017. The overall situation hasn't changed much since then - if anything, the proliferation of log-like APIs in distributed tracing packages has made it even worse. If the authors of this proposal haven't seen the previous doc, it's worth reviewing. I'm not sure if Chris and Peter are still interested in logging, but their perspectives would be valuable here. I particularly liked the very small interface they proposed and the Valuer interface, even if both come at some performance cost. Namespacing I notice that this proposal doesn't include support for namespacing key-value pairs. For example, we might want a gRPC interceptor to produce JSON records like this: { ... standard fields ..., "grpc": {"method": "some.package/Method", ... other gRPC stuff...}} Zap exposes this via Namespace. At least at Uber, this was critical for practical use: like many storage backends, ElasticSearch stops indexing if a field changes type. Implicitly, this forces each binary to have a schema for its log output. It's very hard to do this if dozens of loosely-coordinated packages are writing to a shared, flat keyspace. Namespacing cuts the problem into manageable pieces: each package uses its own name as a namespace, within which it can attempt to adhere to some implicit schema. This problem is significant enough that people have attempted to write static analysis to mitigate it. Of course, not all storage engines have this limitation - but many do. Food for thought. First-class error support The error interface is infamously common in Go code, and logging errors is also common. It seems worth building in special support, rather than forcing users to call err.Error() eagerly. This also opens the door to a richer structured representation of errors. For example, the proposed multi-error support in the stdlib will be nearly unreadable if users shove err.Error() into JSON. If the old x/ errors proposal for automatically capturing stacktraces re-emerges, we'd also want a sane way to include them in structured log output. Zap spends a fair bit of effort to special-case go.uber.org/multierr and github.com/pkg/errors to paper over some of these complexities. Edit: on a second reading, I see the error argument to slog.Error. I still think that it may be valuable to have a func Error(err error) Attr to use with other log levels (particularly debug). Delegating serialization & complex objects It's convenient for end users to let types implement their own serialization logic (a la json.Marshaler or zap.ObjectMarshaler). I don't see an equivalent here. How do you imagine types customizing their representation in logs? Implementing encoding-specific interfaces like json.Marshaler is difficult, since there's no telling what encoding the owner of main prefers. Should packages expose a function that allocates a []log.Attr for each loggable type? Is this wholly out of scope? Do users log each field of interest by hand? Generics My experience with zap is that the mere existence of a faster API, even it's painfully verbose, puts a lot of pressure on developers to use it. Making the lower-allocation API as ergonomic as possible has a disproportionate effect on developer experience. One of my enduring regrets about zap is its fussy API for producing fields. Nowadays, this seems like the sort of problem we ought to be able to solve nicely with a type Attr[T any] struct { key string value T } Minimizing allocations would require a fast path through marshaling that doesn't go through any, which I think would require support for type switches on parametric types. Personally, I'd love to see a sketch of what might be possible with #45380. vet support It'd be nice to also have go vet check for mistakes in the friendlier, alternating key-value API. People don't make many mistakes in practice, but it's nice to have a little extra assurance that you haven't left off an argument or mixed up types. Perhaps there's space for function names as unique as printf and friends, so that the same vet rule could apply to any similarly-named functions? It's great though! Even with the concerns above, I want to reiterate how encouraging this is. Large sections of the Go community seem to really want structured logs, and the current fragmentation is painful. Working around the limitations of a single stdlib implementation will be much better than working around the quirks of the many structured loggers today. Edit: I realize that this proposal isn't a referendum on structured logging in general. Nevertheless, as one of the perpetrators of zap, I do feel compelled to say that I'm personally still not convinced that the fussiness of structured logging is worth it. On balance, I prefer leaving this degree of structure to distributed tracing and Prometheus-style tagged metrics and keeping logs as simple printfs. [?] I seem to be out of step with the rest of the community on this, though, and I do see this proposal as a step forward for structured logging. Beta Was this translation helpful? Give feedback. 22 You must be logged in to vote [?] 28 1 5 replies @jba jba Aug 30, 2022 Maintainer Author - Peter Bourgon and Chris Hines made a proposal with similar goals... I just re-read it. Lots of good ideas. A number of concerns, mostly convenience and performance, pushed us in a different direction. I notice that this proposal doesn't include support for namespacing key-value pairs. Thanks for explaining this Zap feature. I think you could get a similar effect by writing a handler that wraps an existing handler and prepends a fixed string to each key. (After all, not all output formats are hierarchical.) That's not quite what you have, but may be good enough for most uses. I still think that it may be valuable to have a func Error(err error) Attr to use with other log levels (particularly debug). We went back and forth on this, and ultimately decided to leave it out. We can always add it if needed. How do you imagine types customizing their representation in logs? Implementing encoding-specific interfaces like json.Marshaler is difficult, since there's no telling what encoding the owner of main prefers. I think implementing json.Marshaler and encoding.TextMarshaler should suffice. But I agree that a method that returned a []Attr would be more encoding-agnostic. Minimizing allocations would require a fast path through marshaling that doesn't go through any, which I think would require support for type switches on parametric types. Totally agree. It'd be nice to also have go vet check for mistakes in the friendlier, alternating key-value API. We'll probably do that. Beta Was this translation helpful? Give feedback. @akshayjshah akshayjshah Aug 30, 2022 - Thanks for explaining this Zap feature. I think you could get a similar effect by writing a handler that wraps an existing handler and prepends a fixed string to each key. (After all, not all output formats are hierarchical.) That's not quite what you have, but may be good enough for most uses. For sure! Whatever the form of the output, though, IMO it's important to make the namespacing mechanism obvious and easy to use. If it requires writing a custom handler wrapper, very few packages will do it. (I assume that we're aiming to have slog support common scenarios without bespoke wrapper types; otherwise ReplaceAttr could also be a custom handler wrapper.) I think implementing json.Marshaler and encoding.TextMarshaler should suffice. But I agree that a method that returned a []Attr would be more encoding-agnostic. If we're so careful about the performance of the binary writing the logs, presumably we're also interested enabling good end-to-end performance for a log search and ingestion pipeline? In that case, we might want to support binary encodings too - msgpack in particular is fast, doesn't require a schema, and is easy to convert to JSON for manual inspection. If binary encodings are out of scope, then it'd be nice to have a package (on par with encoding/json) for this logfmt-inspired text format. We'll also want to be able to write log processing software in Go, so having a logfmt.Marshaler and logfmt.Unmarshaler would be nice. (The logfmt package can delegate to encoding.TextMarshaler and encoding.TextUnmarshaler if the more specific interfaces aren't implemented, as json does today.) Beta Was this translation helpful? Give feedback. 2 @rabbbit rabbbit Sep 1, 2022 - I'm a biased zap user but would like to +1 @akshayjshah ideas on: * namespacing: it is quite frustrating when you import a new library, it starts logging, and now either all my dashboards are garbage, or just disappear when ELK invisibly stops ingesting. This is painful when you don't control either of the libraries. * specialized func Error(err error) Attr. It's surprisingly pleasant not to even have to deal with naming the error (slog.Info("e", err.Error()) || slog.Info("err", err.Error()) || slog.Info("error", err.Error()) vs just slog.Info(slog.Err, err) Beta Was this translation helpful? Give feedback. 5 @AndrewHarrisSPU edited AndrewHarrisSPU Sep 2, 2022 - Nevertheless, as one of the perpetrators of zap, I do feel compelled to say that I'm personally still not convinced that the fussiness of structured logging is worth it. On balance, I prefer leaving this degree of structure to distributed tracing and Prometheus-style tagged metrics and keeping logs as simple printfs. [?] This made me grin. FWIW, my own experience with zap was that I did end up finding structured logging useful; I enjoyed it a lot more when I switched from zap's sugared logger to DIY, local sugar. Beta Was this translation helpful? Give feedback. 1 @thockin thockin Sep 11, 2022 - Regarding a specific method for types to marshal log-data -- logr defines such a method, too. It seemed "obviously useful", so +1 on that Beta Was this translation helpful? Give feedback. [113] seankhliao Aug 30, 2022 Collaborator - ReplaceAttr func(a Attr) Attr This allows changing or omitting an Attr, but not splitting it out into multiple. At our company, logging calls take a context.Context, and TraceID/SpanID (and other metadata) is extracted from there and injected into log records. We do it this was as it's easier to remember, and span ids aren't yet visible at higher levels (eg wrapping handlers). type Level int I'd like to see it implement encoding.TextMarshaler / encoding.TextUnmarshaler for use in flags type Handler interface I'd like to see a standard handler that forwards to testing.TB.Logf func SetDefault zerolog provides a global logger as a separate package, I've found it quite useful in catching if you use the wrong logger (global vs context-derived). --------------------------------------------------------------------- The other recent (big?) project that I'm aware of in logging standardization is OpenTelemetry's Log Data Model, which has recently been stabilized. There they chose a different mapping of Levels / Severity to numbers, I wonder if it would be worth using the same numbering as theirs? https://github.com/open-telemetry/opentelemetry-specification/blob/ main/specification/logs/data-model.md#field-severitynumber --------------------------------------------------------------------- I'll echo the desire for namespaced key-values, but I think having it on the handler level isn't quite right, as there are cases when you want to write into different namespaces partitioned by business logic rather than hard package boundaries. For JSON it looks halfway possible with Any, maybe there could be another one for Attrs(key string, attrs...Attr) Attr? Beta Was this translation helpful? Give feedback. 4 You must be logged in to vote 8 replies @willfaught willfaught Sep 1, 2022 - Perhaps there should be (*Logger).WithNamespace(string) *Logger and NewLoggerNamespace(string) *Logger that prepend/wrap namespaces around keys. Beta Was this translation helpful? Give feedback. @jba jba Sep 6, 2022 Maintainer Author - I'll echo the desire for namespaced key-values, but I think having it on the handler level isn't quite right, as there are cases when you want to write into different namespaces partitioned by business logic rather than hard package boundaries. Can you give an example of how this would look? Beta Was this translation helpful? Give feedback. @seankhliao seankhliao Sep 6, 2022 Collaborator - I've seen 2 patterns used: * log with dotted key strings and rely on the backend to merge them: in middleware log = log.With("workflow.request_tyoe","...", "request.source", "...") and later log.Info("done", "workflow.result") but this can be somewhat difficult to process properly * update a context object that's passed along and leave it to the final caller or logger processing hooks to expand it into the appropriate fields, the above log would be {"msg":"done", "request": {"source":"..."},"workflow": {"request_type":"...","result":"..."}} would be really nice if somehow interoperated with opentelemetry's attribute package, our developers are already exposed to that for annotating traces/spans. Beta Was this translation helpful? Give feedback. @hherman1 edited hherman1 Sep 11, 2022 - What if there was just a new Attr type that was a namespace attr, and supported passing in a collection of sub-attrs? Seems performant + tiny extension to the existing api Beta Was this translation helpful? Give feedback. @hherman1 hherman1 Sep 11, 2022 - E.g 'sllog.Namespace(ns string, attrs Attr...) Beta Was this translation helpful? Give feedback. View more [122] sagikazarmark Aug 30, 2022 - Thanks for starting this discussion. The sheer amount of logging libraries out there seems to be an indicator that there should be a solution for this in the standard library. A couple thoughts about the proposal: OpenTelemetry I think it would make sense to look at the work the OpenTelemetry working group does...at least to make sure this proposal is not incompatible with what they are working on. I know they don't focus on libraries itself at the moment, but a wire format for logs. Sugared logger Personally, I'm quite fond of Zap's default logger enforcing attributes to be Field instances. Strong type safety sounds like a good default. For people who are willing to sacrifice that can always use the Sugared logger. Maybe reflecting that in this proposal would also make sense by creating separate implementations. Attr I was wondering if it would make sense to make Attr an interface: type Attr interface { Key() string Value() string } That way converting a value to string could be handed off to a custom Attr implementation. I don't know if that would affect allocations though. Maybe they would, but it's good enough for Zap. This would also allow to create separate types instead of using a single Attr with an any value. I was wondering what the purpose of Kind is. Where would it be useful? Context Personally, I prefer extracting information from the context in the logger, not the other way around. Let's say a context travels through a set of layers, each of them adding information to the context that you would like to log. The best place to extract all those information is at the place you actually want to log something. Therefore I'd want to pass the context to the logger somehow, not the other way around. For example, I could implement a handler that extracts all the relevant information from the context. The only question is: how do we pass a context to the logger? Beta Was this translation helpful? Give feedback. 4 You must be logged in to vote 3 10 replies @jba jba Sep 1, 2022 Maintainer Author - I see the motivation for having Logger methods take a context, but we don't want to couple Loggers to contexts. One way to get the same effect is to add the attributes to a new Logger. That is, instead of (or in addition to) ctx = context.WithValue(ctx, k, v) you would write ctx = slog.NewContext(ctx, slog.FromContext(ctx).With(k, v)) Another way is to add the context as an attribute, and write a Handler that looks for it: logger.Info("hello", "ctx", ctx) A third way is to wrap slog in your own API, like github.com/bool64/ ctxd, linked above: func (l *MyLogger) Info(ctx context.Context, msg string, keysAndValues ...any) { l.slogLogger.Info(msg, append(keysAndValues, /* extract from ctx here (or add "ctx", ctx) */)) } Is there a use case not covered by these techniques? Beta Was this translation helpful? Give feedback. 1 @AndrewHarrisSPU AndrewHarrisSPU Sep 1, 2022 - I sort of wonder if there are more general ideas here - one pattern is injecting a logger into something, another is extracting Attrs from something and returning a new logger integrating them. In this sense context is a bit of a special case because it's a dynamic/ untyped K/V store, while for other types the pattern could be made generic or interface-oriented. Beta Was this translation helpful? Give feedback. @akshayjshah akshayjshah Sep 1, 2022 - Is there a use case not covered by these techniques? Having every log line also added to a distributed tracing backend, often via a nearly-identical structured logging API. Wrapping the backend in your own API (like ctxd) works on a module-by-module basis, but it doesn't help ensure that a loosely-coordinated collection of modules does this by default. Being able to set the default handler in slog to both write JSON to stdout (for example) and add data to OpenTracing spans would be very helpful to organizations trying to adopt distributed tracing. Beta Was this translation helpful? Give feedback. @jba jba Sep 2, 2022 Maintainer Author - Being able to set the default handler in slog to both write JSON to stdout (for example) and add data to OpenTracing spans would be very helpful to organizations trying to adopt distributed tracing. Why couldn't you do exactly that (and pass the context down as a Logger attribute)? How would Zap handle it? Beta Was this translation helpful? Give feedback. @sagikazarmark sagikazarmark Sep 3, 2022 - I see the motivation for having Logger methods take a context, but we don't want to couple Loggers to contexts. One way to get the same effect is to add the attributes to a new Logger. That is, instead of (or in addition to) ctx = context.WithValue(ctx, k, v) you would write ctx = slog.NewContext(ctx, slog.FromContext(ctx).With(k, v)) While this works, it requires you to add context values to a logger directly and most probably you would need to add them both to context and the logger, so in reality it would look like: ctx = context.WithValue(ctx, k, v) ctx = slog.NewContext(ctx, slog.FromContext(ctx).With(k, v)) IMO this is a code smell, packs the context with a new logger over and over again unnecessarily. Another way is to add the context as an attribute, and write a Handler that looks for it: logger.Info("hello", "ctx", ctx) Yes, this is getting closer to an ideal solution, but the problem is making sure not the entire context is outputted as is. If Attr were an interface, it could look like this: logger.Info("hello", slog.Context(ctx)) It could implement another interface as well: type Outputer interface { ShouldOutput() bool } If the attr implements this interface and the method returns false then it shouldn't be outputted. The custom attr could also carry some signal to the Handler that it contains a context. This or any other functionally identical solution could work. A third way is to wrap slog in your own API, like github.com/ bool64/ctxd, linked above: func (l *MyLogger) Info(ctx context.Context, msg string, keysAndValues ...any) { l.slogLogger.Info(msg, append(keysAndValues, /* extract from ctx here (or add "ctx", ctx) */)) } This is an alternative that works, but it makes me question the usefulness of the Handler interface. If the primary way to extend the logger is via Handlers, wrapping loggers sounds more like a hack to me, not a first class citizen solution to enhance loggers. Is there a use case not covered by these techniques? Option 3 seems closes to an ideal solution to me out of the three, but it would be nice to have a builtin solution to support context propagation. Beta Was this translation helpful? Give feedback. 1 View more [664] rogpeppe Aug 30, 2022 Collaborator - I think this is promising. A bunch of thoughts: * a few times in the docs, it talks about an Attr "having" a value. I'm not sure that's very helpful (and it wasn't very obvious what it meant to me). How about just calling it "nil" as that's presumably what it is under the hood. Then HasValue becomes redundant - it's just a.Kind() == NilValue * why not make the zero value of LevelRef usable? * I'd like to see more justification for the arbitrary-seeming level constants. Why 10 apart and not 1, 100 or 1000? Why no gap between Info and Debug? It seems to me like the arbitrary numbers make it easier to get things wrong and are less efficient (no trivial mapping to an array) * some mention could be made of how NaN and Infinity are handled when marshaling to JSON. * "Strings are quoted if they contain Unicode space characters" - they should surely be quoted if they contain a quote character too? The quote format should also be defined (presumably Go string syntax, but this could be made clear) * It seems a bit odd to me that the handler constructors are methods on the options, rather than just regular functions that take the options as arguments. If there are any other handlers defined externally, that's the pattern they'll need to use, and as methods, there's a distinct possibility that they might be pulled into a binary even when a json dependency isn't desired. Beta Was this translation helpful? Give feedback. 4 You must be logged in to vote 1 reply @jba jba Sep 5, 2022 Maintainer Author - a few times in the docs, it talks about an Attr "having" a value Agreed. We can remove Attr.HasValue and any talk of an Attr "having" a value. It seems a bit odd to me that the handler constructors are methods on the options, rather than just regular functions that take the options as arguments. This pattern avoids a nil argument to the constructor. It has some precedent in the standard library: net: Listen, ListenConfig.Listen go/printer: Fprint, Config.Fprint go/types: Config.Check (although there is no top-level Check) If there are any other handlers defined externally, that's the pattern they'll need to use Why? Beta Was this translation helpful? Give feedback. [664] rogpeppe Aug 30, 2022 Collaborator - Using generics In one of the comments above, there was mention of the possibility of using generics to mitigate some of the API blowup from having many explicitly different Attr constructors and methods on Attr to retrieve values. I'll expand a bit on my reply above. I think we can use generics to mitigate allocations and reduce the API, even in the absence of # 45380. Here's how the API might look. We'd remove the Duration, Time, etc constructors and methods and replace them with this: // Mk returns an Attr for the supplied value. // Mk does not preserve the exact type of integral values. All signed integers // are converted to int64 and all unsigned integers to uint64. Similarly, // float32s are converted to float64. // // It special-cases the following types to avoid allocations: // // bool // int, int8, int16, etc // uint, uintptr, uint8, uint16, etc // time.Duration // string // time.Time // float32, float64 // // However, named types are preserved. So given // // type Int int // // the expression // // log.Any("k", Int(1)).Value() // // will return Int(1). func Mk[T any](key string, value T) Attr // Value returns the value part of a. It panics if T is not [any] // and doesn't match the kind reported by a.Kind(). // // The above condition implies that T must be one of the // following types: // any, int, uint, time.Duration, time.Time, string, bool, float64 func Value[T any](a Attr) T Another possibility to make the performance implications of creating an Attr a bit clearer might be to define a constraint that allows all the known concrete types, and then keep Any but define the above Mk in terms of that: type Types interface { time.Duration | time.Time | int | uint | ... } func Mk[T Types](key string, value T) Attr Another possible spelling for Mk might be A (or if Attr was renamed to Field, F). As I mentioned in the original comment, it's possible to implement this API without incurring allocations, although there is some performance overhead. Beta Was this translation helpful? Give feedback. 7 You must be logged in to vote 1 4 replies @jba jba Sep 5, 2022 Maintainer Author - Thanks for finding this trick. (I had gotten as far as switching on *T, but never considered that any(val) wouldn't allocate.) One possibility is to replace Any with Any[T]. Any[any] would behave like the current Any. Most or all of the constructors could go away. I'm less convinced of Value[T](Attr). It's much more clumsy to write slog.Value[string](a) than a.String(), and the constraints on T make it no more powerful than the handful of methods we have now. It might be worth it if slog.Value[MyInt](slog.Any("k", 1)) would work, where MyInt's underlying type is int. But that would require switching on ~int. Beta Was this translation helpful? Give feedback. 1 @AndrewHarrisSPU edited AndrewHarrisSPU Sep 6, 2022 - But that would require switching on ~int. FWIW, there's an unsafe trick that bypasses Kind() or switching, and could be used on ~int: func value[T any](rhs any) T { var t T lhs := any(t) lhsPtr := (*uint64)(unsafe.Pointer(&lhs)) rhsPtr := (*uint64)(unsafe.Pointer(&rhs)) if *lhsPtr != *rhsPtr { panic("unsafe assertion") } return rhs.(T) } This must take an any because it relies on the first word of an interface being associated with dynamic type/index into runtime itab table. Any constraint on what an Attr can hold is exogenous to this call. Works on godbolt.org for a lot of things. I don't know if it's a great idea. edit: this is a dumb example, just a type assertion in practice - the trick is really that the unsafe pointer value for rhs can be grabbed and stored ahead of time Beta Was this translation helpful? Give feedback. @jba jba Sep 6, 2022 Maintainer Author - You might see if reflect.TypeOf(lhs) == reflect.TypeOf(rhs) works for you. The implementation of reflect.TypeOf is not much more than what you have. Anyway, this is an exact check, and to make slog.Value[MyInt] (slog.Any("k", 1)) work we need to ask if MyInt's underlying type is int. Beta Was this translation helpful? Give feedback. 1 @AndrewHarrisSPU edited AndrewHarrisSPU Sep 8, 2022 - You might see if reflect.TypeOf(lhs) == reflect.TypeOf(rhs) works for you Pragmatically it's fine. Mostly I've been interested/distracted in exploring what's possible where the FGG/FG paper suggests covariant method receiver types. I feel like the typing and type checking of Attrs is a case where exhaustive pattern matching on a type switch would be reasonable way to structure code, it's just not quite in the language. Another fuzzy idea, in slog this would be a bit nasty: type TypicalInt[T constraints.Signed ] int64 func (TypicalInt[T]) Typical() {} Attrs: type Attr struct { k string n int64 some interface{ Typical() } } func Any[T interface{ Typical() }]( key string, value T ) Attr { if v, ok := any(value).( interface{ UnwrapInt() int64 }); ok { return Attr{key, v.UnwrapInt(), value } // or nil for the last field is possible } return Attr{key, 0, value} } func (attr Attr) Int() int64 { if _, ok := attr.some.(interface{ UnwrapInt() int64 }); ok { return attr.n } panic("oops") } But, downstream, in myPkg importing slog, this is possible, and any TypicalInt is known to be convertible to int64 // package myPkg type myInt slog.TypicalInt[myInt] func(Int) Typical() {} func (n Int) UnwrapInt() int64 { return int64(n) } Dunno if this would actually help to coordinate backends / loggers or just pushes the problem elsewhere. It's messy, and it's type erasure and coupling. edit: fixed exporting Beta Was this translation helpful? Give feedback. [664] rogpeppe Aug 30, 2022 Collaborator - The docs should document what should happen when With is used to attach the same attribute key twice. Does it override the original attribute, get ignored, or is the behaviour undefined? Beta Was this translation helpful? Give feedback. 8 You must be logged in to vote 1 1 reply @jba jba Aug 31, 2022 Maintainer Author - No duplicate checking is done anywhere. Too expensive. The Record we deliver to your Handler will have a raw sequence of Attrs, exactly as you added them. A Handler can of course dedup if it wishes, but the ones we provide do not. I will make this clear in the next version of the doc. Beta Was this translation helpful? Give feedback. 2 [638] willfaught Sep 1, 2022 - "attrs" in func (l *Logger) With(attrs ...any) *Logger should be "args" like in func (l *Logger) Warn(msg string, args ...any) to not suggest that attrs must be Attrs. Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote 1 reply @jba jba Sep 1, 2022 Maintainer Author - Will do. Beta Was this translation helpful? Give feedback. edited [450] mvrhov Sep 1, 2022 - I hate the passing the attributes as key, values e.g kvs ...any from the bottom of the heart. This is going to be a bottomless pit of problems, Where one will mess up the key value pairs and the the hole log line will be messed up. I've messed up the string replacers key value pairs numerous times and Another thing is when logging a lot of data long lines are completely unreadable and one really needs to take a time to see on what goes with what. I really like what zerolog does it with "fluent" calls. and a function for each type. Beta Was this translation helpful? Give feedback. 9 You must be logged in to vote 2 4 replies @vearutop vearutop Sep 1, 2022 - I never had such a problem and I think that was due to using one line per pair convention. slog.Info("message", "foo", 1, "bar", 2, ) Beta Was this translation helpful? Give feedback. 1 @jba jba Sep 1, 2022 Maintainer Author - @mvrhov, I think the vet check will help a lot. It seems that experience with other, similar log packages has shown that this can work. Beta Was this translation helpful? Give feedback. @sagikazarmark sagikazarmark Sep 1, 2022 - Whenever I have to use an API like this, a part of me dies as well. This is where IMO Go takes simplicity too far, instead of making a type system/standard library that's able to represent key-value pairs efficiently. The result is that we are losing build time safety. But that's just my opinion. Beta Was this translation helpful? Give feedback. 7 @sebfan2 sebfan2 Sep 1, 2022 - I agree with this, instead of typed arguments we are relaying a custom and unique formatting style (1 argument the on first line then 2 augment pairs on the following lines) to make the code readable and vetting to ensure it's correct. We also miss out on formatting via standard go fmt. Beta Was this translation helpful? Give feedback. 5 [638] willfaught Sep 2, 2022 - For what it's worth, here's a stab at an alternative to the unstructured k/v design func(string, ...any): func (*Logger) Any(string, any) *Logger func (*Logger) Bool(string, bool) *Logger func (*Logger) Int(string, int) *Logger func (*Logger) String(string, string) *Logger // ... func (*Logger) With(...Field) *Logger func (*Logger) Log(Level, string) func (*Logger) LogDepth(int, Level, string) func (*Logger) Error(string) func (*Logger) Warn(string) func (*Logger) Info(string) func (*Logger) Debug(string) func (*Logger) Namespace(string) *Logger // Or perhaps "Wrap" instead of "Namespace" func Any(string, any) Field func Bool(string, bool) Field func Int(string, int) Field func String(string, string) Field // ... Example use: var pkgLogger *slog.Logger = companyLogger.Namespace("mypkg") // ... var endpointLogger *slog.Logger = pkgLogger.Namespace("myendpoint") // ... if err != nil { var errLogger = endpointLogger.Any("save error", err).Int("userid", user.ID) errLogger = errLogger.Bool("prod", env.Prod) errLogger.Error("cannot save") // Logs with "save error", "userid", and "prod" keys // ... if err != nil { errLogger.Any("cleanup error", err).Error("cannot cleanup") // Logs with "save error", "userid", "prod", and "cleanup error" keys } // ... } // ... if err != nil { endpointLogger.With( slog.Any("error", err), slog.Bool("baz", baz), slog.Int("bar", bar), slog.String("foo", foo), ).Error("cannot do thing") } Highlights: * Specifying keys and values is separate from specifying the level and message. * Specifying k/v's per line and per Logger is the same. * Every level convenience method has the same func(string) signature. * You can avoid allocations with typical use without having to deal with Field directly. * Compile-time checks for well-formed code. No linting required. * Error values are treated like any other interface value with any. Beta Was this translation helpful? Give feedback. 3 You must be logged in to vote 1 5 6 replies @jba jba Sep 3, 2022 Maintainer Author - Specifying keys and values is separate from specifying the level and message. We looked at a fluent API. It ends up being somewhat redundant and confusing when you look at the godoc. For example, you need a name for the type returned by Logger.With, although no one should ever be using that type directly. And to support logging a message with no attributes, you have to repeat all the methods of that type on Logger. Specifying k/v's per line and per Logger is the same. As with the current proposal, in the common case. LogAttrs is an optimization. Many, perhaps most, people will never use it. (At least, that is one way to look at it. We hear the folks who would rather do the extra typing to construct Attrs for more compile-time checking.) Beta Was this translation helpful? Give feedback. @willfaught willfaught Sep 4, 2022 - @deltamualpha: As much of a cop-out as it is to make a recourse to calling something "unidiomatic"... fluent interfaces are very unidiomatic for the standard library. Is the builder pattern un-idiomatic? The fluent design seems just like it with an extra chaining convenience. Although, in this case, the "chaining" is the attachment of fields to a logger, which the original design also has. This also has the (perhaps intentional) effect of making log.Info (sliceOfStrings...), where the slice is built up over time and then dropped into the logger, impossible. I think it would be []any in that case, right? This design accommodates the same accumulation scenario with Logger.With and []Field. Instead of accumulating strings, you accumulate Fields. I guess the fluent part could be left out, and only have Logger.With (...Field). Logger.Bool/etc. are just conveniences for doing that anyway. But I wouldn't favor that, for the same reason we want Info() and Warn() conveniences for Log(). --------------------------------------------------------------------- @jba: For example, you need a name for the type returned by Logger.With, although no one should ever be using that type directly. Doesn't your own Logger.With return a Logger? I'm not sure what you mean. And to support logging a message with no attributes, you have to repeat all the methods of that type on Logger. What's being repeated? Methods like Bool? Where are they being repeated? There's only one logger type in this design: Logger. I'm not sure what you mean. As with the current proposal, in the common case. I meant that they both happen in the same place/way. In your design, we can specify fields for the logger with Logger.With, and for the line with Logger.Log/Info/etc. In this design here, there's only one place to specify fields (Logger.With/Bool/etc.), so there isn't API duplication. Beta Was this translation helpful? Give feedback. @smlx smlx Sep 4, 2022 - We hear the folks who would rather do the extra typing to construct Attrs for more compile-time checking. Yes, please! Beta Was this translation helpful? Give feedback. @jba jba Sep 5, 2022 Maintainer Author - For example, you need a name for the type returned by Logger.With, although no one should ever be using that type directly. Doesn't your own Logger.With return a Logger? I'm not sure what you mean. OK, I didn't read your API carefully enough. If you have With return *Logger, then it has to allocate a new *Logger, and that means every line like logger.With(...).Info(...) will have to allocate, and performance will suffer. As an exercise, you might want to implement your design and benchmark it against Zap. The Zap project has some good benchmarks that you can adapt. When I said "you need a name for the type returned by Logger.With," I was thinking of an API where With returns an intermediate object that avoids allocating. It is possible to get good performance like that, but as I said, the API is messy. So in short, I agree your design is elegant, but I don't see how to make it fast. Beta Was this translation helpful? Give feedback. 3 @daenney daenney Sep 11, 2022 - It ends up being somewhat redundant and confusing when you look at the godoc. If a fluent API is the right fit (regardless of whether that's the case here specifically), we should not be avoiding that "because the godoc ain't pretty". Godoc needs fixing then, it's a valid way to design a package's API. It strikes me as really peculiar that the documentation generator's output would be a factor in the API design at this level. Beta Was this translation helpful? Give feedback. View more edited [446] AndrewHarrisSPU Sep 4, 2022 - should this be "panics if the value is not a time.Time"? func (a Attr) Time() time.Time Time returns the Attr's value as a time.Time. It panics if the value is not a time.Duration. -- I really like the Record/Handler/Logger arrangement. I think it's really sensible for thinking about and capturing middle-end structure. I'm unclear on whether to consider Record's [ time, level, depth, msg ] as Attrs - I think they are all morally Attrs, but are not emitted by the Record.Attrs method? For consistency, this could be useful (I don't think slog's Error severity should be the only place to expect "err"-keyed Attrs): func (r *Record) AddErr(err error) Appends Any("err", err) as with Logger.Error Bikeshedding, could imagine renaming Logger.LogDepth, Logger.LogAttrsDepth -> Logger.Recur, Logger.RecurAttrs. I think the Logger.Log... methods are crowded in documentation, and log.Recur( n, ... ) seems straightforward. The verbosity design doesn't feel immediately intuitive - Why decimal instead of coarse/fine bits? Why use negative verbosities like "WARN-2"? How does this interact with tracing? Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote 1 6 replies @jba jba Sep 5, 2022 Maintainer Author - Should this be "panics if the value is not a time.Time"? Thanks, fixed in the source. I think they are all morally Attrs Yes, but for efficiency they're separate. It is tempting to have a more unified design, but then handlers have to walk the sequence of Attrs to find certain keys (so they can format the record as TIMESTAMP LEVEL MESSAGE ..., for example), and it starts getting expensive. The verbosity design doesn't feel immediately intuitive It's an attempt to accommodate logr. Beta Was this translation helpful? Give feedback. 2 @AndrewHarrisSPU edited AndrewHarrisSPU Sep 5, 2022 - Why "Recur"? I don't get the name. My thinking is that (all? almost all?) Depth calls are paying attention to some kind of recurrence. Maybe Depth is better. It's also an attempt to set the LogDepth methods apart, the way they involve the call stack feels disjoint to me - it's always a little clever, and in various usages can be layers of cleverness. The methods LogDepth, LogAttrsDepth are more involved and finicky than their siblings Log and LogAttr, or their cousins Debug, Info, Warn, Error. This is pretty minor and bike-sheddy :) I can move on. Beta Was this translation helpful? Give feedback. @jhenstridge jhenstridge Sep 8, 2022 - I think the main use case for the Depth calls is for logging within utility functions (e.g. a helper that logs some information about an http.Request). The call site of the Log calls within the utility function is not particularly interesting, but the next frame up is. So the utility function uses LogDepth() to record that. Beta Was this translation helpful? Give feedback. 2 @jba jba Sep 8, 2022 Maintainer Author - Right, either a helper or a complete wrapper of this API. For instance, maybe you prefer to write Log(ctx, level, msg) instead of FromContext(ctx).Log(level, msg). We noticed a lot of wrappers in the wild. Beta Was this translation helpful? Give feedback. @AndrewHarrisSPU AndrewHarrisSPU Sep 8, 2022 - I was probably mistaken about how LogDepth would be commonly used. In the smallest way, nonplussed about the lexicographical closeness LogDepth in the docs or autocomplete, etc. Beta Was this translation helpful? Give feedback. View more [127] jhenstridge Sep 8, 2022 - It looks like this interface would integrate fairly well with systemd-journald's native protocol. In particular, the Handler.With() method looks like it'd make it possible to serialise the common attributes once and use it as a prefix for each log message datagram. One question that I didn't see mentioned is how duplicate attributes should be handled. The API obviously doesn't do anything to prevent them, but is there any behaviour a handler is expected to implement? For example, consider calls like: l.Info("message", "foo", "value1", "foo", "value2") l2 := l.With("foo", "value1") l2.Info("message", "foo", "value2") Are these log messages valid? Is the handler expected to log both values for the attribute? Is it allowed to log only one? If it does only log one value, is it free to choose which one? If it logs multiple values, is it expected to preserve the order? I realise that the answer might be "it is implementation defined", but if that's the case it should probably be spelled out what behaviour callers can rely on. Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote 1 reply @jba jba Sep 8, 2022 Maintainer Author - I think it does have to be left up to the implementation. For speed a Handler must be allowed to ignore duplicates. You can always wrap a Handler in another that pre-processes the list of Attrs however it wants. Beta Was this translation helpful? Give feedback. [494] danielorbach Sep 9, 2022 - Great idea, and a welcome addition to the standard library. I am wondering about support for Fatal, and message formatting. Or, are we supposed to use this package in combination with the existing log package? Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote 1 6 replies @danielorbach danielorbach Sep 9, 2022 - I'm surprised by the distinction. Isn't there a need for a separate level, for non-recoverable errors? This extra level can help automatic tools differentiate usual error-states from circumstances leading to process termination. Beta Was this translation helpful? Give feedback. 2 @willfaught willfaught Sep 9, 2022 - Agreed, a Critical level at least would be helpful for on-call engineers. Beta Was this translation helpful? Give feedback. [?] 1 @AndrewHarrisSPU edited AndrewHarrisSPU Sep 9, 2022 - Isn't there a need for a separate level, for non-recoverable errors? I think the compromise reached is that the Record model can accommodate levels more severe/granular than slog.ErrorLevel. These can be constructed with the Log... methods of a Logger, and the raw text representation of a Level is defined. Eventually, I think the slog.Logger API is unavoidably opinionated but the slog.Record model and slog.Handler interface anticipates a need to wrap or replace or bypass slog.Logger. It's pretty immediate to wrap Logger methods or a Logger instance, and really plausible to write a more lethal alternative to slog.Logger, or wrap a Handler to watch Records and blow up on something too severe (or, really, react to any kind of analysis on information embedded in a Record). Beta Was this translation helpful? Give feedback. @danielorbach danielorbach Sep 10, 2022 - Is this really the expectation? Primal/trivial usage would have packages explicitly rely on a Logger instance, given the package exposes both Handlers and an opinionated Logger. The problem arises when different packages each defines a different Logger alternative. To my opinion, the most important benefit of standard library code is its interoperability. This proposal's strength lies in the binding to Context, allowing (new/adapted) third-party packages to respect loggers set by my code. Note that the Logger's API does not only provide convenience methods to generate a Record struct, but also it holds contextual Attrs set before the call-site to the logging methods. Also, how can I control the text rendering of my own level (in a simple manner)? I fear this entails more than trivially defining another constant of type Level. Beta Was this translation helpful? Give feedback. @AndrewHarrisSPU edited AndrewHarrisSPU Sep 10, 2022 - Is this really the expectation? I think modularity is part of the design, the section on "Interoperating with Other Log Packages" mentions a few scenarios. I thought it was quirky that a Logger is not an interface, and it doesn't satisfy the Handler interface - we can spy on go.dev now, and it turns out a Logger has no state other than the Handler it holds. Note that the Logger's API does not only provide convenience methods to generate a Record struct, but also it holds contextual Attrs set before the call-site to the logging methods. This is where the quirkiness works out well, IMHO - the Logger itself does not immediately hold a set of prefixed Attrs; they are encapsulated by the Handler it immediately wraps. In turn, with the Logger.Handler method available to recover a Handler, it makes it a lot more sane to opt out of the Logger API opinions if/when/where someone has a need to do so. Also, how can I control the text rendering of my own level (in a simple manner)? It might be an unsatisfying approximation, but one could append a bool Attr( "Critical", true ) to a log entry. Not sure how to guarantee that Levels have one and only one textural representation otherwise. Beta Was this translation helpful? Give feedback. View more [345] lmittmann Sep 10, 2022 - Thanks for this interesting discussion. I am wondering if it would be better to make Attr an interface and have unexported concrete implementations for types instead. This would have the benefit, that e.g. Attr.AppendValue would not need to switch on Attr.Kind. Also the API size could be reduced. A quick prototype implementation of this idea is significantly faster. type Attr interface { Key() string Value() any AppendValue(dst []byte) []byte } func Int(key string, val int) Attr { return &intAttr{key: key, val: val} } type intAttr struct { key string val int } func (a *intAttr) Key() string { return a.key } func (a *intAttr) Value() any { return a.val } func (a *intAttr) AppendValue(dst []byte) []byte { return strconv.AppendInt(dst, int64(a.val), 10) } func BenchmarkAttr(b *testing.B) { b.Run("old", func(b *testing.B) { b.ReportAllocs() buf := make([]byte, 0, 128) for i := 0; i < b.N; i++ { attr := slog.Int("key", i) attr.AppendValue(buf) } }) b.Run("new", func(b *testing.B) { b.ReportAllocs() buf := make([]byte, 0, 128) for i := 0; i < b.N; i++ { attr := Int("key", i) attr.AppendValue(buf) } }) } BenchmarkAttr/old-8 47893233 24.30 ns/op 0 B/op 0 allocs/op BenchmarkAttr/new-8 100000000 11.69 ns/op 0 B/op 0 allocs/op Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote 0 replies This comment was marked as off-topic. Sign in to view @willfaught This comment has been hidden. Sign in to view [796] Eun Sep 11, 2022 - I would love to see something like log.Debug("Hello %Name, your balance is %BalanceEUR!", User{ Name: "Joe", Balance: 104, }) Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote 5 1 reply @nfx nfx Sep 11, 2022 - i don't believe it's going to perform well Beta Was this translation helpful? Give feedback. edited [310] code-n-go Sep 11, 2022 - I would love to be able to log a structure, and include something like a 'log:"-"' tag to ensure PII/PCI/PHI content doesn't make it to the output, with defined models. Beta Was this translation helpful? Give feedback. 3 You must be logged in to vote 1 0 replies [147] XANi Sep 11, 2022 - I also think there should be a Panic/Fatal level, else it will just lead to code log.Error("things are very on fire") panic() and I don't exactly want to wonder whether logger I used will have enough time to send/log the error before panic() crashes the app. It also allows said logger to have special treatment of those, like for example you could buffer a bunch of info level logs before sending but panic would trigger buffer flush and only return when the logging message is actually sent As for formatting, the lack of "*f" family of calls will just lead to a lot of log.Info(fmt.Sprintf("some value: %0.2f",value)) or log.Warn(fmt.Sprintf("%d/%d servers alive", alive,total),Int("s_alive",alive) ,Int("s_total",total)) in places where people want to have message that's doesn't have 12 digits of float fractions, is just readable for human operators, or just plainly be padded a little so browsing normal log (and do not make mistake of thinking people will only use it for structured logging, it's nice interface for general logging at a glance) isn't a pain. Beta Was this translation helpful? Give feedback. 5 You must be logged in to vote 1 5 replies @deltamualpha deltamualpha Sep 11, 2022 - But the whole point of structured logging is that you don't do string formatting on the values being logged. Obviously the API can't prevent someone from not using it as intended, but *f variants would be actively encouraging misuse, when someone in that position should just be using the existing log package. Beta Was this translation helpful? Give feedback. 2 @willfaught willfaught Sep 11, 2022 - I think the concern about documenting the "sync" behavior of the logging is valid. I can't recall if this is specified in the design, but it should be. Once slog.Error returns, all the data for it should be written and flushed. Following the call with os.Exit(1) should not lose log data in general. I agree with @deltamualpha on formatting. The point of structured logging is that associated values are, well, structured; you don't have to parse a string to get them out again. What you want is a log line of message="number of servers too low" alive=10 total=20, not message="10/20 servers alive". Beta Was this translation helpful? Give feedback. @AndrewHarrisSPU edited AndrewHarrisSPU Sep 11, 2022 - I think the concern about documenting the "sync" behavior of the logging is valid. I can't recall if this is specified in the design, but it should be. Some sync properties can be implemented in Handlers - both of the provided ones are documented to serialize. The flushing behavior seems trickier - I think Dave Cheney had a nicely written argument on this, generally that it's not really possible to get the semantics of something like Fatal right in the logging library code. Beta Was this translation helpful? Give feedback. @XANi XANi Sep 11, 2022 - But the whole point of structured logging is that you don't do string formatting on the values being logged. Obviously the API can't prevent someone from not using it as intended, but *f variants would be actively encouraging misuse, I just want to have one interface to logging. And I don't think having good messages and structured logging should be exclusive. "It would encourage" argument is not a very good one. Someone would just make a wrapper that's included by thousand packages with only reason being "standard slog is poor in features", just like current situation in logging. Alternatively making something like log.Infof("%d/%d servers alive", Int("s_alive",alive) ,Int("s_total",total)) work would still push people on the the right path without cutting features compared tolog. when someone in that position should just be using the existing log package. log is so simple it's useless. No log levels make it basically unusable from my perspective. Even the smallest app I wrote had a need to sometimes post a warning without killing or panicking the app, or to have a debug log. Beta Was this translation helpful? Give feedback. @willfaught willfaught Sep 11, 2022 - Some sync properties can be implemented in Handlers - both of the provided ones are documented to serialize. The flushing behavior seems trickier - I think Dave Cheney had a nicely written argument on this, generally that it's not really possible to get the semantics of something like Fatal right in the logging library code. It looks like klog has Flush and FlushAndExit funcs that could work equally well. You wouldn't be able to call os.Exit, but you could panic, and handle panics in main() where you call Flush. Beta Was this translation helpful? Give feedback. [902] LukeShu Sep 11, 2022 - In our internal logging library, we started out with a func WithLogger(ctx context.Context, logger Logger) context.Context/func GetLogger(ctx context.Context) Logger API similar to logr's NewContext/FromContext. However, we saw that this often ended up being used wrong, and transitioned from dlog.GetLogger(ctx).Warn ("msg") to dlog.Warn(ctx, "msg") and made GetLogger non-exported: // getLogger returns the logger associated with the Context, or else the fallback logger. // // You may be asking "Why isn't this exported? In some cases there might be debug or trace logging // in loops that where it's important to keep the performance impact as low as possible. As things // stand, each log statement will perform a Context lookup!" // // The reason is: Exporting it introduces the possibility of misuse, and so at this point exporting // it "for performance" would be premature optimization. If we ever do see this causing a // performance problem, then we can export it. But until then, let's make it hard to misuse. // // You see, it was exported back in the days before https://github.com/datawire/apro/pull/1818 (in // fact dlog.GetLogger(ctx).Infoln(...) was the only way to do it for a long time). What we saw with // that was that it's really easy to end up calling `logger = logger.WithField(...)` and ctx = // `dlog.WithField(ctx, ...)` separately and having the separate logger and the ctx drift from // eachother (often, you'll do the former, not updating the ctx, then later someone passes the ctx // to another function, so that function's logger doesn't have the updates). This is a misuse. func getLogger(ctx context.Context) Logger { And so I believe that slog should have top-level functions that take a context as the first argument, in addition to methods on the *Logger object: func Log(ctx context.Context, level Level, message string, kvs ...any) func LogAttrs(ctx context.Context, level Level, message string, attrs ...Attr) func Info(ctx context.Context, message string, kvs ...any) func Warn(ctx context.Context, message string, kvs ...any) func Debug(ctx context.Context, message string, kvs ...any) func Error(ctx context.Context, message string, err error, kvs ...any) Because of the wide use that the standard library would see, I don't believe that having an exported FromContext it would be a premature optimization. However, I believe that its use should be discouraged in favor of the above top-level functions. Perhaps even split them in two packages; log/slog for structured logging without contexts (exposing the *Logger type and the NewContext and FromContext methods, but not the above context-taking functions), and log/clog for context-oriented structured logging (exposing the above context-taking functions, but not NewContext, FromContext, or the *Logger type); so that codebases can easily enforce which paradigm they want to follow by checking which packages get imported. Reference: https://pkg.go.dev/github.com/datawire/dlib/dlog Beta Was this translation helpful? Give feedback. 4 You must be logged in to vote 2 1 1 1 reply @willfaught willfaught Sep 11, 2022 - I think it's important that the level functions and methods match. Do your log methods also take a context? I see a few issues with this: taking a context param implies that logging can be canceled, which should never be done; it requires having a context, which you might not have; it integrates contexts into the core design, assuming that putting loggers into contexts is a core use case, which often times probably won't be the case; and it "infects" the API with contexts, which results in boilerplate code. Beta Was this translation helpful? Give feedback. edited [259] nfx Sep 11, 2022 - Context-based Logging API (extra API) the problem with slog.NewContext/slog.FromContext is the proliferation of slog.FromContext(ctx) into many places of the code, sacrificing readability for the sake of structured logging. Plenty of Go ecosystem uses contexts, so why not make a more native integration option with a less wordy interface? So why not //go:generate context-based API wrappers? We can also generate them into a different (sub)package, so that it's clear we're working with "context-based logging". package clog // context-based structured log. just for illustration purposes. func Debug(ctx context.Context, msg string, args ...any) func Info(ctx context.Context, msg string, args ...any) func Error(ctx context.Context, msg string, err error, args ...any) ... func WithAny(ctx context.Context, key string, value any) context.Context func WithBool(ctx context.Context, key string, value bool) context.Context func WithDuration(ctx context.Context, key string, value time.Duration) context.Context this will help to follow the suite of log4j, which already has MDC on a per-thread basis / (docs) and has a standard way of enabling "per-request" log context. the readability difference between slog.FromContext(ctx).Info("foo") and clog.Info(ctx, "foo") is huge: func handle(w http.ResponseWriter, r *http.Request) { ctx := clog.With(r.Context(), "method", r.Method, "url", r.URL, "traceID", getTraceID(r)) //.. clog.Info(ctx, "something has happened", "success", true) } another practical insight from using structured logging in Go: adding just one attribute (ctx = clog.WithStr(ctx, "source", r.Source())) is the most common case for "reused loggers" and follows ctx = context.WithValue(ctx, foo, bar) pattern. Namespaces namespaces (types, packages) are more of static nature and there should not be WithNamespace(context.Context, string) method. P.S.: args ...any vs args ...Attr is really a matter of taste. Whatever decision you'll make - the community will follow. Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote 2 replies @Merovius edited Merovius Sep 11, 2022 - One issue with this is that it makes logging the most expensive, when it's disabled. Every log line now has to walk the context chain to find the logger it wants to log for. And if there is no such logger, it has to walk the entire chain to find that out. With slog.FromContext, that walk needs to happen once and then can get amortized over multiple log lines. Depending on your willingness to pass around a bare logger, very many. Beta Was this translation helpful? Give feedback. 3 @nfx nfx Sep 11, 2022 - @Merovius that's a good point. But this is why I'm proposing to code-generate it into a sub-package, so that developers can make an explicit choice about it. slog.FromContext/slog.NewContext wouldn't go anywhere. Beta Was this translation helpful? Give feedback. [259] nfx Sep 11, 2022 - IsTraceEnabled / IsDebugEnabled APIs Log4j has isTraceEnabled, that allows saving plenty of compute cycles in practice for a highly-loaded system. We've used it approximately as if slog.IsTraceEnabled() { state := someExpensiveTracingComputation() slog.Trace("intermediate state", "state", state) } and paid special attention to it in code reviews for performance-critical parts of our platform. Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote 1 reply @seankhliao seankhliao Sep 11, 2022 Collaborator - In the current design it would be: if logger.Enabled(slog.DebugLevel) { state := someExpensiveTracingComputation() logger.Debug("intermediate state", "state", state) } Beta Was this translation helpful? Give feedback. edited [880] narqo Sep 11, 2022 - I'm curious to know the rationale behind including the predefined logging levels into the package's API (both as a top-level functions and the Logger type's methods). Was there any survey about practical use-cases, that suggested these four (debug, info, warn, error) are worth adding to the standard library? In my experience, developers often don't make a clear distinguish between "debug", "warn", or "info", mixing all three interchangeably. As far as I recall, Kubernetes project came to the same conclusion, leaving only klog.InfoS and klog.ErrorS functions in their logging package API. I propose to explicitly exclude the question of granular logging levels from the proposed APIs, to make the API surface of the package (should this be a separate package) narrower. Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote 1 reply @hherman1 hherman1 Sep 11, 2022 - I find the debug log level very useful, for more involved inspection of a module, but logs that would be too noisy to include otherwise. However WARN has never been clear to me, and I'm happy to see it removed. I think the proposal is only really committed to numeric log levels, and the predefined ones seem more like conveniences. Beta Was this translation helpful? Give feedback. [259] nfx Sep 11, 2022 - Mass-migration A few years ago, I drove a 40-engineer effort to migrate from "normal logging" to structured logging of tens of thousands of logging statements in a sizeable monorepo. One of the key factors to success was a semi-automated refactoring tool that converted old-style logging to structured dialect. Roughly speaking, it was parsing AST of log statements that resembled log.Printf("[INFO] Created new %s from %s", kind, in.Source()) and converted those into something like slog.Info("Created new from", "kind", kind, "source", in.Source()) of course, the refactoring patch had to be manually tuned by an engineer for messages and variables to make sense. only because of tremendous speed gains by this automated generator could I get buy-in from other teams and convert the critical mass of log statements in a matter of days. Otherwise, it may have taken months or years. It would be great if this initiative included a (pluggable) migration path from the other loggers so that it could be retrofitted into older projects. Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote 0 replies [259] nfx Sep 11, 2022 - Structured errors @jba what I'm missing from this proposal is the way to handle "structured errors", where we want to reduce the cardinality of the "error" field in a log searching tool (splunk, elk, ...). Let's look at the following example: slog.Error("transaction failed", err, "id", t.ID()) which produce approximately the following JSON lines: {"level": "ERROR", "msg": "transaction failed", "error": "reading from 878798: endpoint XYZ returned 400", "id": 347823480923} ... {"level": "ERROR", "msg": "transaction failed", "error": "reading from 2934982: endpoint QWE returned 400", "id": 23489283492} there are a few problems with this JSON: * everything except the error field seems to be structured * it'll be hard to identify in a monitoring system the actual important and most-occurring error, as the reason for an outage. Proposal type LogFieldAdder interface { LogAttrs(r Record) } .. or alternatively type LogFieldAdder interface { LogAttrs() []Attr } so that errors could add context of their own: func (te TxError) []slog.Attr { return []slog.Attr{ slog.Int("account", te.Account}, slog.Str("endpoint", te.Endpoint}, slog.Int("status", te.Response.StatusCode}, } } so that JSON lines become {"level": "ERROR", "msg": "transaction failed", "error": "read failure", "account": 878798, "endpoint": "XYZ", "status": 400, "id": 347823480923} ... {"level": "ERROR", "msg": "transaction failed", "error": "read failure", "account": 2934982, "endpoint": "QWE", "status": 400, "id": 23489283492} what do you think? Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote 1 3 replies @hherman1 hherman1 Sep 11, 2022 - Why wouldn't you just add the relevant attrs on your own when you call the log method? Why have the error object do the adding? Beta Was this translation helpful? Give feedback. @Southclaws Southclaws Sep 11, 2022 - I've had similar frustrations and wrote a library for this similar to CockroachDB's which lets you store key value data into errors and also use contexts to propagate information relevant to logs or to trace spans. I don't think the proposed logging solution needs to change for this though as you can easily build an adapter for any logging library with it. Anyway it's on my profile as Fault and I'd appreciate feedback! Beta Was this translation helpful? Give feedback. @nfx nfx Sep 11, 2022 - @hherman1 because attrs in the context from error may bubble up to the current context Beta Was this translation helpful? Give feedback. Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment Category Discussions Labels None yet 30 participants @jba @evanphx @willfaught @rogpeppe @narqo @LukeShu @XANi @nfx @rabbbit @mvrhov @daenney @hherman1 @Merovius @Eun @smlx @akshayjshah @sethgrid @sagikazarmark @vearutop @Southclaws @deltamualpha and others Add heading text Add bold text, Add italic text, Add a quote, Add code, Insert Link Link Text [ ] URL [ ] Add Add a link, Add a bulleted list, Add a numbered list, Add a task list, Directly mention a user or team Reference an issue or pull request Add heading text Add bold text, Add italic text, Add a bulleted list, Add a numbered list, Add a task list, 1 reacted with thumbs up emoji 1 reacted with thumbs down emoji 1 reacted with laugh emoji 1 reacted with hooray emoji 1 reacted with confused emoji [?] 1 reacted with heart emoji 1 reacted with rocket emoji 1 reacted with eyes emoji Footer (c) 2022 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.