[HN Gopher] Log level 'error' should mean that something needs t...
___________________________________________________________________
Log level 'error' should mean that something needs to be fixed
Author : todsacerdoti
Score : 271 points
Date : 2025-12-17 12:08 UTC (3 days ago)
(HTM) web link (utcc.utoronto.ca)
(TXT) w3m dump (utcc.utoronto.ca)
| shadowgovt wrote:
| This is the standard I use as well. In general, my rule of thumb
| is that if something is logging error, it would have been
| perfectly reasonable for the program to respond by crashing, and
| the only reason it didn't is that it's executing in some kind of
| larger context that wants to stay up in the event of the failure
| of an individual component (like one handler suffering a query
| that hangs it and having to be terminated by its monitoring
| program in a program with multiple threads serving web requests).
| In contrast, something like an ill-formed web query from an
| untrusted source isn't even an error because you can't force
| untrusted sources to send you correctly formed input.
|
| Warning, in contrast, is what I use for a condition that the
| developer predicted and handled but probably indicates the larger
| context is bad, like "this query arrived from a trusted source
| but had a configuration so invalid we had to drop it on the
| floor, or we assumed a default that allowed us to resolve the
| query but that was a massive assumption and you really should
| change the source data to be explicit." Warning is also where I
| put things like "a trusted source is calling a deprecated API,
| and the deprecation notification has been up long enough that
| they really should know better by now."
|
| Where all of this matters is process. Errors trigger pages.
| Warnings get bundled up into a daily report that on-call is
| responsible for following up on, sometimes by filing tickets to
| correct trusted sources and sometimes by reaching out to owners
| of trusted sources and saying "Hey, let's synchronize on your
| team's plan to stop using that API we declared is going away 9
| months ago."
| raldi wrote:
| Right. If staging or the canary is logging errors, you
| block/abort the deploy. If it's logging warnings, that's
| normal.
| nlawalker wrote:
| It seems that the easier rule of thumb, then, is that
| "application logic should never log an error on its own behalf
| unless it terminates immediately after", and that error-level
| log entries should only ever be generated from a higher-level
| context by something else that's monitoring for problems that
| the application code itself didn't anticipate.
| raldi wrote:
| Yes. Examples of non-defects that should not be in the ERROR
| loglevel:
|
| * Database timeout (the database is owned by a separate oncall
| rotation that has alerts when this happens)
|
| * ISE in downstream service (return HTTP 5xx and increment a
| metric but don't emit an error log)
|
| * Network error
|
| * Downstream service overloaded
|
| * Invalid request
|
| Basically, when you make a request to another service and get
| back a status code, your handler should look like:
| logfunc = logger.error if 400 <= status <= 499 and status != 429
| else logger.warning
|
| (Unless you have an SLO with the service about how often you're
| allowed to hit it and they only send 429 when you're over, which
| is how it's supposed to work but sadly rare.)
| jonathrg wrote:
| 4xx is for invalid requests. You wouldn't log a 404 as an error
| raldi wrote:
| I'm talking about codes you receive from services you call
| out to.
| jonathrg wrote:
| Oh that makes sense.
| raldi wrote:
| There are still some special cases, because 404 is used
| for both "There's no endpoint with that name" and
| "There's no record with the ID you tried to look up."
| mewpmewp2 wrote:
| What if user sends some sort of auth token or other type of
| data that you yourself can't validate and third party gives
| you 4xx for it? You won't know ahead of time whether that
| token or data is valid, only after making a request to the
| third party.
| Hizonner wrote:
| > Database timeout (the database is owned by a separate oncall
| rotation that has alerts when this happens)
|
| So people writing software are supposed to guess how your
| organization assigns responsibilities internally? And you're
| _sure_ that the database timeout always happens because there
| 's something wrong with the database, and never because
| something is wrong on your end?
| raldi wrote:
| No; I'm not understanding your point about guessing. Could
| you restate?
|
| As for queries that time out, that should definitely be a
| metric, but not pollute the error loglevel, especially if
| it's something that happens at some noisy rate all the time.
| makeitdouble wrote:
| > the database is owned by a separate oncall rotation
|
| Not OP, but this part hits the same for me.
|
| In the case your client app is killing the DB through too
| many calls (e.g. your cache is not working) you should be
| able to detect it and react, without waiting for the DB
| team to come to you after they investigated the whole
| thing.
|
| But you can't know in advance if the DB connection errors
| are your fault or not, so logging it to cover the worse
| case scenario (you're the cause) is sensible.
| raldi wrote:
| I agree that you should detect this, just through a
| metric rather than putting DB timeouts in the ERROR
| loglevel.
| electroly wrote:
| I think OP is making two separate but related points, a
| general point and a specific point. Both involve guessing
| something that the error-handling code, on the spot, might
| not know.
|
| 1. When I personally see database timeouts at work it's
| rarely the database's fault, 99 times out of 100 it's the
| caller's fault for their crappy query; they should have
| looked at the query plan before deploying it. How is the
| error-handling code supposed to know? I log timeouts (that
| still fail after retry) as errors so someone looks at it
| and we get a stack trace leading me to the bad query. The
| database itself tracks timeout metrics but the log is much
| more immediately useful: it takes me straight to the scene
| of the crime. I think this is OP's primary point: in some
| cases, investigation is required to determine whether it's
| your service's fault or not, and the error-handling code
| doesn't have the information to know that.
|
| 2. As with exceptions vs. return values in code, the low-
| level code often doesn't know how the higher-level caller
| will classify a particular error. A low-level error may or
| may not be a high-level error; the low-level code can't
| know that, but the low-level code is the one doing the
| logging. The low-level logging might even be a third party
| library. This is particularly tricky when code reuse enters
| the picture: the same error might be "page the on-call
| immediately" level for one consumer, but "ignore, this is
| expected" for another consumer.
|
| I think the more general point (that you should avoid
| logging errors for things that aren't your service's fault)
| stands. It's just tricky in some cases.
| Hizonner wrote:
| > No; I'm not understanding your point about guessing.
| Could you restate?
|
| In the general case, the person writing the software has no
| way of knowing that "the database is owned by a separate
| oncall rotation". That's about your organization chart.
|
| Admittedly, they'd be justified in assuming that _somebody_
| is paying attention to the database. On the other hand,
| they really can 't be sure that the database is going to
| report anything useful to anybody at all, or whether it's
| going to report the salient details. The database may not
| even know that the request was ever made. Maybe the
| requests are timing out because they never got there. And
| _definitely_ maybe the requests are timing out because you
| 're sending too many of them.
|
| I mean, no, it doesn't make sense to log a million
| identical messages, but that's rate limiting. It's still an
| error if you can't access your database, and for all you
| know it's an error that your admin will have to fix.
|
| As for metrics, I tend to see those as downstream of logs.
| You compute the metric by counting the log messages. And a
| metric can't say " _this_ particular query failed ". The
| ideal "database timeout" message should give the exact
| operation that timed out.
| zbentley wrote:
| I wish I lived in a world where that worked. Instead, I live in
| a world where most downstream service issues (including
| database failures, network routing misconfigurations, giant
| cloud provider downtime, and more ordinary internal service
| downtime) are observed in the error logs of consuming services
| long before they're detected by the owners of the downstream
| service ... if they ever are.
|
| My rough guess is that 75% of incidents on internal services
| were _only_ reported by service consumers (humans posting in
| channels) across everywhere I've worked. Of the remaining 25%
| that were detected by monitoring, the vast majority were
| detected long after consumers started seeing errors.
|
| All the RCAs and "add more monitoring" sprints in the world
| can't add accountability equivalent to "customers start calling
| you/having tantrums on Twitter within 30sec of a GSO", in other
| words.
|
| The corollary is "internal databases/backend services can be
| more _technically_ important to the proper functioning of your
| business, but frontends /edge APIs/consumers of those backend
| services are more _observably_ important by other people. As a
| result, edge services' users often provide more valuable
| telemetry than backend monitoring."
| raldi wrote:
| But everything you're describing can be done with metrics and
| alerts; there's no need to spam the ERROR loglevel.
| zbentley wrote:
| My point is that just because those problems _can_ be
| solved with better telemetry doesn't mean that is actually
| done in practice. Most organizations do are much more aware
| of /sensitive to failures upstream/at the edge than they
| are in backend services. Once you account for alert
| fatigue, crappy accountability distribution, and
| organizational pressures, even the places that do this well
| often backslide over time.
|
| In brief: drivers don't obey the speed limit and backend
| service operators don't prioritize monitoring. Both groups
| are supposed to do those things, but they don't and we
| should assume they won't change. As a result, it's a good
| idea to wear seatbelts and treat downstream failures as
| urgent errors in the logs of consuming services.
| jillesvangurp wrote:
| Errors mean I get alerted. Zero tolerance on that from my side.
| mfuzzey wrote:
| I think it's difficult to say without knowing how the system is
| deployed and administered. "If a SMTP mailer trying to send email
| to somewhere logs 'cannot contact port 25 on <remote host>', that
| is not an error in the local system"
|
| Maybe or maybe not. If the connection problem is really due to
| the remote host then that's not the problem of the sender. But
| maybe the local network interface is down, maybe there's a local
| firewall rule blocking it,...
|
| If you know the deployment scenario then you can make reasonable
| decisions on logging levels but quite often code is generic and
| can be deployed in multiple configurations so that's hard to do
| greatgib wrote:
| The point is that if your program itself take note of the error
| from the library it is ok. You, as the program owner, can
| decide what to do with it (error log or not).
|
| But if you are the SMTP library and that you unilaterally log
| that as an error. That is an issue.
| zamadatix wrote:
| The counterpoint made above is while what you describe is
| indeed the way the author likes to see it that doesn't
| explain why "an error is something which failed that the
| program was unable to fix automatically" is supposed to be
| any less valid a way to see it. I.e. should error be defined
| as "the program was unable to complete the task you told it
| to do" or only "things which could have worked but you need
| to explicitly change something locally".
|
| I don't even know how to say whether these definitions are
| right or wrong, it's just whatever you feel like it should
| be. The important thing is what your program logs should be
| documented somewhere, the next most important thing is that
| your log levels are self consistent and follow some sort of
| logic, and that I would have done it exactly the same is not
| really important.
|
| At the end of the day, this is just bikeshedding about how to
| collapse ultra specific alerting levels into a few generic
| ones. E.g. RFC 5424 defines 8 separate log levels for syslog
| and, while that's not a ceiling by any means, it's easy to
| see how there's already not really going to be a universally
| agreed way to collapse even just these down to 4 categories.
| hinkley wrote:
| Any robust system isn't going to rely on reading logs to
| figure out what to do about undelivered email anyway. If
| you're doing logistics the failure to send an order
| confirmation needs to show up in your data model in some
| manner. Managing your application or business by logs is
| amateur hour.
|
| There's a whole industry of "we'll manage them for you"
| which is just enabling dysfunction.
| dminuoso wrote:
| This would require a complete new ecosystem and likely new
| language where any degradation of code flow becomes
| communicatable in a standardized and fully documented
| fashion.
|
| The closest we have is something like Java with exceptions in
| type signatures, but we would have to ban any kind of
| exception capture except from final programs, and promote
| basically any logger call int an exception that you could
| remotely suppress.
|
| We could philosophize about a world with compilers made out
| of unobtanium - but in this reality a library author cannot
| know what conditions are fixable or necessitate a fix or not.
| And structured logging lacks has way too many deficiencies to
| make it work from that angle.
| colechristensen wrote:
| How about this:
|
| - An error is an event that _someone_ should act on. Not
| necessarily you. But if it 's not an event that ever needs the
| attention of a person then the severity is less than an error.
|
| Examples: Invalid credentials. HTTP 404 - Not Found, HTTP 403
| Forbidden, (all of the HTTP 400s, by definition)
|
| It's not my problem as a site owner if one of my users entered
| the wrong URL or typed their password wrong, but it's _somebody
| 's_ problem.
|
| A warning is something that A) a person would likely want to
| know and B) wouldn't necessarily need to act on
|
| INFO is for something a person would likely want to know and
| unlikely needs action
|
| DEBUG is for something likely to be helpful
|
| TRACE is for just about anything that happens
|
| EMERG/CRIT are for _significant_ errors of immediate impact
|
| PANIC the sky is falling, I hope you have good running shoes
| DanHulton wrote:
| If you're logging and reporting on ERRORs for 400s, then your
| error triage log is going to be full of things like a user
| entering a password with insufficient complexity or trying to
| sign up with an email address that already exists in your
| system.
|
| Some of these things can be ameliorated with well-behaved UI
| code, but a lot cannot, and if your primary product is the
| API, then you're just going to have scads of ERRORs to triage
| where there's literally nothing you can do.
|
| I'd argue that anything that starts with a 4 is an INFO, and
| if you really wanted to be through, you could set up an alert
| on the frequency of these errors to help you identify if
| there's a broad problem.
| colechristensen wrote:
| You have HTTP logs tracked, you don't need to report them
| twice, once in the HTTP log and once on the backend. You're
| just effectively raising the error to the HTTP server and
| its logs are where the errors live. You don't alert on
| single HTTP 4xx errors because nobody does, you only raise
| on anomalous numbers of HTTP 4xx errors. You _do_ alert on
| HTTP 5xx errors because as "Internal" http errors those
| are on you always.
|
| In other words, of course you don't alert on errors which
| are likely somebody else's problem. You put them in the log
| stream where that makes sense and can be treated
| accordingly.
| adrianmonk wrote:
| > _An error is an event that someone should act on. Not
| necessarily you._
|
| Personally, I'd further qualify that. It should be logged as
| an error _if_ the person who reads the logs would be
| responsible for fixing it.
|
| Suppose you run a photo gallery web site. If a user uploads a
| corrupt JPEG, and the server detects that it's corrupt and
| rejects it, then someone needs to do something, but from the
| point of view of the person who runs the web site, the web
| site behaved correctly. It can't control whether people's
| JPEGs are corrupt. So this shouldn't be categorized as an
| error in the server logs.
|
| But if you let users upload a batch of JPEG files (say a ZIP
| file full of them), you might produce a log file for the user
| to view. And in that log file, it's appropriate to categorize
| it as an error.
| colechristensen wrote:
| That's the difference between an HTTP 4xx and 5xx
|
| 4xx is for client side errors, 5xx is for server side
| errors.
|
| For your situation you'd respond with an HTTP 400 "Bad
| Request" and not an HTTP 500 "Internal Server Error"
| because the problem was with the request not with the
| server.
| Arrowmaster wrote:
| Counter argument. How do you know the user uploaded a
| corrupted image and it didn't get corrupted by your
| internet connection, server hardware, or a bug in your
| software stack?
|
| You cannot accurately assign responsibility until you
| understand the problem.
| solatic wrote:
| > But maybe the local network interface is down, maybe there's
| a local firewall rule blocking it,...
|
| That's exactly why you log it as a warning. People get warned
| all the time about the dangers of smoking. It's important that
| people be warned about smoking; these warnings save lives.
| People should pay attention to warnings, which let them know
| about worrisome concerns that should be heeded. But guess what?
| Everyone has a story about someone who smoked until they were
| 90 and died in a car accident. It is not an _error_ that
| somebody is smoking. Other systems will make their own bloody
| decisions and firewalling you off might be one of them. That is
| _normal_.
|
| What do you _think_ a warning means?
| theli0nheart wrote:
| I agree with this.
|
| Not everything that a library considers an error is an
| application error. If you log an error, something is absolutely
| wrong and requires attention. If you consider such a log as
| "possibly wrong", it should be a warning instead.
| alexwasserman wrote:
| I have been particularly irritated in the past where people use a
| lower log level and include the higher log level string in the
| message, especially where it's then parsed, filtered, and alerted
| on my monitoring.
|
| eg. log level WARN, message "This error is...", but it then trips
| an error in monitoring and pages out.
|
| Probably breaching multiple rules here around not parsing logs
| like that, etc. But it's cropped up so many times I get quite
| annoyed by it.
| jonathrg wrote:
| Stuff like that is a good argument for using structured
| logging, but even if you are just parsing text logs, surely you
| can make the parser be a bit more specific when retrieving the
| log level.
| dragonwriter wrote:
| > I have been particularly irritated in the past where people
| use a lower log level and include the higher log level string
| in the message, especially where it's then parsed, filtered,
| and alerted on my monitoring.
|
| If your parsing, filtering, and monitoring setup parses strings
| that happen to correspond to log level names in positions other
| than that of log levels as having the semantics of log levels,
| then that's a parsing/filtering error, not a logging error.
| vpribish wrote:
| I just started playing in the Erlang ecosystem and they have
| EIGHT levels of logging messages. it seems crazily over-specific,
| but they are the champions of robust systems.
|
| I could live with 4
|
| Error - alert me now.
|
| Warning - examine these later,
|
| Info - important context for investigations.
|
| Debug - usually off in prod.
| regularfry wrote:
| The eight levels in Erlang are inherited from syslog, rather
| than something specific to Erlang itself.
| groundzeros2015 wrote:
| The first one should be crashing.
| dnautics wrote:
| let's say you a bunch of database timeouts in a row. this might
| mean that nothing needs to be fixed. But also, the "thing that
| needs to be fixed" might be "the ethernet cable fell out the back
| of your server".
|
| How do you know?
| raldi wrote:
| You have an alert on what users actually care about, like the
| overall success rate. When it goes off, you check the WARNING
| log and metric dashboard and see that requests are timing out.
| ImPostingOnHN wrote:
| That is a lagging indicator. By the time you're alerted,
| you've already failed by letting users experience an issue.
| raldi wrote:
| What alternative would you propose? Page the oncall
| whenever there's a single query timeout?
| dnautics wrote:
| the alternative i propose is have deep understanding of
| your system before popping off with dumb one size fits
| all rules that don't make sense.
| danaris wrote:
| Well, _yes_. If the cable falls out of the server (or there
| 's a power outage, or a major DDoS attack, or whatever),
| your users are _going_ to experience that before you are
| aware of it. Especially if it 's in the middle of the night
| and you don't have an active night shift.
|
| Expecting arbitrary services to be able to deal with
| _absolutely any_ kind of failure in such a way that users
| never notice is _deeply_ unrealistic.
| makeitdouble wrote:
| > This assumes an error/warning/info/debug set of logging levels
| instead of something more fine grained, but that's how many
| things are these days.
|
| Does it ?
|
| Don't most stacks have an additional level of triaging logs to
| detect anomalies etc ? It can be your New relic/DataDog/Sentry or
| a self made filtering system, but nowadays I'd assume the base
| log levels are only a rough estimate of whether an single event
| has any chance of being problematic.
|
| I'd bet the author also has strong opinions about http error
| codes, and while I empathize, those ships have long sailed.
| rwmj wrote:
| And the second rule is _make all your error messages actionable_.
| By that I mean it should tell me what action to take to fix the
| error (even if that action means hard work, tell me what I have
| to do).
| andoando wrote:
| Error: Possible race condition, rewrite codebase
| morkalork wrote:
| I have written out-of-band sanity checks that have caught
| race conditions, the recommendation is more like "<Thing>
| that should be locked, isn't. Check what was merged and
| deployed in the last 24h, someone ducked it up"
| hyperadvanced wrote:
| This is just plain wrong, I vehemently disagree. What happens
| if a payment fails on my API, and today that means I need to go
| through a 20-step process with this pay provider, my database,
| etc. to correct that. But what's worse is if this error happens
| 11,000 times and I run a script to do my 20 step process 11,000
| times, but it turns out the error was raised in error.
| Additionally, because the error was so explicit about how to
| fix it, I didn't talk to anyone. And of course, the suggested
| fix was out of date because docs lag vs. production software.
| Now I have 11,000 pissed off customers because I was trying to
| be helpful.
| throw3e98 wrote:
| Maybe that makes sense for a single-machine application where
| you also control the hardware. But for a networked/distributed
| system, or software that runs on the user's hardware, the
| action might involve a decision tree, and a log line is a poor
| way to convey that. We use instrumentation, alerting and
| runbooks for that instead, with the runbooks linking into a
| hyperlinked set of articles.
|
| My 3D printer will try to walk you through basic fixes with
| pictures on the device's LCD panel, but for some errors it will
| display a QR code to their wiki which goes into a technical
| troubleshooting guide with complex instructions and tutorial
| videos.
| chongli wrote:
| Suppose I'm writing an http server and the error is caused by a
| flaky power supply causing the disk to lose power when the
| server attempts to read a file that's been requested. How is
| the http server supposed to diagnose this or any other hardware
| fault? Furthermore, why should it even be the http server's
| responsibility to know about hardware issues at all?
| uniq7 wrote:
| The error doesn't need to be extremely specific or point to
| the actual root cause.
|
| In your example,
|
| - "Error while serving file" would be a bad error message,
|
| - "Failed to read file 'foo/bar.html'" would be acceptable,
| and
|
| - "Failed to read file 'foo/bar.html' due to EIO: Underlying
| device error (disk failure, I/O bus error). Please check the
| disk integrity." would be perfect (assuming the http server
| has access to the underlying error produced by the read
| operation).
| Copenjin wrote:
| Some of these replies make me wonder if you have ever written
| any code at all, nonsensical example.
| 1123581321 wrote:
| Can you please explain this? That sounds like identifying bugs
| but not fixing them but I realize you don't mean that. One
| hopes the context information in the error will make it
| actionable when it occurs, never completely successfully, of
| course.
| rwmj wrote:
| Here's an example of a bug that I filed about non-actionable
| error messages: https://github.com/karmab/kcli/issues/456
|
| The first error message was _" No usable public key found,
| which is required for the deployment"_ which doesn't tell me
| what I have to do to correct the problem. Nothing about even
| where it's looking for keys, what is supposed to create the
| key or how I am supposed to create the key.
|
| There are other examples and discussion of what they should
| say in the link.
|
| Edit: Here's another one that I filed:
| https://github.com/containers/podman/issues/20775
| 1123581321 wrote:
| That makes sense and good examples; thanks.
|
| At work, I can think of cases where we error when data
| mismatches between two systems. It's almost always the
| fault of system B but we present the mismatch error
| neutrally. Experienced developers just know to fix B but we
| shouldn't rely on that.
| Copenjin wrote:
| You can hope that the person reading the context will always
| able to understand it like you would have. Bad assumption in
| my experience.
| 1123581321 wrote:
| Quite true. It can be a bad assumption when I'm the one
| trying to understand it weeks later. :)
| magicalhippo wrote:
| This can be difficult or just not possible.
|
| What is possible is to include as much information about what
| the system was trying to do. If there's an file IO error,
| include the the full path name. Saying "file not found" without
| saying _which_ file was not found infuriates me like few other
| things.
|
| If some required configuration option is not defined, include
| the name of the configuration option and from where it tried to
| find said configuration (config files, environment, registry
| etc). And include the detailed error message from the
| underlying system if any.
|
| Regular users won't have a clue how to deal with most errors
| anyway, but by including details at least someone with some
| system knowledge has a chance of figuring out how to fix or
| work around the issue.
| pixl97 wrote:
| So what error do you put if the server is over 500 miles away?
|
| https://web.mit.edu/jemorris/humor/500-miles
|
| Or you can't connect because of a path MTU error.
|
| Or because the TTL is set to low?
|
| Your software at the server level has no idea what's going
| wrong at the network level, all you can send is some kind of
| network problem message.
| Copenjin wrote:
| Exactly. Some applications keep running way after you have long
| gone. If there is useful information to provide give it.
| jayofdoom wrote:
| In OpenStack, we explicitly document what our log levels mean; I
| think this is valuable from both an Operator and Developer
| perspective. If you're a new developer, without a sense of what
| log levels are for, it's very prescriptive and helpful. For an
| operator, it sets expectations.
|
| https://docs.openstack.org/oslo.log/latest/user/guidelines.h...
|
| FWIW, "ERROR: An error has occurred and an administrator should
| research the event." (vs WARNING: Indicates that there might be a
| systemic issue; potential predictive failure notice.)
| quectophoton wrote:
| Thank you, this (and jillesvangurp's comment) sounds way more
| reasonable than the article's suggestion.
|
| If I have a daily cron job that is copying files to a remote
| location (e.g. backups), and the _operation_ fails because for
| some reason the destination is not writable.
|
| Your suggestion would get me _both_ alerts, as I want; the
| article's suggestion would not alert me about the operation
| failing because, after all, it's not something happening in the
| local system, the local program is well configured, and it's
| "working as expected" because it doesn't need neither code nor
| configuration fixing.
| __turbobrew__ wrote:
| Agreed, I don't get the OPs delineation between local and
| non-local error sources. If your code has a job to do it
| doesn't matter if the error was local or non-local, the
| operator needs to know that the code is not doing its job. In
| the case of something like you cannot backup files to a
| remote you can try to contact the humans who own the remote
| or come up with an alternative backup mechanism.
| HarHarVeryFunny wrote:
| I agree with the sentiment, although not sure if "error" is the
| right category/verbiage for actionable logs.
|
| In an ideal world things like logs and alarms (alerting product
| support staff) should certainly cleanly separate things that are
| just informative, useful for the developer, and things that
| require some human intervention.
|
| If you don't do this then it's like "the boy that cried wolf",
| and people will learn to ignore errors and alarms since you've
| trained them to understand that usually no action is needed. It's
| also useful to be able to grep though log files and distinguish
| failures of different categories, not just grep for specific
| failures.
| eterm wrote:
| How I'd personally like to treat them: - Critical
| / Fatal: Unrecoverable without human intervention, someone needs
| to get out of bed, now. - Error : Recoverable without human
| intervention, but not without data / state loss. Must be fixed
| asap. An assumption didn't hold. - Warning: Recoverable
| without intervention. Must have an issue created and prioritised.
| ( If business as usual, this could be downgrading to INFO. )
|
| The main difference therefore between error and warning is, "We
| didn't think this could happen" vs "We thought this might
| happen".
|
| So for example, a failure to parse JSON might be an error if
| you're responsible for generating that serialisation, but might
| be a warning if you're not.
| mewpmewp2 wrote:
| What if you are integrated to a third party app and it gives
| you 5xx once? What do you log it as, and let's say after a
| retry it is fine.
| bqmjjx0kac wrote:
| I would log a warning when an attempt fails, and an error
| when the final attempt fails.
| mewpmewp2 wrote:
| You are not the OP, but I think I was trying to point out
| this example case in relation to their descriptions of
| Error/Warnings.
|
| This scenario may or may not yield in data/state loss, it
| may also be something that you, yourself can't immediately
| fix. And if it's temporary, what is the point of creating
| an issue and prioritizing.
|
| I guess my point is that to any such categorization of
| errors or warnings there are way too many counter examples
| to be able to describe them like that.
|
| So I'd usually think that Errors are something that I would
| heuristically want to quickly react to and investigate
| (e.g. being paged, while Warnings are something I would
| periodically check in (e.g. weekly).
| wredcoll wrote:
| Like so many things in this industry the point is
| establishing a shared meaning for all the humans
| involved, regardless of how uninvolved people think.
|
| That being said, I find tying the level to _expected
| action_ a more useful way to classify them.
| mewpmewp2 wrote:
| But what I also see frequently is people trying to do the
| impossible and idealistic things because they read
| somewhere that something should mean X, when things are
| never so clearly cut, so either it is not such a
| simplistic issue and should be understood as not such a
| simple issue, or there might be a better more practical
| definition for it. We should first start from what are we
| using Logs for. Are we using those for debugging, or so
| we get alerted or both?
|
| If for debugging, the levels seem relevant in the sense
| of how quickly we are able to use that information to
| understand what is going wrong. Out of potential sea of
| logs we want to see first what were the most likely
| culprits for something causing something to go wrong. So
| the higher the log level, the higher likelihood of this
| event causing something to go wrong.
|
| If for alerting, they should reflect on how bad is this
| particular thing happening for the business and would
| help us to set a threshold for when we page or have to
| react to something.
| eterm wrote:
| This might be controversial, but I'd say if it's fine after a
| retry, then it doesn't need a warning.
|
| Because what I'd want to know is _how often does it fail_ ,
| which is a metric not a log.
|
| So expose <third party api failure rate> as a metric not a
| log.
|
| If feeding logs into datadog or similar is the only way
| you're collecting metrics, then you aren't treating your
| observablity with the respect it deserves. Put in real
| counters so you're not just reacting to what catches your eye
| in the logs.
|
| If the third party being down has a knock-on effect to your
| own system functionality / uptime, then it needs to be a
| warning or error, but you should also put in the backlog a
| ticket to de-couple your uptime from that third-party, be it
| retries, queues, or other mitigations ( alternate providers?
| ).
|
| By implementing a retry you planned for that third party to
| be down, so it's just business as usual if it suceeds on
| retry.
| mewpmewp2 wrote:
| > If the third party being down has a knock-on effect to
| your own system functionality / uptime, then it needs to be
| a warning or error, but you should also put in the backlog
| a ticket to de-couple your uptime from that third-party, be
| it retries, queues, or other mitigations ( alternate
| providers? ).
|
| How do you define uptime? What if e.g. it's a social login
| / data linking and that provider is down? You could have
| multiple logins and your own e-mail and password, but you
| still might lose users because the provider is down. How do
| you log that? Or do you only put it as a metric?
|
| You can't always easily replace providers.
| ivan_gammel wrote:
| You may log that or count failures in some metric, but
| the correct answer is to have a health check on third
| party service and an alert when that service is down.
| Logs may help to understand the nature of the incident,
| but they are not the channel through which you are
| informed about such problems.
|
| The different issue is when third party broke the
| contract, so suddenly you get a lot of 4xx or 5xx
| responses, likely unrecoverable. Then you get ERROR level
| messages in the log (because it's unexpected problem) and
| an alert when there's a spike.
| hk__2 wrote:
| > This might be controversial, but I'd say if it's fine
| after a retry, then it doesn't need a warning. > > Because
| what I'd want to know is how often does it fail, which is a
| metric not a log.
|
| It's not controversial; you just want something different.
| _I_ want the opposite: I want to know _why_ / _how_ it
| fails; counting how often it does is secondary. I want a
| log that says "I sent this payload to this API and I got
| this error in return", so that later I can debug if my
| payload was problematic, and/or show it to the third party
| if they need it.
| hamandcheese wrote:
| My main gripe with metrics is that they are not easily
| discoverable like logs are. Even if you capture a list of
| all the metrics emitted from an application, they often
| have zero context and so the semantics are a bit hard to
| decipher.
| kiicia wrote:
| As always ,,it depends"
|
| - info - when this was expected and system/process is
| prepared for that (like automatic retry, fallback to local
| copy, offline mode, event driven with persistent queue etc) -
| warning - when system/process was able to continue but in
| degraded manner, maybe leaving decision to retry to user or
| other part of system, or maybe just relying on someone
| checking logs for unexpected events, this of course depends
| if that external system is required for some action or in
| some way optional - error - when system/process is not able
| to continue and particular action has been stopped
| immediately, this includes situation where retry mechanism is
| not implemented for step required for completion of
| particular action - fatal - you need to restart something,
| either manually or by external watchdog, you don't expect
| this kind of logs for simple 5xx
| marcosdumay wrote:
| Well, the GPs criteria are quite good. But what you should
| actually do depends on a lot more things than the ones you
| wrote in your comment. It could be so irrelevant to only
| deserve a trace log, or so important to get a warning.
|
| Also, you should have event logs you can look to make
| administrative decisions. That information surely fits into
| those, you will want to know about it when deciding to switch
| to another provider or renegotiate something.
| cpburns2009 wrote:
| It really depends on the third party service.
|
| For service A, a 500 error may be common and you just need to
| try again, and a descriptive 400 error indicates the original
| request was actually handled. In these cases I'd log as a
| warning.
|
| For service B, a 500 error may indicate the whole API is
| down, in which case I'd log a warning and not try any more
| requests for 5 minutes.
|
| For service C, a 500 error may be an anomaly and treat it as
| hard error and log as error.
| sysguest wrote:
| hmm maybe we need extra representation?
|
| eg: 2.0 for "trace" / 1.0 for "debug" / 0.0 for "info" / -1.0
| for "warn" / -2.0 for "error that can be handled"
| wredcoll wrote:
| I said this elsewhere, but the point here is what the humans
| involved are supposed to do with this info. Do I literally
| get out of bed on an error log or do I grep for them once or
| twice a month?
| ivan_gammel wrote:
| You should never get out of bed on an error in the log.
| Logs are for retrospective analysis, health checks and
| metrics are for situational awareness, alerts are for
| waking people up.
| RaftPeople wrote:
| > _The main difference therefore between error and warning is,
| "We didn't think this could happen" vs "We thought this might
| happen"._
|
| What about conditions like "we absolutely knew this would
| happen regularly, but it's something that prevents the
| completion of the entire process which is absolutely critical
| to the organization"
|
| The notion of an "error" is very context dependent. We usually
| use it to mean "can not proceed with action that is required
| for the successful completion of this task"
| wizzwizz4 wrote:
| Those conditions would be "Critical", no? The error vs
| warning distinction doesn't apply.
| fhcuvyxu wrote:
| No, many applications need to be fault tolerant.
|
| Crashing your web stack because one route hit an error is a
| dumb idea.
|
| And no, calling it a warning is also dumb idea. It is an
| error.
|
| This article is a navel gazing expedition.
|
| They're kind of right but you can turn any warning into an
| error and vice versa depending on business needs that
| outweigh the technical categorisation.
| arwhatever wrote:
| I like to think of "warning" as something to alert on
| statistically, e.g. incorrect password attempt rate jumps from
| 0.4% of login attempts to 99%.
| masswerk wrote:
| Also, warnings for ambiguous results.
|
| For example, when a process implies a conversion according to
| the contract/convention, but we know that this conversion may
| be not the expected result and the input may be based on
| semantic misconceptions. E.g., assemblers and contextually
| truncated values for operands: while there's no issue with the
| grammar or syntax or intrinsic semantics, a higher level
| misconception may be involved (e.g., regarding address modes),
| resulting in a correct but still non-functional output. So, "In
| this individual case, there may be or may be not an issue.
| Please, check. (Not resolvable on our end.)"
|
| (Disclaimer: I know that this is a very much classic computing
| and that this is now mostly moved to the global TOS, but still,
| it's the classic example for a warning.)
| teo_zero wrote:
| This doesn't resonate with my experience. I place the line
| between a warning and an error whether the operation can or can't
| be completed.
|
| A connection timed out, retrying in 30 secs? That's a warning.
| Gave up connecting after 5 failed attempts? Now that's an error.
|
| I don't care so much if the origin of the error is within the
| program, or the system, or the network. If I can't get what I'm
| asking for, it can't be a mere warning.
| AndroTux wrote:
| "cannot contact port 25 on <remote host>" may very well be a
| configuration _error._ How should the program know?
| HankB99 wrote:
| Would it make sense to consider anything that prevents a
| process from completing it's intended function an error? It
| seems like this message would fall into that category and, as
| you pointed out, could result from a local fault as well.
| kijin wrote:
| SMTP clients are designed to try again with exponential
| backoff. If the final attempt fails and your email gets
| bounced, now that's an error. Until then, it's just a delay,
| business as usual.
| notatoad wrote:
| >How should the program know?
|
| if we're talking about logs from our own applications that we
| have written, the program should know because we can write it
| in a way that it knows.
|
| user-defined config should be verified before it is used. make
| a ping to port 25 to see if it works before you start using
| that config for actual operation. if it fails the verification
| step, that's not an error that needs to be logged.
| 1718627440 wrote:
| So when the random error on a remote party happens at one
| time your system ignores it, bu when it happens at another
| time, it prevents the server from booting? That's a very
| brittle system.
| notatoad wrote:
| log level error prevents your server from booting? i'm
| pretty sure that's not how logging works.
| tcpkump wrote:
| What about when the mail server endpoint has changed, and for
| whatever reason, this configuration wasn't updated? This is a
| common scenario when dealing with legacy infrastructure in my
| experience.
| notatoad wrote:
| the whole point of the essay here is that you should make a
| distinction between errors that you care about and plan to
| fix, and errors that you don't care about and don't intend
| to do anything about. and if you don't intend to do
| anything about it, it shouldn't be logged as error.
|
| i'm following the author's example that an SMTP connection
| error is something you want to investigate and fix. if you
| have a different system with different assumptions where
| your response to a mailserver being unreachable is to
| ignore it, obviously that example doesn't apply for you.
| i'm not saying, and i don't think the author is saying that
| SMTP errors should always or never be logged as errors.
|
| when the mailserver endpoint has changed, you should do the
| thing that makes sense in the context of your application.
| if it's not something that the person responsible for
| reviewing the logs needs to know about, don't log it. if it
| is, then log it.
| mkoubaa wrote:
| To me it's always a neat trick when you're not allowed to use
| print() in production code
| aqme28 wrote:
| I agree with this take in a steady state, but the process of
| building software is just that-- it's a process.
|
| So it's natural for error messages to be expected, as you
| progressively add and then clear up edge cases.
| raldi wrote:
| Exactly: When you're building software, it has lots of defects
| (and, thus, error logging). When it's mature, it should have
| few defects, and thus few error logs, and each one that remains
| is a bug that should be fixed.
| plorkyeran wrote:
| I don't understand why you seem to think you're disagreeing
| with the article? If you're producing a lot of error logs
| because you have bugs that you need to fix then you aren't
| violating the rule that an error log should mean that
| something needs to be fixed.
| raldi wrote:
| I couldn't agree more with the article. What made you think
| I disagreed?
| tgv wrote:
| I log authorization errors as errors. Are they errors? It depends
| on how you read the logs. Perhaps you want to distinguish between
| internal, external and non-attributable errors for easier
| grepping.
| layer8 wrote:
| > When implementing logging, it's important to distinguish
| between an error from the perspective of an individual operation
| and an error from the perspective of the overall program or
| system. Individual operations may well experience errors that are
| not error level log events for the overall program. You could say
| that an operation error is anything that prevents an operation
| from completing successfully, while a program level error is
| something that prevents the program as a whole from working
| right.
|
| This is a nontrivial problem when using properly modularized code
| and libraries that perform logging. They can't tell whether their
| operational error is also a program-level error, which can depend
| on usage context, but they still want to log the operational
| error themselves, in order to provide the details that aren't
| accessible to higher-level code. This lower-level logging has to
| choose _some_ status.
|
| Should only "top-level" code ever log an error? That can make it
| difficult to identify the low-level root causes of a top-level
| failure. It also can hamper modularization, because it means you
| can't repackage one program's high-level code as a library for
| use by other programs, without somehow factoring out the logging
| code again.
| ivan_gammel wrote:
| Libraries should not log on levels above DEBUG, period. If
| there's something worthy for reporting on higher levels, pass
| this information to client code, either as an event, or as an
| exception or error code.
| layer8 wrote:
| From a code modularization point of view, there shouldn't
| really be much of a difference between programs and
| libraries. A program is just a library with a different
| calling convention. I like to structure programs such that
| their actual functionality could be reused as a library in
| another program.
|
| This is difficult to reconcile with libraries only logging on
| a debug level.
| ivan_gammel wrote:
| The main difference is that library is not aware of the
| context of the execution of the code, so cannot decide,
| whether the problem is expected, recoverable or severe.
| layer8 wrote:
| The same is true for programs that are being invoked. The
| program only knows relative to its own purpose, and the
| same is again true for libraries. I don't see the
| difference, other than, as already mentioned, the
| mechanism of program vs. library invocation.
|
| Consider a Smalltalk-like system, or something like TCL,
| that doesn't distinguish between programs and libraries
| regarding invocation mechanism. How would you handle
| logging in that case?
| hrimfaxi wrote:
| The mechanism of invocation is important. Most programs
| allow you to set the logging verbosity at invocation.
| Libraries may provide an interface to do so but their
| entry points tend to be more numerous.
| ivan_gammel wrote:
| The purpose of a program and of a library is different
| and intent of the authors of the code is usually clear
| enough to make the distinction in context. Small
| composable programs aren't interesting case here, they
| shouldn't be verbose anyway even to justify multiple
| logging levels (it's probably just set to on/off using a
| command line argument).
| msteffen wrote:
| Okay, but...most programs are written in Python or Rust
| or something, where invoking library functions is a lot
| safer, more ergonomic, more performant, and more common
| than spawning a subprocess and executing a program in it.
| Like you can't really ignore the human expectations and
| conventions that are brought to bear when your code is
| run (the accommodation of which is arguably most of the
| purpose of programming languages).
|
| When you publish a library, people are going to use it
| more liberally and in a wider range of contexts (which
| are therefore harder to predict, including whether a
| given violation requires human intervention)
| dpark wrote:
| And the program doesn't know if the user is expecting
| failure, either. The library case is not actually much
| different.
|
| It's very reasonable that a logging framework should
| allow higher levels to adjust how logging at lower levels
| is recorded. But saying that libraries should only log
| debug is not. It's very legitimate for a library to log
| "this looks like a problem to me".
| schrodinger wrote:
| I see your point, but disagree on a practical level.
| Libraries are being used while you're in "developer" mode,
| while programs are used in "user" mode (trying awkwardly to
| differentiate between _being_ a developer and currently
| developing code around that library.
|
| Usually a program is being used by the user to accomplish
| something, and if logging is meaningful than either in a
| cli context or a server context. In both cases, errors are
| more often being seen by people/users than by code.
| Therefore printing them to logs make sense.
|
| While a lib is being used by a program. So it has a better
| way to communicate problems with the caller (and
| exceptions, error values, choose the poison of your
| language). But I almost never want a library to start
| logging shit because it's almost guaranteed to not follow
| the same conventions as I do in my program elsewhere.
| Return me the error and let me handle.
|
| It's analogous to how Go has an implicit rule of that a
| library should never let a panic occur outside the library.
| Internally, fine. But at the package boundary, you should
| catch panics and return them as an error. You don't know if
| the caller wants the app to die because it an error in your
| lib!
| 1718627440 wrote:
| Why? Whats wrong with logging it and passing the log object
| to the caller? The caller can still modify the log entry
| however it pleases?
| ivan_gammel wrote:
| Practicality. It is excessive for client code to calibrate
| library logging level. It's ok to do it in logging
| configuration, but having an entry for every library there
| is also excessive. It is reasonable to expect that
| dev/staging may have base level at DEBUG and production
| will have base level at INFO, so that a library following
| the convention will not require extra effort to prevent log
| spam in production. Yes, we have entire logging industry
| around aggregation of terabytes of logs, with associated
| costs, but do you really need that? In other words, are we
| developers too lazy to adapt the sane logging policy, which
| actually requires minimum effort, and will just burn the
| company money for nothing?
| kelnos wrote:
| Eh, as with anything there are always exceptions. I generally
| agree with WARN and ERROR, though I can imagine a few
| situations where it might be appropriate for a library to log
| at those levels. Especially for a warning, like a library
| might emit "WARN Foo not available; falling back to Bar" on
| initialization, or something like that. And I think a library
| is fine logging at INFO (and DEBUG) as much as it wants.
|
| Ultimately, though, it's important to be using a featureful
| logging framework (all the better if there's a "standard" one
| for your language or framework), so the end user can
| enable/disable different levels for different modules
| (including for your library).
| ivan_gammel wrote:
| WARN Foo not available; falling back to Bar
|
| In server contexts this is usually unnecessary noise, if
| it's not change detection. Of course, good logging
| framework will help you to mute irrelevant messages, but as
| I said in another comment here, it's a matter of
| practicality. Library shouldn't create extra effort for its
| users to fine-tune logging output, so it must use
| reasonable defaults.
| hinkley wrote:
| Log4j has the ability to filter log levels by subject matter
| for twenty years. In Java you end up having to use that a lot
| for this reason.
| PartiallyTyped wrote:
| Logging in rust also does that, you can set logging levels
| for individual modules deep within your dependency tree.
| TZubiri wrote:
| Oh that library that gives you a write() wrapper in exchange
| for RCE vulns
| ivan_gammel wrote:
| Log4j is basically a design pattern. If you don't like the
| library, Slf4j/logback are based on the same principles.
| 0x696C6961 wrote:
| Libraries should not log, instead they should allow registering
| hooks which get called with errors and debug info.
| ivan_gammel wrote:
| They can log if platform permits, i.e. when you can set TRACE
| and DEBUG to no-op, but of course it should be done
| reasonably. Having hooks is often an overkill compared to
| this.
| esrauch wrote:
| It doesn't seem to work this way in practice, not least
| because most libraries will be transitive deps of the
| application owner.
|
| I think creating the hooks is very close to just not doing
| anything here, if no one is going to use the hooks anyway
| then you might as well not have them.
| kelnos wrote:
| I think this is useful for libraries in a language like C,
| where there is no standardized logging framework, so there's
| no way for the application to control what the library logs.
| But in a language (Java, Rust, etc.) where there are
| standard, widely-used logging frameworks that give people
| fine-grained control over what gets logged, libraries should
| just use those frameworks.
|
| (Even in C, though... errors should be surfaced as return
| values from functions causing the error, not just logged
| somewhere. Debug info, sure, have a registerable callback for
| that.)
| Blackthorn wrote:
| Libraries should log in a way that is convenient to the
| developer rather than a way that is ideologically consistent.
| Oftentimes, that means logging as we know it.
| Too wrote:
| This is why it's almost always wrong for library functions to
| log anything, even on "errors". Pass the status up through
| return values or exceptions. As a library author you have no
| clue as how an application might use it. Multi threading, retry
| loops and expected failures will turn what's a significant
| event in one context into what's not even worthy of a debug log
| in another. No rule without exceptions of course, one valid
| case could be for example truly slow operations where progress
| reports are expected. Modern tracing telemetry with sampling
| can be another solution for the paranoid.
| echelon wrote:
| You need a tuple: (context, level)
|
| The application owner should be able to adjust the contexts
| up or down. This is the point of ownership and where
| responsibility over which logs matter is handled.
|
| A library author might have ideas and provide useful
| suggestions, but it's ultimately the application owner who
| decides. Some libraries have huge blast radius and their
| `error` might be your `error` too. In other contexts, it
| could just be a warning. Library authors should make a
| reasonable guess about who their customer is and try to
| provide semantic, granular, and controllable failure
| behavior.
|
| As an example, Rust's logging ecosystem provides nice
| facilities for fine-grained tamping down of errors by crate
| (library) or module name. Other languages and logging
| libraries let you do this as well.
|
| That capability just isn't adopted everywhere.
| Izkata wrote:
| Python's built-in logging is the same if used correctly,
| where the library gets a logger based on its module name
| (this part isn't enforced) and the application can add a
| handler to that logger to route the logs differently if
| needed.
| esrauch wrote:
| I think an example where libraries could sensibly log error
| is if you have a condition which is recoverable but may cause
| a significant slowdown, including a potential DoS issue, and
| the application owner can remediate.
|
| You don't want to throw because destroying someone's
| production isn't worth it. You don't want to silent continue
| in that state because realistically there's no way for
| application owner to understand what is happening and why.
| ivan_gammel wrote:
| In such scenarios it makes sense to give clients an
| opportunity to react on such conditions programmatically,
| so just logging is wrong choice and if there's a call back
| to client, client can decide whether to log it and how.
| esrauch wrote:
| It's a nice idea but I've literally never seen it done,
| so I would be interested if you have examples of major
| libraries that do this. Abstractly it doesn't really seem
| to work to me in place of simple logs.
|
| One test case here is that your library has existed for a
| decade and was fast, but Java removed a method that let
| you make it fast, but you can still run slow without that
| API. Java the runtime has a flag that the end use can
| enable to turn it back on a for a stop gap. How do you
| expect this to work in your model, you expect to have an
| onUnnecessarilySlow() callback already set up that all of
| your users have hooked up which is never invoked for a
| decade, and then once it actually happens you start
| calling it and expect it to do something at all sane in
| those systems?
|
| Second example is all of the scenarios where you're some
| transitively used library for many users, it makes and
| callback strategy immediately not work if the person who
| needs to know about the situation and could take action
| is the application owner rather than the people writing
| library code which called you. It would require every
| library to offer these same callbacks and transitively
| propagate things, which would only work if it was just
| such a firm idiomatic pattern in some language ecosystem
| and I don't believe that it is in any language ecosystem.
| ivan_gammel wrote:
| > library has existed for a decade
|
| >but Java removed a method that let you make it fast, but
| you can still run slow without that API
|
| I'd like to see an example of that, because this is
| extremely hypothetical scenario. I don't think any
| library is so advanced to anticipate such scenarios and
| write something to log. And of course Java specifically
| has longer cycle of deprecation and removal. :)
|
| As for your second example, let's say library A is smart
| and can detect certain issues. Library B depending on it
| is at higher abstraction level, so it has enough business
| context to react on them. I don't think it's necessary to
| propagate the problem and leak implementation details in
| this scenario.
| esrauch wrote:
| Protobuf is the example I had in mind. It uses
| sun.misc.Unsafe which is being removed in upcoming Java
| releases, but it has a slow fallback path. It logs a
| warning when it runs if it can tell it's only using the
| fallback path but the fast path is still available if the
| application owner set a flag to turn it back on if they
| want to:
|
| https://github.com/protocolbuffers/protobuf/issues/20760
|
| Java Protobuf also logs a warning now if you can tell you
| are using gencode old enough that it's covered by a DoS
| CVE. They actually did a release that broke compatability
| of the CVE covered gencode but restored it and print a
| warning in a newer release.
| dpark wrote:
| I've written code that followed this model, but it almost
| always just maps to logging anyway, and the rest of the
| time it's narrow options presented in the callback. e.g.
| Retry vs wait vs abort.
|
| It's very rarely realistic that a client would code up
| meaningful paths for every possible failure mode in a
| library. These callbacks are usually reserved for
| expected conditions.
| ivan_gammel wrote:
| > almost always just maps to logging anyway
|
| Yes, that's the point. You log it until you encounter it
| for the first time, then you know more and can do
| something meaningful. E.g. let's say you build an API
| client and library offers callback for HTTP 429. You
| don't expect it to happen, so just log the errors in a
| generic handler in client code, but then after some
| business logic change you hit 429 for the first time. If
| library offers you control over what is going to happen
| next, you may decide how exactly you will retry and what
| happens to your state in between the attempts. If library
| just logs and starts retry cycle, you may get a
| performance hit that will be harder to fix.
| TZubiri wrote:
| We call those warnings, and it's very common to downgrade
| errors to warnings by wrapping an exception and printing
| the trace as you would an exception.
| MobiusHorizons wrote:
| What you are proposing sounds like a nightmare to debug. The
| high level perspective of the operation is of course valuable
| for determining if an investigation is necessary, but the low
| level perspective in the library code is almost always where
| the relevant details are hiding. Not logging these details
| means you are in the dark about anything your abstractions
| are hiding from higher level code (which is usually a lot)
| TZubiri wrote:
| You can log your IO and as long as your functions are
| idempotent that should be enough info to replicate.
| dpark wrote:
| Assuming everything is idempotent is a tall order.
|
| There are a lot of libraries that haven non-idempotent
| actions. There are a lot of inputs that can be
| problematic to log, too.
| Too wrote:
| Simple: include those relevant details in the exceptions
| instead of hiding them.
| layer8 wrote:
| It's not that simple. First, this results in exception
| messages that are a concatenation of multiple levels of
| error escalation. These become difficult to read and have
| to be broken up again in reverse order.
|
| Second, it can lose information about at what exact time
| and in what exact order things happened. For example,
| cleanup operations during stack unwinding can also
| produce log messages, and then it's not clear anymore
| that the original error happened before those.
|
| Even when you include a timestamp at each level, that's
| often not sufficient to establish a unique ordering,
| unless you add some sort of unique counter.
|
| It gets even more complicated when exceptions are
| escalated across thread boundaries.
| Kwpolska wrote:
| Not all problems cause exceptions.
| awesome_dude wrote:
| At the extreme end: If my Javascript frontend is being
| told about a database configuration error happening in
| the backend when a call with specific parameters is made
| - that is a SERIOUS security problem.
|
| Errors are massaged for the reader - a database access
| library will know that a DNS error occurred and that is
| (the first step for debugging) why it cannot connect to
| the specified datastore. The service layer caller does
| not need to know that there is a DNS error, it just needs
| to know that the specified datastore is uncontactable
| (and then it can move on to the approriate resilience
| strategy, retry that same datastore, fallback to a
| different datastore, or tell the API that it cannot
| complete the call at all).
|
| The caller can then decide what to do (typically say
| "Well, I tried, but nothing's happening, have yourself a
| merry 500)
|
| It makes no sense for the Service level to know the
| details of why the database access layer could not
| connect, no more than it makes any sense for the database
| access layer to know why there is a DNS configuration
| error - the database access just needs to log the reasons
| (for humans to investigate), and tell the caller (the
| service layer) that it could not do the task it was asked
| to do.
|
| If the service layer is told that the database access
| layer encountered a DNS problem, what is it going to do?
|
| Nothing, the best it can do is log (tell the humans
| monitoring it) that a DB access call (to a specific DB
| service layer) failed, and try something else, which is a
| generic strategy, one that applies to a host of errors
| that the database call could return.
| cwillu wrote:
| Those details don't belong in the error log level, that's
| what info or trace is for.
| dpark wrote:
| They were replying to a person who says "it's almost
| always wrong for library functions to log anything". Not
| just errors.
| Etherlord87 wrote:
| This seems like such an obvious answer to the problem, your
| program isn't truly modularized if logging is global. If an
| error is unexpected it should bubble all the way up, but if
| it's expected and dealt with, the error message should be
| suppressed or its type changed to a warning.
| dpark wrote:
| I've worked on systems with "modularized" logging. It's
| never been pleasant because investigations involve
| stitching together a bunch of different log sources to
| understand erase actually happened. A global log dump with
| attribution (module/component/file/line) is far easier to
| work with.
| cogman10 wrote:
| Depending on the language and logging framework, debug/trace
| logging can be acceptable in a library. But you have to be
| extra careful to make sure that it's ultimately a no-op.
|
| A common problem in Java is someone will drop a log that
| looks something like this `log.trace("Doing " + foo + " to "
| + bar);`
|
| The problem is, especially in a hot loop, that throw away
| string concatenation can ultimately be a performance problem.
| Especially if `foo` or `bar` have particularly expensive
| `toString` functions.
|
| The proper way to do something like this in java is either
| log.trace("Doing $1 to $2", foo, bar);
|
| or if (log.traceEnabled()) {
| log.trace("Doing " + foo + " to " + bar); }
| TZubiri wrote:
| How about wrapping the log.trace param in a lambda and
| monkeypatching log.trace to take a function that returns a
| string, and of course pushing the conditional to the
| monkeypatched func.
| 01HNNWZ0MV43FF wrote:
| That is why the popular `tracing` crate in Rust uses
| macros for logging instead of functions. If the log level
| is too low, it doesn't evaluate the body of the macro
| renewiltord wrote:
| Conflicting goals for the predominant libraries is what
| causes this. Log4J2 has a rewrite appender that solves the
| problem. But if you want zero-copy etc I don't think there's
| such a solution.
| HendrikHensen wrote:
| > Should only "top-level" code ever log an error? That can make
| it difficult to identify the low-level root causes of a top-
| level failure.
|
| Some languages (e.g. Java) include a stack trace when reporting
| an error, which is extremely useful when logging the error. It
| shows at exactly which point in the code the error was
| generated, and what the full call stack was to get there.
|
| It's a real shame that "modern" languages or "low level"
| languages (e.g. Go, Rust) don't include this out of the box, it
| makes troubleshooting errors in production much more difficult,
| for exactly the reason you mention.
| layer8 wrote:
| The point in the code is not the same information as knowing
| the time, or knowing the order with respect to operations
| performed during stack unwinding. Stacktraces are very
| useful, but they don't replace lower-level logging.
| leni536 wrote:
| I make error logs fail happy path functional/integration tests
| for the backend applications I'm currently writing.
| georgefrowny wrote:
| Easy to say, but there's "yes we know this is wrong but this will
| have to do for now" and "we don't expect to see this in real life
| unless something has gone sideways".
| oofbey wrote:
| At scale the rare events start to happen reliably. Hardware
| failures almost certainly cause ERROR conditions. Network
| glitches.
|
| Our production system pages oncall for any errors. At night it
| will only wake somebody up for a whole bunch of errors. This
| discipline forces us to take a look at every ERROR and decide
| if it is spurious and out of our control or something we can
| deal with. At some point our production system will reach a
| scale where there are errors logged constantly and this
| strategy Durant make sense any more. But for now it helps keep
| our system clean.
| mschuster91 wrote:
| > If error level messages are not such a sign, I can assure you
| that most system administrators will soon come to ignore all
| messages from your program rather than try to sort out the mess,
| and any actual errors will be lost in the noise and never be
| noticed in advance of actual problems becoming obvious.
|
| Bold of you to assume that there are system administrators. All
| too often these days it's "devops" aka some devs you taught how
| to write k8s yamls.
| Too wrote:
| Agree with the post. The job of blackbox is to turn probes into
| metrics. If a probe fails, that should just become a
| probe_success=0 metric. Blackbox did its job and should not log
| an error.
| plandis wrote:
| I agree. Error or higher should result in an alarm and indicates
| that some corrective action needs to be taken.
| BiraIgnacio wrote:
| It means something is wrong, yes. Now, if it's worth fixing
| (granted, most of the time it would), that's another story.
| Kinrany wrote:
| Why are logs usually assumed to be for human consumption only? It
| seems weird to me that log storage usually exists outside of the
| system and isn't a general purpose message bus.
| azov wrote:
| If my system doesn't work - I want to be alerted. If notification
| was supposed to be sent but wasn't - it's an error regardless of
| whether it wasn't sent because of a bug in my code or external
| service being down. It may be a warning if I'm still retrying,
| but if I gave up - it's an error.
|
| "External service down, not my problem, nothing I can do" is
| hardly ever the case - e.g. you may need to switch to a backup
| provider, initiate a support call, or at least try to figure out
| why it's down and for how long.
| 29athrowaway wrote:
| Input errors do not need fixing, so no.
| hedayet wrote:
| I agree with the principle: log level error should mean someone
| needs to fix something.
|
| This post frames the problem almost entirely from a sysadmin-as-
| log-consumer perspective, and concludes that a correctly
| functioning system shouldn't emit error logs at all. That only
| holds if sysadmins are the only "someone" who can act.
|
| In practice, if there is a human who needs to take action -
| whether that's a developer fixing a bug, an infra issue, or
| coordinating with an external dependency - then it's an error.
| The solution isn't to downgrade severity, but to route and notify
| the right owner.
|
| Severity should encode actionability, not just system
| correctness.
| peanut-walrus wrote:
| Disagree. If you have an error that NEEDS fixing, your program
| should exit. Error level logs for operation level errors are
| fine.
| blkflcn3 wrote:
| > What an error log level should mean (a system administrator's
| view)
|
| That says it all:
|
| - Backseat driving
|
| - Not a developer by trade
| dpc_01234 wrote:
| Error log level should be renamed. It's just a terrible name that
| confuses usage.
| jedberg wrote:
| I feel like it's more nuanced than OP writes. Presumably every
| log line comes from something like a try/catch. An edge case was
| identified, and the code did something differently.
|
| Did it do what it was supposed to do, but in a different way or
| defer for retrying later? Then WARN.
|
| Did it fail to do what it needed to do? ERROR
|
| Did it do what it needed to do in the normal way because it was
| totally recoverable? INFO
|
| Did data get destroyed in the process? FATAL
|
| It should be about what the result was, not who will fix it or
| how. Because that might change over time.
| Waterluvian wrote:
| I think this is one of those discussions where there's no one
| right answer (though there's many wrong answers). All you have to
| do is pick a reasonable definition, write it down, socialize it,
| and be consistent when using it.
|
| I think discussions that argue over a specific approach are a
| form of playing checkers.
| rsanek wrote:
| If something needs to be fixed, why is it just a log? How is
| someone supposed to even notice a random error log? At the places
| that I've worked, trying to make alerting be triggered on only
| logs was always quite brittle, it's just not best practice. Throw
| an exception / exit the program if it's something that actually
| needs fixing!
| Copenjin wrote:
| > If something needs to be fixed, why is it just a log?
|
| What he meant is that is an unexpected condition, that should
| have never happened, but that did, so it needs to be fixed.
|
| > How is someone supposed to even notice a random error log?
|
| Logs should be monitored.
|
| > At the places that I've worked, trying to make alerting be
| triggered on only logs was always quite brittle, it's just not
| best practice.
|
| Because the logs sucked. It not common practice, it should be
| best practice.
|
| > Throw an exception / exit the program if it's something that
| actually needs fixing!
|
| I understand the sentiment, but some programs cannot/should not
| exit. Or you have an error in a subsystem that should not bring
| down everything.
|
| I completely agree with the approach of the author, but also
| understand that good logging discipline is rare. I worked in
| many places where logs sucked, they just dumped stuff, and had
| to restructure them.
| mycall wrote:
| Severity is the value and you set thresholds based on context of
| the error type.
| umpalumpaaa wrote:
| What I like about objective-c's error handling approach is that a
| method that can fail is able to tell if a caller considers error
| handling or not. If the passed *error is NULL you know that that
| is no way for a caller to properly handle the error. My
| implementations usually have this logic:
|
| if error == NULL and operationFailed then log error Otherwise Let
| client side do the error handling (in terms of logging)
| Glyptodon wrote:
| I agree errors should be errors. Many things that are logged for
| other reasons should use a different label.
|
| That said, the thing I've cone find being useful as a subcategory
| of error are errors due to data problems vs errors due to other
| issues.
| jmull wrote:
| I encourage people to think a few moments about what to log and
| at what level.
|
| You're kind of telling a story to future potential trouble-
| shooters.
|
| When you don't think about it at all (it doesn't take much), you
| tend to log too much and too little and at the wrong level.
|
| But this article isn't right either. Lower-level components
| typically don't have the context to know whether a particular
| fault requires action or not. And since systems are complex, with
| many levels of abstractions and boxes things live in, actually
| not much is in a position to know this, even to a standard of
| "probably".
| Xss3 wrote:
| Some programs are error resistant and need an additional level:
| Fatal.
|
| A warning can be ignored safely. Warnings may be 'debugging
| enabled, results cannot be certified' or something similar.
|
| An error should not be ignored, an operation is failing, data
| loss may be occurring, etc.
|
| Some users may be okay with that data loss or failing operation.
| Maybe it isnt important to them. If the program continues and
| does not error in the parts that matter to the user, then they
| can ignore it, but it is still objectively an error occurring.
|
| A fatal message cannot be ignored, the system has crashed. Its
| the last thing you see before shutdown is attempted.
___________________________________________________________________
(page generated 2025-12-20 23:00 UTC)