[HN Gopher] Log4j: The pain just keeps going
___________________________________________________________________
Log4j: The pain just keeps going
Author : CrankyBear
Score : 237 points
Date : 2022-07-19 18:03 UTC (1 days ago)
(HTM) web link (thenewstack.io)
(TXT) w3m dump (thenewstack.io)
| neals wrote:
| As a small time hobby / small business server administrator
| (enthusiast) , what's the best practice here?
| AndrewDucker wrote:
| Make sure you have the time to patch/update your applications
| when security issues are raised.
| LaGrange wrote:
| Whoa whoa whoa, slow down, that would mean somebody in the
| organization would have to spend _valuable senior developer
| time_ on reading things.
|
| Yes, that's jaded, but that's what I'd expect. So far I've
| yet to feel like a job _encouraged me_ to pay attention to
| security updates, accessibility standards, or anything like
| that. And it's quite hard to keep myself motivated to do
| that, when it's pretty clear most software products are
| disposable anyway.
| thechao wrote:
| I hired a guy _because_ he loves to be (technically)
| correct about everything, and he does so by reading (and
| REREADING) specs. Gives my team a superpower.
| AndrewDucker wrote:
| You can automate some of this. BlackDuck scans your
| dependencies (if you're using Maven or similar) to check
| what versions of dependencies you're using and if there are
| known CVEs for them. And there are plenty of competitors.
|
| But you still need to patch and test when you find out
| there's an issue.
| kevinmgranger wrote:
| As an alternative / addition to the other suggestions, maintain
| an internal artifact store that syncs with upstream stores, but
| enforce that vulnerable versions aren't present. Block access
| to the upstream artifact stores so builds have to use the
| internal one.
| mikeweiss wrote:
| If want an extremely strong mitigating control that you can
| design into your architecture..... allow internet egress in
| your production environment only to a list of domains/ips in an
| allowlist. This or control egress with an explicit proxy.
|
| If vulnerable systems have no path to the internet at large
| then it's extremely difficult for attackers to know if a system
| is vulnerable to log4j and even harder to actually use it to
| exfiltrate data.
| shagie wrote:
| Put a dependency check in your builds. Raise build errors when
| something is bad. Have it build on a regular basis (could be
| every month)... and then fix things.
|
| https://owasp.org/www-project-dependency-check/
|
| And some examples -
| https://jeremylong.github.io/DependencyCheck/dependency-chec...
| brabel wrote:
| Every case is different... but for the log4shell case in
| particular:
|
| * newer JVM versions protected against the main attack (JNDI),
| so if you were on any JDK version released in the last couple
| of years, you wouldn't be vulnerable. * Java serialization
| filters [1], which are highly recommended to be setup in any
| Java application, would also have prevented the attacks. * not
| including large libraries with lots of unused features could
| have prevented this. In this case, log4j has support for a
| dozen or so things that should've been opt-in (and I think they
| are now) but were enabled by default in every installation.
|
| In summary: actually paying attention to possible attacks and
| ensuring you're defended against them, and reducing the surface
| area for attacks makes your application much safer.
|
| [1] https://docs.oracle.com/javase/10/core/serialization-
| filteri...
| WesolyKubeczek wrote:
| Sometimes people ask me why I'm so skeptical about software
| delivery techniques involving bundling and static compilation.
| This is a thing that people sometimes ask.
| yjftsjthsd-h wrote:
| It's not a question of bundling / static builds, it's a
| question of dependencies and updates. If you build a static
| binary from maintained inputs (say, nix, Gentoo+portage,
| whatever) in CI and automatically distribute it, you're golden.
| If you completely separate a library out and... heck, make the
| user supply their own copy! but don't monitor it for updates,
| don't notify users, don't ship patches ever, then you're in a
| bad spot. Static/dynamic isn't the issue.
| WesolyKubeczek wrote:
| The beauty of the unbundled approach is, though, that you
| don't need to rebuild anything except the vulnerable thing.
| It scales. Having to rebuild the whole world, not so much,
| even if you can do it. Maybe you have access to an infinite
| supply of zero-cost energy, but I don't, and neither does the
| world at large.
|
| When they find a critical bug in a Go module or a Rust crate,
| you usually end up with a litany of packages that had to be
| updated as well because everyone and their crab relies on
| that crate/module.
|
| Count in release managers (or whatever that's called at your
| shop) who pin all dependencies to the most exact minor
| version before shipping, because they have learned to do it
| like a mantra without actually understanding all the "whys"
| of it and just keep religiously applying that hammer
| everywhere.
|
| People keep complaining about how distributions work and how
| the traditional package management sucks all the time (of
| thousands complaining, I estimate that maybe 20 people
| actually know what they are talking about and thus are
| qualified to whine), but they somehow keep forgetting that
| those tools exist for a reason and that they are about to
| rediscover it the hard way. Chesterton's fence and all that.
| 0xbadcafebee wrote:
| It's funny how every Go project seems to statically compile in
| the same libraries. If just one of them has a serious vuln,
| we're talking nearly every Go project in the world having to be
| patched and recompiled, or upgraded with potentially breaking
| changes. And as we know from Log4j, that can be incredibly
| difficult. Just _finding_ all the affected Go apps will be a
| nightmare, and patching will not be as simple as "replacing a
| jar in a zip file".
| chrisseaton wrote:
| > nearly every Go project in the world having to be patched
| and recompiled
|
| Is that a particular problem? What's the difficulty with
| recompiling?
| SomeCallMeTim wrote:
| The advantage of being able to build to a binary is that
| you can then distribute the binary directly instead of
| distributing source and forcing people to rebuild it.
|
| When 98% of your users only have the binary and not the
| source code it means 98% of your users won't even be aware
| that they need a new version that isn't vulnerable.
|
| Remember the vulnerability disclosure will be something
| like "Log4Go has a vulnerability!!", not that the binary
| apps you've downloaded have a vulnerability. So if it
| affects 200 projects, every one of those projects will
| _also_ have to publish vulnerability notifications
| independently.
|
| And frankly 80% of those projects will likely be abandoned
| or at least neglected after a couple of years, leaving many
| people with vulnerable binaries they don't even know to
| check for vulnerabilities.
|
| In other words, they very advantage of Go (easy to
| distribute binaries directly!) is a security disadvantage.
| If you instead installed the app using a package manager,
| the package manager could at least notify you that you have
| a package with a security issue. But that's nothing new;
| security and ease-of-use are often at odds.
| 0xbadcafebee wrote:
| Imagine you are a random person in charge of a computer.
| You hear there's this big vuln affecting Go programs.
|
| 1. How do I find out which programs on my system are Go
| programs? I have a lot of programs scattered all over.
| Someone will have to come up with a method (probably search
| through _$PATH_ for files, run _`strings`_ on it, grep for
| _" Go"_, pray that's enough) to identify them.
|
| 2. They don't list a website, so I have to google the name
| of every program and find their website.
|
| 3. How do I tell which ones are affected by the vuln? I can
| hope their websites tell me, but maybe they're no longer
| maintained, or they haven't updated their READMEs.
|
| 4. If they don't provide a patched version (most probably
| won't), I might have to upgrade, which may break things
| (probably will), assuming they even have an upgradeable
| version that isn't affected.
|
| 5. If I can't upgrade and have to patch, am I even a
| software developer and know how to do that?
|
| 6. If I luckily happen to be one, what system compiled this
| app 8 years ago, with what dependencies, with what version
| of Go, etc? Can I recreate that system and learn their
| build tools to finally patch & recompile? (I am not a Go
| developer and it has taken me days to figure out how to get
| different Go programs to compile, for reasons I don't
| understand)
|
| 7. Once I apply the patch how do I test it? (Log4j had
| multiple iterations of patches/mitigations, some didn't
| work)
|
| If I can't find the source code, can't upgrade it, & can't
| patch it, I am just stuck with a vulnerable version until I
| can somehow replace the program. For the programs I was
| able to identify as vulnerable, anyway.
|
| Log4j was "easy" compared to this, because you could just
| find all the jars on your system and unzip them, find a
| file, replace it, zip it back up. Log4j was actually a
| nightmare though, and Go apps will be worse.
| chrisseaton wrote:
| > How do I find out which programs on my system are Go
| programs?
|
| Well how are these programs getting onto your system?
|
| Presumably via some package manager, or an image builder?
|
| Don't those manage the update for you?
|
| If you're running code that you don't know who wrote,
| with no source code, that you can't patch, that nobody is
| taking responsibility for, then yeah you've got issues.
| But you had issues already if that's the predicament
| you're in. Some of this applies to Java too - obfuscation
| breaks the fixes you described using!
| kelnos wrote:
| > _If you 're running code that you don't know who wrote,
| with no source code, that you can't patch, that nobody is
| taking responsibility for, then yeah you've got issues._
|
| This is the default state of affairs for nearly everyone
| who uses a computer.
|
| We can either throw our hands up and futilely say "well
| they shouldn't do that!", or we can try to solve the
| problem.
| 0xbadcafebee wrote:
| Most people download and install Go programs by hand
| since they're statically compiled, it's their main
| selling point
|
| > If you're running code that you don't know who wrote,
| with no source code, that you can't patch, that nobody is
| taking responsibility for, then yeah you've got issues
|
| 25 million computers around the world still use Windows
| XP. It was launched 21 years ago, EOL 8 years ago. Very
| common actually, especially since most software in the
| world is not open source. People just keep using software
| that works.
| preseinger wrote:
| > You hear there's this big vuln affecting Go programs.
|
| This isn't relevant. What's relevant is if any of the
| specific programs I am responsible for have a
| vulnerability. If so, I need to update those specific
| programs. The notion of implementation language, shared
| library, dependency, these are all abstraction leaks. The
| atomic unit of responsibility, at the machine level, is
| program/service.
| kelnos wrote:
| I think that's the parent's point. The hypothetical big
| announcement that this hypothetical "Log4Go" library has
| a vulnerability will not include a comprehensive list of
| all applications that use the library, so then we have to
| hope that the developers of every single application that
| uses Log4Go hears about the vulnerability, updates their
| software, makes a new release, and then notifies all
| their users somehow.
|
| Alternatively, a more-savvy user has to hear about this
| Log4Go vulnerability and see if any of their applications
| use it and might need an update (which might involve
| contacting the maintainer to _inform them_ that they need
| to update their software). But the vast majority of users
| aren 't even going to hear about the Log4Go
| vulnerability, let alone understand what it means or what
| they need to do to protect themselves.
|
| The safest thing would be for every application to
| dynamically link to a system-provided version of Log4Go,
| and OS maintainers would be the ones to send out an
| update via their OS's auto-update service. This does
| create _other_ problems, but at least they are problems
| that technically-unsophisticated users do not need to
| solve or concern themselves with.
|
| (Of course, there is no mechanism to do this with Go
| modules, so the entire thing is moot.)
|
| This is why I get annoyed when developers say "I have to
| vendor library X into my application, because I'm afraid
| that the system copy will get updated to a version that
| breaks my app". Well then, maybe you shouldn't use a
| library whose author can't abide by semver and provide a
| reasonably-strong promise that minor and patch version
| updates won't break things. Otherwise you are just
| outsourcing your problems to your users.
| zdragnar wrote:
| Easy, just don't write bugs. /s
|
| That said, I think this missing the full picture- we have the
| same problem further down the stack, at the OS level, where
| old machines are running without receiving security updates
| on a regular basis.
|
| I don't know that there's really much of a good answer to the
| problem short of "don't leave your children unattended".
| Unlike real kids and the real world, there's plenty of bad
| actors looking for unattended services to stab.
| skybrian wrote:
| The situation for Go seems to have gotten a bit better with
| Go 1.18. Go binaries now include the versions of all the
| modules in them, and this can be listed with the "go version"
| command.
|
| That won't help you rebuild the binary, but at least you can
| find them.
| jewel wrote:
| That's fantastic. Since Go has gone all-in on static
| compiling, it'd be awesome if they made it possible to
| update a dependency without recompiling (or even having the
| source code for) the rest of the program.
|
| In addition to making it easier to do security updates in
| situations like log4j, it'd also make it easy to comply
| with LGPL license terms.
| er4hn wrote:
| Wouldn't that require the ABI, or at least the API, to be
| the same across versions of the affected library? That
| feels like it may be possible with enough tooling, but
| still very hard to enforce.
| 0xbadcafebee wrote:
| The library being patched needs to use the same ABI, yes.
| So you can usually only patch the same version of a
| library, not upgrade it.
|
| Afaik, the ELF format allows you to replace sections in a
| binary. The code (or data) is packed as "Sections" and
| mapped to memory locations. You use a tool to unpack the
| ELF sections and replace them. The application accesses
| those sections of memory and uses whatever ABI the
| application and library are designed to interact with. As
| long as you patch a section with the same ABI that an
| app/library used before, nobody will notice. And the ELF
| ABI never changes. But of course, not every system uses
| ELFs.
|
| But all this depends on the compiler/linker/etc and how
| they generate sections and store them in the binary (for
| example, is it all compiled into one blob-section, or
| does it store libraries as separate sections, which might
| be inefficient?). I somehow doubt Go does things in such
| a way to allow this kind of patching. They probably
| generate a giant blob full of random functions and record
| versions in a header without the ability to extricate a
| whole library.
| kllrnohj wrote:
| That doesn't seem possible. The benefits of static
| linking are that unused methods, symbols, etc... can be
| stripped as well as cross-library optimizations like in-
| lining. There's not necessarily going to be library lines
| left at all, at least not in any way that looks anything
| like upstream. And it's also going to vary from binary to
| binary, depending on what exactly they used from the
| dependency.
|
| What you're describing would be more akin to shared
| linkage just in a single blob. That seems like it'd be a
| worst-of-both-worlds result.
| kevincox wrote:
| I also like now "single file deployment" is considered a
| feature. It makes it easier for people without proper
| packaging and distribution tools. Of course these people
| without packing and distribution tools will now have a hard
| time finding out everywhere they have copied these binaries
| to.
| LaGrange wrote:
| packageblabla-1.2.3.deb is a single file.
| Beltalowda wrote:
| That will only work on Debian. And that "single file"
| requires also the libfoo-1.5.deb file (must be >=1.5, not
| 1.4 and not higher than 2.0!) etc.
| WesolyKubeczek wrote:
| The thing about "single file deployment" is that it
| shouldn't have external requirements, otherwise it's not
| a single file deployment anymore.
| kelnos wrote:
| ... which is why the parent is (IMO correctly) claiming
| that the grandparent's assertion is incorrect that a
| Debian package is a "single file deployment".
| WesolyKubeczek wrote:
| I think both grandparent you're referring to and its
| parent are lost in the woods.
| shagie wrote:
| I'm not as familiar with Go... I do know that we use a
| dependency scan that looks at the docker images and
| recognizes vulnerable versions of various Java libraries.
|
| With Go, can you look at a compiled binary in an image on the
| artifact repo and identify what libraries it used and if
| those libraries have vulnerabilities?
|
| (and it appears that my question was answered in a sibling
| comment)
| hedora wrote:
| Java's "statically link then scan" approach doesn't really
| work though. (Some sibling threads explain why.)
|
| Even Linux distributions where rebuilding the world is easy
| (like Debian) insist on breaking dependencies out into
| shared libaries, since tracking ad hoc vendored
| dependencies is just too difficult.
|
| For example, sometimes people play dynamic loading and
| renaming tricks to allow multiple versions of the same Java
| library to coexist. That can usually be detected by
| vulnerability scanners, but not always.
| shagie wrote:
| I'm not sure what you mean by "statically link then scan"
| in the Java context. The fingerprint of the vulnerable
| class files are the same no matter where they are stored.
| They could be bundled together, or shoved in a fat jar,
| or exploded into all their classes -- but the class file
| always is distinct and identifiable.
| hedora wrote:
| What if the project recompiled them from source with some
| other version of javac, and prepended "My" to each class
| name?
| shagie wrote:
| I am not familiar with any project that goes through such
| extremes to avoid vulnerability scanners.
|
| I am also not aware of any Java developers who build 3rd
| party libraries from source and avoid using maven (or
| gradle, or the odd one with ivy).
|
| Your "what if..." doesn't describe any workflow that I
| have ever experienced with Java in a professional (and
| even hobbyist) context.
| brabel wrote:
| I think OP is basically describing shading, which is
| actually quite common.
|
| https://maven.apache.org/plugins/maven-shade-plugin/
| shagie wrote:
| That leaves the original class file intact. Dependency
| vulnerability scanners can then identify the class file
| that is problematic.
|
| I have exploded war deployments (
| https://www.jrebel.com/learn/what-exploded-and-packaged-
| depl... ) put in a Tomcat image and the scanner is able
| to identify the libraries that the classes came from and
| their vulnerabilities.
|
| I've also got fat jars for Spring Boot batch jobs that
| are single jars with all of the dependencies packaged in
| them for ease of copying around a single thing on the
| server... and the dependency checker is able issues there
| to based on fingerprints of individual class files.
|
| These work on the "you downloaded this from a known
| package management server". As long as you are using
| maven to download the dependencies, a vulnerability
| checker will be able to identify it. One built into maven
| ( https://owasp.org/www-project-dependency-check/ //
| https://jeremylong.github.io/DependencyCheck/dependency-
| chec... ) will be able to catch these as part of the
| build itself - irrespective of the shading of the final
| artifact.
|
| If you are building from source, and obscuring the class
| names - yes, you can foil an external vulnerability
| checker (and since you're doing it from source, you are
| also not downloading it and thus avoiding a maven plugin
| use)... however, I don't know any projects that go
| through such lengths to compile 3rd party Java code
| themselves rather than using a jar, and are trying to
| evade vulnerability (and license) checkers.
| cesarb wrote:
| > Just finding all the affected Go apps will be a nightmare
|
| This isn't even a new situation. We had the exact same
| problem some time ago with zlib: it was also embedded
| _everywhere_ , and just finding all the affected apps was a
| nightmare. It's the reason most Linux distributions came to
| deeply dislike bundling libraries, instead of using a
| dynamically linked shared copy from a separate package.
| Unfortunately, it seems the lessons from that zlib incident
| are being forgotten, perhaps because too much time has passed
| (IIRC, in the order of decades) since then.
| kelnos wrote:
| > _It 's the reason most Linux distributions came to deeply
| dislike bundling libraries_
|
| Is it, though? Seems like it would be trivial for a
| distribution maintainer to write a script to go through the
| dependency trees of every package, and then automatically
| rebuild any that depend on the vulnerable library. I don't
| think _discovery_ is the problem.
|
| I suspect this is really a legacy of when disk space was
| not so cheap; dynamic linking means smaller binaries
| (usually). And, even today, many distros would probably
| prefer their users to only download a 2MB library update
| from them than have to download 500MB worth of updated
| programs. Bandwidth is still not as cheap as we'd hope.
|
| Edit: just realized you're probably talking about software
| packages vendoring in their own version of a library, not
| about a package statically linking with a system-provided
| copy of the library.
| aorth wrote:
| Surprised nobody mentioned yet: reload4j https://reload4j.qos.ch/
|
| It's a fork of log4j 1.2.17 by the original author that fixes a
| boat load of issues with 1.2.x and is a drop in replacement for
| log4j. Very useful if you can't upgrade to log4x 2.x.
| ChrisMarshallNY wrote:
| _> Welcome to how we use open source code. That's a big reason
| Software Bills of Materials (SBOM) have become essential for
| securing the software supply chain._
|
| I tend to do this, but it's pretty damn easy for me. I have a
| "2-level-dependency" rule that I generally go by, these days.
| i.e., Each dependency can have only one dependency, and that
| dependency can't have any others.
|
| The main way that I deal with it, is to write all my own
| dependencies, and I don't count toolsets, standard libraries, or
| platform frameworks (for better, or for ill).
|
| I really, _really_ want dependencies to work. They are a great
| idea, and I have been promoting modular architecture for decades.
|
| It's just that we are in a "wild west" situation, where we can't
| trust dependencies.
|
| I'm tired of hearing the phrase "It's a solved problem," with a
| reference to a dependency.
|
| I don't have the answer. I just have my own rule, which elicits
| scorn from most modern programmers (I'm painted as a luddite,
| refusing to "get with the times," or as a "tinfoil-hatter").
|
| WFM. YMMV.
| jeroenhd wrote:
| I like the "2-level-dependency" rule, but what do you do when
| one of your dependencies gains more dependencies? Do you stop
| updating and start migrating to something else, or maybe
| maintain a fork of your own?
|
| It's a very nice concept, but I kind of have my doubts about
| maintainability.
| ChrisMarshallNY wrote:
| I use my own dependencies. I've written many of them. Some of
| them, go "deeper," but only in dependencies that I authored.
|
| Here's the dependency manifest from the app I'm working on
| now (From my localization file): // MARK: -
| // MARK: - DO NOT TRANSLATE BELOW THIS LINE - //
| MARK: - "SLUG-VERSION-BMLT"
| = "BMLTiOSLib: 1.4.2"; "SLUG-VERSION-KEYCHAINSWIFT"
| = "KeychainSwift: 20.0.0"; "SLUG-VERSION-
| LGVCLEANTIME" = "LGV_Cleantime: 1.3.5";
| "SLUG-VERSION-UICLEANTIME" =
| "LGV_UICleantime: 1.1.1"; "SLUG-VERSION-AUTOFILL"
| = "RVS_AutofillTextField: 1.3.0"; "SLUG-VERSION-
| GCD" = "RVS_BasicGCDTimer:
| 1.5.0"; "SLUG-VERSION-CHECKBOX"
| = "RVS_Checkbox: 1.2.0"; "SLUG-VERSION-OBSERVER"
| = "RVS_GeneralObserver: 1.1.0"; "SLUG-VERSION-GST"
| = "RVS_Generic_Swift_Toolbox: 1.10.1"; "SLUG-
| VERSION-MB" =
| "RVS_MaskButton: 1.2.0"; "SLUG-VERSION-PP"
| = "RVS_Persistent_Prefs: 1.3.1"; "SLUG-VERSION-UKT"
| = "RVS_UIKit_Toolbox: 1.1.5"; "SLUG-VERSION-
| WHITEDRAGON" = "White Dragon SDK:
| 3.2.2";
|
| The only one of those that I didn't write, was
| KeychainSwift[0]. It makes dealing with the Keychain easy,
| and is a very simple dependency. If it went off the rails,
| I'd write something like it, myself.
|
| All the others, are in my own repos, as top-shelf-quality
| open-source modules.
|
| [0] https://github.com/evgenyneu/keychain-swift
| kelnos wrote:
| The problem I have with adopting something like this is
| that (other than the added time it would take to write and
| maintain all my own dependencies), I don't trust myself to
| avoid writing the same kinds of security bugs that a third-
| party dependency might also write. But those bugs are much
| more likely to be discovered and fixed in the widely-used
| third-party library than in my bespoke library, even if I
| publish that library as open source.
|
| Sure, I would never write these log4j security-related
| bugs, because I would never even think to write such
| obscure functionality into a logging library for my own
| use.
|
| Regardless: frankly, I think anyone who claims they won't
| write security bugs... well, that kind of person is
| probably a bit unrealistic about the flawlessness of own
| abilities, to put it as politely as possible.
| ChrisMarshallNY wrote:
| I don't claim to "never write security bugs."
|
| However, it appears that the norm in the industry, is to
| write lash-up code, as quickly as possible, in the hopes
| on an MVP that will get brought up by a FAANG.
|
| And I am doubtful that most OSS actually gets all that
| "other eyeballs vetting" that everyone talks about. Most
| folks that I know, that use a lot of these dependencies,
| would be unable to understand what goes on in them, let
| alone offer fixes.
|
| In fact, I remember a founder, on this very forum,
| declare that "If you don't get physically sick, looking
| at your MVP code, then you are spending too much time on
| code quality."
|
| That posture doesn't exactly inspire confidence. I'm
| pretty good at what I do, and I tend to "overengineer" my
| security-related stuff, because I don't really have huge
| confidence in that. It's not my forte.
|
| But writing good code, is something that I can do.
|
| Like I said, WFM. YMMV.
| kazinator wrote:
| That's what you get for not minding that a logging library needs
| over 300,000 lines of code.
| croes wrote:
| If I understand this correctly the log4j problem wasn't a bug but
| Java feature at the wrong place
| upupandup wrote:
| It's probable that there are more log4j type of exploits thats
| just waiting to be discovered. It's also becoming hard to tell
| which are private enterprises vs state sponsored cyber attack.
|
| It just seems like there's no solution. If Java is this bad
| imagine how much more attack surface a npm heavy project. Not
| only that the seeming hardware knowledge of state sponsored
| attackers seem sophisticated enough to gain access at the vendor
| level, which forces us to question even the most trusted
| methodologies and dependency management.
| hedora wrote:
| We could learn to write auditably correct software.
|
| Reflections on Trusting Trust explains why writing correct
| software is inadequate:
|
| https://dl.acm.org/doi/10.1145/358198.358210
| robertlagrant wrote:
| It could be that npm libraries' obsession with only doing one
| thing will mean that they are less likely to get the sort of
| feature bloat that caused log4j's current misery.
| spatley wrote:
| The dependencies are knowable in and software that is under
| development. But mitigations may be herculean.
|
| My big question is, how can we exclude features we do not want?
| It seems foolish in retrospect to give your logger exe
| functionality on the server.
|
| Where is my safeLog4j?
| commandlinefan wrote:
| > seems like there's no solution
|
| There is - don't depend on anybody else's software. It just
| means that software development will take longer and cost more.
| upupandup wrote:
| but the cost of rewriting everything that you are currently
| using through dozens of dependencies and third party libs you
| are going to go broke.
|
| Unless you have the budget like the DOD which has its own
| coding language I believe, you are ngmi
|
| Not to mention you would be introducing attack surfaces that
| wont have enough attention because its only being used by
| your team.
|
| At least with a widely used lib, vulnerabilities are caught,
| published (very important) and constantly raising vigilence.
| here you would have to perform your own threat mitigation and
| have bounty programs.
| Beltalowda wrote:
| This assumes that your own software is more secure.
| waynesonfire wrote:
| for what it's worth, the log4j vulnerability was due to
| some obscure feature that nobody uses. unlikely you'd even
| write such code.
| hyperman1 wrote:
| I never understood where log4j 2 was supposed to fit in the
| ecosystem.
|
| * We had log4j 1, which was good enough for most use cases and
| everywhere.
|
| * We had slf4j/logback, which was from the same authors, and
| broke backward compatibility but gave us a fundamentally better
| design.
|
| * We had java.util.logging, which while braindead was builtin to
| Java and hence available everywhere.
|
| *We had commons logging, which was never supposed to escape from
| tomcat, but got overused everywhere anyway.
|
| All of this was more logging frameworks than was healthy for 1
| language, and there are some more smaller ones.
|
| Then apache decides to put new people on log4j, do a backward
| incompatible v2 design that nevertheless is worse than slf4j.
| Why?
| shagie wrote:
| > Then apache decides to put new people on log4j, do a backward
| incompatible v2 design that nevertheless is worse than slf4j.
| Why?
|
| slf4j itself isn't a logging framework. It's a facade to
| logging frameworks.
|
| Simple Logging Facade for Java ( https://www.slf4j.org )
|
| It needs a logging framework behind it - log4j, log4j2,
| logback, commons, JUL.
|
| The question is "why do log4j2?"
|
| Logback went from the log4j1.x path ( https://logback.qos.ch )
|
| Log4j2 has a lot of features that weren't present when the
| project started (
| https://en.wikipedia.org/wiki/Log4j#Apache_Log4j_2 ).
|
| There is a licensing difference between Logback (LGPL) and
| Log4jx (Apache Commons).
| Aperocky wrote:
| > Log4j2 has a lot of features
|
| Apache projects in general have this feature creep seeping
| into every single one of their products that I have used.
|
| It's just hard to trust apache projects now.
| kelnos wrote:
| > _It needs a logging framework behind it - log4j_
|
| Minor nit: log4j doesn't implement the slf4j interface,
| though. If you do want log4j to do your logging, but use
| slf4j in your application, you need to include a "bridge"
| library that adapts slf4j to log4j. And, conversely, if you
| have dependencies that use log4j, but want things to end up
| going through your slf4j-supporting backend (say, logback),
| you need to exclude the "real" log4j jar and include a
| different library that implements log4j's interface and
| forwards calls to slf4j.
| vbezhenar wrote:
| Log4j2 does have some legitimate improvements over logback.
| These days you don't need anything fancy, just spew logs into
| stdout and something will capture it to your ELK, so those
| loggers seem obsolete indeed. Java need standard logging with
| slightly improved ergonomics and that's about it.
| marginalia_nu wrote:
| Yes, Java logging is ridiculously complicated for largely
| historical concerns that just don't make much sense in 2022.
|
| Like what you most likely need is a wrapper for System.out
| that _maybe_ does some clever buffering and backpressure
| management, but the rest is sensibly managed outside of the
| application; whether by logrotate, ELK, or whatever.
|
| Long story short, no modern Java logging framework has any
| sensible reason to be much longer than a few hundred lines of
| code.
| kelnos wrote:
| I don't 100% agree with this, because even when you're
| doing containers and just dumping your log output to
| stdout, you still might be shipping that to a log
| aggregation system that will work a lot better with
| structured logging. So you might want to output JSON or
| some other structured format, and you might want your
| logger to support tagging, markers, stuff like that.
|
| Even then, of course, this kind of logging should be able
| to be done in far fewer lines than most Java logging
| frameworks.
| smarx007 wrote:
| > Java need standard logging with slightly improved
| ergonomics
|
| That's exactly what JDK9+ java.lang.System.Logger is. See
| https://stackoverflow.com/questions/59976828/difference-
| betw...
| ptx wrote:
| It seems to be mainly intended for use internally in the
| JDK, though. From the JEP[1]: "It is not a goal to define a
| general-purpose interface for logging. The service
| interface contains only the minimal set of methods that the
| JDK needs for its own usage."
|
| And from the motivation section: "The proposed service
| enables applications to configure the JDK to use the same
| logging framework as the application: It would only need to
| provide an implementation of the service that returns
| platform loggers that wrap the loggers of the preferred
| logging framework."
|
| [1] https://openjdk.org/jeps/264
| smarx007 wrote:
| Oh, thank you! Learned something today. Plus a lesson to
| rely less on tech blogs and RTFM more!
| layer8 wrote:
| > Then apache decides to put new people on log4j, do a backward
| incompatible v2 design that nevertheless is worse than slf4j.
| Why?
|
| Because the original Log4j author wanted to do his own thing
| for a Log4j successor (slf4j/logback), and the Apache Log4j
| maintainers also had ideas about a next-gen framework and
| wanted to do that.
|
| That's Open Source, everyone can do their own thing if they
| like, and some things will get adopted more than others. Log4j2
| got the most adoption because they had the Log4j "brand" and
| the Apache backing and more company-backed(?) maintainers.
|
| I don't consider java.util.logging to be braindead. It can in
| principle be used as a logging facade like slf4j, but not too
| many do that, and Log4j 1 had come first and kept that
| momentum.
|
| The real mistake was for Java to not have a standard logging
| API earlier. (It took six years after Java 1.0.)
| hyperman1 wrote:
| java.util.logging was braindead because it wanted to know the
| class and method doing the logging. If not provided, it threw
| and catched an exception and filtered the stack trace to
| detect its own caller. Throwing an exception was a very slow
| operation.
|
| I see they did some optimizations to do this work lazily if
| needed. But when log4j 1 was on its peak, that wasn't done,
| and using the built in logger was slow enough to have
| measurable impact .
|
| See https://github.com/openjdk/jdk/blob/6765f902505fbdd02f25b
| 599... at the bottom.
| YetAnotherNick wrote:
| Hearing logging library and next-gen framework in the same
| sentence would never not make me laugh.
| twic wrote:
| I use java.util.logging in a few apps, to cut down on
| dependencies, and i consider it to be fairly braindead. The
| fundamentals are okay, to the extent that they are copies of
| log4j, but there are a bunch of little things that just
| aren't done right, like:
|
| - logging methods taking arrays rather than varargs (this
| would be a trivial change!)
|
| - there being no way to create a SimpleFormatter with a
| custom format in code other than setting a system property
| before creating it
|
| - there being no Logger:getLogger override which takes a
| class, only strings
|
| At some point, i am going to write a little facade which
| wraps jul and fixes the papercuts, which feels like a very
| silly thing to be doing!
| EdwardDiego wrote:
| - the console appender writes to stderr.
| layer8 wrote:
| Those are cosmetic issues, in my book. The basic
| architecture is sound.
|
| In any serious application you will want to have a central
| logging wrapper anyway (I always use one), which
| automatically takes care of the varargs issue.
|
| The implementation of _SimpleFormatter_ is 30 lines that
| you can easily copy and adjust as needed, including with
| your own custom format.
|
| Regarding logger names, I always define an explicit set of
| logger names for the application, because that makes the
| logger names independent from implementation-detail
| subpackages, and any operations-side log filtering based on
| logger names will remains stable, regardless of how you
| refactor your implementation into Java packages. Basically,
| I see the logger names as part of the external interface of
| the application, and therefore I want to have them
| decoupled from the internal package structure, so that mere
| refactorings don't break that interface. You can still
| easily find all program locations that log to a specific
| logger, for example by Find Usages on the respective static
| _Logger_ constant.
| Foobar8568 wrote:
| Long live to SaaS products, no longer our problems hahaha...
| Going to cry in a corner.
| craigching wrote:
| > We are all in this together.
|
| The last line from the article and my favorite from Brazil which
| seems fitting?
| danhab99 wrote:
| I feel like if we as the open source community would require
| commit signing we would be in a safer position. Crypto signing
| doesn't block malware from being introduced, but it would make it
| harder to sneek under our noses.
|
| Currently in open source, you really don't know where your code
| is coming from and who worked on it. Git's "commit author" fields
| don't require any proof of identity, that's what GPG is for.
| vanattab wrote:
| Does anyone know if log4net the .net version of log4j is
| effected?
| WorldMaker wrote:
| log4net was never affected by the log4shell vulnerability as
| there is no .NET equivalent for JNDI [0]. JNDI is remote code
| execution as a service, and entirely a Java idea that that
| would be generally useful.
|
| Though if you are concerned about using a library with "log4"
| in the name and/or shuffled into the Apache graveyard in .NET,
| there are lots of useful replacement libraries.
| Microsoft.Extensions.Logging is "out of the box" in modern .NET
| and a simple NuGet away in older frameworks.
|
| [0]
| https://en.wikipedia.org/wiki/Java_Naming_and_Directory_Inte...
| xtracto wrote:
| Hey, English is not my first language. Is "effected" here the
| right word to use or would "affected" be the correct term? I
| see English speaking people use these two interchangeably so
| much online that I am starting to doubt what I learned in my
| English courses.
| eatsyourtacos wrote:
| "Affected" is the correct term here. People do use those
| wrong all the time. Honestly it's not always that they don't
| know which one (well ok it probably is, but doesn't have to
| be), but I find typing comments online my brain interchanges
| some things like that even though I know which one is
| correct.
| hello_erich wrote:
| "affected" should have been used. It is a common mistake to
| confuse the two.
| aoetalks wrote:
| It should be "affected"
| macintux wrote:
| "Affected" is correct. Fortunately most of the time you can
| use the wrong word and still be understood.
|
| Even with English as my first language, and as a life-long
| pedant, I still have to stop and think each time I use one or
| the other.
| teddyh wrote:
| Native English speakers mostly learned English by sound
| alone, so many such speakers regularly mix up words which
| sound very similar, like it's/its, effected/affected,
| site/sight, etc. Non-native speakers have instead frequently
| learned by mostly reading, and may therefore instead be
| lacking in correct pronunciation, but have no problem in
| distinguishing "it's" and "its", for example.
| NortySpock wrote:
| Affected would have been more correct.
|
| "Affected means that something was influenced or changed
| (e.g. the lyrics affected him).
|
| Effected means that something was brought about or
| facilitated (e.g. she effected the proposed changes)."
|
| It's definitely a subtle change, and, similarly, affect and
| effect tend to be mixed up as well
|
| - from https://prowritingaid.com/affected-vs-
| effected#:~:text=To%20....
|
| I will also recommend the book Uncovering The Logic of
| English, which is primarily aimed at native (American)
| English speakers but has several nods to English-as-a-second-
| language lessons.
|
| Cheers! Signed, an American who was corrected by his father
| quite often and now has an ear for these things.
| alexbakker wrote:
| I'm seeing this as well. While the amount of traffic has
| certainly decreased compared to the first couple of days after
| the CVE was announced, https://log4shell.tools is still being
| used by people every day.
| jacobyoder wrote:
| Had to deal with some automated scan tool telling a client "you
| have log4j - you're vulnerable!" when... we didn't. A project was
| including a dependency on SLF4J and _that_ project pulled in
| slf4j-log4j (IIRC). But it was not configured, wasn 't compiled
| in to the final jar, and ... even it was, it was _very old_ log4j
| (1.1 or 1.2 IIRC?). The vulnerability didn 't affect that older
| version.
|
| We had to spend a week back and forth with
|
| "no, we're not vulnerable".
|
| "But we see log4j is right there in the a file in the directory".
|
| "No, it's not vulnerable and it's not being used and we can't
| reasonably fork the entire dependency just to remove that one
| sub-dependency from our project's dependency"
|
| "But we see log4j is right there in the a file in the directory".
|
| Also, there was no budget to do any sort of rewrite/refactor/etc
| anyway.
| allochthon wrote:
| Typically there's a way to suppress specific warnings in
| systems like these. In your company's situation, I would look
| at moving away from a scanning system if it didn't allow
| overrides like this.
| alfalfasprout wrote:
| So far this is the best approach I've found. The scanning
| tools rarely include that ability but if you build tooling
| around them you can maintain exclusion lists, for particular
| vulnerabilities, library/version pairs, etc.
|
| Unfortunately it does mean there's no getting around having
| someone manually deal with false positives.
| hedora wrote:
| Once you realize this sort of shallow, automated audit exists
| to grease the wheels of some security theater operation, this
| sort of back and forth makes sense.
|
| False positives don't waste the security team's budget, and are
| the only surefire way to justify ongoing expenditures on the
| scanning tool.
| tekchip wrote:
| So somehow false positives negate all the actual positives
| caught and corrected? The only true solution is what? Manual
| audit of everything by some perfect human security
| practitioner? I suppose the same applies to automated
| development tools then.
|
| I will concede there probably are some firms out there acting
| poorly that way. When aren't there? Humans _sigh_. But by in
| large automation and the problems inherent are required.
| horsawlarway wrote:
| I was in this industry for a while (I did 5 years full time
| for a security company).
|
| > I will concede there probably are some firms out there
| acting poorly that way.
|
| This is a _hilarious_ take. Here 's mine: The vast majority
| of these firms are here for liability. They do not provide
| security, they provide security theater so that if/when
| something goes wrong, the client can claim to have followed
| best practices, and that it's not their fault.
|
| This style of theater is _RAMPANT_ in the industry.
| Literally most of them come in with some version of a
| shitty automated tool, often with false positives in the
| 95%+ range, and then you check off checkboxes to make legal
| happy, and ensure that your insurance (or your customer 's
| insurance) will pay if you get compromised.
|
| This is not limited to small companies, but is instead
| mostly how large banks, credit firms, and large enterprises
| work.
|
| The entire damn show is for liability, and half of these
| "Security professionals" can't do anything other than read
| the message coming out of their tool. They are _utterly_
| incompetent when it comes to applying logic to figure out
| whether a specific use of a "risky" tool/language feature
| a genuine problem, vs a god damned fucking regex match from
| their tool on something completely unrelated.
|
| I went in as a naive young dev, I came out with ZERO
| respect for this industry, and a healthy dose of skepticism
| around anything these folks are saying.
|
| Security researchers? Generally fine (although there's a
| new breed of them that simply submits inane non-
| vulnerabilities over and over to attempt to get bug
| bounties from large companies)
|
| Security advice from open source devs? Generally top notch,
| listen to it.
|
| Security advice from that contractor your company paid?
| Expect to have 95% false positives, and they'll miss the 2
| places you _know_ actually have issues. But it 's ok
| because you'll check all the boxes on their sheet and legal
| is happy.
|
| ---
|
| I'm waiting for insurance companies to wise up and stop
| covering companies who are breached. Prices are already
| shooting way up, since it turns out security theater does a
| _very_ bad job stopping real threats, and that 's what this
| industry is right now. Might was well be the TSA of
| software, gonna frisk you real good, and then flunk every
| fucking test.
| beAbU wrote:
| My project's client, a major local bank, mandates pen
| testing before we go live with the product that we're
| building.
|
| Due to time and resourcing constraints, and a fixed
| launch date (don't ask), we had to significantly reduce
| scope. We are currently launching with probably 30% of
| the original scope. Once live we'll iterativley add
| features and grow our product to bring it back to the
| original vision.
|
| Will we have to do pen testing again in the future? Nope.
| Even though a lot of our planned post launch features
| involve integrations with 3rd parties and the exchange of
| sensitive personal data.
|
| Pen testing is 100% a box ticking exercise in my opinion.
| And most of the shops that offer this service are set up
| to provide the service such that large corporates to
| appease legal.
|
| I really don't like pen testing as a practice and
| seriously dislike when it's a go live impediment.
| pas wrote:
| > I really don't like pen testing as a practice
|
| could you elaborate on this a bit? what's wrong with it?
| to me it seems the only logical thing to spend external
| security budget on. (or internal red team if the company
| is so bit it can afford it.)
| rtev wrote:
| What you've described is an issue with the bank's
| procedures, not pentesting.
| kriro wrote:
| Thank you, that is a very interesting take. I've always
| played with the idea of pivoting into the security
| industry because playing hack the box and the like is
| something I do in my free time and enjoy. Maybe I'll just
| do some bug bounties or something for fun instead. I
| think it's a bit different in Europe though at least
| anecdotally there seems to be more code audit and the
| like (more product security) instead of pen testing (more
| network/infrastructure security).
| ylk wrote:
| I'd say don't let yourself be discouraged by GP. Just
| look into a company before you apply. Many have public
| reports and/or security research, both of which you could
| use as indicators.
|
| Here's a repo with lots of public reports by various
| consultancies, you could use that as a starting point:
| https://github.com/juliocesarfort/public-pentesting-
| reports
| hedora wrote:
| You could set your build up to not auto-download
| questionable stuff from untrustworthy machines on the
| Internet.
| paulmd wrote:
| The problem isn't that audits are inherently worthless,
| it's that most of these tools are very low-quality
| implementations of the concept.
|
| In one of my past jobs I was an early-mover on doing a lot
| of our ops on Linux and the audit tools had no concept of
| the backport security model whatsoever. If you were running
| on some kind of LTS distro rather than a current distro, it
| would see "gosh you're 3 minor versions behind, you have a
| ton of unpatched vulnerabilities!", but in actuality you
| didn't, because the fixes got backported into security
| releases (the "31" in something like "3.1.22~ubuntu31" for
| example). But the tool was dumb and it just had a table
| that said "anything under 3.4.14 is vulnerable" even when
| it wasn't necessarily.
|
| OP's "log4j 1.x is not vulnerable to this the first place"
| is a similar thing where either someone forgot to put in a
| value for "min vulnerable version" or the tool doesn't
| understand the concept of min-version at all. To be fair
| that can be genuinely tricky in java, best-case you are
| reading manifest files to try and pick out a version, but
| some legacy stuff doesn't have manifests, _especially_ very
| legacy fat-jar stuff. Yes, that stuff didn 't go away, it's
| still out there in places!
|
| And to be clear this was a long-running issue... it wasn't
| a "oh our tool doesn't understand LTS, I get it" and they
| left me alone... this was every month they'd be back at me
| for a new list of utility vulnerabilities even though I
| repeatedly explained to them that I had set up apt-get to
| auto-upgrade every night and reboot, so we were running
| whatever the latest security backports were available, and
| that just like the last 10 times this was even more false-
| positives.
|
| And again, this isn't just "we have to run it by you no
| matter what" and they leave you alone either. What security
| really wanted was for me to run windows and manually
| install the updates and get back in line with everyone
| else, even though that would have been a _less_ secure
| outcome than my locked-down linux boxes. But this wasn 't
| my day-job at the company so to speak, it was helping out
| another project with some ops and it needed to be as low-
| effort as possible (and to be fair their requirements
| weren't steep, auto-patch-and-reboot was fine by them and
| never caused any breakage during my tenure there).
|
| The root problem is that these sorts of tools aren't a
| substitute for an actual security culture, they're often a
| symptom of a compliance requirement and the whole thing
| turns into a box-checking exercise. Someone is on their ass
| about this because of contractual requirements or PCI
| requirements, and they bought the cheapest thing off the
| shelf that would check the box, and they are implicitly
| showing you here that they don't even understand how to
| interpret the output of that.
| horsawlarway wrote:
| > The problem isn't that audits are inherently worthless,
| it's that most of these tools are very low-quality
| implementations of the concept.
|
| I think the problem is that the current audits _are_
| inherently worthless. I 've never seen another industry
| that would accept a 95% false positive rate from a tool.
| But that's on the low end from my experiences (I've done
| 4 major codebase audits, and worked in the security
| industry for 5 years)
|
| Tools that routinely spit out 1000 plus vulnerabilities,
| and 6 months later you've checked off all of them without
| a single valid vulnerability in the list (but 3 you found
| yourself during that time that were missed entirely by
| the shitty regex that is really the entire tool in
| question).
| bostik wrote:
| > _current audits are inherently worthless_
|
| And that's not helped by the warped incentives. Security
| has to be, fundamentally, assessed as a holistic thing
| and even highly skilled technical people can't really do
| that with modern systems. Now you add in auditors, a
| profession populated by glorified accountants, working
| off of gargantuan checklists.
|
| You invite and reward mediocrity. At best.
|
| Anyone good enough to actually understand and able to
| work with the technology will find a job in the industry,
| for twice the pay. And the same applies to regulators.
| They can't retain technically skilled staff, so they are
| filled with accountants. As a result we get rules and
| regulations, written _by_ technically incompetent
| accountants, _aimed at_ technically incompetent
| accountants.
|
| To fill the gaps we have shitty, false-positive ridden,
| noisy, outright useless tools that generate reports
| intended to _satisfy_ accountants. Security has next to
| nothing to do with that.
|
| And to top it off, we have an entire industry riddled
| with so-called security engineers who know how to run a
| tool (with some marginally helpful StackOverflow
| pointers) and export a report, but can't for their life
| actually interpret, let alone VERIFY, any of it.
|
| That has as much to do with security engineering as
| burning ants with a magnifying glass has to do with
| biology.
| spc476 wrote:
| Back in the late 90s, I was working at a small web hosting
| company (take note). One day, a 500+ page report of a
| recent PCI compliancy check landed on my desk. It was
| nothing but "OMFG! DOMAIN1 RESPONDED TO PING! YOU WILL BE
| PWOWNED! OMFG! DOMAIN1 HAS DNS RECORDS! YOU WILL BE PWONED!
| OMFG! DOMAIN1 HAS A WEB SERVER RUNNING! YOU WILL BE
| PWONED!". Over and over again. For every domain we have.
| Complete and utter garbage report.
|
| Better---just summarize the IPs scanned and report back
| which services were found running on said IPs. Then in an
| appendix, list why each service is (or might be)
| problematic. "Ping? Attackers might be able to figure out
| your network topology and that is bad because blah blah
| blah blah."
|
| But 500 pages of this automatic breathless garbage? Utter
| trash.
| MilStdJunkie wrote:
| Yup, that's us. Same deal exactly. They already stoppered our
| entire team for eight months with a tools audit a few years
| ago, so I hung that dead fish above the conversation when this
| came up. "Uh, ok, don't want to do that, sooooo . .
| recommendations?" "Let's isolate it, toss stuff at it, and see
| if it lights up" "Sounds good to me. FIXED"
| dotnet00 wrote:
| I recall that NVIDIA had to release an update to their CUDA
| Toolkit around when the Log4j vulnerability was announced. They
| weren't using it but the file was being distributed, presumably
| for a similar reason. I'm guessing they might've had similar
| complaints coming in.
| bostik wrote:
| We (read: I) face this on an ongoing basis.
|
| We can't upgrade the entire components that bundled up log4j
| for various reasons, _starting_ from licensing rules. So we
| made the decision to strip out the entire JndiLookup class from
| every project that uses java. Clients do various scans, and
| rely on dumb version string matching and /or banner grabbing.
| We have to routinely point them to our _VERY_ detailed and
| explicit log4j response document, and carefully explain to them
| that their scanners are relying on insufficient detection
| methods.
|
| Security teams are quite content with us giving them detailed
| explanation about the false positive. And then they forget or
| deliberately choose to ignore the lesson and the next time they
| run their scans, get the same false positives again.
|
| Even supposedly state-of-the-art security tools, to this day,
| refuse to actually _verify_ their detections.
| plmpsu wrote:
| Configure your build tool to exclude the transitive dependency.
| thayne wrote:
| I thought Log4j 1.1 was vulnerable, just to a lesser extent,
| and there wasn't a lot of information on it because 1.x is end
| of life
| Danidada wrote:
| Someone said "hey log4j 1.2 wasn't that bad" after the
| log4shell vuln and came up with the idea of reload4j which is
| just log4j 1.2 but with its known vulnerabilities, bugs and
| performance issues fixed. Complete feature freeze besides
| that. Considering that a log tool shouldn't have that many
| features and that log4j 1.2 was being used by a lot of
| companies (just see maven central stats) there is no need to
| add more features to reload4j, which I find kind of cool
| bragr wrote:
| Log4j 1.x is vulnerable, but to different things and not as
| badly.
| cesarb wrote:
| > Log4j 1.x is vulnerable, but to different things
|
| And the most common use of log4j 1.x (logging to the
| console or to a file, with a simple configuration) uses
| none of the vulnerable parts.
| hyperman1 wrote:
| See https://logging.apache.org/log4j/1.2/ . Plenty of
| vulnerabilities. Just different ones
| Natsu wrote:
| EOL libraries are themselves vulnerabilities. It's not really
| good to even have this reachable on the classpath, in cases
| like the grandparent, you should look at removing the class
| from inside the JAR (or removing the JAR itself) if possible
| to ensure that the vulnerable code cannot get called under
| any circumstances.
| Nursie wrote:
| Been in similar situations, it is not fun.
|
| "You have a vulnerable version of netty"
|
| "No, we don't, that's a false positive"
|
| "No see, the tool says you need netty 4.x and that's version
| 1.x, you need to update"
|
| "OK, but the tool is wrong, it's just picking up anything with
| 'netty' in the name, and that component is a wrapper around
| netty for some other thing, it only goes up to 1.8"
|
| "You have to update it to 4.x, this is a vulnerability"
|
| "Do you understand this isn't part of the thing you're
| concerned about"
|
| "I am only concerned about shipping a product without
| vulnerabilities"
|
| "Well this isn't one, because it's a false positive, here's the
| CVE, here's what it applies to, here's the link to this package
| on maven central, it's not part of netty, back off"
|
| "But it's insecure, the tool says so"
|
| We went round and round for about 3 hours until I told the guy
| (an infosec 'pro') to leave me alone until he'd figured out how
| to do his job... he came back to me the next morning for
| another 3 hour slack argument along the exact same lines at
| which point I told him to leave me alone permanently and
| communicate through management if he had anything to say. I'm
| not massively impressed with infosec as a profession after a
| few similar encounters.
| Natsu wrote:
| This is where you could probably just have renamed the file.
| dboreham wrote:
| Same experience several times. Tool picks up examples code
| dependencies from our dependencies is another form of this.
| raesene9 wrote:
| Over-reliance on tools is a bad problem in Infosec, often due
| to the range of areas they have to cover. That said a _good_
| infosec person should always have an understanding of where
| their tools are limited, so they can work around those
| deficiencies.
| d4mi3n wrote:
| Security engineer here. This is sadly common. The grim fact
| seems to be that we have a dearth of information security
| analysts with engineering experience. If they don't have
| "Engineer" or similar in their title, odds are they haven't
| had the pleasure of building or maintaining a non-trivial
| unit of software over more than a quarter or so.
|
| That said, I've seen the inverse problem: engineering staff
| that either don't understand how their dependencies are
| managed (not unreasonable for less experienced teams, NPM for
| example has a tendency for huge dependency trees and this can
| be a hard problem to manage for a big project) or whom are
| for a plethora of reasons incentivized to push back on
| requests for work they don't perceive as important.
|
| The middle ground here requires trust between both parties
| (InfoSec and Engineering org functions). Sadly, not all
| security programs are well managed and not all engineering
| teams have mature practices.
|
| If you have time or capacity, you can get a lot of leverage
| here by educating your security analysts on how your
| technology works (e.g. how the build system functions, how
| your languages of choice share and bundle code). Trust can be
| built between engineering and infosec teams by educating
| where possible. Done well such practices can up-level both
| sides of these discussions and save you stress in the long
| run.
| tetha wrote:
| This is for example why we, as a company, rather took 2
| hours to escalate Log4Shell internally. Which was like 2-3
| hours before it went globally holy shit.
|
| We took the time to analyze the vulnerability, and multiple
| competent people concluded: Yup. This is serious. Then we
| figured out plans, mitigations, and communication paths to
| update the plans and mitigations together with ops, dev and
| other folks. And then we shoved all of that as a large
| document into the larger technical organization as "Holy
| fuck the sky is falling, start patching, here's how to see
| when you have to, here's how you do. Go".
|
| And that's also something I'm currently telling our new
| full-time security engineer. Be a bit mindful in the
| communication of criticality and vunlerability. Like, with
| my team of admins and SREs, you can be like "We're all
| doomed" and someone will be "but we don't deploy that
| feature, calm down. Would you like some tea? What evil
| thing did the CVE say to you?". Or they might escalate
| accordingly, if it looks bad to them, too.
|
| But if you do that to most of our dev-teams? That'll be
| just an unstructured mess, which will be worse than waiting
| a moment to structure a proper response to the threat.
| BrandoElFollito wrote:
| I run a security team in a large company.
|
| What you described is usually the result of some "consulting
| company" (in our case big ones) that drop stuff with zero
| actual knowledge.
|
| Every year we "review" these with the board and it is
| annoying as fuck.
|
| Recently they got a new manager who understands we are in the
| same boat. They have to provide a report with some findings
| and I need to have a secure environment. We talked this over
| and suddenly the board presentation was cool.
|
| As for internal teams, I have a few extraordinary people.
| They are very young and I had to coach them on how the
| vulnerability is actually critical or not depending on the
| context, data, etc.
|
| So not all cybersecurity teams suck :)
| d4mi3n wrote:
| Hi! Question from a fellow practitioner: What tools or
| processes do you and your team use to identify false
| positives in such scans and reports? How do you track out-
| of-band mitigations against real-but-unreachable
| vulnerabilities that have been identified? e.g. A WAF set
| to intercept requests that trigger a specific vuln, or
| network segmentation to limit the scope of a moderate
| vulnerability.
|
| In my experience tuning out false positives and other noise
| is fairly straightforward when you're only looking at CVEs,
| but things rapidly break down when you're trying to
| prioritize work with limited resources (only so many
| security engineers and engineering teams with spare cycles
| on their roadmap).
|
| I have yet to find a good solution here other than manually
| whitelisting specific vulns in some kind of tracker and
| reviewing them whenever the network topology or
| configuration for an application changes.
|
| Curious to hear your experiences, though I suspect this is
| a common and difficult to solve problem.
| commandlinefan wrote:
| I sometimes feel that my job description could be "arguing
| with people who feel passionately about something they don't
| understand".
| dylan604 wrote:
| Isn't that just a description of life in general?
| commandlinefan wrote:
| Not for the people who are arguing with me.
| BrandoElFollito wrote:
| Like I usually say, everyone is a doctor and a football
| coach, and recently a security expert.
|
| I am coming to a point in my 25+ years career where I
| stopped to argue. I just tell them that they are wrong but
| I do not care until they put the company on danger.
|
| Unfortunately this leaves me with fat powerpoint
| presentations which well to be the modern documentation of
| a company.
| d4mi3n wrote:
| Many advisory jobs fall into this bucket. I'm of the
| impression that anybody who reaches senior/staff+ roles end
| up doing this as part of their day to day.
| harveywi wrote:
| Man: What is the secret to eternal happiness?
|
| Guru: To not argue with fools.
|
| Man: I disagree.
|
| Guru: Yes, you are right.
| mbreese wrote:
| It sounds like some of this should be automated. If all the
| infosec person is doing is running a tool against a repo and
| reporting results, that should be automated. When there are
| false positives, that should be annotated in the repo with
| multiple people checking off on it.
|
| I've been impressed by automated bug checking tools in the
| past and I see this as part of the same issue. I don't see
| why this would need an FTE to run code against a tool. CI
| should be enough.
| dboreham wrote:
| It's the "intelligent life form determines the alert is a
| false positive" part that's not automated.
| mbreese wrote:
| Of course that's not automated.
|
| From the parent, it sounded like the their issue wasn't
| figuring out if something was a false positive. It was
| that another "intelligent life form" ran a tool and then
| wouldn't accept when another "intelligent life form"
| assessed that a flag was a false positive.
|
| If the infosec person is just running a tool and
| reporting results, why are they part of the loop? Make
| running the tool mandatory for each git push back to the
| main repo. Then, if/when there is a false positive, allow
| them to pass if they've already been "approved" by some
| means (like a .infosec_ignore file).
| dspillett wrote:
| _> We had to spend a week back and forth with_
|
| We've had similar out of a pen test that noted we were using
| nginx for some of our internal dashboards/tools and the version
| running (1.18 IIRC) was officially EOLed upstream so a high
| security issue.
|
| They initially wouldn't listen to the argument that we use
| stable/LTS Debian/Ubuntu release only, and those hold
| functionality stable (which is one of the key reasons to use
| them) and backport security updates where relevant. We pointed
| them at the package changelogs and they were convinced that
| there was still a vulnerability not patched but for some reason
| didn't want to tell us which, when we finally got that out of
| them it turned out to be one that was introduced in a later
| version so was never relevant in the first place for the one
| the stable repositories included.
|
| You would think a penetration testing company would have a clue
| about something so common...
| raesene9 wrote:
| If it was an unauthenticated test, I'm not surprised. Raising
| findings off of banners is notoriously error prone, but
| pentesters often err on the side of raising things that are
| dubious rather than leaving them out.
|
| If it was a credentialed review, you'd have _hoped_ their
| scanners would plugin to the appropriate debian /ubuntu
| security database and give you a less false positive prone
| set of findings (although with debian based stuff you still
| have the problem of scanners reporting the "unfixed" set
| sometimes)
| cmckn wrote:
| > we can't reasonably fork the entire dependency just to remove
| that one sub-dependency from our project's dependency
|
| If you use Maven, just exclude that transitive dependency from
| the explicit dependency.
|
| But yeah, dumb warnings are dumb. <dependency>
| <groupId>com.example</groupId> <artifactId>my-
| dependency</artifactId> <exclusions>
| <exclusion> <groupId>org.slf4j</groupId>
| <artifactId>slf4j-log4j12</artifactId> </exclusion>
| </exclusions> </dependency>
| EdwardDiego wrote:
| Bingo.
| sporkland wrote:
| Log4j-api isn't vulnerable and the slf4j-log4j bridge depends
| on it. Log4j-core is. All of our internal security scanners
| failed to make this distinction so we had to post filter
| ourselves.
| lcfcjs wrote:
| BrainVirus wrote:
| If you automatically update your dependencies all the time, you
| will constantly get new bugs, issues and sometimes even malware.
|
| If you don't update your dependencies all the time, you will be
| vulnerable to old bugs and issues.
|
| The current software engineering paradigm _has no meaningful
| answer to this_ , no matter what "security experts" tell you. In
| a sane industry this realization would lead to a change of the
| paradigm, but people in our industry seem to be only doubling
| down.
|
| "Use a build tool to do dependency checks!"
|
| And then what?
|
| "Only update libraries if they have vulnerabilities."
|
| So, should this be a manual process where I have to dig through
| obscure warnings every time I build something?
|
| "No, you can automate it!"
|
| Congrats, you've just outsourced the decision making process
| about which versions of libraries to use to some 3d party. You've
| also made your build process reliant on yet another online
| service.
|
| "But it's not a hard dependency, you can still build if the 3d
| party service is off."
|
| So the exact software you compile will depend on whether or not
| you can connect to a 3d party service? Do you understand the
| actual implications of this?
|
| Etc. People with clever advice don't seem to think it through.
| We're fighting fragile complexity of too many tools ducktaped
| together by ducktaping more tools to the whole setup. Again and
| again and again.
| PKop wrote:
| "If you automatically update your dependencies all the time,
| you will constantly get new bugs, issues and sometimes even
| malware.
|
| If you don't update your dependencies all the time, you will be
| vulnerable to old bugs and issues."
|
| >has no meaningful answer to this...a sane industry this
| realization would lead to a change of the paradigm
|
| This sounds like gene mutation and evolutionary selection
| pressure. It's sort of a fundamental aspect of the universe no?
| There may not really be a meaningful "answer" to this in
| principle just various half-measures to muddle along and adapt
| to this reality, survival of the fittest in a sense.
| Spooky23 wrote:
| Yup, you're outsourcing your decisions in exchange for not
| doing the work. You have to understand your risks and needs.
|
| Your boring business systems that will live for 50 years need
| to have boring system level libraries or stuff you maintain.
| Your fast time to market or non-critical systems are optimized
| by speed and cost.
|
| If your service depends on a bunch of downstream stuff, your
| processes need to support upgrade cycles. If you can't keep up
| with that, you need to refactor.
| rowanG077 wrote:
| System level libraries are a huge risk. It makes updating
| much harder leading to many developers just foregoing updates
| unless absolutely necessary. Libraries your software uses
| should be decoupled as much as possible from system level
| libs.
| _jal wrote:
| There is pressure in our org to find ways to have fewer
| dependencies.
|
| Some of this is trivial, and a very good thing - stop using
| bullshit exercises in fragmentation like isEven.
|
| Some of it is very much not, and will apply pressure to find
| vendors who are willing to vet their own bundles of commonly
| used libs.
|
| There will be numerous second order effects of that,
| including the ghettoization of open source that isn't
| "QualStrike Approved(tm)".
| bick_nyers wrote:
| There is likely never going to be an approach that yields 0
| vulnerabilities, so instead we should be more focused on
| minimizing the risk as much as possible. There is a middle
| ground and nuance between never update and always update, just
| as there is a nuance in how much and what you duck tape
| together.
|
| Not sure if it's the best, but my current approach is to target
| major revisions/versions with ABI/API compatibility as viable
| candidates, and update always, but at a lagged rate, ideally
| allowing enough time to pass for vulnerabilities to be surfaced
| in those versions. Log4j is a tough case due to how far back it
| goes version-wise, but you simply can't win them all.
|
| Another important way to minimize risk is to have a strong
| ability to provide a corrective action. If replacing the
| version or the entire dependency of a library you use takes a
| significant engineering effort, then you are arguably at a
| higher risk due to that than you are in your decision to never
| vs. always update.
| agumonkey wrote:
| And there's the tangling of dependencies. One fix in one lib
| may cause a bug in another that isn't maintained.
| paulmd wrote:
| Or you can end up with conflicting versions of transitive
| dependencies.
|
| Or even worse you may have a legacy "fat jar" that you can't
| even manage the transitive dependencies on - they're compiled
| in, and that's that. Something else has a version conflict?
| Tough shit, one or the other is going to break, and depending
| on how bad your build is, you may not even get an explicit
| choice, it may be thrust upon you by the classloader.
|
| Sometimes you can mitigate it somewhat by encapsulating them
| in their own service and just firewalling the hell out of it,
| and modules should help the dependency conflicts somewhat,
| but the problem is these are basically "UXO in the backyard
| garden", they are a passive business risk and sooner or later
| you may get lucky and they go boom even though you did
| nothing wrong.
|
| Despite your best intentions, sometimes you gotta update
| dependencies, and if you allow it to fester, then you risk
| that the time you have to solve the problem is during an
| urgent security crisis. Generally it is better to update them
| on your own timetable than to have it forced upon you like
| that. And the problems and limitations and workarounds only
| become worse and worse over time if you choose to actively
| ignore it.
| agumonkey wrote:
| it's an interesting issue technically.. if only we didn't
| have to suffer its reality :)
| ziddoap wrote:
| > _no matter what "security experts" tell you_
|
| Why the scare quotes?
|
| > _In a sane industry this realization would lead to a change
| of the paradigm, but people in our industry seem to be only
| doubling down._
|
| Maybe because half the time people with extensive
| training/experience in security start to even say something
| they are completely disregarded (like here), or are hamstrung
| by budget because security is viewed as a garbage disposal that
| profits get shoveled into, or because they are hamstrung by
| pushback over every little change because people seem to think
| the main goal of someone doing security is pissing off users.
| citrin_ru wrote:
| There is no free lunch - each line of code is a liability even
| if it comes form a dependency. You wrote a few lines of code
| which depends on 100k LoC library and now you are exposed to
| bugs and vulnerabilities which may exists in these 100k LoC,
| not only in your few lines.
|
| If you using a library you have to spend time on keeping it up
| to date (which includes testing and fixing whatever when wrong
| during an update).
| autokad wrote:
| Lets not forget that promotions are for those who build new
| stuff and those who maintain old stuff don't get promoted.
|
| To add further insult to injury, colleges send SWE/SDEs to the
| work force to learn to 'actually code', while companies have no
| time to train jr developers so they are just thrown in the fire
| and hope for the best.
| xmprt wrote:
| At most companies I've worked for, mentoring and training
| junior engineers is a requirement for promotion. Also, you
| might not get promoted for maintenance but
|
| 1. keeping it running with a healthy userbase and
| demonstrating high impact does, and
|
| 2. if you know how to spin it, then maintenance work can look
| like "building new stuff".
| kelnos wrote:
| > _At most companies I 've worked for, mentoring and
| training junior engineers is a requirement for promotion._
|
| You've worked for some great orgs, then! My experience is
| mostly the opposite. One company I worked for did value
| mentorship, at least somewhat, and it was a factor in
| promotion decisions, but not mentoring others didn't really
| stop people from getting promoted.
| Nicook wrote:
| I think this is more of a "do what I say, and not what I
| do" thing. Nominally most companies want you to mentor
| junior devs, but the worker who spends less time on that,
| and more time on higher visibility work gets the
| promotion priority.
| shepherdjerred wrote:
| > So the exact software you compile will depend on whether or
| not you can connect to a 3d party service? Do you understand
| the actual implications of this?
|
| This is already true with the vast majority of software using
| any an online software repository, e.g. Go, NodeJS, Java, Rust,
| etc.
| philosopher1234 wrote:
| Not Go
| mrcarruthers wrote:
| Still Go. I mean there's no central repository in the way
| there's one for npm, and yes you can point it to any git
| repo as the source for your dependencies, but the reality
| is most are on GitHub. So your central repository is
| GitHub.
| laughingbovine wrote:
| Yeah but you can cache the packages or store them in some
| fashion. I've seen JAR files in git repos far too many
| times...
|
| With security, you need some online service so you can find
| the new CVEs every day.
| kelnos wrote:
| > _With security, you need some online service so you can
| find the new CVEs every day._
|
| Or you just do it manually if that service is down. It's
| weird to me that the argument for not having an automated,
| 3rd-party service is "if it goes down then you'll have to
| do things manually", when the alternative is "you always
| have to do it manually".
|
| If you are comfortable trusting a third-party service to
| tell you when to upgrade, then that is absolutely an
| improvement over doing security updates manually. This is
| why I have unattended-upgrades set up on my Debian systems
| to automatically install updates from Debian Security every
| day. Sure, it may fail for whatever reason, but I am
| certainly not going to take the time to (or even remember
| to) update every day.
| staticassertion wrote:
| Yeah, honestly there's probably just a few libraries
| you're going to have to care about re: keeping up to
| date. Everything else can get updated opportunistically/
| on some cadence. Your exposed attack surface for most
| software is pretty much your TLS library and network
| stack. The more mature you become the more of the attack
| surface you can try to track.
|
| But basically if you just subscribe to a few projects'
| releases you can pretty easily get things pushed to you
| when it matters.
| _dain_ wrote:
| >The current software engineering paradigm has no meaningful
| answer to this, no matter what "security experts" tell you. In
| a sane industry this realization would lead to a change of the
| paradigm, but people in our industry seem to be only doubling
| down.
|
| Capability-based security is one possible answer. The Austral
| language is trying to make this a first-class language feature:
|
| >The problem is that code is overwhelmingly permissionless. Or,
| rather: all code has uniform root permissions. The size of
| today's software ecosystems has introduced a new category of
| security vulnerability: the supply chain attack. An attacker
| adds malware to an innocent library used transitively by
| millions. It is downloaded and run, with the user's
| permissions, on the computers of hundreds of thousands of
| programmers, and afterwards, on application servers.
|
| >The solution is capability-based security. Code should be
| permissioned. To access the console, or the filesystem, or the
| network, libraries should require the capability to do so. Then
| it is evident, from function signatures, what each library is
| able to do, and what level of auditing is required.
|
| https://austral.github.io/spec/rationale-capabilities
| mgsouth wrote:
| There are a few issues here...
|
| 1. Depending upon context, *any* change to state (network
| I/O, console I/O, file I/O, environment variables, etc.) is a
| security vulnerability you can drive a truck through. For
| example: Foo is a very-locked down package that only appends
| text to a file the caller owns. It's formally verified to
| only change the file specified, verified to only append in
| all possible situtations, verified to not have cross-file-
| system identity problems, not subject to file renaming race
| conditions, yada yada yada. Safe? No. "foo('evil_command'
| '~/.bash.rc')".
|
| 2. A useful definition of a "capability" depends upon
| context. A *dynamic* context. Say Foo is intended to be used
| for generating log files. There's no way for a third-party
| library to determine whether the target is actually such a
| thing. It requires manual tuning which fits the operational
| context. Perhaps the file must be named "/var/log/${x}.log".
| Or maybe its "${home}/local/log/${x}.log". Great. Of course,
| now you've got to verify the context-specific defintions. And
| verify the tool that does this verification. And the verify
| the configuration of the tool that does the verification...
| Safe now? No. Context is *dynamic*. "foo('var/log/evil-hack-
| attempts.log' '0.0.0.0/0 tried-to-crash-us')", and your other
| context, the automated blacklisting, locks out the Internet.
|
| 3. This gets very complex very quickly. IMHO, configuring
| Selinux, in a real production environment with real
| personell, is akin to rolling your own cryptography. Can
| people learn to do it? Of course. But the same argument
| applies to writing cryptographic functions. You. Will. Make.
| Huge. Mistakes.
|
| To be clear, this does not mean that capabilities are
| useless. It's obviously critical that they exist. However,
| they ain't magic. You can't "solve security with
| capabilities." There is a point where throwing more of them
| at the problem makes things *worse*.
| shikoba wrote:
| What? Are you crazy? Talking about security to developers!
| How dare you?
| kelnos wrote:
| > _Congrats, you 've just outsourced the decision making
| process about which versions of libraries to use to some 3d
| party._
|
| Aren't we already doing that, to some degree, by watching for
| notifications of vulnerabilities, and then manually patching or
| updating? Sure, with this method, we have the ability to decide
| to ignore a vulnerability, but I'd guess most people aren't
| great at assessing risk (and may not fully understand the
| severity of a vuln, or if their use of the software makes them
| exploitable), and should probably just update whenever a fix
| for a vulnerability comes out. So you might as well automate
| this.
|
| > _So the exact software you compile will depend on whether or
| not you can connect to a 3d party service?_
|
| Anyone who builds against public package registries (Maven
| Central, npm, pypi, crates.io, etc.) already has this problem.
| Some people will go to the effort of putting their own proxy
| cache in front of these registries to insulate themselves from
| downtime, and I expect these sorts of people would do the same
| for a 3rd-party vulnerability updater service.
|
| Regardless, I wouldn't expect this to be a "pull" model,
| wherein you wait until the next time you'd build before getting
| security updates. More likely you would get a notification or
| even an automatically generated pull request (like GitHub's
| dependabot does) when a security update is available. Then it's
| up to you whether you want something else to automatically
| apply those updates, or you can review them yourself and apply
| manually.
|
| I think you're just making a mountain out of a molehill. There
| are several existing options to automate or partially automate
| this, depending on your trust of the relevant third party. And
| you can still do it manually, if you so desire.
| AshamedCaptain wrote:
| I still think that betteer ABI/API stability & control _is_ the
| ideally better solution, but even if I'm a fan of plain old C /
| sonames dependency management, I'll readily admit that very few
| people actually do any type of API promises these days, much
| less API.
| brundolf wrote:
| ABI/API stability != behavioral stability, though. I can give
| you a dependency with exactly the same interface, that will
| still break your software. It's not a solution (even if it
| helps with one piece of the puzzle)
|
| The only complete solution would be static analysis (read:
| type systems) that can guarantee everything about some code's
| contract/behavior relative to the caller. Short of that, it's
| just going to continue being a bunch of manual half-solutions
| like API checks, existing type-level checks, manual changelog
| reading, manual upgrades, testing, etc.
| feffe wrote:
| I think a mindset that could work is to accept software as
| done. This is hard but for foundational stuff it could work
| well. Only fix bugs in such software. Try to set a scope
| what the software should do, build it, then do maintenance
| on it. Supersede it entirely to add new functionality.
|
| That's my armchair take on it.
| seoaeu wrote:
| Adding features to existing software is _much_ cheaper
| than writing entirely new software (not to mention the
| cost to switch!) At the same time, new features in low
| level software can unlock substantial value. Say a new
| API that speeds up your app by 1% or cuts down on further
| development time by a small fraction.
| staticassertion wrote:
| Just patch on a cadence and assume you've got some
| vulnerabilities lying around. When there's something serious
| like log4j, pull the lever to update asap. I think you're
| overcomplicating this.
|
| It'd be cool if devs didn't suck ass at writing code but that's
| the world we live in.
| ClumsyPilot wrote:
| because our industry is obsessed with abstractions - the moto
| is to never fix anything, just pave over it with another
| abstraction.
|
| for example we didnt fix how we build applocations and manage
| dependancies, we invented docker.
|
| we didn't create a secure runtime for the cloud where
| applications can run, instead we virtualised whole operating
| systems
| ActorNightly wrote:
| You are correct, but its not just abstractions, its
| specifically abstractions on a shitty platform.
|
| The whole log4j thing happened is because java is very poorly
| designed from the get go, and one of those points is the
| insistence of making everything an object, including data.
| And of course, you often need to output the string
| representation of data.
|
| From a theory perspective, if you want to rely on language
| features like strong typing to ensure correctness, then you
| have to take it all the way to full provability, where you
| explicitly guarantee bounded input, bounded output, and any
| side effect (where a network call is a side effect, and if
| your code doesn't explicitly state that this side effect will
| happen, it won't run).
| leeoniya wrote:
| > because our industry is obsessed with abstractions
|
| "we do not break userland, period" -- Linus Torvalds
|
| not breaking existing stuff is why many abstractions exist,
| and even why many are put in place early on, to allow for
| some wiggle room without having to break all the things.
| people tend to prefer software that continues to work, above
| all else.
| ClumsyPilot wrote:
| but then someone builds a bank on top of this abstraction,
| and sorts of shit hits the fan.
| pixl97 wrote:
| Working in the industry I do, there are a number of banks
| I will never used based on the terrifying lack of quality
| in the software they write. The SBOM on these projects
| could be printed out and bound as a dictionary there are
| so many packages included.
| leeoniya wrote:
| except, it's turtles all the way down. is POSIX not an
| abstraction? TCP? C/gcc/llvm? glsl?
| kelnos wrote:
| I think we should acknowledge, though, that not all
| abstractions are created equal. Abstractions that hide
| implementation details and allow people to change those
| implementation details without creating churn and extra
| work for their downstream consumers are good.
|
| Abstractions that paper over problems that you are
| unwilling or undisciplined enough to tackle properly are
| usually bad. I think the grandparent was talking about this
| kind.
| leeoniya wrote:
| > Abstractions that paper over problems that you are
| unwilling or undisciplined enough to tackle properly are
| usually bad
|
| some abstractions carry decades of legacy (e.g. linux
| file-system hierarchy), and cannot simply be replaced.
| things that did not connect to the non-existent internet
| did not have to paper over problems that simply could not
| exist at that time (VMs, multi-user, multi-tenant
| hardware). there is plenty of deep software dependencies
| that are no longer maintained which still rely on these
| abstractions and cannot be cheaply/reasonably replaced in
| the field. sure, we have new abstractions since then, but
| only new and actively maintained software can ever make
| use of them, and that will only end up in systems that
| are also actively maintained instead of simply continuing
| to work with exactly zero effort.
| SomeCallMeTim wrote:
| >"Use a build tool to do dependency checks!" > >And then what?
| > >"Only update libraries if they have vulnerabilities." > >So,
| should this be a manual process where I have to dig through
| obscure warnings every time I build something? > >"No, you can
| automate it!" > > Congrats, you've just outsourced the decision
| making process about which versions of libraries to use to some
| 3d party. You've also made your build process reliant on yet
| another online service.
|
| Sorry, this is a total strawman. Automation _doesn 't_ need to
| be integrated with the build in such a way that it creates
| noise, and it doesn't mean you need to let changes be applied
| without human intervention.
|
| I'm hooked up with Snyk which scans all of my dependencies and
| sends me a summary email if it discovers any vulnerabilities.
| It creates PRs that can patch dependency vulnerabilities, and I
| can apply them if I approve of the change. If I don't think the
| change is worth it or the vulnerability is relevant, I can go
| to the Snyk site and mark a vulnerability to be ignored, or
| even tell it to ignore it for a month or whatever. I'm sure
| Snyk isn't unique, either.
|
| Yes, a human should be involved in decisions. No, it doesn't
| need to be noisy.
|
| I'm on Node.js and working on multiple projects with crazy
| numbers of dependencies, and the number of "new
| vulnerabilities" I see is typically a few per month at most.
| Some months there are no new notifications at all. It simply
| doesn't change that quickly, and it's not constant noise unless
| you _ignore_ the notifications, and that 's just bad software
| engineering.
| taeric wrote:
| I don't see how you have really fixed the break much.
| Especially if you can "push back" and say no to an update,
| you are just setting yourself up for when the update is even
| harder to accept. (That is, the further behind on an update
| you are, the harder to take it. Typically.)
|
| And then there is the "I want the fix, but I don't want the
| added attack vector of features that comes with it."
| kelnos wrote:
| Sure, but you're in control, and can decide to do what's
| best for you and your organization. Taking on the technical
| debt of allowing your dependencies to get further and
| further behind isn't a decision I usually make, but it can
| be a valid one depending on the individual circumstances.
| taeric wrote:
| I mean, you aren't wrong. But this is akin to saying you
| don't need a memory safe language, because you can decide
| what is best for your applications memory needs.
|
| That is, I think the ask is more of "why haven't we come
| up with something that is a bit better at mitigating
| risks, here?"
| bastardoperator wrote:
| This is basically what Dependabot on GitHub does too. If it
| finds a vulnerability it creates a PR that you can decide to
| apply. It's automated to the extent that it's easy but it's
| not making or forcing changes on you. A human is still in
| control.
| kodah wrote:
| I think people's experiences will vary with tools like this
| due to company policy. For instance, your company allows you
| to make decisions about whether a vulnerability pertains to
| you, but many don't. I think that's how these tools become
| "noise", because the only option an engineer is given by
| leadership is to comply to the tool.
|
| At this point, I think a lot of problems in software are
| actually misaligned or misincentivized policy gone awry. When
| you read about them on forums they look technical, but
| they're not.
| AndyMcConachie wrote:
| The ease with which software remains mutable after deployment
| is both a blessing and a curse. This is particular and
| intrinsic to software and other digital artifacts. Things based
| in matter do not benefit from this blessing or suffer from this
| curse.
| hinkley wrote:
| When I worked on a code signing app, which is arguably some of
| the highest stakes of almost anything I've worked on, we came
| around to an agreement that one ticket a month would be
| assigned to upgrade libraries, and we rotated that
| responsibility. We didn't stipulate what library, we didn't
| even stipulate which application in the suite (though it was
| assumed that you were likely to chose your primary application
| as the target), so long as something got upgraded.
|
| This policy evolved, if memory serves, after a security issue
| was discovered with the XML library we were using. The fix was
| not back ported to our version, and the versions in between had
| multiple breaking changes, so it was a slog to fix it.
|
| It wasn't even that we were having a production issue, because
| we were still in development and internal testing. Not a single
| 'real' signature had been generated yet. It was all test assets
| and test CA certs. But we had enough wisdom to put 2 and 2
| together and get 4, so we tried to treat the situation as a
| dress rehearsal for some later bug that happened after we
| started playing for keeps. Almost everyone could see this was
| an untenable situation.
|
| There were consequences of course, but we made a bit of
| lemonade in the process. Our integration tests were expanded to
| include a larger percentage of pinning tests for our
| dependencies. Given the gravity of the situation, we needed
| those anyway. We just now had a poster child for doing the
| work.
| agumonkey wrote:
| > we rotated that responsibility
|
| how did that go ? I often wanted to do it but never got to
| submit the idea.
| hinkley wrote:
| A couple didn't get it, a few chose things we might not
| have picked, but that was fine because sooner or later
| having 'dumb' out of date dependencies adds up to real
| problems.
|
| It also took some reminders from the leads to add it in
| instead of letting it slip, but so far it's the least
| broken process I've gotten to use.
| wyager wrote:
| > The current software engineering paradigm has no meaningful
| answer to this
|
| The answer is to stop using garbage software written in garbage
| languages. Of course the current environment is too high-time-
| preference for this to be economical in many industries.
|
| We have lightweight-to-heavyweight formal systems which can
| almost completely or literally completely eliminate issues of
| the type seen with log4j, but due to the marginally higher up-
| front cost of using such systems, only firms with some
| combination of high stakes, low time preference, and foresight
| end up using them.
| ClumsyPilot wrote:
| 'software written in garbage languages'
|
| well, back to the mainframe it is then
| gnulinux wrote:
| Quite the opposite, we have very safe languages now in
| 2022. Any software that's not written in Rust or Haskell
| has to explain the trade-off of being written in an unsafe
| language. Of course the industry will still use unsafe
| languages, but we lack the awareness that we're
| intentionally writing garbage software in garbage languages
| that should be treated as radioactive material in any
| context where safety matters. If you pick a language
| without understanding this point, you're part of the
| problem. If you pick an unsafe language understanding that
| X matters more than safety, then you saw all of this coming
| and you made an educated gamble.
| stonemetal12 wrote:
| These Log4j issues aren't real hardcore exploits. They
| wrote a logger that could run arbitrary code, then were
| surprised when attackers figured out how to run arbitrary
| code.
|
| It isn't a language problem, it is a developer practices
| problem. Developers code like security isn't a thing then
| are confused when their programs aren't secure.
| kelnos wrote:
| > _It isn 't a language problem, it is a developer
| practices problem. Developers code like security isn't a
| thing then are confused when their programs aren't
| secure._
|
| I'd prefer that the "fix" for this isn't "require that
| millions of developers around the world all change their
| behavior, and never screw up in the future".
|
| Languages can still protect against these sorts of bugs.
| Effect systems can describe as a part of the type system
| or function signature what kinds of things a function can
| do. For example, you could have a logging function where
| you know that the only side-effect is that it can write
| to stdout or stderr, or a file. Running arbitrary
| commands would not be allowed, and would cause the
| program to fail to compile.
|
| As a real-world example, I've used the Slick SQL library
| in Scala, and the query type has a type parameter for
| whether the query is an read or a write. It's easy to
| build a simple abstraction on top of your data layer to
| never allow a write to go through, for example, if that's
| something that's useful to prevent bugs. I think one
| thing I did with that was set things up so callers could
| not accidentally write to MySQL replicas; only reads
| could go to them, and writes had to go to the primaries.
|
| Granted, it takes discipline to build and use such a
| system, especially one that has such fine-grained
| effects. (I would expect most are more like a binary
| "does not do I/O" or "does I/O".)
| alcover wrote:
| It isn't a language problem
|
| Exactly. This flamy sub-thread has no point here.
| ClumsyPilot wrote:
| We've had safe lnaguages for decades - for example Ada
|
| https://www.adacore.com/about-ada
| paulmd wrote:
| In what fashion does Rust prevent you from calling valid
| library code that has unexpected behavior?
| throwaway23234 wrote:
| What counts as a garbage language? Java guys will tell you
| it's rust or go, go guys will tell you it's java or rust,
| rust guys will tell you it's java and go.
| wyager wrote:
| I'm not any of those guys, and of those 3 rust is by far
| the least garbage. Garbagicity is a spectrum.
| shikoba wrote:
| I was sure reading your first post that you will propose
| rust. Rust users never cease to amaze me.
| sebzim4500 wrote:
| To be fair there was also a significant chance he would
| think Rust was garbage and only assembly (or if you are
| lucky, C) are acceptable languages.
| EdwardDiego wrote:
| Java person here. Yeah, Go looks a bit crufty, but they're
| gradually undoing the earlier opinionated choices that make
| me think that (Who needs generics? Who needs to tune the
| GC? Oh, our users do), but nah, we mainly hate on Scala
| when we have to maintain anything with Scala deps, just
| that damn ABI that's never compatible.
| recursive wrote:
| > What counts as a garbage language?
|
| I assume it's one with GC.
| joebob42 wrote:
| Wouldn't it be one without gc? Gc collects the garbage
| and disposes of it, whereas in other languages it just
| piles up :D
| Kalium wrote:
| > The current software engineering paradigm has no meaningful
| answer to this, no matter what "security experts" tell you. In
| a sane industry this realization would lead to a change of the
| paradigm, but people in our industry seem to be only doubling
| down.
|
| On the contrary, I believe we do. The answer is _ongoing
| maintenance_.
|
| The basic problem is that people persist in looking at software
| as a fundamentally mechanistic artifact you finish and ship in
| a static environment. Then you move on to the next thing. This
| is fundamentally incorrect. Software engineering is a process
| that takes place in an adversarial, human-driven environment.
| There is fundamentally no automating this away because human
| decision-making work is required.
|
| Any security expert worth their salt will tell you this. As
| long as you persist in trying to view software as a fixed
| artifact in a static environment, you will find there is no
| meaningful answer. Those who come to terms with the dynamic,
| human-driven reality will find that there's a well-understood -
| if inconvenient and expensive - answer.
| secabeen wrote:
| So then what do you do with software that doesn't meet this
| standard? Software that works, that meets a need you have,
| but that is a fixed artifact, or at least will be in the
| foreseeable future. Do you refuse to use said software, and
| instead choose a less useful alternative that is actively
| developed?
| efsavage wrote:
| I believe the point is that _no_ software is a fixed
| artifact. Regardless of the state of its development, it 's
| all reactive to factors outside of its control, whether
| that's other software, users, hardware, etc.
| Veserv wrote:
| That is solved with standard engineering practices.
|
| If you need to make guarantees about your product that are
| dependent on another piece of software, then you just have
| your vendor produce a spec and guarantee conformance to the
| spec, then you audit that the spec fulfills your fixed
| needs and audit that the implementation conforms to the
| spec.
|
| Then, if you made a mistake, your liability is limited to
| your mistakes instead of your mistakes and the mistakes of
| your dependencies.
|
| If, on the other hand, you do not get any guarantees from
| your dependencies, and you fail to achieve your guarantees
| then that is entirely on you as you are the one
| transforming the absence of guarantees into guarantees.
| m_mueller wrote:
| In fact yes. I'd choose a maintained lesser library or an
| in-house implementation over an unmaintained one. This is
| also the main benefit of python, with its huge standard lib
| that effectively gives some maintenance guarantees.
|
| To give you an example: XLSX parsing. For quite a while,
| the most performant library ,xlrd' had a big scary
| unmaintained notice on GitHub, yet people kept recommending
| and using it, often unnoticed through pandas, because they
| don't read docs. I'd refuse to touch that and instead use
| openpyxl, and now newer xlrd releases have even completely
| removed xlsx support.
| Kalium wrote:
| That's a risk management problem. For how long will the
| world around this fixed artifact remain constant? How far
| away is your "foreseeable future" horizon? What happens
| when you're wrong? I have, in fact, chosen to use actively
| developed and maintained systems with fewer features than
| unmaintained EOL'd ones.
|
| When you choose software to solve a problem, you're not
| just choosing software. You're encoding expectations around
| the problem, the environment, and your needs. If all three
| of those can be assured to never change then choosing a
| fixed artifact might be the way forward. Should any of
| those ever change, then your software will either adapt or
| become a less suitable solution.
|
| How often can an organization really be that certain?
| staticassertion wrote:
| That's exactly right - it's about risk. No one has
| "perfect" operations/ security. They have a threat model
| and a risk tolerance.
| SamuelAdams wrote:
| We see this all the time in healthcare. MRI imaging machine
| that cost 500k 20 years ago still works fine but only on
| Windows XP. Cost to rewrite the software for windows 10 is
| more than 500k so rather than paying for that instead they
| firewall the shit out of it so it can only talk to some
| printing / output device.
|
| That's just one example. Medical machines are expensive as
| hell and companies don't want to rebuy or constantly invest
| in something that works and does one thing well.
| bradgessler wrote:
| I was visiting my parents this summer and they have a
| Gameboy from 1991 with a Tetris cartridge, which was
| authored in 1989. They use it regularly and it still
| works! For over 30 years this device has played Tetris
| without requiring any software updates.
| jsiaajdsdaa wrote:
| Better idea: stop logging.
| ClumsyPilot wrote:
| problem: food contaminated
|
| solution: stop eating
| jsiaajdsdaa wrote:
| solution: find uncontaminated food, or make it yourself
| ClumsyPilot wrote:
| option 1 - we and farmers, as professionals are
| responsible for supplying stuff that is not dangerous and
| works as it should
|
| Option 2 - which you are advocating - it's every man for
| himself and we decent into socity of warring tribes of
| hunter hatherer or subsitence farming at best
| jsiaajdsdaa wrote:
| option 2 is the only possible state of reality, because
| it is never possible to guarantee that everything work in
| every instance
|
| let the buyer be aware
| paulmd wrote:
| I mean, how do you know your webserver doesn't have a
| bug? How do you know your OS doesn't? Just write those
| yourself? How do you know yours doesn't? One must still
| verify the key functionality regardless.
|
| But, you're not wrong, some people reach for every
| library and framework they can, and others focus on
| building the most minimal and understandable thing that
| will work. I'm not saying writing your own server in
| assembly, but, avoid unnecessary junk and magic and
| libraries.
|
| It's the classic dilemma: in this world of ours, one can
| either return to monke or progress to crab.
| throwaway23234 wrote:
| the issue is that this can pop up in any library, not just
| logging. It's about keeping your deps up to date (or not) by
| a "3d party" (assuming he mean 3rd party)
| staticassertion wrote:
| Yes, this can pop up in any library. But only because
| developers aren't taught "don't put remote code execution
| into your code". You'd think that would be something that
| someone would teach, but it doesn't really come up.
| Remember that log4j was vulnerable because of a _feature_ -
| it all worked as designed.
| craigching wrote:
| Ironically, logging is one of the ways to help
| mitigate/detect security vulnerabilities.
| MilStdJunkie wrote:
| It's not just logging. L4J is so extensible that people have
| used it for all kinds of things, way waaaaaayyyyy away from
| just logs. So disabling logs won't necessarily cut it.
|
| I am no expert. I ended up indexing all the open source kruft
| I use to hold this ship of fools together, then verified that
| the Log4J pieces were definitely disabled with a bunch of
| monitoring while I tossed stuff at it. I did mention I am not
| an expert, right? I am sure there is a more Pro way to do
| this.
| jsiaajdsdaa wrote:
| F around and find out basically. If you choose to build on
| the house of cards, thou repeath whaet thou soweth or
| something.
| Genbox wrote:
| I was wondering when it would become the norm to call it
| "ducktape". It has been 3 hours and nobody has corrected it.
| Last week I found a 3-pack of "ducktape" in the builders
| market.
|
| I'm not making fun of you or trying to be funny - it is simply
| an observation that I find interesting.
| na85 wrote:
| Where I live there's a brand of duct tape called Duck Tape.
| It's terrible compared to the 3M brand tape, but it's cheap
| so it's popular.
|
| Also nobody that knows what they're doing uses "duct tape" on
| ducts, so the name is pretty meaningless.
| hnov wrote:
| Big-tech has solved this internally. Everything down to the
| linux kernel they're running in prod is versioned and has
| people maintaining it and interacting with upstream (if any).
| This is obviously a full time job for at least a person per
| dependency on average.
| uticus wrote:
| > We're fighting fragile complexity of too many tools ducktaped
| together by ducktaping more tools to the whole setup. Again and
| again and again.
|
| You've brought up three relevant points: 1. The tools are
| _software_ , which strength and weakness is being easy to
| change. 2. The tools rely on abstraction, which strength and
| weakness is summarizing/hiding complexity. 3. Communication
| between publishers/consumers of these tools is hard.
|
| Armchair-coaching I now come up with at least three solutions:
| 1. Re the weakness of #3: ensure when an issue is found at one
| layer, communication happens to maintainers of all other layers
| up and down. 2. Re the weakness of #2: collapse the layers -
| analyze dependencies and don't go past more than x number of
| abstractions. 3. Re the weakness of #1: don't use software.
| voxl wrote:
| And what would you have anyone do?
|
| You could formally spec software and implement it, congrats
| you're now 1000x slower to market and any change you want to
| make is another 1000x investment to spec out and prove.
|
| And let's not even mention the ridiculous idea that open source
| developers be expected to do this. Moreover, this doesn't even
| handle hardware problems!
|
| Okay, but we could just build all the software internally,
| reinvent all the wheels. But how is this any better? Now you
| have even less manpower to fix bugs and you're probably 100x
| slower to market because you're building all these half baked,
| bug ridden libraries.
|
| If anything, the best idea is formally verified sandboxing, so
| that you have strong assurances about certain apps not doing
| bad stuff, but this doesn't solve all problems either.
|
| It's an unsolvable problem in general, which is why no one has
| solved it.
| pclmulqdq wrote:
| It takes a lot longer to build houses that are up to building
| codes, and bridges that have 10x safety factors rather than
| 2x. There's a reason we have building codes and large safety
| factors. Software engineers are discovering this today.
| fisf wrote:
| Software engineers are mostly aware of this. There is just
| little market demand for this level of robustness.
| ruined wrote:
| there was no market demand for building codes, either.
| that's why they're laws
| sebzim4500 wrote:
| Countries that pass laws making it 10x more difficult to
| develop software will not survive the next few decades.
| ClumsyPilot wrote:
| bbut but mah free market gave us fire escapes attached to
| every building in new york, and they were made of wood!
| twunde wrote:
| I know you're being sarcastic, but this reminded me of
| Tanya Reilly's _great_ talk "The history of fire escapes:
| How to fail". There's a recording up at
| https://www.infoq.com/presentations/history-fire-escapes-
| res...
| TheCapn wrote:
| Engineering Code of Ethics dictate that safety of public
| and environment is paramount. I know that's true for my
| P.Eng governing body; it is written into our bylaws. I'm
| sure its the same for most others as well.
|
| I think until Software Engineering catches up in this
| regard with traditional Engineering disciplines we'll
| continue to see issues arise from the software realm. As
| much as the self-titled "Engineers" proclaim it, there's
| more to engineering than design and production.
| kelnos wrote:
| I've always been uncomfortable calling myself a "software
| engineer" for this reason, and usually refer to myself as
| a "software developer". Most of the job listings I see at
| various companies use "engineer" though, and so my
| official title ends up being "engineer", so that's what I
| end up putting on my resume. (Though I guess I don't
| really need to do that; I could put whatever I want.)
| apeace wrote:
| I have a very simple solution, in three parts:
|
| 1) We use very few dependencies. We err on the side of writing
| our own function, library, or UI component rather than
| installing a new dependency, even if a "better" open-source
| version might theoretically be available.
|
| 2) We use a monorepo. We have two Go backends sharing the same
| go.mod, and three Typescript/React frontends sharing the same
| package.json, all in the same repo. An upgrade for one is an
| upgrade for all.
|
| 3) Every first Friday, I update all dependencies to the latest
| version. Including major versions. Yes this is a pain sometimes
| (like when I realized we were on Webpack 4 and the latest was
| Webpack 5), but if you do it consistently, it's usually not
| that much trouble. 1-2 hours per month on average.
|
| In terms of security, I think this is a good middle ground. The
| approach has other benefits than security, too.
| [deleted]
| pixl97 wrote:
| >We use very few dependencies.
|
| While this may solve the "exploit from 3rd party
| dependencies" this says nothing about the security quality of
| your functions. Now instead of the vuln being found in the
| 3rd party package and fixed, the issue remains in your
| program forever, probably getting exploited by nation state
| level actors without your knowledge.
| citrin_ru wrote:
| A benefit of writing own functions - you'll not implement
| something you don't need, which reduces attack surface.
| pixl97 wrote:
| Until your developer decides to write their own
| encryption routines.
| Griffinsauce wrote:
| > 1) We use very few dependencies. We err on the side of
| writing our own function, library, or UI component rather
| than installing a new dependency, even if a "better" open-
| source version might theoretically be available.
|
| This just means you likely have a lower quality version with
| the same problems but without the benefit of an ecosystem to
| find them.
| alfalfasprout wrote:
| We actually are running into this exact problem with running
| production ML workloads and despite anyone's claims it's not a
| solved problem. In fact, this log4j mess is a cake walk in
| comparison.
|
| The issue: ML workloads are very sensitive to the exact version
| of a framework/library used and these frameworks/libraries are
| _not_ stable despite what their semantic version says.
|
| So what do you do when you have a python vulnerability? Best
| case, an update "just works" but in many cases you either need
| to retrain the model (can be extremely expensive or
| inconvenient), rewrite it if APIs changed, or hunt down a
| difference in model output if some downstream dependency broke
| something. Stability of software in the python world is
| abysmal.
| aclatuts wrote:
| I find Python is very prone to a lot of breaking changes even
| with minor version changes. And not just that, even minor
| versions have to match up specifically with another libraries
| minor version. It seems to be a community or ecosystem
| problem with Python, which is why I avoid Python as much as
| possible.
|
| In every other ecosystem when minor version are updated there
| are very rarely issues, and there are little issues with
| compatibility with minor versions of libraries.
| skrtskrt wrote:
| eh, people love to break stuff on minor version changes of
| Go libraries to
| wizofaus wrote:
| At least you find out at compile time...
| btilly wrote:
| My (somewhat dated) experience says that Ruby is all the
| way worse than Python on this.
|
| In part due to a culture that encouraged monkeypatching.
| With the result that what behavior you get depends on who
| last monkeypatched it. So a dependency 3 deep that loads
| the module you expected to load last suddenly causes the
| monkeypatch you were depending on to disappear...
| treeman79 wrote:
| Monkey patching was the greatest thing ever. Until the
| community realized how bad it was. Don't see it much
| these days
| xmprt wrote:
| Is this an issue of code stability or do these libraries have
| leaky abstractions that cause issues with models trained on
| previous version, because the library implementation changed
| and since ML has low explainability, it's hard to tell how
| the change impacts the model performance? eg. making a
| library run faster or use a slightly different parameter
| internally causes some floating point imprecision that
| cascades into wildly different results?
| alfalfasprout wrote:
| Unfortunately it's all of the above. Sometimes the code
| isn't stable (eg; a library method changed), sometimes the
| behavior has changed (eg; same library method but now it
| has a different behavior), and sometimes it has some other
| unwanted effect (the computation produces a totally
| different result, it runs way slower, or something else).
|
| With a service it's all much easier because you can have
| pretty robust integration tests and you're good to go. With
| an ML model it's a lot trickier because even if you have
| sample payloads and responses there might be other payloads
| you didn't think about that could cause issues. Or it might
| rear its ugly head when you finally retrain the model and
| suddenly model performance is crap.
| dotnet00 wrote:
| There's also the issue that subtle bugs are a lot harder to
| find in ML since often the model just adapts (although
| potentially with worse performance). This then breaks
| models when the framework updates.
|
| One recent example that comes to mind is that PyTorch now
| disables Ampere GPUs' TF32 units by default since there
| were some hard to find edge cases where the matrix multiply
| results were completely wrong compared to the expected
| result. This went mostly unnoticed except in a few cases
| where users were trying something a bit more sensitive to
| such issues.
|
| I had a similar issue in one of my own models where there
| was an error in a normalization layer's implementation in
| Tensorflow (it was computing the average over the wrong
| axis iirc). It only became noticeable to me when I came
| back to retrain the model a few months later with an
| updated Tensorflow version and found that the results from
| training were not even comparable.
| einpoklum wrote:
| > So what do you do when you have a python vulnerability?
|
| If you are willing to run other people's arbitrary (Python)
| code, then - vulnerability or no, you should consider the
| system running this code as compromised.
| mrbonner wrote:
| At this point I'm asking if it is ok to go nuclear: just slab
| aspectJ compiler as the last step and do comptime patch the log4j
| remote logging API. Have anyone thought of this? Btw who would
| use remote logging to begin with?
| paulmd wrote:
| > Btw who would use remote logging to begin with?
|
| It's not about remote logging like "we log to somewhere else"
| but rather about including information from remote sources
| (specifically JDNI) in your log messages. If you are ingesting
| logs from a bunch of microservices, it would obviously be
| desirable to know which instance you're in, and just include
| that in the logger pattern. Or if you have multiple containers
| in a single tomcat instance, you might want to have the log
| message include which container it's coming from (especially
| for the console). Multi-container tomcat used to be VERY common
| in pet-style legacy java deployments - it's much slimmer on
| memory, although it does come with the caveat that GC pauses
| affect all containers. But that container/server information
| isn't built into the jar, it's runtime data, and the easiest
| source for getting that information is... connecting to the
| tomcat server via JNDI and pulling instance information.
|
| But oops, they populated the JDNI string values _after_ the
| message was already built, meaning that any log message that
| included a JDNI handle would be executed automatically on its
| own. And JNDI is basically a whole inner-platform, including
| the ability to _classload arbitrary code from remote sources_ ,
| because "lol why not, it's _java enterprise code_ , we're
| building the future here!", so basically any user-controlled
| log message could include a JDNI link to arbitrary code and it
| would be downloaded and run.
|
| (the JDNI being built after the rest of the message was
| populated was _obviously_ the wrong place to do it for exactly
| this reason... that is really the core problem here, _user-
| controlled_ JDNI is absolutely not acceptable given the
| capabilities, but, it 's not like I've never written bad
| code... there but by the grace of god, go I. Although again,
| that one is pretty obviously a problem just conceptually.)
|
| Incidentally though "remote logging" really isn't inherently
| insane as a concept, it's conceptually reasonable that some
| kinds of important messages might want to be logged into Kafka
| or JDBC, and 15 years ago there wasn't the overwhelming
| consensus on "write to disk first, ingest it later" being the
| right way to get it there. If you want to be really really sure
| that certain super-important messages are committed, and you
| don't mind the latency, why not just poke it directly? Then you
| never have to worry about logs getting dropped on disk/etc and
| not making it there. And by running it through log4j you don't
| have to do anything special at an application level, you just
| define certain packages and message levels as being important
| and you don't have to re-invent the wheel on the message-
| building framework/etc. And sure log4j ended up having
| vulnerabilities but your personal implementation may have
| vulnerabilities too - how sure are you about your string
| handling and escaping, could a message spoof what looks like a
| valid log entry? that's a common one. It just turns out that
| JDNI was a vastly bigger attack surface than initially
| realized...
| paulmd wrote:
| Also, btw, just for performance reasons - I wouldn't put a
| JNDI call into a log message anyway - even calling to
| somewhere local, it's not entirely free, and logging is
| highly performance-sensitive. I'm not sure if it's HTTP-based
| or what, maybe it's memory-based if local, but whatever it
| is, doing it thousands of times a second is wasteful.
|
| It's not a "get the container name" call, it's a generic JNDI
| call, those values could change and the output probably can't
| be cached... so just for performance's sake I would get those
| values at application startup and then put them into the
| Mapped Diagnostic Context, which can be accessed via log4j
| pattern formatters. That way you're not calling out for every
| log message.
| bgro wrote:
| When are they going to release Log5j? Or even Log4k to address
| these issues?
| hedora wrote:
| The log4j author already did this about a decade ago. No one
| could understand log4j back then, so the replacement is much,
| much simpler, and API compatible for 99+% of use cases.
|
| Few projects seem to have bothered moving to it, probably
| becuase log4j is such a collosal pain to set up or modify.
| hinkley wrote:
| > No one could understand log4j back then
|
| If you're scoffing at this, I want you to do an experiment.
| Imagine yourself at 26, when you just knew enough to be
| dangerous. Or as the person you're currently mentoring/would
| like to mentor. Now go into your application code, set a
| breakpoint on one of the moderately complex `logger.info()`
| calls, run your application and step into log4j. All the way
| down, until it writes to disk.
|
| If you don't say "what the fuck" out loud at least once,
| you're a soulless monster and should go live in a cave.
|
| If your application has a pretty good architecture, you may
| well discover that the code leading up to the `logger.info()`
| call is substantially less complex than the code below it.
| Just to write some text to the end of a single-writer file.
|
| It chaps my ass that log4js, which claims specifically _not_
| to be a javascript port of log4j, has the same pattern of
| layers and weird indirection (some of which are avoided by
| monkey patching, which is in many respects even worse from a
| legibility standpoint, and absolutely from a discoverability
| standpoint). Much of this is chasing after not implementing
| the same 1 line function 6 times, which is made weirder by
| the fact that you have to write individual unit tests for
| each of the log levels, so you 've already written 95% of the
| duplicated code.
|
| Logging frameworks are a nearly canonical example of DRY as
| an antipattern.
| qw wrote:
| I think Spring Boot defaults to Logback these days.
| bxparks wrote:
| What is that replacement? java.util.logging?
| jffry wrote:
| I believe that replacement is Logback [1] + SLF4J,
| according to the intro section on Wikipedia's log4j article
| [2]
|
| [1] https://logback.qos.ch/reasonsToSwitch.html
|
| [2] https://en.wikipedia.org/wiki/Log4j
___________________________________________________________________
(page generated 2022-07-20 23:01 UTC)