[HN Gopher] Who's at the Helm?
___________________________________________________________________
Who's at the Helm?
Author : dlor
Score : 79 points
Date : 2021-01-05 17:12 UTC (5 hours ago)
(HTM) web link (dlorenc.medium.com)
(TXT) w3m dump (dlorenc.medium.com)
| [deleted]
| uncledave wrote:
| This is an epic mess in the making to be honest and no one takes
| it seriously. It's really not important at all for anyone I have
| spoken to. Most of our stacks are magical fucking unicorns to the
| marketing and decision makers and the majority of the developers
| these days. The level of response I get is akin to "but it's got
| electrolytes" while blindly downloading god knows what into
| production kubernetes clusters. Hell even kubernetes does it. I'm
| waiting for something to waltz in and do some damage. Even worse
| is some of the scanning software which itself ships with CVE
| ridden dependencies. I mentioned this to one of the larger supply
| chain scanning vendors and they didn't even know.
|
| 99% of our tech is like smoking to look cool and being in with
| the crowd. But someone's going to get cancer one day.
| IOT_Apprentice wrote:
| I look upon this and despair. How are average folks supposed to
| trust and use these things? It feels like NPM all over again. I
| keep seeing these kinds of things across modern tech
| stacks/languages and wonder why some brainiac has not looked at
| this in a more general sense of a trusted supply chain and
| audit/update process approach to fix this or come up with some
| transformational approach that at a conceptual level can be
| applied across languages and tech stacks.
| gotem wrote:
| What is NPM?
|
| Nuclear Protocol Magisty?
| jrockway wrote:
| NPM is a popular node.js package manager and package
| repository. https://www.npmjs.com/
| classified wrote:
| Super secret expert tip: The internet can also be used to
| find information.
| jrockway wrote:
| I think that kube-prometheus specifically tries to do too much,
| turning the simple problem of running a few services into its
| own brand-new problem. It attacks the problem of "it's hard to
| write a manifest to run a docker container", which isn't that
| hard, and it replaces it with its own configuration language,
| integrations that don't exist upstream, etc.
|
| The general answer is to avoid helm charts that run more than
| one piece of software. If you want Prometheus, install
| Prometheus. If you want Grafana, install Grafana (and accept
| that you have twice as much work, because you are running twice
| as much software). Helm has a culture of bundling all
| components into one chart, which makes it easy to make a mess.
| You should reject this culture and install the components that
| you need, and accept that it's going to be extra effort --
| running other people's software is hard!
|
| (My full-time job is running our company's open source project
| in the cloud for people that can't get it installed themselves.
| It is a multi-person full-time job. It is not yet trivial to
| run other people's software, even if you talk to those people
| all day and have commit access to the repository.)
|
| Here is a slightly off-topic rant. When I first started using
| Kubernetes, I installed the Prometheus Helm chart. It contains
| a config file that searches for objects in Kubernetes that have
| a "prometheus.io/scrape" annotation to control scraping. I
| didn't know that at the time, and searched in vain for a way to
| scrape multiple containers in the same pod. Everything on the
| Internet said that was impossible, and indeed... it is
| impossible with the random configuration that the Helm chart
| supplied. I wrote my own Prometheus configuration and got
| exactly what I wanted. The abstraction layer that showed up
| without any documentation in the Helm chart caused me infinite
| trouble, whereas if I had to write the scrape configurations
| from the start by myself, this would have never been a problem.
| I wouldn't have gotten started in 1 minute, but I would have
| understood the software I was responsible for operating.
|
| It's dangerous to explode random configuration and code into
| your Kubernetes cluster. Helm is a tool that explodes random
| configuration and code into your Kubernetes cluster. Tread
| carefully!
|
| To answer your original question, "How are average folks
| supposed to trust and use these things?", when I first came to
| the Kubernetes world I saw Helm and immediately thought "this
| doesn't seem like a very good idea to me", and haven't used it
| much. I suspect that most reasonable people will come to the
| same conclusion, but the allure of "get this tedious task done
| in 15 minutes" will snare a few more victims along the way.
| raesene9 wrote:
| Interesting post and illustrative of some of the problems with
| supply chain security.
|
| The more layers of abstraction and packaging that are added the
| harder it becomes for end-users to see what's going on.
|
| One of the big problems is stale packages not keeping all the
| layers under them updated. This problem is likely to get worse as
| time goes by, as people move on from projects and things become
| unmaintained.
|
| One option is to fork the packaging at one level and then take on
| maintenance yourself (e.g. creating/maintaining docker images)
| but then ofc you have all the work of doing that yourself.
|
| Another option (in theory) could be someone selling a service of
| maintained, updated, scanned images/helm charts, but I've not
| seen previous efforts in this area be hugely successful...
|
| hopefully the OSSF will come up with some good solutions in this
| space, but it's fundamentally a time consuming thing to fix and
| maintain.
| xorcist wrote:
| > you have all the work of doing that yourself
|
| There is a difference between unnecessary work and necessary
| work. When you take in a dependency, you become dependent on
| it. That is necessary work.
|
| It's great to highlight that dependencies always mean work.
| There's probably some deeper truth to that, at least in what I
| do. But that doesn't mean all work is bad. After all, work is
| what we put in to make things, well, work.
|
| I "maintain" a bunch of fixes (that is, forks) of upstream
| stuff that I use all around the place. I used to maintain a few
| packages just for recreation. I have selinux dependencies,
| ansible playbooks, system accounts, certificates, and a lot of
| other stuff that I need to maintain in order to make stuff
| work.
|
| I guess I wanted to say don't be afraid to maintain your
| dependencies. Sure, it's work, but not unnecessary work.
| ridruejo wrote:
| " Another option (in theory) could be someone selling a service
| of maintained, updated, scanned images/helm charts"
|
| This is what we aim for with Tanzu Application Catalog
| https://tanzu.vmware.com/application-catalog
|
| We still have a lot of work to do but you can do things like
| providing your own base image that you customize and maintain
| to your specs and we build and test the apps on top of it
| raesene9 wrote:
| That's really interesting and I hope it'll be successful,
| it'd be great to see more solutions tackle this area. I guess
| starting with an entperise price point, means you can
| hopefully get a revenue level to help widen out the product
| set.
| jacques_chester wrote:
| For those who aren't aware, Tanzu Application Catalog is a
| descendant of work done by Bitnami to build and curate OSS.
|
| Disclosure: I work for VMware, but not on this.
| knowhy wrote:
| That seems to be an interesting product. However looking at
| the price [0] that is probably not an option for everyone:
|
| > The Team Edition is priced at $150,000 per year. The Org
| Edition is priced at $995,000 per year
|
| [0]: https://docs.bitnami.com/tanzu-application-
| catalog/faq/get-s...
| jacques_chester wrote:
| Here I rush to add that I also don't work on pricing and
| packaging.
| ridruejo wrote:
| Yes this is mostly an offering oriented towards
| enterprises. There is a new offering coming soon at a
| lower price point
| outworlder wrote:
| Maybe I missed it in the article, but a big (maybe bigger?)
| problem is that all packages are directly sourced from upstream.
|
| Tags can be mutated in most (not all) container registries. A tag
| that you have vetted today could change without your knowledge.
|
| Helm chart versions can also be overwritten.
|
| Ideally, everything should be pulled from repositories you
| control.
| jacques_chester wrote:
| I've been writing a book and I devoted several pages to the
| notion that you should always, _always_ , use fully-digested
| container image references.
|
| Which reminds me: never don't do that. Always use fully-
| digested references. Don't fail to use fully-digested
| references.
| microbass wrote:
| Could you please expand, with an example, if possible?
| jacques_chester wrote:
| Manning allows folks a time-limited preview of their books
| (I think it's ten minutes per day?).
|
| The relevant section is here:
| https://livebook.manning.com/book/knative-in-
| action/chapter-...
|
| The gist is that tags are mutable and deletable. Local
| caching means that every node can disagree on what the tag
| resolves to, and so can the registry. What version
| corresponds to which tag at which location at a given time
| is _not deterministic_.
|
| The only way out is to use fully-digested references. The
| digest means that there can be no mutation of what's being
| pointed to. It will mean the same in every configuration
| and on every node, no matter when the reference is resolved
| and no matter when the reference was created.
| orf wrote:
| It also means every layer has to be kept indefinitely,
| forever. Tags at the very least are pointers that stop
| indefinite and exponential storage growth.
| jacques_chester wrote:
| Tags don't help exponential growth, because you cannot
| reliably reason from a tag-only reference back to the
| _exact_ assets it represents. That 's feasible with
| fully-digested references.
| jacques_chester wrote:
| Without complete asset transparency, these issues will only grow
| worse.
|
| Off and on I have pitched folks the idea that we need some
| concept of "universal asset accounting" or a "universal asset
| graph".
|
| A federatable body of data about assets at various levels of
| detail, abstraction and age.
|
| For example: I should be able to have the digest for a file and
| be able to see anywhere it has ever showed up. I should be able
| to zoom in and out from that pivot point: to debian repositories,
| to container layers, to dependency graphs in NPM, to Helm charts.
| I should be able to see where it lives in my own systems, and
| exactly how it came to be there, from the network operations in
| my own systems all the way back to tickets and commits. For any
| given asset I should be able to discover everything that was
| assembled, transformed, transported and inspected in its scope.
| icefrakker wrote:
| I want is not the same thing as I should be able to.
| lukeschlather wrote:
| From this viewpoint, Helm is really kind of broken-by-design.
| It takes a perfectly good system (Kubernetes Manifests) which
| allows you to exactly specify the exact versions of all the
| containers you want on your cluster, and it wraps it in an
| inscrutable "helm install" invocation which can potentially
| pull containers from anywhere at deploy time.
|
| I've used helm template which generates manifests but
| unfortunately this isn't really the common use case, and it
| doesn't give you that much compared to your favorite text
| templating language.
| jacques_chester wrote:
| I think that an accounting system which encodes uncertainty
| ("it could be any one of these two dozen images") will help
| create pressure on the upstream, as well as downstream
| visibility into who is affected.
| 13security wrote:
| The best description of this I've seen is "chain of custody":
| Like with evidence in criminal proceedings. Docker and Git's
| use of content-addressable hashes helps in this regard, but
| there is still considerable work to be done when tracking
| third-party dependencies.
| jacques_chester wrote:
| We used "chain of custody" as our analogy for buildpacks when
| I was working on them.
|
| It's not completely accurate, though, for the same reason as
| "supply chain". There isn't a single linear sequence of
| agents, actions and assets for a given asset. It's a graph in
| which things can appear many times in many permutations.
| mchmarny wrote:
| "But each layer here adds an element of required trust", how
| often we simply glance over that and assume. In the same time,
| building everything from source is also neither reasonable nor
| 100% secure. Glad smart people like Dan are looking into this.
| jeffbee wrote:
| What's not reasonable about building from source? It really
| seems like a small cost to snapshot a version of third-party
| code into your own source control system. There really aren't
| (or shouldn't be) that many different things in your
| environment.
| mchmarny wrote:
| The amount of dependancies, build environment setup steps,
| across all of the components, and sheer complexity of putting
| the entire stack together from scratch, would exclude a
| substantial portion of the target users... hence, while
| possible, is not reasonable, certainly not on angling basis.
| jacques_chester wrote:
| I worked on buildpacks for a while. Those had to track "only"
| a few hundred upstream dependencies and build them from
| source. It took a fulltime team to build and maintain the
| infrastructure.
|
| The problem here is economic. Because there is no body of
| information about software assets acting as public good, each
| individual is forced to internalise the costs. Either you
| internalise the cost of building everything yourself, or you
| internalise the risk of accepting incoming assets on faith.
| But you _bear a cost_.
| knowhy wrote:
| Building from source seems me the only way to get to a
| reasonable state when it comes to security patching.
|
| However these modern stacks contain a lot of small parts. The
| article gives the kiwigrid/k8s-sidecar as an example. Also,
| well shown in the article, it is not as easy as copying some
| commands to your own Dockerfile. Look at the busybox image
| using glibc from another debian image.
|
| It's not reasonable in the sense that it is a hell lot of
| work and would require more time and effort the average
| devops/sre/whatever (team) has.
|
| It would also go against the promise that Kubernetes would
| make things easy because you could just do helm install
| stable/prometheus
| emerongi wrote:
| At the core of this issue is reproducibility. If you can fully
| reproduce the software you use, you no longer need to trust that
| X binary/container was built properly (although you still need to
| trust the source code itself, as well as the source of the source
| code). You can also always rebuild your software with newer
| dependencies (e.g. if they have received security updates).
|
| It is about time we embraced Guix/Nix.
| jacques_chester wrote:
| I don't agree (or rather, I don't think it's the _only_
| option). Reproducibility is necessary when you are unable to
| trust the accuracy and integrity graph leading up to a
| particular asset. But if you have a trusted, exhaustive,
| tamper-resistant history of how an asset came to be, you can
| skip a rebuild altogether.
___________________________________________________________________
(page generated 2021-01-05 23:01 UTC)