[HN Gopher] Documenting our migration to Docker
___________________________________________________________________
Documenting our migration to Docker
Author : djelal
Score : 76 points
Date : 2021-07-30 11:43 UTC (11 hours ago)
(HTM) web link (www.artifakt.com)
(TXT) w3m dump (www.artifakt.com)
| DrBazza wrote:
| Worth reading just to "TIL" of a dockerfile linter:
| https://hadolint.github.io/hadolint/
| willejs wrote:
| +1 for using hadolint! I have been using it for ages, and I
| highly recommend it. I also sometimes write some Goss tests,
| and even run Trivy at the final stages of the container build
| and delete them once done in the same layer. It takes about 10
| seconds or so longer to build the container, but its baked in
| and helps you catch issues before pushing the container.
| rhacker wrote:
| Someone editorialized the title.
| enlyth wrote:
| It seems the article was submitted by the author, so I don't
| see an issue here
| cle wrote:
| > How does Docker handle asynchronous intermittent processes? Not
| good, actually. It is well known in Docker 101 that thou shall
| not run cron in the same container as the main process.
|
| You can run cron just fine in Docker containers. Whether or not
| that's a good idea depends on what the cron is doing, what your
| application is doing, your runtime requirements, etc. Running
| cron in your container is definitely way simpler than the other
| options involving container scheduling.
| codeflo wrote:
| This. The advice to move cron jobs outside has its place, of
| course. But unless you're running at massive scale, Docker
| becomes easier once you stop worrying about what is and isn't
| "proper container architecture", and just remember that it's
| Linux.
| marcosdumay wrote:
| > just remember that it's Linux
|
| Hum... It's some Linux that appears out of nowhere and may go
| away at any time. This is the reason for most those container
| best practices.
|
| Anyway, this really doesn't translate to "don't", but be sure
| to keep the transiency in mind.
| roenxi wrote:
| A voice in support - it is more important to solve the
| problems that you do have than the problems that the Docker
| community thinks you might have.
|
| But it is important to at least show a little humility and
| recognising what problems they are solving even.
| nonameiguess wrote:
| The reason for 1 container, 1 process depends upon architecting
| your application in such a way that it works with it. Ideally,
| each application thread is stateless and independent of all
| others, most or all of them don't require graceful shutdown,
| and you can encapsulate state elsewhere in persistent volumes.
| If you're just using a single-node container runtime engine,
| it's not a huge benefit, but it does make fearless restarts a
| lot simpler when all you have to do is stop and start the
| container. Think of Erlang style crash-only programming where
| crashing is the only way to stop a process.
|
| When you get into multi-node orchestrators, though, then the
| benefit becomes much more apparent, especially at a scale where
| nodes are guaranteed to fail every now and again. Rescheduling
| to another node is then, again, as simple as launch a new
| container from the same image on a different node. That happens
| a lot faster if each container only needs to start one process.
|
| It also enables you to practice chaos engineering, which is the
| practice of intentionally crashing your own containers every
| few hours. This is a good practice for security, because if an
| attacker ever gains a foothold in one of your containers,
| they'll quickly lose it, but also to just force your
| infrastructure people and application architects to design in
| such a way that you're guaranteed to be treating servers like
| cattle instead of pets, because you're constantly killing them
| and have no choice if you want your application to still work.
| This ensures what you're deploying is robust and repeatable and
| doesn't secretly depend on some specific server state an admin
| achieved at 1 in the morning some random Sunday hopped up on
| Red Bull that he'll never remember and never be able to
| recreate.
| cle wrote:
| There's nothing intrinsic about crons in the same container
| that prevent you from rescheduling containers on other nodes,
| fearlessly restarting, or whacking random containers. What
| they're doing, how they're handling state, and their
| lifecycles might.
|
| Study your requirements. Whether you run your crons in the
| same container or a separate container depends on your
| system's behavior and failure/recovery scenarios. There are
| complexity tradeoffs--moving the cron scheduling outside of
| the container suddenly means you have moving parts outside of
| your container to test, instrument, monitor, and maintain. If
| you're already doing that, it's not too big of a deal, but if
| you aren't then I would think twice before adding all this
| complexity to your architecture, and make sure it's really
| worth it.
|
| (I can't tell from the article which approach works for their
| specific use cases...I'm pushing back against the general
| advice to "avoid cron in containers". I would default to
| using cron, and only move those processes to separate
| containers if you have a concrete reason to.)
| folex wrote:
| I think "1 container = 1 process" is a common misconception.
| There should be a separation of concerns, but there's no reason
| to go to extremes where it doesn't make sense.
|
| S6-overlay people lay it out pretty neat
| https://github.com/just-containers/s6-overlay#the-docker-way
| a_conservative wrote:
| When docker first arrived, people were confused about how to
| use it. I remember seeing lots of people putting an ssh
| daemon in their containers!?! I like the s6-overlay approach,
| it's pragmatic.
|
| I haven't been using containers as much lately, but I wonder
| if s6-overlay's approach could be used to justify including a
| database into the container with an application. Is that a
| good idea?
| Quekid5 wrote:
| > I haven't been using containers as much lately, but I
| wonder if s6-overlay's approach could be used to justify
| including a database into the container with an
| application. Is that a good idea?
|
| The answer is as always: it depends. My rule of thumb here
| would be: would it ever make sense to configure the
| application to use a different (or even just remote)
| database? If so, then the database should have a separate
| container (when using a local db). This applies most of the
| time.
|
| Similarly, if the database itself is an integral part of
| the application which it doesn't make sense to swap out or
| have remote... then by all means just include it and tell
| the user to mount their data volume to /data or whatever.
| Example: A single-system file indexer.
| Groxx wrote:
| > _would it ever make sense to configure the application
| to use a different (or even just remote) database? If so,
| then the database should have a separate container (when
| using a local db)._
|
| This can probably be further simplified to "does it run
| over ports? then probably yes". E.g. you'd separate your
| application and database, but not your database and its
| filesystem.
| Quekid5 wrote:
| I'm not sure I fully understand, but with my assumptions
| in hand:
|
| That's bit of an implementation detail, I think.
|
| If my app is the most amazing file system indexer using
| PostgreSQL behind the scenes, I don't think the "port"
| distinction is relevant... oh, wait. You're thinking of
| ports as in INET vs. plain sockets?
|
| That took me a long time to get. A _very_ technical way
| to put it, but thank you.
| j45 wrote:
| Docker can be as much work as having to reinstall your servers.
|
| Do your infrastructure work up front or later can both be a valid
| decision that can be different for each project depending on a
| lot of factors.
| tyingq wrote:
| I can see a PHP centric shop not feeling all the upside of
| containers. The typical PHP model is pretty simplistic, so it's
| not hard to replicate most of the docker benefits without using
| containers at all.
|
| That is, I can manage and deploy php apps with different php
| versions, database versions, repo versions, etc, without too much
| pain already. Because deploying a PHP app is mostly just dropping
| files into a directory and restarting a web server.
| lkrubner wrote:
| And yet, when I criticized Docker in 2018, in an article much
| discussed here on Hacker News, many of the comments pointed out
| how much something like PHP would benefit.
|
| raziel2p wrote:
|
| " _However, I think the author trivializes the amount of work
| required to make different types of Python /PHP/NodeJS/whatever
| apps all work in a consistent way through configuration
| management, saying "I can just write a bash script or
| makefile." or "just download roles from Ansible Galaxy". This
| is so painfully ignorant and irresponsible that I fail to take
| the article seriously as a whole._"
|
| From here:
|
| https://news.ycombinator.com/item?id=17062288
| kevinstubbs wrote:
| It's almost as if tyingq and raziel2p are two different
| individuals with their own unique viewpoints and opinions...
| It's not a knock on you - it's just that I never understood
| comments like this.
| afarrell wrote:
| If a person says that thing X is easy for them in their
| context, default to believing them.
|
| If a person says that thing X is hard for them in their
| context, default to believing them.
|
| There is no contradiction there.
| danudey wrote:
| We had some software break earlier this week because our
| Ansible role installs Python 3.8.10 as `python` but the
| system had 3.8.6 installed as `python3`. This meant that when
| we ran `/usr/bin/env python` it used the non-system python
| which wasn't picking up packages from
| /usr/lib/python3.8/dist-packages or wherever Ubuntu packages
| are installed.
|
| The default `pip` and `pip3` commands, however, used the
| system python and _did_ read those directories, meaning that
| it kept insisting the given package was installed. It was a
| frustrating amount of digging around that could have been
| avoided in myriad ways, one of which being containers (though
| for a 30-line python script, docker seems like overkill).
| mejutoco wrote:
| To avoid this problem I use virtualenv for each project
| _and_ I point to the pip binary inside that env
| (env/bin/pip or env/bin/python) in my ansible scripts or
| any other commands. I settled on this after running into
| similar problems
| tyingq wrote:
| I can see both sides. If your PHP app depends on non-core
| modules, or resources that might clash with more than one
| instance running (apcu comes to mind, or redis), or a complex
| database layer, etc...maybe docker would help.
| aduitsis wrote:
| Quite correct, both sides are clearly visible and there
| might be some point to using containers anyway.
|
| But there's a bigger issue here. The comments the OP
| received, are heavily worded to the point that they
| explicitly state that the article made them angry.
|
| Regardless of whether containers are suitable or not for
| task X or Y, it is very difficult to have a normal
| conversation, if one's comments can actually make people
| angry.
| skeeter2020 wrote:
| >> Because deploying a PHP app is mostly just dropping files
| into a directory and restarting a web server.
|
| Lots of solutions can't "just restart a web server" as part of
| their deployment.
| maxk42 wrote:
| What do you think the net effect of deploying a new container
| image is? At some point the server goes down and a new copy
| is loaded.
|
| You could always disable opcaching and then you wouldn't even
| have to restart the server to deploy a new PHP instance.
| danudey wrote:
| Depends on how you set it up, but with a lot of container
| orchestration these days, you create a new image then point
| your load balancer at that instead, giving you a clean
| blue/green deploy without downtime.
| icedchai wrote:
| With most PHP setups, you can usually skip the restart part.
| miltondts wrote:
| What exactly do you mean? How do these solutions deal with
| servers unexpectedly going down? I worked on telecom where
| you need the service to be up 24/7 and we restarted all the
| services on deployment. In waves of course.
| 908B64B197 wrote:
| You can always spin a new one and have the load balancer
| redirect requests.
| tyingq wrote:
| Yes, I didn't claim it as an absolute. Lots of solutions can
| restart a web server or a PHP-fpm instance without trouble.
| Or use things other than containers to manage zero downtime.
| Groxx wrote:
| Lots of solutions _don 't want to_ - that's fairly different
| than "can't". I dare say few companies need nigh-100% uptime,
| and it costs an _incredible_ amount both in setup and in
| maintenance to go from "10s outage while updating" to "no
| downtime while updating".
| conradfr wrote:
| Alternatively you can change the symlink to your www
| directory to the new one.
| marcosdumay wrote:
| > different php versions
|
| How do you do that one? Last time I tried (granted, a long time
| ago), apache-mod-php was hardcoded into a single php version,
| and you couldn't have different versions of it on apache
| either.
|
| I always classified php as one of the most problematic
| languages that need containment. There's not only that hard
| dependency on the version, but the language also has a global
| configuration storage that, last time I used it, can not be
| overloaded.
| toast0 wrote:
| I seem to recall being able to do this with some effort in
| the old days. You might have to compile the mod_php's
| yourself though, looks like at least FreeBSD packages have
| different php versions conflict with each other.
|
| If you use php_fpm as others suggest, it would likely be
| easier.
|
| Either way, you need to have some way to indicate which
| version you want; it used to be common on shared hosting to
| have .php3 vs .php4 vs .php5 files to indicate which version
| you wanted; but I think modern PHP tends to have less code
| that doesn't work with newer versions.
| tyingq wrote:
| 2 different running PHP-FPM instances. Then, any web server
| virtual host can choose which one to use. Via SetHandler for
| Apache.
| beberlei wrote:
| You use PHP FPM (Fast CGI) behind Nginx, so you run multiple
| PHP-FPMs each with their own PHP version. This falls out of
| stock Debian/Ubuntu already and can be easily extended to
| even most recent and older PHP versions through PHP Debian
| Maintainer Ondrejs PPA. Currently PHP 7.0 - 7.4 and 8.0
| available via apt-get
| knownjorbist wrote:
| There's a number of ways to deal with it now. Having
| different versions of php installed is the easy part, having
| the webserver send requests to these is a little more tricky.
| Using PHP-FPM makes this considerably easier as you can have
| different versions listening on different unix sockets, for
| example.
| nicoburns wrote:
| I don't think many people who are using containers are using
| apache.
| conradfr wrote:
| Although if you have multiple projects using the same framework
| and libraries on the same server, using containers allows you
| to use opcache preloading.
| chriswarbo wrote:
| > "Whatever is the underlying executing system, Docker can run it
| with the exact same code, byte by byte."
|
| Having recently delved into containers, I don't like this quote.
| It's not _wrong_ , but I think it's also misleading. In
| particular:
|
| > Whatever is the underlying execution system
|
| Docker can only execute containers on Linux. Indeed, Docker
| itself is essentially a UI and management tool for Linux kernel
| features (chroot, bind mounts, etc.). Whilst Docker provides
| software for other platforms, like macOS, those are just UIs
| bundled with a VM/hypervisor running Linux; and it's that Linux
| system which runs the containers.
|
| This may seem pedantic, but I think it's important to understand
| (a) what's actually happening when we run a container, and (b)
| what we can/can't do with such tools. For example, I was confused
| why my containers couldn't run macOS binaries: the whole point of
| containers is that they run inside the host OS, unlike VMs which
| have their own OS. That's certainly true, but I didn't realise
| that when running Docker on macOS, the host is still a VM running
| Linux!
|
| > Docker can run it
|
| Again, pedantic but important: Docker is a UI and management tool
| for containers, which are ultimately run by an underlying Linux
| system (usually via `runc`, or something compatible). I find this
| important since "Docker" provides much more than just running
| containers, e.g. it has Dockerfiles, images, registries, etc.
|
| If someone just wants to run a container, they don't need Docker;
| and I've found Docker to be a more complicated and convoluted way
| of running containers than, say, OCI tools.
|
| For example, I recently wrote an AWS Lambda function which needs
| to run from a custom container (it bundles some tools which are
| larger than Lambda's 50MB code limit; whilst containers have a
| 10GB limit). I originally did this with Docker, which required:
|
| - Building an 'image' containing the software (I actually did
| this with Nix, rather than `docker build`, for reasons I'll give
| below)
|
| - "Loading" that image into Docker
|
| - "Tagging" the image with the URL of an ECR 'image repository'
| (we could include this tag during building, but I find this way
| keeps more distinction between 'building' and 'deploying')
|
| - Running `aws ecr get-login | docker login` to "log in" to
| Docker. This seems ludicrous to me; I hear it's something to do
| with dockerhub compatibility or somesuch; which is still silly,
| since we're not using that.
|
| - Pushing to ECR using Docker
|
| In contrast, I've now switched that project to use OCI images,
| which only requires the following:
|
| - Building an 'image' containing the software. This is just a
| .tar.gz file, plus some .json files which specify the
| "EntryPoint", the SHA256 of the .tar.gz, etc. (all easily created
| with bash + tar + jq)
|
| - Uploading the image to ECR. This can be done with `aws ecr`
| shell commands, but they're quite low-level (e.g. files are
| uploaded 20MB at a time, which needs a loop) so I did this in
| Python using boto3. Note that the 'tag' is still needed, but it's
| just an argument to the 'ecr.put_image' function.
|
| > the exact same code, byte by byte
|
| This is my main problem with the quote. Whilst it might
| theoretically be possible to use Docker "properly", it seems to
| actively _encourage_ awful practices.
|
| For example, to run "the exact same code, byte for byte" we would
| need to actually _get_ those bytes in the first place. Whilst the
| underlying container is simply a directory (known as a "bundle",
| which `runc` actually executes), Docker abstracts over such
| bundles: first we tar them up into "layers", which we then list
| in a JSON file called an "image", which we then "push" to an
| "image repository". To run a container, we "pull" its image from
| a repository, which we reference by a "tag" (essentially a
| filename).
|
| The de facto tag is "latest", which appears in all sorts of
| documentation and tutorials. Tags may get replaced with new
| images at any time by another push, which makes the idea of "the
| exact same code, byte for byte" rather misleading.
|
| (Thankfully some repositories, like ECR, allow 'immutable tags',
| which can never be overwritten. Rather than 'latest', I use
| hashes of an image's content as its tag; so there's never a
| conflict when I push a new image.)
|
| Next, we can ask how those bytes come into existence in the first
| place. In the world of Docker, we don't put files into a .tar.gz;
| instead, we _download an entire Linux distribution_ , then we run
| a bunch of non-deterministic package management commands like
| `apt-get update && apt-get install -y foo`, resulting in a
| massive, unreproducible binary blob.
|
| Whilst it can be argued that Docker will run 'the exact same
| code, byte by byte', it can also be argued that coin tosses are
| deterministic due to Newton's laws of motion (e.g. a reliable
| Docker system is illustrated in Figure 1 of
| http://statweb.stanford.edu/~susan/papers/headswithJ.pdf )
| oftenwrong wrote:
| Docker has a lot of quirks and rough edges that may be
| attributed to its history. It was open-sourced rather hastily
| from dotCloud, and caught on largely due to its appeal as an
| easy-to-use, open, and complete package for containerisation,
| unlike other containerisation ecosystems that already existed.
| In some ways it's an example of a "worse is better" solution
| (which is not a bad thing). Some of the things you mention are
| just the typical sort of bad design decisions that get made
| without the benefit of hindsight. I'm sure we have all made
| similar missteps while designing systems. It's also true that
| some of these flaws made it more approachable, and actually
| helped with adoption. For example, a reproducible image build
| would be great, but it would not be as easy for a newbie to
| throw together as a quasi-shell-script Dockerfile that lets you
| run whatever non-reproducible commands you like. That less-
| opinionated disposition of Docker was part of what made it easy
| to pick up. In the near future we will probably see a new
| containerisation ecosystem rise with an overall better design
| and more disciplined approach. Docker will always have its
| place in history as the implementation that made
| containerisation mainstream.
| djelal wrote:
| Thanks for the detailed answer, appreciate the insights. You
| are right on all points.
|
| Saying "Docker can run" does not mean it will magically do so
| without work and preparation.
|
| They are many ways indeed to take the same Dockerfile, build it
| 10 times and get 10 different Docker images.
|
| People in the "infrastructure as code" ecosystem can relate,
| they faced the same challenges for years.
| yjftsjthsd-h wrote:
| The maddening thing is that Docker _should_ be a lot more
| portable than it is; the actual specs and architecture need
| _something_ to provide the primitives, but it doesn 't have to
| be runc + Linux namespaces. Sure, the MacOS doesn't have those
| primitives, but ex. FreeBSD and Illumos both do. And, in
| fairness, with runj (https://github.com/samuelkarp/runj#runj)
| we might finally be moving in that direction.
| denton-scratch wrote:
| ...to Docker containers.
|
| I don't care to migrate the infrastructure on which I depend to a
| proprietary product, especially one with a record of sudden,
| surprising changes.
|
| It's perfectly possible to containerize without placing your
| faith in Docker.
| yjftsjthsd-h wrote:
| What about Docker is proprietary or subject to surprising
| changes?
| nickstinemates wrote:
| Nothing but FUD here.
___________________________________________________________________
(page generated 2021-07-30 23:01 UTC)