[HN Gopher] Common Infrastructure Errors I've Made
___________________________________________________________________
Common Infrastructure Errors I've Made
Author : eandre
Score : 46 points
Date : 2021-12-03 20:13 UTC (2 hours ago)
(HTM) web link (matduggan.com)
(TXT) w3m dump (matduggan.com)
| timwis wrote:
| Bummer about Python :/ it's my go-to for CLI tools, but I've seen
| that problem too.. pipenv helps, but I wonder if there's a better
| way to package them so they're more future proof.. or do I really
| need to learn go?
| i_like_apis wrote:
| IMO we loose a lot with go: having to compile, loosing the
| interactive shell, etc. Best case you work with a lot of people
| who know how to install python and use pip. Many people whine
| on boards, but it's not that complicated, especially with
| python 3.
| cconstantine wrote:
| > install python and use pip
|
| Now you've given your users 2 issues completely unrelated to
| the problem they're trying to solve. If you can't give your
| users a single simple command, or a single file to download
| your tool is too complicated to install.
| p_l wrote:
| Not to mention all kinds of python version issues,
| especially given that botched 2-to-3 migration means
| there's a lot more Python 2 than there should be.
| shoo wrote:
| > we loose a lot with go: having to compile
|
| If you're concerned about compile time -- Go does a pretty
| reasonable job of caching (including caching unit test
| results). If you're working on a Python project with a large
| number of unit tests, because Python is so slow to execute
| anything, and the go build and test tools are quite fast, it
| wouldn't be that surprising if it was actually faster to
| compile and run the test suite in Go vs running the test
| suite in Python -- particularly for incremental work where
| you make a change in one library and rerun the impacted
| tests.
|
| If you're concerned more about the workflow of needing an
| additional compile step, go has `go run script.go` which lets
| you use go like a scripting language, assuming you're in an
| environment with the go toolchain installed.
| shoo wrote:
| I've worked with Python for over a decade and have worked with
| Go for a few years. The Go tooling and workflow for building
| and deploying applications is much more pleasant than the
| Python ecosystem. Once you know which target operating systems
| and CPU architectures you wish to deploy to, the toolchain
| makes it very easy to cross compile to produce n deployable
| binaries, then the static linking means you generally only need
| to copy 1 binary file to each target, plus your own application
| configuration. If you have a Python background and have done
| some programming with a static type system before, then Go is
| very quick to learn. The Go language itself is a bit annoyingly
| inexpressive for bashing out algorithms, but that probably
| doesn't matter if you're writing CLI tools.
|
| For packaging python stuff, it kind of depends who or what the
| end user of the CLI tool is. I used to work in a small business
| that shipped windows desktop software to customers, a lot of
| the software was written in python. From memory we packaged it
| with py2exe -- so you end up building a zip file containing a
| self-contained executable, the version of the python
| interpreter your tool needs, along with all the python packages
| as well as native libraries. That worked quite reliably, but
| it'd seem rather distasteful to use that approach for sharing
| CLI tools with your colleagues or CI machines in a dev team!
|
| edit: there are pitfalls to deploying go application binaries
| if you try to put them in scratch containers and use libraries
| that assume they can find data such as timezone data provided
| in the usual place in the filesystem by the distribution (there
| will be no such data files in a scratch container unless you
| explicitly copy them in), or build a go linux binary with
| dynamic linking assuming glibc, then try to run it in an alpine
| based environment with musl. So it's not magic. But it's mostly
| pretty nice.
| travisd wrote:
| > Don't write internal cli tools in python
|
| 100%. I stopped writing anything that had to be deployed
| (basically everything except Jupyter notebooks for data stuff) in
| Python because it's truly a nightmare. Go and goreleaser is great
| for writing a CLI (and if it's public, it can auto generate
| binaries and upload to GitHub, create a Homebrew/Scoop bucket,
| etc)
| thunkshift1 wrote:
| Why is python bad for cli applications
| throwaway19937 wrote:
| Packaging and deploying Python apps is an adventure.
| klodolph wrote:
| > Don't migrate an application from the datacenter to the cloud
|
| Reading the actual text of this one I get a different impression,
| but I'm still not sure I agree with this one. Applications can be
| radically different from each other in terms of how they are run.
|
| At one company, we ran a simple application as SaaS for our
| customers or gave them packages to run on-prem. We'd stack
| something like seven SaaS customers on a single set of hardware
| (front-ends and DB servers). The cloud offering was a no-brainer,
| you can just migrate customers one by one to AWS or whatever, or
| spin up a new customer on AWS instead of in our colocation
| center.
|
| Applications have a very wide range of operational complexity.
| Some applications are total beasts--you ask a new engineer to set
| up a test environment as part of on-boarding and it takes them a
| week. Some applications are very svelte, like a single JAR file +
| PostgreSQL database. The operational complexity (complexity of
| running the software) doesn't always correspond to the complexity
| of the code itself or its featureset.
| shoo wrote:
| > I've been involved now in three attempts to do large-scale
| migrations of applications written for a specific datacenter to
| the cloud and every time I have crashed upon the rocks of
| undocumented assumptions about the environment
|
| I've only participated in a single on-prem to cloud migration.
| Some parts of the migration were easy, e.g. moving a postgres
| DB that was running on some on-prem linux server to run in AWS
| RDS. Some parts were rather unpleasant: where you discover that
| a bunch of the application code that runs in worker processes
| assumes it has access to a shared CIFS network share that can
| be used for communication throug the filesystem, and absolute
| file paths to on-prem CIFS network share locations are stored
| in metadata throughout the database. So then your available
| moves for how to migrate the application code and migrate the
| CIFS network share and migrate the data in the database all
| become somewhat tangled together.
| AyyWS wrote:
| I helped migrate an app from on-prem to cloud. During the
| migration we found that the app needed a locally installed
| oracleDB. Well, it violates on-prem best practices and cloud
| best practices. I think migrating just exposes all the
| shortcuts baked into a "craplication."
| dijit wrote:
| Regarding being cloud provider agnostic: it's not always for
| fault tolerance, there can be a couple different reasons.
|
| 1) it gives your company a stronger bargaining position with the
| cloud provider.
|
| Granted, my companies tend to have extremely high spend- but
| being able to shave a dozen or so percent off your bill is enough
| to hire another 50 engineers in my org.
|
| 2) you may end up hitting some kind of unarguable problem.
|
| These could be business driven (my CEO doesn't like yours!),
| technical (GCP is not supported by $vendor) or political (you
| need to make a China version of your product, no GCP in China!)
|
| Everything is trade offs. AWS never worked for us because the
| technical implementation of their hypervisor was not affined to
| CPU cores of the machine, meaning you often compete with other
| VMs on memory bandwidth. -- but AWS works in China (kinda). So my
| solutions support both GCP and AWS as slightly less supported
| backup.
| cconstantine wrote:
| I'd add another reason: Devs need to be able to run stuff
| locally sometimes.
|
| It's neat having a serverless single page app that is hosted in
| s3, served through cloudfront, with lambda's that post messages
| to sqs queues that are read by god knows what else, but what
| happens when there's a bug? How do you test it? You can throw
| more cloud at it and give each dev a way to build their own
| copy of the stack, but that's even more work to manage. Maybe
| localstack behaves the same, but can you integrate it with your
| test framework?
|
| I never took a hard "we must never use aws-only services"
| approach, but having the ability to run something locally was a
| huge plus. Postgres RDS? Totally fine, you don't need amazon to
| run postgres. Redshift? Worth the lock-in given the
| performance. Lambda? Eh, probably not, given that we already
| have a streamlined way to host a webapp.
| wdfx wrote:
| Agree here.
|
| At my $WORK we run all of our backend on AWS but anything
| I've touched has to also run locally.
|
| We use serverless framework, and there are plugins for
| running the lambdas locally as well as for dynamodb, sqs, ses
| and eventbridge.
|
| I think it's a case of choosing your dependencies carefully
| though. I would be wary of integrating against an AWS service
| which does not have an API compatible offline or provided-
| elsewhere alternative.
|
| A case where we fail at this is Cognito. Even our
| 'offline'/local stack has to connect to our dev environment
| for that one.
| dijit wrote:
| Argh. I wish I could upvote this twice.
|
| One of the main reasons we never used google spanner was that
| we can't test it locally.
| cconstantine wrote:
| People don't often think of their local development
| environment as a "platform" that their stuff needs to work
| in, but it really is. In that sense, unless you're hosting
| off of your laptop (please don't!), every app is multi-
| platform.
|
| Every startup I've worked at (and I've been at this for 15+
| years) has moved hosting providers, but I still wouldn't
| put it high on the list of reasons to avoid vendor lock-in.
| If you make sure someone(s) know how the app actually runs,
| and you try to pick stuff you can run locally, the vendor
| lock-in stuff won't be your biggest challenge in the move.
| shoo wrote:
| i agree 100% that you need to structure the project so
| there's a way to develop locally in your dev machine --
| without a network connection -- and run integration tests
| against local versions of services.
|
| looks like google spanner have plugged that workflow gap
| since you evaluated it for your project:
|
| > The Cloud SDK provides a local, in-memory emulator, which
| you can use to develop and test your applications for free
| without creating a GCP Project or a billing account. As the
| emulator stores data only in memory, all state, including
| data, schema, and configs, is lost on restart. The emulator
| offers the same APIs as the Cloud Spanner production
| service and is intended for local development and testing,
| not for production deployments.
|
| https://cloud.google.com/spanner/docs/emulator
|
| although there are https://cloud.google.com/spanner/docs/em
| ulator#limitations_a...
| sneak wrote:
| The guy also recommends using several proprietary AWS services
| in other points.
|
| Then he goes on to advocate against designing for cloud
| flexibility.
|
| This almost feels like AWS marketing.
| iechoz6H wrote:
| > Don't run your own Kubernetes cluster
|
| If we ran our cluster in the cloud we'd be on the hook for
| hundreds of thousands of dollars of additional costs due to the
| high throughput of our service. There are always exceptions to
| any list of rules.
| betaby wrote:
| 0) Don't write software - outsource it to someone smarter.
| barbazoo wrote:
| -1) Don't outsize writing software. Let someone smarter
| outsource it for you.
| torton wrote:
| > Don't write internal cli tools in python
|
| A lot of the advice is good but I take an issue with this one.
| With poetry and docker, packaging Python apps for easy
| consumption is a non-issue. Same for Ruby. If you can get your
| team to standardize on poetry, you might not even need containers
| -- but, honestly, running these tools from CI or automation
| (anywhere) is so useful that you probably want container versions
| anyway.
|
| Golang is not a good fit for exploratory CLIs that work with
| complex data structures and are written for one-off, low-CPU
| consumption purposes -- not for scale-up API services. Just
| having an interactive shell (or `pry` in Ruby, those two are
| identical for the purpose) saved me probably weeks of time.
| Trying to unit test any moderately complex API surface brings me
| to tears when I compare it to trivial object mocking in something
| like Ruby.
|
| Python (or Ruby) are ideal for this and have excellent frameworks
| for CLI tools.
| p_l wrote:
| Ruby is pretty good in this, but Python I'd heavily argue
| against. Maybe in another decade when they manage to
| restabilize what they wrought - it used to be easy.
|
| Python gets more and more complex the further away the people
| running the tool are from fancy recentish distros (Fedora,
| Ubuntu, Arch) or special constrained environments (Nix, running
| the CLI in docker). The moment you have to deal with
| unspecified RHEL version (6 is still reasonably common) or
| derivative, or Mac or Windows, kiss any expectation of python
| packaging being nice "bye bye".
|
| Unless of course you have a platform team that can handle the
| packaging and distribution for you, but then it probably falls
| a bit under "constrained environment".
| chousuke wrote:
| I'd state it as "don't use external dependencies carelessly".
| It applies to more than just Python.
|
| Writing and deploying Python tools is easy if all your
| dependencies come in distro packages.
| p_l wrote:
| And your code is carefully written to run perfectly fine on
| both 2.7 and anything between at least 3.4 and 3.10, but
| might be better to handle as early as 3.0
|
| ... I might have somewhat similar scars to TFA author, I
| guess...
| duped wrote:
| > With poetry and docker, packaging Python apps for easy
| consumption is a non-issue.
|
| This is why you shouldn't write CLI tools in Python, you need
| frigging docker to package them
| 9dev wrote:
| I recently had to package a machine learning Jupyter notebook
| created by a very smart person, albeit a scientist, not a
| developer. Getting the tool somehow into production, making it
| reproducible, testable, and maintainable, has proven to be a
| major headache. Before, I only casually dabbled in the python
| world, but this was the first time I had to care about
| packaging, dependencies, CI and the like. Turns out, as TFA
| said, nobody knows how to package python apps right. For what
| it's worth, I wasn't even able to find some kind of best
| practice to manage friggin dependencies. There's like a myriad
| of package managers, all of them work differently, and nobody
| seemed to had something like redistributing an app to other
| people on their mind. Coming from PHP, JavaScript and Go, this
| was utterly ridiculous to me. Go ahead, tell me I got it all
| wrong and it's really easy using tool Xyz, but if a developer
| with some experience under their belt isn't able to figure this
| out in a few days, things are just broken.
| habitue wrote:
| > With poetry and docker, packaging Python apps for easy
| consumption is a non-issue.
|
| Yeah, apps it's worth doing this for, cli tools it's not.
| grafelic wrote:
| > You spring back to the present day, almost bolting out of your
| chair to object, "Don't do X!". Your colleagues are startled by
| your intense reaction, but they haven't seen the horrors you
| have.
|
| They may be startled, but they almost certainly won't listen. The
| purgatory nature of IT work culture ensures this repetitive
| pattern.
___________________________________________________________________
(page generated 2021-12-03 23:00 UTC)