[HN Gopher] You don't need all that complex/expensive/distractin...
___________________________________________________________________
You don't need all that complex/expensive/distracting
infrastructure (2019)
Author : ddtaylor
Score : 148 points
Date : 2021-03-11 12:01 UTC (11 hours ago)
(HTM) web link (blog.usejournal.com)
(TXT) w3m dump (blog.usejournal.com)
| 1996 wrote:
| So true. HN often admires complexity for complexity sake.
|
| Even without cloudflare, a static website on 5 VPS from 5
| different provider on 5 different continents with a Round Robin
| DNS and a low TTL will get you most of the bangs for bucks
|
| Add geoIP to the mix (like from Azure) and you're golden, even if
| you are just scp'ing files to 5 VPS.
|
| If a DC burns down, remove the A record (or even better: have a
| script do it for you) and reroute to the remaining 4.
|
| Extra bonus: you can fire the DevOps team :)
| jameshart wrote:
| That's funny, sometimes when I visit your site, I get an HTML
| page that references script files and resources that come back
| as 404s. I wonder if it's possible I'm hitting the site during
| a deployment and getting the new HTML page from one of your
| servers, but then the resource requests are going to a
| different server which hasn't received the new version yet.
|
| I guess occasional broken images and script failures are
| unavoidable though. You'd need like, some dev ops engineers to
| solve a problem of that complexity.
| tartoran wrote:
| Why fire the team and not just repurpose them? They're already
| familiarized with some aspects of the product.
| tpetry wrote:
| That's easy if you are just serving static html. But a database
| replicated on 5 continents? That's a lot of latency for
| replication.
| Zelphyr wrote:
| Many people keep saying something to the effect of, "There is a
| right time to scale a system and we need to focus on that." This
| is true. There _is_ a right time to scale up. In my experience,
| you rarely know when that might be except in hindsight.
|
| I worked at a startup once and we built the product in a way that
| it should have been able to scale up pretty well. Then one of the
| co-founders went on a national news channel and started talking
| about our company. Our servers got absolutely slammed with
| traffic. We did the best we could but it didn't go well.
|
| Now, we _should have known_ to expect increased traffic from an
| event like that, and, in fact, we did. We simply didn 't know
| just how much traffic we were going to get, and, honestly,
| couldn't have known that. We had only had a tiny fraction of that
| traffic until that event. That was the best basis for comparison
| we had at that point in time.
|
| We updated and upgraded some things after that and the next time
| he went on that news channel, things were a lot better.
|
| Knowing when to scale up is a hard problem. You have hints, sure,
| but there's never a data point that says, "Ok, you have until
| July to migrate this system from a single VPS to a full
| Kubernetes build-out."
| Zelphyr wrote:
| I would like to add that you'd be foolish to use my post as a
| reason for building a system to scale to high levels
| prematurely. I agree with the original article that doing so
| bogs you down in needless complexity and wastes time and
| resources.
|
| At that startup, we built the system to scale to where we were
| projecting traffic would be within a given time period. If that
| co-founder going on national television hadn't happened, our
| original plan would have been sufficient.
| jameshart wrote:
| There is obviously some point, somewhere between 'just running a
| website out of my spare bedroom' and 'Facebook', where some of
| this infrastructure _does_ become necessary.
|
| It is _very important_ that you realize when you pass one of
| those points.
|
| It doesn't help to just tell people 'you don't need kubernetes,
| you are not Facebook'
|
| There are a lot of companies that aren't Facebook, but they still
| have multimillion dollar revenue streams riding on service
| availability, with compliance obligations and employee paychecks
| on the line, and which need to be able to reliably release
| changes to production code every day, written by multiple
| development teams in different time zones, so... yeah, maybe they
| are a bit beyond just scp'ing their files up to a single Linux
| VPS.
| protonimitate wrote:
| This is a good point. I wish there was more discussion around
| _when_ to consider making these decisions, how to transition,
| what types of systems are in the middle ground between bedroom
| and facebook, etc.
|
| It seems all of the articles/posts that get are attention are
| the "you don't need it ever" opinions, but those are incredibly
| short sighted and assume that everyone is recklessly applying
| complex infrastructure.
|
| In general, it would be nice to have more nuanced discussions
| around these choices rather than the low hanging fruit that is
| always presented.
| jameshart wrote:
| So I agree there's definitely a need for more writing in this
| space (and you'd think it might be particularly of interest
| to incubating startups...)
|
| My top level take on this is that the cues to take things up
| a level in terms of infrastructure diligence are when the
| risk budget you're dealing with is no longer entirely your
| own. Once you have a banker, a lawyer, an insurance policy,
| an investor, a big customer, a government regulator,
| shareholders... ultimately once there's someone you have to
| answer to if you haven't done enough to protect their
| interests.
|
| And risk management isn't just 'what will happen if the
| server catches fire', it's 'how do I guarantee the code
| running in production is the code I think it is?' And 'how do
| I roll back if I push a bug to production?' And 'how can I
| make sure all the current versions of the code come back up
| after patching my servers?'
|
| And it turns out that things like kubernetes can help you
| solve some of those problems. And things like serverless can
| make some of those problems go away completely. But of course
| they come with their own problems! But when those things
| become important enough that you need to solve for them to
| answer the risk aversion of sufficient stakeholders, the cost
| of adopting those technologies starts to look sensible.
| mumblemumble wrote:
| What I tend to see missing is clear explanations of, "What
| problems does this solve?" Usually you see a lot of, "What
| does it do?" which isn't quite the same thing. The framing is
| critical because it's a lot easier for people to decide
| whether or not they have a particular problem. If you frame
| it as, "Do we need this?" though, well, humans are really bad
| at distinguishing need from want.
|
| Take Hadoop. I wasted a lot of time and money on distributed
| computing once, almost entirely because of this mistake. I
| focused on the cool scale-out features, and of course I want
| to be able to scale out. Scale out is an incredibly useful
| thing!
|
| If, on the other hand, I had focused on the problem - "If
| your storage can sustain a transfer rate of X MB/s, then your
| theoretical minimum time to chew through Y MB of data is Y/X
| seconds, and, if Y is big enough, then Y/X becomes
| unacceptable" - then I could easily have said, "Well my Y/X
| is 1/10,000 of the acceptable limit and it's only growing at
| 20% per year, so I guess I don't really need to spend any
| more time thinking about this right now."
| braveyellowtoad wrote:
| Yep. All of us tech folks read sites like hackernews and
| read all about what the latest hot Silicon Valley tech
| startup is doing. Or what Uber is doing. Or what google is
| doing. Or what Amazon is doing. And we want to be cool as
| well so we jump on that new technology. Whereas for the
| significant majority of applications out there, older less
| sexy technology on modern fast computers will almost
| certainly be good enough and probably easier to maintain.
| XorNot wrote:
| Uber is one of those alleged behemoths I'm incredibly
| skeptical about the scaling problems of.
|
| The service they provide can't be anywhere near the
| Facebook/Google league of scale, nor Netflix level of
| data/performance demand.
| edejong wrote:
| We don't have data sheets, like professional electronic
| components have. I guess we value Turing/Church more than
| predictability.
| yawaramin wrote:
| FTA:
|
| > Obviously if you're FAANG-level or some established site
| where that 0.1% downtime translates into vast quantities of
| cash disappearing from your books, this stuff is all great.
| You have the funds to do things "right". Your fancy zero-
| click continuous deployment system saves you
| thousands/millions a year. But at indie maker, "Hey look at
| this cool thing I built ... please, please someone look at it
| (and upvote it on Product Hunt too, thx)" scale -- the scale
| of almost every single site on the net -- that 0.1% is
| vanishingly insignificant.
| richardw wrote:
| No simple advice will cover all the cases but not a bad place
| to start:
|
| https://basecamp.com/gettingreal/04.5-scale-later
| cesaref wrote:
| When you are in a position to pay someone full time to look
| after the web infrastructure of your project, that's when you
| should think this stuff through.
|
| Before that, don't worry about the tech, go for what you
| know, and expect it to need totally re-writing at some point.
| Let's face it, this stuff moves quickly.
| blacktriangle wrote:
| That's the best metric I've heard. When your Heroku bill is
| the same as what a full time ops person would cost, start
| looking to hire.
| jacquesm wrote:
| I would say it has to be at least double that and maybe
| even higher.
| Melkman wrote:
| Also, when cost caused by downtime and updates and/or
| hardware problems exceed the cost of this full time
| engineer. Even then K8s is probable not the next step.
| Simple HA with a load balancer and seperate test and
| production environments plus some automation with an
| orchestration tool works wonders.
| marcosdumay wrote:
| That's not good. That kind of stuff must be derived from
| your business plan, not from the current situation. You
| should know that "if I'm successful at this level, I will
| need this kind of infrastructure" before the problems
| appear, and not be caught by surprise and react in a hush.
| (Granted that reacting in a hush will probably happen
| anyway, but it shouldn't be surprising.)
|
| If your business requires enough infrastructure that this
| kind of tooling is a need at all, it may very well be an
| important variable on whether you can possibly be
| profitable or not.
| marcus_holmes wrote:
| if your business plan has this much detail, you'll spend
| more time updating it to reflect current reality than
| actually shipping features.
|
| Companies that are growing fast enough to need Kubernetes
| are also changing fast and learning fast.
| marcus_holmes wrote:
| This. When you have enough revenue to hire new dev, and
| infrastructure is your most pressing problem to solve with
| that money, hire someone who knows this stuff backwards to
| solve it.
| rualca wrote:
| > There is obviously some point, somewhere between 'just
| running a website out of my spare bedroom' and 'Facebook',
| where some of this infrastructure does become necessary.
|
| Frankly, that blog post reads terribly miopic. The author
| essentially looks at his personal experience setting up his
| single-dev personal project, complains that he spent some time
| jump-starting a basic CICD pipeline, and from there he proceeds
| to argue that no one needs any of that.
|
| Well, how does he expect to check whether their changes break
| or not the deployment? How does he expect to deploy his stuff?
| How does he plan to get multiple commits to play nice prior to
| deployment? How does he expect to roll back changes? How does
| he plan to address request spikes, either from normal traffic
| or attacks? Does he plan to get up at 3AM on a Tuesday to scale
| up deployments or does he plan to let his site crap out during
| the night and just get back to it in the morning?
|
| And somehow we are supposed to believe that this naive belief
| that getting features out of the door is all that matters, and
| basic ops is not needed by anyone?
| thisiszilff wrote:
| Yeah, I kind of agree with the author, but not 100%. The
| takeaway shouldn't be "don't use CI, kubernetes, etc." but
| rather "don't learn something new" when your goal is to get a
| product out the door. If you already know kubernetes, then it
| might seem silly to not use it given how it simplifies
| things. It's only when you don't know kubernetes that you'll
| run into problems.
| rualca wrote:
| > but rather "don't learn something new" when your goal is
| to get a product out the door.
|
| I don't agree at all, because that argument is a poorly
| put-together strawman.
|
| No one ever had to choose between setting up a CICD
| pipeline and churn out features. Ever.
|
| In fact, continuous deployment makes it possible for
| everyone to get features out if the door as fast as they
| possibly can, and ensure deployments are two-way door thing
| instead of a finicky one-way door pray-that-it-works event.
|
| The door where features have go get through is not JIRA,
| but the prod environment. It's inconceivable how some
| people try to pretend that automating operations adds no
| value.
|
| And hell, CICD nowadays is trivial. You know why? Because
| of tools like Kubernetes/Cloudformation/CDK/Ansible/etc.
| tpxl wrote:
| There is a whole lot of time before your first customer
| where you don't need any of this. Getting a MVP out >
| getting your engineering shinies.
|
| In my company I first had servers directly on a bare
| metal box, with half the infra running on my workstation,
| then moved everything to docker-compose, then kubernetes
| and now I'm getting ready to get my CICD pipelines.
| Spooky23 wrote:
| Another factor is the operating model.
|
| People bitching about K8S, systemd, etc miss the point that
| these systems allow a qualified generalist ops admin pop in and
| fix stuff without knowing the app well.
|
| For a little company with <500 servers, it doesn't matter. As
| size and complexity increase, figuring out how the guy left
| last year setup FreeBSD becomes a liability.
| nmfisher wrote:
| Man, I didn't realize 500 servers was still "little company"
| level!
| Spooky23 wrote:
| Marginal cost for servers is very small, and with
| containers, etc, they multiply like bunnies.
|
| Many of our devs are running 8-15 VMs just for their dev
| work. Depending on what you do and how, it's easy to hockey
| stick the count!
| marcosdumay wrote:
| That surfaces a very important communication problem. If
| you consider 500 servers a "small datacenter", then, of
| course, you will say that K8S adds value to small
| operations.
| greggyb wrote:
| A qualified generalist ops admin should be productive on any
| modern UNIX or UNIX-like within a couple weeks.
|
| Figuring out how the guy last year set up FreeBSD consists of
| similar effort and process as figuring out how the guy last
| year set up Debian or CentOS or SUSE.
| Spooky23 wrote:
| It can be challenging. I basically bought my house based on
| figuring that out for people in the old days.
|
| It's not an issue with FreeBSD or anything per se. Usually
| there's a genius superhero guy who does whatever, who burns
| out/quits/otherwise goes away, and the business is stuck.
| jrochkind1 wrote:
| Is there good tooling that's actually in between "just running
| a website out of my spare bedroom" and "kubernetes"?
|
| I think maybe not.
| marcosdumay wrote:
| Machine installation manuals, "Create machine" shell scripts,
| manual docker, script launched docker, VM management tools...
| Well, technically, puppet and it's kind are there too,
| depending on whether you consider them viable or not.
| theptip wrote:
| I found Dockerizing my app really early to be a good bridge
| to k8s.
|
| Build container locally, test, push. On Linux box, pull the
| new app container, stop the old app container, start the new
| app container. Separate nginx reverse proxy also in docker so
| if you're fast enough you don't drop any packets.
|
| I'd say docker is worth using from the get-go, but if you are
| most familiar with raw Linux probably just use that for your
| first iteration.
| didibus wrote:
| You can use something like AWS Application Load Balancer with
| two EC2 hosts behind it. Linode also offers an LB if you
| prefer Linode.
|
| Now do a simple Blue/Green deploy. Deploy to the first host,
| so maybe you use Java so just have a script that shutdowns
| your app and copies over a new uberjar and starts it up
| again. Try it out, if it all works, swap the LB to point to
| this host instead. If it results in errors and failures your
| rollback procedure is to swap it back again to the previous
| host.
|
| You can scale this pretty easily as well, just add more hosts
| behind your Blue/Green groups.
|
| Now the last challenge is actually going to be your DB. You
| can't swap DB instances like that, so that's where something
| like AWS RDS or a managed DB service is useful, cause they
| often offer automatic replica and fail over.
|
| But if you're lazier (or don't have the money) then that,
| just have a third host running PostgressSQL with a scheduled
| backup of your DB to some other durable storage like AWS S3.
| And when you need to update that host, schedule a
| maintainance window for your app/website.
|
| So...
|
| 1) You can start with one single VM and have your DNS point
| directly to that host. Where the host runs your DB and your
| app. Deployments are you ssh-ing, shutting down, copying a
| new uberjar, starting up again. And you turn this into a
| script that does it for you.
|
| 2) Then you add a DB backup cron job to S3 so you don't lose
| all your users data from an incident.
|
| 3) Now if you want to prevent needing downtime when you
| deploy, you add an LB and move your DNS to point to the LB
| and put the host behind the LB, it becomes Green, and add
| another host which will be Blue, move your DB to a third host
| that's not behind the LB. And now do what I said earlier. You
| can now deploy new versions of your app without downtime, and
| do quick rollbacks in case you deploy something that breaks
| your users.
|
| 4) If your user base grows and one host isn't enough anymore,
| you can add more behind the Green and Blue groups. Everything
| else is the same, just your script now needs to swap between
| the groups.
|
| 5) Now you might start to want tests running automatically,
| uberjar being created automatically when you commit, and have
| a beta stage and all that. So you can setup a CI/CD pipeline.
| Have a test host and a test DB. And your app has config to
| know which environment it runs under to detect what DB to
| connect too.
|
| 6) Finally, if your DB becomes a scaling bottleneck due to
| growth, congratulations, you've now becomed a successful
| business and can just hire other people to deal with it.
| bearjaws wrote:
| Serverless compute in your favorite cloud provider IMO.
|
| Been using serverless framework for over a year now, handles
| my deployments, load balancers, API gateway and obviously
| Lambda.
|
| A proper CI/CD setup is only about 8 hours of work with a
| monolith app.
| salmonlogs wrote:
| What about a managed service like EKS or GKE? A minimum k8s
| config is perhaps 20 lines to spin up a service and expose
| the ports - that feels pretty achievable for bedroom coder?
| skrtskrt wrote:
| A couple ansible scripts to deploy the k3s kubernetes
| distribution onto cloud VMs.*
|
| Add LetsEncrypt to the Traefik install for https
|
| Use the default Helm chart to make your k8s deployments for
| your apps - you'll probably only need to tweak a couple
| values to start - ports and ingress rules.
|
| * k3s comes with Traefik pre-baked to listen on ports 80 and
| 443 so you can run it "bare metal" without a cloud provider's
| load balancer service or any special tweaks to Traefik
| config. If you just Helm install any of the standard k8s
| ingress providers like the standard Traefik chart, Consul, or
| Nginx, you have to tweak a bunch to get them working on
| privileged external ports 80 and 443.
| jacquesm wrote:
| Fair point. But people just love to over-complicate things and
| it is far more often that you come across absolutely ridiculous
| infrastructure monstrosities for very small business than that
| you come across the opposite.
| PragmaticPulp wrote:
| I've found that career FOMO drives a lot of the desire to
| overcomplicate infrastructure.
|
| It's hard to look at job listings from prestigious companies
| listing K8s and other buzzwords as requirements without
| feeling some desire to add those buzzwords to your own
| resume.
| tyingq wrote:
| Facebook is kind of an interesting example, as they got pretty
| far into "hyperscale" with mostly PHP, memcached, and Mysql.
| jasode wrote:
| _> Facebook is kind of an interesting example, as they got
| pretty far into "hyperscale" with mostly PHP, memcached, and
| Mysql._
|
| But PHP and MySql are only the surface level tools we _see_.
|
| There is also the _unseen_ and that 's where the immense
| complexity lies hidden under the surface.
|
| E.g. To help with scaling (which Friendster failed at),
| Facebook had separate MySql instances for each school. An
| interview with Facebook co-founder Dustin Moskovitz mentioned
| they had constant stress of emergencies and fires to put out.
| They had tons of custom tooling, Linux scripts, batch jobs,
| automation/orchestration/dashboard, etc and it's very likely
| the combination of all that ends up inventing an "invisible
| internal Kubernetes" without calling it "Kubernetes" to
| manage their server fleet. In other words, if
| Docker/Kubernetes/AWS were around, the early Facebook team
| might have offloaded _some_ of the complexity onto a 3rd-
| party to reduce the all-nighters. But the tools didn 't exist
| yet so they had to do it the hard way and invent "deployment
| orchestration" themselves.
|
| [EDIT to reply]
|
| Thanks for the "Tupperware" comment. A Google search leads to
| a Facebook post saying they renamed it to "Twine":
|
| https://engineering.fb.com/2019/06/06/data-center-
| engineerin...
| jeffffff wrote:
| it's called 'tupperware'
| tyingq wrote:
| That's recent though. I'm saying they scaled up quite a
| long way before doing anything like that. From the outside
| it looks like they didn't really branch out into things
| like this until 2008 or so.
| jasode wrote:
| _> From the outside it looks like they didn't really
| branch out into things like this until 2008 or so._
|
| No, even before the Newsfeed feature rollout of September
| 2006, they already had crazy complexity orchestrating
| multiple servers for many schools. They were only a
| "simple PHP+MySql" website when it was February 2004 with
| only Harvard students on it. Dustin said he was the one
| in charge of spinning up new servers for each new school
| and he said it was a nightmare of engineering effort.
| mattmanser wrote:
| But today, 16 years of Moore's law later, you could now
| have gone way, way, way, way past that.
|
| They probably could have done most of America on a single
| server.
|
| For example, even in 2009 Stack Overflow was running off
| two dedicated servers:
|
| https://stackoverflow.blog/2009/01/12/new-stack-overflow-
| ser...
| Person5478 wrote:
| IIRC, they were doing things like patching bittorrent to
| prefer closer IP's to pull from during their deploys. So
| it would roll through their network from 1 side to the
| other without just completely saturating the entire
| network.
|
| But the point still remains, they had challenges to
| solve, but they were also simple, which made it possible
| TO solve those challenges.
| rualca wrote:
| > Facebook is kind of an interesting example, as they got
| pretty far into "hyperscale" with mostly PHP, memcached, and
| Mysql.
|
| Isn't Facebook known for hiring top world experts such as the
| likes of Andrei Alexandrescu to hyperoptimize all the way
| down their whole stack?
|
| That hardly sounds like running vanilla PHP with your generic
| nem ache and mysql.
|
| And how do you exactly think they get so many 9s in their
| reliability? Is this plausible with the "a single linode box
| ought to do" nonsense?
| tyingq wrote:
| He got there in 2009. Facebook did run for a long time with
| a fairly "vanilla" setup, yes.
|
| A hacker got ahold of their "index.php" in 2007:
|
| https://gist.github.com/nikcub/3833406
|
| It is decidedly not a tour-de-force of technical prowess,
| just typical old school "bag of functions in the same
| namespace" PHP.
| redis_mlc wrote:
| > Facebook is kind of an interesting example, as they got
| pretty far into "hyperscale" with mostly PHP, memcached, and
| Mysql.
|
| Those are fantastic technologies and each is remarkable at
| what it does. Besides that, those are some of the easiest
| skill sets to hire for.
| ahofmann wrote:
| This could be the final post to this discussion. You can't
| really get more to the point than that. We are running a
| large German browser game on individual VPS instances. For
| many years, with PHP, MySQL and Redis. It works and it runs
| just fine.
| bombcar wrote:
| It's not one that just recently had to start migrating from
| Flash, is it?
| bombcar wrote:
| There are ways to quantify it - but a real telling point is if
| your service is down because of the complexity of tooling you
| don't fully understand.
|
| Many problems can be solved with other methods - especially if
| you do NOT have global all-sync'd requirements like Twitter and
| Facebook. Many SaaS (example: Salesforce) are perfectly
| acceptable sharding customers to particular instances thereby
| limiting downtime if something goes wrong.
|
| And always keep in the back of your mind that the largest VPS
| or servers are very performant today - having to scale sideways
| may be masking inefficiencies in the code.
| jameshart wrote:
| Or if your service is down because the single hosting site
| with your Linux box in it is _on fire_
|
| Also, I assume sales force engineers aren't SSHing into each
| shard and running docker-compose manually on each one.
|
| Obviously you need Goldilocks infrastructure - not too
| little, but not too much.
|
| All I'm saying is it doesn't go 'one Linux box, one Linux
| box, one Linux box, reach Facebook size, build your own data
| center'.
| bombcar wrote:
| True - but the datacenter burning still hits you where it
| hurts if your entire Kubernetes cluster is living in that
| datacenter.
|
| Whereas if you make a point of spinning up each Linux box
| on different datacenters (and even perhaps different
| providers) you are at least resilient against that.
|
| Using advanced tooling doesn't remove the need to do actual
| planning beyond "oh I have backups" - you should be listing
| the risks and recoveries for various scenarios - though
| being able to handle "AWS deleted all my servers because I
| returned too many things via Amazon Prime" means you can
| also handle any lesser problem.
| jsilence wrote:
| A whole datacenter on fire, come on, what are the odds?
|
| Oh, wait! Nevermind...
| callmeal wrote:
| >Also, I assume sales force engineers aren't SSHing into
| each shard and running docker-compose manually on each one.
|
| Nothing wrong with that if all you have is 10-15 shards (or
| whatever your annoyance upper limit is).
|
| Might even be faster and cheaper than spending hours
| learning/debugging/figuring out edge cases on something
| else.
| jameshart wrote:
| But there's an upper limit, right? And you'll probably
| hit it somewhere before you pass 100bn USD ARR?
| edoceo wrote:
| IME you'll hit it around $500,000 USD ARR. But again,
| situations vary. I've worked projects where hours long
| scheduled downtime at the weekend was acceptable. So
| easy!
| jacquesm wrote:
| Having an off-site backup of your data is a no-brainer at
| any level of complexity.
| MR4D wrote:
| > There is obviously some point...
|
| My take is simple - if Craigslist doesn't need it, then I
| probably don't either. (I may _want_ it, but that 's
| different).
|
| Definitely not perfect, but it works for me. Obviously
| more/better guideposts would make this a much better yardstick.
| jameshart wrote:
| I duunno exactly what Craigslist is running on today, but
| some slides from 2012[1] include items like running MongoDB
| over "3 shards across 3 node replica sets" with "duplicate
| config in 2nd data center"; a "sharded multi-node" redis
| cluster; multiple "vertically partitioned" MySQL clusters;
| the actual application stack on top of that being mod_perl on
| Apache, and it was all operated by a 'small but growing'
| dedicated team.
|
| It might or might not be notable that the growth of that tech
| team included, one year later, _hiring Larry Wall_ [2].
|
| Given that, I'm not sure how to apply 'if Craigslist doesn't
| need it, I probably don't' as a general principle....
|
| [1] https://www.slideshare.net/jzawodn/living-with-sql-and-
| nosql... [2] http://blog.craigslist.org/2013/10/15/artist-
| formerly-known/
| MR4D wrote:
| I see language as a personal choice, and while I wouldn't
| run perl today (I'm partial to Python, myself), I think
| there stack is fairly simple - tried & true language
| (however quirky it may be), no virtualization, simple metal
| hardware stack, and yet they serve an incredible number of
| users. In that sense, I'm sure hiring Larry was a no-
| brainer.
|
| They are no FB, but they're bigger than almost anything
| else on the web [0]. I doubt most people will build
| anything bigger in their careers. I certainly won't.
|
| A huge caveat might be app features. Clearly running
| something like travelocity could require more pieces, and
| something like a BI app in a fortune 100 company would have
| many more connectors to data sources. But in general,
| they've done pretty well using only newer versions of what
| worked way back in 1999, and to me, that's just incredible.
|
| [0] - https://www.semrush.com/blog/most-visited-websites/
| theptip wrote:
| I'd suggest that when you have paying customers, and your duck
| tape deploy process is starting to feel risky to run during
| business hours, that's a good time to do your first taking of
| stock. Before that point it should be the fastest thing you can
| wire up given your own personal tooling expertise.
|
| (I'd argue that k8s is actually lean enough to be a competitor
| here if you're already an expert and you use a managed service
| like GKE).
|
| Your infra should be in service to some uptime/performance/DX
| goals. Define those (honestly!) and then you can figure out
| what you need to improve.
|
| In particular if no customer has ever noticed downtime, and you
| don't think you'd lose business if someone hit a 30-min outage,
| then you probably don't need to upgrade. Twitter was
| notoriously unreliable in the early days and they made it, for
| example.
| otikik wrote:
| The same applies to the frontend.
| Traubenfuchs wrote:
| Operations guys, "devops" as they are called now, remain the
| arch-wizards with grey beards. They love creating highly company-
| customized, incredibly complex, often times brittle
| infrastructure and deployments based on obscure scripts written
| in non mainstream languages based on plugins and libraries
| without documentation. The same opaqueness as in IT security, but
| in emergencies you can downsize/ignore IT security, but without
| the ops guys your product can not survive.
|
| Kubernetes, Helm and cloud-platform specific quirks were a
| godsend that cemented in this status for at least another decade.
|
| There is probably no job with higher job security in tech than
| this.
| supercon wrote:
| So were it not for this era of cloud platforms and the other
| technical "advancements" you mention, would we somehow be in a
| different place when it comes to maintaining and operating
| software?
|
| As someone in a devops team, I could argue that ops teams are
| rarely the ones deciding what technologies to use. If we follow
| the basic sales->development->maintenance -pipeline, as ops
| your task is simply to keep the service running and secure with
| the tools you are given. Sure there can be added complexity,
| but I doubt it is as systemic/general as you make it seem to
| be.
|
| Anyway, if this is seen as a problem the devs could simply take
| on the responsibility of learning those quirks themselves or
| produce the level of quality software that requires no
| maintanance whatsoever. But for some reason that hasn't
| happened, eventhough the original idea behind devops was to
| blur this line. So until that day comes, I'll sure enjoy
| whatever security one can really have in todays job market.
| Traubenfuchs wrote:
| No, we would simply still use incomprehensible websphere
| setups. Now we got kubernetes as the application server of
| the current decade that's application language/framework
| agnostic.
|
| But it shouldn't be like this. Devops shouldn't have to
| exist. Everything should work like heroku, where even
| mediocre devs can quickly setup proper pipelines.
| breckenedge wrote:
| > There is probably no job with higher job security in tech
| than this.
|
| Will the last person to leave please turn off AWS.
| mercury_craze wrote:
| This isn't totally related to infrastructure but figuring out
| what you should build, from what you could build is something
| I've struggled with as I've moved about the engineering
| management ladder. My current role is a stone's throw from solo
| developer (only developer at 4 person start-up) and while I've
| got total autonomy in the work I complete and the approach I
| take, it's a terrible feeling discovering that you're not super-
| human and you need constraints like almost every engineer that's
| come before you.
|
| Like the author there's been a number of times where I've felt
| self-satisfied completing a piece of technical work, only to
| scratch my head a week later and wonder whether it was worth it.
| I can say with certainty that having a solid CI/CD pipeline has
| saved my bacon a few times so I'd probably recommend that as an
| early investment to make, internal tooling for some of the
| customer registration was a bit of a mistake and not as critical
| as I thought it would be which now feels like a waste of 3 weeks.
|
| I am yet to come up with any meaningful advice to pass on to
| developers in the same position. With time you get better at
| spotting when you can get away with hacking solutions together
| and when to put the time in, having a forward vision for a
| product really helps on focusing your thought and attention and
| time spent on making things visible to the rest of the business
| is rarely time wasted. Awareness of what's going on with the rest
| of the industry is useful, but don't obsess over the new hotness,
| build with what you know works and save your innovation tokens
| for a rainy day.
| waynesonfire wrote:
| I went down this exact same path. Started simplifying everything
| and it did wonders for my velocity. It took me a while to realize
| that we have teams and teams of people maintaining all this crap
| I was trying to spin up myself. It just wasn't worth it.
|
| I went to basic principles, FreeBSD. systemd has taken Linux in
| the wrong direction in my opinion. I don't want anything fancy, I
| don't want to learn another persons resume building framework, I
| want to use technology that has proven longevity.
|
| Our technology space moves so fast, for my personal work, I have
| to leverage the first principle technologies that I have mastered
| over my career. The latest framework may make some problems
| easier to solve, but I don't deploy technology without achieving
| some level of competency. That takes significant time. I'm
| getting wiser with how I spend that time, it's not always easy to
| know.
|
| This process has also taught me why we have this proliferation of
| SaaS solutions. It's tempting, and maybe one day I'll be wise
| enough to realize that this is the route to take. However, I'm
| still able to leverage my core skills and open-source technology
| enough that SaaS solutions don't look attractive to me. I can see
| the benefits either way, so I suppose maybe in five years when I
| reflect I'll know if I made the right investment.
| AlchemistCamp wrote:
| > systemd has taken Linux in the wrong direction in my opinion
|
| Can you elaborate on that a bit? Where does it fall short for
| you and what's your preferred tool?
| nelsonenzo wrote:
| and when he says "you", he means tiny little side projects that
| do nearly nothing.
| dominotw wrote:
| Its not just engineers that want to build complex infrastructure
| though. This a massive misconception and shifting the blame to
| 'nerdy engineers' doing 'nerdy' stuff instead of solving real
| problems.
|
| Startup that worked at raised whole bunch of money and hired tons
| of engineers. They were all tasked to "build something big". eg:
| it was expected of data engineers to setup "big data
| architecture" with complex data-lakes, job schedulers , spark
| clusters, streaming jobs and all sorts of weird stuff. No one
| really understood why we are building all this stuff but we never
| got real answers from the management.
| de6u99er wrote:
| It always depends on the use case and how much vendor lock-in you
| are willing to accept.
| jFriedensreich wrote:
| 3 important factors:
|
| - if i use state of the art production grade technology i can
| talk about it in interviews, if i say oh well i have plenty
| experience running docker compose manually on a server, not so
| much. This is also where new technology can be tested without
| hurting someone, so even if something is totally overkill it can
| still make sense in the overall picture of all projects you work
| on
|
| - ci/cd is maybe not relevant to your users, but its about your
| own sanity, it forces complete documentation in code of what is
| done to build and ship, decouple it from local dev setups and
| make as deterministic as possible. Even small project benefit,
| have you ever come back after 2 years and took hours to remember
| the order of voodoo commands or lost a laptop and then had to
| install specific versions of cli tools?
|
| - the logic is totally changed by the simplicity of faas
| offerings. Nowadays i would maybe not build a gigantic kubernetes
| system, but instead of dealing with vms and servers skip right to
| cloudfare workers or cloud run and have the same benefits of k8
| with even less work and more time to work on the features
| rjbwork wrote:
| >- ci/cd is maybe not relevant to your users, but its about
| your own sanity, it forces complete documentation in code of
| what is done to build and ship, decouple it from local dev
| setups and make as deterministic as possible. Even small
| project benefit, have you ever come back after 2 years and took
| hours to remember the order of voodoo commands or lost a laptop
| and then had to install specific versions of cli tools?
|
| This to me is the biggest advantage of this sort of thing.
| Documentation and notes are great, but I'd rather just have the
| thing-in-itself available that does the job.
| jFriedensreich wrote:
| Yeah and 13 years ago there was a similar argument about git.
| I heard a lot of developers say well its a small project and
| its just me, no way i'm dealing with repos, making commits
| and learn git or mercurial. This completely died out.
| megous wrote:
| I can go back to projects I wrote 10-14 years ago, and the
| same build/deploy scripts still work today. I used standard
| POSIXy or GNU tools back then. They still work the same way
| today.
|
| It really depends on how you pick what you depend on. Would
| some random third party CI/CD infrastructure, I might have
| selected 10 years ago still work or be in business today?
| Hard to say. Maybe I would have picked something that stayed
| supported until this day, maybe not...
| Person5478 wrote:
| CI/CD doesn't require the cloud or containers.
| me_again wrote:
| Perhaps in the resume/interview best to highlight outcomes
| (supported N users with 99.9% availability solo with $x a month
| spend) in favor of tech used. "Evaluated kubernetes and [tech
| du jour]" also gets the keywords in there.
| ummonk wrote:
| I mean, with the amount of time spent doing manual work, you
| often end up spending less time automating it away.
| foobarbecue wrote:
| I find there's a bit of an "if you give a mouse a cookie" effect
| with this stuff.
|
| I recently wanted to organize a data processing pipeline at our
| company. In the past, the system was: there's a bunch of scripts,
| some of them call each other, and sometimes a person logs in and
| kicks off one or more scripts without recording their inputs and
| sending the output to wherever they feel like. I thought
| implementing this on Argo Workflows would allow us to keep track
| of previous runs and generally clarify things.
|
| Well, Argo runs on Kubernetes. Also, it kinda wants your data in
| a bucket. And argo kubernetes wants kustomize and skaffold and or
| / helm and soon I'm installing a package manager that manages
| package managers and I spend half my time managing systems we
| don't really need.
|
| I'm learning I need to to draw the line and deny Argo some of the
| things it wants. No, we don't need to run minio and call our
| files "artefacts." The filesystem works just fine thanks, we
| don't need another layer of it. This means we can't follow some
| of the Argo examples and lose some of its functionality.
| jwr wrote:
| I ran my B2B SaaS on a single (physical) server for the first
| three years or so. I then switched to a setup with three
| production servers, and three staging servers (in a different
| data center), mostly to have the comfort of a distributed
| database, and a staging system to try things out.
|
| Physical servers provide _enormous_ computing capacity these
| days. And managing them isn 't much different from virtual
| servers: you should be using a tool like ansible anyway. The only
| difference is that when I spin up my development cluster I use
| terraform to generate the ansible inventory, and for physical
| servers the inventory is static.
|
| I do not anticipate needing more capacity or scalability anytime
| within my planning horizon.
|
| I keep seeing people investing tons of time and effort into
| things like Kubernetes and I honestly have no idea why. It
| complicates things enormously for no good reason: the benefits,
| if you aren't Google, aren't interesting at all, and the
| drawbacks are huge. The biggest drawback for me (apart from the
| cost of learning, building, and maintaining the infrastructure)
| is the complexity and resulting unpredictable failure modes.
|
| There are things which I'd say are _really_ important for
| infrastructure, though:
|
| * a working and tested backup strategy
|
| * disaster recovery plan (what happens when your entire
| datacenter burns down)
|
| * managing servers via a tool like ansible
|
| * regular testing of various scenarios (including setting up an
| entire system from scratch on new servers, with data recovery
| from backups)
|
| * a database that reliably preserves all of your customers' data
| when any one of your servers goes down (this unfortunately
| excludes many databases and makes the usual "just use Postgres" a
| poor choice, now flame me to high heaven)
|
| I can't imagine a small business where Kubernetes makes business
| sense. Do people seriously have problems with scaling to hundreds
| of machines? Scalability problems with huge traffic peaks that
| actually cause server load? I'm having trouble keeping my CPUs
| busy for any meaningful amount of time, and I don't spend much
| time on optimizations, I just avoid doing stupid things.
| marcosdumay wrote:
| > a database that reliably preserves all of your customers'
| data when any one of your servers goes down (this unfortunately
| excludes many databases and makes the usual "just use Postgres"
| a poor choice, now flame me to high heaven)
|
| You mean as in, backups have the latest data? Every
| transactional DB does that, postgres included. You just have to
| back-up the correct data (what is on the docs, for every one of
| them).
|
| If you mean any DB server can go down at any time and the
| others must take over without losing any data, then your best
| bet is to hire a postgres expert to set it up for you. Both
| Oracle and SQL Server nominally do that, but they have flaws
| that their experts can't correct, and you won't be able to set
| any of them up without acquiring some expertise first.
| mwcampbell wrote:
| > * a database that reliably preserves all of your customers'
| data when any one of your servers goes down (this unfortunately
| excludes many databases and makes the usual "just use Postgres"
| a poor choice, now flame me to high heaven)
|
| What do you suggest instead? My understanding was that
| Postgres, when properly set up with backups and replication, is
| the most reliable open-source database.
| kweinber wrote:
| Here's an oversimplified rule of thumb: Try not to spend more
| than a tenth your time on infrastructure or build if you are
| developing a product. If you find you are spending more than that
| on manual stuff, try to automate etc. to get back below that.
|
| If you are scaling staff, you may want up to a tenth the
| engineering staff to be dedicated to these tasks.... as you scale
| up, more automation to stay at or below that level.
|
| Oversimplified.... but not bad to think about if you think you
| are overdoing it.
|
| If you are way below a tenth.... really think about if you are
| counting all the manual workarounds and pieces of overhead
| honestly....or if you are giving up scalability, reliability or
| flexibility by not controlling your infrastructure.
| twodave wrote:
| > Your goal, when launching a product, is to build a product that
| solves a problem for your users.
|
| This is the main takeaway, in my opinion. I've been at small
| companies who were under-architected "because we can" and large
| companies who were over-architected because someone in the
| leadership chain was enamored with a particular technical choice.
|
| In the end it's a cost curve and depends heavily on the nature of
| your business. I can say personally if the business is even
| mildly complex (not necessarily large, this could mean a web app
| + a mobile app + a single RDBMS) I'd sleep better having some AWS
| (or other cloud) alerts set up to monitor uptime/etc. Others have
| mentioned CI/CD, but again, heavily depends on your needs. I
| worked at one place that survived uploading .zip files to Elastic
| Beanstalk for _years_ to update our PHP website.
|
| For some of the other data compliance/up-time/service
| availability stuff, I'd expect to have the revenue to cover the
| sort of overhead that introduces (otherwise you're just not
| charging enough, or the business isn't really as viable as you
| think).
| nattaylor wrote:
| Maybe it's more important to ask 'why' [do I need this infra]?
|
| Complexity shouldn't be the default but it can be the right
| answer
| Zelphyr wrote:
| Complexity is never the right answer. Complexity, in my
| experience, comes from two things: The person building that
| system doesn't understand the problem well enough and/or
| equates complexity with quality. Neither are good things.
| bovermyer wrote:
| As a "devops" engineer, I can say with total confidence that
| complexity is rarely the right answer. When it is, it's because
| that complexity is the lowest pain point in a calculation that
| has itself become complex.
| agustif wrote:
| Sometimes you can add complexity to the whole stack by adding
| a new shiny layer that brings down complexity where you care
| most about it, in your user-facing UI/UX layer.
|
| Developers can handle complexity via code much more
| efficiently than users operating a site or app...
|
| Also, some of the nice developments of the recent explosion
| of serverless/as a service stuff, is that you can mostly try
| stuff for free, be happy on the free tier if you don't need
| scale, and use that scale when you need it without much
| hassle but paying the overprice of their fat margins.
|
| Also you should probably have an ejection plan in case scale
| + external provider becomes too expensive, you should always
| have a nice idea on how to run your stack if you need it...
| nkotov wrote:
| My team is actually in the process of solving this problem
| ourselves. At the end of the day, k8s is overkill for most
| people, and using the nice UI/UX on top of K8s I feel will have
| devastating effects down the road when you need to troubleshoot,
| manage, and migrate.
| ianpurton wrote:
| I've heard the Kubernetes vs VPS a lot now. I think it's the
| wrong way to think about it.
|
| It's more about a learning curve of best practices.
|
| - Building your application with a CI pipeline is a best
| practice. It helps to ensure your code builds from the repo and
| your tests run.
|
| - Building your app into a container is a best practice. The
| Dockerfile tracks your dependencies as code, and you can version
| control it.
|
| I can tell you from bitter experience these 2 are essential.
|
| Once your app is containerized you have many choices in the way
| you deploy it. You can use PaaS i.e. heroku or azure web apps.
|
| This is orchestration.
|
| If your service has a few containers and maybe some jobs then
| Kubernetes starts to look like a good option. K8 is $50 a month
| on digital ocean, you can run a few projects on that.
|
| Continuous deployment/delivery is a best practice. It takes the
| stress out of releases.
|
| Infrastructure as Code is a best practice. It's immutable
| infrastructure and once you have it gives you so many options in
| the way you manage failover etc.
|
| If you are concerned about uptime during deployments then
| Blue/Green is a great option too.
|
| We're not overcomplicating things, these skills will transfer
| from a side project to a billion dollar corp concerned with
| safety critical software.
|
| So you have to hit a few learning curves, but after that your a
| better developer.
| throw1234651234 wrote:
| There is a fairly in-depth article on which modern deployment
| strategies are useful, and which are a waste of time for mid-size
| companies as of 2020 here:
|
| https://medium.com/swlh/worthwhile-modern-deployment-strateg...
| dang wrote:
| If curious, past threads:
|
| _You Don't Need All That Complex /Expensive/Distracting
| Infrastructure_ - https://news.ycombinator.com/item?id=19358259 -
| March 2019 (121 comments)
|
| _You Don't Need All That Complex /Expensive/Distracting
| Infrastructure_ - https://news.ycombinator.com/item?id=19299393 -
| March 2019 (82 comments)
| Cthulhu_ wrote:
| As always, it depends. Lots of people wish they had Facebook's
| challenges when it comes to scaling. 99% of applications simply
| don't.
|
| I've never seen a company I've worked with go for 'fancy'
| infrastructure because they needed to. It was always because they
| HOPE that they need to. Granted, re-engineering your application
| / architecture for scaling is expensive, but not as expensive as
| building the wrong thing in the first place.
|
| They should all have started off with a monolith and grow from
| there. I mean in all those cases we had just a single,
| 'monolithic' front-end without any problems.
| mring33621 wrote:
| Aspirational Infrastructure!
| durnygbur wrote:
| I don't hear you from inside bastion in autoscaling cluster!
| eric4smith wrote:
| Sometimes a nice dedicated server and Dokku is good enough.
|
| Backups of database offsite, assets in S3 and code in Git means
| you're good for 90% of use cases.
|
| >> git push dokku master
|
| Done.
| fendy3002 wrote:
| I assume that premature infrastructure optimization (complexity,
| complex automation, kubernetes, etc whatever you say) is same
| with premature optimization, both are bad.
|
| However same with optimization, there are some best practices
| that are good to follow, such as avoiding N+1 queries.
|
| For me personally, that will be deploying with docker-compose,
| which usually yml already be created during development, and
| it'll be much easier to migrate later.
| jameshart wrote:
| This is of course a tautology. Premature _anything_ is bad.
| That's what 'premature' means. It means you did it too soon,
| before the optimal time. Doing it later would have been better.
|
| But there _is an optimal time_. And even if you did it too
| soon, if you eventually needed it, at least you've done it.
| Better that than getting to it too late, surely?
| heroprotagonist wrote:
| And then there's the "I've already done it this way a dozen
| times and I'm comfortable with it. There are benefits to it
| and it's not that much extra effort to set it up from the
| beginning and save 10x effort down the line."
| rajatsx wrote:
| That's so true! Not long ago, I wrote an Ansible playbook to
| install my open source software on a DigitalOcean VPS. I devoted
| so much time to learn Ansible as I had zero experience with it. I
| think I almost kept on working on that playbook for almost two
| months. Even after that, I was buggy.
|
| One day, I just woke up and had an epiphany that a BASH script
| could do the same! On top of that, the end users would not have
| to install Ansible in order to run the playbook.
| lukeschlather wrote:
| > Your users to don't care how your code gets onto your server.
|
| I work in B2B and while sure, our users don't necessarily care
| but they also aren't the ones paying us money. The people paying
| us money have a huge set of questions about how code gets on our
| servers and Kubernetes is not everything but it helps a lot
| answering those questions. (Especially using GKE.)
| justizin wrote:
| This article is about a person who needs a brochure. They don't
| even need a Linode VPS.
|
| Also note this starts out by describing a person as making
| "thousands of dollars per month".
|
| If that is your goal, obviously, you should spend as little as
| possible on all of your overhead.
| ic0n0cl4st wrote:
| This past year I've finally been able to hit the $200/hour rate
| as a k8s consultant.
|
| I feel like I'm selling snake oil. I'm working on my 7th
| implementation and _none_ of my customers need kubernetes.
|
| Basically all of their developers and devops engineers want
| kubernetes on their resumes or because it is shiny and what the
| cool kids are using.
|
| My largest implementation was a $3m/year, 9 year old company who
| could run their entire infrastructure on a couple dozen ec2
| instances. However they are under the delusion of "hitting it big
| and needing to be able to scale 100x".
|
| Paid off my student loans this year, and soon my house, thanks to
| superfluous k8s integrations!
| [deleted]
| spaetzleesser wrote:
| "Basically all of their developers and devops engineers want
| kubernetes on their resumes or because it is shiny and what the
| cool kids are using. "
|
| That's what I am doing now. Knowing this stuff makes you much
| more employable and raises your salary. It's a stupid world but
| resume driven development is a rational thing to do and very
| encouraged by hiring practices. For a long time I thought the
| company would appreciate making things simple and cost
| efficient but when I look at the latest hires it's pretty clear
| that the company wouldn't even hire most of the people who keep
| the business running because they are "outdated". You really
| have to keep churning constantly or you fall off the bandwagon.
|
| On the other hand I am pretty new to "modern" server and web
| development and the complexity is just insane. Getting a simple
| API server and frontend going with comprehensive testing,
| CI/CD, containers, dependency injection and whatever is a huge
| effort. It barely leaves you time to think about writing
| efficient quality code because you are so busy with pull
| requests, writing test code and other stuff.
| droopyEyelids wrote:
| You are the real job creator, as there will be more work after
| your migrations than there was before
| jrochkind1 wrote:
| Thank you for this!
|
| What in your opinion is the optimal way to run "a couple dozen
| ec2 instances"?
|
| I'm kind of in that zone.
|
| I don't think "totally manually" works out very well once you
| are in even "one dozen EC2 instances", it's too easy to make
| mistakes, you spend too much time (with downtime) figuring out
| what mistake you made and what it was supposed to be instead.
| Generally too hard to orchestrate changes without downtime.
|
| But is there anything in between totally manual and kubernetes?
|
| I think that's one reason people are reaching for kubernetes,
| they actually _are_ having challenges that need a solution, and
| can 't figure out what the solution is other than kubernetes.
| ic0n0cl4st wrote:
| It's unpopular, but I embrace (AWS) lock-in for small and
| mid-sized infrastructure. AWS' various services solve 99.9%
| of any infrastructure needs. High-bandwidh streaming and
| heavy gpu processing is one of the few places my customers
| find to be cost prohibitive.
|
| For compute I lean towards an immutable infrastructure. I was
| a Puppet / Chef engineer for 8 years, never again. It's
| cleaner to configure apps via some mixture of pre-baked amis,
| docker files and some sort of parameter store. (Usually SSM
| or Vault).
|
| Fargate is a cleaner path but so much more expensive.
| jrochkind1 wrote:
| Thanks for response! Sure, I'm down with AWS lock-in, for
| the sake of argument.
|
| But how do you _manage_ this EC2 fleet and collection of
| other AWS services? Changes happen -- new version of your
| local software, change your mind about some configuration,
| new version of a database, whatever.
|
| OK, it's immutable. But I still need to launch a new EC2,
| and get everyone else that's supposed to know about it to
| know about it and start talking to it instead of the old
| one (or get public traffic to go to it or whatever), and
| then take down the old one once it's safe to do so. (I also
| need to figure out how to construct it as immutable).
|
| Purely manually? Just manual operations in the AWS console?
|
| There may be an obvious answer here I am missing not being
| familiar with the tools. What can be overwhelming here is
| figuring out the "right" way to orchestrate this stuff,
| there are so many options (or sometimes you can't find any
| options but suspect you are not looking in the right place)
| and so many partisans. One reason for kubernetes popularity
| is the succesful awareness marketting, I've heard of it and
| know it is supposed to solve these challenges I'm having
| and don't really understand what the options are and am
| overwhelmed trying to figure it out... but k8 seems
| popular, it must be good?
| watermelon0 wrote:
| Your CI pipeline can build an AMI image with integrated
| Docker image, and a simple systemd script to start it on
| boot.
|
| Deploy step would just be to upgrade AutoScalingGroup
| with the new AMI, in a rolling fashion with
| CloudFormation.
|
| ---
|
| However, in most cases I'd recommend going with ECS +
| Fargate, where you can just deploy your Docker image, and
| don't have to care about the underlying virtual machines.
|
| ---
|
| If you want to go K8s route, EKS + Fargate, (or if you
| want to go cheaper, EKS + EC2 Node Groups), are the best
| way in most cases, but you need to be aware that you will
| have to keep an eye on upgrading EKS versions, Helm
| charts, and other resources you add to the cluster.
|
| I'd say Kubernetes is a good tool if you need it, but
| otherwise I'd start with something that doesn't require
| any maintenance on the ops side.
| ummonk wrote:
| This. I'm not some Unix sysadmin type who wants to do
| everything manually. But the AWS features work out of the
| box for basic deployment, scaling, etc. and Kubernetes
| seems to just add unnecessary operational complexity
| without any value if you don't need any other features
| and are okay with AWS lock-in.
| throwawayzRUU6f wrote:
| Obviously a personal opinion, but Ansible solves this problem
| really well. It does get sluggish once we're talking about
| more than 50 machines, though.
|
| To reliably end up with a working cluster:
|
| - spin up plain operating system VMs
|
| - spin up one more VM for Ansible instigator
|
| - collect the IPs, write them down to Ansible inventory
|
| - pull/upload your custom binaries
|
| - run Ansible playbooks to set the VMs up
|
| Sometimes it's beneficial to have a two-step process. One
| playbook to install rarely changing components. Then make a
| VM snapshot. The second playbook installs your custom
| binaries. Future installations can then start from the
| snapshot.
| deagle50 wrote:
| I've been on the fence about this. If you don't mind, how did
| you get started?
| waynesonfire wrote:
| Oh, I can tell you because I work with these asshats. They
| come in, deploy the technology, usually get promoted and then
| leave. The people left behind have to maintain their garbage
| and because promotions are only given to new feature work
| we're the suckers holding the bag.
| ic0n0cl4st wrote:
| Actually This is exactly how I accidentally started
| consulting on k8s. Three of my customers had engineers who
| advocated for k8s, did 50-75% of an implementation then
| left.
|
| I have told all of my customers they don't need this, but
| they are convinced this is the way of the future.
| deagle50 wrote:
| "Right side of history" is a powerful motivator, for
| better or worse. How did you get the first customer?
| ic0n0cl4st wrote:
| Right side of history? I dunno about anything like that.
| It's just business, show me the money.
|
| There will be a new trend next year or the year after, ad
| nauseum.
|
| I've been consulting on and off for over 20 years now, I
| dont even remember my first 10 customers anymore. I get a
| lot of referrals from customers and from a consulting
| company I sold.
|
| I was primarily focusing on terraform, which is the only
| technology I've actully ever enjoyed professionally, but
| there's just so much stupid $$ in k8s.
| ideamotor wrote:
| Better delete this ;)
| b0afc375b5 wrote:
| Is this one of those rare instances when honesty _isn 't_ the
| best policy?
| statstutor wrote:
| Doubtful. The consultant who is willing to say that
| straight (and with the experience to know it) is going to
| be worth a lot more than $200/h, to the right clients.
| ummonk wrote:
| I rarely see instances where kubernetes actually makes sense
| over just using regular AWS services (load balancing,
| autoscaling, AMIs, etc.) directly. It seems like k8s just adds
| a mostly superfluous extra layer of abstraction with a rather
| user-unfriendly interface.
| salmonlogs wrote:
| Firstly - congrats on making great income by providing valuable
| skills to a hot market.
|
| When you say "none of my customers need kubernetes" is that
| because they are not using any of its benefits, or because they
| could do it another way and kubernetes was one of the options
| rather than the only option?
|
| In my experience very few people are fully exploiting
| everything k8s has to offer, but many people are getting side
| benefits from greater standardisation of deployments,
| consistency for implementation of logging, tracing and
| telemetry, reproducibility of environments and to an extend
| talent retention and development.
|
| I'm very interested to understand where the line is for
| genuinely getting zero benefits of using k8s.
| licebmi__at__ wrote:
| I'm in your position, but in my opinion, there are plenty of
| who are better in kubernetes even if they don't need it. There
| is complexity in the processes and even the scripting people
| use to glue things together, and there are enough places for
| the team to screw up.
| jameshart wrote:
| I'm confused. Isn't a kubernetes cluster just a way to deploy
| software onto a couple of dozen ec2 instances?
|
| Sure it has some accidental complexity, but so does having half
| a dozen different software systems each managing their own
| deployment pathway onto ec2.
|
| There seems to be some belief that kubernetes has to be some
| kind of crazy complex system to operate, when really it's just
| an api for managing which docker processes get spun up.
| 0xbadcafebee wrote:
| > Isn't a kubernetes cluster just a way to deploy software
| onto a couple of dozen ec2 instances?
|
| Sure. And a nuclear power plant is just a way to generate
| electricity.
| arcticfox wrote:
| I am always baffled by this take. I run k8s because it's by
| far the easiest way to do what I want (deploy a handful of
| services to a handful of servers), not because it's cool or
| massively powerful.
|
| In the past few years, k8s simply got easy to get started
| with and it feels like no one noticed.
| BlargMcLarg wrote:
| Most tools are easier than people make them out to be. The
| average dev just isn't that great, and the average company
| overestimates the difficulty of these tools, while burying
| their devs with mindless drone work so they never get the
| time to study these things. The average dev isn't one which
| has studied CS/software dev and/or spends loads of free time
| still investigating tech off the clock, either.
| auganov wrote:
| I'd assume anybody criticizing kubernetes isn't a huge docker
| fan either. I see both used much more as dev ops tools rather
| than real cluster management.
|
| Particularly questionable in organizations that may not have
| any in-house tech expertise. Some contractor, who may be the
| only programmer involved at a given time, comes in and sets
| everything up the kubernetes/docker way. From now on you'll
| always need kubernetes-enabled devs to maintain your thing.
|
| Happened to somebody I know. They believed they had a
| "cluster" with different "servers" but it was a single
| machine. Then I had to explain why they cannot just FTP into
| the server, upload plugin folders (like some tutorial asked)
| and expect it to work. Nor did I have enough time to help out
| since who knows how these dockers mount folders, what's the
| latest "best practice" or what the dev who maintains it
| prefers. There's no way they'll ever need more than one
| machine. I'm pretty sure the system in question does not even
| have a good cluster story in the first place, so at best you
| could scale the database (which usually can do clusters well
| without 3rd party tools).
| throwdbaaway wrote:
| > when really it's just an api for managing which docker
| processes get spun up
|
| On GCP, the goal can be achieved without kubernetes, by
| deploying containers on VMs and MIGs:
|
| https://cloud.google.com/compute/docs/containers/deploying-c.
| ..
|
| Comparing that to kubernetes, the latter is indeed some kind
| of crazy complex system to operate.
|
| The same can also be done either on GCP with non-COS images
| or on AWS, with a little bit of automation:
| https://news.ycombinator.com/item?id=26274033
| jameshart wrote:
| Right. I said k8s is _a_ way. Not _the_ way. There might be
| others. They might have their own trade offs.
| thisiszilff wrote:
| > Comparing that to kubernetes, the latter is indeed some
| kind of crazy complex system to operate.
|
| I think the fair comparison here would be that to google's
| kubernetes engine. Getting a kubernetes cluster up and
| running in GCP is really easy.
| kkoncevicius wrote:
| I don't think this kind of attitude should be encouraged.
| Imagine other professions celebrating things like that.
|
| Surgeon: Some of my patients request bladder surgeries while in
| reality their problems could be solved with a few simple diet
| supplements. But I just do the procedure and take the money.
| Just bought my 3rd Tesla, thanks to superfluous bladder
| operations!
|
| I know selling people shit they don't need is celebrated as an
| achievement nowadays, but feels like it shouldn't be.
| MrPatan wrote:
| I'm not sure I need to engage my imagination very far. Why do
| you think that's not exactly what happens?
| bombcar wrote:
| Tons of work done in various industries is "shitwork" that
| the person doing the work wouldn't normally do except they're
| being paid to do it. At some point you give up arguing with
| the customer.
|
| The dieting is a perfect example - many health issues can be
| solved by losing weight but few people want to make the
| lifestyle changes necessary to do that.
| foobarbecue wrote:
| Is your "shitwork" related to "bullshit jobs"?
| https://en.wikipedia.org/wiki/Bullshit_Jobs
| jnwatson wrote:
| But that's exactly what surgeons and accountants and
| mechanics and consultants and entertainers do. Most work is
| unnecessary. Once you embrace that, it is a lot easier to
| understand how this economy works.
| kavalg wrote:
| I can confirm this. I was offered surgery for my 2 year old
| for a problem that is not life threatening and could
| possibly (but rarely) cure itself. In the end it indeed
| cured itself.
| Zelphyr wrote:
| It already happens in medicine. Of course, a lot of it is
| "I'm afraid of getting sued so I'm going to cover my ass with
| all these tests." but, still.
|
| When I was younger I had a pain in my abdomen I had never
| experienced before. My grandmother told me to go to their
| family doctor who was probably older than they were. He
| pushed on some things and tapped on some things and finally
| diagnosed me. "It's gas. Go home and have a big fart." Sure
| enough, hours later, a big release and the pain was gone.
|
| These days I'd have been sent in for at least an MRI or CAT
| scan to go along with all the blood work that they would do.
|
| Now, I know what some of you are going to say. "But it COULD
| have been something way worse!" Sure, it could have been. But
| this guy had been a doctor for a long time by that point and
| knew from experience that it was gas. Just like when I
| recently had to rebuild a really old server and was getting
| errors that suggested missing class files but, after 25 years
| of doing this, I knew that it was actually missing a PHP
| module.
|
| Less is more and experience is golden.
| rixrax wrote:
| Sounds exactly like the medical industrial complex in the US.
| Opioid crisis and general over medication and over treatment
| just magically happened / keeps happening. Surely no doctor
| ever thought let's give the customer what they are asking.
| jrochkind1 wrote:
| Most people keep themselves from recognizing the problems
| with what they are doing if they are making lots of money off
| it, they convince themselves what they are doing is
| worthwhile after all.
|
| "It is difficult to get a man to understand something when
| his salary depends upon his not understanding it."
|
| I don't think (I hope?) anyone is celebrating "cheating the
| rubes". But I respect GP's ability to be honest about it --
| it's super useful to the rest of us. For instance if we've
| been skeptical of k8, seeing someone who's an expert in it
| say that makes us think ok maybe we're on to something.
|
| Most people in that position would keep their mouths shut,
| and probably keep themselves from admitting to themselves
| what's up.
| ppf wrote:
| How many of the (mostly web-based) technologies that we all
| love to discuss on HN are any different?
| dqpb wrote:
| His reason for why they don't need it was they are 'under the
| delusion of "hitting it big and needing to be able to scale
| 100x".'
|
| To be fair, as a kubernetes consultant, he's probably not
| qualified to determine whether or not they will hit it big.
|
| What he probably is qualified for is to determine if they
| were to serve 100x their current load, would they be well
| served by kubernetes.
| ic0n0cl4st wrote:
| Nothing I do in this world is life or death. Hell, my
| customers neither, mostly they sell or support advertising in
| one fashion or another.
|
| Some founder I work for might make billions on products
| people don't need, why shouldn't I make $50k off of him?
| Isn't that how capitism works?
| throwawayboise wrote:
| I think so. If someone sees enough value in what you
| provide to be willing to pay you, you shouldn't feel bad
| about it. It's not like you're pushing them to adopt k8s.
| They already did that.
|
| It's like someone bought a Mercedes. Nobody needs a
| Mercedes, they could drive a Toyota. But now they need an
| oil change, and that's what you do, and they'll pay you for
| it.
| jaywalk wrote:
| What do you think cosmetic plastic surgeons do?
| BruceBlacklaws wrote:
| The logic behind this article is just bullshit
|
| It's not possible to operate at our scale "using a single Linode
| VPS" or whatever that may mean
| 0xbadcafebee wrote:
| Whether the infra is complex or not isn't going to make a huge
| difference to whether your product makes money. Some of the
| highest-revenue-generating products I have seen were _tire fires_
| under the hood. One product had multiple people working 24 hours
| a day to constantly hose down the multiple levels of problems
| that the incredibly poor design and terrible infrastructure
| created. But through the toil of those people in the backend, the
| product made hundreds of millions a year.
|
| Whether or not you use complex technology, it matters more
| whether it's well designed and well operated. If it's not, you
| will end up with a fire either way. It's just that "simple" fires
| are a lot easier to put out than "complex" ones. (this is
| literally true; a lot of fires should not be put out with water)
| [deleted]
___________________________________________________________________
(page generated 2021-03-11 23:01 UTC)