[HN Gopher] AWS staff spending 'much of their time 'optimizing c...
       ___________________________________________________________________
        
       AWS staff spending 'much of their time 'optimizing customers'
       clouds'
        
       Author : MISTERJerk2U
       Score  : 178 points
       Date   : 2023-04-17 16:57 UTC (6 hours ago)
        
 (HTM) web link (www.theregister.com)
 (TXT) w3m dump (www.theregister.com)
        
       | gumballindie wrote:
       | Of course. AWS apps are a mess and needlessly over complicated.
       | Even if sales went on they'd still spend more time untangling the
       | mess they've made for their clients.
        
       | syrusakbary wrote:
       | TL;DR: they want to rip you off less aggressively, so you don't
       | ditch them on the short term.
       | 
       | Cloud computing has become so expensive that so many companies
       | are moving away from it. And for good reason.
       | 
       | The only providers that are actually reasonable on pricing at the
       | moment are OVH and Hetzner... by an order of magnitude (or two)
       | cheaper than Google Cloud, AWS, Azure, Vercel, etc.
       | 
       | Trusting for any of the latter companies to not rip you off is
       | like trusting a wolf to take care of the sheep herd. It just
       | doesn't make any sense, as their incentives are many times at the
       | opposite spectrum than their customers.
        
         | hot_gril wrote:
         | > Cloud computing has become so expensive that so many
         | companies are moving away from it
         | 
         | Net? Sales figures don't show this.
        
         | 0zemp1c wrote:
         | I expect almost everyone who tries to rebel and go "on prem"
         | will flame out and come running back to the cloud
         | 
         | Ops talent has really dumbed down over the last decade or so
         | (sorry no polite way to say it). Long gone are the days when
         | the ops engineers were talented devs who just liked hanging out
         | at datacenters. Nowadays most of them are just button pushers
         | who are good for standing up SaaS stacks or other simple tasks,
         | but I'd never ask any I've dealt with in the last ten years to
         | build out a datacenter presence from nothing. Most have never
         | set foot in a datacenter. On top of that, most modern ops teams
         | have been downsized (thanks to cloud) that they wouldn't have
         | the hands to get the job done.
         | 
         | Even for good ops people, it can be very hard to capacity-plan
         | and understand what appropriate/affordable/useful hardware is.
         | Most ops folks today have probably never purchased servers for
         | production deployments.
         | 
         | AWS will be happy to welcome these companies back after their
         | on-prem dreams die.
        
           | zer0tonin wrote:
           | Ops talent didn't disappear, it just got centralized in
           | Amazon/Google/Microsoft. Now that those companies are laying
           | off thousands, it is likely that competent ops people with
           | data-center experience will end up on the job market.
        
             | scarface74 wrote:
             | Standard disclaimer: Andy Jassy is my skip * 7 manager.
             | 
             | He has said plenty of times in public statements that less
             | than 5% of all cloud spend is on any cloud provider.
             | 
             | https://accelerationeconomy.com/cloud/amazon-shocker-ceo-
             | jas...
        
           | delfinom wrote:
           | >Ops talent has really dumbed down over the last decade or so
           | 
           | Counterpoint, the ops required for your own infrastructure
           | has greatly dumbed down.
           | 
           | It might surprise you, but OEM hardware solution have greatly
           | evolved and simplified the same.
           | 
           | Now I can setup a core switch running 100 GBs with failovers
           | by just plugging in some cables between them, setting a flag
           | and they immediately start replicating config between
           | themselves.
           | 
           | >Even for good ops people, it can be very hard to capacity-
           | plan and understand what appropriate/affordable/useful
           | hardware is. Most ops folks today have probably never
           | purchased servers for production deployments.
           | 
           | I'll shock you once again, the same way AWS has sales and
           | support departments that help spoonfeed what you need.
           | 
           | So do the hardware OEMs. I can get complete solution
           | walkthroughs with Dell for example.
        
             | 0zemp1c wrote:
             | > Now I can setup a core switch running 100 GBs with
             | failovers by just plugging in some cables
             | 
             | no ops person I have worked with in the last ten years
             | knows what a switch is, and this is over many companies
             | both startup and bigtech
             | 
             | no different for storage, compute...the experience and
             | aptitude gap at most companies, even tech companies, is
             | profound
             | 
             | ops has become devops, netops is no longer in their
             | skillset
        
           | sfink wrote:
           | Counterpoint: I think that just means that on prem needs to
           | close the tooling gap some. Cloud was a centralized point of
           | leverage where improved tooling became worth investing in,
           | but now we're at a point where cloud is vulnerable to
           | somebody following in their footsteps by making on prem not
           | require as much tending.
           | 
           | You still won't be able to do it with idiot monkeys. You just
           | have to enable a small team of strong people. It may be
           | getting harder to find those people, but it's not so bad if
           | you only need a handful of them.
           | 
           | https://oxide.computer/ is shooting for this space, from what
           | I can tell?
        
         | beebmam wrote:
         | Eh, it depends. I've seen such inefficient use of cloud
         | providers. Really depends on what cloud users need to do.
        
       | micromacrofoot wrote:
       | are there enterprise grade products that don't end up spending a
       | large amount of time providing enterprise services?
        
       | epberry wrote:
       | The original promise of the cloud was that it was cheaper than on
       | prem. For many companies this has turned out not to be the case.
       | Having worked on a cost optimization product for the past year
       | (https://vantage.sh) my belief is that the cloud _is_ cheaper
       | than on prem but it is difficult to configure it to be so.
       | 
       | In particular it is difficult and stressful to understand all the
       | knobs to turn within each cloud provider. It's a very AWS thing
       | to put a lot of effort into lowering their customers' bills and I
       | do think it makes for longer lasting business relationships.
        
         | intelVISA wrote:
         | I genuinely struggle to imagine (realistic) scenarios where
         | cloud is cheaper but I am definitely interested in where that
         | would be the case.
        
           | yashap wrote:
           | I think you're almost always gonna pay far more for cloud
           | compute than on-prem. It's pretty standard for companies to
           | have roughly 2x the "peak" load than "minimum" load, so while
           | with on-prem you maybe have to over-provision by ~1.5-2x,
           | that's greatly outweighed with how much more expensive cloud
           | compute is vs. on-prem compute. The only exception is
           | companies with extremely spikey demand - e.g. if your peaks
           | are more like 10x or 100x your troughs, then yeah, cloud is
           | probably cheaper from a pure compute POV.
           | 
           | However, unless you use HUGE amounts of compute, cloud is
           | probably the right choice, because you probably save a tonne
           | on salaries by going the cloud route. It's way easier to
           | properly operate a managed DB than your own (especially in
           | terms of no-downtime upgrades, no-downtime scaling,
           | backup/recovery, etc.), same goes for managed file storage
           | (S3 type things), same goes for managed pub/sub, same goes
           | for managed K8s/whatever, same goes for managed load
           | balancers, etc. For a lot of startups, total cloud spend is
           | equivalent to just a few fulltime Engineer salaries, and they
           | also use a tonne of cloud services, getting that all running
           | well without AWS (or Google Cloud, Azure, whatever) would
           | take way more than a few fulltime Engineer salaries.
           | 
           | FWIW, the last company I worked at (~1000 people), as our AWS
           | bill was getting into the millions annually, we tried
           | migrating off AWS, onto a colocation setup. I didn't work on
           | that project myself, but my understanding is they spent a
           | bunch of engineering effort on it, were still nowhere
           | remotely close to being able to truly move off AWS,
           | ultimately canned the project and moved all the colo stuff
           | back to AWS. I think a lot of engineers underestimate how
           | much effort it is to provide the kind of cloud services
           | Amazon/Google/Microsoft provide, at comparable levels of
           | reliability and ease of use. It's A LOT of effort.
        
           | gopalv wrote:
           | With fixed hardware, you're always planning for max
           | workloads.
           | 
           | The systems are scaled up for the "Black friday" sales event.
           | 
           | With cloud, you will plan for the steady state and just boost
           | it up when you need to.
           | 
           | The anti-pattern is usually that when on-prem hits 80% load,
           | some engineer is usually tapped to weed out all the slow code
           | to push past procurement delays.
           | 
           | While in cloud, people just throw more hardware at it and
           | ignore low hanging performance problems which are costing
           | them money (& unlike the on-prem, fixing it will immediately
           | reflect in the budget the next day).
        
           | VirusNewbie wrote:
           | Four jobs ago I worked on a search engine for a large
           | national sales company. They'd sparingly ran TV ads. When
           | they ran a TV ad their traffic would 10x immediately then a
           | mild smooth increase over the next month or so. Would it have
           | made sense for them to operate on prem machines that could
           | handle their peak capacity, when they only needed their peak
           | for hours per week?
        
           | userbinator wrote:
           | When you need a lot of computing power for a very short time.
        
           | colechristensen wrote:
           | Any use case that has
           | 
           | A) a user base with usage that varies significantly over time
           | 
           | and
           | 
           | B) an architecture that can actually scale significantly up
           | and down during the same time period
           | 
           | So if you have lots of web back ends and business logic and
           | general workers for jobs that can just be shut down when
           | there's no capacity needed, there's a ton of savings
           | available in the cloud.
           | 
           | If you have a big monolithic architecture or some other
           | organization where everything is just on all the time, the
           | cloud makes less sense as a cost savings.
           | 
           | I once wrote an internal chat bot which was used less than 2
           | seconds per day, but was actually supremely useful as it
           | really simplified some process or another and took a fraction
           | of a second to run each time. The _actual_ cost to run this
           | in the cloud was approximately 1 cent per year. If I didn 't
           | have this cloud capability it would be hundreds of dollars
           | per year to host or thousands of capital in machines and
           | maintenance to host myself.
        
           | joshstrange wrote:
           | I have a realistic scenario:
           | 
           | I write event (think food festival) software. 9-10 months out
           | of the year there is almost zero traffic, 1-2 months have
           | mild traffic, ~2 weeks are higher traffic, and then 1-5 days
           | (during the event) are very busy.
           | 
           | I'd be hard pressed to find anything other than my current
           | stack (Lambda/DynamoDB/S3/Route53/APIGateway) that costs as
           | little. In the off-months my costs are ~$5 if that and in the
           | month of the event that might get as high as $15-30. I cannot
           | imagine being able to host it locally/colocated (for the
           | month of the event) for that cheap and that doesn't even
           | touch on the hardware cost (I'm thinking just electricity and
           | co-locating costs).
           | 
           | Yes, I'm a small fish and this is just my side business but
           | for me the cloud is way cheaper. Anything with very
           | spiky/uncertain load can be good candidate for the "cloud"
           | but a big part (IMHO) of making the cloud work is using the
           | managed services and using them smartly. If all you do is
           | spin up EC2 instances then no, the cloud is probably not
           | going to be cheaper in the long run.
        
         | notyourwork wrote:
         | > my belief is that the cloud _is_ cheaper than on prem but it
         | is difficult to configure it to be so.
         | 
         | It takes discrete effort in the core planning of your
         | architecture. You cannot look at AWS resources (or any other
         | cloud provider) as an all-you-can-eat buffet. You have to look
         | at your requirements, look at the viable options that may
         | support your requirements and take into consideration cost
         | associated.
         | 
         | Without looking at cost during design your company/service/team
         | will have a bad time at some point. There are also some really
         | specific things you can do to lower costs but require knowledge
         | and understanding of a myriad of technologies.
        
           | pwarner wrote:
           | you can and should look at them as aN "as little as you can
           | eat buffet" option. the cost transparency is a huge part of
           | the value of cloud, even if it cost the same as on prem, the
           | transparency is huge, but it's even more valuable if you make
           | smart choices with that transparency.
           | 
           | of course often folks just ignore costs until al of the
           | sudden it's a huge pain point.
        
         | rodgerd wrote:
         | > The original promise of the cloud was that it was cheaper
         | than on prem.
         | 
         | As soon as it became clear that this was not the case, the
         | selling point swivelled smoothly to "enablement" and "agility"
         | without missing a beat.
        
         | acdha wrote:
         | I do agree that it's a shrewd move to build relationships but I
         | also don't think that it's clear cut that companies are
         | spending more to be in the cloud versus the cloud costs being
         | much easier to see compared to on prem TCO, especially when it
         | comes to staff time or the costs projects take on when they're
         | limited to the services their IT department is capable of
         | building and operating.
         | 
         | A huge confound here is that in any environment these kind of
         | metrics are proxies for a lot of cultural health, so it's easy
         | to find places hemorrhaging money on AWS without having an easy
         | way to know whether the same managers would have wasted
         | comparable sums internally because e.g. the root cause was
         | letting Accenture send a bunch of 25 year olds to design
         | whatever adds the most expensive terms to their resumes, so
         | while it was AWS today a decade ago it was a $10M Hadoop
         | cluster holding one MacBook Pro's worth of data.
        
         | deanCommie wrote:
         | So long as Software Engineer timing is not respected or tracked
         | (compared to other engineering disciplines), in times of
         | economic downturns, it will always be popular to shit on cloud.
         | 
         | People stop caring about the elasticity, about the operationial
         | overhead to their people, additional 9's of availability, even
         | about feature delivery velocity.
         | 
         | Tech companies turn into accountants looking at X $/Month in
         | the cloud vs Y $/month from "buying a couple of servers".
         | 
         | Then you have "luminaries" like DHH and Elon blogging about how
         | much leaner and better their services are as a result (never
         | mind that twitter is now a buggy mess where everything is
         | eventually consistent and breaks half the time)
        
       | echelon wrote:
       | Use vanilla products (K8S, "RDS", ...) so you can always dangle
       | the threat of going on-prem or to another cloud provider.
       | 
       | Once you use Lambdas and other deep AWS product, you're locked
       | in.
        
         | tyingq wrote:
         | I don't personally feel like Lambdas themselves are the hard
         | lock-in. It's the stuff around them that really gets into your
         | code or forces specific patterns. SQS, SNS, Dynamo, Kinesis,
         | EventBridge, etc.
        
           | throwawayx35fw wrote:
           | In my opinion, SQS shouldn't be on that list.
           | 
           | You save a lot of time and resources just using SQS instead
           | of setting up your own servers. And the lock-in is not as
           | problematic as Lambdas etc.
           | 
           | It doesn't require much code change to move to an alternative
           | message queue (RabbitMQ etc.), so the lock-in is minimal. In
           | my projects, I just need to change a few lines of
           | configuration code and the actual message processing code
           | should remain the same.
        
           | preommr wrote:
           | Which frankly are the essential parts of a robust cloud
           | architecture. Deploying a server is trivial - just dump it on
           | any kind of compute resource. Managing fault-tolerance and
           | monitoring, etc. between those services is the tricky part.
        
             | mschuster91 wrote:
             | IME, not if you're using Terraform with proper
             | modularization and can stick to the commonly available
             | feature set. Have one layer of modules that describe what
             | you _want_ (e.g. a virtual server, database, monitoring)
             | and behind that a separate layer that implements it (e.g.
             | creates an EC2 instance, an RDS database or a CloudWatch
             | alert).
        
               | Karunamon wrote:
               | That's a nonstarter just because of how you describe the
               | various services at each cloud provider. For instance,
               | Google cloud is big into using "projects" as organizers
               | for resources, where the closest analogue in AWS land is
               | an entirely separate account.
        
               | scarface74 wrote:
               | Terraform is not "cloud agnostic" anymore than using
               | Python with the various cloud providers SDKs.
               | 
               | Each provider within TF is tied tightly with the cloud
               | provider.
               | 
               | Standard Disclaimer: yes I work at AWS and I do IAC with
               | the CloudFormation or TF depends on what the customer
               | wants
        
         | roncesvalles wrote:
         | I don't see much difference in vendor lock-in between EKS and
         | Lambda. The abstraction between your Kubernetes application and
         | the underlying platform isn't airtight.
        
         | [deleted]
        
         | master_crab wrote:
         | You could use containerized lambdas. Your application logic
         | would be mostly safe to migrate around as need be.
         | 
         | But then again, migrating from one cloud to another is always a
         | heavy lift no matter how it's constructed.
        
           | vidarh wrote:
           | Having done zero downtime migration from AWS to Google to
           | managed servers (chasing credits, and then moving to Hetzner
           | when the credits ran out): it's not much heavy lifting if you
           | treat each cloud provider as just providing raw compute and
           | storage resources.
        
             | jamsauceismad wrote:
             | It's because you did the heavy lifting up front with how
             | your application is designed and how your infra is
             | provisioned (I'm guessing everything is terraformed or
             | something?).
        
               | vidarh wrote:
               | Everything in Docker containers + a service discovery
               | layer (Consul on that particular case) will get you most
               | of the way there as long as you don't get seduced into
               | relying on cloud specific services.
        
               | scarface74 wrote:
               | And then you're managing Consul clusters (been there done
               | that).
        
               | sleepybrett wrote:
               | Also a great unlock if you do the bulk of your work in
               | kubernetes. Amazon is certainly doing some things in
               | their kube implementation that will make moving suck more
               | (karpenter, some of their other integrations) in an
               | effort to be better and thus sticky... But it's still
               | pretty easy to go from one kube cluster to another.
        
           | croddin wrote:
           | With the lambda web adapter[1], you shouldn't need to make
           | any code changes for web projects, just some dockerfile
           | changes which are only used if it is running as a lambda, so
           | the same image should still work on ecs or another cloud.
           | [1]https://github.com/awslabs/aws-lambda-web-adapter
        
         | hnlmorg wrote:
         | In my experience writing cloud agnostic infra usually ends up
         | more costlier. Not just in monetary terms running the
         | infrastructure, but also in terms of time spent maintaining the
         | infrastructure.
         | 
         | There's generally very little point using the cloud if you're
         | not going to use cloud services.
        
           | aeyes wrote:
           | If all you need is machines then there is still a massive
           | benefit in using the cloud. You can spin up 1000 machines
           | tomorrow without having to buy them, wait for getting them (2
           | months later), rack them, cable them, ....
           | 
           | Especially if you need a couple of racks things get
           | complicated quickly. If you have never racked machines for a
           | week you probably underestimate the amount of time it takes
           | before you can actually run workloads on them.
           | 
           | And if you need a 1000 machines for x years which otherwise
           | you'd put in your own data center you can absolutely go to
           | AWS, GCP, OCI and bid them down against each other.
        
             | hnlmorg wrote:
             | > If all you need is machines then there is still a massive
             | benefit in using the cloud.
             | 
             | "massive" is an overstatement. "some" would be more
             | accurate. And your example is correct, which is why I said
             | "generally" in my post rather than "always" or "never". In
             | fact my current project is using AWS in exactly the way you
             | described.
             | 
             | > If you have never racked machines
             | 
             | I have done. Many times. :)
             | 
             | And there are ways you can get machines at short notice
             | without going down the AWS route. In fact it's actually
             | pretty easy and used to be really common. I used to use all
             | sorts of different VPS and dedicated hosting for personal
             | projects (while managing on prem stuff professionally).
             | These days a lot of those bigger companies are now just
             | wrapped up as part of "cloud services" but there are still
             | some smaller providers out there doing things old school.
             | In fact I have a few GFX cards for machine learning in a
             | London data centre managed by one such independent
             | provider.
             | 
             | > And if you need a 1000 machines for x years which
             | otherwise you'd put in your own data center you can
             | absolutely go to AWS, GCP, OCI and bid them down against
             | each other.
             | 
             | You're now talking about cost optimisation and commitment-
             | based discounts, which is a whole other tangent.
             | 
             | Anyways, my point wasn't that the cloud is more or less
             | expensive than on prem. it was that designing your cloud
             | architecture to be cloud agnostic is more expensive and
             | rarely worthwhile compared with making use of the services
             | offered by whichever cloud provider you're using.
        
         | [deleted]
        
         | sarathyweb wrote:
         | If possible can you suggest me a lof vanilla products. My team
         | is building a platform that allows anyone to spinup instances
         | on any cloud platform. We're planning to add support for DNS
         | management ( Route53, Google cloud DNS& Azure DNS) and storage
         | buckets
        
           | potta_coffee wrote:
           | Just what we need, another platform.
        
       | clarkdale wrote:
       | The title has two grammatical errors with quotes. The corrected
       | form would be:
       | 
       | > AWS staff spending 'much of their time' optimizing customers'
       | clouds
        
       | omneity wrote:
       | 1. At which point does it become cheaper for AWS to offer
       | discounts to customers rather than optimizing their spend?
       | 
       | 2. At which point does it become cheaper for AWS to just simplify
       | their pricing structure?
        
         | belthesar wrote:
         | Optimizing spend for a customer is also optimizing capacity,
         | which is indeed finite. By optimizing spend with right-sizing
         | and such, they lose a very small amount upfront, but can
         | increase customer density, which is a much more sustainable
         | data point for modeling capacity growth. More customers on the
         | platform is more opportunity for larger growth at a sustainable
         | pace that they can scale capacity for.
        
       | alex_lav wrote:
       | If AWS wants to stop doing this, AWS can prioritize making
       | optimizations easier for the customer to do...
        
       | no_wizard wrote:
       | Now if only they'd spend time optimizing their management APIs
       | and SDKs so they were more coherent.
       | 
       | I do not like using the console or having to login to the web
       | portal, but there are things you just can't do from the SDK, or
       | at least, I haven't figured out the right way to do them yet.
       | Entirely Byzantine.
       | 
       | I wonder if they'll ever get around to addressing the mountain of
       | complaints folks have about local dev and emulators too.
        
         | sigjuice wrote:
         | Could you share examples of things that can't be done from the
         | SDK? Thanks!
        
           | no_wizard wrote:
           | Its a mixture of hard to use and things seem to not work as
           | expected.
           | 
           | For instance, I want to do multi region deploys for lambda, I
           | can't just do a simultaneous push to deploy across regions.
           | I'm expected to use this messy switch[0] between the CLI and
           | the CDK to do anything useful. This is a tiny example.
           | 
           | Not to mention the CDK packages have _no_ comments in the
           | shipped files (at least for typescript) so I can 't even
           | introspect the typings and try to understand the purpose of
           | things. The documentation is _atrocious_.
           | 
           | Consider instead Firebase, Supabase or Fly.io, which are
           | _dead simple_ to use.
           | 
           | Developer Experience needs a major overhaul. It should be as
           | easy or easier as Supabase or Firebase. No, _Amplify is not a
           | solution, its too buggy and has zero margin for
           | customization_ plus, its local dev experience is quite bad
           | 
           | It'll probably never happen though, because developers are
           | just end users, not the true customer, the buyer (e.g.
           | management) is. I think that's the real reason why these
           | improvements languish
           | 
           | [0]: https://aws.amazon.com/blogs/compute/deploying-aws-
           | lambda-la...
        
       | DogTweezers wrote:
       | [flagged]
        
       | Anduia wrote:
       | This is more of a side effect of providing self-support
       | solutions, and a trend in Customer Support departments. Amazon
       | Support has been on top of the lists and awards for best support
       | because they have proper KPIs that incentive customer
       | retention/satisfaction over just number of contacts resolved.
       | 
       | So when you have good support/sales staff and most contacts are
       | solved with self-service, what remains are contacts about
       | complicated cases and inquiries.
       | 
       | I imagine that they always had a lot of contacts about optimizing
       | spending, and policies to help and retain those customers
       | accordingly, and now the percentage of time spent in those is
       | higher, probably because the other types of contacts are now
       | self-service.
        
         | crabbone wrote:
         | AWS doesn't even have a way to report a bug... what "on top of
         | the list" are you talking about? Who else is on that list?
         | 
         | AWS support is atrocious, unless you cozy up to them somehow to
         | get special treatment. If you are a "regular" customer, you get
         | zilch. AWS offers best performance for the best price, that and
         | they are the largest service provider is what makes them
         | succeed. They don't need to spend anything on support -- it's
         | not cost effective.
         | 
         | My personal experience: we provided a service through AWS
         | marketplace related to storage. So, we are not your "regular"
         | customer, had to pay them some for accreditation... and here's
         | the thing: it's probably something about how we create EBS
         | snapshots, but about every thousand or so snapshots creation
         | stalls. Doesn't fail, doesn't succeed, just hangs forever.
         | There's nobody to talk to in AWS about this. People who
         | "certified" us (what a joke...) cannot do anything about this
         | problem.
         | 
         | And there's more stuff like that. Sometimes some resources
         | won't delete first 1-2-3 times you try. Some other resources
         | are created, but don't work once in a blue moon. And all you
         | can do about it is just curse and try again.
        
           | PreachSoup wrote:
           | A few years a ago when I worked at a tiny startup(<5) we had
           | direct contact with aws. Not sure if this has changed now
        
             | aeyes wrote:
             | Same experience here, they were even hooking us up with
             | credits. Even when we were tiny they wanted to understand
             | how we used AWS and were available to help with PoCs for
             | products we weren't familiar with.
             | 
             | I interviewed many people in my life and every once in a
             | while a candidate would be an AWS Solutions Architect. To
             | my surprise it turns out that much of their time is spent
             | helping customers implement their cloud workloads.
             | 
             | At $BigCorp when stuff wasn't working I had regularly had
             | AWS bite their nails out trying to understand what was
             | wrong until we got to a working solution.
        
           | evantbyrne wrote:
           | I built a CD service on AWS and I've never had issues with
           | their support. Their automated systems sometimes cause
           | unnecessary support incidents though, and waiting in the
           | default support queue could easily cost a day of downtime for
           | someone. Definitely gives me the impression that they try to
           | steer people towards paying for support by forcing
           | unnecessary interactions.
        
           | neximo64 wrote:
           | AWS has some of the best support out there. You don't even
           | have to 'cozy up'.
           | 
           | It sounds like you didn't use AWS support but you got AWS
           | certified consultants.
        
             | blarghyblarg wrote:
             | Some of the higher tiers of AWS enterprise support are
             | INCREDIBLE. They're close the level of support I'm
             | contracted to provide as a full time IC. Stuff like
             | turnaround within hours? 30 minutes for critical issues? It
             | just sounds like they're not a huge account.
        
               | Retric wrote:
               | Their point was, if you're not a large account you get
               | terrible support. Saying huge accounts great support
               | isn't relevant for the overwhelming majority of AWS
               | customers.
        
               | neximo64 wrote:
               | We don't have a huge account for our AWS. I'm not even
               | sure of the tiers of support. The account manager we had
               | assigned to us was great.
        
           | scarface74 wrote:
           | I work at AWS now (ProServe). But I got my start with AWS at
           | a 60 person company that had business support. The live
           | support was amazing and I often used them as the "easy
           | button"
        
             | trallnag wrote:
             | Hmm, I should use that more often
        
           | ripper1138 wrote:
           | > AWS doesn't even have a way to report a bug...
           | 
           | Customers are able to report literally any issue at all via
           | support. AWS support is a huge organization. On my service
           | team, there were plenty of instances where bugs were reported
           | by customers.
        
             | c7DJTLrn wrote:
             | That sounds pretty nice compared to the GCP world where
             | every issue is your issue and you need to pay for support
             | or forget it. Last time they fucked up, I had to raise the
             | issue in a public bug tracker to which I got one of those
             | "not a bug, wontfix, your problem, goodbye" kind of
             | responses.
        
               | da768 wrote:
               | Whenever I've needed AWS support it was just as aweful as
               | GCP. For example, 12 hour Aurora outage, support was only
               | replying a generic "waiting for engineering team" answer,
               | SLA only refund about $15 although we ended up paying for
               | all the replicas we tried to spin up from backups and
               | wouldn't start. Architect and account manager didn't give
               | a damn about our issue.
        
           | jaggederest wrote:
           | For what it's worth I've worked at multiple companies where
           | we had an AWS technical account rep in our slack channels
           | helping us solve problems.
           | 
           | Agreed that their general support (and some of the design and
           | implementation of their services) makes me want to give up
           | and go outside and touch grass instead of working in tech,
           | but as soon as you reach the level where you are in contact
           | with support directly, they're very responsive. It's a weird
           | dichotomy.
        
             | mrbungie wrote:
             | A "dichotomy" that is explained by big bucks is not a
             | dichotomy but mere business strategy from AWS's part.
        
           | primax wrote:
           | > AWS doesn't even have a way to report a bug... what "on top
           | of the list" are you talking about? Who else is on that list?
           | 
           | You gotta love these assertions at the top of a post that
           | flag people who confidently are saying they have no idea what
           | they're talking about.
           | 
           | You can report bugs via your solution architect, via support,
           | or via the Re:Post bug report tool.
           | 
           | > AWS support is atrocious, unless you cozy up to them
           | somehow to get special treatment. If you are a "regular"
           | customer, you get zilch. AWS offers best performance for the
           | best price, that and they are the largest service provider is
           | what makes them succeed. They don't need to spend anything on
           | support -- it's not cost effective.
           | 
           | Even basic support is great, I've had most of my issues fixed
           | within 8 hours. If you pay for business or enterprise support
           | you'll have a dedicated person you deal with. Try comparing
           | that to GCP...
           | 
           | > My personal experience: we provided a service through AWS
           | marketplace related to storage. So, we are not your "regular"
           | customer, had to pay them some for accreditation... and
           | here's the thing: it's probably something about how we create
           | EBS snapshots, but about every thousand or so snapshots
           | creation stalls. Doesn't fail, doesn't succeed, just hangs
           | forever. There's nobody to talk to in AWS about this. People
           | who "certified" us (what a joke...) cannot do anything about
           | this problem.
           | 
           | If you were a solution partner you would have had a partner
           | solution architect who would take point on this problem for
           | you. This sounds like your organisation didn't accurately
           | share access to this resource with the right people.
           | 
           | > And there's more stuff like that. Sometimes some resources
           | won't delete first 1-2-3 times you try. Some other resources
           | are created, but don't work once in a blue moon. And all you
           | can do about it is just curse and try again.
           | 
           | I've been AWS facing for 11 years and I can count on one hand
           | the times this has happened. Did you file a support ticket?
           | 
           | Source: Work in an AWS consultancy, before that was running
           | tech for a startup on AWS
        
             | Aeolun wrote:
             | > I can count on one hand the times this has happened
             | 
             | You must never have worked with CloudFormation/CDK. What a
             | shitshow. If you need to do anything non-trivial you are
             | basically guaranteed to need a whole ream of dirty hacks
             | and workarounds.
        
       | nfRfqX5n wrote:
       | this is not new. AWS support would always help with this.
        
       | mooreds wrote:
       | Not just their "clouds" but their "cloud spend".
       | 
       | From the article:
       | 
       | > Amazon Web Services sales and support teams are currently
       | "spending much of their time helping customers optimize their AWS
       | spend so they can better weather this uncertain economy."
        
         | [deleted]
        
       | samstave wrote:
       | Personally, this is what I expect of them, specifically for
       | smaller orgs that don't have CISO, SRE or other ops folks needed.
       | 
       | Now, I'm not saying this should be the expected behavior for
       | everyone or every situation, but "aws staff spends majority of
       | time doing what aws staff is hired and skilled to do"
       | 
       | Seems pretty "cost of doing business" to me
        
         | neves wrote:
         | I think that this consumer oriented facet of Amazon is
         | impressive. No other provider will dedicate this time to
         | clients.
        
           | hanselot wrote:
           | I can make a similar argument regarding Jehova's witnesses.
        
             | samstave wrote:
             | Guys! We have another ~~Linux~~ Zealot over here!
        
           | mschuster91 wrote:
           | Oh, I have had good experiences with German providers Manitu
           | and Corpex. Very good service, although the latter almost
           | exclusively deals in managed hosting.
        
       | locustmostest wrote:
       | I believe the next wave of SaaS is products that provide custom
       | deployment, operational support, and cost optimization, where
       | specialist software orgs can provide standard efficiencies and
       | engage AWS for deeper looks.
        
       | rr808 wrote:
       | Lol optimize to help the customer or Amazon?
        
         | [deleted]
        
       | eitally wrote:
       | I was in Google Cloud's partner org for >7 years, managing a team
       | of solution architects pointed at our most strategic partners
       | (Accenture, Deloitte, Infosys, HCL, TCS, etc). I have been saying
       | for nearly the entirety of the time that the way things will pan
       | out for hyperscalers is going to roughly look like this:
       | 
       | AWS = the bazaar of public clouds. A service for nearly
       | everything, but WYSIWYG and quality & consistency will be all
       | over the place, just like amazon.com has become.
       | 
       | Azure = will ultimately own the enterprise. Microsoft is the new
       | IBM and their mature GTM and Professional Services organizations
       | will carry increasing value as cloud services become mostly
       | commoditized. Microsoft has multiple large revenue streams to
       | fund disruptive acquisitions and these will bear fruit through
       | good management. Satya is proving to be an exceptional CEO.
       | 
       | GCP = an also-ran on compute/network/storage, but potentially
       | differentiated on data, analytics & AI if leadership can organize
       | the troops and keep focus on things that matter. Still very rough
       | on the GTM side, and late to the party, making enterprise sales
       | that much harder. Has some unique strengths, but can't rest on
       | its laurels. Execution and focus are key.
       | 
       | Oracle = Making a killing selling an easy-button-to-cloud to
       | existing Oracle on-prem customers, and frankly, this is plenty to
       | keep things moving forward for a while.
       | 
       | IBM = Lots of hopes & dreams associated with RedHat acquisition
       | and GBS/GTS split. It's still a bit early to tell how things will
       | pan out, but IBM isn't playing the same sport as the rest, and
       | has a lot more in common with Oracle than the big three.
       | 
       | My prediction for the next couple of years is that AWS continues
       | to lose market share due to QoS and ultimately being spread too
       | thin (and alienating partners), and all the rest capture it. MSFT
       | is the biggest winner, Google will eventually find a sweet spot
       | (it's already in the "too big to fail" category, and will achieve
       | profitability this year), and ORCL/IBM will see increasing
       | success in their niches but largely avoid direct
       | competition/comparison with the others.
        
         | nextworddev wrote:
         | People underestimate AWS but as long as their S3, EC2, SQS, and
         | Lambda keeps kicking butt, it'd be really hard to displace
         | them.
         | 
         | All roads ultimate lead to storage and compute, which I think
         | AWS does a really good job in.
        
           | danpalmer wrote:
           | Are these all kicking butt? S3 is great, and I don't know
           | much about SQS...
           | 
           | But EC2? Configurations are static, cost optimisation is
           | hard, the spot market is a pain, reserved instances aren't
           | fun to trade-off. Compared to GCP where you get sustained
           | usage discounts which are much simpler to reason about, and
           | you can spec machines in much more dynamic ways.
           | 
           | And Lambda? It's just not a great environment to build
           | applications for. Frameworks are necessary to tame
           | complexity, API gateway adds a ton more complexity, dealing
           | with any state is difficult. Lambda doesn't seem any better
           | than any of the other FaaS offerings. Fargate seems like a
           | better option for most people who want to use serverless, but
           | again that's not unique.
        
             | acdha wrote:
             | AWS compute savings plans are easy to reason about, though,
             | and cover things like Lambda and Fargate, too. GCP's
             | sustained usage discounts are nice but they don't save more
             | than AWS' savings offer and in the last 6 months that's
             | been a lot lower than GCP's cost increases on other
             | services.
        
             | nextworddev wrote:
             | As for Lambda, if ur app gets less than a million requests
             | it's basically free, which I love. API gateway lets me
             | sleep at night because of all it's security features.
             | 
             | EC2 is kind of crappy UX wise but spot instances at cheap
             | af
        
         | primax wrote:
         | > Azure = will ultimately own the enterprise. Microsoft is the
         | new IBM and their mature GTM and Professional Services
         | organizations will carry increasing value as cloud services
         | become mostly commoditized. Microsoft has multiple large
         | revenue streams to fund disruptive acquisitions and these will
         | bear fruit through good management. Satya is proving to be an
         | exceptional CEO.
         | 
         | Given AWS is still growing at 40%+ a year on their huge base,
         | vs Azure who have struggled to hit 30% on a much smaller
         | base... I'd expect the opposite to happen here. Microsoft will
         | end up as the home for companies who 'buy' tech, and Amazon for
         | the companies who 'build' tech.
        
         | ctvo wrote:
         | > AWS = the bazaar of public clouds. A service for nearly
         | everything, but WYSIWYG and quality & consistency will be all
         | over the place
         | 
         | > My prediction for the next couple of years is that AWS
         | continues to lose market share due to QoS and ultimately being
         | spread too thin (and alienating partners), and all the rest
         | capture it.
         | 
         | Need data points.
         | 
         | I've used AWS, GCP, and Azure. AWS is still the only cloud I'd
         | use for business critical infrastructure. There hasn't been a
         | drop off in the quality. I'll take AWS compute (EC2, ECS) over
         | anything at Azure and GCP. Warts and all.
        
       | coding123 wrote:
       | We have some kind of k8s cluster that's just locked into a state
       | where we can't delete it because of some wacky dependencies that
       | no one at the company understands (it was set up by others). So
       | at some point we'll reach out to amazon to have them wipe it.
       | 
       | It's kind of a messy product.
        
       | boesboes wrote:
       | They still have plenty of time to cripple their UI though
        
         | sebazzz wrote:
         | What? The AWS UI is _much_ better than that slow ever-slowing-
         | progress-bars RandomError throwing Azure portal.
        
           | running101 wrote:
           | AWS UI is awful compared to Azure. At least in Azure I can
           | quickly search across regions and subscriptions for cloud
           | resources.
        
           | SketchySeaBeast wrote:
           | That's funny - I primarily have used Azure but whenever I
           | drop into AWS I feel like it's actively punishing me for
           | trying to use the UI.
        
           | abledon wrote:
           | Hard Disagree! Azure Portal is the most clear Cloud UI out
           | there. From the Icons to the Blade/Journey UX scrolling
           | sideways.
        
           | edgyquant wrote:
           | Just because there are worse things doesn't mean the current
           | interface doesn't suck
        
             | hot_gril wrote:
             | It does kinda mean that if _everything_ else is worse.
        
         | mschuster91 wrote:
         | Why would you ever want to use the UI for, other than
         | troubleshooting?! No matter the cloud provider in question,
         | _everything_ should be done through Terraform or another
         | provisioning platform.
         | 
         | Seriously, doing stuff manually on any hosting provider has
         | become a crimson red flag for me. Even at home for my private
         | stuff, I have ansible scripts for every piece of
         | infrastructure. No more manually set up raspberrys or what
         | ever.
        
       | nixcraft wrote:
       | A few weeks back helped someone move from EC2/ELB/RDS to
       | Lightsail Vm/LB/Database The same app and stuff, but the price is
       | now fixed[0]. So many apps can use something like Lightsail and
       | cut costs. I am surprised AWS don't promote lightsail. They also
       | have a weird OS selection and still have CentOS 8, which is no
       | longer supported. They treat Lightsail as a second-class citizen.
       | 
       | [0] https://aws.amazon.com/lightsail/pricing/
        
         | nik736 wrote:
         | Lightsail is heavily throttled, so I wouldn't use it for
         | anything "serious".
        
           | rstupek wrote:
           | Do you have any evidence that supports this?
        
           | Sohcahtoa82 wrote:
           | Lightsail instances are just burstable (t2/t3) instances
           | renamed.
        
         | danpalmer wrote:
         | I think DigitalOcean is a far better Lightsail implementation.
         | 
         | Both products trade predictable performance and support for
         | predictable, low, pricing.
         | 
         | However DigitalOcean is a much slicker product. It's both more
         | capable due to the wider range of services, and simpler due to
         | not having all the AWS cruft. It's still not really appropriate
         | for big services (we had outages due to poor DO network
         | management, poor support), but great for what it is.
        
           | Sohcahtoa82 wrote:
           | I won't use DO anymore simply because they took my droplet
           | offline because it got DDoSed.
           | 
           | The attack only lasted a couple minutes and I was able to
           | weather the storm just fine. I only used it as an IRC
           | bouncer. During the attack, my IRC ping went up, but nothing
           | crashed until I got disconnected, couldn't reconnect, and had
           | an e-mail in my box from DO saying they took my droplet down
           | to protect their network.
        
       | coredog64 wrote:
       | We'll see if it stays this way, but the goal of ProServe and
       | Enterprise Support is to ensure business outcomes rather than
       | directly drive cloud spend.
        
         | Macha wrote:
         | Ehh, at some point the automatic suggestions for lambda,
         | athena, sagemaker, whatever is at least a little interested in
         | getting customer lockin/stickiness.
        
           | scarface74 wrote:
           | I work in ProServe. But even when I was in doing regular old
           | enterprise architecture on prem, with Azure, etc. You're
           | always locked into your infrastructure and the cost to
           | migrate any large infrastructure no matter how "agnostic" you
           | try to stay comes with major cost both real and opportunity
           | and risk of bugs and down time.
           | 
           | Show me your "agnostic" solution and I can show you plenty of
           | ways that you tied yourself to your choices. Not to mention
           | all of the organizational constraints and coordination -
           | architecture review boards, planning committees, project
           | management organizations, etc.
        
       | silisili wrote:
       | This sounds like a good thing. I work for a <1000 employee firm,
       | and AFAIK we're spending in 6 figures monthly with AWS. I cannot
       | even imagine what huge places spend there.
       | 
       | All it takes is bad support, severe price undercutting, or
       | degrading service and we're off to another, or a split. By having
       | people at least feign caring, giving support, and giving
       | suggestions, they're locking us in and getting way more back in
       | revenue than what it's costing them, at least from us.
        
         | MuffinFlavored wrote:
         | > I work for a <1000 employee firm, and AFAIK we're spending in
         | 6 figures monthly with AWS.
         | 
         | The argument for 6 figure cloud spend was always "it'll help
         | you not need as many infrastructure-related dev ops engineers",
         | right? Would you say that's true for your org?
         | 
         | Every org I've ever worked at has had 6 figure monthly cloud
         | bill, and the 6 figure monthly payroll bill related to teams
         | working with cloud infrastructure
        
           | crabbone wrote:
           | It's kind of even worse: since public cloud became very
           | popular, the profile of infrastructure programmers changed. A
           | lot of knowledge necessary to run own infrastructure
           | disappeared, being replaced by the knowledge of how to
           | operate the service.
           | 
           | So, for example, things like PXE boot, which would be useful
           | in your own datacenter, but don't work (well) in clouds s.a.
           | AWS become mystery. Obviously, this isn't limited to just
           | PXE.
           | 
           | And since public cloud's goal is to provide service to as
           | many customers as possible while having as few staff infra
           | engineers, the total expertise among general public
           | disappears. Having worked in the infra business for a while
           | -- it's increasingly more difficult to hire infra engineers.
        
             | senko wrote:
             | > since public cloud became very popular, the profile of
             | infrastructure programmers changed. A lot of knowledge
             | necessary to run own infrastructure disappeared, being
             | replaced by the knowledge of how to operate the service.
             | 
             | It was ever thus. Previously it was CISCO Solution
             | Specialists and HP/UX Most Valuable Engineers and the like,
             | and all the knowledge beyond the basics was completely tied
             | to the vendor. In enterprise IT, it's still like that for
             | Microsoft.
        
             | Kye wrote:
             | Is there some institutional barrier to training new hires?
             | It seems like the inevitable result if you need an
             | increasingly niche skill. The situation doesn't seem like
             | it'll get better on its own.
        
               | bcrosby95 wrote:
               | Train by who? How do you start this process? If you never
               | had the expertise, or more unfortunately had it but lost
               | it, it's very expensive to gain it.
        
           | dragonwriter wrote:
           | A six figure payroll bill related to teams working with cloud
           | infrastructure? So, a low-single-digit number of engineers?
        
           | renewiltord wrote:
           | The real cost in most places is in opportunity cost and lead
           | time. e.g. you want to do a thing but you can't because you
           | don't have the physical resources available. That can be very
           | frustrating for people trying out things that will produce
           | novel revenue in the org. Reducing the cost of
           | experimentation is quite valuable in a larger organization.
           | 
           | For something like our HFT firm, we see high sustained
           | utilization so we have an on-prem cluster of nodes that we
           | run. They're faster (more powerful CPUs), have more RAM, and
           | you can run fiber optic interconnect that's, in practice,
           | faster than on AWS. We still use AWS extensively, though.
        
             | candiodari wrote:
             | Except that for the same cost in cloud you can't just have
             | double the resources in dedicated servers, but like 3x to
             | 10x depending. And if you do colo, even more.
             | 
             | I get that if you have management that won't keep even 10%
             | spare capacity online you can't work like this, but they
             | will also "tightly control cloud spending", won't they? Not
             | sure.
             | 
             | Cloud only really makes sense at really small scale.
        
               | renewiltord wrote:
               | Everyone's use cases are different. But I think I'm
               | making the right trade-offs here overall (I am management
               | in this world).
        
             | nostrebored wrote:
             | You can use fiber for hft through AWS as well, but you lose
             | a lot of the benefits of cloud when you have a sustained
             | workload that doesn't benefit from being fault tolerant
             | because the system would be down anyway.
        
           | bastardoperator wrote:
           | Considering the cost of the infra it would take to replicate
           | some of these services, not to mention bandwidth, DC, and
           | electricity costs, you're probably still saving money. You're
           | absolutely saving on Network Ops and Data Center Ops not to
           | mention the huge investment in gear one has to make. A single
           | server can cost over 100K. Network gear can costs way more.
           | The fact that you don't have to make those investments is the
           | allure. At a certain point, you might outgrow someones else's
           | cloud and build your own, but that divide is a fairly large
           | threshold to cross.
        
             | scarmig wrote:
             | And as far as problems go, "we're so successful that it
             | makes financial sense to build our own on-prem cloud" is a
             | pretty good problem to have.
        
               | xmprt wrote:
               | I always figured it was the other way around. When you're
               | small it's pretty easy to get by with a stupidly simple
               | solution but as you grow you end up needing to spend much
               | more to build something scalable and at that point, using
               | the cloud makes sense. The biggest success that cloud
               | providers have had is convincing users that they need to
               | spend $100k and that a much simpler $5k solution that's
               | built using off the shelf components just won't cut it.
        
               | foobarian wrote:
               | The most painful is having to run multiple data centers
               | for HA. Double or triple the price right there.
        
               | traceroute66 wrote:
               | > The most painful is having to run multiple data centers
               | for HA. Double or triple the price right there.
               | 
               | Ok, let's make one thing patently clear: ITS THE SAME IN
               | THE CLOUD
               | 
               | All the cloud vendors will tell you need to have stuff
               | replicated in multiple "Availability Zones" or "Regions".
               | 
               | And yup, the nickle & diming nature of the cloud means
               | that's going to cost you double or triple.
        
               | HardlyCurious wrote:
               | I see the cloud mostly for startup-ish companies hoping
               | to grow rapidly but which want to avoid large upfront
               | expenses to be ready for said growth.
               | 
               | A stable company where growth as a percentage isn't
               | likely to be significant can run things cheaper on their
               | own in most cases. At least if you consider the cost of
               | the inevitable departure from the cloud provider either
               | to switch another or to go on-prem. And if you aren't
               | willing to make that exit, you can guarantee your cloud
               | provider won't stop cranking up the fees until the threat
               | of you leaving surfaces.
        
               | scarmig wrote:
               | I think this is a pretty key point. If a business is
               | going through any kind of rapid change, cloud providers
               | offer a lot of off-the-shelf help for that, be it ability
               | to scale, hosted infrastructure, or PoPs in new
               | geographies. If the company is relatively static with
               | easily predictable future requirements, you can get a lot
               | more bang-for-your-buck by handling things on your own
               | and developing your own in-house expertise.
        
               | icefo wrote:
               | There is also a third approach that is the best if you
               | have a predictible base load with surges sometimes imo:
               | hybrid cloud
               | 
               | You basically run the base load in your own data center
               | and the surges go to the cloud. My university is
               | evaluating this because sometimes you have multiple labs
               | that need a lot of compute resources at the same time and
               | local compute cluster has finite capacity.
        
             | phpisthebest wrote:
             | >>A single server can cost over 100K. Network gear can
             | costs way more.
             | 
             | "Can" sure... likely to for most orgs no. Certainly not for
             | someone running a 5 figure cloud bill...
             | 
             | as a reference, our standard compute server we use on prem
             | is $12,000 including a 5 year support contract.
             | 
             | > At a certain point, you might outgrow someones else's
             | cloud and build your own,
             | 
             | If your buying $100K servers, and $100K network devices, I
             | pretty sure you are at the point
        
               | baq wrote:
               | if you're buying either, you're paying millions in
               | salaries to technicians and sysadmins and
               | DBAs^W^W^Wdevops and SREs.
               | 
               | i've been on both sides of the fence and it's a case of
               | 'grass is always greener on the other side'. the truth
               | is, running any sort of non-trivial infra is ducking
               | expensive.
        
               | bcrosby95 wrote:
               | This is what always made me chuckle a bit. I have a half
               | dozen friends who were sysadmins 20 years ago and today
               | they're "DevOps".
               | 
               | Precisely zero have been put out of work.
        
               | throwawayapples wrote:
               | Yep. It's not like either sysadmin or devops are harder
               | than or easier than the other... they're just different.
               | 
               | Actually, seems like managing k8s is an order of time
               | expenditure greater than managing an old-school F5 with a
               | bunch of Unixy web servers behind it.
        
               | primax wrote:
               | When I was a sysadmin it was definitely harder.
               | 
               | Devops gives the benefit of cows instead of pets, and a
               | ton of reusable work. So I get why it's more 'valuable'
               | from a remuneration basis.
        
               | SoftTalker wrote:
               | Yes, time and complexity.
               | 
               | Managing a switch and servers is a piece of cake compared
               | to managing k8s, IMO.
        
               | aeyes wrote:
               | Depends on how much abstraction you have, I have seen big
               | companies where deploying code is basically like using
               | Heroku. As an engineer responsible for a couple of
               | services you don't need to know or care if this code is
               | running on bare metal, Mesos, K8s and you care even less
               | about the data center.
               | 
               | I come from this old world of managing switches and
               | servers and today we definitely need a lot less people to
               | run code in production. I used to work at a company with
               | ~2000 machines in physical data centers before
               | containerization, this required a huge infra team - I'm
               | sure that today I could support the same workloads with
               | half the team.
        
             | dimitrios1 wrote:
             | Having worked half my career at places with their own data
             | centers and self ran infra, and the other half with mostly
             | cloud based solutions, I have a theory.
             | 
             | Perhaps we are designing far more complicated solutions now
             | to leverage these cloud services, whereas having the
             | constraints of a self operated data center and
             | infrastructure necessitates more ingenuity to achieve
             | similar results.
             | 
             | We used to do so much more with just a few pieces of
             | infrastructure, like our RDBMS's, as one example. It was
             | amazing to me how many scenarios we solved with just a
             | couple of solid vertically scaled database servers with
             | active-active failovers, Redis, an on-prem load balancer,
             | and some webservers (later, self hosted containerization
             | software). We used to design for as few infrastructure
             | pieces as possible, now it seems like that is rarely a
             | constraint people have in their minds anymore.
        
               | notyourwork wrote:
               | Amen, I'm becoming an old grumpy engineer on my team for
               | constantly asking why we need yet another <insert cloud
               | technology here>. I'm not against new technology but I am
               | against not considering what we have and how it may
               | already solve the problem without adding wider breadth to
               | our operational surface area. And it's every single damn
               | year now because now cloud providers string their own
               | cloud primitives together to form new cloud services.
        
               | datavirtue wrote:
               | Could have just called an API but instead we fired an SNS
               | event. Sigh.
        
               | notyourwork wrote:
               | How many times I've had this discussion. Let's publish a
               | notification, and let's have the message receiver call
               | some API. Why not just call the API from the place where
               | you want to publish the message? Because we need this SNS
               | message queue.
        
               | phpisthebest wrote:
               | it is the classic find a problem to use our solution, or
               | XY problem
               | 
               | As an example I have seen many times people attempt to
               | find a reason to use k8s because the industry says they
               | should instead of looking at what they need to do and
               | then determining if k8s is the best for that application
        
               | chubs wrote:
               | Maybe they're looking for an excuse to gain k8s
               | experience to bolster their resume? If most startups
               | fail, might as well gain some skills out of the current
               | one? Perhaps it doesn't benefit the startup though,
               | inflating complexity, infra spend, and slowing
               | productivity.
        
             | jbverschoor wrote:
             | A single server which can probably serve your complete
             | customer base could / might cost less than 5k
        
               | nathancahill wrote:
               | As I recall, StackOverflow runs (ran?) on 6 large but not
               | massive servers.
        
               | ctvo wrote:
               | They also used the dotnet stack (Windows Server, IIS,
               | MSSQL, dotnet), and optimized the heck out of everything.
               | They're not the typical use case.
               | 
               | (I'm not saying dotnet allowed them to get by on N low
               | digit servers. I'm saying those folks are atypical)
        
               | bcrosby95 wrote:
               | We did very similar with a Java stack without even trying
               | really. Competitors using things like Ruby and went all
               | in on distributed messes had hundreds of servers but we
               | had about 15. It does require you to be aware of
               | performance, but I wouldn't call it difficult or
               | particularly time consuming.
        
               | l33t233372 wrote:
               | What are you saying by pointing out that they use the
               | dotnet stack?
               | 
               | It's interesting trivia to be sure, but I'm wondering if
               | you were making a point with that
        
               | drchickensalad wrote:
               | Have you seen the performance of standard ruby and python
               | web frameworks in comparison? It's a massive difference
        
               | birdyrooster wrote:
               | Add in redundancy and that number quadruples bc you need
               | to manage shared state and 2-3x the hardware
        
               | Spivak wrote:
               | I am gonna go out on a limb and say that given that
               | they're talking about replication they mean server rack
               | which _definitely_ is not $100k /mo but can pretty easily
               | be $100k up-front.
        
         | tpmx wrote:
         | <incoherent-stuff/>
        
           | sebzim4500 wrote:
           | Total spend = 1k * 100k = 100M
           | 
           | AWS spend = 500k * 12 = 6M
           | 
           | AWS spend / Total spend = 6%
           | 
           | Where the hell does 0.1% come from?
        
             | tpmx wrote:
             | Yeah, that was broken. Please ignore. (Sorry!)
        
           | [deleted]
        
         | gowld wrote:
         | > All it takes is [bad things] and we're off to another
         | 
         | > By [avoiding those bad things] they're locking us in
         | 
         | Providing a good product at a good price is "locking in" ?
        
           | silisili wrote:
           | Well sure. We're happy with the service and support, so keep
           | deploying more and more. It's not locking in like a gym
           | membership sure, more of a trust relationship. And the more
           | we deploy and use, the harder it would be to jump ship.
        
             | paulmd wrote:
             | Egress costs alone make leaving pretty much a non-starter
             | once you're entrenched.
             | 
             | Leaving really involves running two systems in parallel for
             | a bit and gradually doing a changeover - blue-green in
             | production between two different clouds. Which is not
             | cheap/free either, you are actually going to _increase_
             | your costs significantly as you leave.
        
               | robocat wrote:
               | Appropriately sneaky: https://aws.amazon.com/snowmobile/
               | appears[1] to allow export of petabytes of data but then
               | the FAQ reneges "Snowmobile does not support data
               | export"[2].
               | 
               | [1] tagline: "Migrate or transport exabyte-scale datasets
               | into and out of AWS".
               | 
               | [2] "Q: Can I export data from AWS with Snowmobile?" A:
               | No. in
               | https://aws.amazon.com/snowmobile/faqs/#Using_Snowmobile
        
               | renewiltord wrote:
               | The answer, for the curious, is to use Snowball Edge.
        
           | factormeta wrote:
           | Hmm, don't think that applies all the time. Herznet is way
           | cheaper than AWS, and in most cases, I don't know why
           | companies won't use it instead of AWS. The only reason people
           | use it from what I know is that helps them with the next job,
           | because everyone is using AWS. Kinda feels like ms and oracle
           | in the 90's.
        
             | [deleted]
        
             | jonfromsf wrote:
             | Herzner is way cheaper. But it doesn't have a database
             | service or an S3 analog. Both of those are needed if you
             | want to avoid spending a lot of money on operations staff.
        
             | mschuster91 wrote:
             | There are more factors than "nobody got fired for buying
             | IBM" regarding AWS/Azure/GCE:
             | 
             | - the Big Three have data centers all over the world. No
             | other provider can match that. (Obviously: if 99% of your
             | customers come from the EU, Hetzner, OVH and friends will
             | be cheaper)
             | 
             | - there is _a ton_ of prior experience in managing and
             | provisioning infrastructure on the Big Three. You got
             | Terraform providers covering every oh so tiny resource they
             | have, and for literally every problem you got dozens upon
             | dozens of stackoverflow /serverfault posts.
             | 
             | - the Big Three have allll sorts of managed services. No
             | matter if you need cheap blob hosting, virtual servers,
             | bare metal servers _without_ paying setup fees, AI training
             | servers, VPNs, global load balancing /caching, data
             | storage/archival, databases, "serverless" hosting, email,
             | SMS, IoT communication, Active Directory and other
             | identity/authentication/authorization solutions, business
             | data analytics, whatever - it's all a one-stop-shop. No
             | dozens of vendors, bills, payment schedules, GDPR
             | processing agreements, budget issues - one vendor, one
             | bureaucracy, everything you need.
             | 
             | It's really bad that the EU never woke up to the rise of
             | the Big Three and say, for example, financed the
             | development of OpenStack to the tune that it could be used
             | by domestic providers, because now the Big Three have all
             | but eliminated the smaller competition.
        
             | justinclift wrote:
             | Hetzner's dedicated servers have a limit of 10 firewall
             | rules for their (hardware based) equivalent of AWS Security
             | Groups.
             | 
             | And some of those rules already have mandatory entries, so
             | you'll likely only have around 7-8 actually usable.
             | 
             | "Just use the firewall in your OS" is a workaround, but
             | doesn't fully cover the same scenario's. That's why AWS has
             | Security Groups after all.
             | 
             | This limitation means we have to use smaller sized servers
             | (64/128GB ram), rather than bigger ones which would
             | otherwise be more cost effective.
             | 
             | It's a real pita for us, and Hetzner have shown no interest
             | in ever addressing the problem. :(
             | 
             | That being said, it's all still a lot cheaper than AWS, as
             | Hetzner doesn't charge for bandwidth with their dedicated
             | servers. :)
        
         | Aperocky wrote:
         | It seems your infra spend is fairly contained compared just to
         | the human cost of your organization.
        
         | andreygrehov wrote:
         | Any chance to contact you? Would love to ask a few questions.
        
       | fwungy wrote:
       | IBM's Hybrid Cloud concept is promising.
       | 
       | Run your sensitive or expensive bulk resources like dbs onprem
       | and go cloud for the rest. You can often save big just bringing
       | your db onsite.
        
         | fortylove wrote:
         | The concept is promising, agreed. But like most (all?) recent
         | IBM products, the failure point will be in the execution of the
         | concept. IBM is a marketing company at this point, not a
         | serious technology company.
        
           | acdha wrote:
           | Execution and rent-seeking - their culture is based on high
           | margins so they tend to design things which will put a lot of
           | hard to migrate logic into their app and even when they know
           | they're competing against more affordable options the prices
           | will be eye-watering. I assume that the whales who do buy
           | generate enough revenue to make up for the lost sales but it
           | doesn't seem like a good long-term strategy.
        
         | running101 wrote:
         | Works if latency isn't an issue between app and db, which it
         | often times is.
        
           | danpalmer wrote:
           | I did a cloud migration from a local datacenter to GCP and DB
           | latency was the big issue for us. Latency from our DC to GCP
           | was ~10ms, so an average page load with 10 queries added
           | 100ms overhead which was mostly unacceptable.
           | 
           | Big enterprisey applications like this would love to only be
           | doing 10 queries, 30+ is much more likely, and I bet they
           | won't be moving from 1 DC to 1 cloud region, so 10ms is also
           | unlikely. Back of the envelope suggests more like 1s of
           | additional latency.
           | 
           | That would barely work for a big enough OLAP deployment, with
           | all sorts of varied consumers, let alone OLTP.
        
         | electroly wrote:
         | In the AWS world, you can do this by colocating servers at an
         | AWS Direct Connect facility. You get a fiber connection from
         | your rack to AWS's rack inside the building, and then AWS
         | backhauls it on their network to the AWS Region. There are such
         | colocation facilities located within a few miles of any major
         | AWS Region for low latency (~1 msec) connections.
        
       ___________________________________________________________________
       (page generated 2023-04-17 23:01 UTC)