[HN Gopher] CTO Headaches: Top cloud-to-cloud migration woes (an...
___________________________________________________________________
CTO Headaches: Top cloud-to-cloud migration woes (and how to solve
them)
Author : llarsson
Score : 59 points
Date : 2021-02-17 08:20 UTC (12 hours ago)
(HTM) web link (www.cncf.io)
(TXT) w3m dump (www.cncf.io)
| rsync wrote:
| I don't know if this will penetrate upward to the CTO level, but
| 'rclone'[1] is an open source, standards based solution to cloud-
| to-cloud migration.
|
| You can use rclone - and all its powers - without installing or
| running it locally: ssh user@rsync.net rclone
| s3:/some/bucket gdrive:/blah
|
| ... as there is a cloud storage provider with rclone _built into
| the platform_.[2]
|
| [1] https://rclone.org/
|
| [2] https://www.rsync.net/products/universal.html
| llarsson wrote:
| Yeah, that migrates data files from A to B. And that has
| tremendous value, as your command line shows!
|
| But, it does nothing for making, e.g., the message bus service
| that AWS offers compatible with the one Azure offers, though,
| so your application will surely know something's up when it
| doesn't work any more.
| kirktrue wrote:
| I would hope a chief _technical_ officer with their salt would
| realize how powerful this setup is. Unfortunately, the lack of
| a cute name, branding, and marketing engine behind it may cause
| some to dismiss it as 'it's too simple, it must be missing
| something.'
| sidlls wrote:
| At a certain scale anything purporting to be as simple as a
| one-liner CLI program is suspect.
|
| Larger organizations get _messy_ , it's (almost) never as
| simple as "clone from S3 bucket X to drive folder Y".
| tyingq wrote:
| To me, if you're using AWS, GCP, or Azure, you should be using
| their "hard to migrate away" services. That's the value of the
| brand name cloud platforms. Plumbing that's done for you.
|
| If you're going to the trouble of being agnostic, then you might
| as well deploy to a way cheaper bare metal datacenter and enjoy
| market rate storage, compute and egress.
|
| Running cloud agnostic on AWS, GCP, etc, is the worst of both
| worlds.
| fatnoah wrote:
| >If you're going to the trouble of being agnostic, then you
| might as well deploy to a way cheaper bare metal datacenter and
| enjoy market rate storage, compute and egress.
|
| Interesting. I wonder a which scale this becomes true. It's
| been a long time, so I'm not current on rates, but I've been
| involved with a couple migrations from datacenter to AWS and
| AWS was substantially cheaper for the same level of infra in
| both cases. These were smaller use cases, though.
| Terretta wrote:
| Absolutely, and not said enough -- but do this in a _risk
| managed_ way. Hold yourself accountable to really give that
| thought. Terretta doctrine on this is "Aggressively _exploit_
| CSP differentiation, with a _credible_ migration plan you keep
| in the drawer."
|
| To borrow an AWS metaphor on "One way doors versus two way
| doors" decisions, do pre-think, but don't pre-code. Just think
| as you design the house about whether you might someday want to
| go through that door another way. It's inexpensive now to make
| design decisions that make that easier later.
|
| At an enterprise, fully understand the BCP or business
| continuity plan. You will likely be stunned how long functions
| are considered OK to be offline for regulator-compliant BCP
| planning purposes, relative to the SLAs that I.T. gets yelled
| at about.
|
| Your job before committing to vendor lock in is to write down a
| plausible plan to get out of the lock in that fits (when effort
| is doubled twice) inside the BCP plan window, and maintain that
| plan as part of the platform documentation for the capability
| in question.
|
| When you study the economics of re-investment by Azure, Google,
| and AWS, it's impossible for an enterprise to achieve the same
| capability-evolution curve without floating on that rising
| tide. The hard part is to figure out how to not get capsized by
| change waves or get swamped if the CSP pulls the plug in your
| boat. These are part engineering, part
| business/procurement/legal problems, but much less costly than
| holding yourself to either a lowest common denominator or fully
| abstracted principle.
| didip wrote:
| I respectfully disagree.
|
| There's a sweet spot where companies can use the lowest
| foundational building blocks, build their own stacks on top,
| and save resources (comparatively to running your own
| datacenter).
|
| Things like EC2, EKS, S3, and RDS Aurora can be considered
| foundational but you don't really have to use Athena,
| CloudWatch, or RedShift (they are super expensive anyway).
| rubiquity wrote:
| Disclosure: I work at AWS, and might surprise you that I
| disagree. Opinions my own.
|
| I see this line get touted a lot but it is almost never
| accompanied by any depth. It sounds good, but what does it
| actually mean in practice? Do you have concrete examples of
| what using the best parts of AWS, GCP, and Azure are? From
| where I'm sitting, the best parts of the cloud were created
| 10-15 years ago: On-demand compute (without dealing with the
| joys of subnetting, DHCP, and configuring Linux interfaces),
| object and block storage, managed databases/DNS/load balancers.
| The newer features tend to lack the things you really need in a
| production ready service: monitoring, debugging, configuration,
| etc.
|
| The right answer to this multi-cloud question is to stop having
| so many different components in your stack. Be painstakingly
| deliberate about what you adopt and have someone who can
| articulate (ideally in writing for posterity) why you're
| adopting it. Pick a few foundational primitives and master them
| operationally. One relational database, one web server, one
| load balancer, one cache, zero message queues ;).
|
| (Spoiler alert: this is what all of the cloud vendors
| themselves do internally.)
| dastbe wrote:
| (also work at aws)
|
| i think that spoiler alert is a little overgeneralized.
|
| first, there's plenty of advances in those categories in the
| past 5 or so years (lambda, fargate, qldb, a bunch more) that
| move the needle significantly on the versions of things that
| existed 10-15 years ago. we didn't collectively get
| everything right by 2012, and i heartily endorse customers
| using things that take away operational complexity but are
| less "complete" compared to the lower level counterparts.
|
| second, you missed mentioning that circular dependencies is a
| very real reason for why cloud vendor service teams can't
| (not won't) use services that would otherwise benefit them.
| this is a problem that is much less apparent to customers of
| clouds because, to them, the suite of services is provided as
| a single flat layer of options.
|
| i do agree that you want to make sure any service you use is
| "complete" for you, but that completeness can exist on a
| spectrum and sometimes getting onboard early and driving
| their roadmap can benefit you greatly. i also agree that you
| want to be conservative with how many services you do use,
| because each one without exception will become a learning
| experience.
| Tehnix wrote:
| > From where I'm sitting, the best parts of the cloud were
| created 10-15 years ago
|
| I just moved from a startup that was running things on
| services that were either created or only really matured
| within the past three years, to a different startup running
| on tech from 10-15 years ago.
|
| The differences in Ops overhead and scalability is huuuuuuuge
| to say the least.
|
| Modern infrastructure on AWS for me would be:
|
| - Lambda
|
| - DynamoDB
|
| - S3
|
| - Fargate
|
| - EKS to a certain extent
|
| - And extremely excited about Serverless Aurora V2
|
| Older infrastructure with massive ops overhead:
|
| - EC2
|
| - ElasticBeanstalk
|
| - RDS MySQL/PostgreSQL
|
| Every year at AWS re:Invent I see the needle moving into more
| serverless stuff, and for a very very good reason.
|
| To maybe take up some of your points:
|
| > On-demand compute
|
| EC2 and RDS are the absolute bare minimum you can do for on-
| demand compute and was the initial introduction to the
| elasticity of the cloud. It's slow and cumbersome to scale up
| and down, and also very costly to do so (RDS is basically a
| x2 of you instance size every time).
|
| That's skipping automatic scale - for RDS? Forget about it.
|
| > The newer features tend to lack the things you really need
| in a production ready service: monitoring, debugging,
| configuration, etc
|
| State of anything running on EC2: You are 100% responsible
| for getting your metrics and logs out of the instances, you
| get close to worthless OS metrics by default, that might tell
| you info about some scaling events, but no insights into you
| actual applciation.
|
| State of serverless: Built-in integrations with CloudWatch
| for monitoring and XRay for profiling. Out of the box you
| have error reporting on Lambdas along with DLQ and much more.
|
| > without dealing with the joys of subnetting, DHCP, and
| configuring Linux interfaces
|
| I honestly hold the opinion that if you deal with things at
| this level of the infrastructure stack, then the "modern
| cloud" is not relevant to you (AWS, Azure, and GCloud), but
| you would fit better or something like Hetzner or with
| dedicated servers.
| tyingq wrote:
| _" It sounds good, but what does it actually mean in
| practice?"_
|
| I think what I was saying is pretty straightforward...
|
| _" If you're going to the trouble of being agnostic"_
|
| Meaning that if you're trying to build some sort of facade
| that makes AWS and GCP look the same, you might as well just
| make a facade that points at something cheaper than AWS/GCP.
| AWS/GCP markup for things like internet egress, storage, and
| compute are high, partially because of all that plumbing. Any
| facade puts you in a lowest common denominator situation
| where you're paying for lots of stuff then not using it.
|
| Or, in short, "cloud agnostic" running on cloud is a weird
| idea that throws out lots of the value add. Perhaps there are
| some exceptions for very simple architectures, as you hint.
| rubiquity wrote:
| > I think what I was saying is pretty straightforward...
|
| I was asking you to give specific examples.
|
| > Perhaps there are some exceptions for very simple
| architectures, as you hint.
|
| Simplicity should be the goal and the starting point.
| Terretta wrote:
| S3 and DynamoDB would be the two I'd mention first (since
| know it or not, folks are relying on them anyway), then
| things like Fargate, QLDB, and PrivateLink.
|
| Agree on the simplicity point, though the key is not just
| simplicity for coding, or for communicating about the
| architecture, but emphasizing simplicity of operation and
| honesty at the skills, abilities, and criticality of
| operating and resolving things under stress.
|
| Several years as AWS CAB member show this topic is hotly
| debated. I think there's this "third way", to
| purposefully exploit the different DNA of the top CSPs.
|
| I choose that word "exploit" to appeal to the board room,
| to feel they get more than they have to pay for (which I
| think is true, thanks to CSPs' focused spend). A more
| friendly way of putting it from CSP point of view is
| "leverage what we're best at". Rare case of genuine win-
| win.
| kirktrue wrote:
| What about the case where you're not migrating _away_ , but do
| need to support multiple clouds for regulatory, failover,
| customer requirements, etc. reasons? Is there a way to still
| leverage the first-class features of, say, AWS and GCP without
| resorting to one of these issues:
|
| - death by abstraction of cloud differences
|
| - teams incompletely reimplementing features on each service
|
| - giving up and going to bare metal
|
| ?
| llarsson wrote:
| Not really a way to leverage the different services by cloud
| vendors, because the services really are different. So it is
| either the abstractions you mention, or going with self-
| hosted. And the latter cases, you really start to appreciate
| cloud native tech such as what you find at the CNCF.
| closeparen wrote:
| S3, SQS, RDS, etc. are some damn nice primitives and also have
| analogues across all the clouds. The headcount to run their
| open source equivalents to the same SLAs is not free either.
| villasv wrote:
| I've always said this to our CTO. If we were going to use AWS,
| we should go all in. Once we stopped trying too hard to avoid
| lock in, we stopped halfassing our architecture.
| AtlasBarfed wrote:
| And if AWS hikes rates (or reduces discounts) on you once
| their algorithmic AI determines your switching cost exceeds
| the rate hike?
|
| "Oh they would NEVER do that"
|
| And these gee-whiz heavy-binding services would NEVER steal
| your data.
|
| The strange thing at work is people bend over backwards to
| use AWS stuff:
|
| "I made this virtual network card that I can detach and
| attach to another VM when we want to keep the same address
| for a box!"
|
| How about we use DNS.
|
| AWS services are great for things like their Postgres as a
| service and other things that fundamentally have standards
| based access.
|
| But the others like big data streams will be a bit more
| convenient in the beginning and then you're beholden to their
| supported features and the fact you are handing them
| visibility to your data.
| villasv wrote:
| Nice straw-man. Keep HN being HN, two thumbs up.
| Tehnix wrote:
| > And if AWS hikes rates (or reduces discounts) on you once
| their algorithmic AI determines your switching cost exceeds
| the rate hike?
|
| > "Oh they would NEVER do that"
|
| Interestingly, what you insinuate is exactly the opposite
| of actual reality when you look at what has happened over
| the years: things have gotten CHEAPER.
|
| Off the top of my head:
|
| - Lambdas are now billed per 1ms instead of per 100ms which
| will be a significant cost reduction for _everyone_ using
| Lambda
|
| - EKS started at $200 / month for the cluster and is now at
| ~$72 / month
|
| - Each bump in EC2 instance generation is always cheaper
| for the same or more performance
|
| - DynamoDB added on-demand capacity which is at face value
| more expensive, but allows you to scale to 0 and will
| usually result in an overall cost reduction
|
| - This December they announced GP3 which can scale storage
| and IOPS independently allowing you to much cheaper scale
| up your needs for more IOPS without paying for TBs of
| unused storage
|
| and probably many many more.
| llarsson wrote:
| OP here.
|
| I agree with you: if you know your deployment is, and always
| shall be, on e.g. AWS, you essentially lose out on great
| integration if you try to be cloud agnostic.
|
| However, as in the introduction:
|
| "Whether motivated by the need for a multi-cloud strategy,
| expenditure minimization, legislative or regulatory demands, or
| simply to get closer to end users, many organizations find
| themselves migrating from one cloud to another."
|
| I work with many customers that migrate away from the big US
| providers because GDPR and the "Schrems II" ruling makes it
| legally questionable to continue to use these services for
| certain data processing.
|
| Or ones that have been bitten hard by cloud outages, causing
| them to fail to deliver service to their customers. Getting a
| tiny rebate on your cloud bill that month does not begin to
| cover the lost reputation due to downtime.
| rataata_jr wrote:
| Thank you. That brought a new perspective to me.
| motohagiography wrote:
| Seconded, I thought it was very insightful.
| reacharavindh wrote:
| " But what I've also seen is that if you are using a managed
| Kubernetes service, your users and permissions handling is
| perhaps tied not to Kubernetes role-based access control (RBAC)
| features, but rather, to cloud-specific offerings. Like AWS
| Identity and Access Management (IAM). Great service, but ties you
| to the AWS platform. "
|
| - precisely why Kubernetes as a proxy for "if we build on K8s, we
| can easily move across clouds" from an infrastructure point of
| view is not true. If you develop your application for K8S, then
| yeah, by all means good for you. But, don't say your reason for
| taking on the complexity of Kubernetes as a platform is the
| ability to become multi-cloud easily.
| booleanbetrayal wrote:
| Four days ago, AWS rolled generalized OIDC support for their
| cluster auth. Presumably, this makes it easier to transition
| providers since the user store can remain consistent.
|
| https://aws.amazon.com/blogs/containers/introducing-oidc-ide...
___________________________________________________________________
(page generated 2021-02-17 21:02 UTC)