[HN Gopher] Pulumi Insights - AI generated IaC programs
___________________________________________________________________
Pulumi Insights - AI generated IaC programs
Author : kaotown
Score : 80 points
Date : 2023-04-13 18:26 UTC (4 hours ago)
(HTM) web link (www.pulumi.com)
(TXT) w3m dump (www.pulumi.com)
| fdgsdfogijq wrote:
| CDK/Pulumi are going to be used by LLMs to spin up website infra
| exactly to spec. Devs will fill in the business logic interfaces.
| Going to be wild
| shadycuz wrote:
| Yeah but ChatGPT can already write Terraform and Cloudformation
| really well. So this doesn't seem that special besides it's
| already baked into the pulumi eco system.
| smt88 wrote:
| Why would you need LLMs for a boilerplate web application?
| verdverm wrote:
| There is a "need" vs "want" aspect here, but often the
| boilerplates that we reach for become out of date. In theory,
| an LLM producing these could remain up to date without
| maintenance, key phrase "in theory"
| trufas wrote:
| There's no guarantee the boilerplate an LLM spits out will
| be up to date. It'll almost definitely have some outdated
| code in it's dataset that it can reference.
| verdverm wrote:
| yes, a basic LLM is susceptible to this. The other major
| issue is that they will generate different output, even
| with the same input.
|
| There is work going into giving the LLMs access to
| external data & systems. This is my basis for saying they
| will be able to stay up to date.
|
| I have a very different approach I'm working on:
| https://docs.hofstadter.io/getting-started/creators
| (human-made blueprints that can be started from and later
| updated to bring in the upstream changes)
| gtirloni wrote:
| _> exactly to spec_
|
| And that spec is?
| fdgsdfogijq wrote:
| Natural language :)
| JohnMakin wrote:
| but how will it be maintained? "IAC" seems like a bit of a
| misnomer here, if I'm understanding correctly. Now, if Pulumi
| generated terraform for me to maintain, that'd be a different
| matter, but this article seems to just produce cloud resources
| based on LLM prompts, which are not deterministic at all.
| twalla wrote:
| It produces pulumi programs which are the equivalent of a
| terraform module or main.tf file. Whether or not said
| programs are deterministic is another matter.
| dmattia wrote:
| Pulumi is an IAC tool similar to Terraform (it actually
| usually calls terraform providers over gRPC under the hood),
| just written in languages like Typescript/Python. I think the
| intention would be to use this AI prompt thing to generate
| pulumi and then to insert it into your codebase, just like
| you'd do with terraform, and then it becomes deterministic.
|
| I've used ChatGPT and other tools to generate Pulumi before,
| so I'm not totally sure how this specific tool from Pulumi is
| different, but I'd guess they've somehow trained it more
| specifically on their sdks/docs or something
| JohnMakin wrote:
| thanks for the clarification
| rcarr wrote:
| I had a really bad experience with Pulumi in February/March that
| probably cost me around 4 weeks of lost dev time, possibly more.
| Context: Not a DevOps guy, created a serverless site using
| Typescript, created the infrastructure for it on AWS console and
| then decided to try and replicate it using Pulumi to get some IaC
| skills.
|
| Issues:
|
| - The majority of the documentation/examples assume you are going
| to be writing all your code in one big long index.js file rather
| than the micro stacks approach. No idea why this is considering
| no-one organises their code this way in any other part of a
| coding project so don't know why micro stack approach would not
| be the default approach.
|
| - Major issues getting it to work with typescript/ts-node/tsx
| correctly. Could only manage to do this if I used a Pulumi
| Automation Runtime program rather than a Pulumi Automation Local
| program which was super awkward as it meant I lost access to
| using the CLI.
|
| - No way of testing any serverless function that called another
| serverless function. Would have to use something like LocalStack
| if you wanted to do this (and that seemed like a nightmare) or
| use a npm module that didn't seem like a safe long term bet. Not
| entirely pulumi's fault but if you want to do this it can be done
| with SST or with Amazon SAM.
|
| In the end I ended up giving up on Pulumi and will be rewriting
| the entire infrastructure code using SST. If I had an app that
| wasn't serverless I'd consider using it again but definitely
| wouldn't give it another shot for a serverless app until it
| either has similar testing functionality for serverless functions
| as SST or if there's an official way of interfacing with the SST
| framework. I'd also probably hold out until monorepo and
| Typescript support was better.
|
| My frustration levels were elevated going into the month as I
| built the site using a new framework that went from 1.0 to 2.0
| mid development and had to learn the idiosyncrasies of that.
| However the month from hell with Pulumi was enough to send me
| over the edge and I'm now taking a few weeks off from coding to
| get my patience back again.
|
| Oh and I've no idea what this AI thing is like but when I tried
| to get ChatGPT to answer questions on Pulumi it was useless. It
| would flat out lie and make up classes and functions that didn't
| exist.
| cube2222 wrote:
| Hey, I'm curious about the pulumi-ai cli.
|
| Specifically, did you solve the problem of stale API information?
|
| What I mean is that using gpt-4 to generate code is generally
| very straightforward, but due to the knowledge cutoff it won't
| know about i.e. new AWS APIs like Lambda URLs.
|
| Is this something you've managed to solve? Or is it just the
| "even with that knowledge cutoff there's enough value" situation?
| verdverm wrote:
| There are attempts at solving this issue more generally, by
| giving the GPTs access to external sources of information, and
| ofc the right prompts & chaining
| AaronFriel wrote:
| Pulumian here - this is something I'm working on and hopefully
| we'll have more to share soon.
|
| We're in a good position here in that our providers have rich
| schemas: https://raw.githubusercontent.com/pulumi/pulumi-
| kubernetes/m...
|
| However our larger providers, primarily cloud platforms, have
| schemas much larger than the context length of the model. So
| the trick is scoping that down to the necessary & sufficient
| amount of data into a prompt, whether via plugin (not yet
| available via API), preprocessing the prompt, or using a
| langchain-esque approach.
|
| As Karoly Zsolnai-Feher[1] says, "what a time to be alive!"
|
| [1] of Two Minute Papers fame:
| https://www.patreon.com/TwoMinutePapers
| cube2222 wrote:
| Thanks for the explanation and good luck, then! Very curious
| to see what you come up with.
| awinter-py wrote:
| ohhhh this is really smart; every terraform user has at some
| point run up against the limits of it not being a language and
| flirted with pulumi. natural excuse now to get started, esp if
| the auto import works
|
| targeting translation between declarative-ish languages is a
| natural day-1 application for LLMs
|
| declarative languages generally good, bc 1) it's relatively easy
| to check them, 2) they are low-boilerplate (less to get wrong +
| better support for small token memories)
|
| wonder if this backfires bc pulumi is more powerful, giving the
| LLM more chances to get in trouble? like there are cases where in
| order to check a pulumi program you would need something like
| TLA+ with a good model of the system you are targeting
| c7DJTLrn wrote:
| Sorry to be offtopic, but I've been using Pulumi at work for the
| past 6 months and I'm really not impressed. It's basically just
| Terraform but worse, with a million ways to declare your
| infrastructure instead of just one. Infrastructure people tend
| not to write the best code and from my observation the extra
| freedom of an imperative language just makes stuff even more
| complex and harder to maintain. It's also much harder to automate
| than Terraform, I am not aware of any equivalent to Atlantis.
|
| Also, Pulumi previews (equivalent to plans) are complete
| bullshit. If you don't write your code carefully, resources can
| be created and removed and you won't know it's going to happen
| until you start applying... it's an engineer's worst nightmare
| when a tool lies.
| mdeeks wrote:
| Atlantis is basically just a terraform oriented command runner.
| I believe you can create a custom Atlantis workflow that'll
| just run `pulumi preview` or `pulumi up`. That's how Atlantis
| supports terragrunt as well:
| https://www.runatlantis.io/docs/custom-workflows.html#terrag...
|
| Our experience with Terraform is exactly the same as what
| you're describing with Pulumi. We have hundreds of stacks and
| they are all implemented differently because they are written
| by a hundred different people with varying knowledge of
| Terraform. Terraform is a goofy language and it's very hard to
| do even simple things like conditionals. There is also a
| massive amount of copy/paste because people have no idea how to
| set up their provider or remote states and terraform has no
| real way to make those DRY.
| i_play_stax wrote:
| Well said. The constraints of Terraform (HCL2) are a blessing.
| verdverm wrote:
| I've been replacing my TF with CUE -> tf.json
|
| I find it to be the best of both worlds
| nostrebored wrote:
| I think the fact that there are "infrastructure people" vending
| infra for devs is usually evidence of a mistake. If you have
| platform engineers, sure, that makes a ton of sense. But adding
| friction to the developer workflow of waiting for someone else
| to do something that's an API call away is strange
| verdverm wrote:
| It's very valuable to have someone on your team thinking
| about infra all the time.
|
| We know that we are constantly pulled in many directions in
| our industry, and often we take shortcuts to get the work out
| the door. Infra is not a place you typically want to take
| shortcuts. Burdening devs with the infra | ops
| responsibilities is a sure way to security incidents and
| inflated costs.
|
| It does provide a good market for consultants|contractors to
| come in and clean up afterwards.
|
| If we look at this job separation in a different analogy, why
| do we typically separate FE & BE development? b/c people can
| only be expected to be proficient in so much of the stack?
| And you typically want someone around who is proficient for
| each part of the stack?
| re-thc wrote:
| > If we look at this job separation in a different analogy,
| why do we typically separate FE & BE development? b/c
| people can only be expected to be proficient in so much of
| the stack? And you typically want someone around who is
| proficient for each part of the stack?
|
| It's just been a convenient way to divide up the work? When
| FE & BE don't work together and aren't aware of each other
| it's just another mess. And then people invented Backend
| For Frontend to deal with it or GraphQL. We're just adding
| more layers and abstractions and complexities on top.
|
| It's valuable to have someone to be dedicated to
| infrastructure. It's even more valuable for everyone to be
| aware of the whole ecosystem. No 1 lives in a silo.
| verdverm wrote:
| Truth, I've built a lot of automation for a CI / ops
| system. The devs have not taken the time to learn it,
| because business needs them to deliver value to the user.
| This certainly adds pain at times, but overall the
| business is better off with specialization and experts.
|
| In an ideal world, it would be great for everyone to be
| aware, but people have limited time and the systems
| naturally grow large enough that you would have to spend
| all of your time just keeping up with the changes.
| c7DJTLrn wrote:
| I see what you're saying but I also don't. Developers can't
| be expected to maintain the infrastructure too, there's an
| immense amount of work involved to keep it reliable and
| secure.
| dmattia wrote:
| I've also made the switch from managing a few thousand
| Terraform modules to handling most app-code things in Pulumi
| and have run into some of these limitations.
|
| With Terraform + Terragrunt + Atlantis, we created
| https://github.com/transcend-io/terragrunt-atlantis-config and
| had an extremely robust and easy to use flow for updating all
| infra code.
|
| We've since moved to an approach where more of our
| infra/security things are managed in Terraform (like Guardduty,
| SSO, Github repo settings, etc.) maintained by more devops
| folks, and our app code is mostly in Pulumi (lambdas, Fargate,
| CloudFront/CloudFlare CDNs, etc.). To accomplish this without
| something like Atlantis, we moved the app code infra
| deployments from being deployed continuously pre-merge via
| Atlantis to being deployed via `pulumi up` calls in our normal
| CI flows, so like right next to where we build the docker
| images and restart ECS services, as an example.
|
| Overall I actually really love this flow. It is so, so much
| easier to create multi-regional infra in Pulumi with a quick
| for loop, and it's also much easier to do things like run
| esbuild over our code in typescript, and then bundle the output
| of that call and send it up to a Lambda function all from
| pulumi/typescript without needing separate build steps or to do
| things like using terragrunt pre-hooks or Docker build steps
| inside terraform provisioners, which I always found slow and
| clunky.
|
| I would agree that Pulumi's plans are a disappointment though,
| exactly as you said.
|
| Overall I've been happy with the change, and we've seen some
| improvements I think in the velocity that developers can launch
| services that meet our requirements.
| cube2222 wrote:
| You can automate both Terraform and Pulumi in a similar way to
| Atlantis using Spacelift[0] (though generally with a lot of
| additional features and customizability thrown in vs Atlantis;
| but you can also accomplish basically the exact Atlantis flow).
|
| It's a CI/CD system specialized in Infrastructure-as-Code.
|
| [0]: https://spacelift.io
|
| Disclaimer: Software Engineer at Spacelift, though I'm
| recommending it not just because I work there, but because I
| think it's legitimately a very good product.
| c7DJTLrn wrote:
| Thanks for reminding me about this, we came across it a while
| ago. I should book a call. Do you guys support on-premise
| hosting?
| cube2222 wrote:
| Yes!
|
| We support using the SaaS version with self-hosted workers
| and privately hosted VCS systems, as well as (released just
| last week[0]!) a fully self-hosted version of Spacelift
| (specifically, bring your own AWS).
|
| [0]: https://spacelift.io/blog/introducing-spacelift-self-
| hosted
| sandGorgon wrote:
| just wondering - have you tried CDKTF ? what do you think of
| that directions
| evboyle wrote:
| Engineering Manager for the Pulumi Cloud here.
|
| We have an equivalent of Atlantis called Pulumi Deployments[1].
| The benefit of the Deployments platform is that it is entirely
| API driven. In addition to defining CI/CD and deployment in
| configuration/code, we offer APIs that let you do this
| programmatically. Great for platform automation where you are
| setting up hundreds or thousands of stacks.
|
| In addition to `git push` workflows, we also support other
| deployment triggers such as a REST API. This is pretty unique,
| and let's you do things like build RESTful infrastructure APIs
| [2] ontop of the deployments platform.
|
| - [1] https://www.pulumi.com/blog/pulumi-deployments-platform-
| auto... - [2] https://github.com/pulumi/deploy-
| demos/tree/main/deployment-...
| verdverm wrote:
| Agreement, I do not understand this backwards movement in the
| DevOps world. My hypothesis is that they are catering to a
| different group, i.e. enabling developers to do Ops, who don't
| want to learn TF and want to use their preferred language.
| DevOps first practitioners are in short supply, so it makes
| sense there is a market for this.
| re-thc wrote:
| Why is it backwards? Is making it more accessible and
| inclusive backwards? So these "DevOps practitioners" who are
| so different according to you never used Python or any
| programming language?
|
| We should enable everyone to at least aware of Ops and be
| able to contribute. Why does it need to be gated behind a
| special language i.e. HCL?
|
| A lot of times things go rogue exactly because developers
| don't understand and claim to not have a need to understand
| because it's not their job. Ultimately the code runs on the
| infrastructure provisioned just like how we live on Earth
| altogether.
|
| Just like moving to recycling and clean energy the only way
| is to go at it together and not create more divide.
| trallnag wrote:
| Maybe it shines if your infrastructure is really dynamic?
| c7DJTLrn wrote:
| The project I'm on involves dynamic infrastructure. I don't
| see any benefit Pulumi provides over Terraform in this
| regard. In fact, Terraform's module system is more convenient
| if you want to do per-tenant infra versioning for example.
| dmattia wrote:
| You've used Pulumi extensively and don't see a single
| benefit?
|
| Here's setting up Guardduty in AWS in multiple regions
| using a terraform module: https://github.com/gruntwork-
| io/terraform-aws-security/blob/...
|
| In pulumi, it would be:
|
| ``` import * as pulumi from "@pulumi/pulumi"; import * as
| aws from "@pulumi/aws";
|
| [ "us-east-1", "us-east-2", "us-west-1", "us-west-2", "ap-
| south-1", "ap-northeast-2", "ap-southeast-1", "ap-
| southeast-2", "ap-northeast-1", "ca-central-1", "eu-
| central-1", "eu-west-1", "eu-west-2", "eu-west-3", "sa-
| east-1", ].forEach((region) => { const provider = new
| aws.Provider(`aws-provider-${region}`, { region: region,
| }); const guardDuty = new
| aws.guardduty.Detector(`detector-${region}`, {
| enable: true, }, { provider: provider });
|
| }); ```
| verdverm wrote:
| If I use Python for my Pulumi, how could I reuse the work
| of a peer who uses JS for Pulumi?
|
| Do we need to have multiple "mirrors" of our internal
| infra modules? Or do we need multiple language runtimes
| in our deployment runner in CI?
|
| With TF, there is one language and one binary
| dmattia wrote:
| You would have separate JS and Python stacks that could
| share outputs, similar to a `terraform_remote_state` data
| source in terraform-land. There's also a yaml "language"
| support if you just want to use a config language:
| https://www.pulumi.com/docs/intro/languages/yaml/, but I
| have never messed with this, as we really just use
| typescript.
|
| As somebody who maintains public terraform providers
| though, I do want to point out that "With TF, there is
| one language and one binary" is not totally accurate.
| Each provider you install must be installed separately,
| and runs separately as its own process, and the core
| terraform library calls out to the providers via gRPC.
| You can verify this by running `ps` during your next
| terraform run and seeing all the different providers on
| their own processes.
|
| Pulumi actually uses the same gRPC setup, and can even
| communicate with Terraform providers. So the only real
| different part binary wise is the "core" terraform binary
| being replaced by a different "core" pulumi binary based
| on the language you are using, but the rest of the
| providers will all be the same, and you'll run multiple
| languages either way.
|
| If your company only supports one or a few languages in
| your normal production code, you can also just use those
| languages for Pulumi, and still get one language and one
| binary. There's nothing forcing you to use multiple
| different languages.
| verdverm wrote:
| It's still an extra dimension of complexity to support N
| language runtimes. Won't each of them still need the same
| TF providers?
|
| With TF, sure there is a call to fetch those providers &
| modules, but still, I don't have to install N language
| runtimes, which needs its own IaC to build, thus adding
| another process to the mix
|
| I would assume most eng orgs try to limit their Pulumi to
| one language to skirt this issue
| jaxxstorm wrote:
| You build a multi language component:
| https://www.pulumi.com/blog/pulumiup-pulumi-packages-
| multi-l...
|
| You can then publish a language SDK for your supported
| languages, so npm for node packages, pip for python.
|
| The invoking peer needs to have the language they're
| using runtime locally
|
| The python boilerplater is here:
| https://github.com/pulumi/pulumi-component-provider-py-
| boile...
|
| Here's an example component built in Go:
| https://github.com/jaxxstorm/pulumi-productionapp
|
| Then in examples you can see it being using:
| https://github.com/jaxxstorm/pulumi-
| productionapp/tree/main/...
| verdverm wrote:
| Thanks for that blog link! I have been asking this
| question for a while and you are the first to provide an
| actual useful answer.
|
| Can I pass arguments to these components in other
| languages? Still unclear
|
| It looks to limit, or create specific points, of where
| the reuse can happen. Like it is still impossible to use
| a helper function from one runtime in another. So we
| still need to maintain utilities in multiple languages
|
| Either way, it looks like a lot more complexity for
| anyone who goes down this path
| re-thc wrote:
| > If I use Python for my Pulumi, how could I reuse the
| work of a peer who uses JS for Pulumi?
|
| Standardize on the language(s). It's important. Pick 1
| for Pulumi and standardize on it. Even better pick 1 or 2
| for the whole company and standardize on it. In a large
| company...
|
| > Or do we need multiple language runtimes in our
| deployment runner in CI?
|
| Already does happen.
|
| But back to the origin question: you can create Pulumi
| packages that can generate usage in different languages
| as required.
| [deleted]
| c7DJTLrn wrote:
| The link is 404 but maybe you have a point there. I don't
| think this alone is enough to make me want to use Pulumi
| over Terraform though :)
| jaxxstorm wrote:
| Pulumi employee here!
|
| Worth noting that while lots of insights are leveraging LLMs/AI,
| it's only one pillar of the launch today.
|
| The cross cloud search and analytics are really valuable
| additions to the product. On day 1, being able to export all your
| resource definitions to BI intelligence tools and drill down into
| specific areas of interest is something that has traditionally
| been very difficult.
|
| On a personal note, being able to simply search for "who spun up
| this EC2 instance" without traversing through cloudtrail is a
| godsend, too.
___________________________________________________________________
(page generated 2023-04-13 23:02 UTC)