[HN Gopher] Create Terraform files using Python scripts
___________________________________________________________________
Create Terraform files using Python scripts
Author : sharjeelsayed
Score : 57 points
Date : 2021-01-02 08:16 UTC (14 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| programmerslave wrote:
| CDK from AWS usurps cloud formation
| antonvs wrote:
| CDK still uses CloudFormation to provision resources, so
| "usurps" is not quite right.
| milkshakes wrote:
| we've been doing this a while with streamalert. nice to see it
| generalized though.
| 0xbadcafebee wrote:
| Does anyone know if there's OSS projects we could use to replace
| Terraform with Python + boto3? I specifically do not want to use
| Pulumi. The dependency validation wouldn't be too hard to
| implement, but all the inter-resource and feature-specific logic
| would need to be duplicated.
|
| The reason I won't use Pulumi is 1) licensing and 2) corporate
| ownership. I'd rather use CloudFormation.
| vr46 wrote:
| Code to generate DSL to call APIs? This is somewhat outdated now,
| although I wouldn't have even used it five years ago TBH.
| Abstraction pile-up.
| busterarm wrote:
| I feel like a lot of people are missing the fact that
| Terraform's value is in being a generic way to interact with a
| vast number of APIs.
|
| But then when I look at things like Pulumi, I'm reminded that
| software engineers have their hammer and tend to see all
| problems as nails.
|
| That's not a knock on the profession, seeing as I am one, but
| writing Terraform serves business needs that writing and
| deploying software does not. It takes a certain amount of
| maturity across the industry to not tunnel vision and write
| code to solve every problem.
| beastman82 wrote:
| This exists because terraform DSL, yaml, and Json are not
| programming languages and are thus tragic choices for making
| declarative statements
| BaronSamedi wrote:
| After many years working with these tools I agree. Declarative
| and template DSLs are OK for basic things but you quickly bump
| into their limitations when you want to do advanced
| configurations. At this point administrators should know how to
| code so being able to use a normal programming language with
| IaC modules should be the standard.
|
| Amazon's CDK for AWS is a step in the right direction.
| Microsoft's new Bicep for Azure misses the mark in my view
| because it is yet another DSL and not a real programming
| language.
| hardwaresofton wrote:
| I have been saying this for a while now -- as much as I respect
| Hashicorp and admire their engineering, HCL is the biggest
| problem with the tools they have that it's used in -- biggest
| of which being Terraform.
|
| Don't use a DSL where a full programming language is what you
| need. I can see that they probably wanted to be able to skip
| writing 2/3 language CDKs out of the box, and DSLs probably hit
| on the need for simplicity really early, but infrastructure
| code is somewhere you really want to be able to drop into the
| full expressive power of a programming language.
|
| I use Pulumi[0] for my projects, and while I doubt people will
| be able to get it as much adoption as I think it should have in
| corporate environments, Terraform has introduced the CDK[1]
| which is a similar approach. Have a talk with your engineers
| about complexity (and how not to strangle the rest of the team
| with it) and stop using DSLs in places you need more expressive
| power.
|
| I think a good indication of when you need to stop using DSLs
| or when you should maybe realize a DSL wasn't the right way to
| go is when you start doing things like writing control flow.
| You could can get away with DSLs if you were using
| s-expressions, but that is an exception that proves the rule
| (because there happens to a family of languages that treat
| s-expressions as syntax).
|
| [0]: https://www.pulumi.com
|
| [1]: https://www.hashicorp.com/blog/cdk-for-terraform-enabling-
| py...
| busterarm wrote:
| The moment you need control flow to define your resources,
| I'd argue that you're verging away from the realm of
| declarative infrastructure.
|
| I'm using Terraform to manage 10^4 machines in combination
| with sane CI/CD, Bash/JQ (for dealing with Terraform
| outputs), Packer and Ansible. Everytime I see somebody
| reaching to a full programming language to define their
| infrastructure, they seem to be doing too much with one tool.
|
| Terraform should merely provision things and in that role I
| find it fine as is. Preferred, even.
| 0xbadcafebee wrote:
| I really dislike the idea of declarative infrastructure.
| It's literally a program that is designed to do one thing,
| but will change a million things in order to do that one
| thing. It's Configuration Management for Infrastructure.
| Yet so many people have this idea that it's something else,
| like it's supposedly simpler or more effective.
|
| Saying _I want an S3 bucket named "foo"_ is the same as
| aws s3 list-buckets | grep "foo" || aws s3 create-bucket
| --bucket "foo"
|
| Did I need a big fat declarative infrastructure system to
| make that? No. But people want more complexity and
| features, and they want to make it _look_ simple. So they
| write big fat applications and libraries to do that. The
| idea that there 's some inherently superior difference of
| "declarative programming" over "regular programming" is
| giving people the idea that wrapping everything up in
| interfaces somehow removes the complexity, or somehow ends
| up in a better program.
| busterarm wrote:
| Yes but AWS CLI commands change over time and don't have
| a native way of maintaining which version of the CLI you
| use. Also, you have to maintain that knowledge for
| however many things you have to do across however many
| providers.
|
| The point of Terraform isn't to add complexity, it's to
| have a general way of interacting with a vast number of
| APIs that's effectively the same and to abstract away the
| tribal knowledge of knowing how each individual API
| works.
|
| On the same provider version, you generally can expect
| Terraform to work the same over time (okay this is less
| true for say Google provider...) as the CLI keeps
| evolving.
|
| It's still helpful to understand the providers and their
| CLIs, but Terraform is a substantial force multipler
| because of how generic it is across the absurdly long
| list of APIs that it talks to. That is what its value is.
| hardwaresofton wrote:
| > The moment you need control flow to define your
| resources, I'd argue that you're verging away from the
| realm of declarative infrastructure.
|
| Declarative infrastructure shouldn't be pursued for it's
| own sake -- what I want is efficient, and simple to manage
| infrastructure automation. The declarative nature is
| awesome, but once you start doing plumbing of variables and
| complexity from one static script from another, the
| cognitive load of keeping this all in line is better
| managed with a programming language in my opinion, you're
| just choosing bash/jq/awk/etc instead of a different
| language.
|
| I think "the way the declarations are made must be static
| files" is dogmatic or at least limiting for me. Yes it is
| absolutely the simplest way to view what's present, but the
| problem is when someone goes into change any of this they
| will be dealing with your bolted-together complexity (even
| if it's not very complex).
|
| > I'm using Terraform to manage 10^4 machines in
| combination with sane CI/CD, Bash/JQ (for dealing with
| Terraform outputs), Packer and Ansible. Everytime I see
| somebody reaching to a full programming language to define
| their infrastructure, they seem to be doing too much with
| one tool.
|
| > Terraform should merely provision things and in that role
| I find it fine as is. Preferred, even.
|
| I can't argue with the efficiency and efficacy of your
| setup, but I don't think much of this has to do with what
| we were discussing -- Pulumi does not seek to do the jobs
| of those other tools -- it's not going to build your VM
| images or do provisioning (unless you use it that way like
| with terraform[0]).
|
| Here's a concrete example of a benefit I got form using
| Pulumi over terraform recently, in some code working with
| SES: import * as fs from "fs";
| // ... more imports and other lines // Email
| Access key const emailAccessKey = new
| aws.iam.AccessKey( `${stack}-ses-access-key`,
| {user: emailUser.name} ); export const
| emailUserSMTPPassword = emailAccessKey.sesSmtpPasswordV4;
| export const emailUserSecret =
| emailAccessKey.encryptedSecret; // Write the
| smtp username and password out to a local secret file
| const apiSecretsDir = path.join(__dirname, "secrets",
| "api", stack); const smtpUsernameFilePath =
| path.resolve(path.join(apiSecretsDir,
| "SES_USERNAME.secret")); const smtpPasswordFilePath
| = path.resolve(path.join(apiSecretsDir,
| "SES_PASSWORD.secret"));
| emailAccessKey.sesSmtpPasswordV4.apply(password => {
| console.log(`Writing SES SMTP username to
| [${smtpUsernameFilePath}]`);
| fs.writeFileSync(smtpUsernameFilePath, emailUsername);
| console.log(`Writing SES SMTP password to
| [${smtpPasswordFilePath}]`);
| fs.writeFileSync(smtpPasswordFilePath, password);
| });
|
| I wanted to write information out to a file... So I just
| did, and that was it. No need to reach for the stack output
| later and pipe it anywhere -- any time pulumi runs it will
| update that variable if/when it changes, and the next tool
| (which requires the file at that path to be present) will
| continue on without knowing a thing.
|
| I can't say that this is perfect Pulumi code (ex. I could
| have defined a custom Resource to do this for me), but I
| have saved myself having to do the plumbing with bash
| scripts and terraform output awk-ing, and the information
| goes just where I want it (NOTE: the secrets folder is
| encrypted with git-crypt[1]). When someone comes to this
| file (ses.ts), they're going to be able to easily trace
| where these values where generated -- similar with bash
| scripts, but now they don't have to be a bash/awk/jq master
| to manipulate information. There are definitely _some_
| gotchas to using Pulumi (like the `.apply` there), but in
| the end, I 'd prefer to make changes like this in a
| consistent language I like (Typescript).
|
| My toolkit looks very similar to you, except I basically
| only use make + kubectl + pulumi + ansible (rarely, because
| of the kind of servers I rent).
|
| [0]: https://www.terraform.io/docs/provisioners/
|
| [1]: https://www.agwa.name/projects/git-crypt/
| busterarm wrote:
| You can just write information out to files in Terraform
| with no stress.
|
| In terraform resources this is what that looks like:
| resource aws_iam_user email { name = "email"
| } resource aws_iam_access_key email {
| user = aws_iam_user.email.name }
| resource local_file smtp_password { content =
| aws_iam_access_key.email.ses_smtp_password_v4
| filename = "SES_PASSWORD.secret" }
|
| So what's the plumbing that you would have to do? Under
| the hood, Pulumi is using the Terraform providers...
|
| (I left out username, because I don't see where you're
| setting emailUsername)
|
| In my pipelines I don't bother writing out to files
| things that are in terraform state. I just create an
| output for that state (potentially set to sensitive) and
| then use that output in my CI/CD. Remote state stays
| encrypted and without wide access and I don't have to
| worry about secrets being in files anywhere.
|
| That's where the bash scripts do things with outputs. It
| could by python or whatever, it doesn't matter really.
| But with bash I can easily just set variables to
| `terraform output -json | jq <select output &/|| do
| stuff>`.
|
| Mainly all I do is write terraform outputs to vault (i
| have simple bash automation to do all of this) and then I
| can use the Vault secrets in other CI/CD pipelines.
| throwaway894345 wrote:
| It's not flow control so much as sane, expression-based
| generation of resources.Terraform has been evolving toward
| the dynamic with features like for_each, but these features
| have awful ergonomics compared to something like list
| comprehensions. Similarly, sometimes you want to reuse some
| pattern, but the unit for reuse in terraform is the modules
| which involves a lot of ceremony, so you don't reach for it
| as often as you might if the unit of reuse was a simple
| function definition.
|
| I don't especially care if Terraform is a simple static
| language for generating resources and the DRY-ness comes
| from an exterior Python/etc script that generates the
| Terraform files or whether the requisite dynamism is built
| into the terraform language itself, but make no mistake,
| the dynamism is absolutely essential for maintainable
| terraform.
| WatchDog wrote:
| Oh wow, I had no idea that terraform had implemented a CDK
| frontend. That's awesome. I've been writing a bunch of
| regular CDK code, which is AWS specific, I loathe HCL, and I
| was a bit disappointed by the quality of pulumi when I
| evaluated it. Il have to check out the terraform CDK support.
| temikus wrote:
| Care to share your thoughts on Pulumi quality? I'm asking
| since I'm currently looking at their TypeScript
| implementation to use in prod and would love to know any
| issues that may pop up.
| cyberpunk wrote:
| I'm super curious about pulumi -- I just inherited a mess of
| terragrunt at a new gig and was plannning to rewrite it
| anyway, is it mature enough for a reasonably standard aws/eks
| setup?
| hardwaresofton wrote:
| As much as I like Pulumi, no one gets fired for choosing
| Terraform. You're going to be able to find the most help,
| discussion, and stuff for Terraform. Terraform and it's
| ecosystem is the safer choice, especially when the CDK is
| an option.
|
| On the other hand I can say that a bunch of Pulumi's stuff
| is built on terraform underneath the covers so I'm not sure
| how far behind they are but it probably isn't by much. In
| my limited use (for example deploying SES stuff) I found it
| a pleasure to use and didn't find anything that terraform
| did that it didn't (again, likely due to pulumi being able
| to utilize terraform providers under the covers).
|
| > I just inherited a mess of terragrunt at a new gig and
| was plannning to rewrite it anyway
|
| Hope you're really sure this is where you want to use your
| effort tokens -- I'm not sure how much of a mess with it,
| but these tools are so new, it might be worth seeing if you
| can de-clutter it without a complete rewrite. Or maybe it's
| small enough that a complete rewrite is relatively low
| friction... Either way, there be dragons.
| mixedCase wrote:
| > but infrastructure code is somewhere you really want to be
| able to drop into the full expressive power of a programming
| language.
|
| Please expand on what this expressive power you _need_ is. I
| see things like Pulumi and I can only think of IaC codebases
| that end up being turing tarpits in the hands of developers
| that don 't get what the whole point of the declarative model
| is, putting a bunch of untestable arbitrary IO and high
| cyclomatic complexity in the middle of determining what is
| going to be deployed.
|
| And this is definitely not an argument against having better
| languages than HCL, Dhall would certainly be a step up that
| doesn't give the developer an arsenal of footguns to move
| fast and wreak havoc on the underlying foundations of a
| business. But I want to know what needs you have that you
| feel are worth the risk of putting a bull in the china shop.
| eat_veggies wrote:
| It seems like Nix and NixOps are designed for this type of
| thing, since they're built on a functional language instead of
| yaml. But I haven't played with them too much yet -- anyone
| have experience using them?
| zerubeus wrote:
| I love you
| fishnchips wrote:
| Not sure about that. It's a clever trick alright but it adds a
| possibly unnecessary layer of leaky abstraction and makes code
| reuse more difficult when working with "vanilla" Terraform. If
| you really want to use a full programming language with infra as
| code, I'd suggest looking into Pulumi. Or, if you want to (or
| need to) stick to Terraform, try the official CDK.
| [deleted]
| TabTwo wrote:
| Now we need some Perl scripts to generate the Python scripts.
| sunshinerag wrote:
| and then we need a DSL to get rid of all the imperative Perl.
| erik_seaberg wrote:
| I feel I should take this opportunity to confess that I once
| used M4 to generate HCL, before I figured out Terraform JSON.
| ohuhu wrote:
| Planning any support for Azure?
| pram wrote:
| Terraform added a lot of these procedural functions natively in
| the time since this was created.
|
| At any rate if you want to stick with Python, I'd say Pulumi
| would be a safer choice these days.
| jpalomaki wrote:
| Yes, Pulumi uses the Terraform providers, and allows you script
| your stuff in Typescript/Python/Java/C#.
|
| https://www.pulumi.com/
| verdverm wrote:
| How do Pulumi models in different languages interact or
| interop?
| 7sigma wrote:
| Not an expert but it is possible to mix and match with the
| new automation api. The example below mixes typescript with
| go.
|
| https://github.com/pulumi/automation-api-
| examples/tree/main/...
| verdverm wrote:
| Looks like you get a runtime explosion mad then some.
| Your CI will need all of the runtimes used plus the extra
| code to handle multiple language runtimes in Pulumi via
| the automation api.
|
| Anyhow, thanks for the link.
| jpalomaki wrote:
| I guess typically you would pick one language and use
| that throughout the Pulumi project.
| verdverm wrote:
| As would I. I see this becoming a pain for the operator
| of the build systems in a mutli-team environment.
| ramaro wrote:
| Kapitan allows you to achieve the same using (python) Kadet to
| compile JSON tf files https://kapitan.dev/compile/#kadet
| mirekrusin wrote:
| I find cue lang [1] very practical/useful/terse, their idea of
| lattice based type system is very intuitive and expressive with
| very little code.
|
| [1] https://cuelang.org/
| codewithcheese wrote:
| So happy (because cue is a great solution) and surprised (it's
| niche atm) to see cue mentioned!
|
| I invite readers to try the cue/Kubernetes tutorial, its
| inspired:
| https://github.com/cuelang/cue/blob/v0.2.2/doc/tutorial/kube...
|
| Here the language creator explains the inspiration and the
| concept, then he works through the above tutorial
| https://www.youtube.com/watch?v=b3fhA12KS48
| verdverm wrote:
| With Cue, there is the possibility to import TFs definitions
| into Cue, write your IaC, and output to JSON. Like parent says,
| Cue feels like the right way to approach this IaC problem with
| a purpose built language. It was designed to manage this
| complexity.
| kesor wrote:
| Why not just use Python to create the "things" in whenever you
| need to create them in the first place?
|
| Just write the code taking idempotency in mind, and no reason to
| use Terraform at all.
| vbernat wrote:
| You will hand up recreating a lot of what Terraform is doing.
| It is not a matter of just creating resources if they don't
| exist. Notably, modifying existing resources in the right order
| is not trivial.
| fennecfoxen wrote:
| You'd think it would be possible to make like Python bindings
| or something though.
| mixedCase wrote:
| > Just write the code taking idempotency in mind
|
| In the real world, this is akin to "just don't write bugs".
| dkdk8283 wrote:
| What I don't get about "modern" development is lack of complexity
| management.
|
| It feels like 8 degrees of work to write HCL that any component
| developer can do.
|
| What maddened me was the lack of support for the count parameter
| for modules. Made me rage. But not enough to switch to tools like
| this or troposphere.
|
| Not to mention the supply chain implications and security risk
| that comes with it.
| erik_seaberg wrote:
| I don't know about the author's issues, but I wrote something
| like this because early versions of HCL were _very_ limited.
| E.g., you couldn 't write a module that cranks out autoscaled
| DynamoDB tables per region because you could only pass scalars
| to modules, not entire key schemata or tag sets.
|
| I really wish Terraform itself were written in Python or
| something similarly hackable without rebuilding the binary for
| everyone. I've glanced at CDK but it looks like building blocks
| for reinventing our own Terraform.
| [deleted]
| busterarm wrote:
| > the lack of support for the count parameter for modules. Made
| me rage.
|
| Supported as of Terraform 0.13.
|
| It was on the roadmap for a really long time, but because of
| early design decisions took a while to get there.
| birdyrooster wrote:
| Did someone just put me in a time machine to 10 years ago?
___________________________________________________________________
(page generated 2021-01-02 23:02 UTC)