https://hub.qovery.com/guides/engineering/terraform-not-the-golden-hammer/

 GuidesDocsTutorialsForum
Web ConsoleHome 
[ ]
[                    ]
 

  * Guides
  * Docs
  * Tutorials
  * Forum
  * Web Console
  * Home
  *  

engineering

Terraform is Not the Golden Hammer

Feedbacks about mixed usages (Cloud Providers, Kubernetes...)
type: engineeringtechnology: terraform
[deimosfr]
Pierre M.
Pierre is an SRE, and CTO of Qovery. He has 15+ years of experience
in R&D. From the financial to the Ad-Tech industry, he has a strong
knowledge in distributed and highly-reliable systems. He's also the
MariaDB High Performance book author.
Stats
9 min read
Updated Sep 17th, 2021
Contents

  * Overview
  * How we used Terraform
      + HCL
      + GitOps and team usage
      + tfstate
      + Helm management
  * Problems facing
      + Heterogeneous resources management
      + (Too) Strong dependencies
      + No automatic reconciliation
      + Automatic import
  * Advises and suggestion
      + Split
      + Outsource
  * Conclusion

Terraform is probably the most used tool to deploy cloud services.
It's a fantastic tool, easily usable, with descriptive language (DSL)
called HCL, team-oriented, supporting tons of cloud providers, etc.

On paper, it's an attractive solution. And it's easy to start
delegating more and more responsibilities to Terraform, as it's like
a swiss knife; it knows how to perform several kinds of actions
against several varieties of technologies.

Qovery is a platform to help developers to deploy their app on their
cloud account in a few minutes (check it out). Before deploying an
app, Qovery needs to deploy a few services (cloud provider side)
where the app code will be hosted. To do so, we decided to use
Terraform. The main reasons are:

  * Terraform is the industry standard to deploy cloud services.
  * Qovery Engine is open source (https://github.com/Qovery/engine),
    and we wanted to use something that anyone could easily
    contribute to.
  * Terraform is maintained by HashiCorp and by Cloud providers
    themself (trust of good quality and integration)

At the beginning of Qovery, we took shortcuts. We needed to go fast.
Using Terraform as the golden hammer was our shortcut. Based on our
past experiences, we knew the golden hammer didn't exist. We've seen
many companies struggling when they start needing customization. In
the end, you pay the price of using non-adapted tools!

So we were playing with the clock, as we knew it wouldn't fit in the
mid/long run but did not precisely know when it would happen.

This article is a return of experience, explaining where, when, and
how you should use Terraform.

#How we used Terraform

#HCL

First thing to understand is how Terraform works. It's a DSL as I
mentionned earlier, the code looks like this:

Copy

resource "scaleway_k8s_cluster" "kubernetes_cluster" {
  name    = var.kubernetes_cluster_name
  version = var.scaleway_ks_version
  cni     = "cilium"

  autoscaler_config {
    disable_scale_down = true
    estimator = "binpacking"
    scale_down_delay_after_add = "10m"
    balance_similar_node_groups = true
  }

  auto_upgrade {
    enable = true
    maintenance_window_day = "any"
    maintenance_window_start_hour = 3
  }
}

As you can see, it's easily readable and understandable. It supports
AWS, DigitalOcean, Scaleway, and so many other cloud providers.

#GitOps and team usage

You can add this kind of code in a git repository and work with your
team members on the same codebase.

When you run terraform against the Terraform code you've written, it
will generate a tfstate file locally containing the information of
what it has managed, keeping track of what it owns.

Working with Terraform in a team with parallel deployments is not the
default Terraform behavior. You will have to configure a remote
backend (s3+dynamodb, for example) to store the tfstate file.

Copy

terraform {
  backend "s3" {
    access_key = xxx
    secret_key = xxx
    bucket = xxx
    key =xxx.tfstate"
    dynamodb_table = xxx
    region = xxx
  }
}

You'll then have a shared lock mechanism to avoid more than one
person applying a change to the same resources.

#tfstate

When you run terraform, it first refreshes the content of the state
file, comparing what is deployed and what is stored into the tfstate
file. It allows Terraform only to perform change actions on what is
different from the tfstate file. It's very efficient.

#Helm management

Helm doesn't only know how to work with several cloud providers but
also knows how to talk to Kubernetes, Helm...the list is...HUGE! As
you can see on the provider list (https://registry.terraform.io/
browse/providers), there are +1.3k providers available!

So we were using it for Helm. Why? Because it's super useful to
create something on a cloud provider (like an IAM account), get the
results from Terraform, and directly inject them as Helm variables.

To show how easy it is:

Copy

# Create user and attach policy

resource "aws_iam_user" "iam_eks_loki" {
  name = "qovery-logs-${var.kubernetes_cluster_id}"
  tags = local.tags_eks
}

resource "aws_iam_access_key" "iam_eks_loki" {
  user    = aws_iam_user.iam_eks_loki.name
}

resource "aws_iam_policy" "loki_s3_policy" {
  name = aws_iam_user.iam_eks_loki.name
  description = "Policy for logs storage"

  policy = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}
POLICY
}

resource "aws_iam_user_policy_attachment" "s3_loki_attachment" {
  user       = aws_iam_user.iam_eks_loki.name
  policy_arn = aws_iam_policy.loki_s3_policy.arn
}

# Deploy chart with user API credentials

resource "helm_release" "loki" {
  name = "loki"
  chart = "common/charts/loki"
  namespace = "logging"
  create_namespace = true
  atomic = true

  set {
    name = "config.storage_config.aws.access_key_id"
    value = aws_iam_access_key.iam_eks_loki.id
  }

  set {
    name = "config.storage_config.aws.secret_access_key"
    value = aws_iam_access_key.iam_eks_loki.secret
  }

 ...

  depends_on = [
    aws_iam_user.iam_eks_loki,
    aws_iam_access_key.iam_eks_loki,
    aws_s3_bucket.loki_bucket,
    aws_iam_policy.loki_s3_policy,
    aws_iam_user_policy_attachment.s3_loki_attachment,
    aws_eks_cluster.eks_cluster,
  ]
}

And it supports the removal and upgrades for sure!

#Problems facing

At this time, we had the golden hammer, we were super happy about
what we achieved with the time invested. We were able to deploy on
cloud providers (AWS/DigitalOcean), use Cloudflare, deploy with Helm
and perform some operations with Kubernetes. Everything only with
Terraform! So what could go wrong?

#Heterogeneous resources management

The way Terraform manages resources is not homogeneous. Here are a
few examples:

  * When you run Terraform against AWS on the subnets part, it will
    create (anytime you deploy) the missing subnets
  * For some resources like RDS or EKS, it won't check if the
    resource already exists or not. So if it's missing, nothing is
    going to happen as it's marked are deployed in the tfstate file
  * Same for Helm chart deployed, for example, they are marked as
    deployed, so no update will be performed on it until you change
    something

So until you experienced one of those cases, it's hard to know if a
resource (which is not there) will be re-created or not.

#(Too) Strong dependencies

Let me give a frustrating example, let's say I want to deploy:

 1. A Kubernetes cluster (EKS) on AWS
 2. DNS name on Cloudflare
 3. Helm charts on this EKS cluster

I specify dependencies in Terraform with this exact order. I run the
"terraform apply" command to deploy this stack. A few min later,
wowww it works, that is amazing, I'm super excited!

A few days later, I need more resources, so I update the number of
worker nodes in EKS. I run once again the "terraform apply" command.
But for some reason, Cloudflare API doesn't answer and I got
completely stuck there without the possibility to update with
Terraform this field because of linked dependencies.

Same for Helm, I've multiplied the number of charts I wanted to
deploy. If for some reason I have a problem with some, I may be
unable to update values I wanted to update on others' charts, even if
it shouldn't be that important.

I just wanted a dependency order of deployment, not a so hard
dependency between all of them for any kind of updates.

The link between all declared is strong, so strong that you may be
blocked until the problem is resolved (by a third-party provider or a
manual fix from you). In case of issues, when you need to go fast, it
can be a real issue, drastically slowing down the resolution of your
problems.

#No automatic reconciliation

Those who already used a configuration manager (Puppet, Ansible,
Chef...), are familiar with the automatic reconciliation. Just run it
against your infrastructure, and you'll be sure about the end result
if you have any doubts. You'll get what you've asked for!

On terraform it's different, because of the tfstate. All the deployed
elements are stored in the tfstate, re-running terraform won't update
resources that are supposed to be in a specific state but are not.

This is where the biggest behavior comes in with Terraform compared
to configuration managers.

#Automatic import

When you deploy resources and something goes wrong (like an API
returns a timeout failure, but in the end, you have your resource
deployed), Terraform won't store the info in the state file as it
shouldn't exist. Unfortunately, the resource exists and next time you
will run "terraform apply", you'll face a "resource already exists"
message.

There is, unfortunately, no way to automatically recover it
automatically. You need to "import" each resource one by one (https:/
/www.terraform.io/docs/cli/commands/import.html).

This is not convenient at all if you have a team of dedicated Ops/
DevOps/SRE managing it, who will fix it manually. But in the case
your want it to be 100% automatic, it is a problem.

#Advises and suggestion

#Split

If you want to let Terraform manage several kinds of different
resources, I strongly advise to split into different state files and
do not link them all together.

This is not convenient at first because you're losing the strong link
between all of them, but you can overcome this issue with data
sources (https://www.terraform.io/docs/language/data-sources/
index.html).

You'll also need some tooling around it to manage the flow (in which
order all of them have to run).

#Outsource

This is the choice we've made at Qovery, we only kept the minimum
useful to Terraform, so the Cloud part.

Everything Helm/Kubernetes related are managed by our Engine. This
has a lot of advantages (will talk about it in a dedicated post):

  * more flexibility
  * restricting Terraform to what he's perfect at
  * linked resources are strong, and they are really strong
    cloud-only provider side
  * we better fine-grained manage helm lifecycle

#Conclusion

If today, someone is asking me: "Should I use Terraform to deploy
cloud providers infrastructures or services?". I would definitively
say "YES".

But I'll mention that depending on how strong should be the
automatization requested behind it, splitting or delegating some
parts of what should be achieved. It has to come in the balance at a
very early stage.

 
Engineering

Qovery combines the power of Kubernetes, the reliability of AWS, and
the simplicity of Heroku to deploy your apps.

                

Resources

  * Documentation
  * Guides
  * Tutorials
  * Engineering
  * Pricing
  * Enterprise
  * API
  * Github

Community

  * Discord
  * Forum
  * Community call
  * Goodies
  * Roadmap

Company

  * Blog
  * Jobs
  * Team
  * Investors
  * Contact

(c) 2021 DESIGNED BY QOVERY | BACKED BY TECHSTARS | PROUD SILVER MEMBER
OF CNCF AND LINUX FOUNDATION | QOVERY BY BIRDSIGHT - ALL RIGHTS
RESERVED