Terraform Controller Practices

Terraform Controller is a Kubernetes Controller for Terraform.

Architecture

---

Get started

---

Github: https://github.com/oam-dev/terraform-controller

Standalone Terraform Controller

---

Install Kubernetes Terraform Controller Chart

$ helm repo add kubevela-addons https://charts.kubevela.net/addons
"kubevela-addons" has been added to your repositories

$ helm upgrade --install terraform-controller -n terraform --create-namespace kubevela-addons/terraform-controller
Release "terraform-controller" does not exist. Installing it now.
NAME: terraform-controller
LAST DEPLOYED: Fri Mar 11 15:08:57 2022
NAMESPACE: terraform
STATUS: deployed
REVISION: 1
TEST SUITE: None

[blazehu@MacBook ~]$ kubectl get all -n terraform
NAME                                        READY   STATUS    RESTARTS   AGE
pod/terraform-controller-557d4b8869-nv28x   1/1     Running   0          22s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/terraform-controller   1/1     1            1           22s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/terraform-controller-557d4b8869   1         1         1       22s
[blazehu@MacBook ~]$ kubectl get crd | grep terraform
configurations.terraform.core.oam.dev            2022-03-11T07:08:27Z
providers.terraform.core.oam.dev                 2022-03-11T07:08:27Z

Authenticate Cloud Provider and Create a Cloud Resource

---

下面验证腾讯云的 cos bucket 创建和销毁，准备以下文件：

.
├── provider.yaml                      # provider 描述文件
├── configuration_hcl_example.yaml     # cos bucket 描述文件
└── tencent-credentials.conf           # 腾讯云相关 secret 信息（secretid & secretkey）

provider.yaml

apiVersion: terraform.core.oam.dev/v1beta1
kind: Provider
metadata:
  name: tencent
spec:
  provider: tencent
  region: ap-nanjing
  credentials:
    source: Secret
    secretRef:
      namespace: vela-system
      name: tencent-account-creds
      key: credentials

configuration_hcl_example.yaml

apiVersion: terraform.core.oam.dev/v1beta1
kind: Configuration
metadata:
  name: tencent-cos-hcl
spec:
  hcl: |
    terraform {
      required_providers {
        tencentcloud = {
          source  = "tencentcloudstack/tencentcloud"
          version = "1.60.5"
        }
      }
    }

    resource "tencentcloud_cos_bucket" "mycos" {
      bucket = "blazehu-test-125834470x"
      acl    = "private"
    }
  providerRef:
    name: tencent

tencent-credentials.conf

secretID: 
secretKey:

create secret

通过 tencent-credentials.conf 文件创建 secret ：

kubectl create secret generic tencent-account-creds -n vela-system --from-file=credentials=tencent-credentials.conf

create provider

[blazehu@MacBook ~]$ kubectl apply -f provider.yaml
provider.terraform.core.oam.dev/tencent created
[blazehu@MacBook ~]$ kubectl get provider.terraform.core.oam.dev
NAME      STATE   AGE
tencent   ready   3m41s

create configuration

[blazehu@MacBook ~]$ kubectl apply -f configuration_hcl_example.yaml
configuration.terraform.core.oam.dev/tencent-cos-hcl created
[blazehu@MacBook ~]$ kubectl get configuration.terraform.core.oam.dev
NAME              STATE                     AGE
tencent-cos-hcl   ProvisioningAndChecking   13s

terraform-controller 会拉起 Job 跑 terraform 命令来创建，源码：

[blazehu@MacBook ~]$ k get job
NAME                    COMPLETIONS   DURATION   AGE
tencent-cos-hcl-apply   0/1           3s         3s
[blazehu@MacBook ~]$ k get po
NAME                          READY   STATUS    RESTARTS   AGE
tencent-cos-hcl-apply-fhpsg   1/1     Running   0          7s
.
.
.
.
.
.
[blazehu@MacBook ~]$ k get configuration.terraform.core.oam.dev
NAME              STATE       AGE
tencent-cos-hcl   Available   4m16s

腾讯云控制台可以观测到 cos bucket 成功创建。

tip: 执行成功后 state 相关内容写入了 secret 里。

delete configuration

[blazehu@MacBook ~]$ k delete -f configuration_hcl_example.yaml
configuration.terraform.core.oam.dev "tencent-cos-hcl" deleted

[blazehu@MacBook ~]$ k get job
NAME                      COMPLETIONS   DURATION   AGE
tencent-cos-hcl-apply     1/1           2m55s      5m11s
tencent-cos-hcl-destroy   0/1           7s         7s
[blazehu@MacBook ~]$ k get pod
NAME                            READY   STATUS      RESTARTS   AGE
tencent-cos-hcl-apply-qnfnx     0/1     Completed   3          3m30s
tencent-cos-hcl-destroy-vt5b2   0/1     Completed   0          10s

腾讯云控制台可以观测到 cos bucket 已经被回收处理。

优化

因为每一个新的操作都是启动一个新的 Pod 去执行 terraform init...，由于网络问题， Initializing provider plugins 经常失败，于是这里使用 cache ，构建一个新的 job 镜像然后更新 release。

FROM oamdev/docker-terraform:1.1.2
LABEL maintainer="blazehu"
ENV TF_PLUGIN_CACHE_DIR /.terraform.d/plugin-cache
COPY registry.terraform.io /.terraform.d/plugin-cache/registry.terraform.io

upgrade release

[blazehu@MacBook ~]$ helm upgrade --install terraform-controller -n terraform --create-namespace kubevela-addons/terraform-controller --set terraformImage=docker-terraform:2.0

tips: 也可以是用 sidecar 注入 pod 来使用 cache （openkruise）

Reference documentation

• https://developer.aliyun.com/article/724349
• https://github.com/oam-dev/terraform-controller/blob/master/getting-started.md