Terraform Plugin Development

Terraform is an open source resource orchestration tool based on Golang, which allows users to manage and configure any infrastructure, the infrastructure of public and private cloud services, and external services.

Overview

Terraform is logically split into two main parts:

• Terraform Core: This is the Terraform binary that communicates with plugins to manage infrastructure resources. It provides a common interface that allows you to leverage many different cloud providers, databases, services, and in-house solutions.
• Terraform Plugins: These are executable binaries written in Go that communicate with Terraform Core over an RPC interface. Each plugin exposes an implementation for a specific service, such as the AWS provider or the cloud-init provider. Terraform currently supports one type of Plugin called providers.
  

Get Started

Clone these template repositories on GitHub: terraform-provider-scaffolding (SDKv2)

Steps:

1. clone the terraform-provider-scaffolding (SDKv2).
2. explore development environment, modify GNUmakefile.
3. define the provider、data_source、resource schema.
4. write code for cos bucket CRUD (internal/provider dir) and acceptance tests.
5. test the provider.
6. generate the provider documentation.

Requirements

• Terraform >= 0.13.x
• Go >= 1.15

Building The Provider

To compile the provider, run go install. This will build the provider and put the provider binary in the $GOPATH/bin directory.

Generate the Provider Documentation

To generate or update documentation, run go generate.

Acceptance tests

In order to run the full suite of Acceptance tests, run make testacc.

Note: Acceptance tests create real resources, and often cost money to run.

Directory Structure

Take cos bucket as an example, modify the directory structure as follows.

terraform-provider-cos
├── README.md     
├── GNUmakefile
├── CHANGELOG.md                                变更日志
├── LICENSE                                     授权信息   
├── main.go                                     程序入口文件
├── docs                                        文档目录
├── examples                                    示例配置文件目录
├── internal                                    Provider核心目录
│   └── provider
│       ├── data_source_cos_bucket.go           bucket查询
│       ├── data_source_cos_bucket_test.go
│       ├── provider.go                         Provider核心文件
│       ├── provider_test.go 
│       ├── resource_cos_bucket.go              bucket资源管理
│       ├── resource_cos_bucket_test.go
│       └── service_cos_bucket.go               封装的bucket相关Service
├── go.mod
├── go.sum
├── terraform-registry-manifest.json
└── tools
    └── tools.go

The structure is mainly divided into five parts:

• main.go, plugin entry.
• provider.go, attributes used to describe plugins, such as: configured key, supported resource list, callback - configuration.
• data_source_*.go, read calls, mainly query interfaces.
• resource_*.go, write calls, including resource addition, deletion, modification and query interfaces.
• service_*.go, public methods divided by resource categories.

Explore your development environment

TEST?=$$(go list ./... | grep -v 'vendor')
HOSTNAME=blazehu.com
NAMESPACE=edu
NAME=cos
BINARY=terraform-provider-${NAME}_v${VERSION}
OS_ARCH=darwin_amd64
VERSION=0.1
#BINARY=terraform-provider-${NAME}_v${VERSION}.exe
#OS_ARCH=windows_amd64
default: testacc

build:
	go build -o ${BINARY}

test:
	go test -i $(TEST) || exit 1
	echo $(TEST) | xargs -t -n4 go test $(TESTARGS) -timeout=30s -parallel=4

# Run acceptance tests
.PHONY: testacc
testacc:
	TF_ACC=1 go test ./... -v $(TESTARGS) -timeout 120m

# Install
.PHONY: install
install: build # Build manager binary.
	mkdir -p ~/.terraform.d/plugins/${HOSTNAME}/${NAMESPACE}/${NAME}/${VERSION}/${OS_ARCH}
	mv ${BINARY} ~/.terraform.d/plugins/${HOSTNAME}/${NAMESPACE}/${NAME}/${VERSION}/${OS_ARCH}

Define provider schema

func New(version string) func() *schema.Provider {
	return func() *schema.Provider {
		p := &schema.Provider{
			DataSourcesMap: map[string]*schema.Resource{
				"cos_bucket_data_source": dataSourceCosBucket(),
			},
			ResourcesMap: map[string]*schema.Resource{
				"cos_bucket_resource": resourceCosBucket(),
			},
			Schema: map[string]*schema.Schema{
				"secret_id": &schema.Schema{
					Type:        schema.TypeString,
					Required:    true,
					Sensitive:   true,
					DefaultFunc: schema.EnvDefaultFunc("SECRET_ID", nil),
				},
				"secret_key": &schema.Schema{
					Type:        schema.TypeString,
					Required:    true,
					Sensitive:   true,
					DefaultFunc: schema.EnvDefaultFunc("SECRET_KEY", nil),
				},
				"region": &schema.Schema{
					Type:        schema.TypeString,
					Required:    true,
					Sensitive:   true,
					DefaultFunc: schema.EnvDefaultFunc("REGION", nil),
				},
			},
			ConfigureContextFunc: providerConfigure,
		}

		return p
	}
}

func providerConfigure(ctx context.Context, d *schema.ResourceData) (interface{}, diag.Diagnostics) {
	region := d.Get("region").(string)
	secretId := d.Get("secret_id").(string)
	secretKey := d.Get("secret_key").(string)

	// Warning or errors can be collected in a slice type
	var diags diag.Diagnostics

	c := &CosClient{
		Region:    region,
		SecretId:  secretId,
		SecretKey: secretKey,
	}

	return c, diags
}

Define bucket data resource schema

resource_cos_bucket.go

func resourceCosBucket() *schema.Resource {
	return &schema.Resource{
		// This description is used by the documentation generator and the language server.
		Description: "Sample resource in the Terraform provider cos.",

		CreateContext: resourceCosBucketCreate,
		ReadContext:   resourceCosBucketRead,
		UpdateContext: resourceCosBucketUpdate,
		DeleteContext: resourceCosBucketDelete,

		Schema: map[string]*schema.Schema{
			"name": {
				// This description is used by the documentation generator and the language server.
				Description: "cos bucket name.",
				Type:        schema.TypeString,
				Required:    true,
			},
			"acl": {
				Description: "cos bucket acl.",
				Type:        schema.TypeString,
				Default:     "private",
				Optional:    true,
			},
			"update_at": {
				Description: "cos bucket create time",
				Type:        schema.TypeString,
				Computed:    true,
			},
		},
	}
}

data_source_cos_bucket.go

func dataSourceCosBucket() *schema.Resource {
	return &schema.Resource{
		// This description is used by the documentation generator and the language server.
		Description: "Sample data source in the Terraform provider cos.",

		ReadContext: dataSourceCosBucketRead,

		Schema: map[string]*schema.Schema{
			"name": {
				// This description is used by the documentation generator and the language server.
				Description: "cos bucket name.",
				Type:        schema.TypeString,
				Required:    true,
			},
			"owner": {
				// This description is used by the documentation generator and the language server.
				Description: "cos bucket owner.",
				Type:        schema.TypeString,
				Computed:    true,
			},
		},
	}
}

Implement Complex Read

func dataSourceCosBucketRead(ctx context.Context, d *schema.ResourceData, meta interface{}) diag.Diagnostics {
	// use the meta value to retrieve your client from the provider configure method
	client := meta.(*CosClient)

	var diags diag.Diagnostics

	name := d.Get("name").(string)
	owner, err := client.GetACLOwner(name)
	if err != nil {
		return diag.Errorf(fmt.Sprintf("get cos bucket owner failed. msg: %s", err.Error()))
	}
	d.SetId(name)
	if err = d.Set("owner", owner); err != nil {
		tflog.Error(ctx, err.Error())
	}

	return diags
}

Implement Create

func resourceCosBucketCreate(ctx context.Context, d *schema.ResourceData, meta interface{}) diag.Diagnostics {
	// use the meta value to retrieve your client from the provider configure method
	client := meta.(*CosClient)

	var diags diag.Diagnostics

	name := d.Get("name").(string)
	acl := d.Get("acl").(string)

	if err := client.Put(name, acl); err != nil {
		return diag.Errorf(fmt.Sprintf("created cos bucket failed. msg: %s", err.Error()))
	}
	d.SetId(name)
	if err := d.Set("update_at", time.Now().Format(time.RFC3339)); err != nil {
		tflog.Error(ctx, err.Error())
	}

	tflog.Info(ctx, fmt.Sprintf("created a cos bucket, name: %s, region: %s", name, client.Region))
	// write logs using the tflog package
	// see https://pkg.go.dev/github.com/hashicorp/terraform-plugin-log/tflog
	// for more information
	return diags
}

Implement Update

func resourceCosBucketUpdate(ctx context.Context, d *schema.ResourceData, meta interface{}) diag.Diagnostics {
	// use the meta value to retrieve your client from the provider configure method
	client := meta.(*CosClient)

	var diags diag.Diagnostics

	if d.HasChange("acl") {
		name := d.Get("name").(string)
		acl := d.Get("acl").(string)

		if err := client.PutACL(name, acl); err != nil {
			return diag.Errorf(fmt.Sprintf("update cos bucket acl failed. msg: %s", err.Error()))
		}
		if err := d.Set("update_at", time.Now().Format(time.RFC3339)); err != nil {
			tflog.Error(ctx, err.Error())
		}
	}

	return diags
}

Implement Delete

func resourceCosBucketDelete(ctx context.Context, d *schema.ResourceData, meta interface{}) diag.Diagnostics {
	// use the meta value to retrieve your client from the provider configure method
	client := meta.(*CosClient)

	var diags diag.Diagnostics

	name := d.Get("name").(string)
	if err := client.Delete(name); err != nil {
		return diag.Errorf(fmt.Sprintf("delete cos bucket failed. msg: %s", err.Error()))
	}
	d.SetId("")
	tflog.Info(ctx, fmt.Sprintf("delete a cos bucket, name: %s", name))

	return diags
}

Implement Cos Bucket Service

import (
	"context"
	"fmt"
	"github.com/tencentyun/cos-go-sdk-v5"
	"net/http"
	"net/url"
)

type CosClient struct {
	SecretId  string
	SecretKey string
	Region    string
}

func (c *CosClient) client(name string) *cos.Client {
	url, _ := url.Parse(fmt.Sprintf("https://%s.cos.%s.myqcloud.com", name, c.Region))
	b := &cos.BaseURL{BucketURL: url}
	return cos.NewClient(b, &http.Client{
		Transport: &cos.AuthorizationTransport{
			SecretID:  c.SecretId,
			SecretKey: c.SecretKey,
		},
	})
}

func (c *CosClient) Put(name, acl string) error {
	opt := &cos.BucketPutOptions{
		XCosACL: acl,
	}
	_, err := c.client(name).Bucket.Put(context.Background(), opt)
	return err
}

func (c *CosClient) GetACLOwner(name string) (string, error) {
	acl, _, err := c.client(name).Bucket.GetACL(context.Background())
	return acl.Owner.DisplayName, err
}

func (c *CosClient) PutACL(name, acl string) error {
	opt := &cos.BucketPutACLOptions{
		Header: &cos.ACLHeaderOptions{
			//private，public-read，public-read-write
			XCosACL: acl,
		},
	}
	_, err := c.client(name).Bucket.PutACL(context.Background(), opt)
	return err
}

func (c *CosClient) Delete(name string) error {
	_, err := c.client(name).Bucket.Delete(context.Background())
	return err
}

Test the provider

1. make install

   [blazehu@MacBook ~]$ make install
   go build -o terraform-provider-cos_v0.1
   mkdir -p ~/.terraform.d/plugins/blazehu.com/edu/cos/0.1/darwin_amd64
   mv terraform-provider-cos_v0.1 ~/.terraform.d/plugins/blazehu.com/edu/cos/0.1/darwin_amd64

2. write demo.tf

   terraform {
     required_providers {
       cos = {
         source  = "blazehu.com/edu/cos"
         version = "0.1"
       }
     }
   }
   
   provider "cos" {
     region = "ap-shanghai"
   }
   
   resource "cos_bucket_resource" "demo" {
     name = "terraform-1251762279"
     acl  = "private"
     # acl = "public-read-write"
   }
   
   data "cos_bucket_data_source" "test" {
     name = cos_bucket_resource.demo.id
   }

3. run terraform init

   Initializing the backend...
   
   Initializing provider plugins...
   - Finding blazehu.com/edu/cos versions matching "0.1.0"...
   - Installing blazehu.com/edu/cos v0.1.0...
   - Installed blazehu.com/edu/cos v0.1.0 (unauthenticated)
   
   Terraform has created a lock file .terraform.lock.hcl to record the provider
   selections it made above. Include this file in your version control repository
   so that Terraform can guarantee to make the same selections by default when
   you run "terraform init" in the future.
   
   Terraform has been successfully initialized!

4. run terraform apply -auto-approve, create a cos bucket

   Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
     + create
    <= read (data resources)
   
   Terraform will perform the following actions:
   
     # data.cos_bucket_data_source.test will be read during apply
     # (config refers to values not yet known)
    <= data "cos_bucket_data_source" "test"  {
         + id    = (known after apply)
         + name  = (known after apply)
         + owner = (known after apply)
       }
   
     # cos_bucket_resource.demo will be created
     + resource "cos_bucket_resource" "demo" {
         + acl       = "public-read-write"
         + id        = (known after apply)
         + name      = "terraform-1251762279"
         + update_at = (known after apply)
       }
   
   Plan: 1 to add, 0 to change, 0 to destroy.
   cos_bucket_resource.demo: Creating...
   cos_bucket_resource.demo: Creation complete after 1s [id=terraform-1251762279]
   data.cos_bucket_data_source.test: Reading...
   data.cos_bucket_data_source.test: Read complete after 0s [id=terraform-1251762279]
   
   Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

5. change acl to public-read-write, run terraform apply -auto-approve

   cos_bucket_resource.demo: Refreshing state... [id=terraform-1251762279]
   
   Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
     ~ update in-place
    <= read (data resources)
   
   Terraform will perform the following actions:
   
     # data.cos_bucket_data_source.test will be read during apply
     # (config refers to values not yet known)
    <= data "cos_bucket_data_source" "test"  {
         ~ id    = "terraform-1251762279" -> (known after apply)
           name  = "terraform-1251762279"
         ~ owner = "qcs::cam::uin/794369159:uin/794369159" -> (known after apply)
       }
   
     # cos_bucket_resource.demo will be updated in-place
     ~ resource "cos_bucket_resource" "demo" {
         ~ acl       = "public-read-write" -> "private"
           id        = "terraform-1251762279"
           name      = "terraform-1251762279"
           # (1 unchanged attribute hidden)
       }
   
   Plan: 0 to add, 1 to change, 0 to destroy.
   cos_bucket_resource.demo: Modifying... [id=terraform-1251762279]
   cos_bucket_resource.demo: Modifications complete after 1s [id=terraform-1251762279]
   data.cos_bucket_data_source.test: Reading... [id=terraform-1251762279]
   data.cos_bucket_data_source.test: Read complete after 0s [id=terraform-1251762279]
   
   Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

6. verify if no changes, run terraform apply -auto-approve

   cos_bucket_resource.demo: Refreshing state... [id=terraform-1251762279]
   
   No changes. Your infrastructure matches the configuration.
   
   Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
   
   Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

7. destroy the resources, run terraform destroy -auto-approve

   cos_bucket_resource.demo: Refreshing state... [id=terraform-1251762279]
   
   Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
     - destroy
   
   Terraform will perform the following actions:
   
     # cos_bucket_resource.demo will be destroyed
     - resource "cos_bucket_resource" "demo" {
         - acl       = "private" -> null
         - id        = "terraform-1251762279" -> null
         - name      = "terraform-1251762279" -> null
         - update_at = "2022-01-29T11:28:31+08:00" -> null
       }
   
   Plan: 0 to add, 0 to change, 1 to destroy.
   cos_bucket_resource.demo: Destroying... [id=terraform-1251762279]
   cos_bucket_resource.demo: Destruction complete after 1s
   
   Destroy complete! Resources: 1 destroyed.

   Tip: You can also retrieve detailed Terraform and provider logs by setting the environment variable TF_LOG. Please include a detailed logs with any bug reports so the author can identify and address the bug. To learn more about log levels and how to interpret a crash log, refer to the Debugging Terraform Documentation.

Reference documentation

• https://www.terraform.io/plugin
• https://learn.hashicorp.com/collections/terraform/providers
• https://cloud.tencent.com/developer/article/1067230