Blog
24 September 2021
Rustam Akhmetkabirov, software engineer

Using Terraformer to adapt existing AWS infrastructure to deploy it with Terraform

This article will share our experience importing and adapting infrastructure configurations deployed manually to AWS to the Terraform format. You may ask, what’s the point of all this? The reasons are many: ensuring fault tolerance, making horizontal and vertical scaling easier, etc. Let’s discuss them.

Why I might need that

The automated infrastructure management solutions allow you to:

  • quickly recreate the infrastructure;
  • maintain the changes history;
  • control the software versions used;

It’s not just a matter of convenience. Lack of such automated solutions directly impacts your business apps’ operation, e.g., the speed of infrastructure problems troubleshooting, its overall fault tolerance, and uptime. This revelation led to the emergence of the IaC approach and its proliferation.

The configuration management systems (Ansible, Chef, etc.) solve this problem at the software services level. However, they do not cover the lower (hardware) level that also needs attention.

Most of our readers are probably aware of solutions covering the hardware level (e.g., Terraform). It comes in handy when you create an infrastructure from scratch. But what if our production cluster includes dozens or even hundreds of components? In this case, writing deployment scripts for each element would require a tremendous effort.

In this article, we will take a different approach and try to import the existing production infrastructure. Next, we will adapt the imported configurations to enable a seamless transition to automated management.

Planning and preparing

We have examined various tools for importing configurations (Terraform itself, the AWS CLI tool, Terraforming) and settled on Terraformer. We liked the completeness and accuracy of the output data produced by Terraformer and its ease of use. The author of Terraformer pursued the goals mentioned above when he faced a similar challenge. In the end, he has created a tool with the capabilities we needed.

It is worth mentioning that we planned to import and adapt the configurations in several unrelated regions. All actions had to be carried sequentially and affect only one region at a time. Thus, we decided to implement a fully operational environment and verify its functionality in a testing environment. We will use a Docker container because it allows for quick deployment of an environment that is tied to specific software versions.

The Docker image will be built and run using the werf tool and its built-in syntax (called Stapel) instead of Dockerfile. However, our preference is motivated by a more convenient/habitual approach and is not due to any architectural constraints meaning that you can easily use plain Docker as well.

We will use a basic Ubuntu image and install Terraform and the Terraformer plugin in it. We will also need to configure Terraform and add a file with AWS credentials. Here is the Stapel manifest for werf (you can efficiently rewrite it to plain Dockerfile if necessary):

configVersion: 1
project: terraform
---
 
{{- $params := dict -}}
{{- $_ := set $params "UbuntuVersion" "18.04" -}}
{{- $_ := set $params "UbuntuCodename" "bionic" -}}
{{- $_ := set $params "TerraformVersion" "0.13.6" -}}
{{- $_ := set $params "TerraformerVersion" "0.8.10" -}}
{{- $_ := set $params "WorkDir" "/opt/terraform" -}}
{{- $_ := set $params "AWSDefaultOutput" "json" -}}
 
{{- $_ := env "AWS_SECRET_ACCESS_KEY" | set $ "AWSSecretAccessKey" -}}
{{- $_ := env "AWS_ACCESS_KEY_ID" | set $ "AWSAccessKeyId" }}
{{- $_ := env "AWS_REGION" | set $ "AWSRegion" }}
{{- $_ := env "CI_ENVIRONMENT_SLUG" | set $ "Environment" }}
 
---
image: terraform
from: ubuntu:{{ $params.UbuntuVersion }}
git:
- add: /
 to: "{{ $params.WorkDir }}"
 owner: terraform
 group: terraform
 excludePaths:
 - "*.tfstate"
 - "*.tfstate.backup"
 - "*.bak"
 - ".gitlab-ci.yml"
 stageDependencies:
   setup:
   - "regions/*"
   - "states/*"
ansible:
 beforeInstall:
 - name: "Install essential utils"
   apt:
     name:
     - tzdata
     - apt-transport-https
     - curl
     - locales
     - locales-all
     - unzip
     - python
     - groff
     - vim
     update_cache: yes
 - name: "Remove old timezone symlink"
   file:
     state: absent
     path: "/etc/localtime"
 - name: "Set timezone"
   file:
     src: /usr/share/zoneinfo/Europe/Tallinn
     dest: /etc/localtime
     owner: root
     group: root
     state: link
 - name: "Create non-root main application user"
   user:
     name: terraform
     comment: "Non-root main application user"
     uid: 7000
     shell: /bin/bash
     home: {{ $params.WorkDir }}
 - name: "Remove excess docs and man files"
   file:
     path: "{{`{{ item }}`}}"
     state: absent
   with_items:
   - /usr/share/doc/
   - /usr/share/man/
 - name: "Disable docs and man files installation in dpkg"
   copy:
     content: |
       path-exclude="/usr/share/doc/*"
     dest: "/etc/dpkg/dpkg.cfg.d/01_nodoc"
 - name: "Generate ru_RU.UTF-8 default locale"
   locale_gen:
     name: ru_RU.UTF-8
     state: present 
 install:
 - name: "Download terraform"
   get_url:
     url: https://releases.hashicorp.com/terraform/{{ $params.TerraformVersion }}/terraform_{{ $params.TerraformVersion }}_linux_amd64.zip
     dest: /usr/src/terraform_{{ $params.TerraformVersion }}_linux_amd64.zip
     mode: 0644
 - name: "Install terraform"
   shell: |
     unzip /usr/src/terraform_{{ $params.TerraformVersion }}_linux_amd64.zip -d /usr/local/bin
     terraform -install-autocomplete
 - name: "Install terraformer provider"
   get_url:
     url: https://github.com/GoogleCloudPlatform/terraformer/releases/download/{{ $params.TerraformerVersion }}/terraformer-aws-linux-amd64
     dest: /usr/local/bin/terraformer
     mode: 0755
 - name: "Make AWS config files"
   shell: |
     set -e
     mkdir "{{ $params.WorkDir }}/.aws"
     touch "{{ $params.WorkDir }}/.aws/credentials"
     touch "{{ $params.WorkDir }}/.aws/config"
     chown -R 7000:7000 "{{ $params.WorkDir }}/.aws"
     chmod 0700 "{{ $params.WorkDir }}/.aws"
     chmod 0600 "{{ $params.WorkDir }}/.aws/credentials"
 beforeSetup:
 - name: "Write AWS credentials"
   shell: |
     set -e
     printf "[default]\noutput = %s\nregion = %s\n" "{{ $params.AWSDefaultOutput }}" "{{ $.AWSRegion }}" >> "{{ $params.WorkDir }}/.aws/config"
     printf "[default]\naws_access_key_id = %s\naws_secret_access_key = %s\n" "{{ $.AWSAccessKeyId }}" "{{ $.AWSSecretAccessKey }}" >> "{{ $params.WorkDir }}/.aws/credentials"
 - name: "Create region directories"
   file:
     path: "{{`{{ item }}`}}"
     state: directory
     owner: 7000
     group: 7000
     mode: 0775
   with_items:
     - "{{ $params.WorkDir }}/regions/{{ $.Environment }}"
     - "{{ $params.WorkDir }}/states/{{ $.Environment }}"
 setup:
 - name: "Init"
   shell: |
     terraform init
   args:
     chdir: "{{ $params.WorkDir }}"
   become: true
   become_user: terraform
 beforeSetupCacheVersion: "{{ $.AWSRegion }}-{{ $.AWSAccessKeyId }}"
 setupCacheVersion: "{{ $.Environment }}-{{ $.AWSRegion }}-{{ $.AWSAccessKeyId }}"

Now it is time to build an image (do not forget to provide the AWS IAM user credentials). You can specify them manually each time you run the command or define them in the CI environment:

# werf build
CI_ENVIRONMENT_SLUG="eu" AWS_SECRET_ACCESS_KEY="XXX" AWS_ACCESS_KEY_ID="YYY" AWS_REGION="eu-central-1" werf build --stages-storage :local

Start the container using the image we built and the same variable values. Note that the difference in values will result in a configuration mismatch error. Since we interact with the existing infrastructure through abstraction layers, you must use the same parameters we used for building the image:

# werf run terraform
CI_ENVIRONMENT_SLUG="eu" AWS_SECRET_ACCESS_KEY="XXX" AWS_ACCESS_KEY_ID="YYY" AWS_REGION="eu-central-1" werf run --stages-storage :local --docker-options="--rm -ti -w /opt/terraform/ -u terraform" terraform -- /bin/bash

The implementation

Initialization

The Terraform initialization stage for installing the provider plugin can be included in the image. Yes, the size of the image will grow. However, you will get a fully functional image you can repeatedly use (which is essential for our main task).

To pre-install a plugin, create a providers.tf file and put it to the corresponding directory in the image:

terraform {
 required_providers {
   aws = {
     source  = "hashicorp/aws"
     version = "~> 3.25"
   }
 }
}
 
provider "aws" {
 profile = "default"
}

Now, initialize the Terraform configuration. If the provider plugin is installed during the building stage, the init command goes last (and you do not need to invoke it manually).

~$ terraform init
Initializing the backend...
~~~
Terraform has been successfully initialized!

Importing

Now it is time to import the resources:

~$ terraformer import aws --path-pattern="{output}/" --compact=true --regions=eu-central-1 --resources=elasticache,rds
aws importing region eu-central-1
aws importing... elasticache
                ~~~
aws importing... rds
                ~~~
aws Connecting.... 
aws save 
aws save tfstate

In our example, we use the following keys:

  • aws — the provider name;
  • --path-pattern — the path to the configuration files generated by the utility;
  • --compact=true — writing configurations for all types of resources into one file. Note that the description of resources, variables, states will be saved to different files;
  • --regions — the provider’s region;
  • --resources — types of resources to import.

Change the current directory to the one created by the import utility, and run init in this directory (to initialize the imported resource configuration):

~$ cd generated/
~/generated$ terraform init

Initializing the backend...
                ~~~
Terraform has been successfully initialized!

Now it is time to do some planning with the imported configuration:

~/generated$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.terraform_remote_state.local: Refreshing state...
                ~~~

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_db_instance.tfer--db1 will be updated in-place
  ~ resource "aws_db_instance" "tfer--db1" {
                ~~~
      + delete_automated_backups              = true
                ~~~
      + skip_final_snapshot                   = false
                ~~~
    }

  # aws_db_instance.tfer--db2 will be updated in-place
  ~ resource "aws_db_instance" "tfer--db2" {
                ~~~
      + delete_automated_backups              = true
                ~~~
      + skip_final_snapshot                   = false
                ~~~
    }

Plan: 0 to add, 2 to change, 0 to destroy.

Planning revealed inconsistencies between the state detected during import and the current resource configuration. And Terraform let us know that it is going to change two aws_db_instance resources. This can be, for example, due to a difference of available sets of instructions of the import utility and Terraform. This situation is quite normal given that the tools we use are developed by different vendors, and one is an abstraction on top of the other. Also, there may be some lag in the available functionality.

In our case, the imported resource configuration and the tfstate file are missing the following two parameters in aws_db_instance resources:

  • delete_automated_backups
  • skip_final_snapshot

To resolve conflicts, let’s find out the current values of these parameters using one of AWS interfaces: web-gui or aws-cli. Add these parameters to the resource config file:

~/generated$ vim resources.tf

  delete_automated_backups = "false"
            ~~~
  skip_final_snapshot = "false"

~/generated$ vim terraform.tfstate

  "delete_automated_backups": "false",
            ~~~
  "skip_final_snapshot": "false",

… and run the planning for the imported configuration once again:

~/generated$ terraform plan
Refreshing Terraform state in-memory prior to plan...

This time, the planning completes successfully:

0 to add, 0 to change, 0 to destroy

Well, the preparatory steps for the adaptation of resources are complete, and the configuration is ready to be applied.

Adding to and merging configurations

There may be situations when you need to update the resource configuration.

Let’s imagine, for example, that we need to add another type of resource to the configuration. To do this, switch to the source directory and import the configuration of this resource type. Note that the path-pattern directory should be different from the previous one since Terraformer does not rename files used for saving the import results. In other words, it will simply overwrite the existing files if you forget to change the directory.

Let’s add the EC2 resources with the NodeRole=node tag to our imported configuration using the --filter key:

~/generated$ cd ../
~$ terraformer import aws --path-pattern="./ec2/" --compact=true --regions=eu-central-1 --resources=ec2_instance --filter="Name=tags.NodeRole;Value=node"
aws importing region eu-central-1
aws importing... ec2_instance
Refreshing state... aws_instance.tfer--i-002D-03f57062-node-1
Refreshing state... aws_instance.tfer--i-002D-009fc9e6-node-2
Refreshing state... aws_instance.tfer--i-002D-0b4b4c7c-node-3
aws Connecting.... 
aws save 
aws save tfstate

Switch to the directory with the imported config and run the terraform init command:

~$ cd ec2/
~/ec2$ terraform init

Initializing the backend...

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Finding hashicorp/aws versions matching "~> 3.29.0"...
- Finding latest version of -/aws...
- Installing -/aws v3.29.0...
- Installed -/aws v3.29.0 (signed by HashiCorp)
- Installing hashicorp/aws v3.29.0...
- Installed hashicorp/aws v3.29.0 (signed by HashiCorp)

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* -/aws: version = "~> 3.29.0"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Perform the planning:

~/ec2$ terraform plan

Make necessary changes (in the manner mentioned above) to the config so that the planning completes successfully:

0 to add, 0 to change, 0 to destroy

… and merge resources.tf files by copying and pasting configuration blocks.

The tfstate files are merged using the built-in Terraform command, state mv. Note that Terraform can only add the state to the target file for a single resource at a time. Therefore, if you have several resources, you need to repeat this operation for each one of them.

~/ec2$ cd ../generated/
~/generated$ terraform state mv -state=../ec2/terraform.tfstate -state-out=terraform.tfstate 'tfer--i-002D-03f57062-node-1' 'tfer--i-002D-03f57062-node-1'

Applying the configuration

It’s time to apply the configuration. You must ensure that no changing or destroying (per Terrafrom’s terminology) tasks are planned. After all, even minor and harmless changes can cause the instance to restart. And in some cases, the resource can even be recreated from scratch. That is why you must strive for the following cherished line: “Plan: 0 to add, 0 to change, 0 to destroy.”

~/regions/eu$ terraform apply
                ~~~
Terraform will perform the following actions:

Plan: 0 to add, 0 to change, 0 to destroy.
                ~~~
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Migrating the state to S3

You can upload the tfstate file to S3 storage for convenience since Terraform will handle all further actions with this state file. The beauty of storing the file in S3 is that it will be blocked during use. As a result, a team of engineers can work simultaneously on a massive infrastructure without the risk of conflicts when launching critical operations. Also, with S3, you can store the file reliably (and version it, too).

Migrating the tfstate file starts with the creation of a separate directory. It will contain the configuration files intended for creating resources responsible for storing the state.

Let’s create a provider configuration first:

provider "aws" {
  region = "eu-central-1"
}

terraform {
  required_providers {
    aws = {
      version = "~> 3.28.0"
    }
  }
}

… and the aws_s3_bucket configuration. This resource provisions the dedicated storage in S3:

resource "aws_s3_bucket" "terraform_state" {
  bucket = "eu-terraform-state"
  lifecycle {
    prevent_destroy = true
  }
  versioning {
    enabled = true
  }
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

It uses the following parameters:

  • bucket — the name of the storage directory;
  • lifecycle { prevent_destroy = true } — prevents the deletion of the resource;
  • versioning { enabled = true } — enables version control for the terraform.tfstate file on the part of the S3 storage;
  • server_side_encryption_configuration — encryption settings.

Let’s also add the aws_dynamodb_table resource to the config. This AWS DynamoDB table stores information about terraform.tfstate locks:

resource "aws_dynamodb_table" "terraform_locks" {
  name = "eu-terraform-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
}

Here, we have:

  • name — the name of the table for storing information about locks;
  • billing_mode — in the PAY_PER_REQUEST billing mode, the payment amount is calculated based on the number of requests to the table;
  • hash_key — the primary key of the table.

Initialize the configuration, then run the plan and apply commands:

~$ terraform init
Initializing the backend...
                ~~~
Terraform has been successfully initialized!

~/states/eu$ terraform plan
Terraform will perform the following actions:

  # aws_dynamodb_table.terraform_locks will be created

  # aws_s3_bucket.terraform_state will be created

Plan: 2 to add, 0 to change, 0 to destroy.

~/states/eu$ terraform apply
Terraform will perform the following actions:

  # aws_dynamodb_table.terraform_locks will be created

  # aws_s3_bucket.terraform_state will be created

Plan: 2 to add, 0 to change, 0 to destroy.

aws_dynamodb_table.terraform_locks: Creating...
aws_s3_bucket.terraform_state: Creating…

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Switch to the directory containing the AWS resource configuration and create a backend.tf file in it with the following lines:

~$ cd ../../regions/eu
~$ vim backend.tf
terraform {
 backend "s3" {
   bucket = "eu-terraform-state"
   key = "terraform.tfstate"
   region = "eu-central-1"
   dynamodb_table = "eu-terraform-locks"
   encrypt = true
 }
}

Any attempt to do something with the current resource configuration will result in an error:

Backend reinitialization required. Please run "terraform init".
Reason: Initial configuration of the requested backend "s3"

To get rid of it, you need to re-run the initialization. Terraform will connect to the S3 backend created earlier and request the terraform.tfstate file. Since the new backend is empty, it will prompt if you want to copy the local state file to S3:

~/regions/eu$ terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Releasing state lock. This may take a few moments...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Terraform has been successfully initialized!

And, lastly, plan the final version of the configuration:

~/regions/eu$ terraform plan
Refreshing Terraform state in-memory prior to plan...
                ~~~
No changes. Infrastructure is up-to-date.

Summary

We have successfully adapted the configuration of the target resources. You can save it to a Git repository to enable versioning and (some) decentralization. The state file is also kept in the S3 backend with versioning and session locking enabled.

All the steps discussed in this article can help you re-create the infrastructure in the shortest possible time if some incident destroys your environment.

Of course, these measures are only a tiny part of those required to fail-proof your application. You also need to keep in mind dynamic data (and their recovery), as well as the ability to migrate between different providers, among other things.

Conclusion

You can import the resource configuration of a particular cloud provider in many ways. The practice shows that any approach involves customization, and you must tailor it to suit your specific tools. Nevertheless, Terraformer helped us to complete the task. Of course, it took us time to research, implement, and debug the process. But once it was complete, we were able to import resource configurations of several clusters within the planned time frames and without any problems on the infrastructure’s side.

When planning such tasks, you need to consider the rationale (or even test the feasibility) of such an import, given the efforts involved. Chances are, for a small number of resources, you’d instead write Terraform configurations from scratch or import the resources one by one using the standard tools and then adapt them. However, automation helps when a large number of resources are involved. With it, you can perform a seemingly impossible task.