Terraform code structures

Terragrunt code structures

Questions related to Terraform code structure are by far the most frequent in the community. Everyone thought about the best code structure for the project at some point also.

How should I structure my Terraform configurations?

This is one of the questions where lots of solutions exist and it is very hard to give universal advice, so let's start with understanding what are we dealing with.

The official Terraform documentation describes all aspects of configuration in details. Read it carefully to understand the rest of this section.

This section describes key concepts which are used inside the book.

Resource

Resource is aws_vpc, aws_db_instance, etc. A resource belongs to a provider, accepts arguments, outputs attributes, and has a lifecycle. A resource can be created, retrieved, updated, and deleted.

Resource module

Resource module is a collection of connected resources which together perform the common action (for e.g., creates VPC, subnets, NAT gateway, etc). It depends on provider configuration, which can be defined in it, or in higher-level structures (e.g., in infrastructure module).

Infrastructure module

An infrastructure module is a collection of resource modules, which can be logically not connected, but in the current situation/project/setup serves the same purpose. It defines the configuration for providers, which is passed to the downstream resource modules and to resources. It is normally limited to work in one entity per logical separator (e.g., AWS Region, Google Project).

For example, module uses resource modules like and to manage the infrastructure required for running on .

Another example is module where multiple modules by are being used together to manage the infrastructure as well as using Docker resources to build, push, and deploy Docker images. All in one set.

Composition

Composition is a collection of infrastructure modules, which can span across several logically separated areas (e.g.., AWS Regions, several AWS accounts). Composition is used to describe the complete infrastructure required for the whole organization or project.

A composition consists of infrastructure modules, which consist of resources modules, which implement individual resources.

Data source

Data source performs a read-only operation and is dependant on provider configuration, it is used in a resource module and an infrastructure module.

Data source terraform_remote_state acts as a glue for higher-level modules and compositions.

The data source allows an external program to act as a data source, exposing arbitrary data for use elsewhere in the Terraform configuration. Here is an example from the where the filename is computed by calling an external Python script.

The data source makes an HTTP GET request to the given URL and exports information about the response which is often useful to get information from endpoints where a native Terraform provider does not exist.

Remote state

Store for each infrastructure module and composition in a remote backend, configured with ACLs, versioning, and logging. This single, authoritative source of truth keeps environments consistent and typically includes disaster-recovery features such as automated backups. Managing state locally can lead to collaboration issues and race conditions when multiple developers run Terraform at the same time, resulting in unpredictable outcomes.

Provider, provisioner, etc

Providers, provisioners, and a few other terms are described very well in the official documentation and there is no point to repeat it here. To my opinion, they have little to do with writing good Terraform modules.

Why so difficult?

While individual resources are like atoms in the infrastructure, resource modules are molecules (consisting of atoms). A module is the smallest versioned and shareable unit. It has an exact list of arguments, implement basic logic for such a unit to do the required function. e.g., module creates aws_security_group and aws_security_group_rule resources based on input. This resource module by itself can be used together with other modules to create the infrastructure module.

Access to data across molecules (resource modules and infrastructure modules) is performed using the modules' outputs and data sources.

Access between compositions is often performed using remote state data sources. There are .

When putting concepts described above in pseudo-relations it may look like this:

is powerful (if not the most powerful out there now) and one of the most used tools which allow management of infrastructure as code. It allows developers to do a lot of things and does not restrict them from doing things in ways that will be hard to support or integrate with.

Some information described in this book may not seem like the best practices. I know this, and to help readers to separate what are established best practices and what is just another opinionated way of doing things, I sometimes use hints to provide some context and icons to specify the level of maturity on each subsection related to best practices.

The book was started in sunny Madrid in 2018, available for free here at .

A few years later it has been updated with more actual best practices available with Terraform 1.0. Eventually, this book should contain most of the indisputable best practices and recommendations for Terraform users.

Source: https://github.com/antonbabenko/terraform-best-practices/tree/master/examples/small-terraform

This example contains code as an example of structuring Terraform configurations for a small-size infrastructure, where no external dependencies are used.

Source:

This example contains code as an example of structuring Terraform configurations for a medium-size infrastructure which uses:

• 2 AWS accounts

• 2 separate environments (prod and stage which share nothing). Each environment lives in a separate AWS account

There is also a workshop for people who want to practice some of the things described in this guide.

The content is here - https://github.com/antonbabenko/terraform-best-practices-workshop

What are the tools I should be aware of and consider using?

• Terragrunt - Orchestration tool

• - Code linter

• - Version manager

• - A modern composable framework for Terraform backed by YAML

• - HashiCorp plugin for the version manager

• - Pull Request automation

• - Collection of git hooks for Terraform to be used with

• - Cloud cost estimates for Terraform in pull requests. Works with Terragrunt, Atlantis and pre-commit-terraform too.

What are the solutions to with modules?

Versions of resource and infrastructure modules should be specified. Providers should be configured outside of modules, but only in composition. Version of providers and Terraform can be locked also.

There is no master dependency management tool, but there are some tips to make dependency specifications less problematic. For example, can be used to automate dependency updates. Dependabot creates pull requests to keep your dependencies secure and up-to-date. Dependabot supports Terraform configurations.

Source:

This example contains code as an example of structuring Terraform configurations for a large-size infrastructure which uses:

• 2 AWS accounts

• 2 regions

- List of people who work with Terraform very actively and can tell you a lot (if you ask them).

- A community of individuals who actively share their Terraform knowledge through content, events, and open collaboration.

- Curated list of resources on HashiCorp's Terraform.

- "Your Weekly Dose of Terraform" YouTube channel by Anton Babenko. Live streams with reviews, interviews, Q&A, live coding, and some hacking with Terraform.

Use locals to specify explicit dependencies between resources

Helpful way to give a hint to Terraform that some resources should be deleted before even when there is no direct dependency in Terraform configurations.

Formatting

Terraform’s terraform fmt command enforces the canonical style for configuration files. The tool is intentionally opinionated and non-configurable, guaranteeing a uniform format across codebases so reviewers can focus on substance rather than style. Integrate it with to validate and format code automatically before it reaches version control.

For example:

In CI pipelines, use terraform fmt -check to verify compliance. It exits with status 0 when all files are correctly formatted; otherwise, it returns a non-zero code and lists the offending files. Centralizing formatting in this way removes merge friction and enforces a consistent standard across teams.

Editor Configuration

• Use .editorconfig: helps maintain consistent coding styles for multiple developers working on the same project across various editors and IDEs. Include an .editorconfig file in your repositories to maintain consistent whitespace and indentation.

Example .editorconfig:

Documentation

Automatically generated documentation

is a framework for managing and maintaining multi-language pre-commit hooks. It is written in Python and is a powerful tool to do something automatically on a developer's machine before code is committed to a git repository. Normally, it is used to run linters and format code (see ).

With Terraform configurations pre-commit can be used to format and validate code, as well as to update documentation.

Check out the to familiarize yourself with it, and existing repositories (eg, ) where this is used already.

terraform-docs

is a tool that does the generation of documentation from Terraform modules in various output formats. You can run it manually (without pre-commit hooks), or use to get the documentation updated automatically.

Comment style

Use # for comments. Avoid // or block comments.

Example:

Section Headers: Delimit section headers in code with # ----- or ###### for clarity.

Example:

@todo: Document module versions, release, GH actions

Resources

3. Blog post by :

General conventions

1. Use _ (underscore) instead of - (dash) everywhere (in resource names, data source names, variable names, outputs, etc).

2. Prefer to use lowercase letters and numbers (even though UTF-8 is supported).

Resource and data source arguments

1. Do not repeat resource type in resource name (not partially, nor completely):

1. Resource name should be named this if there is no more descriptive and general name available, or if the resource module creates a single resource of this type (eg, in there is a single resource of type aws_nat_gateway and multiple resources of typeaws_route_table, so aws_nat_gateway should be named this and aws_route_table should have more descriptive names - like private, public, database).

Code examples of resource

Usage of count / for_each

Placement of tags

Conditions in count

Variables

1. Don't reinvent the wheel in resource modules: use name, description, and default value for variables as defined in the "Argument Reference" section for the resource you are working with.

2. Support for validation in variables is rather limited (e.g. can't access other variables or do lookups if using a version before Terraform 1.9). Plan accordingly because in many cases this feature is useless.

Outputs

Make outputs consistent and understandable outside of its scope (when a user is using a module it should be obvious what type and attribute of the value it returns).

1. The name of output should describe the property it contains and be less free-form than you would normally want.

2. Good structure for the name of output looks like {name}_{type}_{attribute} , where:

   1. {name} is a resource or data source name

Code examples of output

Return at most one ID of security group:

When having multiple resources of the same type, this should be omitted in the name of output:

Use plural name if the returning value is a list