Terraform: Set Up S3 Cross-Region Replication from Unencrypted Buckets

Terraform provides a reliable method to replicate S3 buckets across regions even when the source buckets are unencrypted. This guide explains how to configure cross-region replication with Terraform, detailing necessary preparations, code structure, and testing practices.

Overview

This guide explains the steps required to establish replication between S3 buckets. The approach uses Terraform to define and manage the AWS infrastructure. Setting up replication across regions improves data availability and ensures backup copies exist in separate geographical locations.

Prerequisites

Before starting the configuration, ensure that you have the following:

AWS Account: Active AWS credentials with permissions to create S3 buckets and configure replication.
Terraform Installed: A recent version of Terraform on your machine.
S3 Buckets: Two buckets are needed: a source bucket in one region and a destination bucket in another region.
IAM Roles and Policies: Policies that allow access to both the source and target buckets.

Ensure the buckets are already set up in AWS. The source bucket does not need encryption for replication to work with this configuration. The replication role should grant the source bucket permission to write objects to the destination bucket.

Setting Up the Terraform Configuration

The Terraform configuration is organized into several parts. Below is a breakdown of the file structure and key components:

Providers: Specify AWS as the provider and configure the regions for each bucket.
Resources: Define the source and destination buckets, along with their configurations.
IAM Roles and Policies: Create an IAM role with policies that permit S3 replication.
Replication Configuration: Apply replication rules to the source bucket.

Providers and Regions

The Terraform configuration must define two providers if the buckets are in different regions. Use the alias feature to differentiate between them. An example configuration is:

provider "aws" {
  region = "us-east-1"
}

provider "aws" {
  alias  = "secondary"
  region = "us-west-2"
}

Defining S3 Buckets

Create the source bucket in one region and the destination bucket in the other region. Specify versioning as replication requires it. An example setup is as follows:

resource "aws_s3_bucket" "source_bucket" {
  bucket = "example-source-bucket"
  versioning {
    enabled = true
  }
}

resource "aws_s3_bucket" "destination_bucket" {
  provider = aws.secondary
  bucket   = "example-destination-bucket"
  versioning {
    enabled = true
  }
}

Configuring IAM Role and Policies

The IAM role enables the source bucket to replicate objects to the destination bucket. Create a role and attach a policy similar to this:

resource "aws_iam_role" "replication_role" {
  name = "s3_replication_role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action    = "sts:AssumeRole"
        Effect    = "Allow"
        Principal = {
          Service = "s3.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_policy" "replication_policy" {
  name   = "s3_replication_policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "s3:GetReplicationConfiguration",
          "s3:ListBucket"
        ]
        Effect   = "Allow"
        Resource = [
          aws_s3_bucket.source_bucket.arn
        ]
      },
      {
        Action = [
          "s3:GetObjectVersion",
          "s3:GetObjectVersionAcl"
        ]
        Effect   = "Allow"
        Resource = [
          "${aws_s3_bucket.source_bucket.arn}/*"
        ]
      },
      {
        Action = [
          "s3:ReplicateObject",
          "s3:ReplicateDelete",
          "s3:ReplicateTags"
        ]
        Effect   = "Allow"
        Resource = [
          "${aws_s3_bucket.destination_bucket.arn}/*"
        ]
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "attach_replication_policy" {
  role       = aws_iam_role.replication_role.name
  policy_arn = aws_iam_policy.replication_policy.arn
}

Adding Replication Configuration to the Source Bucket

Attach the replication configuration to the source bucket by referencing the IAM role. The configuration includes rules that indicate the target bucket and conditions under which replication occurs. An example configuration is:

resource "aws_s3_bucket_replication_configuration" "replication" {
  bucket = aws_s3_bucket.source_bucket.id

  role = aws_iam_role.replication_role.arn

  rules {
    id     = "ReplicationRule"
    status = "Enabled"

    destination {
      bucket        = aws_s3_bucket.destination_bucket.arn
      storage_class = "STANDARD"
    }

    filter {
      prefix = ""
    }
  }
}

Verification and Testing

After applying the configuration with terraform apply, check that the following items are correctly set:

Versioning Enabled: Both buckets must have versioning activated.
IAM Role Permissions: Confirm that the IAM role has permissions to replicate objects.
Replication Rules: Verify the replication configuration in the source bucket.

A quick test can be performed by uploading an object to the source bucket. The object should appear in the destination bucket within a few minutes.

Troubleshooting

Permissions Issues: Validate that the IAM role and policies are correctly attached and allow the required actions.
Bucket Versioning: Confirm that versioning is active on both buckets; replication will fail without it.
Region Mismatch: Ensure that the source and destination buckets are specified correctly in their respective providers.

Final Thoughts

Using Terraform to configure S3 cross-region replication from unencrypted buckets improves data redundancy and regional availability. This configuration keeps your replication process automated and manageable through code. The method outlined in this guide provides a clear, maintainable approach to cross-region replication, ensuring that backups exist in another region and that your data remains accessible even if one region faces issues.

By following this setup, you obtain a structured and effective replication mechanism, which allows for consistent management of AWS infrastructure with Terraform. This article presents a detailed walk-through without unnecessary content, providing a straightforward method to set up cross-region replication that meets your site’s technical needs.

Cloud & Infrastructure