Automate AWS Infrastructure with Terraform & Jenkins – GitOps

In this article, we will learn about Infrastructure As Code (IAC).

We will provision AWS infrastructure using Terraform & Jenkins. We are using Jenkins for creating a pipeline and deploying terraform code.

In this example, we are creating an s3 bucket and ec2 instance using Terraform. You can similarly add any other resource.

Infrastructure as Code – It is the process of configuring resources automatically in a user interface as opposed to manually managing infrastructure in a file, or files.

Gitops – GitOps is a framework for operations that adds infrastructure automation to DevOps best practices, including version control, collaboration, compliance, and CI/CD, which are often used for application development.

Terraform – Terraform is HashiCorp’s infrastructure-as-code service. It is a tool for safely modifying and managing infrastructure in a repeatable manner.

Jenkins- An open-source DevOps solution for continuous integration/continuous delivery and deployment (CI/CD) automation that uses the Java programming language. Implementation of pipelines, often known as CI/CD workflows, uses it.

Terraform state files are created locally by default. This is not ideal when multiple people are working on a project. We will use Amazon S3 to store remote state files with DynamoDB state locking and consistency checking.

Your state files will be stored in the S3 bucket. The DynamoDB table allows you to lock the state file to prevent multiple people from writing to it at the same time.

Steps:

1. Create AWS account and get the details of AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY
2. Install AWS CLI on Linux system
3. Create s3 bucket to store terraform state
4. Create DynamoDB table to lock your Terraform state file
5. Create GitHub repo to store terraform config files and Jenkinsfile
6. Install Jenkins
7. Create Jenkins credentials for github and aws (AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY)
8. Create and run Jenkins’s pipeline

Procedure:

1. Create an AWS account and get the details of AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY

Generate new Access ID and Secret or access existing one using –
http://console.aws.amazon.com
Copy or Download of AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY

2. Install AWS CLI on the Linux system

Follow the below steps on your Linux system to download and install aws cli –

$ python –version
$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
$ unzip awscli-bundle.zip
$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
$ which aws
$ aws configure

3. Create s3 bucket to store the terraform state and enable versioning
Create aws s3 bucket –

$ aws s3 ls
$ aws s3api create-bucket --bucket "tfs-s3-bkt1" --region us-east-1
$ aws s3 ls
$ aws s3api put-bucket-versioning --bucket "tfs-s3-bkt1" --versioning-configuration 
Status=Enabled

Output –

[prayag@panel tf-aws]$ aws s3 ls
[prayag@panel tf-aws]$ aws s3api create-bucket –bucket “tfs-s3-bkt1” –region us-east-1
{
“Location”: “/tfs-s3-bkt1”
}

[prayag@panel tf-aws]$ aws s3 ls
2022-07-13 14:35:46 tfs-s3-bkt1
[prayag@panel tf-aws]$ aws s3api put-bucket-versioning –bucket “tfs-s3-bkt1” –versioning-configuration Status=Enabled
You can also create s3 bucket using AWS GUI Console

4. Create a DynamoDB table to lock your Terraform state file

$ aws dynamodb create-table –table-name tf-backend-lock –attribute-definitions AttributeName=LockID,AttributeType=S –key-schema AttributeName=LockID,KeyType=HASH –provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5
Output ->

[prayag@panel tf-aws]$ aws dynamodb create-table –table-name tf-backend-lock –attribute-definitions AttributeName=LockID,AttributeType=S –key-schema AttributeName=LockID,KeyType=HASH –provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=

{
“TableDescription”: {
“AttributeDefinitions”: [
{
“AttributeName”: “LockID”,
“AttributeType”: “S”
}
],
“TableName”: “tf-backend-lock”,
“KeySchema”: [
{
“AttributeName”: “LockID”,
“KeyType”: “HASH”
}
],
“TableStatus”: “CREATING”,
“CreationDateTime”: “2022-07-13T14:39:33.285000+05:30”,
“ProvisionedThroughput”: {
“NumberOfDecreasesToday”: 0,
“ReadCapacityUnits”: 5,
“WriteCapacityUnits”: 5
},
“TableSizeBytes”: 0,
“ItemCount”: 0,
“TableArn”: “arn:aws:dynamodb:us-east-1:768149447218:table/tf-backend-lock”,
“TableId”: “eab0955e-1870-45c3-96b7-82810f987b3b”
}
}
[prayag@panel tf-aws]$5
You can also create DynamoDB table using AWS GUI Console

5. Create a GitHub repo to store terraform config files and Jenkinsfile

You can clone my repo –

$ git clone https://github.com/prayag-sangode/tf-aws.git
We are using s3 bucket and dynamodb table which we created in earlier steps in provider.tf

$ cat provider.tf
terraform {
backend “s3” {
bucket = “tfs-s3-bkt1”
key = “terraform.tstate”
region = “us-east-1”
dynamodb_table = “tf-backend-lock”
}

}
provider “aws” {
region = “${var.region}”
}

6. Install Jenkins

On Jenkins host install git & terraform –
$ sudo yum install -y yum-utils
$ sudo yum-config-manager –add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
$ sudo yum -y install terraform
$ sudo yum -y install git
Access Jenkins using http://IP:8080
In Jenkins install terraform and aws plugin

7. Create Jenkins credentials for GitHub and aws (AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY)

Create GitHub Credentials.
Create AWS credentials in Jenkins –
In secret add your key – AWS_ACCESS_KEY_ID. Select kind as Secret text
In secret enter AWS_SECRET_ACCESS_KEY. Select kind as Secret text
So, we have below 3 credentials –
No alt text provided for this image
In Jenkinsfile we are using AWS credentials variables as below –
pipeline {

environment {

AWS_ACCESS_KEY_ID = credentials(‘AWS_ACCESS_KEY_ID’)

AWS_SECRET_ACCESS_KEY = credentials(‘AWS_SECRET_ACCESS_KEY’)

}

agent any

tools {

terraform ‘terraform’

}

8. Create and run Jenkins’s pipeline

Create Pipeline, add github credentials for repo and add github repo
Run pipeline and check if build is successful –
Check AWS console if EC2 instance and S3 bucket is created –
Check AWS console if terraform state file exists in s3 bucket (used in terraform backend ) –
Check DynamoDB lock table item –

I hope you found this to be useful in some way.  I’ll be back with some more interesting new cloud and Devops articles soon