Building IaC Pipeline on AWS with Security Fully Integrate

In my last article (Link) we reviewed Infrastructure-as-code (IaC), why it’s important, why many companies are using it, and possible ways to add security to the IaC pipeline.

We know cloud developers often find themselves in high-pressure scenarios, forced to meet deadlines, and deliver fast — and that may push them to not pay full attention to coding and configuration. Now, without proper configurations, organizations are prone to security breaches. The bottom line is, it’s important to ensure architectural best practices are followed across the development process, even on the tightest of timelines.

Now lets work on a real example of how to build engineering best practices into the IaC pipelines automatically for an AWS environment through hundreds of checks that align to the AWS Well-Architected Framework (security, cost optimization, performance, operational excellence, and reliability) and other compliance standards.

Here is a high-level view of what we will be working on:

Initial Requirements

As part of this scenario, you will need the following items to be able to build this environment:

Let's start building your first Infrastructure-as-Code pipeline with Best-Practices Checks!

1 — Install the Security plugin in the IDE and enable the API Token to scan CloudFormation Templates

After installed the VSCode IDE remember to install the Cloud Conformity Security Plugin as you can see here:

VSCode Marketplace— Cloud Conformity Security Plugin LINK

Cloud One — Conformity Template Scanner Extension

After you create your account in Cloud One — Conformity using this link here (Create Account), login into your account, and generate an API Token.

In Conformity — Click in the username on the top rights -> User Settings -> API Keys -> New API Key to create an API Key to be used in the VSCode Plugin. Remember to copy the key and save it safely. You won't be able to get it again.

Copy the API Key and go back to VSCode 💻

1. Click on the Extensions icon (left side) and click in Extension Settings ⚙️ for the Cloud Conformity Template Scanner entry.

2. Select Edit in settings.json on the Cc. ApiKey section.

3.Input the API Key you generated on the previous step and save.

Now you will be able to scan the CloudFormation templates based on hundreds of checks that help you comply with the AWS Well-Architected Framework among others Standards & Frameworks to ensure you are building superior cloud infrastructure.

Here is one example of a CloudFormation template not following engineering best practices for you to test it:

To test the extension, open the CloudFormation Template above with VSCode and open the Command Pallet with:

  • MAC OS: ⇧ + ⌘ + P
  • Windows/Linux: Ctrl+Shift+P

Search for Cloud One Conformity: Scan the Current Open Template and hit <Enter> it will automatically start scanning your CloudFormation template:

The result will appear in a second tab called Scan Result like the image below. You can also check out the Cloud Knowledge Base, which can help you understand more about any best practice check violations and how to remediate/fix it in your CloudFormation template or in Production environments:

Awesome the first stage for an IaC Security automation is done. 😃🤖💻

2 — Create a CI/CD pipeline using AWS tools and integrate the Conformity Template Scanner into it

In the first stage, we show how you could scan before committing a new version of the code to the code repository, but sometimes the developer can forget to do it and commit the CFT with some issues. Now we will be able to create a security gateway to help you prevent new resources from not following best practices recommended by your company.

This can help you to monitor and audit any new change in real-time before you deploy in the AWS environment. 🤯

In this chapter, we will show how to use Template Scanner in the CI/CD pipeline with AWS CodeCommit, AWS CodeBuild, AWS CodePipeline. Let's do it 🛫.

CodeCommit will help us to host our code as a code repository. Many people and companies across the globe use GitHub, GitLab, BitBucket for this purpose. To allow your VSCode to push code to CodeCommit we will need to do some configuration.

Creating a Git Repository on AWS CodeCommit

  • Create IAM user with permission to CodeCommit and only access via SSH key. (You can see details about Git permission and recommendation here in this link from AWS)

If you go to Security credentials -> HTTPS Git credentials for AWS CodeCommit you will be able to generate

Download the credentials and secure keep it.

Additional information from AWS about these steps -> LINK

  • Create AWS CodeCommit repository.

You can run git clone in your machine and easily bring the git config to your computer.

git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/{YOUR REPOSITORY NAME}

Now you could build your own CloudFormation or use the bad CloudFormation Template that I shared before to push to AWS Codecommit.

git add .
git commit -m "First Commit"
git push

Now that you could push your first commit to AWS CodeCommit let's move to AWS CodeBuild and AWS CodePipeline.

Creating a CodeBuild to scan your CloudFormation templates Automatically

AWS CodeBuild is a very similar solution to GitHub Actions, Azure DevOps, and GitLab. It is a CI/CD technology that you can use to build new projects, run automation, and deploy new applications or infrastructures.

We will use CodeBuild with a specific container image to run the Conformity Template Scanner to recognized possible issues before you deploy the new IaC in production.

  • Create a Build Project in AWS CodeBuild

In my case, I will be using a Standard Image from AWS, but you could use other Manage Image or build your own for this kind of automation.

Here is the configuration for the image below:

Environment image: Managed Image
Operationg System: Amazon Linux 2
Runtime: Standard
Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
Image version: Always use the latest
Environment type: Linux
Service Role: New
Role Name: <NEW ROLE NAME>

Here is the configuration for the image below:

Environment Variables
CC_API_KEY = <YOUR API KEY FROM CLOUD ONE - CONFORMITY>
CC_REGION = <REGION SELECTED TO CREATE YOUR CONFORMITY TENANT>
CC_RISK_LEVEL = <RISK LEVEL NOT BE ACCEPT>
CFN_TEMPLATE_FILE_LOCATION = <YOUR CLOUDFORMATION TEMPLATE PATH>
STACK_NAME = <THE CLOUDFORMATION STACK NAME>

Here is the configuration for the image below:

Build specifications: Insert build commands
Build Commands:
version: 0.2phases:
install:
runtime-versions:
python: 3.7
pre_build:
commands:
- pip3 install awscli --upgrade --user
build:
commands:
- pip3 install -r https://raw.githubusercontent.com/OzNetNerd/Cloud-Conformity-Pipeline-Scanner/master/requirements.txt
- wget https://raw.githubusercontent.com/OzNetNerd/Cloud-Conformity-Pipeline-Scanner/master/src/scanner.py
- CC_API_KEY=`jq -r '.CC_API_KEY' <<< $CC_API_KEY`
- python3 scanner.py

post_build:
commands:
- aws cloudformation deploy --template-file $CFN_TEMPLATE_FILE_LOCATION --stack-name $STACK_NAME --no-fail-on-empty-changeset

Buildspec.yml GitHub link -https://raw.githubusercontent.com/fernandostc/IaC-Security-Automation/master/buildspec.yml

Noted: The template scanner that I'm using in this example was built by Will Robison — Solution Architect from Trend Micro — Here is the GitHub link from the project: https://github.com/OzNetNerd/Cloud-Conformity-Pipeline-Scanner

Now click "Create build project" to complete the process. 😃💻

Create a Secret Key inside Secret Manager to keep the Cloud One — Conformity — API Key

Now click the “Store” to complete the process. 😃💻

Important: Remember to create a Policy to attach to your role service used in the CodeBuild with the following permission to allow the CodeBuild to get the value from the SecretManager:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:us-east-1:<AWS Account ID>:secret:<Secret Key ARN>"
}
]
}

Creating AWS CodePipeline to automate trigger to run the Template Scan

AWS CodePipeline will help you to automate the process to run the Template Scanner every time that you make some update in your IaC. You could create multiple stages in CodePipeline, but our example will be a simple one just to make It very simple to understand.

Let's create a new pipeline on AWS CodePipeline

You can define the name for the Pipeline and select New service role if you don't have any service role defined before for it.

Now you will need to define the following parameters based on the CodeCommit that we created before:

Source Provider: <AWS CodeCommit>Repository Name: <CloudOneConformity or the repository name that you gave before>Branch Name: <master>

We need to define the information about the Build process now:

Build provider: <AWS CodeBuild>Project Name: <IaC-Security-Automation or the repository name that you gave before>

In this case, we are not planning to use Deploy because we are doing the deploy inside CodeBuild as part one of the stages. You can click on "skip the deploy stage".

You can review the configuration and after click on "Create pipeline"

Testing the automated pipeline

1 — Create your CloudFormation template or use this one here for the testing process: Link

2 — Scan your CloudFormation template with Cloud One — Conformity plugin and check the result.

Check that still had some issues to resolve before committing to production

3— You can commit it to check if it's working inthe pipeline just as a test before fixing all the issues in the CloudFormation template

4 — Commit the new code to CodeCommit:

git add .
git commit -m "Test Automation"
git push

5 — Commit the new code to CodeCommit. It will kick off the Code Pipeline.

After complete the automation process, you will be able to see that it succeeded in both stages.

NOTE: If you have any issue check the build process, go to the last build and see the build logs. It will help a lot to debug if you face some issues

Conclusion

Now you have a fully automated IaC pipeline with AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, and Cloud One — Conformity to review any possible misconfiguration during the process to create your CloudFormation template.

Hopefully, this will help you and your team to ensure architectural best practices are followed across the development process to continue to rapidly innovate in the cloud. If you want to learn more don’t hesitate to reach out to me!

Acknowledgment

I want to say a BIG thank you to some people that helped me with fantastic feedback to improve this article:

  • Raphael Bottino
  • Melissa Clow

If this post was helpful, please click the clap 👏 button below a few times 😉👍! ⬇

I'm a Computer Engineer 👨‍💻 with a passion for Cybersecurity, DevOps, and Cloud. When I'm not at my 💻 , I'm traveling and taking photos across the globe 🌎

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store