Cloud-Native CI/CD Workflows in AWS: 3 Use Cases

AWS is one of the cloud computing providers that offers production-ready CI/CD services. Let’s review some basic examples of how you can build a solid CI/CD for most adopted cloud-native architectures in AWS.

Dockerized Microservices

You can industrialize your Dockerized microservices deployment with the below architecture:

Figure 1: CI/CD pipeline for Dockerized microservices

The pipeline is divided into multiple stages:

• Checkout: Fetching the latest code changes from GitHub repository.
• Pre-integration tests: Running the following tests:
  • Unit Tests: Test each piece of your source code. You could also output your coverage and validate that you’re meeting your code coverage requirements.
  • Quality Tests: Check that the code is well formatted and follows best practices.
  • Security Tests: Inspect source code to uncover common security vulnerabilities.
• Build: Building a Docker image based on a Dockerfile that is stored in a code repository.
• Push: Using the semver tagging scheme to tag the Docker image so that it can roll back to a specific version. Once the image is tagged, it will be stored in a private registry like Amazon Elastic Container Registry (Amazon ECR). 
• Deploy: Using AWS CLI or Kubernetes command line, deploy the new changes to Amazon Elastic Kubernetes Service (Amazon EKS).

Note: A Slack notification will be sent whenever something happens to the builds (success or failure).

Serverless Functions

The following diagram describes how to set up a highly resilient and fault-tolerant CI/CD pipeline in order to automate the deployment process of your Lambda-based serverless application:

Figure 2: CI/CD workflow for Lambda-based applications

The workflow consists of the following steps:

• Checkout: Pull the latest commits from the GitHub repository.
• Pre-build: Install the needed dependencies.
• Tests: Launch unit, quality, and security checks against the Lambda handler to ensure your codebase is in a bug-free state and that it is allowing developers to detect and fix integration problems continuously.
• Build: Build a deployment package (ZIP file) from the source code.
• Push: Store the deployment package to an Amazon S3 bucket. A Git SHA commit ID can be used as a deployment package version number. This makes it easier to track different versions built throughout the development cycle, as well as to roll back to a specific commit if things go wrong. 
• Post-build: Update the Lambda function’s code with the new deployment package and create a new version.

Machine Learning Pipelines

AWS provides a cloud-based environment to develop, train, test, deploy, manage, and monitor machine learning models. The figure below shows how to use this environment to build an automated deployment workflow in order to increase the productivity of data science teams.

Figure 3: CI/CD workflow for machine learning models

A classic CI/CD workflow for a machine learning pipeline is divided into the following phases:

• Prepare: Data orchestration, extraction, cleaning, and versioning (Jupyter Notebook files).
• Build: Building the training model, packaging, and storing the generated model to an Amazon S3 bucket.
• Train: Building a Docker image to train the generated model on a dataset and further serve prediction. Distributed training can be used to reduce the training time.
• Test/tuning: Testing the data model on Amazon SageMaker. You can configure SageMaker to automatically tune your model by adjusting multiple combinations of algorithm parameters. The model must meet at least a given quality threshold; if not, the data team will be notified.
• Deploy: Once training is complete, the model can be deployed to SageMaker endpoints for real-time predictions. Advanced deployment strategies, such as A/B testing, can be used to test the accuracy of different machine learning models.

Final Note

To summarize, here are some basic guidelines on how to implement and maintain a CI/CD pipeline to best serve your organization’s needs:

• Start with the smallest pain points and attack bigger ones as you go (KISS principle).
• Keep in mind security automation (DevSecOps).
• Implement tracking and version control tools to achieve better visibility of your project.
• Optimize your CI/CD pipeline build time and release often.

As an AWS Advanced Consulting Partner, Media Temple can help you get the most from your AWS cloud. Reach out anytime.