CI/CD Enabler Framework for Google Cloud Platform

Introduction

Software systems in a typical enterprise environment are quite complex (& grows more complex over time). It becomes a humongous task to coordinate different activities such as code & configuration changes especially when a large team is involved in development and operations. DevOps in software is seen as a cultural movement to ease out operational challenges related to integration and deployment.

Continuous Integration (CI) is an underlying technique for DevOps. According to the CI principle, if a task takes a lot of time and energy, it should be done more often, making it significantly less painful at each step. Continuous Deployment (CD) is the last leg to make software releases available to staging/production environments. Enterprises are finding it crucial to employ a CI/CD pipeline in a software project(s) to ensure quality and timely deliveries.

Features of GCP’s Cloud Build Service:

Google Cloud Platform (GCP) provides Cloud Build service as a toolchain to kick start the CI for a project. Some of its features are listed below:

• Integrates with existing source control repository i.e. GitHub, Bitbucket, Cloud Source Repositories, etc.
• Executes builds on GCP infrastructure i.e. VMs (GCE), serverless compute, Kubernetes (GKE), or Firebase
• Supports customization and control for the build, test, and deploy workflows

Limitations of GCP’s Cloud Build Service

However, Cloud Build does not support deployment/ setup creation natively and build artifacts management. It needs to be extended (or coupled with other services) to provide Continuous Deployment and hence have a complete CI/ CD pipeline. Some of the GCP services that are useful in this regard are Container Registry (for build artifact management) and Deployment Manager (Infrastructure as Code/Infrastructure deployment service)

Building a CI/CD Pipeline for a GCP Project:

To build a complete CI/CD pipeline for a GCP-based project, some work is required for deployment/ setup creation and artefact management. This is a cumbersome task, and DevOps engineers may find it hard to make quick progress using the available services (out of the box). In such a situation, having an enabling framework that helps build the CI/CD pipeline without significant effort is very useful.

HSC’s team has been working on GCP-based projects and has developed a CI/CD pipeline enabler framework. This framework additionally supports notification for sharing the progress of the pipeline. The developed framework helps in the realization of the CI/CD pipeline shown below:

CI/CD Pipeline enabler framework

The framework helps in executing the workflow shown below:

The key components of a typical workflow are briefly discussed below:

• Cloud Source Repo: This is part of the target project while enabling the CI/CD.
• Terminal: DevOps engineers need to apply a tag in a defined format after code check-in.
• Cloud Build: This framework creates two Cloud Build triggers (one for Dev/Staging & one for Prod) and a configuration file as expected by Cloud Build. Cloud Build service triggers match the tag and invoke the framework-provided CI/CD program. This program takes care of the complete CI/CD pipeline with the help of other GCP services.
• Cloud Pub/Sub: This communicates the result of the pipeline through a Pub/Sub cloud resource with the help of an MS Teams plugin to the MS Teams channel.

The project lead/manager is responsible for enabling the CI/CD pipeline using this framework, and the DevOps team simply triggers the pipeline by applying a git TAG after they check-in their changes.

HSC's Expertise

HSC has been using this framework in multiple projects and has helped enable CI/CD. These projects encompass a complete AI/ML application lifecycle, i.e. ETL, model training, testing/ validation, evaluation, deployment, monitoring, and a web app to see the application results.

Implementing a CI/CD pipeline helps reduce the risks in each build and makes a software application available to prospective customers faster than ever before. Interested in deploying a CI/CD pipeline. Early bug detection to avoid any last-minute surprises, greater visibility of new builds, test cases, and issues arising due to the same, and achieving faster feedback to ensure flawless code delivery is a few added advantages of deploying the CI/CD pipeline framework.

References:

1. CI principle
2. Cloud Build
3. Reference CI Pipeline
4. Reference CD Pipeline
5. DM Best Practices