Create CI/CD Pipeline in GitLab in under 10 mins

Shivani Shinde

Cloud & DevOps

Tags:

CI/CD

cicd-pipeline

gitlab-ci

Why Chose GitLab Over Other CI tools?

If there are many tools available in the market, like CircleCI, Github Actions, Travis CI, etc., what makes GitLab CI so special? The easiest way for you to decide if GitLab CI is right for you is to take a look at following use-case:

GitLab knocks it out of the park when it comes to code collaboration and version control. Monitoring the entire code repository along with all branches becomes manageable. With other popular tools like Jenkins, you can only monitor some branches. If your development teams are spread across multiple locations globally, GitLab serves a good purpose. Regarding price, while Jenkins is free, you need to have a subscription to use all of Gitlab’s features.

In GitLab, every branch can contain the gitlab-ci.yml file, which makes it easy to modify the workflows. For example, if you want to run unit tests on branch A and perform functional testing on branch B, you can simply modify the YAML configuration for CI/CD, and the runner will take care of running the job for you. Here is a comprehensive list of Pros and Cons of Gitlab to help you make a better decision.

Intro

GitLab is an open-source collaboration platform that provides powerful features beyond hosting a code repository. You can track issues, host packages and registries, maintain Wikis, set up continuous integration (CI) and continuous deployment (CD) pipelines, and more.

In this tutorial, you will configure a pipeline with three stages: build, deploy, test. The pipeline will run for each commit pushed to the repository.

GitLab and CI/CD

As we all are aware, a fully-fledged CI/CD pipeline primarily includes the following stages:

Build
Test
Deploy

Here is a pictorial representation of how GitLab covers CI and CD:

Let’s take a look at an example of an automation testing pipeline. Here, CI empowers test automation and CD automates the release process to various environments. The below image perfectly demonstrates the entire flow.

‍

Let’s create the basic 3-stage pipeline

Step 1: Create a project > Create a blank project

Visit gitlab.com and create your account if you don't have one already. Once done, click “New Project,” and on the following screen, click “Create Blank Project.” Name it My First Project, leave other settings to default for now, and click Create.
Alternatively, if you already have your codebase in GitLab, proceed to Step 2.

Step 2: Create a GitLab YAML

‍To create a pipeline in GitLab, we need to define it in a YAML file. This yaml file should reside in the root directory of your project and should be named gitlab-ci.yml. GitLab provides a set of predefined keywords that are used to define a pipeline.

In order to design a basic pipeline, let’s understand the structure of a pipeline. If you are already familiar with the basic structure given below, you may want to jump below to the advanced pipeline outline for various environments.

The hierarchy in GitLab has Pipeline > Stages > Jobs as shown below. The Source or SRC is often a git commit or a CRON job, which triggers the pipeline on a defined branch.

Now, let’s understand the commonly used keywords to design a pipeline:

stages: This is used to define stages in the pipeline.
variables: Here you can define the environment variables that can be accessed in all the jobs.
before_script: This is a list of commands to be executed before each job. For example: creating specific directories, logging, etc.
artifacts: If your job creates any artifacts, you can mention the path to find them here.
after_script: This is a list of commands to be executed after each job. For example: cleanup.
tags: This is a tag/label to identify the runner or a GitLab agent to assign your jobs to. If the tags are not specified, the jobs run on shared runners.
needs: If you want your jobs to be executed in a certain order or you want a particular job to be executed before the current job, then you can set this value to the specific job name.
only/except: These keywords are used to control when the job should be added to the pipeline. Use ‘only’ to define when a job should be added, whereas ‘except’ is used to define when a job should not be added. Alternatively, the ‘rules’ keyword is also used to add/exclude jobs based on conditions.

You can find more keywords here.

Let’s create a sample YAML file.

	stages:
	- build
	- deploy
	- test

	variables:
	RAILS_ENV: "test"
	NODE_ENV: "test"
	GIT_STRATEGY: "clone"
	CHROME_VERSION: "103"
	DOCKER_VERSION: "20.10.14"

	build-job:
	stage: build
	script:
	- echo "Check node version and build your binary or docker image."
	- node -v
	- bash buildScript.sh

	deploy-code:
	stage: deploy
	needs: build-job
	script:
	- echo "Deploy your code "
	- cd to/your/desired/folder
	- bash deployScript.sh

	test-code:
	stage: test
	needs: deploy-code
	script:
	- echo "Run your tests here."
	- cd to/your/desired/folder
	- npm run test

view raw .yaml hosted with ❤ by GitHub

As you can see, if you have your scripts in a bash file, you can run them from here providing the correct path.

Once your YAML is ready, commit the file.

Step 3: Check Pipeline Status

‍Navigate to CICD > Pipelines from the left navigation bar. You can check the status of the pipeline on this page.

Here, you can check the commit ID, branch, the user who triggered the pipeline, stages, and their status.

If you click on the status, you will get a detailed view of pipeline execution.

‍

If you click on a job under any stage, you can check console logs in detail.

‍

If you have any artifacts created in your pipeline jobs, you can find them by clicking on the 3 dots for the pipeline instance.

‍

‍

Advanced Pipeline Outline

For an advanced pipeline that consists of various environments, you can refer to the below YAML. Simply remove the echo statements and replace them with your set of commands.

	image: your-repo:tag
	variables:
	DOCKER_DRIVER: overlay2
	DOCKER_TLS_CERTDIR: ""
	DOCKER_HOST: tcp://localhost:2375
	SAST_DISABLE_DIND: "true"
	DS_DISABLE_DIND: "false"
	GOCACHE: "$CI_PROJECT_DIR/.cache"
	cache: # this section is used to cache libraries etc between pipeline runs thus reducing the amount of time required for pipeline to run
	key: ${CI_PROJECT_NAME}
	paths:
	- cache-path/
	#include:
	#- #echo "You can add other projects here."
	#- #project: "some/other/important/project"
	#ref: main
	#file: "src/project.yml"
	default:
	tags:
	- your-common-instance-tag
	stages:
	- build
	- test
	- deploy_dev
	- dev_tests
	- deploy_qa
	- qa_tests
	- rollback_qa
	- prod_gate
	- deploy_prod
	- rollback_prod
	- cleanup
	build:
	stage: build
	services:
	- docker:19.03.0-dind
	before_script:
	- echo "Run your pre-build commnadss here"
	- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
	script:
	- docker build -t $CI_REGISTRY/repo:$DOCKER_IMAGE_TAG --build-arg GITLAB_USER=$GITLAB_USER --build-arg GITLAB_PASSWORD=$GITLAB_PASSWORD -f ./Dockerfile .
	- docker push $CI_REGISTRY/repo:$DOCKER_IMAGE_TAG
	- echo "Run your builds here"
	unit_test:
	stage: test
	image: your-repo:tag
	script:
	- echo "Run your unit tests here"
	linting:
	stage: test
	image: your-repo:tag
	script:
	- echo "Run your linting tests here"
	sast:
	stage: test
	image: your-repo:tag
	script:
	- echo "Run your Static application security testing here "
	deploy_dev:
	stage: deploy_dev
	image: your-repo:tag
	before_script:
	- source file.sh
	- export VARIABLE="$VALUE"
	- echo "deploy on dev"
	script:
	- echo "deploy on dev"
	after_script:
	#if deployment fails run rollback on dev
	- echo "Things to do after deployment is run"
	only:
	- master #Depends on your branching strategy
	integration_test_dev:
	stage: dev_tests
	image: your-repo:tag
	script:
	- echo "run test on dev"
	only:
	- master
	allow_failure: true # In case failures are allowed
	deploy_qa:
	stage: deploy_qa
	image: your-repo:tag
	before_script:
	- source file.sh
	- export VARIABLE="$VALUE"
	- echo "deploy on qa"
	script:
	- echo "deploy on qa
	after_script:
	#if deployment fails run rollback on qa
	- echo "Things to do after deployment script is complete "
	only:
	- master
	needs: ["integration_test_dev", "deploy_dev"]
	allow_failure: false
	integration_test_qa:
	stage: qa_tests
	image: your-repo:tag
	script:
	- echo "deploy on qa
	only:
	- master
	allow_failure: true # in case you want to allow failures

	rollback_qa:
	stage: rollback_qa
	image: your-repo:tag
	before_script:
	- echo "Things to rollback after qa integration failure"
	script:
	- echo "Steps to rollback"
	after_script:
	- echo "Things to do after rollback"
	only:
	- master
	needs:
	[
	"deploy_qa",
	]
	when: on_failure #This will run in case the qa deploy job fails
	allow_failure: false

	prod_gate: # this is manual gate for prod approval
	before_script:
	- echo "your commands here"
	stage: prod_gate
	only:
	- master
	needs:
	- deploy_qa
	when: manual


	deploy_prod:
	stage: deploy_prod
	image: your-repo:tag
	tags:
	- some-tag
	before_script:
	- source file.sh
	- echo "your commands here"
	script:
	- echo "your commands here"
	after_script:
	#if deployment fails
	- echo "your commands here"
	only:
	- master
	needs: [ "deploy_qa"]
	allow_failure: false
	rollback_prod: # This stage should be run only when prod deployment fails
	stage: rollback_prod
	image: your-repo:tag
	before_script:
	- export VARIABLE="$VALUE"
	- echo "your commands here"
	script:
	- echo "your commands here"
	only:
	- master
	needs: [ "deploy_prod"]
	allow_failure: false
	when: on_failure
	cleanup:
	stage: cleanup
	script:
	- echo "run cleanup"
	- rm -rf .cache/
	when: always

view raw .yaml hosted with ❤ by GitHub

Conclusion

If you have worked with Jenkins, you know the pain points of working with groovy code. Thus, GitLab CI makes it easy to design, understand, and maintain the pipeline code.

Here are some pros and cons of using GitLab CI that will help you decide if this is the right tool for you!

Thank you! Your submission has been received!

Oops! Something went wrong while submitting the form.

Create CI/CD Pipeline in GitLab in under 10 mins

Why Chose GitLab Over Other CI tools?

Intro

In this tutorial, you will configure a pipeline with three stages: build, deploy, test. The pipeline will run for each commit pushed to the repository.

GitLab and CI/CD

As we all are aware, a fully-fledged CI/CD pipeline primarily includes the following stages:

Build
Test
Deploy

Here is a pictorial representation of how GitLab covers CI and CD:

‍

Let’s create the basic 3-stage pipeline

Step 1: Create a project > Create a blank project

Step 2: Create a GitLab YAML

The hierarchy in GitLab has Pipeline > Stages > Jobs as shown below. The Source or SRC is often a git commit or a CRON job, which triggers the pipeline on a defined branch.

Now, let’s understand the commonly used keywords to design a pipeline:

stages: This is used to define stages in the pipeline.
variables: Here you can define the environment variables that can be accessed in all the jobs.
before_script: This is a list of commands to be executed before each job. For example: creating specific directories, logging, etc.
artifacts: If your job creates any artifacts, you can mention the path to find them here.
after_script: This is a list of commands to be executed after each job. For example: cleanup.
tags: This is a tag/label to identify the runner or a GitLab agent to assign your jobs to. If the tags are not specified, the jobs run on shared runners.
needs: If you want your jobs to be executed in a certain order or you want a particular job to be executed before the current job, then you can set this value to the specific job name.
only/except: These keywords are used to control when the job should be added to the pipeline. Use ‘only’ to define when a job should be added, whereas ‘except’ is used to define when a job should not be added. Alternatively, the ‘rules’ keyword is also used to add/exclude jobs based on conditions.

You can find more keywords here.

Let’s create a sample YAML file.

	stages:
	- build
	- deploy
	- test

	variables:
	RAILS_ENV: "test"
	NODE_ENV: "test"
	GIT_STRATEGY: "clone"
	CHROME_VERSION: "103"
	DOCKER_VERSION: "20.10.14"

	build-job:
	stage: build
	script:
	- echo "Check node version and build your binary or docker image."
	- node -v
	- bash buildScript.sh

	deploy-code:
	stage: deploy
	needs: build-job
	script:
	- echo "Deploy your code "
	- cd to/your/desired/folder
	- bash deployScript.sh

	test-code:
	stage: test
	needs: deploy-code
	script:
	- echo "Run your tests here."
	- cd to/your/desired/folder
	- npm run test

view raw .yaml hosted with ❤ by GitHub

As you can see, if you have your scripts in a bash file, you can run them from here providing the correct path.

Once your YAML is ready, commit the file.

Step 3: Check Pipeline Status

‍Navigate to CICD > Pipelines from the left navigation bar. You can check the status of the pipeline on this page.

Here, you can check the commit ID, branch, the user who triggered the pipeline, stages, and their status.

If you click on the status, you will get a detailed view of pipeline execution.

‍

If you click on a job under any stage, you can check console logs in detail.

‍

If you have any artifacts created in your pipeline jobs, you can find them by clicking on the 3 dots for the pipeline instance.

‍

‍

Advanced Pipeline Outline

For an advanced pipeline that consists of various environments, you can refer to the below YAML. Simply remove the echo statements and replace them with your set of commands.

	image: your-repo:tag
	variables:
	DOCKER_DRIVER: overlay2
	DOCKER_TLS_CERTDIR: ""
	DOCKER_HOST: tcp://localhost:2375
	SAST_DISABLE_DIND: "true"
	DS_DISABLE_DIND: "false"
	GOCACHE: "$CI_PROJECT_DIR/.cache"
	cache: # this section is used to cache libraries etc between pipeline runs thus reducing the amount of time required for pipeline to run
	key: ${CI_PROJECT_NAME}
	paths:
	- cache-path/
	#include:
	#- #echo "You can add other projects here."
	#- #project: "some/other/important/project"
	#ref: main
	#file: "src/project.yml"
	default:
	tags:
	- your-common-instance-tag
	stages:
	- build
	- test
	- deploy_dev
	- dev_tests
	- deploy_qa
	- qa_tests
	- rollback_qa
	- prod_gate
	- deploy_prod
	- rollback_prod
	- cleanup
	build:
	stage: build
	services:
	- docker:19.03.0-dind
	before_script:
	- echo "Run your pre-build commnadss here"
	- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
	script:
	- docker build -t $CI_REGISTRY/repo:$DOCKER_IMAGE_TAG --build-arg GITLAB_USER=$GITLAB_USER --build-arg GITLAB_PASSWORD=$GITLAB_PASSWORD -f ./Dockerfile .
	- docker push $CI_REGISTRY/repo:$DOCKER_IMAGE_TAG
	- echo "Run your builds here"
	unit_test:
	stage: test
	image: your-repo:tag
	script:
	- echo "Run your unit tests here"
	linting:
	stage: test
	image: your-repo:tag
	script:
	- echo "Run your linting tests here"
	sast:
	stage: test
	image: your-repo:tag
	script:
	- echo "Run your Static application security testing here "
	deploy_dev:
	stage: deploy_dev
	image: your-repo:tag
	before_script:
	- source file.sh
	- export VARIABLE="$VALUE"
	- echo "deploy on dev"
	script:
	- echo "deploy on dev"
	after_script:
	#if deployment fails run rollback on dev
	- echo "Things to do after deployment is run"
	only:
	- master #Depends on your branching strategy
	integration_test_dev:
	stage: dev_tests
	image: your-repo:tag
	script:
	- echo "run test on dev"
	only:
	- master
	allow_failure: true # In case failures are allowed
	deploy_qa:
	stage: deploy_qa
	image: your-repo:tag
	before_script:
	- source file.sh
	- export VARIABLE="$VALUE"
	- echo "deploy on qa"
	script:
	- echo "deploy on qa
	after_script:
	#if deployment fails run rollback on qa
	- echo "Things to do after deployment script is complete "
	only:
	- master
	needs: ["integration_test_dev", "deploy_dev"]
	allow_failure: false
	integration_test_qa:
	stage: qa_tests
	image: your-repo:tag
	script:
	- echo "deploy on qa
	only:
	- master
	allow_failure: true # in case you want to allow failures

	rollback_qa:
	stage: rollback_qa
	image: your-repo:tag
	before_script:
	- echo "Things to rollback after qa integration failure"
	script:
	- echo "Steps to rollback"
	after_script:
	- echo "Things to do after rollback"
	only:
	- master
	needs:
	[
	"deploy_qa",
	]
	when: on_failure #This will run in case the qa deploy job fails
	allow_failure: false

	prod_gate: # this is manual gate for prod approval
	before_script:
	- echo "your commands here"
	stage: prod_gate
	only:
	- master
	needs:
	- deploy_qa
	when: manual


	deploy_prod:
	stage: deploy_prod
	image: your-repo:tag
	tags:
	- some-tag
	before_script:
	- source file.sh
	- echo "your commands here"
	script:
	- echo "your commands here"
	after_script:
	#if deployment fails
	- echo "your commands here"
	only:
	- master
	needs: [ "deploy_qa"]
	allow_failure: false
	rollback_prod: # This stage should be run only when prod deployment fails
	stage: rollback_prod
	image: your-repo:tag
	before_script:
	- export VARIABLE="$VALUE"
	- echo "your commands here"
	script:
	- echo "your commands here"
	only:
	- master
	needs: [ "deploy_prod"]
	allow_failure: false
	when: on_failure
	cleanup:
	stage: cleanup
	script:
	- echo "run cleanup"
	- rm -rf .cache/
	when: always

view raw .yaml hosted with ❤ by GitHub

Conclusion

If you have worked with Jenkins, you know the pain points of working with groovy code. Thus, GitLab CI makes it easy to design, understand, and maintain the pipeline code.

Here are some pros and cons of using GitLab CI that will help you decide if this is the right tool for you!

CI/CD

cicd-pipeline

gitlab-ci

About the Author

Did you like the blog? If yes, we're sure you'll also like to work with the people who write them - our best-in-class engineering team.

We're looking for talented developers who are passionate about new emerging technologies. If that's you, get in touch with us.

Explore current openings

Subscribe to get the latest technology updates

Create CI/CD Pipeline in GitLab in under 10 mins

Shivani Shinde

Why Chose GitLab Over Other CI tools?

Intro

GitLab and CI/CD

Let’s create the basic 3-stage pipeline

Step 1: Create a project > Create a blank project

Step 2: Create a GitLab YAML

Step 3: Check Pipeline Status

Advanced Pipeline Outline

Conclusion

MORE POSTS BY THIS AUTHOR

Shivani Shinde

You may also like

Linux Internals of Kubernetes Networking

Shiwam Jaiswal

Strategies for Cost Optimization Across Amazon EKS Clusters

Saurabh Taneja

Mastering Prow: A Guide to Developing Your Own Plugin for Kubernetes CI/CD Workflow

Bhavya Jain

Create CI/CD Pipeline in GitLab in under 10 mins

Why Chose GitLab Over Other CI tools?

Intro

GitLab and CI/CD

Let’s create the basic 3-stage pipeline

Step 1: Create a project > Create a blank project

Step 2: Create a GitLab YAML

Step 3: Check Pipeline Status

Advanced Pipeline Outline

Conclusion

About the Author

Did you like the blog? If yes, we're sure you'll also like to work with the people who write them - our best-in-class engineering team.

We're looking for talented developers who are passionate about new emerging technologies. If that's you, get in touch with us.

About Velotio

Subscribe to get the latest technology updates

Related Posts

Linux Internals of Kubernetes Networking

Strategies for Cost Optimization Across Amazon EKS Clusters

Mastering Prow: A Guide to Developing Your Own Plugin for Kubernetes CI/CD Workflow

Simplifying MySQL Sharding with ProxySQL: A Step-by-Step Guide

Streamline Kubernetes Storage Upgrades

Unlocking Key Insights in NATS Development: My Journey from Novice to Expert - Part 1

Unveiling the Magic of Kubernetes: Exploring Pod Priority, Priority Classes, and Pod Preemption

How to deploy GitHub Actions Self-Hosted Runners on Kubernetes

How to Setup HashiCorp Vault HA Cluster with Integrated Storage (Raft)

How To Get Started With Logging On Kubernetes?

Acquiring Temporary AWS Credentials with Browser Navigated Authentication

How to Avoid Screwing Up CI/CD: Best Practices for DevOps Team

How to Make Your Terminal More Productive with Z-Shell (ZSH)

Setting Up A Robust Authentication Environment For OpenSSH Using QR Code PAM

Hacking Your Way Around AWS IAM Roles

Monitoring a Docker Container with Elasticsearch, Kibana, and Metricbeat

Autoscaling in Kubernetes using HPA and VPA

Managing a TLS Certificate for Kubernetes Admission Webhook

Prow + Kubernetes - A Perfect Combination To Execute CI/CD At Scale

Building A Containerized Microservice in Golang: A Step-by-step Guide

Kubernetes Migration: How To Move Data Freely Across Clusters

OPA On Kubernetes: An Introduction For Beginners

To Go Serverless Or Not Is The Question

Ensure Continuous Delivery On Kubernetes With GitOps’ Argo CD

How To Implement Chaos Engineering For Microservices Using Istio

Helm 3: A More Secured and Simpler Kubernetes Package Manager

An Introduction To Cloudflare Workers And Cloudflare KV store

Getting Started With Kubernetes Operators (Golang Based) - Part 3

Getting Started With Kubernetes Operators (Ansible Based) - Part 2

Getting Started With Kubernetes Operators (Helm Based) - Part 1

How to Write Jenkinsfile for Angular and .Net Based Applications

Kubernetes CSI in Action: Explained with Features and Use Cases

A Comprehensive Tutorial to Implementing OpenTracing With Jaeger

The Ultimate Guide to Disaster Recovery for Your Kubernetes Clusters

Know Everything About Spinnaker & How to Deploy Using Kubernetes Engine

Mesosphere DC/OS Masterclass : Tips and Tricks to Make Life Easier

Managing Secrets Using AWS Systems Manager Parameter Store and IAM Roles

Taking Amazon's Elastic Kubernetes Service for a Spin

Extending Kubernetes APIs with Custom Resource Definitions (CRDs)

Jenkins X - A Cloud-native Approach to CI/CD

Demystifying High Availability in Kubernetes Using Kubeadm

Exploring Upgrade Strategies for Stateful Sets in Kubernetes

Learn How to Quickly Setup Istio Using GKE and its Applications