This blog focuses on migrating Kubernetes clusters from one cloud provider to another. We will be migrating our entire data from Google Kubernetes Engine to Azure Kubernetes Service using Velero.

Prerequisite

• A Kubernetes cluster > 1.10

Setup Velero with Restic Integration

Velero consists of a client installed on your local computer and a server that runs in your Kubernetes cluster, like Helm. 

Installing Velero Client

You can find the latest release corresponding to your OS and system and download Velero from there:

CODE: https://gist.github.com/velotiotech/5eaf73002a337090f6c1447f781c9110.js

Extract the tarball (change the version depending on yours) and move the Velero binary to /usr/local/bin

CODE: https://gist.github.com/velotiotech/4eea8b553e7a55bde9e0b164d77eeec7.js

Create a Bucket for Velero on GCP

Velero needs an object storage bucket where it will store the backup. Create a GCS bucket using:

CODE: https://gist.github.com/velotiotech/d653e3a2198c8fb9c023171cf36e1441.js

Create a Service Account for Velero

CODE: https://gist.github.com/velotiotech/fb6e48a089d44274ca6e0f7a2773a825.js

Install Velero Server on GKE and AKS

Use the --use-restic flag on the Velero install command to install restic integration.

CODE: https://gist.github.com/velotiotech/fd4696bd5c76803fcc6d58970533717f.js

After that, you can see a DaemonSet of restic and deployment of Velero in your Kubernetes cluster.

CODE: https://gist.github.com/velotiotech/0c9cdd85e08391ef315a34e6081d9c26.js

Restic Components

In addition, there are three more Custom Resource Definitions and their associated controllers to provide restic support.

Restic Repository

• Maintain the complete lifecycle for Velero’s restic repositories.
• Restic lifecycle commands such as restic init check and prune are handled by this CRD controller.

PodVolumeBackup

• This CRD backs up the persistent volume based on the annotated pod in selected namespaces. 
• This controller executes backup commands on the pod to initialize backups. 

PodVolumeRestore

• This controller restores the respective pods that were inside restic backups. And this controller is responsible for the restore commands execution.

Backup an application on GKE

For this blog post, we are considering that Kubernetes already has an application that is using persistent volumes. Or you can install Wordpress as an example as explained here.

We will perform GKE Persistent disk migration to Azure Persistent Disk using Velero. 

Follow the below steps:

1. To back up, the deployment or statefulset checks for the volume name that is mounted to backup that particular persistent volume. For example, here pods need to be annotated with Volume Name “data”.

CODE: https://gist.github.com/velotiotech/a20e7fa0012e978876832ef41d040c1f.js

2. Annotate the pods with the volume names, you’d like to take the backup of and only those volumes will be backed up: 

CODE: https://gist.github.com/velotiotech/f0352394e64239a6494cbacbe9b12eab.js

For example, 

CODE: https://gist.github.com/velotiotech/ca00fb3f5278ba1eeca4017b68bdcaec.js

3. Take a backup of the entire namespace in which the application is running. You can also specify multiple namespaces or skip this flag to backup all namespaces by default.
   We are going to backup only one namespace in this blog.

CODE: https://gist.github.com/velotiotech/9e1ee13cb1a92abd971306da243c1c5c.js

4. Monitor the progress of backup:

CODE: https://gist.github.com/velotiotech/4648d864f83154b88bc9807a4ff95419.js     

Once the backup is complete, you can list it using:

CODE: https://gist.github.com/velotiotech/3f40998478d601302b73978cb05284e5.js

You can also check the backup on GCP Portal under Storage.
Select the bucket you created and you should see a similar directory structure:

Restore the application to AKS

Follow the below steps to restore the backup:

1. Make sure to have the same StorageClass available in Azure as used by GKE Persistent Volumes. For example, if the Storage Class of the PVs is “persistent-ssd”, create the same on AKS using below template:

CODE: https://gist.github.com/velotiotech/6114f426ed85fabb375ec04e1a547e50.js

2. Run Velero restore.

CODE: https://gist.github.com/velotiotech/1ca36f3b2b4d346b698f43b210fc3e61.js

You can monitor the progress of restore:

CODE: https://gist.github.com/velotiotech/af1b0bcefbce86aa683276445ad2086f.js

You can also check on GCP Portal, a new folder “restores” is created under the bucket.

In some time, you should be able to see that the application namespace is back and Wordpress and MySQL pods are running again.

Troubleshooting

For any errors/issues related to Velero, you may find below commands helpful for debugging purposes:

CODE: https://gist.github.com/velotiotech/f19ae7ed8a0041e494cc27f8eab75378.js

Conclusion

The migration of persistent workloads across Kubernetes clusters on different cloud providers is difficult. This became possible by using restic integration with the Velero backup tool. This tool is still said to be in beta quality as mentioned on the official site. I have performed GKE to AKS migration and it went successfully. You can try other combinations of different cloud providers for migrations.

The only drawback of using Velero to migrate data is if your data is too huge, it may take a while to complete migration. It took me almost a day to migrate a 350 GB disk from GKE to AKS. But, if your data is comparatively less, this should be a very efficient and hassle-free way to migrate it.