.svg)
.svg)
This blog focuses on migrating Kubernetes clusters from one cloud provider to another. We will be migrating our entire data from Google Kubernetes Engine to Azure Kubernetes Service using Velero.
Velero consists of a client installed on your local computer and a server that runs in your Kubernetes cluster, like Helm.
You can find the latest release corresponding to your OS and system and download Velero from there:
| $ wget | |
| https://github.com/vmware-tanzu/velero/releases/download/v1.3.1/velero-v1.3.1-linux-amd64.tar.gz |
Extract the tarball (change the version depending on yours) and move the Velero binary to /usr/local/bin
| $ tar -xvzf velero-v0.11.0-darwin-amd64.tar.gz | |
| $ sudo mv velero /usr/local/bin/ | |
| $ velero help |
Velero needs an object storage bucket where it will store the backup. Create a GCS bucket using:
| gsutil mb gs://<bucket-name |
| # Create a Service Account | |
| gcloud iam service-accounts create velero --display-name "Velero service account" | |
| SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list --filter="displayName:Velero service account" --format 'value(email)') | |
| #Define Permissions for the Service Account | |
| ROLE_PERMISSIONS=( | |
| compute.disks.get | |
| compute.disks.create | |
| compute.disks.createSnapshot | |
| compute.snapshots.get | |
| compute.snapshots.create | |
| compute.snapshots.useReadOnly | |
| compute.snapshots.delete | |
| compute.zones.get | |
| ) | |
| # Create a Role for Velero | |
| PROJECT_ID=$(gcloud config get-value project) | |
| gcloud iam roles create velero.server \ | |
| --project $PROJECT_ID \ | |
| --title "Velero Server" \ | |
| --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")" | |
| # Create a Role Binding for Velero | |
| gcloud projects add-iam-policy-binding $PROJECT_ID \ | |
| --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \ | |
| --role projects/$PROJECT_ID/roles/velero.server | |
| gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin | |
| # Generate Service Key file for Velero and save it for later | |
| gcloud iam service-accounts keys create credentials-velero \ | |
| --iam-account $SERVICE_ACCOUNT_EMAIL |
Use the --use-restic flag on the Velero install command to install restic integration.
| $ velero install \ | |
| --use-restic \ | |
| --bucket \ | |
| --provider gcp \ | |
| --secret-file \ | |
| --use-volume-snapshots=false \ | |
| --plugins=--plugins restic/restic | |
| $ velero plugin add velero/velero-plugin-for-gcp:v1.0.1 | |
| $ velero plugin add velero/velero-plugin-for-microsoft-azure:v1.0.0 |
After that, you can see a DaemonSet of restic and deployment of Velero in your Kubernetes cluster.
| $ kubectl get po -n velero |
In addition, there are three more Custom Resource Definitions and their associated controllers to provide restic support.
Restic Repository
PodVolumeBackup
PodVolumeRestore
For this blog post, we are considering that Kubernetes already has an application that is using persistent volumes. Or you can install Wordpress as an example as explained here.
We will perform GKE Persistent disk migration to Azure Persistent Disk using Velero.
Follow the below steps:
| volumes: | |
| - name: data | |
| persistentVolumeClaim: | |
| claimName: mongodb |
| $ kubectl -n NAMESPACE annotate pod/POD_NAME backup.velero.io/backup-volumes=VOLUME_NAME1,VOLUME_NAME2 |
For example,
| $ kubectl -n application annotate pod/wordpress-pod backup.velero.io/backup-volumes=data |
| $ velero backup create testbackup --include-namespaces application |
| $ velero backup describe testbackup --details |
Once the backup is complete, you can list it using:
| $ velero backup get |
You can also check the backup on GCP Portal under Storage.
Select the bucket you created and you should see a similar directory structure:
Follow the below steps to restore the backup:
| kind: StorageClass | |
| apiVersion: storage.k8s.io/v1 | |
| metadata: | |
| name: persistent-ssd // same name as GKE storageclass name | |
| provisioner: kubernetes.io/azure-disk | |
| parameters: | |
| storageaccounttype: Premium_LRS | |
| kind: Managed |
| $ velero restore create testrestore --from-backup testbackup |
You can monitor the progress of restore:
| $ velero restore describe testrestore --details |
You can also check on GCP Portal, a new folder “restores” is created under the bucket.
In some time, you should be able to see that the application namespace is back and Wordpress and MySQL pods are running again.
For any errors/issues related to Velero, you may find below commands helpful for debugging purposes:
| # Describe the backup to see the status | |
| $ velero backup describe testbackup --details | |
| # Check backup logs, and look for errors if any | |
| $ velero backup logs testbackup | |
| # Describe the restore to see the status | |
| $ velero restore describe testrestore --details | |
| # Check restore logs, and look for errors if any | |
| $ velero restore logs testrestore | |
| # Check velero and restic pod logs, and look for errors if any | |
| $ kubectl -n velero logs VELERO_POD_NAME/RESTIC_POD_NAME | |
| NOTE: You can change the default log-level to debug mode by adding --log-level=debug as an argument to the container command in the velero pod template spec. | |
| # Describe the BackupStorageLocation resource and look for any errors in Events | |
| $ kubectl describe BackupStorageLocation default -n velero |
The migration of persistent workloads across Kubernetes clusters on different cloud providers is difficult. This became possible by using restic integration with the Velero backup tool. This tool is still said to be in beta quality as mentioned on the official site. I have performed GKE to AKS migration and it went successfully. You can try other combinations of different cloud providers for migrations.
The only drawback of using Velero to migrate data is if your data is too huge, it may take a while to complete migration. It took me almost a day to migrate a 350 GB disk from GKE to AKS. But, if your data is comparatively less, this should be a very efficient and hassle-free way to migrate it.
Did you like the blog? If yes, we're sure you'll also like to work with the people who write them - our best-in-class engineering team.
We're looking for talented developers who are passionate about new emerging technologies. If that's you, get in touch with us.
This blog focuses on migrating Kubernetes clusters from one cloud provider to another. We will be migrating our entire data from Google Kubernetes Engine to Azure Kubernetes Service using Velero.
Velero consists of a client installed on your local computer and a server that runs in your Kubernetes cluster, like Helm.
You can find the latest release corresponding to your OS and system and download Velero from there:
| $ wget | |
| https://github.com/vmware-tanzu/velero/releases/download/v1.3.1/velero-v1.3.1-linux-amd64.tar.gz |
Extract the tarball (change the version depending on yours) and move the Velero binary to /usr/local/bin
| $ tar -xvzf velero-v0.11.0-darwin-amd64.tar.gz | |
| $ sudo mv velero /usr/local/bin/ | |
| $ velero help |
Velero needs an object storage bucket where it will store the backup. Create a GCS bucket using:
| gsutil mb gs://<bucket-name |
| # Create a Service Account | |
| gcloud iam service-accounts create velero --display-name "Velero service account" | |
| SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list --filter="displayName:Velero service account" --format 'value(email)') | |
| #Define Permissions for the Service Account | |
| ROLE_PERMISSIONS=( | |
| compute.disks.get | |
| compute.disks.create | |
| compute.disks.createSnapshot | |
| compute.snapshots.get | |
| compute.snapshots.create | |
| compute.snapshots.useReadOnly | |
| compute.snapshots.delete | |
| compute.zones.get | |
| ) | |
| # Create a Role for Velero | |
| PROJECT_ID=$(gcloud config get-value project) | |
| gcloud iam roles create velero.server \ | |
| --project $PROJECT_ID \ | |
| --title "Velero Server" \ | |
| --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")" | |
| # Create a Role Binding for Velero | |
| gcloud projects add-iam-policy-binding $PROJECT_ID \ | |
| --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \ | |
| --role projects/$PROJECT_ID/roles/velero.server | |
| gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin | |
| # Generate Service Key file for Velero and save it for later | |
| gcloud iam service-accounts keys create credentials-velero \ | |
| --iam-account $SERVICE_ACCOUNT_EMAIL |
Use the --use-restic flag on the Velero install command to install restic integration.
| $ velero install \ | |
| --use-restic \ | |
| --bucket \ | |
| --provider gcp \ | |
| --secret-file \ | |
| --use-volume-snapshots=false \ | |
| --plugins=--plugins restic/restic | |
| $ velero plugin add velero/velero-plugin-for-gcp:v1.0.1 | |
| $ velero plugin add velero/velero-plugin-for-microsoft-azure:v1.0.0 |
After that, you can see a DaemonSet of restic and deployment of Velero in your Kubernetes cluster.
| $ kubectl get po -n velero |
In addition, there are three more Custom Resource Definitions and their associated controllers to provide restic support.
Restic Repository
PodVolumeBackup
PodVolumeRestore
For this blog post, we are considering that Kubernetes already has an application that is using persistent volumes. Or you can install Wordpress as an example as explained here.
We will perform GKE Persistent disk migration to Azure Persistent Disk using Velero.
Follow the below steps:
| volumes: | |
| - name: data | |
| persistentVolumeClaim: | |
| claimName: mongodb |
| $ kubectl -n NAMESPACE annotate pod/POD_NAME backup.velero.io/backup-volumes=VOLUME_NAME1,VOLUME_NAME2 |
For example,
| $ kubectl -n application annotate pod/wordpress-pod backup.velero.io/backup-volumes=data |
| $ velero backup create testbackup --include-namespaces application |
| $ velero backup describe testbackup --details |
Once the backup is complete, you can list it using:
| $ velero backup get |
You can also check the backup on GCP Portal under Storage.
Select the bucket you created and you should see a similar directory structure:
Follow the below steps to restore the backup:
| kind: StorageClass | |
| apiVersion: storage.k8s.io/v1 | |
| metadata: | |
| name: persistent-ssd // same name as GKE storageclass name | |
| provisioner: kubernetes.io/azure-disk | |
| parameters: | |
| storageaccounttype: Premium_LRS | |
| kind: Managed |
| $ velero restore create testrestore --from-backup testbackup |
You can monitor the progress of restore:
| $ velero restore describe testrestore --details |
You can also check on GCP Portal, a new folder “restores” is created under the bucket.
In some time, you should be able to see that the application namespace is back and Wordpress and MySQL pods are running again.
For any errors/issues related to Velero, you may find below commands helpful for debugging purposes:
| # Describe the backup to see the status | |
| $ velero backup describe testbackup --details | |
| # Check backup logs, and look for errors if any | |
| $ velero backup logs testbackup | |
| # Describe the restore to see the status | |
| $ velero restore describe testrestore --details | |
| # Check restore logs, and look for errors if any | |
| $ velero restore logs testrestore | |
| # Check velero and restic pod logs, and look for errors if any | |
| $ kubectl -n velero logs VELERO_POD_NAME/RESTIC_POD_NAME | |
| NOTE: You can change the default log-level to debug mode by adding --log-level=debug as an argument to the container command in the velero pod template spec. | |
| # Describe the BackupStorageLocation resource and look for any errors in Events | |
| $ kubectl describe BackupStorageLocation default -n velero |
The migration of persistent workloads across Kubernetes clusters on different cloud providers is difficult. This became possible by using restic integration with the Velero backup tool. This tool is still said to be in beta quality as mentioned on the official site. I have performed GKE to AKS migration and it went successfully. You can try other combinations of different cloud providers for migrations.
The only drawback of using Velero to migrate data is if your data is too huge, it may take a while to complete migration. It took me almost a day to migrate a 350 GB disk from GKE to AKS. But, if your data is comparatively less, this should be a very efficient and hassle-free way to migrate it.
.svg)
.svg)
