Introduction:

Custom resources definition (CRD) is a powerful feature introduced in Kubernetes 1.7 which enables users to add their own/custom objects to the Kubernetes cluster and use it like any other native Kubernetes objects. In this blog post, we will see how we can add a custom resource to a Kubernetes cluster using the command line as well as using the Golang client library thus also learning how to programmatically interact with a Kubernetes cluster.

What is a Custom Resource Definition (CRD)?

In the Kubernetes API, every resource is an endpoint to store API objects of certain kind. For example, the built-in service resource contains a collection of service objects. The standard Kubernetes distribution ships with many inbuilt API objects/resources. CRD comes into picture when we want to introduce our own object into the Kubernetes cluster to full fill our requirements. Once we create a CRD in Kubernetes we can use it like any other native Kubernetes object thus leveraging all the features of Kubernetes like its CLI, security, API services, RBAC etc.

The custom resource created is also stored in the etcd cluster with proper replication and lifecycle management. CRD allows us to use all the functionalities provided by a Kubernetes cluster for our custom objects and saves us the overhead of implementing them on our own.

How to register a CRD using command line interface (CLI)

Step-1: Create a CRD definition file sslconfig-crd.yaml

CODE: https://gist.github.com/velotiotech/baacb972a46af7ea18ecac2e43853337.js

Here we are creating a custom resource definition for an object of kind SslConfig. This object allows us to store the SSL configuration information for a domain. As we can see under the validation section specifying the cert, key and the domain are mandatory for creating objects of this kind, along with this we can store other information like the provider of the certificate etc. The name metadata that we specify must be spec.names.plural+"."+spec.group.

An API group (blog.velotio.com here) is a collection of API objects which are logically related to each other. We have also specified version for our custom objects (spec.version), as the definition of the object is expected to change/evolve in future so it's better to start with alpha so that the users of the object knows that the definition might change later. In the scope, we have specified Namespaced, by default a custom resource name is clustered scoped. 

CODE: https://gist.github.com/velotiotech/35d487e0ccf5d7ceb5cfbed1169840bc.js

Step-2:  Create objects using the definition we created above

CODE: https://gist.github.com/velotiotech/c8334618d9692a0eb2adfd4172614ce8.js

CODE: https://gist.github.com/velotiotech/b7321beb52dbfcaf130244d895cd5ad9.js

Along with the mandatory fields cert, key and domain, we have also stored the information of the provider ( certifying authority ) of the cert.

How to register a CRD programmatically using client-go

Client-go project provides us with packages using which we can easily create go client and access the Kubernetes cluster.  For creating a client first we need to create a connection with the API server.
How we connect to the API server depends on whether we will be accessing it from within the cluster (our code running in the Kubernetes cluster itself) or if our code is running outside the cluster (locally)

If the code is running outside the cluster then we need to provide either the path of the config file or URL of the Kubernetes proxy server running on the cluster.

CODE: https://gist.github.com/velotiotech/774e111162e6239fa80a7d0a1753b5c1.js

OR

CODE: https://gist.github.com/velotiotech/8d268c714e2e0a7893289b63821e3ef9.js

When the code is to be run as a part of the cluster then we can simply use

CODE: https://gist.github.com/velotiotech/ee9c00b960f31462088162ca66a3c144.js

Once the connection is established we can use it to create clientset. For accessing Kubernetes objects, generally the clientset from the client-go project is used, but for CRD related operations we need to use the clientset from apiextensions-apiserver project. 

apiextension "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"

CODE: https://gist.github.com/velotiotech/b8d4a9233f378d8610df9280f3abeaa9.js

Now we can use the client to make the API call which will create the CRD for us.

CODE: https://gist.github.com/velotiotech/cbdf87b529c406a63822c7d4f99a9f3b.js

In the create CRD function, we first create the definition of our custom object and then pass it to the create method which creates it in our cluster. Just like we did while creating our definition using CLI, here also we set the parameters like version, group, kind etc.

Once our definition is ready we can create objects of its type just like we did earlier using the CLI. First we need to define our object.

CODE: https://gist.github.com/velotiotech/3dd135ff6cbcda723a6d3e128d67f167.js

Kubernetes API conventions suggests that each object must have two nested object fields that govern the object’s configuration: the object spec and the object status. Objects must also have metadata associated with them. The custom objects that we define here comply with these standards. It is also recommended to create a list type for every type thus we have also created a SslConfigList struct.

Now we need to write a function which will create a custom client which is aware of the new resource that we have created.

CODE: https://gist.github.com/velotiotech/44dccc9f33968e7e3b95f7f6bfcec44f.js

Building the custom client library

Once we have registered our custom resource definition with the Kubernetes cluster we can create objects of its type using the Kubernetes cli as we did earlier but for creating controllers for these objects or for developing some custom functionalities around them we need to build a client library also using which we can access them from go API. For native Kubernetes objects, this type of library is provided for each object.

CODE: https://gist.github.com/velotiotech/a8852c7b918eaa3414d997a9a33d8f37.js

We can add more methods like watch, update status etc. Their implementation will also be similar to the methods we have defined above. For looking at the methods available for various Kubernetes objects like pod, node etc. we can refer to the v1 package.

Putting all things together

Now in our main function we will get all the things together.

CODE: https://gist.github.com/velotiotech/4edba98890a0bb7e7ad529c535de986c.js

Now if we run our code then our custom resource definition will get created in the Kubernetes cluster and also an object of its type will be there just like with the cli. The docker image akash125/crdblog is build using the code discussed above it can be directly pulled from docker hub and run in a Kubernetes cluster. After the image is run successfully, the CRD definition that we discussed above will get created in the cluster along with an object of its type. We can verify the same using the CLI the way we did earlier, we can also check the logs of the pod running the docker image to verify it. The complete code is available here.

Conclusion and future work

We learned how to create a custom resource definition and objects using Kubernetes command line interface as well as the Golang client. We also learned how to programmatically access a Kubernetes cluster, using which we can build some really cool stuff on Kubernetes, we can now also create custom controllers for our resources which continuously watches the cluster for various life cycle events of our object and takes desired action accordingly. To read more about CRD refer the following links:

• https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
• https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/
• https://blog.openshift.com/kubernetes-custom-resources-grow-up-in-v1-10/
• https://medium.com/@trstringer/create-kubernetes-controllers-for-core-and-custom-resources-62fc35ad64a3