• United States+1
  • United Kingdom+44
  • Afghanistan (‫افغانستان‬‎)+93
  • Albania (Shqipëri)+355
  • Algeria (‫الجزائر‬‎)+213
  • American Samoa+1684
  • Andorra+376
  • Angola+244
  • Anguilla+1264
  • Antigua and Barbuda+1268
  • Argentina+54
  • Armenia (Հայաստան)+374
  • Aruba+297
  • Australia+61
  • Austria (Österreich)+43
  • Azerbaijan (Azərbaycan)+994
  • Bahamas+1242
  • Bahrain (‫البحرين‬‎)+973
  • Bangladesh (বাংলাদেশ)+880
  • Barbados+1246
  • Belarus (Беларусь)+375
  • Belgium (België)+32
  • Belize+501
  • Benin (Bénin)+229
  • Bermuda+1441
  • Bhutan (འབྲུག)+975
  • Bolivia+591
  • Bosnia and Herzegovina (Босна и Херцеговина)+387
  • Botswana+267
  • Brazil (Brasil)+55
  • British Indian Ocean Territory+246
  • British Virgin Islands+1284
  • Brunei+673
  • Bulgaria (България)+359
  • Burkina Faso+226
  • Burundi (Uburundi)+257
  • Cambodia (កម្ពុជា)+855
  • Cameroon (Cameroun)+237
  • Canada+1
  • Cape Verde (Kabu Verdi)+238
  • Caribbean Netherlands+599
  • Cayman Islands+1345
  • Central African Republic (République centrafricaine)+236
  • Chad (Tchad)+235
  • Chile+56
  • China (中国)+86
  • Christmas Island+61
  • Cocos (Keeling) Islands+61
  • Colombia+57
  • Comoros (‫جزر القمر‬‎)+269
  • Congo (DRC) (Jamhuri ya Kidemokrasia ya Kongo)+243
  • Congo (Republic) (Congo-Brazzaville)+242
  • Cook Islands+682
  • Costa Rica+506
  • Côte d’Ivoire+225
  • Croatia (Hrvatska)+385
  • Cuba+53
  • Curaçao+599
  • Cyprus (Κύπρος)+357
  • Czech Republic (Česká republika)+420
  • Denmark (Danmark)+45
  • Djibouti+253
  • Dominica+1767
  • Dominican Republic (República Dominicana)+1
  • Ecuador+593
  • Egypt (‫مصر‬‎)+20
  • El Salvador+503
  • Equatorial Guinea (Guinea Ecuatorial)+240
  • Eritrea+291
  • Estonia (Eesti)+372
  • Ethiopia+251
  • Falkland Islands (Islas Malvinas)+500
  • Faroe Islands (Føroyar)+298
  • Fiji+679
  • Finland (Suomi)+358
  • France+33
  • French Guiana (Guyane française)+594
  • French Polynesia (Polynésie française)+689
  • Gabon+241
  • Gambia+220
  • Georgia (საქართველო)+995
  • Germany (Deutschland)+49
  • Ghana (Gaana)+233
  • Gibraltar+350
  • Greece (Ελλάδα)+30
  • Greenland (Kalaallit Nunaat)+299
  • Grenada+1473
  • Guadeloupe+590
  • Guam+1671
  • Guatemala+502
  • Guernsey+44
  • Guinea (Guinée)+224
  • Guinea-Bissau (Guiné Bissau)+245
  • Guyana+592
  • Haiti+509
  • Honduras+504
  • Hong Kong (香港)+852
  • Hungary (Magyarország)+36
  • Iceland (Ísland)+354
  • India (भारत)+91
  • Indonesia+62
  • Iran (‫ایران‬‎)+98
  • Iraq (‫العراق‬‎)+964
  • Ireland+353
  • Isle of Man+44
  • Israel (‫ישראל‬‎)+972
  • Italy (Italia)+39
  • Jamaica+1876
  • Japan (日本)+81
  • Jersey+44
  • Jordan (‫الأردن‬‎)+962
  • Kazakhstan (Казахстан)+7
  • Kenya+254
  • Kiribati+686
  • Kosovo+383
  • Kuwait (‫الكويت‬‎)+965
  • Kyrgyzstan (Кыргызстан)+996
  • Laos (ລາວ)+856
  • Latvia (Latvija)+371
  • Lebanon (‫لبنان‬‎)+961
  • Lesotho+266
  • Liberia+231
  • Libya (‫ليبيا‬‎)+218
  • Liechtenstein+423
  • Lithuania (Lietuva)+370
  • Luxembourg+352
  • Macau (澳門)+853
  • Macedonia (FYROM) (Македонија)+389
  • Madagascar (Madagasikara)+261
  • Malawi+265
  • Malaysia+60
  • Maldives+960
  • Mali+223
  • Malta+356
  • Marshall Islands+692
  • Martinique+596
  • Mauritania (‫موريتانيا‬‎)+222
  • Mauritius (Moris)+230
  • Mayotte+262
  • Mexico (México)+52
  • Micronesia+691
  • Moldova (Republica Moldova)+373
  • Monaco+377
  • Mongolia (Монгол)+976
  • Montenegro (Crna Gora)+382
  • Montserrat+1664
  • Morocco (‫المغرب‬‎)+212
  • Mozambique (Moçambique)+258
  • Myanmar (Burma) (မြန်မာ)+95
  • Namibia (Namibië)+264
  • Nauru+674
  • Nepal (नेपाल)+977
  • Netherlands (Nederland)+31
  • New Caledonia (Nouvelle-Calédonie)+687
  • New Zealand+64
  • Nicaragua+505
  • Niger (Nijar)+227
  • Nigeria+234
  • Niue+683
  • Norfolk Island+672
  • North Korea (조선 민주주의 인민 공화국)+850
  • Northern Mariana Islands+1670
  • Norway (Norge)+47
  • Oman (‫عُمان‬‎)+968
  • Pakistan (‫پاکستان‬‎)+92
  • Palau+680
  • Palestine (‫فلسطين‬‎)+970
  • Panama (Panamá)+507
  • Papua New Guinea+675
  • Paraguay+595
  • Peru (Perú)+51
  • Philippines+63
  • Poland (Polska)+48
  • Portugal+351
  • Puerto Rico+1
  • Qatar (‫قطر‬‎)+974
  • Réunion (La Réunion)+262
  • Romania (România)+40
  • Russia (Россия)+7
  • Rwanda+250
  • Saint Barthélemy (Saint-Barthélemy)+590
  • Saint Helena+290
  • Saint Kitts and Nevis+1869
  • Saint Lucia+1758
  • Saint Martin (Saint-Martin (partie française))+590
  • Saint Pierre and Miquelon (Saint-Pierre-et-Miquelon)+508
  • Saint Vincent and the Grenadines+1784
  • Samoa+685
  • San Marino+378
  • São Tomé and Príncipe (São Tomé e Príncipe)+239
  • Saudi Arabia (‫المملكة العربية السعودية‬‎)+966
  • Senegal (Sénégal)+221
  • Serbia (Србија)+381
  • Seychelles+248
  • Sierra Leone+232
  • Singapore+65
  • Sint Maarten+1721
  • Slovakia (Slovensko)+421
  • Slovenia (Slovenija)+386
  • Solomon Islands+677
  • Somalia (Soomaaliya)+252
  • South Africa+27
  • South Korea (대한민국)+82
  • South Sudan (‫جنوب السودان‬‎)+211
  • Spain (España)+34
  • Sri Lanka (ශ්‍රී ලංකාව)+94
  • Sudan (‫السودان‬‎)+249
  • Suriname+597
  • Svalbard and Jan Mayen+47
  • Swaziland+268
  • Sweden (Sverige)+46
  • Switzerland (Schweiz)+41
  • Syria (‫سوريا‬‎)+963
  • Taiwan (台灣)+886
  • Tajikistan+992
  • Tanzania+255
  • Thailand (ไทย)+66
  • Timor-Leste+670
  • Togo+228
  • Tokelau+690
  • Tonga+676
  • Trinidad and Tobago+1868
  • Tunisia (‫تونس‬‎)+216
  • Turkey (Türkiye)+90
  • Turkmenistan+993
  • Turks and Caicos Islands+1649
  • Tuvalu+688
  • U.S. Virgin Islands+1340
  • Uganda+256
  • Ukraine (Україна)+380
  • United Arab Emirates (‫الإمارات العربية المتحدة‬‎)+971
  • United Kingdom+44
  • United States+1
  • Uruguay+598
  • Uzbekistan (Oʻzbekiston)+998
  • Vanuatu+678
  • Vatican City (Città del Vaticano)+39
  • Venezuela+58
  • Vietnam (Việt Nam)+84
  • Wallis and Futuna+681
  • Western Sahara (‫الصحراء الغربية‬‎)+212
  • Yemen (‫اليمن‬‎)+967
  • Zambia+260
  • Zimbabwe+263
  • Åland Islands+358
Thanks! We'll be in touch in the next 12 hours
Oops! Something went wrong while submitting the form.

Extending Kubernetes APIs with Custom Resource Definitions (CRDs)

Introduction:

Custom resources definition (CRD) is a powerful feature introduced in Kubernetes 1.7 which enables users to add their own/custom objects to the Kubernetes cluster and use it like any other native Kubernetes objects. In this blog post, we will see how we can add a custom resource to a Kubernetes cluster using the command line as well as using the Golang client library thus also learning how to programmatically interact with a Kubernetes cluster.

What is a Custom Resource Definition (CRD)?

In the Kubernetes API, every resource is an endpoint to store API objects of certain kind. For example, the built-in service resource contains a collection of service objects. The standard Kubernetes distribution ships with many inbuilt API objects/resources. CRD comes into picture when we want to introduce our own object into the Kubernetes cluster to full fill our requirements. Once we create a CRD in Kubernetes we can use it like any other native Kubernetes object thus leveraging all the features of Kubernetes like its CLI, security, API services, RBAC etc.

The custom resource created is also stored in the etcd cluster with proper replication and lifecycle management. CRD allows us to use all the functionalities provided by a Kubernetes cluster for our custom objects and saves us the overhead of implementing them on our own.

How to register a CRD using command line interface (CLI)

Step-1: Create a CRD definition file sslconfig-crd.yaml

apiVersion: "apiextensions.k8s.io/v1beta1"
kind: "CustomResourceDefinition"
metadata:
name: "sslconfigs.blog.velotio.com"
spec:
group: "blog.velotio.com"
version: "v1alpha1"
scope: "Namespaced"
names:
plural: "sslconfigs"
singular: "sslconfig"
kind: "SslConfig"
validation:
openAPIV3Schema:
required: ["spec"]
properties:
spec:
required: ["cert","key","domain"]
properties:
cert:
type: "string"
minimum: 1
key:
type: "string"
minimum: 1
domain:
type: "string"
minimum: 1

Here we are creating a custom resource definition for an object of kind SslConfig. This object allows us to store the SSL configuration information for a domain. As we can see under the validation section specifying the cert, key and the domain are mandatory for creating objects of this kind, along with this we can store other information like the provider of the certificate etc. The name metadata that we specify must be spec.names.plural+"."+spec.group.

An API group (blog.velotio.com here) is a collection of API objects which are logically related to each other. We have also specified version for our custom objects (spec.version), as the definition of the object is expected to change/evolve in future so it's better to start with alpha so that the users of the object knows that the definition might change later. In the scope, we have specified Namespaced, by default a custom resource name is clustered scoped. 

# kubectl create -f crd.yaml
# kubectl get crd NAME AGE sslconfigs.blog.velotio.com 5s
view raw crd.sh hosted with ❤ by GitHub

Step-2:  Create objects using the definition we created above

apiVersion: "blog.velotio.com/v1alpha1"
kind: "SslConfig"
metadata:
name: "sslconfig-velotio.com"
spec:
cert: "my cert file"
key : "my private key"
domain: "*.velotio.com"
provider: "digicert"

# kubectl create -f crd-obj.yaml
# kubectl get sslconfig NAME AGE sslconfig-velotio.com 12s
view raw create_crd.sh hosted with ❤ by GitHub

Along with the mandatory fields cert, key and domain, we have also stored the information of the provider ( certifying authority ) of the cert.

How to register a CRD programmatically using client-go

Client-go project provides us with packages using which we can easily create go client and access the Kubernetes cluster.  For creating a client first we need to create a connection with the API server.
How we connect to the API server depends on whether we will be accessing it from within the cluster (our code running in the Kubernetes cluster itself) or if our code is running outside the cluster (locally)

If the code is running outside the cluster then we need to provide either the path of the config file or URL of the Kubernetes proxy server running on the cluster.

kubeconfig := filepath.Join(
os.Getenv("HOME"), ".kube", "config",
)
config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
if err != nil {
log.Fatal(err)
}
view raw proxy.sh hosted with ❤ by GitHub

OR

var (
// Set during build
version string
proxyURL = flag.String("proxy", "",
`If specified, it is assumed that a kubctl proxy server is running on the
given url and creates a proxy client. In case it is not given InCluster
kubernetes setup will be used`)
)
if *proxyURL != "" {
config, err = clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
&clientcmd.ClientConfigLoadingRules{},
&clientcmd.ConfigOverrides{
ClusterInfo: clientcmdapi.Cluster{
Server: *proxyURL,
},
}).ClientConfig()
if err != nil {
glog.Fatalf("error creating client configuration: %v", err)
}
view raw proxy_server.sh hosted with ❤ by GitHub

When the code is to be run as a part of the cluster then we can simply use

import "k8s.io/client-go/rest"  ...  rest.InClusterConfig()
view raw run_cluster.sh hosted with ❤ by GitHub

Once the connection is established we can use it to create clientset. For accessing Kubernetes objects, generally the clientset from the client-go project is used, but for CRD related operations we need to use the clientset from apiextensions-apiserver project

apiextension "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"

kubeClient, err := apiextension.NewForConfig(config)
if err != nil {
glog.Fatalf("Failed to create client: %v.", err)
}

Now we can use the client to make the API call which will create the CRD for us.

package v1alpha1
import (
"reflect"
apiextensionv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1"
apiextension "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
apierrors "k8s.io/apimachinery/pkg/api/errors"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
const (
CRDPlural string = "sslconfigs"
CRDGroup string = "blog.velotio.com"
CRDVersion string = "v1alpha1"
FullCRDName string = CRDPlural + "." + CRDGroup
)
func CreateCRD(clientset apiextension.Interface) error {
crd := &apiextensionv1beta1.CustomResourceDefinition{
ObjectMeta: meta_v1.ObjectMeta{Name: FullCRDName},
Spec: apiextensionv1beta1.CustomResourceDefinitionSpec{
Group: CRDGroup,
Version: CRDVersion,
Scope: apiextensionv1beta1.NamespaceScoped,
Names: apiextensionv1beta1.CustomResourceDefinitionNames{
Plural: CRDPlural,
Kind: reflect.TypeOf(SslConfig{}).Name(),
},
},
}
_, err := clientset.ApiextensionsV1beta1().CustomResourceDefinitions().Create(crd)
if err != nil && apierrors.IsAlreadyExists(err) {
return nil
}
return err
}
view raw createcrd.go hosted with ❤ by GitHub

In the create CRD function, we first create the definition of our custom object and then pass it to the create method which creates it in our cluster. Just like we did while creating our definition using CLI, here also we set the parameters like version, group, kind etc.

Once our definition is ready we can create objects of its type just like we did earlier using the CLI. First we need to define our object.

package v1alpha1
import meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
type SslConfig struct {
meta_v1.TypeMeta `json:",inline"`
meta_v1.ObjectMeta `json:"metadata"`
Spec SslConfigSpec `json:"spec"`
Status SslConfigStatus `json:"status,omitempty"`
}
type SslConfigSpec struct {
Cert string `json:"cert"`
Key string `json:"key"`
Domain string `json:"domain"`
}
type SslConfigStatus struct {
State string `json:"state,omitempty"`
Message string `json:"message,omitempty"`
}
type SslConfigList struct {
meta_v1.TypeMeta `json:",inline"`
meta_v1.ListMeta `json:"metadata"`
Items []SslConfig `json:"items"`
}
view raw type.go hosted with ❤ by GitHub

Kubernetes API conventions suggests that each object must have two nested object fields that govern the object’s configuration: the object spec and the object status. Objects must also have metadata associated with them. The custom objects that we define here comply with these standards. It is also recommended to create a list type for every type thus we have also created a SslConfigList struct.

Now we need to write a function which will create a custom client which is aware of the new resource that we have created.

package v1alpha1
import (
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/runtime/serializer"
"k8s.io/client-go/rest"
)
var SchemeGroupVersion = schema.GroupVersion{Group: CRDGroup, Version: CRDVersion}
func addKnownTypes(scheme *runtime.Scheme) error {
scheme.AddKnownTypes(SchemeGroupVersion,
&SslConfig{},
&SslConfigList{},
)
meta_v1.AddToGroupVersion(scheme, SchemeGroupVersion)
return nil
}
func NewClient(cfg *rest.Config) (*SslConfigV1Alpha1Client, error) {
scheme := runtime.NewScheme()
SchemeBuilder := runtime.NewSchemeBuilder(addKnownTypes)
if err := SchemeBuilder.AddToScheme(scheme); err != nil {
return nil, err
}
config := *cfg
config.GroupVersion = &SchemeGroupVersion
config.APIPath = "/apis"
config.ContentType = runtime.ContentTypeJSON
config.NegotiatedSerializer = serializer.DirectCodecFactory{CodecFactory: serializer.NewCodecFactory(scheme)}
client, err := rest.RESTClientFor(&config)
if err != nil {
return nil, err
}
return &SslConfigV1Alpha1Client{restClient: client}, nil
}
view raw register.go hosted with ❤ by GitHub

Building the custom client library

Once we have registered our custom resource definition with the Kubernetes cluster we can create objects of its type using the Kubernetes cli as we did earlier but for creating controllers for these objects or for developing some custom functionalities around them we need to build a client library also using which we can access them from go API. For native Kubernetes objects, this type of library is provided for each object.

package v1alpha1
import (
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/rest"
)
func (c *SslConfigV1Alpha1Client) SslConfigs(namespace string) SslConfigInterface {
return &sslConfigclient{
client: c.restClient,
ns: namespace,
}
}
type SslConfigV1Alpha1Client struct {
restClient rest.Interface
}
type SslConfigInterface interface {
Create(obj *SslConfig) (*SslConfig, error)
Update(obj *SslConfig) (*SslConfig, error)
Delete(name string, options *meta_v1.DeleteOptions) error
Get(name string) (*SslConfig, error)
}
type sslConfigclient struct {
client rest.Interface
ns string
}
func (c *sslConfigclient) Create(obj *SslConfig) (*SslConfig, error) {
result := &SslConfig{}
err := c.client.Post().
Namespace(c.ns).Resource("sslconfigs").
Body(obj).Do().Into(result)
return result, err
}
func (c *sslConfigclient) Update(obj *SslConfig) (*SslConfig, error) {
result := &SslConfig{}
err := c.client.Put().
Namespace(c.ns).Resource("sslconfigs").
Body(obj).Do().Into(result)
return result, err
}
func (c *sslConfigclient) Delete(name string, options *meta_v1.DeleteOptions) error {
return c.client.Delete().
Namespace(c.ns).Resource("sslconfigs").
Name(name).Body(options).Do().
Error()
}
func (c *sslConfigclient) Get(name string) (*SslConfig, error) {
result := &SslConfig{}
err := c.client.Get().
Namespace(c.ns).Resource("sslconfigs").
Name(name).Do().Into(result)
return result, err
}
view raw sslconfig.go hosted with ❤ by GitHub

We can add more methods like watch, update status etc. Their implementation will also be similar to the methods we have defined above. For looking at the methods available for various Kubernetes objects like pod, node etc. we can refer to the v1 package.

Putting all things together

Now in our main function we will get all the things together.

package main
import (
"flag"
"fmt"
"time"
"blog.velotio.com/crd-blog/v1alpha1"
"github.com/golang/glog"
apiextension "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
clientcmdapi "k8s.io/client-go/tools/clientcmd/api"
)
var (
// Set during build
version string
proxyURL = flag.String("proxy", "",
`If specified, it is assumed that a kubctl proxy server is running on the
given url and creates a proxy client. In case it is not given InCluster
kubernetes setup will be used`)
)
func main() {
flag.Parse()
var err error
var config *rest.Config
if *proxyURL != "" {
config, err = clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
&clientcmd.ClientConfigLoadingRules{},
&clientcmd.ConfigOverrides{
ClusterInfo: clientcmdapi.Cluster{
Server: *proxyURL,
},
}).ClientConfig()
if err != nil {
glog.Fatalf("error creating client configuration: %v", err)
}
} else {
if config, err = rest.InClusterConfig(); err != nil {
glog.Fatalf("error creating client configuration: %v", err)
}
}
kubeClient, err := apiextension.NewForConfig(config)
if err != nil {
glog.Fatalf("Failed to create client: %v", err)
}
// Create the CRD
err = v1alpha1.CreateCRD(kubeClient)
if err != nil {
glog.Fatalf("Failed to create crd: %v", err)
}
// Wait for the CRD to be created before we use it.
time.Sleep(5 * time.Second)
// Create a new clientset which include our CRD schema
crdclient, err := v1alpha1.NewClient(config)
if err != nil {
panic(err)
}
// Create a new SslConfig object
SslConfig := &v1alpha1.SslConfig{
ObjectMeta: meta_v1.ObjectMeta{
Name: "sslconfigobj",
Labels: map[string]string{"mylabel": "test"},
},
Spec: v1alpha1.SslConfigSpec{
Cert: "my-cert",
Key: "my-key",
Domain: "*.velotio.com",
},
Status: v1alpha1.SslConfigStatus{
State: "created",
Message: "Created, not processed yet",
},
}
// Create the SslConfig object we create above in the k8s cluster
resp, err := crdclient.SslConfigs("default").Create(SslConfig)
if err != nil {
fmt.Printf("error while creating object: %v\n", err)
} else {
fmt.Printf("object created: %v\n", resp)
}
obj, err := crdclient.SslConfigs("default").Get(SslConfig.ObjectMeta.Name)
if err != nil {
glog.Infof("error while getting the object %v\n", err)
}
fmt.Printf("SslConfig Objects Found: \n%v\n", obj)
select {}
}
view raw main.go hosted with ❤ by GitHub

Now if we run our code then our custom resource definition will get created in the Kubernetes cluster and also an object of its type will be there just like with the cli. The docker image akash125/crdblog is build using the code discussed above it can be directly pulled from docker hub and run in a Kubernetes cluster. After the image is run successfully, the CRD definition that we discussed above will get created in the cluster along with an object of its type. We can verify the same using the CLI the way we did earlier, we can also check the logs of the pod running the docker image to verify it. The complete code is available here.

Conclusion and future work

We learned how to create a custom resource definition and objects using Kubernetes command line interface as well as the Golang client. We also learned how to programmatically access a Kubernetes cluster, using which we can build some really cool stuff on Kubernetes, we can now also create custom controllers for our resources which continuously watches the cluster for various life cycle events of our object and takes desired action accordingly. To read more about CRD refer the following links:

Get the latest engineering blogs delivered straight to your inbox.
No spam. Only expert insights.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Did you like the blog? If yes, we're sure you'll also like to work with the people who write them - our best-in-class engineering team.

We're looking for talented developers who are passionate about new emerging technologies. If that's you, get in touch with us.

Explore current openings

Extending Kubernetes APIs with Custom Resource Definitions (CRDs)

Introduction:

Custom resources definition (CRD) is a powerful feature introduced in Kubernetes 1.7 which enables users to add their own/custom objects to the Kubernetes cluster and use it like any other native Kubernetes objects. In this blog post, we will see how we can add a custom resource to a Kubernetes cluster using the command line as well as using the Golang client library thus also learning how to programmatically interact with a Kubernetes cluster.

What is a Custom Resource Definition (CRD)?

In the Kubernetes API, every resource is an endpoint to store API objects of certain kind. For example, the built-in service resource contains a collection of service objects. The standard Kubernetes distribution ships with many inbuilt API objects/resources. CRD comes into picture when we want to introduce our own object into the Kubernetes cluster to full fill our requirements. Once we create a CRD in Kubernetes we can use it like any other native Kubernetes object thus leveraging all the features of Kubernetes like its CLI, security, API services, RBAC etc.

The custom resource created is also stored in the etcd cluster with proper replication and lifecycle management. CRD allows us to use all the functionalities provided by a Kubernetes cluster for our custom objects and saves us the overhead of implementing them on our own.

How to register a CRD using command line interface (CLI)

Step-1: Create a CRD definition file sslconfig-crd.yaml

apiVersion: "apiextensions.k8s.io/v1beta1"
kind: "CustomResourceDefinition"
metadata:
name: "sslconfigs.blog.velotio.com"
spec:
group: "blog.velotio.com"
version: "v1alpha1"
scope: "Namespaced"
names:
plural: "sslconfigs"
singular: "sslconfig"
kind: "SslConfig"
validation:
openAPIV3Schema:
required: ["spec"]
properties:
spec:
required: ["cert","key","domain"]
properties:
cert:
type: "string"
minimum: 1
key:
type: "string"
minimum: 1
domain:
type: "string"
minimum: 1

Here we are creating a custom resource definition for an object of kind SslConfig. This object allows us to store the SSL configuration information for a domain. As we can see under the validation section specifying the cert, key and the domain are mandatory for creating objects of this kind, along with this we can store other information like the provider of the certificate etc. The name metadata that we specify must be spec.names.plural+"."+spec.group.

An API group (blog.velotio.com here) is a collection of API objects which are logically related to each other. We have also specified version for our custom objects (spec.version), as the definition of the object is expected to change/evolve in future so it's better to start with alpha so that the users of the object knows that the definition might change later. In the scope, we have specified Namespaced, by default a custom resource name is clustered scoped. 

# kubectl create -f crd.yaml
# kubectl get crd NAME AGE sslconfigs.blog.velotio.com 5s
view raw crd.sh hosted with ❤ by GitHub

Step-2:  Create objects using the definition we created above

apiVersion: "blog.velotio.com/v1alpha1"
kind: "SslConfig"
metadata:
name: "sslconfig-velotio.com"
spec:
cert: "my cert file"
key : "my private key"
domain: "*.velotio.com"
provider: "digicert"

# kubectl create -f crd-obj.yaml
# kubectl get sslconfig NAME AGE sslconfig-velotio.com 12s
view raw create_crd.sh hosted with ❤ by GitHub

Along with the mandatory fields cert, key and domain, we have also stored the information of the provider ( certifying authority ) of the cert.

How to register a CRD programmatically using client-go

Client-go project provides us with packages using which we can easily create go client and access the Kubernetes cluster.  For creating a client first we need to create a connection with the API server.
How we connect to the API server depends on whether we will be accessing it from within the cluster (our code running in the Kubernetes cluster itself) or if our code is running outside the cluster (locally)

If the code is running outside the cluster then we need to provide either the path of the config file or URL of the Kubernetes proxy server running on the cluster.

kubeconfig := filepath.Join(
os.Getenv("HOME"), ".kube", "config",
)
config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
if err != nil {
log.Fatal(err)
}
view raw proxy.sh hosted with ❤ by GitHub

OR

var (
// Set during build
version string
proxyURL = flag.String("proxy", "",
`If specified, it is assumed that a kubctl proxy server is running on the
given url and creates a proxy client. In case it is not given InCluster
kubernetes setup will be used`)
)
if *proxyURL != "" {
config, err = clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
&clientcmd.ClientConfigLoadingRules{},
&clientcmd.ConfigOverrides{
ClusterInfo: clientcmdapi.Cluster{
Server: *proxyURL,
},
}).ClientConfig()
if err != nil {
glog.Fatalf("error creating client configuration: %v", err)
}
view raw proxy_server.sh hosted with ❤ by GitHub

When the code is to be run as a part of the cluster then we can simply use

import "k8s.io/client-go/rest"  ...  rest.InClusterConfig()
view raw run_cluster.sh hosted with ❤ by GitHub

Once the connection is established we can use it to create clientset. For accessing Kubernetes objects, generally the clientset from the client-go project is used, but for CRD related operations we need to use the clientset from apiextensions-apiserver project

apiextension "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"

kubeClient, err := apiextension.NewForConfig(config)
if err != nil {
glog.Fatalf("Failed to create client: %v.", err)
}

Now we can use the client to make the API call which will create the CRD for us.

package v1alpha1
import (
"reflect"
apiextensionv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1"
apiextension "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
apierrors "k8s.io/apimachinery/pkg/api/errors"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
const (
CRDPlural string = "sslconfigs"
CRDGroup string = "blog.velotio.com"
CRDVersion string = "v1alpha1"
FullCRDName string = CRDPlural + "." + CRDGroup
)
func CreateCRD(clientset apiextension.Interface) error {
crd := &apiextensionv1beta1.CustomResourceDefinition{
ObjectMeta: meta_v1.ObjectMeta{Name: FullCRDName},
Spec: apiextensionv1beta1.CustomResourceDefinitionSpec{
Group: CRDGroup,
Version: CRDVersion,
Scope: apiextensionv1beta1.NamespaceScoped,
Names: apiextensionv1beta1.CustomResourceDefinitionNames{
Plural: CRDPlural,
Kind: reflect.TypeOf(SslConfig{}).Name(),
},
},
}
_, err := clientset.ApiextensionsV1beta1().CustomResourceDefinitions().Create(crd)
if err != nil && apierrors.IsAlreadyExists(err) {
return nil
}
return err
}
view raw createcrd.go hosted with ❤ by GitHub

In the create CRD function, we first create the definition of our custom object and then pass it to the create method which creates it in our cluster. Just like we did while creating our definition using CLI, here also we set the parameters like version, group, kind etc.

Once our definition is ready we can create objects of its type just like we did earlier using the CLI. First we need to define our object.

package v1alpha1
import meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
type SslConfig struct {
meta_v1.TypeMeta `json:",inline"`
meta_v1.ObjectMeta `json:"metadata"`
Spec SslConfigSpec `json:"spec"`
Status SslConfigStatus `json:"status,omitempty"`
}
type SslConfigSpec struct {
Cert string `json:"cert"`
Key string `json:"key"`
Domain string `json:"domain"`
}
type SslConfigStatus struct {
State string `json:"state,omitempty"`
Message string `json:"message,omitempty"`
}
type SslConfigList struct {
meta_v1.TypeMeta `json:",inline"`
meta_v1.ListMeta `json:"metadata"`
Items []SslConfig `json:"items"`
}
view raw type.go hosted with ❤ by GitHub

Kubernetes API conventions suggests that each object must have two nested object fields that govern the object’s configuration: the object spec and the object status. Objects must also have metadata associated with them. The custom objects that we define here comply with these standards. It is also recommended to create a list type for every type thus we have also created a SslConfigList struct.

Now we need to write a function which will create a custom client which is aware of the new resource that we have created.

package v1alpha1
import (
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/runtime/serializer"
"k8s.io/client-go/rest"
)
var SchemeGroupVersion = schema.GroupVersion{Group: CRDGroup, Version: CRDVersion}
func addKnownTypes(scheme *runtime.Scheme) error {
scheme.AddKnownTypes(SchemeGroupVersion,
&SslConfig{},
&SslConfigList{},
)
meta_v1.AddToGroupVersion(scheme, SchemeGroupVersion)
return nil
}
func NewClient(cfg *rest.Config) (*SslConfigV1Alpha1Client, error) {
scheme := runtime.NewScheme()
SchemeBuilder := runtime.NewSchemeBuilder(addKnownTypes)
if err := SchemeBuilder.AddToScheme(scheme); err != nil {
return nil, err
}
config := *cfg
config.GroupVersion = &SchemeGroupVersion
config.APIPath = "/apis"
config.ContentType = runtime.ContentTypeJSON
config.NegotiatedSerializer = serializer.DirectCodecFactory{CodecFactory: serializer.NewCodecFactory(scheme)}
client, err := rest.RESTClientFor(&config)
if err != nil {
return nil, err
}
return &SslConfigV1Alpha1Client{restClient: client}, nil
}
view raw register.go hosted with ❤ by GitHub

Building the custom client library

Once we have registered our custom resource definition with the Kubernetes cluster we can create objects of its type using the Kubernetes cli as we did earlier but for creating controllers for these objects or for developing some custom functionalities around them we need to build a client library also using which we can access them from go API. For native Kubernetes objects, this type of library is provided for each object.

package v1alpha1
import (
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/rest"
)
func (c *SslConfigV1Alpha1Client) SslConfigs(namespace string) SslConfigInterface {
return &sslConfigclient{
client: c.restClient,
ns: namespace,
}
}
type SslConfigV1Alpha1Client struct {
restClient rest.Interface
}
type SslConfigInterface interface {
Create(obj *SslConfig) (*SslConfig, error)
Update(obj *SslConfig) (*SslConfig, error)
Delete(name string, options *meta_v1.DeleteOptions) error
Get(name string) (*SslConfig, error)
}
type sslConfigclient struct {
client rest.Interface
ns string
}
func (c *sslConfigclient) Create(obj *SslConfig) (*SslConfig, error) {
result := &SslConfig{}
err := c.client.Post().
Namespace(c.ns).Resource("sslconfigs").
Body(obj).Do().Into(result)
return result, err
}
func (c *sslConfigclient) Update(obj *SslConfig) (*SslConfig, error) {
result := &SslConfig{}
err := c.client.Put().
Namespace(c.ns).Resource("sslconfigs").
Body(obj).Do().Into(result)
return result, err
}
func (c *sslConfigclient) Delete(name string, options *meta_v1.DeleteOptions) error {
return c.client.Delete().
Namespace(c.ns).Resource("sslconfigs").
Name(name).Body(options).Do().
Error()
}
func (c *sslConfigclient) Get(name string) (*SslConfig, error) {
result := &SslConfig{}
err := c.client.Get().
Namespace(c.ns).Resource("sslconfigs").
Name(name).Do().Into(result)
return result, err
}
view raw sslconfig.go hosted with ❤ by GitHub

We can add more methods like watch, update status etc. Their implementation will also be similar to the methods we have defined above. For looking at the methods available for various Kubernetes objects like pod, node etc. we can refer to the v1 package.

Putting all things together

Now in our main function we will get all the things together.

package main
import (
"flag"
"fmt"
"time"
"blog.velotio.com/crd-blog/v1alpha1"
"github.com/golang/glog"
apiextension "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
clientcmdapi "k8s.io/client-go/tools/clientcmd/api"
)
var (
// Set during build
version string
proxyURL = flag.String("proxy", "",
`If specified, it is assumed that a kubctl proxy server is running on the
given url and creates a proxy client. In case it is not given InCluster
kubernetes setup will be used`)
)
func main() {
flag.Parse()
var err error
var config *rest.Config
if *proxyURL != "" {
config, err = clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
&clientcmd.ClientConfigLoadingRules{},
&clientcmd.ConfigOverrides{
ClusterInfo: clientcmdapi.Cluster{
Server: *proxyURL,
},
}).ClientConfig()
if err != nil {
glog.Fatalf("error creating client configuration: %v", err)
}
} else {
if config, err = rest.InClusterConfig(); err != nil {
glog.Fatalf("error creating client configuration: %v", err)
}
}
kubeClient, err := apiextension.NewForConfig(config)
if err != nil {
glog.Fatalf("Failed to create client: %v", err)
}
// Create the CRD
err = v1alpha1.CreateCRD(kubeClient)
if err != nil {
glog.Fatalf("Failed to create crd: %v", err)
}
// Wait for the CRD to be created before we use it.
time.Sleep(5 * time.Second)
// Create a new clientset which include our CRD schema
crdclient, err := v1alpha1.NewClient(config)
if err != nil {
panic(err)
}
// Create a new SslConfig object
SslConfig := &v1alpha1.SslConfig{
ObjectMeta: meta_v1.ObjectMeta{
Name: "sslconfigobj",
Labels: map[string]string{"mylabel": "test"},
},
Spec: v1alpha1.SslConfigSpec{
Cert: "my-cert",
Key: "my-key",
Domain: "*.velotio.com",
},
Status: v1alpha1.SslConfigStatus{
State: "created",
Message: "Created, not processed yet",
},
}
// Create the SslConfig object we create above in the k8s cluster
resp, err := crdclient.SslConfigs("default").Create(SslConfig)
if err != nil {
fmt.Printf("error while creating object: %v\n", err)
} else {
fmt.Printf("object created: %v\n", resp)
}
obj, err := crdclient.SslConfigs("default").Get(SslConfig.ObjectMeta.Name)
if err != nil {
glog.Infof("error while getting the object %v\n", err)
}
fmt.Printf("SslConfig Objects Found: \n%v\n", obj)
select {}
}
view raw main.go hosted with ❤ by GitHub

Now if we run our code then our custom resource definition will get created in the Kubernetes cluster and also an object of its type will be there just like with the cli. The docker image akash125/crdblog is build using the code discussed above it can be directly pulled from docker hub and run in a Kubernetes cluster. After the image is run successfully, the CRD definition that we discussed above will get created in the cluster along with an object of its type. We can verify the same using the CLI the way we did earlier, we can also check the logs of the pod running the docker image to verify it. The complete code is available here.

Conclusion and future work

We learned how to create a custom resource definition and objects using Kubernetes command line interface as well as the Golang client. We also learned how to programmatically access a Kubernetes cluster, using which we can build some really cool stuff on Kubernetes, we can now also create custom controllers for our resources which continuously watches the cluster for various life cycle events of our object and takes desired action accordingly. To read more about CRD refer the following links:

Did you like the blog? If yes, we're sure you'll also like to work with the people who write them - our best-in-class engineering team.

We're looking for talented developers who are passionate about new emerging technologies. If that's you, get in touch with us.

Explore current openings