Overview

We will cover how to set up mutual TLS authentication and authorization on Amazon MSK.

Amazon MSK is a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data. You can enable client authentication with TLS for connections and client authorization from your applications to your Amazon MSK brokers and ZooKeeper nodes. 

Prerequisites

• Terraform: For creating a private CA and MSK Cluster
• AWS CLI: For creating TLS certificates (the user must have access to create a private CA, issue certificates, and create MSK cluster)

Setup TLS authentication and authorization

To use client authentication with TLS on MSK, you need to create the following resources:

• AWS Private CA
• MSK cluster with TLS encryption enabled
• Client certificates

Create AWS Private CA

AWS Private CA can be either in the same AWS account as your cluster, or in a different account. For information about AWS Private CAs, see Creating and Managing a AWS Private CA. In this setup, we will use Terraform to create a private CA.

Steps to create Private CA

1. Run below Terraform code to create the Private CA.

CODE: https://gist.github.com/velotiotech/93cb17d300b017736b163b247a5ae843.js

2. Once the private CA is created, install the certificate from the AWS console.

Steps to install the certificate.

• If you are not already on the CA's details page, open the AWS Private CA console at https://console.aws.amazon.com/acm-pca/home. On the private certificate authorities page, choose a root CA that you have created with the certificate status as Pending or Active.
• Choose Actions, and installthe  CA certificate to open the Install root CA certificate page.
• Under Specify the root CA certificate parameters, specify the following certificate parameters:

• Validity — Specifies the expiration date and time for the CA certificate. The AWS Private CA default validity period for a root CA certificate is ten years.
• Signature algorithm — Specifies the signing algorithm to use when the root CA issues new certificates. Available options vary according to the AWS Region where you are creating the CA. For more information, see Compatible signing algorithms, Supported cryptographic algorithms, and SigningAlgorithm in CertificateAuthorityConfiguration.

• SHA256 RSA

• Review your settings to make sure they’re correct, then choose Confirm and install.        
• The details page for the CA displays the status of the installation (success or failure) at the top. If the installation was successful, the newly completed root CA displays a status of Active in the General pane.

Create an MSK cluster that supports TLS client authentication.

Note: We highly recommend using independent AWS Private CA for each MSK cluster when you use mutual TLS to control access. Doing so will ensure that TLS certificates signed by PCAs only authenticate with a single MSK cluster.

Run the below Terraform code to create MSK cluster

Note: Update attributes as per the requirement and configurations.

CODE: https://gist.github.com/velotiotech/eb53c26076bad4021f40f6d7015c1d4d.js

It will take 15-20 minutes to create the MSK cluster. 

Note: Since the bootstrap URL will be used to communicate with the MSK cluster using the Kafka CLI or SDKs, save it from the Terraform output.

Create TLS certificates using previously created AWS Private CA

We will create two certificates, one is for admin access, and the other one is for client access. For creating certificates, a common name (CN) is required. The CN is used as a principal while granting permissions through kafka ACLs

Create admin TLS certificate

Steps to create TLS certificate

1. Generate CSR and key.

CODE: https://gist.github.com/velotiotech/0eb72247a18ea0326d3020241bd61d97.js

2. Issue certificate using previously created private CA (replace <CA_ARN> with the ARN of the AWS Private CA that you created).

CODE: https://gist.github.com/velotiotech/c3af62b4ac8866c8ab59336cb3d75842.js

3. Get the certificate ARN issued in the previous step.

CODE: https://gist.github.com/velotiotech/1803bde3cdea17cd9c40468d29e03d2f.js

4. Export the certificate in pkcs12 format.

CODE: https://gist.github.com/velotiotech/58cd0ddb0adf3a9d8d4bbee75443cfa6.js

Create client TLS certificate

1. Generate CSR and key

CODE: https://gist.github.com/velotiotech/c9877fe1a83d82e880c8085496134234.js

2. Issue certificate using previously created private CA (replace <CA_ARN> with the ARN of the AWS Private CA that you created).

CODE: https://gist.github.com/velotiotech/5c81954c2eb017e4cf451f055b4a72d0.js

3. Get certificate ARN issue in the previous step.

CODE: https://gist.github.com/velotiotech/656990a0f665d471123e915c3141570e.js

4. Export the certificate in pkcs12 format.

CODE: https://gist.github.com/velotiotech/dbc8200e4f3bcd3eade6c9831d7e0060.js

Setup a client machine to interact with the MSK cluster

1. Create an Amazon EC2 instance to use as a client machine. For simplicity, create this instance in the same VPC you used for the cluster. See Step 3: Create a client machine for an example of how to create such a client machine.
2. Copy previously created certificates admin.p12 and client.p12 into the client machine.
3. Install java8+ on the client machine.
4. Download Kafka binaries and extract

https://archive.apache.org/dist/kafka/3.4.0/kafka_2.13-3.4.0.tgz

5. Create admin and client configuration files for authentication  and authorization.

CODE: https://gist.github.com/velotiotech/62ea110fcdd8de79669e95cccc274ce1.js

Test Authentication and Authorization using ACLs

Create Admin ACLs for granting admin access to clusters, topics, and groups

By default, the MSK cluster will allow everyone if no ACL is found. Here the Admin ACL will be the first ACL. The Admin user ("User:CN=admin") will leverage on Admin ACL to grant permissions to Client User("User:CN=client").

ACL for managing cluster operations (Admin ACL).

CODE: https://gist.github.com/velotiotech/1a675ab5def2a26e692c6ce98d3a9a37.js

ACL for managing topics permissions (Admin ACL).

CODE: https://gist.github.com/velotiotech/b468011150e41fde6fe2937b4587749a.js

ACL for managing group permissions (Admin ACL).

CODE: https://gist.github.com/velotiotech/c086be686a5458e6659dc01784fdaf73.js

Create a topic.

CODE: https://gist.github.com/velotiotech/e7ce4c2205c37687f05617c292a8dff6.js

List topic and check the topic is created.

CODE: https://gist.github.com/velotiotech/29564a197f1aa16315a20c77310628f3.js

Grant write permission to the topic so that client (producer) can publish messages to the topic (Use admin user for granting access to client).

CODE: https://gist.github.com/velotiotech/3554c5e450cdb652e5a4b98bb73a4f74.js

Publish messages to the topic using client user.

CODE: https://gist.github.com/velotiotech/583a6cf5ffd5147823a33d3199eff2f0.js

Consume messages.

Note: If you try to consume messages from the topic using a consumer group, you will get a group authorization error since the client user is not authorized to access groups.

CODE: https://gist.github.com/velotiotech/ae1e0382b47ecdd187de77ff5f0c0bdf.js

Grant group permission to the client user.

CODE: https://gist.github.com/velotiotech/e634d6e1e8ab10643ec707adb6a591d7.js

After providing group access, the client user should be able to consume messages from the topic using a consumer group.

This way, you can manage client access to the topics and groups.

Additional Commands 

List ACL.

CODE: https://gist.github.com/velotiotech/8009434f7c07d8416653bca781583eb5.js

Delete ACL (Create and delete ACL commands are same except –aad/–remove argument).

CODE: https://gist.github.com/velotiotech/6faf165cb0a807c5bf3ebc1cfa4f7036.js

Conclusion

AWS MSK eases the effort of managing independently hosted Kafka clusters. Users can scale Kafka brokers and storage as necessary. MSK supports TLS encryption and allows users to create TLS connections from the application to Amazon MSK brokers and ZooKeeper nodes with the help of the AWS Private CA, which enables users to create certificates for authentication.