In the first part of this blog, we saw how to install standalone redis service on DC/OS with Persistent storage using RexRay and AWS EBS volumes.

A single server is a single point of failure in every system, so to ensure high availability of redis database, we can deploy a master-slave cluster of Redis servers. In this blog, we will see how to setup such 6 node (3 master, 3 slave) Redis cluster and persist data using RexRay and AWS EBS volumes. After that we will see how to import existing data into this cluster.

Redis Cluster

It is form of replicated Redis servers in multi-master architecture. All the data is sharded into 16384 buckets, where every master node is assigned subset of buckets out of them (generally evenly sharded) and each master replicated by its slaves.  It provides more resilience and scaling for production grade deployments where heavy workload is expected. Applications can connect to any node in cluster mode and the request will be redirected to respective master node.

Objective: To create a Redis cluster with number of services in DCOC environment with persistent storage and import the existing Redis dump.rdb data to the cluster.

Prerequisites :  

• Make sure rexray component is running and is in a healthy state for DCOS cluster.

Steps:

• As per Redis doc, the minimal cluster should have at least 3 master and 3 slave nodes, so making it a total 6 Redis services.
• All services will use similar json configuration except changes in names of service, external volume, and port mappings.
• We will deploy one Redis service for each Redis cluster node and once all services are running, we will form cluster among them.
• We will use host network for Redis node containers, for that we will restrict Redis nodes to run on particular node. This will help us to troubleshoot cluster (fixed IP, so we can restart Redis node any time without data loss).
• Using host network adds a prerequisites that number of dcos nodes >= number of Redis nodes.

1. First create Redis node services on DCOS:
2. Click on the Add button in Services tab of DCOS UI

• Click on JSON configuration

• Add below json config for Redis service, change the values which are written in BLOCK letters with # as prefix and suffix.
• #NODENAME# - Name of Redis node (Ex. redis-node-1)
• #NODEHOSTIP# - IP of dcos node on which this Redis node will run. This ip must be unique for each Redis node. (Ex. 10.2.12.23)
• #VOLUMENAME# - Name of persistent volume, Give name to identify volume on AWS EBS (Ex. <dcos cluster="" name="">-redis-node-<node number="">)</node></dcos>
• #NODEVIP# - VIP For the Redis node. It must be ‘Redis’ for first Redis node, for others it can be the same as NODENAME (Ex. redis-node-2)

CODE: https://gist.github.com/velotiotech/f51520210ab7f368582e102403bae6ed.js

• After updating the highlighted fields, copy above json to json configuration box, click on ‘Review & Run’ button in the right corner, this will start the service with above configuration.
• Once above service is UP and Running, then repeat the step 2 to 4 for each Redis node with respective values for highlighted fields.
• So if we go with 6 node cluster, at the end we will have 6 Redis nodes UP and Running, like:

Note: Since we are using external volume for persistent storage, we can not scale our services, i.e. each service will only one instance max. If we try to scale, we will get below error :

2. Form the Redis cluster between Redis node services:

• To create or manage Redis-cluster, first deploy redis-cluster-util container on DCOS using below json config:

CODE: https://gist.github.com/velotiotech/5dab10b51d13a520826c1eac7aebc322.js

This will run service as :

• Get the IP addresses of all Redis nodes to form the cluster, as Redis-cluster can not be created with node's hostname / dns. This is an open issue.

Since we are using host network, we need the dcos node IP on which Redis nodes are running.

Get all Redis nodes IP using:

CODE: https://gist.github.com/velotiotech/b00a40567241c51b53368ae60b0aae74.js

Here Redis-node is the prefix used for all Redis nodes.

Note the output of this command, we will use it in further steps.

• Get the node where redis-cluster-util container is running and ssh to dcos node using:

CODE: https://gist.github.com/velotiotech/74845079a1cb1218382fcde8391ecf60.js

• Now find the docker container id of redis-cluster-util and exec it using:

CODE: https://gist.github.com/velotiotech/d77e6a74c5c7d27b792662e9c91774a8.js

• No we are inside the redis-cluster-util container. Run below command to form Redis cluster.

CODE: https://gist.github.com/velotiotech/22ad6aaf5fa39b0da8eb6145de0a2a84.js

• Here use the Redis nodes IP addresses which retrieved in step 2.

CODE: https://gist.github.com/velotiotech/c04282316b63e7f70929c40e3709b86e.js

• Parameters:
• The option --replicas 1 means that we want a slave for every master created.
• The other arguments are the list of addresses (host:port) of the instances we want to use to create the new cluster.
• Output:
• Select ‘yes’ when it prompts to set the slot configuration shown.
• Run below command to check the status of the newly create cluster

CODE: https://gist.github.com/velotiotech/d04a7202744c922de6de13c4abc5bca9.js

• Parameters:
• host:port of any node from the cluster.
• Output:
• If all OK, it will show OK with status, else it will show ERR with the error message.

3. Import existing dump.rdb to Redis cluster

• At this point, all the Redis nodes should be empty and each one should have an ID and some assigned slots:

Before reuse an existing dump data, we have to reshard all slots to one instance. We specify the number of slots to move (all, so 16384), the id we move to (here Node 1 - 10.0.1.90:6379) and where we take these slots from (all other nodes).

CODE: https://gist.github.com/velotiotech/e35eb32089b00f8569de1bb1eb089705.js

Parameters:

host:port of any node from the cluster.

Output:

It will prompt for number of slots to move - here all. i.e 16384

Receiving node id - here id of node 10.0.1.90:6379 (redis-node-1)

Source node IDs  - here all, as we want to shard  all slots to one node.

And prompt to proceed - press ‘yes’

• Now check again node 10.0.1.90:6379  

CODE: https://gist.github.com/velotiotech/26201ab7408765a4e76114f2b3eea654.js

Parameters: host:port of any node from the cluster.

Output: it will show all (16384) slots moved to node 10.0.1.90:6379

• Next step is Importing our existing Redis dump data.  

Now copy the existing dump.rdb to our redis-cluster-util container using below steps:

- Copy existing dump.rdb to dcos node on which redis-cluster-util container is running. Can use scp from any other public server to dcos node.

- Now we have dump .rdb in our dcos node, copy this dump.rdb to redis-cluster-util container using below command:

CODE: https://gist.github.com/velotiotech/dd90b53320bcd18fa8206478887640ac.js

Now we have dump.rdb in our redis-cluster-util container, we can import it to our Redis cluster. Execute and go to the redis-cluster-util container using:

CODE: https://gist.github.com/velotiotech/03b241c12d1506915ec4b89888bb8c70.js

It will execute redis-cluster-util container which is already running and start its bash cmd.

Run below command to import dump.rdb to Redis cluster:

CODE: https://gist.github.com/velotiotech/1688127715d87ed9a9c62ad4f8ad3b4c.js

Parameters:

Path to dump.rdb

host:port of any node from the cluster.

Output:

If successful, you’ll see something like:

CODE: https://gist.github.com/velotiotech/4d16ab70ba90c22ef0511deaf9dcd0cd.js

as well as this in the Redis server logs:

CODE: https://gist.github.com/velotiotech/e00890f4e1cc67e4d51dbb2354e608ad.js

WARNING:
Like our Oracle DB instance can have multiple databases, similarly Redis saves keys in keyspaces.
Now when Redis is in cluster mode, it does not accept the dumps which has more than one keyspaces. As per documentation:
‍
“Redis Cluster does not support multiple databases like the stand alone version of Redis. There is just database 0 and the SELECT command is not allowed. “‍

So while importing such multi-keyspace Redis dump, server fails while starting on below issue :

CODE: https://gist.github.com/velotiotech/ab78b1975756dcfa8e618a2c86066a6b.js

There is redis-cli command “MOVE” to move keys from one keyspace to another keyspace.

Also can run below command to move all the keys from keyspace 1 to keyspace 0 :

CODE: https://gist.github.com/velotiotech/6f0a804e04837c09ec7f0ec65feeac84.js

• Verify import status, using below commands : (inside redis-cluster-util container)

CODE: https://gist.github.com/velotiotech/9c53a765de8349e71b0b4e5fc99d4eb2.js

It will run Redis info command on node 10.0.1.90:6379 and fetch keyspace information, like below:

# Keyspace
db0:keys=33283,expires=0,avg_ttl=0

• Now reshard all the slots to all instances evenly

The reshard command will again list the existing nodes, their IDs and the assigned slots.

CODE: https://gist.github.com/velotiotech/8e3a41bce4f6bdfc0e1baf72aa36a17e.js

Parameters:

host:port of any node from the cluster.

Output:

It will prompt for number of slots to move - here (16384 /3 Masters = 5461)

Receiving node id - here id of master node 2  

Source node IDs  - id of first instance which has currently all the slots. (master 1)

And prompt to proceed - press ‘yes’

Repeat above step and for receiving node id, give id of master node 3.

• After the above step, all 3 masters will have equal slots and imported keys will be distributed among the master nodes.
• Put keys to cluster for verification

CODE: https://gist.github.com/velotiotech/a695b2b8a5f6eaadd551e13db87df845.js

Above error shows that server saved this key to instance 10.0.9.203:6379, so client redirected it. To follow redirection, use flag “-c” which says it is a cluster mode, like:

CODE: https://gist.github.com/velotiotech/396cf7a41813099aa8a8b5addecc317c.js

Redis Entrypoint

Application entrypoint for Redis cluster is mostly depends how your Redis client handles cluster support. Generally connecting to one of master nodes should do the work.

Use below host:port in your applications :

redis.marathon.l4lb.thisdcos.directory:6379

Automation of Redis Cluster Creation

We have automation script in place to deploy 6 node Redis cluster and form a cluster between them.

Script location: Github

• It deploys 6 marathon apps for 6 Redis nodes. All nodes are deployed on different nodes with CLUSTER_NAME as prefix to volume name.

• Once all nodes are up and running, it deploys redis-cluster-util app which will be used to form Redis cluster.
• Then it will print the Redis nodes and their IP addresses and prompt the user to proceed cluster creation.
• If user selects to proceed, it will run redis-cluster-util app and create the cluster using IP addresses collected. Util container will prompt for some input that the user has to select.

Conclusion

We learned about Redis cluster deployment on DCOS with Persistent storage using RexRay. We also learned how rexray automatically manages volumes over aws ebs and how to integrate them in DCOS apps/services. We saw how to use redis-cluster-util container to manage Redis cluster for different purposes, like forming cluster, resharding, importing existing dump.rdb data etc. Finally, we looked at the automation part of whole cluster setup using dcos cli and bash.

Reference

• Persistent Volumes
• Storage Management Solution
• Redis Cluster Official Tutorial
• Redis Cluster Dockerfile
• Redis Cluster Util Dockerfile
• Redis Clients