Thanks! We'll be in touch in the next 12 hours
Oops! Something went wrong while submitting the form.

Installing Redis Service in DC/OS With Persistent Storage Using AWS Volumes

Parvez Kazi

Cloud & DevOps

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker.

It supports various data structures such as Strings, Hashes, Lists, Sets etc. DCOS offers Redis as a service

Why Do We Use External Persistent Storage for Redis Mesos Containers?

Since Redis is an in-memory database, an instance/service restart will result in loss of data. To counter this, it is always advisable to snapshot the Redis in-memory database from time to time.

This helps Redis instance to recover from the point in time failure.

In DCOS, Redis is deployed as a stateless service. To make it a stateful and persistent data, we can configure local volumes or external volumes.

The disadvantage of having a local volume mapped to Mesos containers is when a slave node goes down, your local volume becomes unavailable, and the data loss occurs.

However, with external persistent volumes, as they are available on each node of the DCOS cluster, a slave node failure does not impact the data availability.

Rex-Ray

REX-Ray is an open source, storage management solution designed to support container runtimes such as Docker and Mesos.

REX-Ray enables stateful applications such as databases to persist and maintain its data after the life cycle of the container has ended. Built-in high availability enables orchestrators such as Docker Swarm, Kubernetes, and Mesos Frameworks like Marathon to automatically orchestrate storage tasks between hosts in a cluster.

Built on top of the libStorage framework, REX-Ray's simplified architecture consists of a single binary and runs as a stateless service on every host using a configuration file to orchestrate multiple storage platforms.

Objective: To create a Redis service in DC/OS environment with persistent storage.

Warning: The Persistent Volume feature is still in beta Phase for DC/OS Version 1.11.

Prerequisites:

  • Make sure the rexray service is running and is in a healthy state for the cluster.
Rexray Service

Steps:

  • Click on the Add button in Services component of DC/OS GUI.
DCOS UI
  • Click on JSON Configuration.  
JSON Configuration

Note: For persistent storage, below code should be added in the normal Redis service configuration JSON file to mount external persistent volumes.

CODE: https://gist.github.com/velotiotech/2a23b12f1d96805c1abc8dcd25a3f940.js

  • Make sure the service is up and in a running state:
Running Redis Service

If you look closely, the service was suspended and respawned on a different slave node. We populated the database with dummy data and saved the snapshot in the data directory.

When the service did come upon a different node 10.0.3.204, the data persisted and the volume was visible on the new node.

CODE: https://gist.github.com/velotiotech/8adadb858ad605f74d8014423a1e6048.js

  •  Check the volume tab :
Check the Volume Tab

Note: For external volumes, the status will be unavailable. This is an issue with DC/OS.

The Entire Service JSON file:

CODE: https://gist.github.com/velotiotech/13fa86a37a02bc92f4724c8d439b6e89.js

Redis entrypoint

To connect with Redis service, use below host:port in your applications:

redis.marathon.l4lb.thisdcos.directory:6379

Conclusion

We learned about Standalone Redis Service deployment from DCOS catalog on DCOS.  Also, we saw how to add Persistent storage to it using RexRay. We also learned how RexRay automatically manages volumes over AWS ebs and how to integrate them in DCOS apps/services.  Finally, we saw how other applications can communicate with this Redis service.

References

Get the latest engineering blogs delivered straight to your inbox.
No spam. Only expert insights.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Did you like the blog? If yes, we're sure you'll also like to work with the people who write them - our best-in-class engineering team.

We're looking for talented developers who are passionate about new emerging technologies. If that's you, get in touch with us.

Explore current openings

Installing Redis Service in DC/OS With Persistent Storage Using AWS Volumes

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker.

It supports various data structures such as Strings, Hashes, Lists, Sets etc. DCOS offers Redis as a service

Why Do We Use External Persistent Storage for Redis Mesos Containers?

Since Redis is an in-memory database, an instance/service restart will result in loss of data. To counter this, it is always advisable to snapshot the Redis in-memory database from time to time.

This helps Redis instance to recover from the point in time failure.

In DCOS, Redis is deployed as a stateless service. To make it a stateful and persistent data, we can configure local volumes or external volumes.

The disadvantage of having a local volume mapped to Mesos containers is when a slave node goes down, your local volume becomes unavailable, and the data loss occurs.

However, with external persistent volumes, as they are available on each node of the DCOS cluster, a slave node failure does not impact the data availability.

Rex-Ray

REX-Ray is an open source, storage management solution designed to support container runtimes such as Docker and Mesos.

REX-Ray enables stateful applications such as databases to persist and maintain its data after the life cycle of the container has ended. Built-in high availability enables orchestrators such as Docker Swarm, Kubernetes, and Mesos Frameworks like Marathon to automatically orchestrate storage tasks between hosts in a cluster.

Built on top of the libStorage framework, REX-Ray's simplified architecture consists of a single binary and runs as a stateless service on every host using a configuration file to orchestrate multiple storage platforms.

Objective: To create a Redis service in DC/OS environment with persistent storage.

Warning: The Persistent Volume feature is still in beta Phase for DC/OS Version 1.11.

Prerequisites:

  • Make sure the rexray service is running and is in a healthy state for the cluster.
Rexray Service

Steps:

  • Click on the Add button in Services component of DC/OS GUI.
DCOS UI
  • Click on JSON Configuration.  
JSON Configuration

Note: For persistent storage, below code should be added in the normal Redis service configuration JSON file to mount external persistent volumes.

CODE: https://gist.github.com/velotiotech/2a23b12f1d96805c1abc8dcd25a3f940.js

  • Make sure the service is up and in a running state:
Running Redis Service

If you look closely, the service was suspended and respawned on a different slave node. We populated the database with dummy data and saved the snapshot in the data directory.

When the service did come upon a different node 10.0.3.204, the data persisted and the volume was visible on the new node.

CODE: https://gist.github.com/velotiotech/8adadb858ad605f74d8014423a1e6048.js

  •  Check the volume tab :
Check the Volume Tab

Note: For external volumes, the status will be unavailable. This is an issue with DC/OS.

The Entire Service JSON file:

CODE: https://gist.github.com/velotiotech/13fa86a37a02bc92f4724c8d439b6e89.js

Redis entrypoint

To connect with Redis service, use below host:port in your applications:

redis.marathon.l4lb.thisdcos.directory:6379

Conclusion

We learned about Standalone Redis Service deployment from DCOS catalog on DCOS.  Also, we saw how to add Persistent storage to it using RexRay. We also learned how RexRay automatically manages volumes over AWS ebs and how to integrate them in DCOS apps/services.  Finally, we saw how other applications can communicate with this Redis service.

References

Did you like the blog? If yes, we're sure you'll also like to work with the people who write them - our best-in-class engineering team.

We're looking for talented developers who are passionate about new emerging technologies. If that's you, get in touch with us.

Explore current openings