Multitenant HA Redis on AWS

In this post I'll demonstrate one of the many ways to setup a multitenant and highly available Redis cluster using Amazon Web Services, OpenVZ containers, Open vSwitch with GRE tunneling, HAProxy and keepalived on CentOS 6.5.

This is what the architecture looks like:


I'll use one VPC, with two Subnets, 3 EC2 instances for the Redis and Sentinel containers, and two more EC2 instances for the HAProxy and keepalived proxy layer.

Each OpenVZ container will be part of an isolated network, the entry point to which is the HAProxy node. This is achieved by using Open vSwitch with a mesh of GRE tunnels.

First create the VPC subnets and the EC2 instances, then on the Redis instances install the OVZ Kernel:
After you reboot into the OVZ kernel install OVS and the high performance SR-IOV ixgbevf NIC driver:
To enable the SR-IOV on the instances stop them first then using the aws cli run:
Start the instances back and create the GRE tunnels between them, to enable the isolated communication between the OVZ containers:
Next, we need to ensure that each new OVZ container virtual interface is added to the OVS bridge we just created during start, by creating the following two scripts:
Line 42 is the one that does that and needs to be changed if you are using the script provided with OVZ, as it uses the standard Linux bridge instead.

On each host, create a container and install Redis and Sentinel inside of it: I created my own Debian Wheezy template using debootstrap, but you can use upstream templates as well.

The Redis and Sentinel configs for the cluster follow:
Make sure the virtual interface of the container has been added to the OVS bridge (line 19):
With this the 3 Redis/Sentinel nodes are complete. Next let's move on to EC2 Node 4 and 5.

First, create the same GRE tunnels to the other 3 EC2 nodes as shown previously, the main difference here is that the bridge interface will have an IP that's part of the GRE mesh subnet to allow the other hosts to reach the containers:
Keepliaved will be using unicast messages between Node 4 and Node 5. When the state of the node changes it will trigger a script defined on line 29, which will re-assign the private IP from one of the EC2 instances to the other, and keepliaved will raise the IP and send an unsolicited ARP broadcast. For this to work we need to have the aws cli installed (lines 3-5), and auth working (lines 71).

If Node 4 and 5 have a public IP you should be able to connect to the current Redis master, as decided by the Sentinels by using the redis-cli command. To test a failover just shut down redis-server on the current master container and watch the sentinels promote a new master and HAProxy detecting the change (lines 107-113).