Meet keepalived - High Availability and Load Balancing in One

in my quest to make my services highly available i started to look for ways to fail over an ip address from one machine to another when i started on this quest i thought i'd find something large and complex or require additional hardware or something that was only available to the highly trained system admins but basic aha doesn't need to be complex or require additional hardware it can be done with a small service and a little bit of config and that's when i discovered keep a live d keepa live id is a framework for both load balancing and high availability that implements the virtual router redundancy protocol or vrrp for short this is a protocol that you see in some routers and has been implemented inside of keepalfd it creates a virtual ip or otherwise known as a vip or floating ip that acts as a gateway that routes traffic to all participating hosts and this vip can provide high availability in this type of setup in this configuration one node is set to active and one or many nodes are set to passive and all nodes listen and communicate on the vrrp network and only the master in this configuration responds to requests in the event that the master fails vrrp communication between the nodes it will quickly fail over to a standby host making it the new master and now controlling the vip and overall limiting the downtime and increasing the availability of your services worth mentioning too that keep alive d can also be configured in a load balancer configuration in this configuration both nodes have a virtual ip and a load balancer serving traffic from real servers behind them in this scenario the traffic from the client is routed to the load balancer and then to the real server and then the real server itself responds directly to the client this configuration provides a high level of availability because any one of your load balancers or real servers could fail simultaneously while still keeping your services alive this is a more complex configuration for sure but it illustrates how flexible and how reliable keep alive d can be and i know this might sound difficult to set up but it's much easier than you think so today we'll be covering keep a live d and setting it up in a high availability scenario we'll install and configure it in an active passive mode with a vip so that you can supply high availability services to your network we'll conduct a few tests to make sure your services fail over properly and then we'll walk through some advanced use cases for keepa live tea with something like pie hole and we'll do all this with a few simple commands and config that i'll be providing to you and you can find the link to that in the description below and if that sounds good to you don't forget to give this video a thumbs up and subscribe if you haven't already so what do you need we'll need a minimum of at least two servers now these can be any flavor of linux you like but i'm gonna run this on ubuntu server and that's about all you need once you have these two servers up and running you want to make sure that you have a dedicated ip address for each now this can be with a static ip address that you assign on each node individually or it can be a dhcp reservation that gets handed out to these two nodes or you can take the belt and suspenders approach and apply both but either way you'll want to make sure that these ip addresses on these nodes do not change next you'll want to make note of the ip address you can find this by typing ipspace a and as you can see on my nodes the first one is 192 168 30. 31 and the other one is 192 168 30. 32 next we'll want to make sure that our app repositories are up to date and you can do this by running sudo apt update next we'll want to run sudo apt install keep a live d and i've also found on some systems you might need lib ipsec 13 which is a network utility for linux then after that's installed all we need to do now is configure keep a live d so first we'll focus on the first node in this group we'll want to create a config file at etsy keep alive d keep alive d.

conf and this file doesn't exist yet so don't be surprised if you don't see any configuration inside in the configuration that goes inside of that file should look like this now there are probably a lot of properties in here you don't understand so let's talk about them so first we have the rrp underscore instance so this is our instance of vrrp and it's an identifier to point back to it and we're going to name ours vi underscore 1. this can be anything you want but it needs to be consistent across all of our nodes next we're defining the default state when this instance stands up and the default state we want this one to be is master now the other option is backup but you'll see that later but in this configuration this node is going to be the master if both are in a good state next is the network interface that we'll be running on so here i have ens18 now how did i find that if you remember when we found the ip address for this node we could also see the interface that this one is running on so the interface i have is ns18 so we'll just want to plug it into there next is virtual router id and this can be anything as long as they match across instances so i'm just choosing 55. next is priority and this is how it determines which one will be master and generally speaking the backup which we'll talk about here in a little bit should be lower than the priority on the master so i'm choosing 150.

next is advert int or the advertisement interval and i'm setting it to 1. and this is actually the default value but i like to explicitly set it to 1 so i know it's there later if i need to tweak it but this is the interval or how often it will advertise itself to the other nodes and next are properties for unicast now this isn't really needed it will actually multicast if you don't have this but i'd rather set up unicast and define the nodes that it's going to advertise to so i'm going to set the unicast source ip this is a typo to this machine or this node and then unicast peer is going to be our other node which is 32. and next up is our authentication block and this is so we can supply authentication to our vrrp instances and this isn't necessary but it's a really good idea to set this with an auth type and a password for it and this can be anything you want it to be so keep it secure but i'm setting the odd type as password and the off pass as some random string and this will need to match on all the nodes that we're configuring and this last block is virtual ip address or rvip so this is the virtual ip that's going to create that all of our other services will communicate with and this should also be an ip address on your network that's not already taken so in this case i'm saying that my vip is 182 168 30.

100 and before we apply this configuration it's worth mentioning that keep it live d supports a lot of different configurations you can find this on their documentation page so if you're thinking of doing something more advanced like emailing when something goes wrong or a track script to see if the service is alive check out the documentation for more info on that so now that we have our configuration looking pretty good let's copy this and apply it to our first node and then save it now we can enable keep a live d using sudo systemctl ctl enable dash dash now keepa live d dot service and if we want to check the status we run sudo system ctl status keepa live d. service and we can see here it's active and running one thing that caught my eye in the logs and i'm glad i got this is right here line 14 truncating off pass to eight characters so that means it only supports up to eight characters now if you use the same password on both machines it really doesn't matter but we should probably limit that password to eight characters so rather than having keep a live d truncate that password for me and having a debugging nightmare sometime in the future let's just change that right now to eight characters one two three four five six seven eight and let's update the password now in our config and save it then we'll restart this service then we'll check the status again and now we can see it's not truncating our password okay let's take care of the other node now so our second instance is almost a copy and paste of the first we'll just need to update a couple of things so we'll want to keep the same instance of vi underscore one we'll want to change the state to backup we'll keep the interface as the same interface unless your machine has a different interface name but i showed you how to get that the virtual router id is going to be the same and then for priority we'll actually lower it to 100. this kind of seems backwards to me because normally a lower priority means a higher priority but what i learned is actually you want a higher number on your master than you do on your backup i guess it kind of makes sense after saying it out loud now so we'll just want to make sure that this is lower than our master so our master was 150 we'll set this one to 100.

the advertising interval we can keep as one and then so our unicast source ip is now this machine that we're on and this machine the second node is dot 32 and our pier that we're unicasting to is dot 31 next in the authentication block we're going to keep the same thing auth type is password and the auth pass is our password remember this needs to be eight characters we just learned right before this and next our virtual ip address is going to be that same vip so 192 168 so now we can copy this configuration and then go into our second node make sure you're in your second node check the ip address this one is dot 32 then we'll want to create or edit this configuration file and then we'll paste our contents inside and then we'll save it then we'll want to enable keep a live d on this machine then we'll want to check its status so the status looks good now that both of those machines are up and configured let's do some quick diagnostics first we'll ping our master node and we can see here that's responding then we'll ping our backup node and we can see that's responding now we'll ping our vip and you can see that's responding so congratulations so far keepa id is up and working as it should we're pinging this vip and we're getting a response back now that response is coming from our masternode but let's make sure we have a highly available vip right now so on the right side i'm pinging our vip and on the left i'm remoted into our master and on the bottom left i'm remoted into our backup so let's actually stop the service that's running on the first one so sudo systemctl stop keepalived. service and if we stop that we can see that our vip is still responding and just to prove our point let's actually stop it on the backup as well so if we stop it on the backup you can see that on the right side now the vip is no longer responding so let's actually start it back up and you can see it's responding again and let's start it on our master and of course it's still responding but when i started it on the master now that one actually took over so how do i actually know which one is the master right now in serving traffic well if you remote into either one of those and look at the status or the logs we could see here in the output this machine right here which is the backup entered backup state initially then became master and then became backup state so i'll keep this open on the right and on my master i will actually stop it now stopping the service and if we look at the logs again we can see it's entering master state so let's start this back up on the master now if we look at the logs again we can see it received a master advertisement letting it know that hey the master's back up and now we're entering a backup state so pretty awesome so far we have high availability ping i know not the greatest let's apply this to something simple like nginx so in this example i'm going to set up nginx on both nodes but really this could be any service that you have on two nodes that you want to provide high availability with so in this example i'm going to spin up nginx really quick using docker now this doesn't have to be docker doesn't have to be containerized it can be anything but i'm using docker because it's easy to spin up and it's easy to clean up with afterwards but if we run this command this should spin up nginx and i've created a really simple index. html that i'm mapping to and again all of this will be in the documentation if you want to try this out so if we spin that up we should now have nginx running on port 8080 on this machine and if we go to that first node we can see hello world one so this is actually working and once when our second node our backup node we're going to do the same thing except for the index.

html that i create here now i'm going to say hello world 2. so in here it's going to say hello from secondary node as the title which you don't see and then i'm going to say hello world 2. save that and then i'm going to run that same docker command we ran before which is spinning up nginx mapping a volume to nginx to the inside of an nginx html folder which maps our index.

html and then some ports just going to pull it down and spin it up and once that's up you can go to the ip address of your second node but now we see hello world 2. so this is hitting that ip address directly so this means both nodes are up individually but we're not hitting the vip yet so let's hit the bib so the vip if you recall was dot 100 and here we see the vip responding with hello world one so this means the traffic's being served from our master node which is exactly what we want now let's introduce a chaos monkey and let's stop the first note so on the left side you can see i'm hitting the vip and we're getting hello world one and now let's stop keep alive d on our primary node and let's refresh and now we're getting hello world from the second node super awesome so let's start this back up refresh and now one takes back over so super awesome now we have a highly available nginx and this is great unless you're not using nginx so let's talk about some real world scenarios in my home lab services tour i talked about how i had three buy hole instances i have my primary and my secondary and my tertiary but i have three so the first one is a virtual machine the second one is also a virtual machine but the third one's actually a raspberry pi but you might be asking how do you supply three dns's to your client don't they just have a primary and secondary yes i think that's the right answer so yes generally speaking you have a primary and secondary dns but the way that i've set this up is that my secondary dns is actually my vip so that vip that we just talked about your virtual ip address that floating ip can be your secondary dns so let's imagine for a second that you've installed and configured keep a live d on both of your pi hole instances you would actually do the same thing you would configure a node 1 and a node 2. you would configure it in an active and passive mode you would create a vrrp instance you would set the state of one of those machines to master and the other one to backup you would check to make sure you have the right interface router id priority advertisement interval and the same idea with our unicast source ip and our unicast peers so exactly the same configuration this will work for anything and you'd want to configure authentication and your odd type and your password the same exact thing and also then a vip so here's the important piece right here if you're doing this with two pi hole instances this would be your secondary dns so let's say for instance currently your secondary dns is 192.

168. 100.