so uh so thanks for uh Deo for coming here uh and he's going to give a talk on raft a consensus protocol on the lines of pus and Zab uh he's been working on this with professor John Hout for the last you know like I don't know two years two years uh he's just finished his uh PhD recently uh you know I think we should just give him a round of applause for that um so over to you thanks uh I'll just jump right in so I'm going to go over kind of an intro to
consensus and then uh deeper on the raft algorithm itself what what is consensus it's a bit hard to Define but it's agreement on shared state so you have multiple servers you want them to kind of agree and present a consistent view of some State uh and yet you want to be able to tolerate failures so a typical consensus cluster can recover from server failures autonomously like here in a five server cluster uh if any majority of the servers are available the whole system will be available fully operational so you can lose two servers no problem
lose the third server you'll you'll no longer be available but you'll maintain consistency at all times and this is key to building consistent storage systems uh so any fault tolerant consistent system system uh should have consensus somewhere inside of it um if you Peak inside at the systems you'll usually see uh two different things you might see a to-do that says uh this look hard so let's let's leave this for later eliminate this single point of failure sometime later or worse you might see some sort of ad hoc algorithm where you know consensus look too
hard people kind of try approximate it and you end up with excuses like this case is rare and typically occurs as a result of a network partition with replication lag you know something that never happens in the real world um or in a in a properly built system you'll at some level you'll find a consensus algorithm so either paos or raft or Zab uh this might be built into the system might be a library it might even be a network service so zookeepers quite popular here uh where you you access a network service that provides
the consensus you need and don't have to complicate your own uh application so consensus you typically use it to build what's called a replicated State machine and it's a replicated State machine that you then use in your larger system this kind of the replicated State machine AR uh architecture where by state machine what I mean is just some data structure that you want your clients to interact with and here I'm showing kind of a simple map like X to one y is two uh something that'll change over time deterministic and then uh so I'm showing
a cluster of three servers here where each server has its own copy of the state machine but the system as a whole is going to provide the illusion that there's just one coh State machine each server has its own State machine instead of taking uh commands to modify that directly from clients the state machine just each state machine plays commands from a log so first apply X gets three then y gets two Etc the idea is that if all the logs on all the different servers look the same have the same series of commands then
all of the state machines will end up in the same state and produce the same ass so for clients so that's how you ride the illusion of a single system it's the job of a consensus module to ensure that to ensure proper log replication so to take commands from clients to order order them consistently into the log to persist them there uh so for most of the talk I'll just focus on providing a replicated log the state machine really kind of the application on top um and as I mentioned before the system will make progress
as long as any majority of the servers are up uh that's for a typical algorithm like paos ra zap and uh it can handle servers coming and going messages kind of being arbitrarily delayed or lost uh but no bantine failures so servers can't apply you so then how do you use replicated State machines in a real system well there's kind of two categories for this one would be use it for to manage your top level system configuration um and the other is to replicate an entire database state so here here in the top level system
configuration you'll have exactly one replicated State machine and it might manage things like what server are in the cluster um you know what uh what shards of my database if each server have or what what part of the namespace is you server in charge of so on the left you can imagine a system where uh most of your clust or all all of your cluster servers just interact directly with the replicated State machine to communicate so maybe they're uh pulling work off of a work CU and and that that work CU is the state machine
um on the right right you see a common simplification where uh you want some active entity to manage your cluster but you don't want that thing to be a s single point of failure this is actually uh this is my use case uh so I started raft as part of ramcloud um and we have a single cluster coordinator that you know if in in the past if it failed um that was it for your whole system so instead what we do now store all of that coordinator the critical coordinator state in a replicated State machine
and then there standby coordinators that uh can be elected and pull that state out if they need to and then then they can take the role of coordinator a very different approach um is if you want to take all of your database State and put it manage it using consensus so this this the approach taken by spanner megga store uh Hydra base where you kind of partition all your data and you assign each partition to a replicated State machine uh and quite often um you know some of the members of your replicated each replicated State
machine might span the globe so that has the advantage that uh kind of each partition can autonomously recover from failures and uh without any involvement of some Central Authority and if you need to undergo operations across partitions in an atomic way you can kind of use a a two-face permit protocol pretty simply because you know that none of these replicated State machines will fail or it's highly unlikely for them to fail um that's kind of how replicated State machines are used in practice now getting back to the details of how they're implemented um historically it's
it's been the paxos protocol that's basically synonymous with consensus today as created by Leslie Lort the series touring Award winner and unfortunately there's two key problems with paxos I'll summarize them using these quotes So an nsdi reviewer wrote The Dirty Little Secret of the nsdi community is that at most five people really truly understand every part of paxos wink as in paxos is really hard to understand even the one of the top systems conference full of academics uh we don't get it and it's a fundamental protocol for distributed systems and then the chubby authors so
chubby is a replicated State machine uh inside of Google they wrote there are significant gaps between the description of the pxs of algorithm and the needs of a real world system the final system will be based on an unproven protocol so not only is this thing really hard to understand it's actually really hard to implement in a real system um these are the problems that we tried to address with raft we we designed ref specifically for understandability we wanted our our goal was to find the best algorithm for building real practical systems so it had
to be correct had to be complete had to perform well these are things that paos already did but it also had to be understandable so that you could teach it and so that you could implement it at a reasonable cost and then at every step of the way when we reached a design decision we asked what would be easier to understand or what would be easier to explain that's kind of the criteria we Ed uh to explore a new design what we ended up with is a fundamentally different decomp decomposition from what paxos does so
raft relies pretty heavily on a leader whereas that's kind of an afterthought in taxos uh the algorithm is has less complexity in its state space so it kind of it's it operates in a more restricted form which makes it simpler to understand and overall it just has much less mechanism uh so you'll see in a minute how ra is actually quite compact these things uh things again lead to easier to teach easier to implement so on the teaching we actually did a study to compare um you know if is raft actually easier for people to
understand uh I don't have time to get into the details of the study but basically we we took a classroom of students at Stanford and another classroom at Berkeley and we taught them both paxos and raft uh using some video lectures you can actually find these on YouTube and on the left uh you can see we quiz the students after and 33 of the 43 students actually scored higher than raft after spending about the same time learning it um then on the right we we asked them you know if you had to implement one of
these or you had to explain one of these to a fellow student which one would you choose paxos or rra and nearly everyone said WRA in that case so it's really encouraging for the understandability uh as I said I don't have time to get into the details I think what's more important to do in this talk is to try to just teach you raft and you can then decide for yourself so I'll be using a visualization to do that um but first kind of an overview so that we're not too lost so we have to
split SP up into three parts uh leader election log replication that's kind of what happens during normal operation and then safety to tie these things together and actually create a consistent system so in leader election the idea is to uh select one of the servers to act as the cluster leader and if that server were to go down we want to detect a crash and select a new leader to take its place in log replication uh it's that Leader's job to take commands from clients these are the commands that are ultimately destined for for the
replicated State machine append those commands to the leader log and then make the leaders make the other server log match the leaders the leader will replicate its log to the other servers and override any inconsistencies the third section is safety um where so we add a restriction on leader election here so that only a server with an upto-date log can become leader and given that uh when the leader replicates its log to the other servers and the other servers just override any inconsistencies uh that actually is safe since the leader log is actually known to
be good let me flip over okay there there's going to be a little best uh feel free to pause me but I can't really go back so leader election keep your eye out on server two it's about to Time Out start an election and become leader so there it's timing out and now it's leader and it's first job is to send a round of heartbeats to the cluster that was a full leader election it happens in one round uh that was a little too quick to understand so I'm going to go into it in more
detail now so you'll note that uh This Server is leader in term two terms are used to distinguish leaders newer leaders from older leaders it's kind of a logical clock um so I'm going to crash F two now and uh you'll see fever 3 become leader in term three but server three first times out and increments its term to term three that's the number here and also the pink color um you notice that the other fers are still on term to uh that's because we're you know we're in a Distributive system here where not uh
nothing can happen atomically across the whole cluster and so uh these servers will stay in term two for now but every message that's sent in raft actually includes the sender's term number and the recipient will adopt the largest term number it's ever seen so as soon as these other servers see uh receive these messages they will also adopt term three okay and uh you feel these dots around server 3 um those represent votes so server 3 has entered What's called the candidate State it's trying to become leader itself and to do so it needs to
collect votes from a majority of the cluster uh servers are kind of greedy so the first thing they do is vote for themselves that's why it's already got one dot filled in and so um the these these messages you see uh they're called request vote rpcs and so the purpose is for server server 3 is asking the other servers please can I have your vote okay so fast learning just a little bit um now these three other servers have received the messages so they updated themselves to term three and they all voted for Server 3
it's the plus sign you see um so server is allowed to vote at most once per term and for now it's just going to vote first come first sered but as soon as server three learns that it has received the majority of the votes then it'll become leader in term three and it will its first job as Leaders to send out a round of heartbeats there are the heartbeats and that leader election is done okay so that those two were like good normal case elections that's what you would expect to happen uh 90 95% of
the time what we needed though was for one of those servers to time out ahead of the others so if that doesn't happen we've got what's called a split vote and I'll show you that now uh the note server one and server five they're they're about to time out at the exact same time and they're going to campaign for leadership uh in term four so they're both candidates now and they both voted for themselves and they both broadcast out their request vote messages uh unfortunately for them it just so happens that each of them reaches
one of the other servers first so each of them is actually only going to collect one more vote that'll leave them with two out of five votes not enough for to become leader well that's kind of unfortunate you can see they're trying to get server 2's boat but I killed server 2 um so that's not going to work so now server one server five are kind of stuck in this candidate State uh the cluster has no leader what do we do well in R we do this simple stupid thing which is just wait for another
timeout to go by and in this case you'll see fiver 3 at actually times out in term five and is able to collect votes in term five become leader so how do we how do we kind of encourage that behavior well we randomize those timeouts so it's not uh you know it's not that all the servers are synchronized using the same timeouts in fact when they reset their timeout you might have noticed it resets randomly somewhere between halfway and all the way around the circle uh and that gives the servers enough of a that that
creates enough of a range that it's highly likely for one server to time out first actually less so in this visualization than in a real system um so nor normal case elections would proceed in just one term uh the the ones that don't will typically just res in one split vote and then will be resolved quickly um that's it for elections yep uh so we developed it for a landan uh the same Concepts apply to a when but you just might need to scale the timeouts maybe can um at a first approximation you're right let
me let me hold off on comparing with pxo and and zookeeper at to the end of the T I I like I said I'll come back to leader election and change it slightly later okay so going on to log replication um so you can see this thing on the right that's that's the logs in the cluster um but a a five server cluster but I I've actually killed off three of the servers so that they're not in the way so just focusing on servers one and two right now uh server one actually uh became leader
I guess sometime before I killed off server 3 and um and and a client came up to Ser or three clients came up to server one and they each asked uh the leader to uh please here here's my command please apply it to the state machine so the leaders ended up with three entries in its log um this is a leader of term two and when it receives ENT uh command from a client it'll create an entry and write its own term number in that entry so that's why uh these entries are all blue with
term two in them so let's go through through log replication um I'll allow server one to just replicate that first entry now now server two has the same entry let me explain the dot and the arrow so the arrow is called the next Index this is the position in the log where the leader will send the next entry to that server uh and to make progress you want that to be you know the point where they diverge and then uh the dot is the match index and that means that the leader knows that it and
the follower agree all the way up through the dot so here they agree up through that first entry and the leader knows to send uh the second entry next that's actually what what the message is about to do but you can see that proceed um the visualization goes one entry at a time but you can you could tweak it to do batching pretty easily uh okay so so now the leader and server two logs are uh are identical you notice these entries still have dashed lines around them that means they're not committed yet they're actually
they're not yet uh necessarily durable so they haven't been those commands haven't been acknowledged to clients yet um you can imagine uh to give you an idea of the kind of thinking you need to do you can imagine if server one and server two were to go down servers 3 four and five came up um well 3 four and five don't even have a copy of that entry but they could elect a leader amongst themselves and those entries might be lost forever so that's why they're not safe yet to to make them safe we actually
have to replicate them to one more server so I've brought up server three and now the leader has replicated that first entry to server three and that match the match index has gone up um so at this point the leader has marked the first entry committed in its own log and a leader gets to mark an entry committed that it created once that entry has reached a majority of the cluster so here uh server one knows the entry is committed the other servers actually don't know it yet um you know's the the same entry still
dashed on their server on their in their logs uh and and so the leader actually part of this replication RPC is it's called append entries uh part of the append entries RPC is to include the commit index so this message down to server 3 is going to inform server 3 of the entry's commitment and the next message to server two will'll do the same you let this advance for a while you'll see we get uh replication three ways out and eventually everyone learns that every entry is committed that's great that's kind of uh normal case
log replication and now let's get into the somewhat harder cases so there's two inconsistencies that can arise because leaders go up and down and uh switch over so so you can have missing entries and you can have extraneous entries and I'll go through both of those um same cluster but a FB Ed quite a bit so let's see Server one uh I it it used to be our leader um had a c a couple more clients come up to it and say hey please replicate these entries before it got a chance to do that though
it failed um and then fiver 4 came back up and fiver two of our new lead R um so now server one's down for a while and it's got those extraneous entries that we'll get to later but or missing entries um uh server 2 is now our leader so uh you know what's what's the difference between the leader log and the leader log it's kind of trivial uh but but actually while while we're here um so you can use the concept of the match index on the leader to record how far the dis has gotten
in persisting entries it's actually a a pretty useful optimization um okay so so server two our leader has initialized all the next indices to be just past the end of its log um you'll see that works really well for Server 3 and not so well for Server 4 that missing three of these entries on this on this first heartbeat um you see Server three replies positively server four replies negatively what's that about well aend entries includes a simple consistency check so if next index is pointing at index 4 then the append entries will say hey
do you at index three have an entry with term two so the leader kind of looking to see if the prior index matches in order to build uh an induction so um f 3 says why yes yes I do have an index with term two there whereas server 4 says no I actually I don't um so for Server three we're going to see this match index Advance all the way up uh and for Server 4 instead you can see the next increment next index decrements down finally they they agree on the empty log and they
build their way back up from there okay so fast forwarded a little more now uh where I had uh submitted a request to server two and and that request actually got committed on server 2 three and four Mark committed everywhere uh and now I brought back fer one but you kind of see the trouble here we need to preserve this pink entry because it's committed we've told clients that it's going to stick around forever and yet server one has these extraneous entries that uh never got acknowledged aren't really important but they're in the way so
what happens is uh when fer 2 and of pen entries messages over to server one well they include that same consistency check and server one's just going to overwrite the entries that are in the way so it kind of has a built-in assertion that uh this index 3 is going to match and given that given that also given that server two is the leader of a higher term so it has more recent information fre one's just going to truncate the end of its log and overwrite all that information so now we've uh resolve that those
extraneous entries yeah so the uh the match index is just initialized to zero but it it jumps forward quite uh you know pretty pretty quickly um so for for a brand new leader until you figure out where the logs diverge you won't be able to commit any new entries but that's a it's a pretty brief period of time uh well consider like server five in this case doesn't actually have uh those committed entries yet I I like it wouldn't I I I guess it wouldn't it wouldn't affect the operation of the algorithm in any way
yeah so right so the uh I guess I didn't say the each server State machine applies as far as the last committed entry and no further so server one would not have applied uh the dashed entries and and all of these servers would have would now apply all of the entries um no so it's only the leader that decides when entries are committed they they're just they don't until until an entry is committed it's only in the log and it's persisted to dis but it's not it won't affect the state machine and it won't be
externalized in any way we can we can come back to that later okay so so we've gone through log replication but the the normal case where uh you know kind of in this situation where everyone's next index is pretty well aligned takes uh one round one round trip to half the cluster for the leader to commit an entry and now uh I'll move on to the third section on safety so let me start by motivating the problem same cluster but I've killed server 2 and brought up server 5 again but wouldn't it be bad if
server 5 happened to time out first and try to become leader um the problem is server 5 has a completely empty log and if it became leader uh it doesn't know that they're F the committed entries out there it might just replace them with entries of its own and then uh all consistency would be lost what happens is when it uh becomes candidate and it requests votes those request vote messages actually include information about the candidate's log and a server won't vote for a candidate whose log is less up to date than its own so
in this case uh server 5 isn't a ble to get any votes at all except for its own so it can't get uh the majority needed to become leader instead what happens is again we wait another timeout server three uh will timeout for term five and become leader in term five and its log is eligible so it can collect votes as normal but what what exactly is that rule that information about uh the candidates log well it's the length of the log but more importantly it's the term of the last log entry so the terms
the uh so here here term four is the more significant bit where uh this top server would not vote for the middle server because it's the term of its last entry is larger and the middle server wouldn't vote for the bottom server since uh the middle serers log is strictly longer when you take take that rule in aggregate uh what it gets you is that if you don't have all the committed entries then you can't become leader so any any server that goes through the candidate phase and is able to collect a majority of the
votes already has all the entries we care about in its log that's what makes rap safe let me flip back to my [Music] SLS so to review things we covered there in leader election uh the cluster uses heartbeats and timeouts to detect crashes and these timeouts are actually randomized so that we can avoid split votes the majority voting guarantees that we've got at most one liter per term uh it's kind of a key property that you need maintain consistency in log replication you saw how the log how the leader can take commands from clients and
append them to its log and then replicate its log to other servers those other servers will just overwrite any inconsistencies there's this built-in consistency check with the term of the previous entry which is used to make sure that logs don't diverge in funny ways we can't can't possibly create a mish mash in ra um if two logs have the same entry in one position then the entire prefix before that will also match and then in the safety section we tied that together so that only Leaders with all committed entries in the logs but we can
only elect a leader if it has all committed entries in its log already and then when it does overwrite inconsistencies that that's actually say uh one thing I touched on but didn't go through the example was that um I said a leader that uh that creates an entry can mark that entry committed once it's replicated to a majority of the cluster new leaders and raft actually uh if they inherit entries from prior terms they won't commit those right away until they're able to push one of their own entries out you can see the paper for
the sample on that kind of take some time to walk through yeah no the the absolute last one in their log even if it's uncommitted absolutely um and and there's always there's sort of always that potential in a distributed system that you actually do some work and then the acknowledgement gets lost uh except in raft it can guess happen in the other direction or wa no no no I mean I mean what what can happen if uh the cluster moves on and yet later actually does the work but it's kind of too late to acknowledge
the quient uh you you cannot roll back a committed entry uhhuh right for for now um there there is a there are some rare cases where that one could still be rolled back uh I think so so just conservatively if something's if something's committed then we will absolutely keep it for all time if it's not committed don't know uh the way you decide is is there any possibility that a server could become leader without having that entry oh next SL I think oh uh it's a fun Benchmark I'm showing here a question was you know
how often are these split votes going to happen how much randomization do you need to avoid them so I tried to create the absolute worst case situation uh on a on a gigabit Ethernet landan with uh five servers just try to synchronize those servers with using uh heartbeats and then have them time out at instead of choosing randomly the fixed 150 millisecond timeout and if we do that then basically every election takes longer than 10 seconds um so just you know going through what is about like 60 split votes at a minimum per election if
you give the same cluster just a little bit of Randomness um that's really all it takes so so we'd recommend something conserv conservative L like choose the timeouts randomly between 150 and 300 milliseconds and then basically uh all of your elections will complete within a couple hundred milliseconds and uh the fraction of Elections that have split votes is down here in the in the 4% or so of those 4% uh 96% will complete in the next term Etc so it converges quite quickly do that answer your question right so it's it's not it's not the
one with the highest term it's uh it's one that is high highest compar that's backward it's a it's the majority of uh most suitable logs that could become elected leader so so of any five server cluster at least three of them will always be eligible is right right so yeah so under asymmetric Network partitions I won't guarantee availability um and if if you do want to operate in those cases uh there's you know there there's something you can do to the protocol to make it a bit more robust see my thesis for that but it's
really uh the hope is that you have bidirectional communication and if you don't can't really make guarantees oh going forward uh it's hard to evaluate kind of how easy raft is to implement but one piece of evidence I do have is that um there's dozens of implementations in progress some of these are at companies some of these are people in their spare time uh you know not not all of these are production ready by any means but the paper was just published in June for the first time so it's uh and these guys got a
head start but it's still pretty early um and some you can see these are in a wide variety of languages um that's I think that's because the people doing it in their spare time pick their new favorite language to to use uh some of these like go raft are uh are libraries some of them like log cabin uh which is my implementation in C++ are kind of full-blown Network Services um and another one I wanted to to touch on is uh Facebook's Hydra base I think they're the first example of people kind of uh using
raft for replicating entire database State and they're they're doing so in a cross Data Center Way um the image I stole from them seems to be hard to read on this projector but the idea is they would take uh here here you've got three different data centers um and they want to create a consistent view of hdfs and so they've got they're actually showing five rra servers um five and some of them are and their their state machine in this example is hdfs so some of their R servers actually don't have state machines I think
that's because uh probably running a full-blown hdfs node is expensive but they do they do participate in uh the the log replic application so that they can be uh relied on in case of a crash so these two data centers each uh they each have two servers in them uh but only one state machine and then this data center is just purely for replication purposes um actually I I don't know much about uh any other deviations they they've made from raft but I thought I'd throw this out there in case it comes up in the
Q&A so to wrap up the talk um consensus is needed for building all these consistent storage systems but until now at least it was widely regarded as difficult um know paxel really hard to learn really hard to implement so we designed raft specifically for understandability the user study showed that it's easier to teach in classrooms we think it's a better foundation for building practical systems because it's it's a lot closer to uh what it you know the replicated State machine replicated log approach which is what you actually want in a system as a better problem
decomposition using a a strong leader and a lot less mechanism you know those those four messages the request vote request and response and the append entries request response get you through the entirety of the core algorithm uh and cluster membership changes there there are quite a bit uh more there are several more topics that you need for a a full-blown system so cluster membership changes uh would be you know say I start with a three fer cluster how do I and I change my mind up to a five server cluster or how do I replace
servers if they fail you actually need to do that through the consensus algorithm because you're changing the definition of majority uh so to do that safely you want to automate that uh log compaction what what do we do when the what do we do when the raft lug grows too big uh can't can't keep just a pending commands to it for all the time so that uh the tech report goes through kind of a snapshotting approach and that's generalized in my my thesis um so you can do things like level DB or SS tables uh
client interaction it's actually often overlooked but it's uh source of bugs is how do clients interact with the system get the consistency guarantees that we actually want and especially how do you implement readon operations with the same level of consistency and then of course there's more evaluation that I didn't cover here um all right I'll take questions and there uh there's a mic where I can try to repeat your [Music] questions can you compare it at the high level how this is different than you know like virtual synony extanded virtual synony because I see a
lot of difference in there I cannot no I'm not really familiar with the virtual synchrony work I think there's uh there's some claim that there's equivalence to to consensus there but I know yeah because from there um what I remember was that uh you know for each of the election they bump up the terms and so that you know all the so the idea of a term number I can say is uh it's just fundamental to the problem so paxos calls it a proposal number zookeeper calls it an EPA viap replication calls it a view
but this this idea that um we're not relying on actual like wall time here to decide who's stale and who's current so you need to uh somehow propagate that around using the logical clock and and uh in all of these algorithms once you adopt a higher number you're going to start either ignoring or rejecting messages from with a lower number than I may be misremembering but I thought when I first saw this work a year ago whenever there were some blog posts did you guys use TLA plus to verify some of this or am I
misremember yeah um so I I have a formal spec written in TLA plus I also tried to use some of the TLA proof system to automatically prove it and then backed off and did more of a englishy proof um is englishy a technical term I'm kidding what I um so what what prompted the back off then like I saw the list of other stuff in your thesis and tech report like membership change compaction was it the core algorithm you backed off or was it when you started to add all these features you need to be
real that you backed off no it's even in in just the core algorithm so um it it I got up to like 4,000 lines of TLA proof and uh it was unmaintainable and part of that might be my failing like have no training and formal methods um but I I think part of like I I don't know that that's the best tool for the job um one one example is like there's no type system in TLA so kind of every step of the way you need to say yeah I added I added one to that
so it's still an integer and only then can you actually get the tools to to agree with you um I I discussed that in the correctness chapter in my thesis do you want to read more so two questions one is u i see similarities and I don't really see any specific differences with Zab uh is that something that you can touch um anything that is outstanding I know that it's more or less similar at a very high level but I wanton like is there something really specific that differentiates from Zab sure I think um you
at a high level kind of you snap replication Zab and raft are pretty close uh at kind of a low level because we did push on understandability so hard uh you see that like ref has four message types where Zab has 10 uh so it's just kind of uh if you're pushing on understandability you need to motivate every piece of the algorithm and if there's something you can't motivate that's probably because it shouldn't be there um I I don't want to you know zookeeper is really widely used it seems to work I uh it's appears
to be correct and everything I don't want to say anything too bad about them but I think uh it's one one difference is like the academic approach and the industry approach where uh we were trying to find an algorithm that optimizes understandability they were trying to find an algorithm that works so the other thing is you did mention about the clients uh so is there anything special that is being done here in terms of uh how to guarantee because one of the real problems that we found uh with zuker is it's very hard to in
some cases to make sure that a right was successful right [Music] um so I I mean a lot of that stuff fall on uh implementations and how how they expose their apis and stuff uh one one thing that I think is particularly hard about zookeeper is that if you're disconnected from the server you happen to be talking to then uh kind of all information is lost so at least in my implementation I I tried uh like sessions are a a concept that's shared across servers and so a client can can reconnect that way um this
is this is stuff zookeeper could do but for the clients as well how to implement a client or is it mainly no no no not even not a tlaa spec or anything I have a chapter describing it uh I see my thesis I have code yeah yeah yeah yeah yeah go follow that um yeah so the there's dozens of implementations of raft in progress uh and and dozens that are permissively licensed uh that's excluding Hydra base but everything else I talked about question thank you so we'll be having a lunch so if you're interested in
further questioning questions about raft you can join