How Raft servers agree
The leader of your five-server store drops off the network. Within about one election timeout, another server takes over, and your next SET x=5 is safe once three of the five have written it down.
The leader goes quiet, and a follower's timer runs out
Your servers act like a committee. The chair keeps sending every member a short “still here” memo. When s1 drops off the network, the memos stop. Each member has its own timer, and s3's runs out first.
The memo is a heartbeat: an AppendEntries call with no entries. A follower that hears none within its election timeout becomes a candidate and asks the others for votes. Each server picks its timeout at random, for example from 150–300 ms, so usually only one fires.
The candidate asks for votes, and three of five make it leader
Each election gets a new term number, here term 4. A member gives one vote per term, to the first candidate who asks. Every member also keeps a copy of the committee's notebook. It says no if the candidate's copy is behind its own.
The candidate adds 1 to currentTerm, votes for itself and sends RequestVote. A log is ahead if its last entry has a later term, or the same term and a higher index. With no majority, the timers run out again, and a new term starts.
| The voter sees | Vote |
|---|---|
a term older than its own | no |
votedFor names someone else this term | no |
a log behind its own, by lastLogTerm, then lastLogIndex | no |
| none of these | yes |
Any two majorities of five share a server, so one term never has two leaders.
The leader writes your command and sends it to every follower
You send SET x=5 to s3. The chair writes it as line 7 and sends a memo. The memo also says what line 6 should look like. A member whose line 6 matches copies line 7. Otherwise it says no, and the chair resends from an earlier line.
A follower refuses new entries unless its log has prevLogTerm at prevLogIndex. After a no, the leader lowers its nextIndex for that follower and retries. So two logs that agree on one entry agree on everything before it.
AppendEntries { // example values
term: 4, leaderId: "s3",
prevLogIndex: 6, prevLogTerm: 3,
entries: [{ term: 4, command: "SET x=5" }],
leaderCommit: 6
}
A line is final once three of five have it
Now s2, s3 and s4 hold line 7. Any later chair needs a vote from one of them, and they refuse a candidate whose notebook is behind. So line 7 can never be lost. The chair carries it out, sets x to 5 and answers you OK.
The leader tracks a matchIndex per follower. It moves commitIndex to 7, the highest entry from its own term that a majority holds. Later memos carry leaderCommit, and followers apply up to it, in log order.
A write is done when three of five have it, not all five.
The old leader comes back, sees a higher term and steps down
s1's network returns, and it still thinks it chairs term 3. While cut off, it wrote SET y=1 as its own line 7. Nobody else has that line, and nobody got an OK for it.
A server that sees a higher term in any request or reply takes that term and becomes a follower. s3's next memo carries line 7 from term 4. s1's line 7 has the same index but a different term, so s1 deletes it and copies s3's.
Before
- s1 thinks it leads term 3.
- Its line 7 is
SET y=1, on 1 of 5.
After
- s1 follows s3 in term 4.
- Its line 7 is
SET x=5, like s3's.
Before you rely on it
- catchA cut-off leader still thinks it leads, as s1 did. If it answers reads from its own copy, a read can be stale. Before it answers a read, a Raft leader must hear back from a majority in one heartbeat round.
- ask nextWhy 3 or 5 servers, not 4? Four need three for a majority, so they survive one failure, like three. Five survive two.
- ask nextCan a new leader commit an old leader's line by counting copies? No. It counts copies only for lines from its own term. Older lines become final along with a later line from its term (the paper's Figure 8).
- ask nextWhat if the leader commits but crashes before it answers? The client retries with the new leader, so the command can run twice. Clients tag each command with a serial number, and the state machine skips repeats.
- simplifiedLeft out: membership changes, snapshots, disk writes and how clients find the leader. Server names, terms and line numbers are examples.
- sourcesOngaro and Ousterhout, In Search of an Understandable Consensus Algorithm (Extended Version), Figure 2 and sections 5, 8 and 9.3; raft.github.io.