Distributed systems · explained simply

How Raft servers agree

The leader of your five-server store drops off the network. Within about one election timeout, another server takes over, and your next SET x=5 is safe once three of the five have written it down.

heartbeats stop→ election timeout→ RequestVote→ AppendEntries→ 3 of 5→ apply→ step down
s1 leader heartbeat election timeout s3 · 150–300 ms
1

The leader goes quiet, and a follower's timer runs out

Your servers act like a committee. The chair keeps sending every member a short “still here” memo. When s1 drops off the network, the memos stop. Each member has its own timer, and s3's runs out first.

The memo is a heartbeat: an AppendEntries call with no entries. A follower that hears none within its election timeout becomes a candidate and asks the others for votes. Each server picks its timeout at random, for example from 150–300 ms, so usually only one fires.

committee = cluster chair = leader member = follower “still here” memo = heartbeat timer = election timeout
s3 candidate term 4 RequestVote s3 s2 s4 s5 s1 3 of 5 = leader
2

The candidate asks for votes, and three of five make it leader

Each election gets a new term number, here term 4. A member gives one vote per term, to the first candidate who asks. Every member also keeps a copy of the committee's notebook. It says no if the candidate's copy is behind its own.

The candidate adds 1 to currentTerm, votes for itself and sends RequestVote. A log is ahead if its last entry has a later term, or the same term and a higher index. With no majority, the timers run out again, and a new term starts.

The voter seesVote
a term older than its ownno
votedFor names someone else this termno
a log behind its own, by lastLogTerm, then lastLogIndexno
none of theseyes
notebook = log vote = voteGranted

Any two majorities of five share a server, so one term never has two leaders.

s3 · leader 5 t2 6 t3 7 t4 SET x=5 s4 5 t2 6 t3 prev entry 6 · t3 AppendEntries
3

The leader writes your command and sends it to every follower

You send SET x=5 to s3. The chair writes it as line 7 and sends a memo. The memo also says what line 6 should look like. A member whose line 6 matches copies line 7. Otherwise it says no, and the chair resends from an earlier line.

A follower refuses new entries unless its log has prevLogTerm at prevLogIndex. After a no, the leader lowers its nextIndex for that follower and retries. So two logs that agree on one entry agree on everything before it.

AppendEntries {               // example values
  term: 4, leaderId: "s3",
  prevLogIndex: 6, prevLogTerm: 3,
  entries: [{ term: 4, command: "SET x=5" }],
  leaderCommit: 6
}
line = log entry memo = AppendEntries
5 6 7 s1 t2 t3 t3 s2 t2 t3 t4 s3 t2 t3 t4 s4 t2 t3 t4 s5 t2 t3 commitIndex = 7 apply x = 5 state machine then replies OK
4

A line is final once three of five have it

Now s2, s3 and s4 hold line 7. Any later chair needs a vote from one of them, and they refuse a candidate whose notebook is behind. So line 7 can never be lost. The chair carries it out, sets x to 5 and answers you OK.

The leader tracks a matchIndex per follower. It moves commitIndex to 7, the highest entry from its own term that a majority holds. Later memos carry leaderCommit, and followers apply up to it, in log order.

final = committed carry out = apply to state machine

A write is done when three of five have it, not all five.

term 4 from s3 s1 · follower 5 t2 6 t3 7 t3 SET y=1 7 t4 SET x=5
5

The old leader comes back, sees a higher term and steps down

s1's network returns, and it still thinks it chairs term 3. While cut off, it wrote SET y=1 as its own line 7. Nobody else has that line, and nobody got an OK for it.

A server that sees a higher term in any request or reply takes that term and becomes a follower. s3's next memo carries line 7 from term 4. s1's line 7 has the same index but a different term, so s1 deletes it and copies s3's.

Before

  • s1 thinks it leads term 3.
  • Its line 7 is SET y=1, on 1 of 5.

After

  • s1 follows s3 in term 4.
  • Its line 7 is SET x=5, like s3's.

Before you rely on it