Does a two-site Kubernetes cluster need three etcd voters?

Three control-plane VMs feel like three chances to survive a bad night. They are not, and the distance between those two beliefs is what turns a tidy three-node design into a cluster that stops accepting writes when one building loses power. A VPN link between the sites makes the machines reachable from each other and hides the problem rather than fixing it. Votes are counted per etcd member, and two of those members can still sit behind one switch, one UPS and one uplink.

Three control-plane nodes, one failure domain

You run kubectl get nodes, see three control-plane entries, and file the cluster as highly available. That count describes schedulable capacity, not survivability. In kubeadm kept without an external etcd, each control-plane node also hosts a member, so three nodes really do give you three voters. Whether those voters fail independently is a separate question that the node list cannot answer.

Two VMs on one hypervisor, two racks on one power feed, two uplinks from one ISP: each pair is a single domain wearing two hostnames. Write the placement down and mark what every voter shares — hypervisor, PDU, top-of-rack switch, storage array, NAT device, VPN concentrator. Then remove each shared object on paper and recount the votes that remain. A cluster that loses two voters to one failed host has been a single-voter cluster all along; nobody had asked it the question yet.

What a two-site split can and cannot survive

With three voters placed two and one across two sites, the larger side keeps quorum when the other site vanishes and the single-voter side never does. That asymmetry is arithmetic, not tuning. Any majority-based etcd cluster spread over exactly two independent sites fails the test "lose either site and keep committing", because both sides would have to hold more than half the votes at the same time.

The table below is the quickest way to see which failures you actually bought. Read each row as: if this whole row disappears tonight, can the survivors still commit a write?

Why the tunnel never adds a vote

A VPN is transport, not a member. It can make a remote voter reachable and it can turn a partition into something that looks like a slow timeout, and that is the whole of its contribution to quorum. What it does add is cost, because every commit has to cross the link and etcd pays that round trip on writes that touch the raft log. A path that flaps converts replication into retries and elections.

Partitions are where people expect magic, so be precise about them. If the link drops while the old leader sits on the one-voter side, that side cannot commit and the leader steps down once the election timeout expires, while the two-voter side elects a replacement. The etcd failure-mode guide describes the behaviour plainly: continued service on the majority side, no second writable cluster on the isolated side.

Decide which outage you are buying

Two plus one is a legitimate design when Site A is the primary and you accept that losing it means losing control-plane writes until someone acts. What turns it into a defect is leaving that decision implicit. Put it in the runbook: which site your control plane survives, and what an operator does when the other site goes dark at 03:00.

If the requirement is instead "either site may fail", the placement must hold a majority in each site, and three voters over two sites cannot do that. You either add a genuinely independent third domain — a small region in another provider works if the path to it is stable — or split into two clusters with separate etcd and accept the operational cost of that split. No witness shortcut avoids this: etcd has no lightweight arbiter member, so a third voter is a full member with its own disk, memory and network demands. Drawing a box labelled witness on the diagram does not create one.

Quorum is not the same as service

etcd quorum keeps the API server able to accept writes, and that is all it keeps. Load balancer health checks, ingress, CNI reachability across the tunnel, volume attachment zones and application replication each need their own answer. A cluster can hold a clean majority and still serve nothing useful to users.

Then confirm the membership you believe you have, because diagrams drift:

kubectl -n kube-system get pods -l component=etcd -o wide
kubectl get endpoints kube-apiserver -n default
etcdctl member list -w table

Compare the member list against the placement drawing. If two members resolve to the same host or the same rack, the drawing is wrong rather than the cluster, and resizing around a mistaken topology only buys a more expensive version of the same outage.

Before signing off on any three-voter design, answer four questions: how many voters lose power together, whose site loss still leaves a majority, where the leader sits when the tunnel drops, and what the surviving site can still do. A "not sure" on any of them means the cluster is not highly available yet. It is three machines that have been lucky.