KoutenDB v0.13 Implementation Roadmap

Theme

v0.13 removes the cluster transaction landing node as a single point of failure without adding consensus traffic to ordinary ring-local reads and writes.

The scope is deliberately narrow:

Replicate committed cluster transaction intent, fence coordinator ownership, and make failover explicit and recoverable.

This is not a Raft implementation and does not make every KoutenDB operation a globally synchronous transaction.

Coordinator Invariants

The implementation must preserve these rules:

  1. A cluster transaction is acknowledged only after its complete intent is durable on the configured coordinator and standby.
  2. Transaction IDs include a monotonic coordinator epoch and a persistent local sequence, so different coordinator generations cannot reuse an identity.
  3. Only the coordinator named by the current epoch may apply a fenced intent.
  4. Promotion requires an explicit maintenance drain and a reachable cluster majority. KoutenDB does not automatically guess through a network partition.
  5. A promoted standby re-replicates every pending intent to its newly configured standby before applying it.
  6. A recovered stale primary cannot make the current standby accept an old-epoch commit.
  7. Client bootstrap and coordinator discovery tolerate an unavailable first endpoint but reject conflicting metadata at the highest visible epoch.

Implemented Scope

  • persisted coordinatorEpoch, coordinatorNode, and coordinatorReplica;
  • epoch-encoded, collision-resistant cluster transaction IDs;
  • synchronous durable intent mirroring before commit acknowledgement;
  • replication markers bound to both coordinator epoch and standby node;
  • additive fenced APPLYTXF, TXMIRROR, and promotion wire operations;
  • explicit, majority-gated standby promotion through the Nim API and CLI;
  • automatic client discovery of the highest non-conflicting coordinator epoch;
  • bootstrap topology discovery through any reachable configured endpoint;
  • coordinator role, assignment, replica-health, mirror success/failure, and pending-intent metrics;
  • WAL replay and compaction preservation of coordinator and mirror state.

Failure Matrix

scripts/coordinator_failover_smoke.sh exercises a three-node persistent cluster with strong durability:

  • refuse a successful commit acknowledgement while the standby is unavailable, for both normal termination and SIGKILL, then retry the same transaction identity after recovery;
  • accept identical mirror and completion replay while rejecting a conflicting payload under the same transaction ID;
  • lose the standby after commit mirroring but before its completion acknowledgement, then verify automatic convergence after restart;
  • stop the owner of a committed transaction;
  • verify the primary and standby both retain the pending intent;
  • restart the standby from its own WAL and verify the intent remains pending;
  • crash the primary coordinator;
  • verify that one surviving node cannot resume a new epoch without quorum;
  • recover a second node, promote the standby, and resume through majority;
  • re-replicate the old pending intent to the new standby;
  • apply the intent to the recovered owner and verify the exact payload;
  • verify ring cardinality remains exactly one after retry and failover;
  • restart the old primary and verify that its stale-epoch commit is rejected;
  • verify that new transaction IDs carry the promoted epoch.

The release gate also rejects undrained promotion, invalid assignments, same-epoch conflicts, epoch rollback, partial staging, quorum without the assigned standby, and orphan cluster-transaction WAL records. The complete test-to-invariant table is maintained in coordinator-failover.md.

Explicit Boundaries

  • Promotion is operator-driven, not automatic.
  • The peer list remains static; this work does not implement service discovery.
  • The coordinator standby protects cluster transaction landing intent. It does not duplicate every ring-local record.
  • Cross-Universe convergence remains the responsibility of Universe sync.
  • Mixed-version rolling upgrade policy remains separate work.

Operational configuration and recovery commands are documented in Coordinator Failover.