KoutenDB v0.12 Implementation And Validation Record

This document records the implementation and validation completed for the KoutenDB v0.12 development cycle, followed by explicitly separate follow-on candidates.

Theme

v0.11 made ring locality part of the persistent disk-read layout. The next cycle should make that layout easier to maintain and observe without weakening its crash-safety or operator control.

The primary direction is:

Make disk-backed locality self-maintaining within explicit limits, then validate the complete path once instead of repeating long endurance runs after every intermediate change.

Delivered In v0.12

1. Bounded Automatic Ring Packing (Implemented)

KoutenDB already reports per-ring stale ratios and provides explicit pack-ring and pack-recommended commands. The next step is an opt-in maintenance scheduler that uses the same diagnostics.

The implemented scheduler supports:

  • stale-ratio and stale-record thresholds;
  • maximum rings, bytes, or elapsed time per maintenance run;
  • an optional maintenance window;
  • dry-run and status output using the same decision logic as execution;
  • durable reporting of success, skip, interruption, and failure reasons;
  • clean restart after process termination;
  • a default-off mode so opening a database never starts unpredictable work.

The scheduler does not turn into an unbounded background compactor. It is off by default, requires positive ring, byte, and elapsed limits, and runs through the server’s existing single-owner maintenance path. maintenance-plan, maintenance-run, and maintenance-status expose the same decision model for manual operation. Atomic status publication and manifest-last segment activation preserve the previous complete generation after interruption.

Coverage includes exact budget boundaries, elapsed and byte interruption, process-restart status recovery, malformed status rejection, fully deleted rings, UTC maintenance windows, server metrics, and offline reopen/verify. The additive C ABI exposes disk-backed open options, segment diagnostics, and the same plan/run/status/recovery path without changing the existing ABI v2 layouts.

2. Generation Checkpoints And Snapshots

Status: implemented for the v0.12 cycle.

A checkpoint should identify one consistent storage generation across the WAL, ring-segment manifest, metadata, and rebuildable sidecar indexes.

The initial scope should provide:

  • an atomic checkpoint identity and WAL high-water mark;
  • verification that every referenced generation is complete and checksummed;
  • restart and restore from a selected complete checkpoint;
  • cleanup rules that never remove the last verified generation;
  • a machine-readable status suitable for backup orchestration.

This complements backup and restore. It does not need to become a full point-in-time recovery service in the first implementation.

The implemented scope publishes koutendb-checkpoint-v1 directories through a staging-directory rename. Each directory contains a compact WAL, complete ring-local segment/index generations, a checksummed inventory, and a completion marker written last. Verification rejects missing or unreferenced files, checksum and size changes, unsafe paths and symlinks, WAL replay failures, segment/index mismatches, and logical-count drift. Restore stages and verifies the whole generation before atomically replacing the destination. Retention never deletes the final verified generation and preserves invalid generations for diagnosis.

The implementation is available through Nim APIs, CLI commands, and additive C ABI v2 JSON functions. It is a selected-generation recovery boundary, not continuous point-in-time recovery or managed backup scheduling.

3. Prometheus-Compatible Operational Metrics

Status: implemented for the v0.12 cycle.

Existing machine-readable counters should be available in Prometheus text format through a stable operational surface.

Initial metrics should cover:

  • segment hits and WAL fallbacks;
  • segment/index bytes and active generation counts;
  • stale records, stale ratios, and recommended rings;
  • pack attempts, outcomes, durations, and bytes rewritten;
  • checkpoint age and verification state;
  • WAL growth, item/ring counts, and capacity-limit failures;
  • request latency and error counters already exposed by cluster metrics.

Metric names and labels must avoid unbounded ring-name cardinality by default.

The implemented surface preserves legacy key/value lines and adds Prometheus and OpenMetrics output through the Nim API, CLI, and additive C ABI v2 calls. It covers node/request/error, WAL and segment-layout, automatic maintenance, capacity guardrail, and aggregate checkpoint health metrics. Ring names, checkpoint IDs, and arbitrary error text are excluded from labels. Metrics collection uses a ring-count-bounded segment snapshot rather than scanning all records.

4. Explainable Maintenance Decisions

Status: implemented for the v0.12 cycle.

Operators should be able to answer why a ring was packed, skipped, or read from the WAL. Diagnostics should expose stable reason codes rather than requiring log parsing.

This includes:

  • pack recommendation reasons;
  • WAL fallback reasons;
  • inactive-generation cleanup results;
  • capacity guardrail failures;
  • checkpoint eligibility and rejection reasons.

Maintenance JSON now carries additive reasonCode, stopReasonCode, and inactive-generation cleanup codes while preserving existing text fields. WAL fallback counters are separated into point-read, ring-scan, and bounded-window reasons. Checkpoint status adds a bounded reasonCode alongside the detailed human-readable reason, and guardrail metrics use four fixed rejection labels.

Follow-On Candidates

These are important, but they should not make the v0.12 persistence scope unbounded.

Cluster Transaction Coordinator Redundancy

Remove the current node0 landing single point of failure with fenced ownership and recoverable handoff. KoutenDB should not add consensus round trips to every ordinary write merely to imitate a different database architecture.

Incremental Import And Change Feeds

Allow an existing PostgreSQL, MySQL, or application event stream to place only changed records into KoutenDB. This supports adoption with an existing system of record while KoutenDB serves locality-oriented retrieval.

Multi-Context Locality

Explore a public model for reading the same stored data through multiple context axes, such as time, version, source, or application view, without duplicating the payload. Detailed graph/lens semantics remain a separate design track and are not required for v0.12.

Validation Roadmap

1. Unit And Invariant Matrix

Continue extending deterministic tests around:

  • writes, updates, deletes, backfills, pack, and reopen;
  • identical logical result sets before and after maintenance;
  • segment corruption, wrong index offsets, and whole-ring WAL fallback;
  • exact threshold boundaries and invalid maintenance budgets;
  • checkpoint creation, rejection, cleanup, and restore;
  • failure isolation between unrelated rings.

2. Accelerated Churn Test

Status: runner implemented; 120,000-operation validation completed.

Before another 72-hour run, execute a one-to-three-hour high-density workload that maximizes state transitions rather than wall-clock time.

It should repeatedly perform:

  • multi-ring writes, updates, deletes, and backfills;
  • manual and scheduled pack cycles;
  • reopen and verification cycles;
  • checkpoint creation and cleanup;
  • backup and restore comparisons.

The implemented manual runner records p50/p95/p99 latency, RSS, WAL bytes, segment/index bytes, recommended-ring counts, generations, fallback counts, and compact logical-result digests. Correctness is checked by exact sorted result-set equality against an independent in-memory model, including after maintenance, reopen, checkpoint, backup, and restore operations. A finite series of bounded maintenance passes must also drain the final backlog.

The operation-bounded runner passed 120,000 churn operations on 2026-08-07, including 1,202 exact logical-state comparisons, 482 bounded maintenance runs, 120 verified checkpoints, 60 close/reopen cycles, and 30 backup/restore comparisons. It finished with 8,054 live records across 17 rings, no duplicate or missing logical records, zero segment-to-WAL fallbacks, a drained maintenance backlog, and a successful final offline verification. The full environment, command, and measured latency telemetry are recorded in accelerated-churn-testing.md.

This operation-bounded evidence maximizes state transitions. It does not replace the process-crash, concurrency, storage-failure, container, or final 72-hour matrices below.

3. Process-Crash Matrix

Status: implemented for the v0.12 cycle.

Process-level termination coverage now extends beyond the writer loop. The matrix sends an actual SIGKILL at pack, checkpoint, and backup boundaries, including:

  • during segment output;
  • after segment replacement but before index replacement;
  • after both generation files are durable but before manifest activation;
  • immediately after manifest activation;
  • during inactive-generation cleanup;
  • during backup/checkpoint publication.

After every crash, exactly one complete generation must remain usable and the logical result must match WAL recovery.

The matrix covers nine exact publication points: segment output, data-file publication, index publication, manifest activation, inactive-generation cleanup, checkpoint publication before and after rename, and backup publication before and after replacement. It verifies unrelated-ring isolation, old-or-new generation visibility, checkpoint listing isolation, backup restore contents, retry after restart, temporary-file cleanup, and zero segment-to-WAL fallbacks after a recovery pack. The hooks exist only in the dedicated -d:koutenTestCrashPoints build and are absent from release builds.

This is process-termination evidence. It does not claim power-loss or storage device durability; those failure modes remain in the storage-failure matrix.

4. Concurrency And Backpressure

Status: implemented for the v0.12 cycle.

The process matrix runs multiple readers and writers while automatic maintenance, metrics reads, and snapshot barriers are active. It validates:

  • no deadlocks or duplicated committed mutations;
  • bounded queues and explicit overload errors;
  • stable lock/fencing behavior;
  • no partial results during segment fallback;
  • bounded memory growth under slow clients.

The CI-sized matrix uses four writer processes, three reader processes, and one metrics/snapshot observer against a strong-durability disk-backed node. Writers put and update the same ring with applied acknowledgements. Final online and offline checks require exact payload and ID sets, monotonic reader counts, duplicate-free pagination, successful segment verification, and automatic maintenance activity.

Connection admission, partial request bodies, oversized ring-list pages, and a client that stops reading a multi-megabyte response are tested separately. Accepted sockets now have receive and send deadlines, ring-list pages have a fixed maximum, and rejected admission increments connectionsRejected. Plaintext clients receive ERR overloaded; TLS connections are closed before the handshake when the hard transport limit is already full. The test also found and fixed cluster Nim API writes that did not persist the ring name, which previously made name-based embedded reads disappear after reopening the same data directory even though the records remained intact.

5. Storage-Failure Injection

Status: implemented for the v0.12 cycle.

scripts/storage_failure_matrix.sh covers deterministic disk-full and short WAL writes, real segment-directory permission loss, a missing active segment, a damaged generation manifest, and index-only offset corruption. Every case uses an independent strong-durability disk-backed store and verifies exact logical state after reopen.

All direct Store WAL mutations now pass through one error boundary. A write or flush failure poisons that Store handle, rejects later mutations, and does not publish the failed record or metadata change into in-memory state. A torn final record is repaired to the last checksummed WAL boundary on reopen. Derived segment/index/manifest damage is rebuilt from the authoritative WAL; a valid-looking wrong index offset falls back only the affected record and increments the bounded fallback reason counter. The disk-full and short-write cases are deterministic test injection, not a claim that every filesystem and device failure mode has been reproduced.

6. Upgrade And Migration Compatibility

Status: implemented for the v0.12 cycle.

tests/fixtures/ now keeps immutable stores produced by the tagged v0.10.1 and v0.11.0 release code. The former contains the legacy generation-zero segment layout; the latter contains manifest-selected generation-one segments and sidecar indexes. scripts/upgrade_fixture_matrix.sh copies each fixture before opening it and verifies:

  • open and WAL replay on the new version;
  • legacy generation-zero segment reads;
  • explicit pack into the current format;
  • JSONL dump/import as the stable pre-v1 migration boundary;
  • generation checkpoint verification and restore;
  • offline verification after upgrade, restore, and JSONL import.

The matrix compares exact release-generated dumps for direct upgrades and a portable logical projection for JSONL imports, where IDs are intentionally reissued. It preserves raw, JSON, NIF, and BIF payloads plus vectors, metadata, updates, deletes, and transaction results.

7. Container And Security Validation

Status: implemented and locally validated for the v0.12 cycle.

scripts/container_security_matrix.sh validates strong disk-backed named-volume persistence across restart and container replacement, network interruption and recovery, TLS, ID/password/secret-key authentication, ring authorization, credential rotation, offline verification, and persistent audit evidence. Plaintext, foreign-CA, hostname-mismatch, expired-certificate, invalid credential, missing-secret, and denied-ring paths fail closed.

This remains a manual Docker gate rather than a default CI job because it builds an image and creates isolated containers, networks, and volumes. Its cleanup trap removes all test resources. See container-security-validation.md for the matrix and local result.

8. External Driver Compatibility

Status: implemented and clean across all five external drivers.

scripts/external_driver_matrix.sh runs the separately maintained Rust, JavaScript/TypeScript, PHP, C++, and Python driver suites and applies the same strong disk-backed TLS, authentication, restart, and failure matrix to all five. Driver release work remains in each external repository, while the core provides the compatibility target.

The 2026-08-13 clean run passed the Rust, JavaScript/TypeScript, PHP, C++, and Python suites plus verified TLS/auth CRUD, rejection cases, persistent restart, offline segment verification, and audit-evidence checks. Rust follows eager connection failure semantics, and Python follows owner-bearing FWD responses and routed multi-node BGET. See external-driver-validation.md.

9. Final 72-Hour Endurance Gate

Status: completed successfully for the v0.12 release.

The final 72-hour run exercised the feature-frozen v0.12 storage and maintenance path. The report captured:

  • p50/p95/p99 point, ring, stellar, and retrieval latency;
  • memory and disk growth;
  • WAL fallback and pack outcome counters;
  • stale ratios and generation counts;
  • cluster queues and errors;
  • final offline verification and restore equivalence.

The harness ran three strong-durability disk-backed nodes, bounded automatic packing, update/delete/backfill churn, point/ring/stellar/retrieval reads, rolling latency percentiles, RSS/data-size sampling, and a queue convergence barrier. After shutdown it performs segment-aware verification, creates and verifies one generation checkpoint per node, restores each into a fresh directory, verifies the restored stores, and compares exact sorted JSONL dumps.

The 2026-08-10 through 2026-08-13 run completed 4,213,187 mixed operations with zero client errors. All queues converged to zero, all source and restored stores passed offline verification, and every restored sorted JSONL dump was byte-for-byte equal to its source. Full counters and conditions are recorded in soak-testing.md. The 72-hour test remains a manual release gate and does not run in normal CI.

  1. Implement bounded automatic packing.
  2. Implement generation checkpoints and maintenance diagnostics.
  3. Add Prometheus-compatible metrics.
  4. Run the accelerated churn, crash, concurrency, storage-failure, and upgrade matrices.
  5. Run container, TLS/auth, and external-driver compatibility tests.
  6. Freeze the release candidate and run one final 72-hour endurance test.

Exit Criteria

The v0.12 release met these exit criteria:

  • maintenance work is opt-in, bounded, observable, and restart-safe;
  • pack/checkpoint interruption cannot replace a complete generation with an incomplete one;
  • complete logical result sets remain identical across maintenance and recovery;
  • capacity growth and fallback behavior are explainable through metrics;
  • upgrade fixtures and backup/restore verification pass;
  • the accelerated and final endurance reports show no unbounded growth or unexplained latency degradation.

Non-Goals

The v0.12 persistence cycle should not require:

  • unbounded automatic compaction;
  • consensus on every ordinary write;
  • a general-purpose graph database inside KoutenDB;
  • a 72-hour CI job;
  • claiming multi-region certification from a local endurance run.