KoutenDB v0.10 Roadmap

This roadmap describes the intended v0.10 direction. It is not a release promise. Items can move between releases if implementation, testing, or user feedback changes the priority.

KoutenDB v0.9 proved two important points:

  • large generated validation can run at multi-million record scale;
  • locality-aware reads can express related-data bundles that are awkward for conventional key/value or relational query shapes.

The v0.10 goal is to turn that progress into something easier to evaluate in small production-like deployments.

Theme

Operational readiness for locality-aware NoSQL.

v0.10 should not only make KoutenDB faster. It should make KoutenDB easier to trust, inspect, recover, and validate before a user puts real application data behind it.

Primary Workstreams

1. Operational Doctor And Verify

Add an operator-facing diagnosis path that can be run before startup, after an incident, or during deployment checks.

Initial implementation status: started. operationalVerify(dataDir) and kouten verify --data=DIR now exercise persistent store open/replay, data-dir lock acquisition, metadata counts, locality metrics, and optional segment rebuild verification. kouten doctor --data=DIR uses the same operational path, while kouten doctor without --data remains the optional dependency setup doctor. kouten verify --backup=DIR and kouten doctor --backup=DIR now route through the strict backup verifier. kouten verify --server-config=FILE and kouten doctor --server-config=FILE validate koutend JSON startup configuration, including auth/TLS/file sanity. --metrics and --json provide machine-readable output for operational automation. --max-wal-bytes=N and --max-segment-files=N, --max-items=N, and --max-rings=N let deployment checks fail early when a trial exceeds operator-defined capacity thresholds. Cluster maintenance primitives are also started: kouten drain, kouten snapshot, and kouten resume expose admin-only read-only drain, flush/report, and resume controls for rolling maintenance and backup windows. Optional local soak tooling is also started: examples/soak_72h.sh runs a configurable three-node mixed workload for long-running pre-release validation and leaves JSONL progress, node logs, final metrics, snapshot output, and offline verify reports.

Candidate commands:

  • kouten doctor
  • kouten verify
  • kouten verify --data=DIR
  • kouten verify --segments
  • kouten verify --backup=DIR

Checks should cover:

  • data directory ownership and lock state;
  • WAL header, record length, and checksum validity;
  • segment files and rebuildability;
  • ring metadata consistency;
  • galaxy identity;
  • auth/TLS configuration sanity;
  • backup readability;
  • broad-scan guardrail configuration.

The output should be human-readable by default and machine-readable with a metrics or JSON option.

2. Backup, Restore, And Recovery Drills

KoutenDB already has backup and restore primitives. v0.10 should make the operational path clearer and easier to prove.

Target improvements:

  • documented backup / restore / verify flow;
  • explicit failure-drill demos;
  • restore into a fresh data directory;
  • segment rebuild after restore;
  • WAL repair behavior examples;
  • encrypted backup verification examples;
  • recovery smoke scripts that are safe to run locally.

The goal is to show that KoutenDB can be tested like a database, not only tried like a library.

3. Quotas And Guardrails

Production trials need predictable failure modes. KoutenDB should reject or warn before an accidental workload turns into an uncontrolled scan, oversized payload, or runaway import.

Initial implementation status: started. Embedded handles now expose opt-in KoutenGuardrails for write-path caps on payload bytes, vector dimensions, ring count, and records per ring. The defaults are disabled so existing NoSQL-style workloads keep their current behavior.

Candidate guardrails:

  • max payload bytes; initial embedded API done;
  • max ring count; initial embedded API and verify --max-rings=N done;
  • max records per ring; initial embedded API done;
  • max broad-scan candidates;
  • max request body bytes;
  • max import batch bytes;
  • max WAL bytes threshold; verify --max-wal-bytes=N done;
  • max segment file count threshold; verify --max-segment-files=N done;
  • max item count threshold; verify --max-items=N done;
  • max segment rebuild bytes warning threshold.

These should be configurable globally and, where it makes sense, per galaxy or ring profile.

4. Access And Change Audit

KoutenDB already tracks some operational counters and universe/warp state. v0.10 should add a clearer audit boundary for deployment evaluation.

Candidate audit events:

  • auth success/failure;
  • denied ring access;
  • write/update/delete;
  • backup/restore/compact/segment rebuild;
  • config load;
  • broad-scan warning;
  • universe sync apply/skip/dead-letter;
  • lock acquire/release/failure.

This does not need to become a heavy enterprise audit subsystem in v0.10. A focused append-only audit log with tests is enough to make production trials safer.

Initial implementation status: started. Persistent embedded stores now append kouten.audit.jsonl for direct write/update/delete, backup, restore, compact, and guardrail denial events. Persistent koutend nodes also append server-side auth success/failure, authz-denied, retrieve-denied, and broad-scan-denied events. This is still a focused operational audit trail, not a full enterprise audit policy engine.

5. Production Config And Docker Compose Examples

Many users need to see how a system should be placed before they evaluate it. v0.10 should add realistic examples rather than only small command snippets.

Initial implementation status: started. examples/compose/operational-trial.compose.yml now provides a local production-like trial with an authenticated persistent node, a healthcheck, a tools profile, offline verify --data, backup, verify --backup, and audit JSONL inspection. See docs/operational-trials.md. koutend --config=FILE and KOUTEN_SERVER_CONFIG=FILE now load server defaults from JSON, with CLI flags overriding the file. This gives production examples a single file for node id, peers, data directory, durability, auth, ring prefixes, roles, and TLS paths.

Candidate examples:

  • single persistent node;
  • TLS + username/password/secret-key;
  • backup volume;
  • recovery verification;
  • universe sync between two local placements;
  • read-heavy RAG corpus;
  • user-detail application data layout;
  • Docker Compose examples for local production-like trials.

Large benchmarks should remain manual. CI should keep fast smoke coverage and not generate multi-million record datasets by default.

6. Bulk Load And Segment Pack Hardening

v0.9 showed that KoutenDB can run large generated validations, but it also made bulk import and segment packing the next obvious performance target.

Target areas:

  • JSONL import throughput;
  • batch commit size behavior;
  • many-ring metadata overhead;
  • segment pack scheduling;
  • segment rebuild after restore;
  • metrics for import and pack stages;
  • clear guidance for memory-backed vs disk-backed validation.

This is both a performance task and an operations task: users should know when to import, when to pack, and how to verify the resulting layout.

7. Locality Read Hardening

KoutenDB should continue to protect the read path that is most central to its design: bounded nearby data retrieval.

Target areas:

  • stellar reads with per-subring limits and sorts;
  • bounded ring-window reads;
  • pinpoint entity reads such as users/<id>;
  • related-data bundle reads;
  • metrics for candidate count, returned count, ring count, and latency;
  • adversarial locality validation with writes, deletes, backfills, and compact.

The goal is not to claim universal superiority over PostgreSQL, Redis, or other databases. The goal is to make the locality-shaped workload measurable and hard to dismiss.

Non-Goals For v0.10

The following are useful, but should not block v0.10 unless user feedback changes the priority:

  • full managed-service control plane;
  • dynamic online cluster membership;
  • cluster transaction coordinator redundancy;
  • full MVCC;
  • LangChain / LlamaIndex adapters;
  • WASM browser storage;
  • official Unity / Unreal assets;
  • complete Prometheus / Datadog integrations.

Release Bar

v0.10 should be considered ready when:

  • doctor / verify cover the main data-dir, WAL, segment, backup, and config failure modes;
  • admin-only drain / snapshot / resume controls exist and are smoke-tested;
  • backup/restore/failure-drill examples are documented and smoke-tested;
  • guardrails exist for the highest-risk accidental workloads;
  • audit events cover auth, writes, deletes, restore/compact, and broad scans;
  • production-like Docker Compose examples exist;
  • the large validation path remains manual, documented, and reproducible;
  • CI stays fast and does not include multi-million record benchmarks by default.

Expected Message

The release message should be:

KoutenDB v0.10 makes locality-aware NoSQL easier to evaluate operationally: diagnose it, verify it, back it up, restore it, guard it, and run realistic production-like demos without turning large benchmarks into CI requirements.