KoutenDB Status / Roadmap
KoutenDB Status / Roadmap
This is the canonical English status document. Translations are secondary references and may lag behind this file.
Release checklist: release-checklist.md
Translations:
- Japanese: koutendb-status.ja.md
- German: koutendb-status.de.md
- French: koutendb-status.fr.md
- Chinese: koutendb-status.zh.md
- Korean: koutendb-status.ko.md
Core DB
| Feature | Status | Notes |
|---|---|---|
| Embedded DB | Done | open(dataDir=...) and memory-only mode |
| put / get | Done | Ring-scoped writes and ID-based reads |
| ORM foundation API | Done | Embedded APIs plus cluster PoC for update, JSON patch, deleteById, listByRing, countByRing; canonical data is expected to live in one galaxy/ring, with alternate views handled by ring hierarchy, naming, import rules, and retrieval profiles; driver exposure is still pending |
| Warp belt | PoC | WAL-backed delayed patch queue: enqueueWarp, warpStep, warpDrain; scans specified rings in registration order and drops merge patches onto matching JSON documents; includes minimal attempts / retryAt / maxAttempts / ack / dead-letter state plus acked-job cleanup; FlowBrigade and FlowLogbook adapters are planned instead of core dependencies; server scheduling is still pending |
| JSON document query | Done | GraphQL-style selection |
| Payload codecs | PoC | Per-record raw / json / nif / bif metadata survives WAL, cluster transport, handoff, transactions, Universe sync, and retrieval. NIF/BIF encoding/decoding remains outside the core; optional adapter: koutendb-nif backed by nifkit |
| Prepared selection | Done | Reusable validated projection tree in embedded mode plus a bounded server-side parse cache for cluster queries |
| Vector retrieve | Done | FAISS bridge is the intended production vector path. Exact backend remains for small datasets, tests, and fallback |
| Ring / hierarchy | Done | ring = "a/b/c" and child-ring expansion |
| Galaxy isolation | Done | Separate data dir / peer list / credential boundary |
| Atlas / ring map | Done | atlas() and kouten atlas |
| Galaxy/ring description | Done | Atlas map annotations, not payload text |
| Time orbit | PoC | Embedded ring-local 60-bit millisecond orbit for log/event/time-series placement. Includes persisted profiles, putTime / readTime, and kouten time-orbit/time-put/time-get; cluster profile administration is still pending. Design note: time-orbit.md |
| Retrieval tuning profile | Done | amount / scope / depth |
| FAISS vector backend | PoC | Dynamic bridge via libkouten_faiss.so; default fetch tag is FAISS v1.14.3, exact commit pinning is optional via KOUTEN_FAISS_COMMIT; kouten doctor, bridge build, tests/tapi.nim, and examples/vector_backend_bench.sh are verified locally |
| FAISS GPU backend | Not planned for core | KoutenDB is designed to reduce the search set before ANN/LLM work. Needing GPU FAISS is treated as a placement or retrieval-profile issue first |
| Retrieval planner | PoC | Deterministic heuristic planner. Stronger planner claims require larger real-corpus benchmarks and further tuning |
| WASM browser embedded | Post-v0.1 candidate | Browser state boundary / IndexedDB / OPFS |
Persistence / Operations
| Feature | Status | Notes |
|---|---|---|
| Append-only WAL | Done | Batched flush by default; durStrong / --durability=strong adds flush + fsync write boundaries. New WAL files use a magic/version header and per-record length + CRC32 wrappers; legacy pre-v1.0 WAL remains readable for migration |
| Reopen recovery | Done | Items / vectors / ring metadata / descriptions |
| Transaction | Done | Embedded atomic transaction plus all-or-nothing batchPutAtomic, batchUpdateAtomic, and batchDeleteAtomic helpers |
| Cooperative coordinate locks | Done | Embedded opt-in ring and stellar locks for high-integrity workflows; normal NoSQL read/write paths do not check locks |
| Cluster transaction landing | PoC | node0 landing. scripts/cluster_tx_smoke.sh covers apply smoke; scripts/cluster_failure_smoke.sh covers owner crash/restart retry; redundancy is not implemented yet |
| Cluster CRUD/list/count | PoC | update, deleteById, JSON patch, listByRing, countByRing use landing intents or node fan-out; scripts/cluster_tx_smoke.sh covers smoke |
| Compact | Done | Rebuilds WAL from live records |
| Backup / restore | Done | Backup as compacted WAL and restore into another data dir |
| Dump / import-jsonl | Done | NoSQL JSONL import rules. This is the stable human-readable migration boundary while the pre-v1.0 internal WAL format can still evolve |
| Universe sync outbox | PoC | WAL-backed eventual sync event queue with idempotent apply, ack/prune, transaction-backed putSynced, prune-safe monotonic source ids, latest-only pending coalescing, delayed timestamp apply windows, retryAt / maxAttempts / dead-letter state, kouten universe-export / universe-apply JSONL handoff, one-shot kouten universe-sync between local data dirs, remote --peers delivery via UAPPLY, and universe-status operational counters. It is a durable scheduler boundary, not immediate global consistency |
| Crash recovery tests | Partial | Torn WAL tail repair, versioned-WAL checksum mismatch refusal, mid-file WAL corruption refusal, compact interruption, partial commit cases |
| Strong durability / fsync knob | Done | open(dataDir=..., durability=durStrong) and koutend --durability=strong; store/API tests cover reopen, transaction, compact |
| Core test suite | Done | scripts/test_core.sh runs orbital core, selection, field, store, and public API tests |
| Full smoke suite | Done | scripts/test_all_smoke.sh runs core tests plus cluster tx, failure retry, authz, wire fuzz, recovery, and remote universe sync smoke; driver compatibility is opt-in |
| Generation snapshot / checkpoint | Planned | Generational snapshot and checkpoint are pending; encrypted backup is available |
| Kubernetes manifests | Planned | liveness/readiness, PVC, rolling restart |
Cluster / Network
| Feature | Status | Notes |
|---|---|---|
| Static cluster | Done | koutend --id --peers |
| Deterministic locate | Done | E(id,t) -> node |
| Handoff / forwarder | PoC | Slow tick integration exists. Fully distributed forwarder placement is not done |
| Driver-friendly wire | Done | PUTR/GETID/QRYID; WIREVER exposes the current protocol version and CODECS exposes payload formats. Compatibility policy is documented in docs/protocol-compatibility.md |
| Health / metrics / rings | Done | CLI and wire protocol; metrics include uptime, request/error/auth counters, connection counts, WAL bytes, warp backlog, universe apply counters, cluster tx backlog, and storage/ring counts |
| Authn + secret key | Done | username/password/secret-key; unusable credential combinations fail at startup |
| TLS | Done | Standard TLS transport for koutend and CLI/client connections when built with -d:ssl; scripts/cluster_tls_smoke.sh covers authenticated TLS, secret-key transport, JSON put/get, and plain-client rejection |
| Authz / RBAC | PoC | koutend --allow-ring=prefix[,prefix...] and --role=user:password:reader|writer|admin[:prefixes]; scripts/cluster_authz_smoke.sh and scripts/cluster_rbac_smoke.sh cover prefix and role matrix behavior |
| Wire fuzz smoke | Done | scripts/cluster_wire_fuzz_smoke.sh runs deterministic malformed-frame cases, including oversized headers and deep JSON, and verifies the cluster stays healthy |
| Server resource guardrails | Partial | Accepted sockets have a body-read timeout and fixed active-connection cap; fuller request-deadline and per-query cost controls remain planned |
| Bounded server retrieve | Done | koutend keeps only the current top candidates up to request budget while scanning local vectors instead of retaining every matching payload before truncation |
| Dynamic membership / epoch migration | Foundation | Current peer list is still static at runtime, but v0.6 adds explicit arc tables, weighted arcs, deterministic virtual arcs, topology validation, and remapFraction so membership changes can be modeled with less unnecessary remapping than naive mod nNodes. Online rebalance workflow is still planned |
| Cluster transaction coordinator redundancy | Planned | Remove node0 as a single point of failure |
| Read-your-writes for cluster tx | PoC | get/query/batchGet fallback to node0 landing intent before owner apply; cluster smoke covers update/delete |
| Fault-tolerance improvements | Planned | Post-v0.1 work; universe sync outbox is now the first durable eventual-convergence primitive |
| Multi-VM / multi-AZ benchmark | Planned | Real-world latency and failure behavior |
Drivers / Bindings
| Target | Status | Notes |
|---|---|---|
| Nim API | Done | Native public API |
| C ABI | Done | ABI version / last error / put/get/retrieve/batch/atlas plus additive codec-aware put/get calls; C ABI vectors are host-native float arrays, while TCP wire vectors are canonical little-endian float32 |
| JavaScript / TypeScript | Published | npm koutendb v0.1.3; repository puffball1567/koutendb-js; Node-API C ABI wrapper with TypeScript API |
| Bun | Partial | The npm package uses Node-API and includes Bun compatibility verification, but Bun support remains experimental |
| Rust | Published | crates.io koutendb v0.1.3; repository puffball1567/koutendb-rust; C ABI wrapper |
| Python | Published | PyPI koutendb v0.1.3; repository puffball1567/koutendb-python; native TCP wire driver |
| Go | Done | C ABI wrapper minimal |
| PHP | Published | Packagist koutendb/koutendb v0.1.2; repository puffball1567/koutendb-php; FFI / C ABI wrapper with Docker smoke |
| Swift | Done | SwiftPM C ABI wrapper with Linux Docker smoke |
| C# minimal | Done | OSS generic C# wrapper. Unity official asset is separate |
| C++ | Released | Repository puffball1567/koutendb-cpp v0.1.1; C++17 C ABI wrapper with CMake smoke; Unreal official plugin is separate |
| Kotlin-first JVM | Done | JNI / C ABI wrapper with Docker smoke |
| React Native / WASM local state | Post-v0.1 candidate | Browser / React Native state boundary; handled with the WASM line, not before Kotlin |
| Driver discovery CLI | Done | kouten driver list/info/install prints official driver metadata and setup commands without executing remote scripts |
| Driver compatibility test suite | Partial | scripts/driver_compat.sh; Docker-backed PHP / Swift / Kotlin are opt-in and verified |
| Package publishing | Partial | nimble install koutendb, cargo add koutendb, npm install koutendb, composer require koutendb/koutendb, and python3 -m pip install koutendb are available. NuGet, Maven, Go, SwiftPM, and other registry packages remain future work |
Benchmarks / Demos
| Item | Status | Notes |
|---|---|---|
| Working-set bench | Done | scanned/query reduction |
| Memory-pressure bench | Done | estimated candidate memory/query |
| RAG-style bench | Done | recall retained while tokens/query are reduced |
| AI/RAG JSONL case study | Done | examples/ai_rag_case_study.sh generates a deterministic multi-ring JSONL corpus, imports it, and compares global / routed / wrong-ring retrieval |
| PostgreSQL comparison | Done | Limited reference comparison |
| Redis comparison | Done | Smoke test with conditions and limits documented |
| C ABI bench | Done | examples/cbench.c |
| Docker case study | Partial | memory pressure / PHP / Swift smoke |
| Unique data model demo | Done | examples/stellar_data_model_demo.sh demonstrates separate rings, stellar attach/detach, narrowed reads, and non-copy visibility changes |
| Cluster transaction smoke | Partial | scripts/cluster_tx_smoke.sh starts 3 local nodes and verifies apply / retrieve |
| Cluster failure retry smoke | Partial | scripts/cluster_failure_smoke.sh kills the owner node, verifies the intent remains pending, restarts the owner, and verifies retry apply |
| Universe sync demo | Done | examples/universe_sync_demo.sh builds a small source/target pair, demonstrates API-level sync, then demonstrates the CLI export/sync/prune boundary. scripts/universe_sync_failure_smoke.sh verifies malformed JSONL handling, replay idempotency, and explicit ack/prune. scripts/universe_sync_remote_smoke.sh verifies remote --peers delivery and target-down retry behavior |
| Payload codec demos | Done | examples/payload_codecs_demo.sh covers embedded persistence and prepared selection; examples/payload_codecs_cluster_demo.sh covers codec negotiation and legacy wire-header compatibility |
| Crash / failure case study | Partial | Store-level WAL tail repair, mid-file WAL corruption refusal, compact interruption, partial commit, and cluster owner crash/restart retry are covered |
| Multi-node cloud case study | Planned | VM/AZ, latency, failover behavior |
| Prometheus / Datadog exporter | Post-v0.1 candidate | Core exposes key/value metrics now; OpenMetrics / Datadog collector should be added outside the core server loop |
| State boundary demo | Post-v0.1 candidate | browser/RN local-global state demo |
Security / Safety
| Item | Status | Notes |
|---|---|---|
| Username/password auth | Done | koutend and driver path; user without password fails closed |
| Secret key gate | Done | ID/password alone can be insufficient; secret-key without user/password fails closed |
| nimsodium encryption primitive | Partial | Used for auth transport; scope may expand |
| Galaxy isolation | Done | Limits blast radius by galaxy |
| TLS | Done | Standard TCP transport TLS is implemented for -d:ssl builds; certificate rotation and managed CA workflows remain operational work |
| Ring/galaxy authz | PoC | Ring prefix authorization is implemented for named-ring wire operations; richer role policy is pending |
| Backup encryption | Done | backupEncrypted / restoreEncryptedBackup and kouten backup-encrypted / restore-encrypted use nimsodium secretbox |
| General audit log | Planned | Full append-only access/change audit for enterprise / regulated workloads. Warp jobs already persist attempts / retryAt / ack / dead-letter state, but that is job state, not a database-wide audit log |
| Threat model document | Draft | docs/threat-model.md covers assets, trust boundaries, current controls, and known gaps |
Post-v0.1 Roadmap Candidates
These are candidates for v0.2 and later releases. They are not all scoped to a single v0.2.0 milestone.
- WASM browser embedded
- IndexedDB / OPFS persistence
- React hooks / browser state boundary
- React Native / WASM local state module
- Unity official asset
- Unreal official plugin
- package publishing workflows for remaining language drivers
- API reference documentation
- Prometheus / OpenMetrics and Datadog metrics adapters
- Fault-tolerance improvements
Managed Service Readiness Gaps
KoutenDB should be able to become a managed service in the same operational category as hosted cache, document, search, or AI-context databases. Some managed-service requirements are already expressible through KoutenDB concepts: replication-style redundancy maps to universes, logical isolation maps to galaxies, read scope maps to rings, and backup verification maps to recovery universes.
The following items are the remaining implementation candidates that are not fully covered by the current concepts or code:
| Candidate | Why it is needed |
|---|---|
| Durable eventual universe sync | Universes currently cover recovery topology. A managed service also needs live delayed convergence between same-name galaxies across universes without global commit waits. |
| Ring apply policy | Managed deployments need per-ring behavior such as latest-only, append-only, bounded-history, and delayed timestamp apply. This keeps consistency rules explicit without making the whole DB strongly serializable. |
| Read-your-writes across local pending state | Local users should not feel universe-sync delay. The cluster landing-intent fallback is a start; universe-level pending overlays are still missing. |
| Dynamic node replacement | Managed services must replace failed or upgraded nodes without manual peer-list surgery. Current clusters use static peers. |
| Cluster coordinator redundancy | Node0 is currently the transaction landing zone. Managed service readiness needs the coordinator role to survive node replacement or failover. |
| TLS and certificate rotation | Username/password/secret-key auth exists, but managed public or VPC deployments need transport TLS and rotation workflows. |
| Secret rotation | authProfiles reference external secrets, but the server and drivers need an explicit rotation story for username/password/secret-key credentials. |
| Point-in-time recovery / generation checkpoints | Backup/restore exists. Managed services normally require recoverable generations, restore-point selection, and verification before promotion. |
| Drain / quiesce / snapshot barrier | Rolling maintenance and consistent managed backups need a control-plane hook to stop accepting new writes, flush durable state, and report readiness. |
| OpenMetrics / CloudWatch / Datadog adapters | KoutenDB exposes key/value metrics, but managed integrations need standard exporters or collectors. |
| Quotas and capacity guardrails | Galaxy isolation exists, but managed multi-tenant operation needs limits for WAL bytes, item count, ring count, payload size, and connection pressure. |
| Protocol / storage compatibility policy | Managed upgrades need clear compatibility rules for wire protocol, WAL records, snapshots, and drivers. |
These gaps define the boundary between a promising server database and a provider-ready managed database. KoutenDB should not copy every Redis, RDS, or ElastiCache mechanism one-to-one; it should provide equivalent operational outcomes where KoutenDB’s universe / galaxy / ring model already gives a simpler or more natural shape.