RocheDB x Shelfer Integration

RocheDB and Shelfer address the same cost problem at different layers. RocheDB alone is not positioned as a complete answer to AI infrastructure consumption. The strategy is a toolchain in which each component removes a different kind of waste, reducing search scope, transfer volume, prompt tokens, worker fanout, retries, and unsafe context inclusion in aggregate.

RocheDB reduces the candidate set before data reaches an LLM or reranker through placement, rings, halos, galaxies, vector backends, and retrieval statistics. Shelfer handles MCP execution boundaries, Delivery, worker routing, token/cost budgets, RAG utility, and audit.

Therefore RocheDB should not become Shelfer’s internal database. The default integration shape is that Shelfer sees RocheDB as a RAG source, document source, or retrieval source.

corpus / events / documents
  -> RocheDB galaxy / ring / vector backend
  -> RocheDB retrieval envelope
  -> Shelfer RAG source adapter
  -> Shelfer Delivery / policy / metrics / audit
  -> LLM or allowlisted MCP workers
  -> Shelfer utility feedback
  -> RocheDB routing and ring policy tuning

Responsibility Split

Layer Responsibility
RocheDB document / vector storage, ring scoped retrieval, galaxy isolation, candidate reduction, payload projection, auth boundary
Shelfer MCP tools/resources/prompts, Delivery queues, worker allowlists, token/cost budgets, RAG/resource utility, review and audit ledgers
prompt/content guard untrusted RAG/tool/worker text inspection before LLM context inclusion
flow-control layer retry, backoff, circuit breaking, bulkheads, deadlines, token/cost budgets
Adapter stable JSON envelope, policy hints, metric mapping, retry/backoff integration

RocheDB should not expose Shelfer-specific runtime internals in its public core. Shelfer should not depend on RocheDB’s orbital internals. The shared contract is a small retrieval envelope plus metrics.

Retrieval Envelope Contract

Shelfer is expected to release after RocheDB, so RocheDB should stabilize an adapter-neutral retrieval envelope first. Shelfer can later implement a consumer adapter or plugin against this contract.

Canonical constants:

Name Value
RetrievalEnvelopeSchema rochedb.retrieval.v1
RetrievalEnvelopeVersion 1

A RocheDB-backed source returns an envelope shaped like this:

{
  "schema": "rochedb.retrieval.v1",
  "version": 1,
  "source": {
    "provider": "rochedb",
    "galaxy": "tenant-a",
    "ring": "docs/security",
    "backend": "exact",
    "sourceType": "document"
  },
  "query": {
    "mode": "vector",
    "budget": 8,
    "ringScoped": true
  },
  "plan": {
    "strategy": "ring-scoped",
    "baseRing": "docs/security",
    "amount": "raNormal",
    "scope": "ssTight",
    "depth": "sdNormal",
    "ringScoped": true,
    "budget": 8,
    "focus": 0,
    "topRings": 0,
    "effectiveTopRings": 0,
    "branchBudget": 0,
    "maxDepth": 0,
    "includeChildren": false,
    "reason": "explicit ring scope",
    "selectedRings": ["docs/security"],
    "prunedRings": []
  },
  "chunks": [
    {
      "id": "0000000000000001:00000002",
      "payload": "...",
      "score": 0.93,
      "estimatedTokens": 120,
      "ring": "docs/security",
      "sourceUri": "rochedb://tenant-a/docs/security/0000000000000001:00000002"
    }
  ],
  "stats": {
    "totalVectors": 1000000,
    "scanned": 12000,
    "skippedVectors": 988000,
    "returned": 8,
    "ringsTouched": 1,
    "fanoutNodes": 2,
    "payloadBytes": 3900,
    "estimatedTokens": 975,
    "candidateReduction": 0.988
  },
  "policyHints": {
    "resourceKind": "rag",
    "resourceScope": "topic",
    "retentionClass": "normal",
    "contextReusable": true,
    "dataLabel": "internal"
  }
}

This keeps the contract useful for Shelfer, HTTP clients, and future drivers without making Shelfer a hard dependency of RocheDB.

RocheDB exposes:

  • retrievalEnvelope(...): produce the canonical JSON envelope.
  • RetrievalEnvelopeSchema: schema identifier.
  • RetrievalEnvelopeVersion: integer schema version.
  • retrievalEnvelopeValidationErrors(env): return compatibility errors.
  • isValidRetrievalEnvelope(env): convenience boolean.

Required Fields

Path Meaning
schema / version Contract identity and compatibility version
source.provider Usually rochedb; adapters may namespace derived providers
source.galaxy Isolation / tenant boundary when available
source.ring Routed ring / topic namespace when available
source.backend vbExact, cluster, future faiss, etc.
source.sourceType document, event, code, metric, etc.
query.mode Retrieval mode such as vector
query.budget Requested result budget
query.ringScoped Whether the query was explicitly scoped to a ring
plan Executed retrieval plan. Human-facing fields are amount, scope, and depth; numeric fields are internal diagnostics
plan.ringFeatures Query-aware ring candidates used by the planner, including centroid similarity, ring count, and base/sibling/descendant flags
chunks[] Ordered retrieval candidates
stats Candidate reduction and payload/token estimates
policyHints Advisory metadata for runtime policy

Compatibility Rules

  • RocheDB core may add optional fields to v1 envelopes.
  • RocheDB must not remove or rename required v1 fields without a new schema.
  • Consumers must ignore unknown fields.
  • Consumers should reject envelopes that fail retrievalEnvelopeValidationErrors.
  • Policy hints are advisory. They do not change auth, galaxy isolation, or runtime policy by themselves.
  • estimatedTokens is an approximation for routing and measurement, not a billing source of truth.

Retrieval Plan Tuning

RocheDB should expose query execution as a plan, similar in spirit to SQL EXPLAIN and optimizer hints. The first version is deliberately non-destructive: it changes the retrieval plan, not physical data placement.

The plan is the right place to represent ring collision / satellite / bridge behavior before allowing any physical merge or re-parenting.

Examples:

Situation Plan-level expression
Explicit ring search strategy = "ring-scoped", selectedRings = [baseRing]
Broad search with bounded fanout strategy = "top-rings", effectiveTopRings > 0
Future parent/child traversal strategy = "hierarchical-ring", includeChildren = true, maxDepth > 0
Collision-like co-selection selected sibling rings in the same plan
Bridge behavior include both branches for a query family without moving data
Satellite behavior prefer a child ring when entering a parent ring

Current implementation:

  • Ring names separated by / are registered as a hierarchy when written.
  • depth = sdDeep / sdVeryDeep expands a base ring into child rings in the retrieval plan.
  • scope = ssNear / ssWide / ssAll expands a ring to siblings under the same parent.
  • The exact vector backend can search multiple selected rings in one pass.
  • Ring names are persisted in the WAL and hierarchy is restored after reopen.
  • Persistent embedded stores can compact the WAL with compact / roche compact --data=DIR, keeping only live records and metadata.
  • Persistent embedded stores can create and restore compact WAL backups with backup / restoreBackup or roche backup --data=DIR --backup=DIR and roche restore --backup=DIR --data=DIR.
  • Persistent embedded stores can export readable JSON Lines dumps with dump or roche dump --data=DIR --out=FILE; this is for inspection and migration, not crash recovery.
  • External NoSQL JSON Lines exports can be imported with routing rules: importJsonl(..., ringField = "tenant", ringPrefix = "tenant/", payloadField = "body", vecField = "embedding") or roche import-jsonl --data=DIR --in=FILE --ring-field=tenant. The selected ring is created automatically, so imports can distribute existing documents into RocheDB’s ring hierarchy during ingestion.
  • The builtin planner ranks expanded candidates with deterministic DB-local features: base ring priority, centroid similarity to the query, optional utility, and ring count.
  • Planner selection is deterministic heuristic ranking inside RocheDB. Model optimizers stay outside the read path; agents should use atlas, stats, and explain output to recommend profiles or ring changes.
  • Physical merge, split, re-parent, bridge creation, and collision automation are still advisory/future work.

The human-facing tuning words are:

Field Choices Meaning
amount raFew, raNormal, raMany, raAllUseful How many useful chunks to return
scope ssTight, ssNear, ssWide, ssAll How widely RocheDB may search across related rings
depth sdShallow, sdNormal, sdDeep, sdVeryDeep How deeply RocheDB may descend through ring hierarchy

The numeric fields (budget, focus, effectiveTopRings, branchBudget, maxDepth) are diagnostics and low-level override points. Application code should normally use the words above.

This gives each deployment room to tune for its own workload:

  • low-latency systems can use amount = raFew, scope = ssTight, depth = sdShallow;
  • recall-sensitive systems can use amount = raMany, scope = ssWide, depth = sdDeep;
  • deep research systems can use amount = raAllUseful, scope = ssAll, depth = sdVeryDeep;
  • tenant-sensitive systems can forbid cross-galaxy or cross-label plan expansion;
  • Shelfer can compare plan variants through utility and context-investment metrics.

Physical rewrite operations such as merge, split, or re-parent should remain explicit policy decisions. Plan changes are safe to A/B test and easy to roll back.

RocheDB exposes named tuning profiles for this purpose:

db.configureSearchProfile("short",
  SearchProfile(amount: raFew, scope: ssTight, depth: sdShallow,
                note: "short answer"))

db.configureSearchProfile("wide",
  SearchProfile(amount: raMany, scope: ssWide, depth: sdDeep,
                note: "wider answer"))

let fast = db.retrievalEnvelopeTuned(queryVec, ring = "docs/security",
                                     profile = "short")
let broad = db.retrievalEnvelopeTuned(queryVec, ring = "docs/security",
                                      profile = "wide")

The analogy to RDB tuning is:

RDB concept RocheDB equivalent
SQL text retrieval request + base ring
optimizer plan RetrievalPlan
index choice ring / future vector backend / future hierarchy traversal choice
optimizer hint SearchProfile / internal RetrievalTuning profile
workload advisor external agent using atlas and stats, not RocheDB read path
EXPLAIN envelope plan + stats
table statistics ring summaries, ring metrics, candidate reduction
query plan regression test quality-fixed rag-bench / Shelfer utility comparison

Shelfer Metric Mapping

RocheDB already exposes RetrieveStats fields that map naturally to Shelfer’s RAG/resource utility and context investment metrics.

RocheDB field Shelfer usage
chunks[].id selectedResources, usedItemIds, ignoredItemIds
source.galaxy resourceProvider = "rochedb:<galaxy>" or tenant label
source.ring ragNamespace / resourceNamespace
source.backend ragProfile / resourceProfile
stats.estimatedTokens contextLoadTokens
baseline tokens - routed tokens qualityAdjustedSavedTokens, when quality is unchanged
candidateReduction retrieval efficiency signal
fanoutNodes / ringsTouched routing pressure and source-selection signal

Shelfer records whether the returned context actually changed the answer, which chunks were used, and which chunks were ignored. RocheDB should use that feedback to tune ring policies and default budgets, not to silently change authorization or runtime policy.

Safety Boundary

RocheDB retrieval results are data cargo. Shelfer should treat them as untrusted RAG content until runtime policy accepts them.

  • Auth, secret keys, and galaxy isolation remain RocheDB-side access controls.
  • Shelfer applies worker allowlists, labels, Delivery policy, and audit.
  • Prompt/content inspection can run before RocheDB chunks enter an LLM context.
  • Plugin or adapter metadata is advisory until a Shelfer host explicitly adopts it.
  • Utility feedback must not bypass tenant, galaxy, or credential boundaries.

Implementation Order

  1. Keep RocheDB core independent and emit the stable retrieval envelope.
  2. Add a small RocheDB RAG adapter for Shelfer as a separate module or plugin.
  3. Map RocheDB RetrieveStats into Shelfer RAG/resource utility records.
  4. Add a feedback loop that recommends ring budgets and source routing.
  5. Measure quality-fixed reductions with roche rag-bench and Shelfer context investment reports.

The important KPI is not only latency. For AI workloads the target is:

same recall / answer quality
with fewer scanned vectors, fewer returned chunks, fewer prompt tokens,
less reranker work, and less worker fanout.