Skip to content

Data layer & RAG

Emma's authoritative application data and retrieval pipeline run in Docker on inva-local-01 under ~inva/emma/data/ and ~inva/emma/rag/. The store is Postgres 16.14 + pgvector 0.8.5 plus a dual-Redis pair; retrieval is a hybrid FTS + vector pipeline enforced under row-level security. Secrets come from GCP Secret Manager via agentops-secret — never inline.

Containers

Container Image Config Bind
postgres Postgres 16.14 + pgvector 0.8.5 PGDATA on NVMe; shared_buffers=4GB, effective_cache_size=24GB localhost
redis-queue Redis 1 GB, noeviction, AOF — durable queue / outbox drain localhost
redis-cache Redis 2 GB, allkeys-lru — ephemeral session/cache localhost

Schema (schema/001_memory_rag.sql)

  • knowledge_chunks — durable RAG corpus. Each row carries a ts tsvector (GIN index, full-text search) and an embedding vector(768) (HNSW cosine index, m=16, ef_construction=200). This is what makes hybrid retrieval possible.
  • memory_items — durable classified memory (see Emma memory); same dual FTS + vector indexing.
  • project_documents — document-level records.
  • audit_events — append-only, monthly-partitioned, INSERT-only trigger (no UPDATE/DELETE).
  • outbox — transactional outbox; work is committed atomically with the source change and drained by a worker (via redis-queue).

Isolation (RLS)

Row-level security is enabled and forced on the tenant tables. Policies enforce:

  • Org isolation — rows are visible only within the caller's org_id.
  • personal_private — owner-only rows, invisible even to same-org peers.

The caller sets app.user_id, app.org_id, app.role GUCs per transaction; the application connects as the non-owner emma_app role (cannot bypass RLS). Verified live: cross-org leak blocked, personal_private owner-only, denial when no authorized source exists.

RAG pipeline (rag/rag.py)

Hybrid retrieval with grounded generation, all under RLS in a single transaction:

  1. Embed the query with nomic-embed-text (768-d).
  2. Retrieve candidates two ways over knowledge_chunks:
  3. FTS via the ts tsvector (GIN);
  4. Vector nearest-neighbour via the HNSW cosine index.
  5. Fuse the two rankings with Reciprocal Rank Fusion (RRF).
  6. Generate a grounded answer with llama3.1:8b, emitting citations to source chunks.
  7. Record audit_events + outbox in the same transaction.

Verified: correct grounded citation; cross-org leak blocked (Alice cannot see org B's "Northwind"); personal_private owner-only; denial when no authorized source; audit + outbox written. Embeddings are computed off the request path for ingestion (outbox → worker); query-time embed is ~120 ms.

Connection pooling

Add PgBouncer in transaction mode when the Emma application connects, so many short-lived app connections multiplex onto a small Postgres backend pool. Tracked as an open item on inva-local-01.

Backups

PGDATA lives on the NVMe hot tier; pg backups, WAL archive, and snapshots land on the /srv/data cold tier (1.8 TB HDD). Because the WG management link is intermittent and the box holds authoritative application data, on-box backups plus the outbound serving path are the durability strategy.