Skip to content

Model routing

Emma routes each turn through an LLMGateway that prefers local models and escalates to external APIs only when needed and permitted. Tier 0 is the local Ollama fleet on inva-local-01; Tier 1 is external API escalation behind a data-governance gate.

Tier 0 — local (Ollama)

Served at http://10.30.0.4:11434 over the overlay. The gateway selects a mode rather than a fixed model — see Local AI serving for benchmarks and tuning.

Mode Model Profile
Quality (default) qwen2.5:14b, num_ctx=2048, tuned ~47 tok/s, kept resident; rare concurrent requests queue
Concurrency (founder peaks) llama3.1:8b 100 % GPU, 130 agg tok/s at 4 parallel
Embeddings nomic-embed-text 768-d, off the request path (outbox → worker)

The RTX 3080's 10 GB VRAM is the ceiling: the router swaps between quality and concurrency modes rather than co-residing both models with the embedder.

Tier 1 — external API escalation

The gateway escalates hard, low-confidence, or large-context turns to external providers:

  • Anthropic Claude — e.g. a Haiku-class model to classify/route and a Sonnet-class model to synthesize.
  • OpenAI — alternative escalation target.

API key required — not a subscription

External escalation needs a provider API key (emma-model-api-key, currently empty). A consumer/human subscription cannot be used as a backend. Until a key is provisioned, Emma runs local-only.

Data-governance gate

Restricted and personal_private data stays local-only — it is never sent to an external provider. The governance gate blocks escalation for those turns regardless of difficulty.

Serving profile (benchmark-backed)

Matches the owner directive (quality-first, low concurrency, good queue):

  • Default: quality mode (qwen2.5:14b), resident, queue-on-contention (OLLAMA_MAX_QUEUE=256) with a "thinking…" UX.
  • Peaks: flip to concurrency mode (llama3.1:8b); set OLLAMA_NUM_PARALLEL=4 for peak-heavy deployments.
  • Escalate: only hard/low-confidence/large-context turns, only when a key exists, only when the governance gate allows.

See the full benchmark matrix.

Open items

  • Cut Emma's ModelProvider to local as the default backend.
  • Provision emma-model-api-key (via GCP Secret Manager) to enable Tier 1 escalation.
  • Optionally pull qwen2.5-coder:7b / qwen2.5:3b (GGUF-curl path) for dev/classify tiers.