Local AI serving (Ollama on inva-local-01)

Local model serving runs on inva-local-01 — an RTX 3080 (10 GB VRAM), driver 595.71.05, CUDA 13.2. Ollama serves the fleet at http://10.30.0.4:11434 over the WireGuard overlay. This is Tier 0 of Emma's model routing; external APIs are the escalation tier.

Model fleet

Model	Role
qwen2.5:14b	Quality tier — default Emma reasoning/synthesis
llama3.1:8b	Fast / concurrent tier — founder chat under load; RAG grounded answers
nomic-embed-text	RAG embeddings (768-d)

VRAM is the hard ceiling (10 GB): the 14B weights (~8.9 GB) sit at 93 % GPU. The fleet is not co-resident — the app router swaps between the quality and concurrency modes. Models are stored under /data/ollama-models (NVMe hot tier).

Tuning (env flags)

Applied as an Ollama service drop-in:

Flag	Value	Effect
OLLAMA_FLASH_ATTENTION	1	Flash attention
OLLAMA_KV_CACHE_TYPE	q8_0	Quantized KV cache (VRAM headroom)
OLLAMA_NUM_PARALLEL	2	Concurrent request slots (raise to 4 for peak-heavy)
OLLAMA_MAX_LOADED_MODELS	2	Cap resident models
OLLAMA_MAX_QUEUE	256	Queue depth (rare concurrent requests queue, not fail)
OLLAMA_KEEP_ALIVE	30m	Keep the quality model resident

Combined with num_ctx=2048, flash-attention + q8 KV cache gave the 14B a 6× speedup (7.8 → 47 tok/s).

Benchmark matrix (RTX 3080, 10 GB)

Model	Config	Throughput	GPU placement	Load	Use
qwen2.5:14b	default (untuned)	7.8 tok/s	90 % GPU (10 % CPU spill)	19 s	— (shows why tuning matters)
qwen2.5:14b	tuned, num_ctx=2048	47 tok/s	93 % GPU	5.6 s	Quality tier
llama3.1:8b	single-stream	108 tok/s	100 % GPU	warm	Fast tier
llama3.1:8b	4 concurrent	130 agg tok/s	100 % GPU	warm	Concurrent founder chat
nomic-embed-text	sequential	8.1 emb/s (768-d)	GPU	warm	RAG embeddings

Key results:

1. Tuning gave the 14B a 6× speedup (7.8 → 47 tok/s) — flash-attention + q8 KV cache + 2048 ctx.
2. The pre-tuning serialization is fixed: 4-concurrent 8B rose from ~90 → 130 agg tok/s (now exceeds single-stream).
3. 10 GB VRAM is the hard cap: the 14B leaves the model at 93 % GPU. A smaller quant (Q3_K_M) would be 100 % GPU at some quality cost; 47 tok/s at 93 % is a good quality tier.

Recommended serving profile

Matches the owner directive (quality-first, low concurrency, good queue):

• Default quality tier: qwen2.5:14b, num_ctx=2048, tuning on → ~47 tok/s. Keep resident (KEEP_ALIVE=30m). Rare concurrent requests queue (OLLAMA_MAX_QUEUE=256) with a "thinking…" UX.
• Concurrency tier (founder peaks): flip the app router to llama3.1:8b (100 % GPU, 130 agg tok/s at 4 parallel). For peak-heavy deployments set OLLAMA_NUM_PARALLEL=4 (trades some VRAM).
• RAG embeddings: nomic-embed-text — batch ingestion off the request path (outbox → worker); query-time embed ~120 ms.

Gotchas

Ollama multipart pull corrupts on this link

On the residential link, Ollama's 16-way multipart pull produces random digest mismatches on large blobs. Workaround: single-stream curl the GGUF from Hugging Face → ollama create -f Modelfile (bypasses the parallel downloader). This is how the 14B landed. Run ollama create in a clean foreground shell — it panics under a systemd-run root env.

WireGuard link drops

The residential-NAT WG link drops periodically; self-heal reconnects in ~40 s. Management SSH is affected; serving is not — cloudflared dials outbound. Authoritative data on the box therefore needs backups + the outbound serving path.