Local LLM Playground

Problem

Cloud LLM APIs are powerful but expensive and unsuitable for sensitive infrastructure data. Engineers need a fast local loop for prompt iteration.

Architecture

1. 1React UI with streaming SSE client
2. 2Next.js API route as thin gateway
3. 3Ollama runtime on local GPU/CPU
4. 4Model registry with pull/list/switch

Screenshots

Lessons Learned

• Streaming tokens early improves perceived latency more than raw tokens/sec.
• Model switching needs explicit context resets to avoid silent prompt bleed.
• Quantized 7B–14B models are enough for most DevOps assistant workflows.