lmgo-v2

中文版 README

lmgo-v2 is a Windows system tray app that wraps llama.cpp server in router mode, providing dynamic model switching without restarts. The app itself is backend-agnostic — pair it with any llama-server build (CUDA, ROCm, Vulkan, CPU).

System Requirements

• OS: Windows 11
• Architecture: x86_64
• llama-server: Any backend — grab a pre-built binary from llama.cpp releases, or for AMD GPUs use llamacpp-rocm.

Features

• Router Mode: Single llama-server process serves all models, switches on-demand via API
• Dynamic Model Switching: No restart required — change the model field in your API request
• System Tray: Minimal tray menu to open Web UI, toggle auto-start, and refresh config
• Auto-start on Boot: Option to start with Windows via Startup folder shortcut
• Active Model Display: Shows which model is currently loaded in the tray menu
• Standard OpenAI API: GET /v1/models, POST /v1/chat/completions — any OpenAI-compatible client works
• INI Model Config: models.ini defines models with per-model parameters (speculative decoding, mmproj, KV cache, etc.)

Quick Start

1. Download lmgo-v2.exe from releases
2. Place it in an empty folder
3. Run it — a tray icon appears, config.json and models.ini are generated
4. Edit models.ini to define your models (paths, parameters)

Configuration

config.json

{
  "autoStartEnabled": false,
  "port": 19966,
  "pollInterval": 2
}

Field	Description
autoStartEnabled	Auto-start with Windows
port	llama-server HTTP port
pollInterval	How often (seconds) to check which model is loaded for the tray display

models.ini

[*] section contains global defaults inherited by all models. Each model section defines per-model overrides:

[*]
host = 0.0.0.0
no-host = true
ngl = all
flash-attn = on
ctx-size = 131072
batch-size = 4096
ubatch-size = 4096
threads = 0
threads-batch = 0
cache-type-k = f16
cache-type-v = f16
kv-offload = true
no-mmap = true
direct-io = true
mlock = true
split-mode = layer
main-gpu = 0

[llama-3-8b]
model = D:/LLM/Llama-3-8B-Instruct.gguf
ctx-size = 8192
temp = 0.7

[gemma-4-12b]
model = D:/LLM/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf
ctx-size = 262144
model-draft = D:/LLM/mtp-gemma-4-12B.gguf
spec-type = draft-mtp
spec-draft-n-max = 4
mmproj = D:/LLM/mmproj-F16.gguf

The section name is the model identifier used in API requests. All llama-server CLI flags are valid INI keys (remove the leading --).

API Usage

GET  http://localhost:19966/v1/models
POST http://localhost:19966/v1/chat/completions

Switch models by changing the model field:

curl http://localhost:19966/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma-4-12b","messages":[{"role":"user","content":"Hello"}]}'

from openai import OpenAI
client = OpenAI(base_url="http://localhost:19966/v1", api_key="not-needed")
client.chat.completions.create(
    model="gemma-4-12b",
    messages=[{"role":"user","content":"Hello"}],
)

Building from Source

1. Download the latest llama.cpp server package from llama.cpp releases (e.g. llama-b*-bin-win-cuda-cu12.4-x64.zip) and place it in the project root
2. Run:

go mod tidy
go build -ldflags "-s -w -H windowsgui" .

The .zip is embedded into the binary via //go:embed and extracted to server/ on first run. To use a different build (e.g. ROCm), replace the .zip file before building.

Related

• lmgo — Previous version (process-based model management + lmc TUI)
• llamacpp-rocm — AMD ROCm builds of llama.cpp