# Serving with vLLM

## `I` Overview

Semioserve is myModel's catalogue-driven vLLM launcher.
It resolves one or more models from a YAML catalogue, builds the `vllm serve` command line from each model's `build_args`, supervises the subprocess, and exposes a thin management API on a separate port.

Modern vLLM serves the OpenAI API natively (`vllm serve`), so Semioserve no longer reimplements inference endpoints — chat completions, embeddings, and tool calls are handled entirely by vLLM's built-in server.
Semioserve's job is orchestration: pick models from the catalogue, launch them, monitor their health, and front the management API with auth and TLS when you run in production.

```
                    ┌─────────────────────────────────────────┐
   client ─────────▶│ management API (Quart :5000)             │
                    │  /models  /health                         │
                    ├─────────────────────────────────────────┤
                    │ vLLM server :5001  (model A)              │
                    │ vLLM server :5002  (model B)              │
                    └─────────────────────────────────────────┘
```

## `II` The Model Catalogue

The catalogue is a plain YAML file that maps human-readable model names to HuggingFace model IDs, build arguments, and metadata.
Two seed catalogues ship with the repo:

| File                                      | Contents                                                |
| ----------------------------------------- | ------------------------------------------------------- |
| `mymodel/catalogue/data/optimizers.yaml`  | Generative / transformer models (CHAT, CODE, DRAW, ...) |
| `mymodel/catalogue/data/classifiers.yaml` | Reductive models (EMBED, SCORE, SEGMENT, ...)           |

### What a ModelCard is

Every entry in the catalogue is a `ModelCard` — a Pydantic model that holds the persisted, HuggingFace-backed metadata record for one inference model.
Key fields:

| Field        | Type                   | Description                                          |
| ------------ | ---------------------- | ---------------------------------------------------- |
| `myname`     | `str`                  | System name (e.g. `gemma_md`, `deepseek-coder_sm`)   |
| `hfname`     | `str`                  | HuggingFace repo id (e.g. `google/gemma-2-9b-it`)    |
| `verb`       | `Verb`                 | The model's function (CHAT, EMBED, CODE, ...)        |
| `modality`   | `(Modality, Modality)` | Input / output modality pair                         |
| `engine`     | `str`                  | Model framework; `remote` for non-local models       |
| `family`     | `str`                  | Optional family name for grouping related models     |
| `size`       | `Size`                 | Coarse parameter-count bucket (XS, SM, MD, LG, XL)   |
| `f_size`     | `str`                  | Total file size on disk (e.g. `29GB`)                |
| `params`     | `str`                  | Number of parameters (e.g. `7B`)                     |
| `weight`     | `int`                  | Weight datatype bit width (16, 32, ...)              |
| `build_args` | `dict`                 | Engine-init kwargs passed to `vllm serve`            |
| `infer_args` | `dict`                 | Inference-time params (temperature, max_tokens, ...) |

### The Verb enum

A `Verb` describes the input-to-output transformation a model performs.
It is the primary grouping key in the catalogue YAML — each verb is a top-level section:

```yaml
CHAT:
    aquila_md:
        hfname: BAAI/AquilaChat2-7B
        ...
EMBED:
    ...
```

Common verbs include:

| Verb       | Input        | Output       | Example use               |
| ---------- | ------------ | ------------ | ------------------------- |
| `CHAT`     | Text / Image | Text / Image | Conversational assistants |
| `CODE`     | Text / Code  | Code         | Code completion           |
| `EMBED`    | Text         | Vector       | Sentence embeddings       |
| `COMPLETE` | Text         | Text         | Text completion           |
| `DRAW`     | Text         | Image        | Text-to-image generation  |
| `SEE`      | Image        | Text         | Image captioning          |
| `SPEAK`    | Text         | Speech       | Text-to-speech            |

The full enum lives in `mymodel/catalogue/ontology.py`.

### The Modality flag

`Modality` is a `Flag` enum representing datatypes a model can consume or produce (TEXT, CODE, IMAGE, AUDIO, VIDEO, ...).
Every `Verb` is defined by an `(input, output)` modality pair, so models cluster naturally by the transformation they perform.

Modality pairs are written as `input -> output` strings in the catalogue:

```yaml
modality: text -> vector
```

If omitted, the modality defaults to the verb's own input/output pair.

### The Size enum

`Size` is a coarse parameter-count bucket parsed from a model's name suffix:

| Size | Suffix | Typical VRAM |
| ---- | ------ | ------------ |
| `XS` | `_xs`  | < 1 GB       |
| `SM` | `_sm`  | 1–4 GB       |
| `MD` | `_md`  | 4–12 GB      |
| `LG` | `_lg`  | 12–30 GB     |
| `XL` | `_xl`  | > 30 GB      |

The suffix is parsed automatically from the model name (e.g. `gemma_md`).
If no suffix is present, `Size(0)` (no size) is used.

### How the Registry works

The `Registry` class owns one YAML catalogue file and its live `ModelCard` instances.

```python
from mymodel.catalogue import Registry, OPTIMIZERS

registry = Registry(OPTIMIZERS)
# registry.models is a dict[str, ModelCard]
```

The Registry can:

- **Load** a catalogue YAML into `ModelCard` instances (`Registry.load()`)
- **Fetch** metadata from HuggingFace (`Registry.refresh_hf_data()`) — social
  fields (downloads, likes, license), technical fields (tags, architecture,
  files), and config details are populated automatically
- **Add** a new model (`Registry.add()`) — creates the card, fetches HF data,
  and persists it to the YAML index
- **Write** the catalogue back to disk (`Registry.write_index()`)
- **Resolve** effective build/infer args through layered defaults
  (`Registry.resolve_args()`)

### Listing available models

```sh
# List all model names in the catalogue
uv run python -c "
from mymodel.catalogue import Registry, OPTIMIZERS
r = Registry(OPTIMIZERS)
for name in sorted(r.models):
    print(name)
"
```

Or browse the YAML directly:

```sh
grep -E '^[A-Z]' mymodel/catalogue/data/optimizers.yaml
```

### Adding a new model

Use the Python API to add a model — it fetches HuggingFace metadata and writes the catalogue automatically:

```python
import asyncio
from mymodel.catalogue import Registry, OPTIMIZERS

async def main():
    r = Registry(OPTIMIZERS)
    card = await r.add(
        hfname='google/gemma-2-9b-it',
        myname='gemma_md',           # your system name
        verb='CHAT',                 # the Verb
        family='',                   # optional family grouping
    )
    print(f'Added {card.myname} ({card.params}, {card.f_size})')

asyncio.run(main())
```

The `add` method asserts the model does not already exist, creates a `ModelCard`, populates it from HuggingFace, registers it, and writes the index.

## `III` Launching a Model

### How serve.py works

The `mymodel.serve` module is the entry point for launching catalogued models as vLLM-native OpenAI servers.

At a high level:

1. **Parse arguments** — model names (or `VERB:name` specs), host, port, dev
   mode, directory, and catalogue path
2. **Load the catalogue** — instantiate a `Registry` from the YAML file
3. **Resolve models** — each requested model is looked up by name; if the name
   is not in the catalogue, a `KeyError` is raised
4. **Build the command** — each model's `build_args` dict is rendered as
   `vllm serve` CLI flags via the `_to_flags()` helper
5. **Launch subprocesses** — each model gets its own vLLM server on an
   incrementing port (base port + 1, + 2, ...)
6. **Serve the management API** — a Quart app on the base port exposes
   `/models` and `/health`

### OpsStance: dev vs production

`OpsStance` controls deployment posture:

| Mode  | Auth                       | TLS client-auth | Use case          |
| ----- | -------------------------- | --------------- | ----------------- |
| `DEV` | Disabled on management API | Relaxed         | Local development |
| `PRD` | Enabled (JWT via Keycloak) | Enforced        | Production        |

In dev mode, the management API is served without authentication.
In production mode (`--dev_mode` not set), an `AuthMiddleware` wraps the ASGI app and verifies a Keycloak-issued JWT (HS512) on every HTTP/websocket connection.

### The `_to_flags` function

`build_args` is a dict of key-value pairs.
`_to_flags()` converts it to a list of CLI arguments:

- `True` values become bare flags: `--tensor-parallel-size` (if value is `True`)
- `False` and `None` values are skipped
- Other values become `--key value` pairs

For example:

```python
build_args = {'dtype': 'bfloat16', 'tensor_parallel_size': 1, 'enable_lora': True}
# → ['--dtype', 'bfloat16', '--tensor-parallel-size', '1', '--enable-lora']
```

Underscores in keys are converted to hyphens (`tensor_parallel_size` → `tensor-parallel-size`).

### The vllm serve command

The full argument vector built by `VLLMServer.argv()`:

```
vllm serve <model_ref> \
  --served-model-name <myname> \
  --host <host> \
  --port <port> \
  [--api-key <api_key>] \
  <build_args flags...>
```

`model_ref` is the local weights path if it exists on disk, otherwise the HuggingFace id for vLLM to fetch.
`--served-model-name` is the card's `myname` — this is the model name clients use when sending requests.

### Launching in dev mode

```sh
# Default: dev mode, localhost:5000 management, models on 5001+
uv run python -m mymodel gemma_md
```

Multiple models:

```sh
uv run python -m mymodel gemma_md deepseek-coder_sm
# gemma_md → port 5001, deepseek-coder_sm → port 5002, management on 5000
```

With explicit options:

```sh
uv run python -m mymodel gemma_md \
  --host 0.0.0.0 \
  --port 8000 \
  --catalogue mymodel/catalogue/data/optimizers.yaml
```

### Launching in production mode

The `--dev_mode` flag is `store_true` with a default derived from the `NUCLEUS_MODE` environment variable.
Production mode is active when `NUCLEUS_MODE` does not start with `dev` **and** `--dev_mode` is not passed.

```sh
# Production mode: enables JWT auth on the management API
NUCLEUS_MODE=prd uv run python -m mymodel gemma_md
```

```sh
# Explicitly force dev mode regardless of NUCLEUS_MODE
uv run python -m mymodel gemma_md --dev_mode
```

### The management API

Two endpoints are exposed on the management port:

| Endpoint  | Method | Description                                          |
| --------- | ------ | ---------------------------------------------------- |
| `/models` | GET    | Active servers (with live health) and full catalogue |
| `/health` | GET    | Quick health check: `{"status": "ok", "models": N}`  |

```sh
curl http://localhost:5000/health
# {"status": "ok", "models": 1}

curl http://localhost:5000/models
# {"active": {"gemma_md": {"verb": "chat", "url": "http://localhost:5001/v1", "healthy": true}}, "catalogue": [...]}
```

### SSL/TLS configuration

If a certificate pair exists under `<directory>/ssl/server.crt` and `<directory>/ssl/server.key`, the management API is served over TLS.

In dev mode, an optional CA certificate (`ssl/ca.crt`) is loaded for client-auth relaxation.
In production mode, client-auth CA is not applied — the JWT middleware handles authentication instead.

To enable TLS:

```sh
mkdir -p ~/.my/ssl
# Place your cert and key
cp server.crt ~/.my/ssl/server.crt
cp server.key ~/.my/ssl/server.key
# Optional: CA cert for dev-mode client auth
cp ca.crt ~/.my/ssl/ca.crt
```

Without a cert pair, the management API falls back to plain HTTP (typical in development).

### Subprocess supervision

Each vLLM server runs as an async subprocess managed by the `VLLMServer` class.

- **Start**: `VLLMServer.start()` launches `vllm serve` as a subprocess
- **Health check**: `VLLMServer.healthy()` polls `http://<host>:<port>/health`
- **Shutdown**: `VLLMServer.stop()` sends SIGTERM to the subprocess and waits
  for it to exit
- **Signal handling**: SIGTERM to the management process triggers a graceful
  shutdown of all model subprocesses

## `IV` GPU Memory and Model Sizing

### Choosing a model that fits your GPU

The `Size` enum gives a coarse hint, but actual VRAM consumption depends on several factors.

| Size | Typical params | Approximate VRAM (FP16) |
| ---- | -------------- | ----------------------- |
| `XS` | < 1B           | < 2 GB                  |
| `SM` | 1B–7B          | 2–14 GB                 |
| `MD` | 7B–13B         | 14–26 GB                |
| `LG` | 13B–70B        | 26–140 GB               |
| `XL` | > 70B          | > 140 GB                |

### Key fields

| Field    | What it tells you                                |
| -------- | ------------------------------------------------ |
| `f_size` | Total on-disk file size of the model weights     |
| `params` | Parameter count (e.g. `7B`, `406M`)              |
| `weight` | Weight datatype bit width (16 = FP16, 32 = FP32) |
| `size`   | Coarse bucket from the name suffix               |

### Weight datatype considerations

The `weight` field indicates the bit width of the stored parameters.
FP16 (16-bit) models use roughly half the VRAM of FP32 (32-bit) models for the same parameter count.
A 7B model in FP16 needs about 14 GB of VRAM; in FP32 it needs about 28 GB.

### How build_args affect memory

The `build_args` dict is passed directly to `vllm serve`, so any vLLM flag that controls memory usage goes here:

| Flag                     | Effect                                            |
| ------------------------ | ------------------------------------------------- |
| `dtype`                  | Weight dtype (`bfloat16`, `float16`, `auto`)      |
| `tensor_parallel_size`   | Split model across N GPUs                         |
| `gpu_memory_utilization` | Fraction of GPU memory vLLM may use (default 0.9) |
| `max_model_len`          | Maximum context length (longer = more KV cache)   |
| `quantization`           | Weight quantization method (`awq`, `gptq`, ...)   |
| `enable_lora`            | Enable LoRA adapter hot-swapping                  |

For example, to serve a model with tensor parallelism across two GPUs and bfloat16 weights:

```yaml
build_args:
  dtype: bfloat16
  tensor_parallel_size: 2
```

## `V` Connecting Through the Gateway

The LiteLLM gateway (`~/my/apps/gate/gateway/`) fronts your local vLLM servers with a single OpenAI-compatible endpoint on `http://localhost:4000/v1`.
Clients point at the gateway and select a model by alias.

### The local-coder alias

The gateway's `~/my/apps/gate/gateway/config.yaml` includes an entry for locally-served vLLM models:

```yaml
model_list:
    - model_name: local-coder
      litellm_params:
        model: openai/local-coder          # must match vLLM --served-model-name
        api_base: os.environ/LOCAL_VLLM_API_BASE
        api_key: os.environ/LOCAL_VLLM_API_KEY
```

The `model:` field (after the `openai/` prefix) must match the `--served-model-name` that vLLM was launched with — which is the catalogue card's `myname`.
So if you launch `gemma_md`, the gateway alias `local-coder` will route to it only if the `model:` field is `openai/gemma_md`.

### Setting api_base

`LOCAL_VLLM_API_BASE` points to the vLLM server's OpenAI endpoint:

```sh
export LOCAL_VLLM_API_BASE=http://localhost:8000/v1
export LOCAL_VLLM_API_KEY=your-api-key
```

When running the gateway inside a container, use `host.docker.internal:8000` instead of `localhost:8000` to reach the host's vLLM server.

### Testing the connection

```sh
# Start the gateway
cd ~/my/apps/gate/gateway && task up

# Verify it is live
curl http://localhost:4000/health/liveliness

# Send a test request through the gateway
curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_TOKEN" \
  -d '{
    "model": "local-coder",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 128
  }'
```

See {ref}`gateway` for the full routing configuration, remote providers, and secret management.

## `VI` Troubleshooting

### GPU out of memory

```
ValueError: NCCL or CUDA out of memory
```

vLLM tries to use 90% of GPU VRAM by default.
If you are running multiple models or other GPU workloads, reduce `gpu_memory_utilization`:

```yaml
build_args:
  gpu_memory_utilization: 0.5   # use only 50% of VRAM
```

Or switch to a smaller `Size` bucket model, enable quantization, or use tensor parallelism across multiple GPUs.

### Model not found in catalogue

```
KeyError: "Model 'my_model' is not in the catalogue optimizers.yaml."
```

The model name you passed must match a `myname` in the catalogue YAML.
Check spelling and verify with:

```sh
uv run python -c "
from mymodel.catalogue import Registry, OPTIMIZERS
r = Registry(OPTIMIZERS)
print(sorted(r.models))
"
```

If the model genuinely is missing, add it with `Registry.add()` (see [Adding a new model](#adding-a-new-model)).

### Port conflicts

If the default ports (5000 for management, 5001+ for models) are already in use, specify alternatives:

```sh
uv run python -m mymodel gemma_md --port 9000
# Management on 9000, model server on 9001
```

Or set the `NUCLEUS_PORT` environment variable:

```sh
NUCLEUS_PORT=9000 uv run python -m mymodel gemma_md
```

### vLLM startup failures

**`vllm` not on PATH:**

```
RuntimeError: `vllm` is not on PATH; install the serving extra to launch models.
```

Install vLLM in your environment (`uv sync` should pull it in, or install it manually: `pip install vllm`).

**Model download fails:**

vLLM fetches models from HuggingFace when local weights are not found.
Ensure `HF_TOKEN` is set if the model is gated, and that you have network access to `huggingface.co`.

**Health check never passes:**

vLLM can take 30–60 seconds to load weights and initialize the KV cache.
Check the vLLM subprocess logs for errors.
The management API's `/health` endpoint returns immediately; the `/models` endpoint reports per-model health by polling `http://<host>:<port>/health`.

**Management API returns 401 in production mode:**

In production mode (`NUCLEUS_MODE=prd`), the management API requires a valid Keycloak-issued JWT.
Switch to dev mode for local debugging: `NUCLEUS_MODE=dev` or pass `--dev_mode`.