Serving with vLLM#

I Overview#

Semioserve is myModel’s catalogue-driven vLLM launcher. It resolves one or more models from a YAML catalogue, builds the vllm serve command line from each model’s build_args, supervises the subprocess, and exposes a thin management API on a separate port.

Modern vLLM serves the OpenAI API natively (vllm serve), so Semioserve no longer reimplements inference endpoints — chat completions, embeddings, and tool calls are handled entirely by vLLM’s built-in server. Semioserve’s job is orchestration: pick models from the catalogue, launch them, monitor their health, and front the management API with auth and TLS when you run in production.

                    ┌─────────────────────────────────────────┐
   client ─────────▶│ management API (Quart :5000)             │
                    │  /models  /health                         │
                    ├─────────────────────────────────────────┤
                    │ vLLM server :5001  (model A)              │
                    │ vLLM server :5002  (model B)              │
                    └─────────────────────────────────────────┘

II The Model Catalogue#

The catalogue is a plain YAML file that maps human-readable model names to HuggingFace model IDs, build arguments, and metadata. Two seed catalogues ship with the repo:

File

Contents

mymodel/catalogue/data/optimizers.yaml

Generative / transformer models (CHAT, CODE, DRAW, …)

mymodel/catalogue/data/classifiers.yaml

Reductive models (EMBED, SCORE, SEGMENT, …)

What a ModelCard is#

Every entry in the catalogue is a ModelCard — a Pydantic model that holds the persisted, HuggingFace-backed metadata record for one inference model. Key fields:

Field

Type

Description

myname

str

System name (e.g. gemma_md, deepseek-coder_sm)

hfname

str

HuggingFace repo id (e.g. google/gemma-2-9b-it)

verb

Verb

The model’s function (CHAT, EMBED, CODE, …)

modality

(Modality, Modality)

Input / output modality pair

engine

str

Model framework; remote for non-local models

family

str

Optional family name for grouping related models

size

Size

Coarse parameter-count bucket (XS, SM, MD, LG, XL)

f_size

str

Total file size on disk (e.g. 29GB)

params

str

Number of parameters (e.g. 7B)

weight

int

Weight datatype bit width (16, 32, …)

build_args

dict

Engine-init kwargs passed to vllm serve

infer_args

dict

Inference-time params (temperature, max_tokens, …)

The Verb enum#

A Verb describes the input-to-output transformation a model performs. It is the primary grouping key in the catalogue YAML — each verb is a top-level section:

CHAT:
    aquila_md:
        hfname: BAAI/AquilaChat2-7B
        ...
EMBED:
    ...

Common verbs include:

Verb

Input

Output

Example use

CHAT

Text / Image

Text / Image

Conversational assistants

CODE

Text / Code

Code

Code completion

EMBED

Text

Vector

Sentence embeddings

COMPLETE

Text

Text

Text completion

DRAW

Text

Image

Text-to-image generation

SEE

Image

Text

Image captioning

SPEAK

Text

Speech

Text-to-speech

The full enum lives in mymodel/catalogue/ontology.py.

The Modality flag#

Modality is a Flag enum representing datatypes a model can consume or produce (TEXT, CODE, IMAGE, AUDIO, VIDEO, …). Every Verb is defined by an (input, output) modality pair, so models cluster naturally by the transformation they perform.

Modality pairs are written as input -> output strings in the catalogue:

modality: text -> vector

If omitted, the modality defaults to the verb’s own input/output pair.

The Size enum#

Size is a coarse parameter-count bucket parsed from a model’s name suffix:

Size

Suffix

Typical VRAM

XS

_xs

< 1 GB

SM

_sm

1–4 GB

MD

_md

4–12 GB

LG

_lg

12–30 GB

XL

_xl

> 30 GB

The suffix is parsed automatically from the model name (e.g. gemma_md). If no suffix is present, Size(0) (no size) is used.

How the Registry works#

The Registry class owns one YAML catalogue file and its live ModelCard instances.

from mymodel.catalogue import Registry, OPTIMIZERS

registry = Registry(OPTIMIZERS)
# registry.models is a dict[str, ModelCard]

The Registry can:

  • Load a catalogue YAML into ModelCard instances (Registry.load())

  • Fetch metadata from HuggingFace (Registry.refresh_hf_data()) — social fields (downloads, likes, license), technical fields (tags, architecture, files), and config details are populated automatically

  • Add a new model (Registry.add()) — creates the card, fetches HF data, and persists it to the YAML index

  • Write the catalogue back to disk (Registry.write_index())

  • Resolve effective build/infer args through layered defaults (Registry.resolve_args())

Listing available models#

# List all model names in the catalogue
uv run python -c "
from mymodel.catalogue import Registry, OPTIMIZERS
r = Registry(OPTIMIZERS)
for name in sorted(r.models):
    print(name)
"

Or browse the YAML directly:

grep -E '^[A-Z]' mymodel/catalogue/data/optimizers.yaml

Adding a new model#

Use the Python API to add a model — it fetches HuggingFace metadata and writes the catalogue automatically:

import asyncio
from mymodel.catalogue import Registry, OPTIMIZERS

async def main():
    r = Registry(OPTIMIZERS)
    card = await r.add(
        hfname='google/gemma-2-9b-it',
        myname='gemma_md',           # your system name
        verb='CHAT',                 # the Verb
        family='',                   # optional family grouping
    )
    print(f'Added {card.myname} ({card.params}, {card.f_size})')

asyncio.run(main())

The add method asserts the model does not already exist, creates a ModelCard, populates it from HuggingFace, registers it, and writes the index.

III Launching a Model#

How serve.py works#

The mymodel.serve module is the entry point for launching catalogued models as vLLM-native OpenAI servers.

At a high level:

  1. Parse arguments — model names (or VERB:name specs), host, port, dev mode, directory, and catalogue path

  2. Load the catalogue — instantiate a Registry from the YAML file

  3. Resolve models — each requested model is looked up by name; if the name is not in the catalogue, a KeyError is raised

  4. Build the command — each model’s build_args dict is rendered as vllm serve CLI flags via the _to_flags() helper

  5. Launch subprocesses — each model gets its own vLLM server on an incrementing port (base port + 1, + 2, …)

  6. Serve the management API — a Quart app on the base port exposes /models and /health

OpsStance: dev vs production#

OpsStance controls deployment posture:

Mode

Auth

TLS client-auth

Use case

DEV

Disabled on management API

Relaxed

Local development

PRD

Enabled (JWT via Keycloak)

Enforced

Production

In dev mode, the management API is served without authentication. In production mode (--dev_mode not set), an AuthMiddleware wraps the ASGI app and verifies a Keycloak-issued JWT (HS512) on every HTTP/websocket connection.

The _to_flags function#

build_args is a dict of key-value pairs. _to_flags() converts it to a list of CLI arguments:

  • True values become bare flags: --tensor-parallel-size (if value is True)

  • False and None values are skipped

  • Other values become --key value pairs

For example:

build_args = {'dtype': 'bfloat16', 'tensor_parallel_size': 1, 'enable_lora': True}
# → ['--dtype', 'bfloat16', '--tensor-parallel-size', '1', '--enable-lora']

Underscores in keys are converted to hyphens (tensor_parallel_size → tensor-parallel-size).

The vllm serve command#

The full argument vector built by VLLMServer.argv():

vllm serve <model_ref> \
  --served-model-name <myname> \
  --host <host> \
  --port <port> \
  [--api-key <api_key>] \
  <build_args flags...>

model_ref is the local weights path if it exists on disk, otherwise the HuggingFace id for vLLM to fetch. --served-model-name is the card’s myname — this is the model name clients use when sending requests.

Launching in dev mode#

# Default: dev mode, localhost:5000 management, models on 5001+
uv run python -m mymodel gemma_md

Multiple models:

uv run python -m mymodel gemma_md deepseek-coder_sm
# gemma_md → port 5001, deepseek-coder_sm → port 5002, management on 5000

With explicit options:

uv run python -m mymodel gemma_md \
  --host 0.0.0.0 \
  --port 8000 \
  --catalogue mymodel/catalogue/data/optimizers.yaml

Launching in production mode#

The --dev_mode flag is store_true with a default derived from the NUCLEUS_MODE environment variable. Production mode is active when NUCLEUS_MODE does not start with dev and --dev_mode is not passed.

# Production mode: enables JWT auth on the management API
NUCLEUS_MODE=prd uv run python -m mymodel gemma_md

# Explicitly force dev mode regardless of NUCLEUS_MODE
uv run python -m mymodel gemma_md --dev_mode

The management API#

Two endpoints are exposed on the management port:

Endpoint

Method

Description

/models

GET

Active servers (with live health) and full catalogue

/health

GET

Quick health check: {"status": "ok", "models": N}

 
curl http://localhost:5000/health
# {"status": "ok", "models": 1}

curl http://localhost:5000/models
# {"active": {"gemma_md": {"verb": "chat", "url": "http://localhost:5001/v1", "healthy": true}}, "catalogue": [...]}

SSL/TLS configuration#

If a certificate pair exists under <directory>/ssl/server.crt and <directory>/ssl/server.key, the management API is served over TLS.

In dev mode, an optional CA certificate (ssl/ca.crt) is loaded for client-auth relaxation. In production mode, client-auth CA is not applied — the JWT middleware handles authentication instead.

To enable TLS:

mkdir -p ~/.my/ssl
# Place your cert and key
cp server.crt ~/.my/ssl/server.crt
cp server.key ~/.my/ssl/server.key
# Optional: CA cert for dev-mode client auth
cp ca.crt ~/.my/ssl/ca.crt

Without a cert pair, the management API falls back to plain HTTP (typical in development).

Subprocess supervision#

Each vLLM server runs as an async subprocess managed by the VLLMServer class.

  • Start: VLLMServer.start() launches vllm serve as a subprocess

  • Health check: VLLMServer.healthy() polls http://<host>:<port>/health

  • Shutdown: VLLMServer.stop() sends SIGTERM to the subprocess and waits for it to exit

  • Signal handling: SIGTERM to the management process triggers a graceful shutdown of all model subprocesses

IV GPU Memory and Model Sizing#

Choosing a model that fits your GPU#

The Size enum gives a coarse hint, but actual VRAM consumption depends on several factors.

Size

Typical params

Approximate VRAM (FP16)

XS

< 1B

< 2 GB

SM

1B–7B

2–14 GB

MD

7B–13B

14–26 GB

LG

13B–70B

26–140 GB

XL

> 70B

> 140 GB

Key fields#

Field

What it tells you

f_size

Total on-disk file size of the model weights

params

Parameter count (e.g. 7B, 406M)

weight

Weight datatype bit width (16 = FP16, 32 = FP32)

size

Coarse bucket from the name suffix

Weight datatype considerations#

The weight field indicates the bit width of the stored parameters. FP16 (16-bit) models use roughly half the VRAM of FP32 (32-bit) models for the same parameter count. A 7B model in FP16 needs about 14 GB of VRAM; in FP32 it needs about 28 GB.

How build_args affect memory#

The build_args dict is passed directly to vllm serve, so any vLLM flag that controls memory usage goes here:

Flag

Effect

dtype

Weight dtype (bfloat16, float16, auto)

tensor_parallel_size

Split model across N GPUs

gpu_memory_utilization

Fraction of GPU memory vLLM may use (default 0.9)

max_model_len

Maximum context length (longer = more KV cache)

quantization

Weight quantization method (awq, gptq, …)

enable_lora

Enable LoRA adapter hot-swapping

For example, to serve a model with tensor parallelism across two GPUs and bfloat16 weights:

build_args:
  dtype: bfloat16
  tensor_parallel_size: 2

V Connecting Through the Gateway#

The LiteLLM gateway (~/my/apps/gate/gateway/) fronts your local vLLM servers with a single OpenAI-compatible endpoint on http://localhost:4000/v1. Clients point at the gateway and select a model by alias.

The local-coder alias#

The gateway’s ~/my/apps/gate/gateway/config.yaml includes an entry for locally-served vLLM models:

model_list:
    - model_name: local-coder
      litellm_params:
        model: openai/local-coder          # must match vLLM --served-model-name
        api_base: os.environ/LOCAL_VLLM_API_BASE
        api_key: os.environ/LOCAL_VLLM_API_KEY

The model: field (after the openai/ prefix) must match the --served-model-name that vLLM was launched with — which is the catalogue card’s myname. So if you launch gemma_md, the gateway alias local-coder will route to it only if the model: field is openai/gemma_md.

Setting api_base#

LOCAL_VLLM_API_BASE points to the vLLM server’s OpenAI endpoint:

export LOCAL_VLLM_API_BASE=http://localhost:8000/v1
export LOCAL_VLLM_API_KEY=your-api-key

When running the gateway inside a container, use host.docker.internal:8000 instead of localhost:8000 to reach the host’s vLLM server.

Testing the connection#

# Start the gateway
cd ~/my/apps/gate/gateway && task up

# Verify it is live
curl http://localhost:4000/health/liveliness

# Send a test request through the gateway
curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_TOKEN" \
  -d '{
    "model": "local-coder",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 128
  }'

See The LiteLLM Gateway for the full routing configuration, remote providers, and secret management.

VI Troubleshooting#

GPU out of memory#

ValueError: NCCL or CUDA out of memory

vLLM tries to use 90% of GPU VRAM by default. If you are running multiple models or other GPU workloads, reduce gpu_memory_utilization:

build_args:
  gpu_memory_utilization: 0.5   # use only 50% of VRAM

Or switch to a smaller Size bucket model, enable quantization, or use tensor parallelism across multiple GPUs.

Model not found in catalogue#

KeyError: "Model 'my_model' is not in the catalogue optimizers.yaml."

The model name you passed must match a myname in the catalogue YAML. Check spelling and verify with:

uv run python -c "
from mymodel.catalogue import Registry, OPTIMIZERS
r = Registry(OPTIMIZERS)
print(sorted(r.models))
"

If the model genuinely is missing, add it with Registry.add() (see Adding a new model).

Port conflicts#

If the default ports (5000 for management, 5001+ for models) are already in use, specify alternatives:

uv run python -m mymodel gemma_md --port 9000
# Management on 9000, model server on 9001

Or set the NUCLEUS_PORT environment variable:

NUCLEUS_PORT=9000 uv run python -m mymodel gemma_md

vLLM startup failures#

vllm not on PATH:

RuntimeError: `vllm` is not on PATH; install the serving extra to launch models.

Install vLLM in your environment (uv sync should pull it in, or install it manually: pip install vllm).

Model download fails:

vLLM fetches models from HuggingFace when local weights are not found. Ensure HF_TOKEN is set if the model is gated, and that you have network access to huggingface.co.

Health check never passes:

vLLM can take 30–60 seconds to load weights and initialize the KV cache. Check the vLLM subprocess logs for errors. The management API’s /health endpoint returns immediately; the /models endpoint reports per-model health by polling http://<host>:<port>/health.

Management API returns 401 in production mode:

In production mode (NUCLEUS_MODE=prd), the management API requires a valid Keycloak-issued JWT. Switch to dev mode for local debugging: NUCLEUS_MODE=dev or pass --dev_mode.