Python API

This page documents the programmatic surface of the mymodel package. It covers the server entry points, the management HTTP API contract, the configuration knobs, and the error behavior.

The mymodel package contains no inference endpoints of its own. Chat completions, embeddings, and streaming are served by the vLLM subprocesses that the launcher starts. See Serving with vLLM for the operational guide and Model Catalogue for the catalogue class reference.

I Module Layout

Module	Public symbols	Purpose
mymodel.serve	Semioserve, VLLMServer, OpsStance, main	Catalogue-driven launcher and management API
mymodel.web	AuthMiddleware, SSLConfig	JWT authentication and TLS configuration
mymodel.common	MY_LOCAL, NUCLEUS_HOST, NUCLEUS_PORT, …	Shared constants resolved from the environment
mymodel.catalogue	Registry, ModelCard, Verb, Modality, …	The model catalogue (Model Catalogue)

mymodel/__init__.py is empty, so import symbols from their own modules, for example from mymodel.serve import Semioserve.

II Starting the Server

From the command line

The only entry point is the module runner. mymodel/__main__.py delegates to mymodel.serve.main(), and pyproject.toml defines no console script.

uv run python -m mymodel <model> [<model> ...] [options]

Each model spec is a catalogue name, optionally prefixed with its verb as VERB:name. Only the part after the first colon is used for the lookup, so CHAT:gemma_md and gemma_md resolve to the same card. A name that is not in the catalogue raises a KeyError at startup, before any server listens.

Argument	Default	Description
models	(required, one or more)	Catalogue model names to serve, one vLLM server each
--host	$NUCLEUS_HOST or localhost	Host for the management API and the vLLM servers
--port	$NUCLEUS_PORT or 5000	Management API port
--dev_mode	$NUCLEUS_MODE starts with dev	Development mode (disables JWT auth)
--directory	$MY_LOCAL	Local root directory
--catalogue	mymodel/catalogue/data/optimizers.yaml	Catalogue YAML file

Each model gets its own port. The first model listens on --port + 1, the second on --port + 2, and so on.

From Python

The same launcher is a class, so embedding it in another program is direct:

import asyncio
from mymodel.serve import Semioserve

asyncio.run(Semioserve(models=['gemma_md']).run())

Semioserve is a singleton. Its __new__ returns the existing instance on repeated construction, because the Quart routes read the instance as a singleton. The constructor accepts the same knobs as the CLI: models, host, port, dev_mode, directory, and catalogue.

What happens at startup

1. Validate the root directory layout. The directories models, logs, metrics, and data must already exist under --directory, or startup raises an AssertionError.

2. Load the catalogue YAML file into a Registry.

3. Resolve each requested model spec to a ModelCard, raising KeyError for unknown names.

4. Create one VLLMServer per model, with ports assigned in request order.

5. Configure TLS for the management API if a certificate pair exists under <directory>/ssl.

6. Launch each vllm serve subprocess, then serve the management API until SIGTERM arrives. SIGTERM stops every vLLM subprocess before the process exits.

VLLMServer.start() does not block on readiness. A large model can take tens of seconds to load. GET /models reports the live health of each server in the meantime.

III The Management HTTP API

The management API is a Quart application bound to --host:--port. It exposes two GET endpoints. Quart serializes the returned dicts as JSON.

Endpoint	Method	Description
/health	GET	Liveness of the management API itself
/models	GET	Active servers with live health, plus the catalogue

GET /health

Returns a static payload. It does not probe the vLLM servers.

{"status": "ok", "models": 1}

The models field counts the configured vLLM servers, healthy or not.

GET /models

Polls each vLLM server’s /health endpoint and reports the result per model.

{
  "active": {
    "gemma_md": {
      "verb": "chat",
      "url": "http://localhost:5001/v1",
      "healthy": true
    }
  },
  "catalogue": ["gemma_md", "..."]
}

• active maps each served model name to its verb, its OpenAI-compatible base URL, and a live health boolean.

• catalogue lists every model name in the loaded catalogue, sorted, whether served or not.

A vLLM server that is still loading reports "healthy": false until its /health endpoint answers with status 200.

Error shapes

The management API defines a small error surface:

• 401 Unauthorized. In production mode, a missing, malformed, expired, or mis-issued token gets a 401 with a text/plain body of Unauthorized.

• Startup errors. An unknown model name raises KeyError and a missing root directory raises AssertionError. Both abort the process before the API listens, so they never surface as HTTP responses.

• vLLM failures. Errors from the model servers themselves appear in the subprocess logs, not in the management API. The only signal the API gives is "healthy": false from GET /models.

Streaming

The management API has no streaming endpoints. Streaming inference, such as server-sent events from /v1/chat/completions with "stream": true, is served by the vLLM subprocesses directly.

IV Configuration

Environment variables

Variable	Default	Effect
NUCLEUS_MODE	dev	Selects the OpsStance. Anything not starting with dev is production
NUCLEUS_HOST	localhost	Default for --host
NUCLEUS_PORT	5000	Default for --port, parsed as an integer
MY_LOCAL	(required)	Default for --directory. Validated as an existing directory at import
LOCAL_VLLM_API_KEY	(empty)	When set, passed to every vLLM server as --api-key
KEYCLOAK_SECRET	(required in production)	HMAC secret for JWT verification
KEYCLOAK_ISSUER	(required in production)	Expected iss claim of accepted tokens

CLI arguments take precedence over the environment defaults they mirror.

Directory layout

The root directory must contain models, logs, metrics, and data before startup. Local model weights live under models. A card whose weights exist on disk is served from that path, otherwise vLLM fetches the HuggingFace id.

TLS

If both <directory>/ssl/server.crt and <directory>/ssl/server.key exist, the management API is served over TLS. In development mode an optional <directory>/ssl/ca.crt is also loaded into the server configuration. Without a certificate pair, the API falls back to plain HTTP. See Auth & TLS for certificate setup.

V Authentication

Authentication is posture-dependent, controlled by OpsStance:

• Development (--dev_mode, the default while NUCLEUS_MODE starts with dev). The management API serves without authentication.

• Production (NUCLEUS_MODE set to anything else, without --dev_mode). AuthMiddleware wraps the ASGI application and verifies a Keycloak-issued JWT on every connection.

Production clients pass the token as an Authorization: Bearer <token> header on HTTP requests, or as an ?auth=<token> query parameter on websocket connections. Tokens are HMAC-signed with HS512 and verified against KEYCLOAK_SECRET. The iss claim must equal KEYCLOAK_ISSUER, the aud claim must be nexus, and the token must be unexpired. Failures return the 401 shape described above.

Full key issuance and hardening steps live in Auth & TLS.

VI Examples

Query the management API with curl:

curl http://localhost:5000/health
curl http://localhost:5000/models

In production mode, add the token header:

curl -H "Authorization: Bearer $TOKEN" http://localhost:5000/models

Use the package’s existing aiohttp dependency to make the same calls from Python:

import asyncio
import aiohttp

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('http://localhost:5000/models') as resp:
            status = await resp.json()
    for name, server in status['active'].items():
        print(f"{name}: {server['url']} healthy={server['healthy']}")

asyncio.run(main())

VII See also

• Serving with vLLM: the operational guide for catalogue concepts, GPU sizing, and troubleshooting.

• Model Catalogue: the Registry and ModelCard API for listing, adding, and enriching models.

• The LiteLLM Gateway: the proxy that fronts the vLLM servers with one OpenAI-compatible endpoint.

• Auth & TLS: JWT issuance, TLS setup, and production hardening.