# Python API

This page documents the programmatic surface of the `mymodel` package.
It covers the server entry points, the management HTTP API contract, the configuration knobs, and the error behavior.

The `mymodel` package contains no inference endpoints of its own.
Chat completions, embeddings, and streaming are served by the vLLM subprocesses that the launcher starts.
See [Serving with vLLM](serving) for the operational guide and [Model Catalogue](catalogue) for the catalogue class reference.

## `I` Module Layout

| Module              | Public symbols                                   | Purpose                                            |
| ------------------- | ------------------------------------------------ | -------------------------------------------------- |
| `mymodel.serve`     | `Semioserve`, `VLLMServer`, `OpsStance`, `main`  | Catalogue-driven launcher and management API       |
| `mymodel.web`       | `AuthMiddleware`, `SSLConfig`                    | JWT authentication and TLS configuration           |
| `mymodel.common`    | `MY_LOCAL`, `NUCLEUS_HOST`, `NUCLEUS_PORT`, ...  | Shared constants resolved from the environment     |
| `mymodel.catalogue` | `Registry`, `ModelCard`, `Verb`, `Modality`, ... | The model catalogue ([Model Catalogue](catalogue)) |

`mymodel/__init__.py` is empty, so import symbols from their own modules, for example `from mymodel.serve import Semioserve`.

## `II` Starting the Server

### From the command line

The only entry point is the module runner.
`mymodel/__main__.py` delegates to `mymodel.serve.main()`, and `pyproject.toml` defines no console script.

```sh
uv run python -m mymodel <model> [<model> ...] [options]
```

Each model spec is a catalogue name, optionally prefixed with its verb as `VERB:name`.
Only the part after the first colon is used for the lookup, so `CHAT:gemma_md` and `gemma_md` resolve to the same card.
A name that is not in the catalogue raises a `KeyError` at startup, before any server listens.

| Argument      | Default                                  | Description                                          |
| ------------- | ---------------------------------------- | ---------------------------------------------------- |
| `models`      | *(required, one or more)*                | Catalogue model names to serve, one vLLM server each |
| `--host`      | `$NUCLEUS_HOST` or `localhost`           | Host for the management API and the vLLM servers     |
| `--port`      | `$NUCLEUS_PORT` or `5000`                | Management API port                                  |
| `--dev_mode`  | `$NUCLEUS_MODE` starts with `dev`        | Development mode (disables JWT auth)                 |
| `--directory` | `$MY_LOCAL`                              | Local root directory                                 |
| `--catalogue` | `mymodel/catalogue/data/optimizers.yaml` | Catalogue YAML file                                  |

Each model gets its own port.
The first model listens on `--port` + 1, the second on `--port` + 2, and so on.

### From Python

The same launcher is a class, so embedding it in another program is direct:

```python
import asyncio
from mymodel.serve import Semioserve

asyncio.run(Semioserve(models=['gemma_md']).run())
```

`Semioserve` is a singleton.
Its `__new__` returns the existing instance on repeated construction, because the Quart routes read the instance as a singleton.
The constructor accepts the same knobs as the CLI: `models`, `host`, `port`, `dev_mode`, `directory`, and `catalogue`.

### What happens at startup

1. Validate the root directory layout.
   The directories `models`, `logs`, `metrics`, and `data` must already exist under `--directory`, or startup raises an `AssertionError`.
2. Load the catalogue YAML file into a `Registry`.
3. Resolve each requested model spec to a `ModelCard`, raising `KeyError` for unknown names.
4. Create one `VLLMServer` per model, with ports assigned in request order.
5. Configure TLS for the management API if a certificate pair exists under `<directory>/ssl`.
6. Launch each `vllm serve` subprocess, then serve the management API until SIGTERM arrives.
   SIGTERM stops every vLLM subprocess before the process exits.

`VLLMServer.start()` does not block on readiness.
A large model can take tens of seconds to load.
`GET /models` reports the live health of each server in the meantime.

## `III` The Management HTTP API

The management API is a Quart application bound to `--host`:`--port`.
It exposes two GET endpoints.
Quart serializes the returned dicts as JSON.

| Endpoint  | Method | Description                                         |
| --------- | ------ | --------------------------------------------------- |
| `/health` | GET    | Liveness of the management API itself               |
| `/models` | GET    | Active servers with live health, plus the catalogue |

### GET /health

Returns a static payload.
It does not probe the vLLM servers.

```json
{"status": "ok", "models": 1}
```

The `models` field counts the configured vLLM servers, healthy or not.

### GET /models

Polls each vLLM server's `/health` endpoint and reports the result per model.

```json
{
  "active": {
    "gemma_md": {
      "verb": "chat",
      "url": "http://localhost:5001/v1",
      "healthy": true
    }
  },
  "catalogue": ["gemma_md", "..."]
}
```

- `active` maps each served model name to its verb, its OpenAI-compatible base URL, and a live health boolean.
- `catalogue` lists every model name in the loaded catalogue, sorted, whether served or not.

A vLLM server that is still loading reports `"healthy": false` until its `/health` endpoint answers with status 200.

### Error shapes

The management API defines a small error surface:

- **401 Unauthorized.** In production mode, a missing, malformed, expired, or mis-issued token gets a `401` with a `text/plain` body of `Unauthorized`.
- **Startup errors.** An unknown model name raises `KeyError` and a missing root directory raises `AssertionError`.
  Both abort the process before the API listens, so they never surface as HTTP responses.
- **vLLM failures.** Errors from the model servers themselves appear in the subprocess logs, not in the management API.
  The only signal the API gives is `"healthy": false` from `GET /models`.

### Streaming

The management API has no streaming endpoints.
Streaming inference, such as server-sent events from `/v1/chat/completions` with `"stream": true`, is served by the vLLM subprocesses directly.

## `IV` Configuration

### Environment variables

| Variable             | Default                    | Effect                                                                  |
| -------------------- | -------------------------- | ----------------------------------------------------------------------- |
| `NUCLEUS_MODE`       | `dev`                      | Selects the `OpsStance`. Anything not starting with `dev` is production |
| `NUCLEUS_HOST`       | `localhost`                | Default for `--host`                                                    |
| `NUCLEUS_PORT`       | `5000`                     | Default for `--port`, parsed as an integer                              |
| `MY_LOCAL`           | *(required)*               | Default for `--directory`. Validated as an existing directory at import |
| `LOCAL_VLLM_API_KEY` | *(empty)*                  | When set, passed to every vLLM server as `--api-key`                    |
| `KEYCLOAK_SECRET`    | *(required in production)* | HMAC secret for JWT verification                                        |
| `KEYCLOAK_ISSUER`    | *(required in production)* | Expected `iss` claim of accepted tokens                                 |

CLI arguments take precedence over the environment defaults they mirror.

### Directory layout

The root directory must contain `models`, `logs`, `metrics`, and `data` before startup.
Local model weights live under `models`.
A card whose weights exist on disk is served from that path, otherwise vLLM fetches the HuggingFace id.

### TLS

If both `<directory>/ssl/server.crt` and `<directory>/ssl/server.key` exist, the management API is served over TLS.
In development mode an optional `<directory>/ssl/ca.crt` is also loaded into the server configuration.
Without a certificate pair, the API falls back to plain HTTP.
See [Auth & TLS](auth) for certificate setup.

## `V` Authentication

Authentication is posture-dependent, controlled by `OpsStance`:

- **Development** (`--dev_mode`, the default while `NUCLEUS_MODE` starts with `dev`).
  The management API serves without authentication.
- **Production** (`NUCLEUS_MODE` set to anything else, without `--dev_mode`).
  `AuthMiddleware` wraps the ASGI application and verifies a Keycloak-issued JWT on every connection.

Production clients pass the token as an `Authorization: Bearer <token>` header on HTTP requests, or as an `?auth=<token>` query parameter on websocket connections.
Tokens are HMAC-signed with `HS512` and verified against `KEYCLOAK_SECRET`.
The `iss` claim must equal `KEYCLOAK_ISSUER`, the `aud` claim must be `nexus`, and the token must be unexpired.
Failures return the 401 shape described above.

Full key issuance and hardening steps live in [Auth & TLS](auth).

## `VI` Examples

Query the management API with curl:

```sh
curl http://localhost:5000/health
curl http://localhost:5000/models
```

In production mode, add the token header:

```sh
curl -H "Authorization: Bearer $TOKEN" http://localhost:5000/models
```

Use the package's existing `aiohttp` dependency to make the same calls from Python:

```python
import asyncio
import aiohttp

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('http://localhost:5000/models') as resp:
            status = await resp.json()
    for name, server in status['active'].items():
        print(f"{name}: {server['url']} healthy={server['healthy']}")

asyncio.run(main())
```

## `VII` See also

- [Serving with vLLM](serving): the operational guide for catalogue concepts, GPU sizing, and troubleshooting.
- [Model Catalogue](catalogue): the `Registry` and `ModelCard` API for listing, adding, and enriching models.
- [The LiteLLM Gateway](gateway.md): the proxy that fronts the vLLM servers with one OpenAI-compatible endpoint.
- [Auth & TLS](auth): JWT issuance, TLS setup, and production hardening.