Auth & TLS#

I Overview#

myModel protects two separate surfaces with two different mechanisms:

  • The management API (Quart, default port 5000) verifies a Keycloak-issued JWT bearer token. This layer is active in production mode only.

  • The vLLM inference servers (ports 5001 and up) use a static API key passed to vllm serve. This layer is active whenever $LOCAL_VLLM_API_KEY is set.

TLS is a third, independent layer. The management API serves HTTPS when a certificate pair exists under <directory>/ssl, and plain HTTP otherwise.

Development mode is the default. It disables the JWT layer entirely, so the management API accepts unauthenticated requests. Production mode wraps the whole ASGI app in AuthMiddleware, so every management route requires a valid token, including /health.

II Running in production mode#

The OpsStance enum in mymodel/serve.py controls the deployment posture. The launcher selects it from two inputs:

  • The NUCLEUS_MODE environment variable (default dev, from mymodel/common.py).

  • The --dev_mode CLI flag, which forces development mode regardless of NUCLEUS_MODE.

Production mode is active when NUCLEUS_MODE does not start with dev and --dev_mode is not passed:

NUCLEUS_MODE=prd uv run python -m mymodel gemma_md

The JWT layer reads its configuration from the process environment:

Variable

Purpose

Read by

KEYCLOAK_SECRET

HMAC key used to verify token signatures

AuthMiddleware

KEYCLOAK_ISSUER

Expected value of the token’s iss claim

AuthMiddleware

NUCLEUS_MODE

Deployment posture (dev prefix means development)

mymodel/common.py

The my package’s env interface reads these variables. At import time, the interface also loads the nearest .env file via python-dotenv. An unset variable reads as the empty string.

Warning

Never run production mode with KEYCLOAK_SECRET unset. The verifier would then use the empty string as the HMAC key, and a token signed with an empty key would pass verification.

III The JWT middleware#

AuthMiddleware (mymodel/web.py) is an ASGI middleware that verifies a Keycloak-issued JWT on every connection. It is vendored from the fleet’s ai monorepo, with the logfire dependency replaced by stdlib logging. Semioserve.build_app() attaches it only in production mode.

Token extraction#

The middleware extracts the token differently per connection type:

  • HTTP: from the Authorization header. The header must match Bearer <token> exactly (case-sensitive, a single space), because the match is a fullmatch against the pattern Bearer (?P<token>\S+).

  • Websocket: from the auth query parameter, as in ws://host:port/path?auth=<token>.

The management API currently defines only HTTP routes (/models and /health), so the websocket path is dormant.

Verification steps#

verify_token() applies these checks in order:

  1. A token is present at all.

  2. jwt.decode() verifies the signature with KEYCLOAK_SECRET and the HS512 algorithm (HMAC-SHA-512). The algorithm is a constructor parameter, but Semioserve always uses the default.

  3. The iss claim equals $KEYCLOAK_ISSUER.

  4. The exp claim is not in the past. PyJWT already rejects expired tokens during decode, so this explicit check is a redundant second pass.

  5. The aud claim equals the hardcoded string nexus.

A token that passes every check lets the request through to the Quart app. The middleware logs the authenticated client address at INFO level.

IV Failure modes#

Rejection looks the same to the client for every failure cause. The specific reason appears only in the server logs, at ERROR level.

Cause

HTTP response

Websocket response

Missing or malformed Authorization header

401, text/plain body Unauthorized

Close code 4000, reason Authentication failed

Missing auth query parameter

(n/a)

Close code 4000, reason Authentication failed

Bad signature or malformed JWT

401, body Unauthorized

Close code 4000

Expired token

401, body Unauthorized

Close code 4000

Wrong issuer (iss) or audience (aud)

401, body Unauthorized

Close code 4000

Two edge cases deserve honest treatment:

  • Missing claims crash instead of rejecting cleanly. The verifier indexes payload['iss'], payload['exp'], and payload['aud'] directly. A well-formed token that lacks one of these claims raises an uncaught KeyError. This surfaces as a server error rather than a clean 401.

  • The manual expiry check is unreachable for standard tokens. PyJWT validates exp during jwt.decode() and raises first. The explicit check in verify_token() only matters for tokens where PyJWT skipped expiry, which the current call never produces.

V TLS for the management API#

The launcher looks for a certificate pair under <directory>/ssl, where <directory> is the --directory CLI argument (default: the $MY_LOCAL path):

  • server.crt: the server certificate.

  • server.key: the server private key.

  • ca.crt: an optional CA certificate, loaded only in development mode.

When both server.crt and server.key exist, SSLConfig (mymodel/web.py) builds a Hypercorn config from them. The SSL context enforces TLS 1.2 as the minimum version. In production mode the CA certificate is not applied, since the JWT middleware handles authentication.

When the pair is absent, the management API falls back to plain HTTP. This is the typical setup for local development behind loopback.

VI The inference-port API key#

The JWT layer protects only the management API. The vLLM servers that run the actual inference have their own, separate mechanism. When $LOCAL_VLLM_API_KEY is set, Semioserve passes it to each vllm serve subprocess as --api-key. vLLM then requires that key as a bearer token on its OpenAI-compatible endpoints. When the variable is unset, the flag is omitted and the inference ports accept unauthenticated requests.

This key is also how the fleet’s LiteLLM gateway reaches a local vLLM server. The gateway lives in its own repository (~/my/apps/gate/gateway, localhost:4000) and sends $LOCAL_VLLM_API_KEY as the upstream credential for its local routes. Three distinct credentials therefore appear in a full deployment:

  • LITELLM_TOKEN: client to gateway (managed by the gateway repo, not this one).

  • LOCAL_VLLM_API_KEY: gateway to vLLM inference servers (this repo, optional).

  • Keycloak JWTs: clients to the management API (this repo, production mode only).

VII Current status and limitations#

  • AuthMiddleware and SSLConfig have no test coverage. No file under tests/ exercises the middleware, token verification, or the TLS configuration.

  • The websocket branch of the middleware is dormant, because the management API has no websocket routes.

  • The audience value nexus is hardcoded. Operators must configure their Keycloak client to issue tokens with that audience.

For the management API endpoints themselves, see Python API. For the launcher that wires all of this together, see Serving with vLLM.