Quickstart

I Introduction

myModel is a collection of instruments for deploying LLMs locally. It combines:

• a catalogue-driven vLLM launcher (serve.py);

• a llama.cpp-based ternary model launcher (ternary.py);

• local vLLM and Ternary serving profiles (vllm/, ternary/);

• container deployment artifacts (pods/);

• an optional QLoRA tuning pipeline (pods/tune/).

Together, these tools form one cohesive group, but they are not a single product.

The core workflow follows this sequence. First, you pick models from a YAML catalogue. Next, you launch them with vLLM. Finally, you front them with a LiteLLM proxy that exposes a single OpenAI-compatible endpoint.

Clients (Droid, Claude Code, Cursor, curl) point at the gateway and select a model by alias. The gateway itself lives in its own repository, ~/my/apps/gate. See The LiteLLM Gateway for the split and what stays here.

II Prerequisites

Requirement	Version	Notes
Python	3.13+	Required for the mymodel package
NVIDIA GPU	CUDA-capable	An RTX 4090 is used as the example throughout
podman or docker	latest	For container deployments in pods/
Task	taskfile.dev	The build system (task commands)
uv	latest	Python package management (uv sync)

Verify your environment:

python3 --version   # 3.13+
nvidia-smi          # confirm GPU is visible
task --version      # confirm Task is installed
uv --version        # confirm uv is installed

III Installation

Clone the repository and sync the Python environment:

git clone https://gitlab.com/doering-ai/apps/model.git
cd model
uv sync

uv sync installs all runtime dependencies (vLLM, transformers, pydantic, etc.) and the development tooling (ruff, pyrefly, pytest).

Verify the mymodel package is importable:

uv run python -c "import mymodel; print('ok')"

IV Your first model deployment

The catalogue concept

myModel does not hard-code model paths. Instead, it reads a catalogue — a YAML file that maps human-readable names to HuggingFace model IDs, build arguments, and metadata. Two seed catalogues ship with the repo:

• mymodel/catalogue/data/optimizers.yaml — generative/transformer models (CHAT, CODE, DRAW, …)

• mymodel/catalogue/data/classifiers.yaml — reductive models (EMBED, SCORE, SEGMENT, …)

Each entry is a ModelCard placed in the ontology by its Verb (the input→output transformation) and Modality pair. For example, a CHAT verb model takes text/image in and produces text out.

List available models

The catalogue is a plain YAML file, so you can browse it directly:

# See all top-level verb sections (CHAT, CODE, DRAW, EMBED, etc.)
grep -E '^[A-Z]' mymodel/catalogue/data/optimizers.yaml

Or use the Python API to enumerate model names:

uv run python -c "
from mymodel.catalogue import Registry, OPTIMIZERS
r = Registry(OPTIMIZERS)
for name in sorted(r.models):
    print(name)
"

Launch a model with vLLM

The serve module launches one or more catalogued models as vLLM-native OpenAI servers:

uv run python -m mymodel <model_name>

For example, to serve a chat model:

uv run python -m mymodel gemma_md

This starts a vLLM server on a port (default 5001 for the first model) and a thin management API on port 5000. The vLLM server exposes the standard OpenAI API at http://localhost:<port>/v1.

Test it with curl

Once the server is healthy, send a chat completion request:

curl http://localhost:5001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma_md",
    "messages": [{"role": "user", "content": "Hello, what model are you?"}],
    "max_tokens": 128
  }'

Check the management API for server health and catalogue state:

curl http://localhost:5000/health
curl http://localhost:5000/models

V Setting up the gateway

What the gateway does

Important

The gateway lives in ~/my/apps/gate (gate/gateway/), not here. See The LiteLLM Gateway for the split. This section only covers connecting a client to the local-coder route this repo serves. Full gateway setup (secrets, task up, harness wiring, telemetry) is in that repo’s gateway/README.md.

The LiteLLM gateway is a proxy that exposes a single OpenAI-compatible endpoint on http://localhost:4000/v1. Clients point at this one URL and select a model by alias. The gateway holds all upstream credentials, so you configure secrets once.

client (Droid/Claude Code/curl) ──Bearer LITELLM_TOKEN──▶ gateway :4000 ─┬─▶ local vLLM :8000
                                                                          ├─▶ OpenRouter (remote)
                                                                          └─▶ Ternary Bonsai :8001 (session-only)

Minimal config to route to the local vLLM

The gateway’s routing table is ~/my/apps/gate/gateway/config.yaml. A minimal entry to route to a local vLLM server looks like:

model_list:
  - model_name: local-coder
    litellm_params:
      model: openai/local-coder          # must match vLLM --served-model-name
      api_base: http://localhost:8000/v1
      api_key: os.environ/LOCAL_VLLM_API_KEY

Start the gateway

cd ~/my/apps/gate/gateway
task up

Verify it’s live:

curl http://localhost:4000/health/liveliness

Connect a client

With the gateway running, any OpenAI-compatible client can point at it:

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_TOKEN" \
  -d '{
    "model": "local-coder",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 128
  }'

VI Connecting a harness

The gateway speaks both OpenAI Chat Completions (/v1/chat/completions) and Anthropic Messages (/v1/messages). LiteLLM automatically translates the Anthropic surface, so most tools work without extra configuration. Full instructions for every harness (Claude Code, Droid, opencode, Kilo Code, Dirac, Warp) live in ~/my/apps/gate/gateway/README.md. The essentials are below.

Claude Code

Claude Code speaks the Anthropic Messages API, which LiteLLM auto-translates:

export ANTHROPIC_BASE_URL="http://localhost:4000"
export ANTHROPIC_AUTH_TOKEN="$LITELLM_TOKEN"
export ANTHROPIC_SMALL_FAST_MODEL="glm-5.2"
claude --model glm-5.2

The claude-glm shell function (~/my/apps/gate/gateway/claude-glm.sh) handles the secret injection and TTY passthrough for interactive sessions.

Factory Droid

Droid uses the OpenAI-compatible surface:

# Source the droid-glm helper for interactive use:
source ~/my/apps/gate/gateway/droid-glm.sh
droid-glm

Simple curl client

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_TOKEN" \
  -d '{"model": "glm-5.2", "messages": [{"role": "user", "content": "Hello!"}]}' \
  --no-buffer

VII Next steps

Guide	What you will learn
Serving with vLLM	The catalogue system, build arguments, and multi-model serving
The LiteLLM Gateway	Full routing config, remote providers, and secret management
Container Deployments	podman containers for chat, reading, tuning, and development
Model Catalogue	How to add models, the verb/modality ontology, and YAML schema
Fine-tuning	QLoRA pipeline configuration and the smoke test
Ternary Bonsai	The guarded llama.cpp advisor session
Python API	Direct programmatic access to the catalogue and serving APIs
Auth & TLS	JWT auth, SSL/TLS configuration, and production hardening