# Quickstart

## `I` Introduction

myModel is a collection of instruments for deploying LLMs locally.
It combines:

- a catalogue-driven vLLM launcher (`serve.py`);
- a llama.cpp-based ternary model launcher (`ternary.py`);
- local vLLM and Ternary serving profiles (`vllm/`, `ternary/`);
- container deployment artifacts (`pods/`);
- an optional QLoRA tuning pipeline (`pods/tune/`).

Together, these tools form one cohesive group, but they are not a single product.

The core workflow follows this sequence.
First, you pick models from a YAML catalogue.
Next, you launch them with vLLM.
Finally, you front them with a LiteLLM proxy that exposes a single OpenAI-compatible endpoint.

Clients (Droid, Claude Code, Cursor, curl) point at the gateway and select a model by alias.
The gateway itself lives in its own repository, `~/my/apps/gate`.
See [The LiteLLM Gateway](gateway.md) for the split and what stays here.

## `II` Prerequisites

| Requirement      | Version                              | Notes                                         |
| ---------------- | ------------------------------------ | --------------------------------------------- |
| Python           | 3.13+                                | Required for the `mymodel` package            |
| NVIDIA GPU       | CUDA-capable                         | An RTX 4090 is used as the example throughout |
| podman or docker | latest                               | For container deployments in `pods/`          |
| Task             | [taskfile.dev](https://taskfile.dev) | The build system (`task` commands)            |
| uv               | latest                               | Python package management (`uv sync`)         |

Verify your environment:

```sh
python3 --version   # 3.13+
nvidia-smi          # confirm GPU is visible
task --version      # confirm Task is installed
uv --version        # confirm uv is installed
```

## `III` Installation

Clone the repository and sync the Python environment:

```sh
git clone https://gitlab.com/doering-ai/apps/model.git
cd model
uv sync
```

`uv sync` installs all runtime dependencies (vLLM, transformers, pydantic, etc.) and the development tooling (ruff, pyrefly, pytest).

Verify the `mymodel` package is importable:

```sh
uv run python -c "import mymodel; print('ok')"
```

## `IV` Your first model deployment

### The catalogue concept

myModel does not hard-code model paths.
Instead, it reads a **catalogue** — a YAML file that maps human-readable names to HuggingFace model IDs, build arguments, and metadata.
Two seed catalogues ship with the repo:

- `mymodel/catalogue/data/optimizers.yaml` — generative/transformer models (CHAT, CODE, DRAW, ...)
- `mymodel/catalogue/data/classifiers.yaml` — reductive models (EMBED, SCORE, SEGMENT, ...)

Each entry is a `ModelCard` placed in the ontology by its `Verb` (the input→output transformation) and `Modality` pair.
For example, a `CHAT` verb model takes text/image in and produces text out.

### List available models

The catalogue is a plain YAML file, so you can browse it directly:

```sh
# See all top-level verb sections (CHAT, CODE, DRAW, EMBED, etc.)
grep -E '^[A-Z]' mymodel/catalogue/data/optimizers.yaml
```

Or use the Python API to enumerate model names:

```sh
uv run python -c "
from mymodel.catalogue import Registry, OPTIMIZERS
r = Registry(OPTIMIZERS)
for name in sorted(r.models):
    print(name)
"
```

### Launch a model with vLLM

The `serve` module launches one or more catalogued models as vLLM-native OpenAI servers:

```sh
uv run python -m mymodel <model_name>
```

For example, to serve a chat model:

```sh
uv run python -m mymodel gemma_md
```

This starts a vLLM server on a port (default `5001` for the first model) and a thin management API on port `5000`.
The vLLM server exposes the standard OpenAI API at `http://localhost:<port>/v1`.

### Test it with curl

Once the server is healthy, send a chat completion request:

```sh
curl http://localhost:5001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma_md",
    "messages": [{"role": "user", "content": "Hello, what model are you?"}],
    "max_tokens": 128
  }'
```

Check the management API for server health and catalogue state:

```sh
curl http://localhost:5000/health
curl http://localhost:5000/models
```

## `V` Setting up the gateway

### What the gateway does

```{important}
The gateway lives in `~/my/apps/gate` (`gate/gateway/`), not here.
See [The LiteLLM Gateway](gateway.md) for the split.
This section only covers connecting a client to the local-coder route this repo serves.
Full gateway setup (secrets, `task up`, harness wiring, telemetry) is in that repo's `gateway/README.md`.
```

The LiteLLM gateway is a proxy that exposes a single OpenAI-compatible endpoint on `http://localhost:4000/v1`.
Clients point at this one URL and select a model by alias.
The gateway holds all upstream credentials, so you configure secrets once.

```
client (Droid/Claude Code/curl) ──Bearer LITELLM_TOKEN──▶ gateway :4000 ─┬─▶ local vLLM :8000
                                                                          ├─▶ OpenRouter (remote)
                                                                          └─▶ Ternary Bonsai :8001 (session-only)
```

### Minimal config to route to the local vLLM

The gateway's routing table is `~/my/apps/gate/gateway/config.yaml`.
A minimal entry to route to a local vLLM server looks like:

```yaml
model_list:
  - model_name: local-coder
    litellm_params:
      model: openai/local-coder          # must match vLLM --served-model-name
      api_base: http://localhost:8000/v1
      api_key: os.environ/LOCAL_VLLM_API_KEY
```

### Start the gateway

```sh
cd ~/my/apps/gate/gateway
task up
```

Verify it's live:

```sh
curl http://localhost:4000/health/liveliness
```

### Connect a client

With the gateway running, any OpenAI-compatible client can point at it:

```sh
curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_TOKEN" \
  -d '{
    "model": "local-coder",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 128
  }'
```

## `VI` Connecting a harness

The gateway speaks both OpenAI Chat Completions (`/v1/chat/completions`) and Anthropic Messages (`/v1/messages`).
LiteLLM automatically translates the Anthropic surface, so most tools work without extra configuration.
Full instructions for every harness (Claude Code, Droid, opencode, Kilo Code, Dirac, Warp) live in `~/my/apps/gate/gateway/README.md`.
The essentials are below.

### Claude Code

Claude Code speaks the Anthropic Messages API, which LiteLLM auto-translates:

```sh
export ANTHROPIC_BASE_URL="http://localhost:4000"
export ANTHROPIC_AUTH_TOKEN="$LITELLM_TOKEN"
export ANTHROPIC_SMALL_FAST_MODEL="glm-5.2"
claude --model glm-5.2
```

The `claude-glm` shell function (`~/my/apps/gate/gateway/claude-glm.sh`) handles the secret injection and TTY passthrough for interactive sessions.

### Factory Droid

Droid uses the OpenAI-compatible surface:

```sh
# Source the droid-glm helper for interactive use:
source ~/my/apps/gate/gateway/droid-glm.sh
droid-glm
```

### Simple curl client

```sh
curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_TOKEN" \
  -d '{"model": "glm-5.2", "messages": [{"role": "user", "content": "Hello!"}]}' \
  --no-buffer
```

## `VII` Next steps

| Guide                             | What you will learn                                            |
| --------------------------------- | -------------------------------------------------------------- |
| [Serving with vLLM](serving)      | The catalogue system, build arguments, and multi-model serving |
| [The LiteLLM Gateway](gateway.md) | Full routing config, remote providers, and secret management   |
| [Container Deployments](pods)     | podman containers for chat, reading, tuning, and development   |
| [Model Catalogue](catalogue)      | How to add models, the verb/modality ontology, and YAML schema |
| [Fine-tuning](tuning)             | QLoRA pipeline configuration and the smoke test                |
| [Ternary Bonsai](ternary)         | The guarded llama.cpp advisor session                          |
| [Python API](api)                 | Direct programmatic access to the catalogue and serving APIs   |
| [Auth & TLS](auth)                | JWT auth, SSL/TLS configuration, and production hardening      |