# Container Deployments

## `I` Overview

Pods are pre-built container deployment artifacts for different model serving scenarios in myModel.
Each pod is a self-contained directory under `pods/` with a `Containerfile`, a compose config, and launch scripts.
They let you deploy models in containers without writing your own Dockerfiles.

If you are new to containers: a **Containerfile** (called a `Dockerfile` in Docker parlance) is a recipe for building a container image, and a **compose file** (`compose.yaml`) declares how to run that image, including GPU access, ports, and volumes. myModel uses `podman` (rootless, daemonless) instead of `docker`, but the file formats are compatible.

```
pods/
├── chat/        # vLLM inference server
├── read/        # Marker PDF/OCR server
├── tune/        # QLoRA training environment
├── dev/         # CUDA development environment
└── README.md    # Container image reference
```

The four pods cover the most common workflows: serving a model for chat, processing documents, fine-tuning, and developing against the CUDA stack.
You can mix and match them or build your own (see [§VII](#vii-building-custom-pods)).

## `II` Container images reference

myModel builds on upstream container images from NVIDIA and vLLM.
The table below is an expanded view of `pods/README.md`, listing the images used or referenced by the pods.

| Image                                                | Variant            | Use case                                          | Notes                                                                            |
| ---------------------------------------------------- | ------------------ | ------------------------------------------------- | -------------------------------------------------------------------------------- |
| `nvcr.io/nvidia/cuda:13.0.2-devel-ubuntu24.04`       | CUDA devel         | Building GPU-dependent Python packages (read pod) | Ubuntu 24.04, CUDA 13.0.2                                                        |
| `nvcr.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04` | cuDNN devel        | Same but with cuDNN pre-installed                 | Required for Marker's torchvision/torch stack                                    |
| `docker.io/vllm/vllm-openai:latest`                  | First-party vLLM   | Running the OpenAI-compatible vLLM server         | [Docs](https://github.com/vllm-project/vllm/blob/main/docs/deployment/docker.md) |
| `docker.io/vllm/vllm-openai:v0.11.2`                 | Pinned first-party | Chat pod default image                            | Pinned for reproducibility                                                       |
| `nvcr.io/nvidia/vllm:25.11-py3`                      | NVIDIA's vLLM      | Alternative vLLM image from NGC                   | Ubuntu 24.04, CUDA 13.0.2, Python 3.12                                           |

**Compatibility notes:**

- **podman**: The pods use `security_opt: [label=disable]` and `devices: [nvidia.com/gpu=0]`,
  which are podman-compatible. NVIDIA CDI must be configured on the host.
- **Platform**: All images target `linux/amd64` unless overridden. The chat pod's `compose.yaml`
  pins `platform: linux/arm64` for the default configuration (adjust to your hardware).

## `III` Chat pod: vLLM inference server

The chat pod runs [vLLM](https://github.com/vllm-project/vllm) as an OpenAI-compatible HTTP server inside a container.
This is the fastest path to serving a model: you provide a HuggingFace model ID and the container handles the rest.

### The Containerfile

`pods/chat/Containerfile` is a thin layer on top of `docker.io/vllm/vllm-openai:v0.11.2`.
The file is mostly commented-out scaffolding showing where you would add custom dependencies:

```dockerfile
FROM docker.io/vllm/vllm-openai:v0.11.2

# Dependency installation example:
# RUN uv pip install --system vllm[audio]
```

Uncomment or add `RUN` lines to install extra Python packages.
For most deployments, the base image alone is sufficient.

### The compose.yaml

```yaml
services:
    chat-server:
        image: docker.io/vllm/vllm-openai:v0.11.2
        platform: linux/arm64
        runtime: nvidia
        ipc: host
        ports:
            - 8000:8000
        environment:
            - HF_TOKEN=${HF_TOKEN}
            - MODEL_ID=${MODEL_ID:-google/gemma-3-12b-it}
        volumes:
            - $HF_HOME:/root/.cache/huggingface
        security_opt:
            - label=disable
        command: ["--model", "${MODEL_ID:-google/gemma-3-12b-it}"]
        deploy:
            resources:
                reservations:
                    devices:
                        - driver: nvidia
                          count: all
                          capabilities: [gpu]
```

| Setting                       | Purpose                                                     |
| ----------------------------- | ----------------------------------------------------------- |
| `runtime: nvidia`             | NVIDIA container runtime for GPU access                     |
| `ipc: host`                   | Share host IPC namespace (vLLM recommendation for perf)     |
| `ports: 8000:8000`            | Expose the OpenAI API on host port 8000                     |
| `HF_TOKEN`                    | HuggingFace access token (for gated models)                 |
| `MODEL_ID`                    | The HuggingFace model ID to serve (defaults to Gemma 3 12B) |
| `volumes`                     | Bind-mount the HF cache so model weights persist            |
| `security_opt: label=disable` | Disable SELinux label enforcement (podman compatibility)    |
| `deploy.resources`            | Reserve all NVIDIA GPUs for the container                   |

### The run script

`pods/chat/run.sh` is a one-liner equivalent to the compose file for ad-hoc use:

```bash
podman run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --env "HF_TOKEN=${HF_TOKEN}" \
    --security-opt label=disable \
    docker.io/vllm/vllm-openai --model google/gemma-3-12b-it
```

### Build and run

```sh
cd pods/chat
podman build -t mymodel-chat .

export HF_TOKEN=hf_xxxxxxxxxxxx
podman compose -f compose.yaml up
# or: bash run.sh
```

### Customize the model

Set `MODEL_ID` before launching to serve a different model:

```sh
MODEL_ID=microsoft/Phi-3-mini-4k-instruct podman compose -f compose.yaml up
```

### Test it

```sh
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "google/gemma-3-12b-it", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 64}'
```

### The reference container file

`pods/chat/vLLM__reference.container` is the full upstream vLLM Dockerfile (22 KB), included for reference.
It shows how the official image is built from scratch (multi-stage CUDA build, wheel compilation, DeepGEMM, EP kernels, FlashInfer).
You do not need to build from this file, but it is an invaluable reference if you need to patch vLLM or add custom kernels.

For more on serving models outside of containers, see [Serving with vLLM](serving).

## `IV` Read pod: OCR/PDF processing

The read pod runs a [Marker](https://github.com/VikParuchuri/marker) server for converting PDF and image files into structured markdown.
Unlike the chat pod, this pod builds a custom image from the NVIDIA CUDA base.

### The marker-server.py

`pods/read/marker-server.py` (11 KB) distributes document processing across multiple GPUs using Python multiprocessing.
Key features:

- **Multi-GPU parallelism**: One `multiprocessing.Pool` per GPU; each worker pins itself via
  `CUDA_VISIBLE_DEVICES`.
- **Configurable LLM backend**: Optionally routes to OpenAI, Claude, Gemini, Vertex, Azure, or
  Ollama for validation and correction.
- **Modal integration**: Designed to run on Modal (serverless GPU) with 4x T4 GPUs, but the core
  logic works locally on any CUDA-capable machine.
- **I/O**: Files in a mounted intake directory (PDF, JPG, PNG, TIFF); output is extracted markdown
  text, images, and metadata per document.

### The Containerfile

`pods/read/Containerfile` is a multi-stage build from the NVIDIA CUDA cuDNN base:

```dockerfile
FROM nvcr.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04 AS base
RUN python3 -m pip install uv

# GPU packages: marker-pdf, torch, torchvision
FROM base as marker-base
RUN $UV_INSTALL marker-pdf pillow pytorch torchvision torchaudio

# Server packages: FastAPI, granian (ASGI server)
FROM marker-base as py-base
RUN $UV_INSTALL fastapi granian uvloop sqlmodel

# Final image
FROM py-base as final-read
COPY ./marker-server.py /usr/local/bin/marker_server
ENTRYPOINT ["marker_server", "--port", "8000"]
```

It uses `uv` for fast package resolution with cache mounts for reproducible, layer-cached builds.

### The compose.yaml

```yaml
services:
    my-reader:
        build: .
        image: mymodel-read:latest
        ports:
            - 8000:8000
        devices:
            - nvidia.com/gpu=0
        security_opt:
            - label=disable
        env_file:
            - ${MY_ENV_FILE}
        volumes:
            - ${MY_DATA}/ocr_intake:/data/intake:ro
            - ${MY_DATA}/ocr_output:/data/output:rw
            - datalab-cache:/root/.cache/datalab/:rw
```

| Setting                     | Purpose                                                      |
| --------------------------- | ------------------------------------------------------------ |
| `build: .`                  | Build from the local `Containerfile` (not a pre-built image) |
| `devices: nvidia.com/gpu=0` | Mount GPU 0 via NVIDIA CDI                                   |
| `volumes`                   | Intake (read-only), output (read-write), cache               |
| `env_file`                  | Load secrets (API keys, model paths) from an env file        |

### Build and run

```sh
cd pods/read
podman build -t mymodel-read:latest .

export MY_ENV_FILE=.env
export MY_DATA=/path/to/your/data
podman compose -f compose.yaml up
```

### Send documents for processing

Place PDFs or images in your intake directory, then call the server:

```sh
curl http://localhost:8000/convert \
  -H "Content-Type: application/json" \
  -d '{"file": "/data/intake/document.pdf", "output_format": "markdown"}'
```

## `V` Tune pod: QLoRA fine-tuning

The tune pod provides a standalone training environment for [QLoRA](https://arxiv.org/abs/2305.14314) fine-tuning of LLMs.
It is a deliberately separate `uv` project because `unsloth`'s tight torch/transformers pins are unsatisfiable when co-resolved with `my-model`'s vLLM dependency under the repo's `exclude-newer` guard.

The tune pod takes teacher-trace JSONL datasets and produces LoRA adapters that can be hot-loaded into a running vLLM server.
It uses [Unsloth](https://github.com/unslothai/unsloth) for memory-efficient 4-bit quantized training with gradient checkpointing.

### Separate uv project structure

```
pods/tune/
├── pyproject.toml     # Standalone uv project (my-model-tune)
├── README.md           # Usage and adapter serving notes
└── train.py            # QLoRA training entrypoint (7 KB)
```

`pyproject.toml` declares its own dependencies (`datasets`, `trl`, `unsloth`) and pins Python to `>=3.13,<3.14`.
It does **not** depend on `my-model`; instead, `train.py` bootstraps the sibling package via an explicit `sys.path.insert`:

```python
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
```

This lets `train.py` import the pure, unit-tested `mymodel.tune` modules (config, dataset schema, prechecks) without joining the two dependency trees.

### The train.py script

`train.py` is the QLoRA entrypoint: trace JSONL in, LoRA adapter out.
Key CLI arguments:

| Argument          | Purpose                                                           |
| ----------------- | ----------------------------------------------------------------- |
| `--config PATH`   | Path to a `TuneConfig` file (.json/.yaml) declaring the run       |
| `--smoke`         | Run a tiny synthetic 5-step QLoRA on Qwen3-0.6B (proves the loop) |
| `--max-steps N`   | Override the config step cap                                      |
| `--skip-precheck` | Skip the busy-machine refusal (cloud/burst runs only)             |

The script loads and validates the dataset, runs a machine precheck that refuses to start if the GPU or RAM is busy, loads the base model in 4-bit via Unsloth's `FastLanguageModel`, applies LoRA to the standard Qwen/Llama attention and MLP modules, runs `SFTTrainer` from `trl` with bf16 training, and saves the adapter with a `run-manifest.json`.

### How to use it

```sh
# 1. Resolve the environment (first run downloads torch — big)
task tune:sync

# 2. Smoke test: 5-step QLoRA on Qwen3-0.6B with a synthetic dataset
task tune:smoke

# 3. Real training run
task tune:train -- --config runs/my-run.json
```

### GPU requirements and memory

The precheck refuses to start while the GPU or RAM is busy.
Real runs should carry a memory scope (house `salt` memory-guard rule) to protect the desktop:

```sh
systemd-run --user --scope -p MemoryMax=40G \
  task tune:train -- --config runs/my-run.json
```

### Serving the adapter

Write adapters to `${HF_HOME:-~/local/models}/loras/<adapter>` (the `TuneConfig.output_dir` default).
That path rides the HF-cache bind mount into the local vLLM container.
Start vLLM with the student base and LoRA enabled, then hot-load:

```sh
VLLM_MODEL=org/exact-base-checkpoint VLLM_TOOL_PARSER=hermes task vllm:up
curl -s http://localhost:8000/v1/load_lora_adapter \
  -H "Authorization: Bearer $LOCAL_VLLM_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"lora_name": "example-adapter", "lora_path": "/root/.cache/huggingface/loras/example-adapter"}'
```

Requests then select the adapter by `"model": "example-adapter"`.
See [Fine-tuning](tuning) for the full tuning guide.

## `VI` Dev pod: CUDA development

The dev pod provides a bare CUDA development environment for exploring the container ecosystem, building custom images, or testing GPU-dependent code without the vLLM or Marker overhead.

`pods/dev/cuda.yaml` launches the NVIDIA CUDA cuDNN devel image with GPU access and a bash shell:

```yaml
services:
    cuda-dev:
        image: nvcr.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04
        devices:
            - nvidia.com/gpu=0
        security_opt:
            - label=disable
        command: ['/bin/bash']
```

### When to use this vs the chat pod

Use the dev pod when you need to build or compile GPU-dependent packages from source (flash-attention, custom CUDA kernels), explore the CUDA toolkit, or debug GPU issues in isolation.
Use the chat pod when you just want to serve a model.
The dev pod has no model server, no HTTP endpoint, and no pre-installed ML libraries.

### How to use it

```sh
cd pods/dev
podman compose -f cuda.yaml run --rm cuda-dev
```

This drops you into a bash shell inside the container with GPU access.

## `VII` Building custom pods

Creating your own pod follows the same three-file pattern used by the existing pods:

```
pods/my-pod/
├── Containerfile    # How to build the image
├── compose.yaml     # How to run the container
└── run.sh           # (optional) Ad-hoc launch script
```

### Minimal example

**`Containerfile`:**

```dockerfile
FROM nvcr.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04

RUN python3 -m pip install uv
RUN uv pip install --system torch fastapi uvicorn

COPY server.py /app/server.py
WORKDIR /app

ENTRYPOINT ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8000"]
```

**`compose.yaml`:**

```yaml
services:
    my-server:
        build: .
        image: mymodel-mypod:latest
        ports:
            - 8000:8000
        devices:
            - nvidia.com/gpu=0
        security_opt:
            - label=disable
        volumes:
            - ~/.cache/huggingface:/root/.cache/huggingface
```

**Build and run:**

```sh
cd pods/my-pod
podman build -t mymodel-mypod:latest .
podman compose -f compose.yaml up
```

### Wiring GPU access

**Podman with NVIDIA CDI** (recommended):

```yaml
devices:
    - nvidia.com/gpu=0
```

CDI (Container Device Interface) is the modern, portable way to expose GPUs.
Configure it on the host with:

```sh
nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
```

**Docker / docker-compose with NVIDIA runtime:**

```yaml
deploy:
    resources:
        reservations:
            devices:
                - driver: nvidia
                  count: all
                  capabilities: [gpu]
```

### Exposing ports

Map container ports to the host in the compose file:

```yaml
ports:
    - 8000:8000   # host:container
```

For multiple services, use different host ports to avoid conflicts.

### Mounting model directories

Bind-mount the HuggingFace cache so model weights persist across restarts:

```yaml
volumes:
    - ~/.cache/huggingface:/root/.cache/huggingface
```

For custom data directories:

```yaml
volumes:
    - /path/to/host/data:/data:ro    # read-only intake
    - /path/to/host/out:/out:rw      # read-write output
```

## `VIII` Podman vs Docker

myModel prefers **podman** over docker.
Both use the same `Containerfile` and `compose.yaml` formats, but there are practical differences.

### Rootless podman advantages

- **No daemon**: podman runs as a regular process. No `dockerd` consuming resources at idle.
- **Rootless by default**: containers run as your user, not root. Better security isolation.
- **Systemd integration**: `podman generate systemd` creates unit files for managing containers as
  user services.
- **Drop-in compatibility**: `podman build`, `podman run`, and `podman compose` mirror the docker
  CLI for most workflows.

### NVIDIA CDI configuration

Both podman and docker can use NVIDIA CDI for GPU access.
CDI is preferred over the legacy `--gpus all` flag because it is runtime-agnostic:

```sh
# Generate the CDI spec (once)
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

# Verify GPU access
podman run --rm --device nvidia.com/gpu=0 \
  nvcr.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04 nvidia-smi
```

### Host networking considerations

The gateway (see [The LiteLLM Gateway](gateway.md)) uses podman with host networking (`--net host`) so that the gateway and local vLLM servers share `localhost`.
This avoids port-mapping overhead and simplifies TLS termination.
When running containers on the default bridge network, use explicit port mappings: `podman run -p 8000:8000 ...`.

### Boot persistence for rootless containers

Rootless podman containers do not survive reboot by default.
To persist a container as a user service:

```sh
# Generate a systemd unit file
podman generate systemd --files --name my-container

# Install as a user service
mkdir -p ~/.config/systemd/user/
cp container-my-container.service ~/.config/systemd/user/

# Enable lingering so the service starts at boot
loginctl enable-linger $USER

systemctl --user daemon-reload
systemctl --user enable --now container-my-container.service
```

This ensures your model-serving container restarts automatically after a reboot.