Container Deployments#

`I` Overview#

Pods are pre-built container deployment artifacts for different model serving scenarios in myModel. Each pod is a self-contained directory under pods/ with a Containerfile, a compose config, and launch scripts. They let you deploy models in containers without writing your own Dockerfiles.

If you are new to containers: a Containerfile (called a Dockerfile in Docker parlance) is a recipe for building a container image, and a compose file (compose.yaml) declares how to run that image, including GPU access, ports, and volumes. myModel uses podman (rootless, daemonless) instead of docker, but the file formats are compatible.

pods/
├── chat/        # vLLM inference server
├── read/        # Marker PDF/OCR server
├── tune/        # QLoRA training environment
├── dev/         # CUDA development environment
└── README.md    # Container image reference

The four pods cover the most common workflows: serving a model for chat, processing documents, fine-tuning, and developing against the CUDA stack. You can mix and match them or build your own (see §VII).

`II` Container images reference#

myModel builds on upstream container images from NVIDIA and vLLM. The table below is an expanded view of pods/README.md, listing the images used or referenced by the pods.

Image	Variant	Use case	Notes
`nvcr.io/nvidia/cuda:13.0.2-devel-ubuntu24.04`	CUDA devel	Building GPU-dependent Python packages (read pod)	Ubuntu 24.04, CUDA 13.0.2
`nvcr.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04`	cuDNN devel	Same but with cuDNN pre-installed	Required for Marker’s torchvision/torch stack
`docker.io/vllm/vllm-openai:latest`	First-party vLLM	Running the OpenAI-compatible vLLM server	Docs
`docker.io/vllm/vllm-openai:v0.11.2`	Pinned first-party	Chat pod default image	Pinned for reproducibility
`nvcr.io/nvidia/vllm:25.11-py3`	NVIDIA’s vLLM	Alternative vLLM image from NGC	Ubuntu 24.04, CUDA 13.0.2, Python 3.12

Compatibility notes:

podman: The pods use security_opt: [label=disable] and devices: [nvidia.com/gpu=0], which are podman-compatible. NVIDIA CDI must be configured on the host.
Platform: All images target linux/amd64 unless overridden. The chat pod’s compose.yaml pins platform: linux/arm64 for the default configuration (adjust to your hardware).

`III` Chat pod: vLLM inference server#

The chat pod runs vLLM as an OpenAI-compatible HTTP server inside a container. This is the fastest path to serving a model: you provide a HuggingFace model ID and the container handles the rest.

The Containerfile#

pods/chat/Containerfile is a thin layer on top of docker.io/vllm/vllm-openai:v0.11.2. The file is mostly commented-out scaffolding showing where you would add custom dependencies:

FROM docker.io/vllm/vllm-openai:v0.11.2

# Dependency installation example:
# RUN uv pip install --system vllm[audio]

Uncomment or add RUN lines to install extra Python packages. For most deployments, the base image alone is sufficient.

The compose.yaml#

services:
    chat-server:
        image: docker.io/vllm/vllm-openai:v0.11.2
        platform: linux/arm64
        runtime: nvidia
        ipc: host
        ports:
            - 8000:8000
        environment:
            - HF_TOKEN=${HF_TOKEN}
            - MODEL_ID=${MODEL_ID:-google/gemma-3-12b-it}
        volumes:
            - $HF_HOME:/root/.cache/huggingface
        security_opt:
            - label=disable
        command: ["--model", "${MODEL_ID:-google/gemma-3-12b-it}"]
        deploy:
            resources:
                reservations:
                    devices:
                        - driver: nvidia
                          count: all
                          capabilities: [gpu]

Setting	Purpose
`runtime: nvidia`	NVIDIA container runtime for GPU access
`ipc: host`	Share host IPC namespace (vLLM recommendation for perf)
`ports: 8000:8000`	Expose the OpenAI API on host port 8000
`HF_TOKEN`	HuggingFace access token (for gated models)
`MODEL_ID`	The HuggingFace model ID to serve (defaults to Gemma 3 12B)
`volumes`	Bind-mount the HF cache so model weights persist
`security_opt: label=disable`	Disable SELinux label enforcement (podman compatibility)
`deploy.resources`	Reserve all NVIDIA GPUs for the container

The run script#

pods/chat/run.sh is a one-liner equivalent to the compose file for ad-hoc use:

podman run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --env "HF_TOKEN=${HF_TOKEN}" \
    --security-opt label=disable \
    docker.io/vllm/vllm-openai --model google/gemma-3-12b-it

Build and run#

cd pods/chat
podman build -t mymodel-chat .

export HF_TOKEN=hf_xxxxxxxxxxxx
podman compose -f compose.yaml up
# or: bash run.sh

Customize the model#

Set MODEL_ID before launching to serve a different model:

MODEL_ID=microsoft/Phi-3-mini-4k-instruct podman compose -f compose.yaml up

Test it#

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "google/gemma-3-12b-it", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 64}'

The reference container file#

pods/chat/vLLM__reference.container is the full upstream vLLM Dockerfile (22 KB), included for reference. It shows how the official image is built from scratch (multi-stage CUDA build, wheel compilation, DeepGEMM, EP kernels, FlashInfer). You do not need to build from this file, but it is an invaluable reference if you need to patch vLLM or add custom kernels.

For more on serving models outside of containers, see Serving with vLLM.

`IV` Read pod: OCR/PDF processing#

The read pod runs a Marker server for converting PDF and image files into structured markdown. Unlike the chat pod, this pod builds a custom image from the NVIDIA CUDA base.

The marker-server.py #

pods/read/marker-server.py (11 KB) distributes document processing across multiple GPUs using Python multiprocessing. Key features:

Multi-GPU parallelism: One multiprocessing.Pool per GPU; each worker pins itself via CUDA_VISIBLE_DEVICES.
Configurable LLM backend: Optionally routes to OpenAI, Claude, Gemini, Vertex, Azure, or Ollama for validation and correction.
Modal integration: Designed to run on Modal (serverless GPU) with 4x T4 GPUs, but the core logic works locally on any CUDA-capable machine.
I/O: Files in a mounted intake directory (PDF, JPG, PNG, TIFF); output is extracted markdown text, images, and metadata per document.

The Containerfile#

pods/read/Containerfile is a multi-stage build from the NVIDIA CUDA cuDNN base:

FROM nvcr.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04 AS base
RUN python3 -m pip install uv

# GPU packages: marker-pdf, torch, torchvision
FROM base as marker-base
RUN $UV_INSTALL marker-pdf pillow pytorch torchvision torchaudio

# Server packages: FastAPI, granian (ASGI server)
FROM marker-base as py-base
RUN $UV_INSTALL fastapi granian uvloop sqlmodel

# Final image
FROM py-base as final-read
COPY ./marker-server.py /usr/local/bin/marker_server
ENTRYPOINT ["marker_server", "--port", "8000"]

It uses uv for fast package resolution with cache mounts for reproducible, layer-cached builds.

The compose.yaml#

services:
    my-reader:
        build: .
        image: mymodel-read:latest
        ports:
            - 8000:8000
        devices:
            - nvidia.com/gpu=0
        security_opt:
            - label=disable
        env_file:
            - ${MY_ENV_FILE}
        volumes:
            - ${MY_DATA}/ocr_intake:/data/intake:ro
            - ${MY_DATA}/ocr_output:/data/output:rw
            - datalab-cache:/root/.cache/datalab/:rw

Setting	Purpose
`build: .`	Build from the local `Containerfile` (not a pre-built image)
`devices: nvidia.com/gpu=0`	Mount GPU 0 via NVIDIA CDI
`volumes`	Intake (read-only), output (read-write), cache
`env_file`	Load secrets (API keys, model paths) from an env file

Build and run#

cd pods/read
podman build -t mymodel-read:latest .

export MY_ENV_FILE=.env
export MY_DATA=/path/to/your/data
podman compose -f compose.yaml up

Send documents for processing#

Place PDFs or images in your intake directory, then call the server:

curl http://localhost:8000/convert \
  -H "Content-Type: application/json" \
  -d '{"file": "/data/intake/document.pdf", "output_format": "markdown"}'

`V` Tune pod: QLoRA fine-tuning#

The tune pod provides a standalone training environment for QLoRA fine-tuning of LLMs. It is a deliberately separate uv project because unsloth’s tight torch/transformers pins are unsatisfiable when co-resolved with my-model’s vLLM dependency under the repo’s exclude-newer guard.

The tune pod takes teacher-trace JSONL datasets and produces LoRA adapters that can be hot-loaded into a running vLLM server. It uses Unsloth for memory-efficient 4-bit quantized training with gradient checkpointing.

Separate uv project structure#

pods/tune/
├── pyproject.toml     # Standalone uv project (my-model-tune)
├── README.md           # Usage and adapter serving notes
└── train.py            # QLoRA training entrypoint (7 KB)

pyproject.toml declares its own dependencies (datasets, trl, unsloth) and pins Python to >=3.13,<3.14. It does not depend on my-model; instead, train.py bootstraps the sibling package via an explicit sys.path.insert:

REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))

This lets train.py import the pure, unit-tested mymodel.tune modules (config, dataset schema, prechecks) without joining the two dependency trees.

The train.py script#

train.py is the QLoRA entrypoint: trace JSONL in, LoRA adapter out. Key CLI arguments:

Argument	Purpose
`--config PATH`	Path to a `TuneConfig` file (.json/.yaml) declaring the run
`--smoke`	Run a tiny synthetic 5-step QLoRA on Qwen3-0.6B (proves the loop)
`--max-steps N`	Override the config step cap
`--skip-precheck`	Skip the busy-machine refusal (cloud/burst runs only)

The script loads and validates the dataset, runs a machine precheck that refuses to start if the GPU or RAM is busy, loads the base model in 4-bit via Unsloth’s FastLanguageModel, applies LoRA to the standard Qwen/Llama attention and MLP modules, runs SFTTrainer from trl with bf16 training, and saves the adapter with a run-manifest.json.

How to use it#

# 1. Resolve the environment (first run downloads torch — big)
task tune:sync

# 2. Smoke test: 5-step QLoRA on Qwen3-0.6B with a synthetic dataset
task tune:smoke

# 3. Real training run
task tune:train -- --config runs/my-run.json

GPU requirements and memory#

The precheck refuses to start while the GPU or RAM is busy. Real runs should carry a memory scope (house salt memory-guard rule) to protect the desktop:

systemd-run --user --scope -p MemoryMax=40G \
  task tune:train -- --config runs/my-run.json

Serving the adapter#

Write adapters to ${HF_HOME:-~/local/models}/loras/<adapter> (the TuneConfig.output_dir default). That path rides the HF-cache bind mount into the local vLLM container. Start vLLM with the student base and LoRA enabled, then hot-load:

VLLM_MODEL=org/exact-base-checkpoint VLLM_TOOL_PARSER=hermes task vllm:up
curl -s http://localhost:8000/v1/load_lora_adapter \
  -H "Authorization: Bearer $LOCAL_VLLM_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"lora_name": "example-adapter", "lora_path": "/root/.cache/huggingface/loras/example-adapter"}'

Requests then select the adapter by "model": "example-adapter". See Fine-tuning for the full tuning guide.

`VI` Dev pod: CUDA development#

The dev pod provides a bare CUDA development environment for exploring the container ecosystem, building custom images, or testing GPU-dependent code without the vLLM or Marker overhead.

pods/dev/cuda.yaml launches the NVIDIA CUDA cuDNN devel image with GPU access and a bash shell:

services:
    cuda-dev:
        image: nvcr.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04
        devices:
            - nvidia.com/gpu=0
        security_opt:
            - label=disable
        command: ['/bin/bash']

When to use this vs the chat pod#

Use the dev pod when you need to build or compile GPU-dependent packages from source (flash-attention, custom CUDA kernels), explore the CUDA toolkit, or debug GPU issues in isolation. Use the chat pod when you just want to serve a model. The dev pod has no model server, no HTTP endpoint, and no pre-installed ML libraries.

How to use it#

cd pods/dev
podman compose -f cuda.yaml run --rm cuda-dev

This drops you into a bash shell inside the container with GPU access.

`VII` Building custom pods#

Creating your own pod follows the same three-file pattern used by the existing pods:

pods/my-pod/
├── Containerfile    # How to build the image
├── compose.yaml     # How to run the container
└── run.sh           # (optional) Ad-hoc launch script

Minimal example#

Containerfile:

FROM nvcr.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04

RUN python3 -m pip install uv
RUN uv pip install --system torch fastapi uvicorn

COPY server.py /app/server.py
WORKDIR /app

ENTRYPOINT ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8000"]

compose.yaml:

services:
    my-server:
        build: .
        image: mymodel-mypod:latest
        ports:
            - 8000:8000
        devices:
            - nvidia.com/gpu=0
        security_opt:
            - label=disable
        volumes:
            - ~/.cache/huggingface:/root/.cache/huggingface

Build and run:

cd pods/my-pod
podman build -t mymodel-mypod:latest .
podman compose -f compose.yaml up

Wiring GPU access#

Podman with NVIDIA CDI (recommended):

devices:
    - nvidia.com/gpu=0

CDI (Container Device Interface) is the modern, portable way to expose GPUs. Configure it on the host with:

nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Docker / docker-compose with NVIDIA runtime:

deploy:
    resources:
        reservations:
            devices:
                - driver: nvidia
                  count: all
                  capabilities: [gpu]

Exposing ports#

Map container ports to the host in the compose file:

ports:
    - 8000:8000   # host:container

For multiple services, use different host ports to avoid conflicts.

Mounting model directories#

Bind-mount the HuggingFace cache so model weights persist across restarts:

volumes:
    - ~/.cache/huggingface:/root/.cache/huggingface

For custom data directories:

volumes:
    - /path/to/host/data:/data:ro    # read-only intake
    - /path/to/host/out:/out:rw      # read-write output

`VIII` Podman vs Docker#

myModel prefers podman over docker. Both use the same Containerfile and compose.yaml formats, but there are practical differences.

Rootless podman advantages#

No daemon: podman runs as a regular process. No dockerd consuming resources at idle.
Rootless by default: containers run as your user, not root. Better security isolation.
Systemd integration: podman generate systemd creates unit files for managing containers as user services.
Drop-in compatibility: podman build, podman run, and podman compose mirror the docker CLI for most workflows.

NVIDIA CDI configuration#

Both podman and docker can use NVIDIA CDI for GPU access. CDI is preferred over the legacy --gpus all flag because it is runtime-agnostic:

# Generate the CDI spec (once)
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

# Verify GPU access
podman run --rm --device nvidia.com/gpu=0 \
  nvcr.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04 nvidia-smi

Host networking considerations#

The gateway (see The LiteLLM Gateway) uses podman with host networking (--net host) so that the gateway and local vLLM servers share localhost. This avoids port-mapping overhead and simplifies TLS termination. When running containers on the default bridge network, use explicit port mappings: podman run -p 8000:8000 ....

Boot persistence for rootless containers#

Rootless podman containers do not survive reboot by default. To persist a container as a user service:

# Generate a systemd unit file
podman generate systemd --files --name my-container

# Install as a user service
mkdir -p ~/.config/systemd/user/
cp container-my-container.service ~/.config/systemd/user/

# Enable lingering so the service starts at boot
loginctl enable-linger $USER

systemctl --user daemon-reload
systemctl --user enable --now container-my-container.service

This ensures your model-serving container restarts automatically after a reboot.

Container Deployments#

I Overview#

II Container images reference#

III Chat pod: vLLM inference server#

The Containerfile#

The compose.yaml#

The run script#

Build and run#

Customize the model#

Test it#

The reference container file#

IV Read pod: OCR/PDF processing#

The marker-server.py#

The Containerfile#

The compose.yaml#

Build and run#

Send documents for processing#

V Tune pod: QLoRA fine-tuning#

Separate uv project structure#

The train.py script#

How to use it#

GPU requirements and memory#

Serving the adapter#

VI Dev pod: CUDA development#

When to use this vs the chat pod#

How to use it#

VII Building custom pods#

Minimal example#

Wiring GPU access#

Exposing ports#

Mounting model directories#

VIII Podman vs Docker#

Rootless podman advantages#

NVIDIA CDI configuration#

Host networking considerations#

Boot persistence for rootless containers#

`I` Overview#

`II` Container images reference#

`III` Chat pod: vLLM inference server#

`IV` Read pod: OCR/PDF processing#

The marker-server.py #

`V` Tune pod: QLoRA fine-tuning#

`VI` Dev pod: CUDA development#

`VII` Building custom pods#

`VIII` Podman vs Docker#