myModel: Local LLM inference and serving

myModel is a collection of tools for deploying local LLMs across multiple runtimes (vLLM, llama.cpp) and platforms, with an optional tuning pipeline. An external LiteLLM gateway fronts the local servers. See The LiteLLM Gateway. Together, these tools form one cohesive group, but they are not a single product. They work together to serve models over HTTP from your own hardware.

Install

The source repository is public. Install from Git with Python 3.13 or newer:

uv tool install git+https://gitlab.com/doering-ai/apps/model.git

Or clone and sync the development environment:

git clone https://gitlab.com/doering-ai/apps/model.git
cd model
uv sync

See the Quickstart for a full walkthrough.

• 1. Quickstart
  • I Introduction
  • II Prerequisites
  • III Installation
  • IV Your first model deployment
  • V Setting up the gateway
  • VI Connecting a harness
  • VII Next steps
• 2. Serving with vLLM
  • I Overview
  • II The Model Catalogue
  • III Launching a Model
  • IV GPU Memory and Model Sizing
  • V Connecting Through the Gateway
  • VI Troubleshooting
• 3. The LiteLLM Gateway
  • What myModel still owns
  • Connecting a harness
• 4. Container Deployments
  • I Overview
  • II Container images reference
  • III Chat pod: vLLM inference server
  • IV Read pod: OCR/PDF processing
  • V Tune pod: QLoRA fine-tuning
  • VI Dev pod: CUDA development
  • VII Building custom pods
  • VIII Podman vs Docker
• 5. Model Catalogue
  • What is the Model Catalogue?
  • The Ontology: Verbs, Modalities, and Sizes
  • The ModelCard
  • The Registry
  • Creating a New Model Entry
  • Using the Catalogue with serve.py
  • HuggingFace Integration
• 6. Fine-tuning
  • I Overview
  • II How a run works
  • III The dataset: teacher-trace JSONL
  • IV Run configuration: TuneConfig
  • V Running a training run
  • VI The machine precheck
  • VII Outputs and serving the adapter
  • VIII Limits
• 7. Ternary Bonsai
  • I Overview
  • II The launcher
  • III The systemd unit
  • IV Configuration
  • V Operating a session
  • VI The A100 backend seam
  • VII See also
• 8. Python API
  • I Module Layout
  • II Starting the Server
  • III The Management HTTP API
  • IV Configuration
  • V Authentication
  • VI Examples
  • VII See also
• 9. Auth & TLS
  • I Overview
  • II Running in production mode
  • III The JWT middleware
  • IV Failure modes
  • V TLS for the management API
  • VI The inference-port API key
  • VII Current status and limitations