The LiteLLM Gateway#

Important

The LiteLLM gateway no longer lives in this repository. It moved to its own repository, ~/my/apps/gate (gate/gateway/), on 2026-08-08. This repo’s old gateway/ directory was removed.

The gateway is operational infrastructure shared by every harness in the fleet. It handles routing, metering, and fleet-mode policy. It listens on localhost:4000 and reaches this repo’s local servers over loopback (:8000, :8001). It never runs inference itself.

The full reference lives in ~/my/apps/gate/gateway/README.md and ~/my/apps/gate/gateway/docs/. It covers config, secrets, harness wiring, telemetry, and troubleshooting.

What myModel still owns#

  • vllm/ provides local vLLM on the RTX 4090. The gateway exposes it as the local-coder model alias. See Serving and vllm/README.md. Start it with task vllm:up from this repo’s root (no gateway secret needed).

  • ternary/ provides the session-bounded Ternary Bonsai 27B profile. The gateway exposes it as local-bonsai-27b. See ternary/README.md. Start it with task ternary:up.

Connecting a harness#

Point any tool at http://localhost:4000/v1 with LITELLM_TOKEN as the bearer key. Claude Code uses http://localhost:4000 for its Anthropic surface. Select local-coder or local-bonsai-27b by alias, the same as any other gateway route. Harness wiring for each tool lives in ~/my/apps/gate/gateway/README.md.

See also: Quickstart for the myModel walkthrough, and myGate for the gateway itself.