Ternary Bonsai

I Overview

The Ternary Bonsai service is the local quality-floor runtime for active advisor and critic review sessions. It serves one pinned artifact, the Ternary Bonsai 27B model in Q2_0 GGUF form, through a pinned llama.cpp build. The raw endpoint binds only to 127.0.0.1:8001, and the LiteLLM gateway exposes it as local-bonsai-27b.

The service is session-bounded by design. You start it for a review session and stop it afterward. It has no [Install] target, so systemd never starts it at boot, and it forcibly ends any forgotten session after six hours.

Three layers cooperate:

Layer	Role
mymodel/ternary.py	The launcher. Validates every input, then execs the exact verified binary and model
ternary/config.env	The operational session bounds (ports, budgets, deadlines)
ternary/ternary-bonsai-27b.service	The systemd user unit that runs the launcher under cgroup and sandbox limits

The launcher holds all immutable identity: artifact paths, the pinned PrismML runtime revision, and the pinned SHA-256 digests. The PrismML runtime is the llama.cpp source tree under ~/.local/share/prism-bonsai/llama.cpp, pinned to one commit. config.env cannot override any of these values, and tests/test_ternary.py rejects a profile that tries.

Important

The trust boundary is a single-principal workstation. The committed loopback token prevents accidental cross-wiring between local services. It is not isolation from another process running as you. The project venv and its CUDA/NCCL packages count as trusted, package-managed inputs.

II The launcher

mymodel.ternary is a single-file worker built on pydantic and the standard library alone. Its Worker model validates one session-bounded server, and its CLI exposes four commands:

Command	Effect
preflight	Validate immutable inputs and GPU admission, then exit
serve	Run preflight, re-verify the opened inodes, then exec the server
status	Report health from the loopback endpoint, exit 1 when unhealthy
wait	Block until the endpoint answers, with a 120-second default deadline

The preflight gate

Preflight fails closed: any failed check raises RuntimeError before the GPU is committed. The checks run in this order:

1. The host stays 127.0.0.1 and parallelism stays 1.

2. The llama-server binary exists and is executable, and the model file exists.

3. Each runtime-library directory exists, belongs to root or to you, and is not group- or world-writable.

4. The binary SHA-256 matches the pinned digest.

5. The model SHA-256 matches the pinned digest.

6. git rev-parse HEAD in the runtime tree matches the pinned PrismML commit.

7. nvidia-smi reports at least 16,000 MiB of free VRAM (the admission floor).

8. No competing process holds GPU compute. The display processes kwin_wayland and Xorg may hold a small CUDA context and do not block admission.

9. The loopback port binds successfully.

Every nvidia-smi probe also fails closed on empty or malformed output, not only on bad values.

Verified exec

serve does not trust the paths preflight checked. It opens the binary and model with O_NOFOLLOW, hashes the open file descriptors, and compares each digest against the pinned value again. Hashing the open inode defeats a path swap between check and exec. It then calls os.execve on the verified binary descriptor, passing the model as /proc/self/fd/<fd>.

The child environment is minimal. The launcher forwards an explicit allowlist of non-secret variables (HOME, PATH, USER, cache paths, locale) and adds LD_LIBRARY_PATH plus LLAMA_API_KEY. Gateway credentials such as LITELLM_TOKEN never cross into the runtime, and the API key never appears on the command line.

The server contract

The launcher builds a one-slot, bounded-reasoning command:

Setting	Value	Purpose
--host / --port	127.0.0.1 / 8001	Loopback ingress only
--alias	ternary-bonsai-27b	The model name clients send
--ctx-size	262144	Full model context
--parallel	1	One request at a time
--reasoning-budget	65536	Bounded reasoning effort
--timeout	1800	Thirty-minute per-request deadline
--cache-type-k / -v	q8_0	Quantized KV cache
--cache-ram	0	Prompt cache disabled
--n-gpu-layers all	fixed	Full GPU offload
--flash-attn on	fixed	Flash attention
--reasoning-format	deepseek	Reasoning content format

III The systemd unit

ternary/ternary-bonsai-27b.service is a user unit of Type=exec that runs the launcher three times:

1. ExecStartPre runs preflight, so a failed gate stops the unit before exec.

2. ExecStart runs serve, which replaces the process with the verified llama-server.

3. ExecStartPost runs wait --timeout 120, so the unit reports active only when the endpoint answers.

Session bounds

Directive	Value	Effect
RuntimeMaxSec	6h	Forgotten sessions end after six hours
StartLimitIntervalSec/Burst	6h/3	Crash loops stop after three starts
Restart	no	No automatic restart
MemoryHigh / MemoryMax	20G / 28G	cgroup memory throttle and hard cap
TasksMax	512	Thread and process cap
OOMPolicy	stop	OOM kills the unit, not the desktop
KillMode	control-group	Stop kills the whole process group
KillSignal / FinalKillSignal	SIGINT / SIGKILL	Graceful stop, then force after TimeoutStopSec=45s

Sandboxing

The unit hides the host from the runtime. ProtectHome=tmpfs mounts an empty home, and explicit BindReadOnlyPaths expose only the four trees the runtime needs: the project venv, the uv-managed Python tree behind the venv interpreter symlink, the mymodel package, and the PrismML tree. ProtectSystem=strict, PrivateTmp, NoNewPrivileges, and the kernel and syscall restrictions complete the profile.

IV Configuration

ternary/config.env holds the operational session bounds. Each variable maps to a Worker field with the same default:

Variable	Default	Meaning
TERNARY_PROJECT	repo path	Project root the unit runs from
TERNARY_MODEL_ALIAS	ternary-bonsai-27b	Server alias
TERNARY_HOST	127.0.0.1	Must stay loopback
TERNARY_PORT	8001	Loopback port
TERNARY_CONTEXT_TOKENS	262144	Context size
TERNARY_PARALLEL	1	Must stay 1
TERNARY_REASONING_BUDGET	65536	Reasoning budget in tokens
TERNARY_REQUEST_TIMEOUT	1800	Per-request deadline in seconds
TERNARY_MIN_GPU_FREE_MIB	16000	GPU admission floor in MiB
LOCAL_GGUF_TOKEN	local-bonsai-key	Loopback token, sent as LLAMA_API_KEY

After you change the profile, run task ternary:install to refresh the unit symlink and reload systemd, then repeat the preflight.

V Operating a session

The ternary:* Taskfile namespace drives the whole lifecycle:

1. Run task ternary:preflight to validate the host before you commit the GPU.

2. Run task ternary:up to install the user-unit symlink and start the service. This task never enables the unit.

3. Run task ternary:status to see the systemd state and the live health endpoint.

4. Run task ternary:logs to follow the journal.

5. Run task ternary:down to stop the service when the session ends.

The served endpoint is http://127.0.0.1:8001/v1, OpenAI-compatible, one request at a time.

Corpus supervises each local merge-gate review in a dedicated process group with a one-hour default ceiling, and a deadline never counts as a successful review. For an operator-attached session that intentionally needs longer, raise the finite CORPUS_CRITIC_REVIEW_TIMEOUT_SECONDS ceiling for that invocation. Keep the service’s thirty-minute per-request bound unchanged.

VI The A100 backend seam

The local RTX 4090 is not the only planned home for this contract. An operator-attached Colab A100 backend boundary may implement the same ternary-bonsai-27b OpenAI-compatible contract when the local GPU is insufficient.

That backend is not built yet. This repo ships only the local service, and nothing here creates or keeps cloud runtimes alive. Backend selection stays explicit for each active session.

VII See also

• Quickstart for the gateway route that fronts this service.

• The LiteLLM Gateway for the local-bonsai-27b alias and client wiring.

• Serving with vLLM for the always-on local serving path, which the session service does not replace.