# Fine-tuning

## `I` Overview

myModel includes a pilot pipeline for fine-tuning a local base model on traces from a stronger teacher model.
The pipeline uses **QLoRA**: the base model stays frozen in 4-bit quantization, and training updates only a small **LoRA adapter** (low-rank matrices attached to the attention and MLP projections).
The input is a JSONL dataset of teacher LM calls, and the output is one adapter directory that vLLM can serve next to the base model.

This is finetune pilot 1: the training loop works end to end and is proven by a smoke test, but the pipeline has no evaluation stage of its own yet.

The implementation splits into two halves:

- `mymodel/tune/` holds the pure, unit-tested half: the run config (`config.py`), the trace dataset schema (`dataset.py`), and the machine prechecks (`precheck.py`).
  These modules use only the standard library plus pydantic.
- `pods/tune/` holds the GPU half: a standalone `uv` project whose `train.py` entrypoint runs the actual training with Unsloth and TRL.
  It imports the `mymodel.tune` modules across the package boundary through an explicit `sys.path` bootstrap.

The two halves live in separate environments on purpose.
Unsloth pins torch and transformers tightly, and those pins cannot co-resolve with `my-model`'s vLLM dependency tree under the repo's `exclude-newer` guard.
See [Container Deployments](pods) for the pod layout.

## `II` How a run works

One training run performs these steps in order:

1. Check that the machine is idle enough to train (see [§VI](#vi-the-machine-precheck)).
2. Load and validate the trace dataset. Malformed lines are reported and skipped.
3. Load the base model in 4-bit through Unsloth's `FastLanguageModel`.
4. Attach LoRA adapters to the Qwen/Llama projection modules with gradient checkpointing.
5. Render each conversation through the model's own chat template into plain text.
6. Train with TRL's `SFTTrainer` in bf16.
7. Save the adapter and tokenizer, then write a run manifest (see [§VII](#vii-outputs-and-serving-the-adapter)).

Heavy imports load lazily, so `--help` and the precheck return instantly even on a cold environment.

## `III` The dataset: teacher-trace JSONL

The dataset is a JSONL file with one teacher LM call per line, in the format that `critic export-traces` (corpus repo) writes:

```json
{"messages": [{"role": "system", "content": "You are a code reviewer."}, {"role": "user", "content": "Review this diff."}], "completion": "Looks solid.", "meta": {"ref": "ex-1", "model": "deepseek-chat", "cost_usd": 0.01}}
```

Each record carries three fields:

| Field        | Content                                                                                   |
| ------------ | ----------------------------------------------------------------------------------------- |
| `messages`   | The chat turns the teacher received (roles: system, user, assistant, tool). At least one. |
| `completion` | The text the teacher returned. Must not be empty.                                         |
| `meta`       | Provenance: example `ref`, teacher `model`, and `cost_usd`. All optional.                 |

During training, `to_training_messages` appends the completion as the final assistant turn, which makes it the supervised target.
Unknown extra keys are ignored, but unknown roles and empty completions fail validation.

The loader `load_traces` returns the valid records plus a list of per-line error strings.
`train.py` prints the count of skipped malformed lines and continues. It exits only when no valid records remain.
The `trace_stats` function summarizes the corpus (record count, distinct refs, character totals, per-model counts) for the run manifest.

The exporter writes train-split examples only.
The frozen holdout stays unseen for `critic eval`, so do not add holdout traces to a training dataset.

## `IV` Run configuration: `TuneConfig`

One small file declares a run completely, so runs are reproducible and reviewable.
`TuneConfig.from_file` reads JSON, or YAML when `pyyaml` is installed. The main package environment has it, and the lean pod environment should use JSON.

Three fields are required:

| Field          | Meaning                                                                                       |
| -------------- | --------------------------------------------------------------------------------------------- |
| `base_model`   | HuggingFace ID of the student base. Must match the serving base exactly. There is no default. |
| `adapter_name` | Name of the produced adapter. Also its output subdirectory.                                   |
| `dataset_path` | Path to the trace JSONL. `~` expands at load time.                                            |

The remaining fields carry recipe defaults:

| Field           | Default                | Meaning                                                                                             |
| --------------- | ---------------------- | --------------------------------------------------------------------------------------------------- |
| `output_dir`    | `~/local/models/loras` | Parent directory for adapter output.                                                                |
| `lora_r`        | 16                     | LoRA rank.                                                                                          |
| `lora_alpha`    | 32                     | LoRA alpha. The default keeps the 2x-rank ratio from the template.                                  |
| `lora_dropout`  | 0.05                   | LoRA dropout.                                                                                       |
| `max_seq_len`   | 8192                   | Maximum training sequence length. The binding VRAM knob on a 24 GB card.                            |
| `epochs`        | 1.0                    | Passes over the dataset. Ignored when `max_steps` is set.                                           |
| `max_steps`     | none                   | Hard step cap. Smoke runs set this small.                                                           |
| `learning_rate` | 2e-4                   | AdamW peak learning rate.                                                                           |
| `batch_size`    | 2                      | Per-device micro-batch size.                                                                        |
| `grad_accum`    | 8                      | Gradient-accumulation steps.                                                                        |
| `seed`          | 3407                   | RNG seed for the whole run.                                                                         |
| `load_in_4bit`  | true                   | QLoRA (4-bit base) instead of bf16 LoRA. 4-bit is what fits models of 4B parameters and up locally. |

The effective batch size is `batch_size * grad_accum`.
Unknown keys fail validation, so a typoed field name cannot silently train with defaults.
Numeric bounds are validated as well: for example `lora_r` must be at least 1 and `max_seq_len` at least 128.

A minimal run config looks like this:

```json
{
    "base_model": "Qwen/Qwen3-4B-Instruct-2507",
    "adapter_name": "example-adapter",
    "dataset_path": "~/local/traces/critic-train.jsonl",
    "max_steps": 200
}
```

## `V` Running a training run

The Taskfile exposes three tasks, all rooted at `pods/tune/`:

1. Resolve the pod environment with `task tune:sync`. The first run downloads torch, so it is large.
2. Prove the loop with `task tune:smoke`. This runs a 5-step QLoRA on Qwen3-0.6B against a tiny synthetic dataset in a temporary directory.
3. Write a run config and start training with `task tune:train -- --config runs/my-run.json`.

On this box, wrap real runs in a memory scope (the house `salt` memory-guard rule):

```sh
systemd-run --user --scope -p MemoryMax=40G task tune:train -- --config runs/my-run.json
```

`train.py` takes these arguments:

| Argument          | Meaning                                                                             |
| ----------------- | ----------------------------------------------------------------------------------- |
| `--config PATH`   | Path to a `TuneConfig` file (`.json` or `.yaml`).                                   |
| `--smoke`         | Run the synthetic 5-step loop instead. Mutually exclusive with `--config`.          |
| `--max-steps N`   | Override the step cap from the config.                                              |
| `--skip-precheck` | Skip the busy-machine refusal. For cloud or burst runs where the box is not shared. |

## `VI` The machine precheck

Before any training, `assert_ready` refuses to start on a busy box.
The training GPU (device 0) is shared with the desktop, and system RAM is the box's real pressure point.
The precheck reads `nvidia-smi` and `/proc/meminfo`, then raises `PrecheckError` listing every violated threshold:

| Check           | Refusal threshold          |
| --------------- | -------------------------- |
| GPU memory      | More than 6144 MiB in use  |
| GPU utilization | More than 25 percent       |
| System RAM      | Less than 12 GiB available |

The memory threshold tolerates the desktop (KWin/Xwayland hold about 2 to 3 GB) but rejects transient jobs such as marker OCR.
Pass `--skip-precheck` only for cloud or burst runs on an unshared machine.

## `VII` Outputs and serving the adapter

A run writes to `output_dir / adapter_name` (the `TuneConfig.adapter_dir` property).
The directory holds the saved adapter, the tokenizer, and a `run-manifest.json` recording the full config, dataset stats, final training loss, step count, wall-clock seconds, peak VRAM, and the adapter path.

The default `output_dir` sits under the HuggingFace cache path, so adapters ride the HF-cache bind mount into the local vLLM container.
To serve one, start vLLM on the exact student base with LoRA enabled (the `enable_lora` build arg, see [Serving with vLLM](serving)), then load the adapter at runtime:

```sh
VLLM_MODEL=org/exact-base-checkpoint VLLM_TOOL_PARSER=hermes task vllm:up
curl -s http://localhost:8000/v1/load_lora_adapter \
  -H "Authorization: Bearer $LOCAL_VLLM_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"lora_name": "example-adapter", "lora_path": "/root/.cache/huggingface/loras/example-adapter"}'
```

Requests then select the adapter with `"model": "example-adapter"`.
Adding a gateway alias for an adapter is a separate, human-release-gated change under `policies/versioning.md`.

## `VIII` Limits

- The LoRA target modules are fixed to the Unsloth-recommended Qwen/Llama set (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`).
  Other architectures need a code change, not a config change.
- The `base_model` must match the serving base checkpoint exactly, or the adapter will not apply.
- The pipeline has no evaluation stage. Quality measurement happens outside it, through `critic eval` on the frozen holdout.
- The smoke test proves the mechanics on a 0.6B model. It says nothing about adapter quality on real traces.