Fine-tuning

I Overview

myModel includes a pilot pipeline for fine-tuning a local base model on traces from a stronger teacher model. The pipeline uses QLoRA: the base model stays frozen in 4-bit quantization, and training updates only a small LoRA adapter (low-rank matrices attached to the attention and MLP projections). The input is a JSONL dataset of teacher LM calls, and the output is one adapter directory that vLLM can serve next to the base model.

This is finetune pilot 1: the training loop works end to end and is proven by a smoke test, but the pipeline has no evaluation stage of its own yet.

The implementation splits into two halves:

• mymodel/tune/ holds the pure, unit-tested half: the run config (config.py), the trace dataset schema (dataset.py), and the machine prechecks (precheck.py). These modules use only the standard library plus pydantic.

• pods/tune/ holds the GPU half: a standalone uv project whose train.py entrypoint runs the actual training with Unsloth and TRL. It imports the mymodel.tune modules across the package boundary through an explicit sys.path bootstrap.

The two halves live in separate environments on purpose. Unsloth pins torch and transformers tightly, and those pins cannot co-resolve with my-model’s vLLM dependency tree under the repo’s exclude-newer guard. See Container Deployments for the pod layout.

II How a run works

One training run performs these steps in order:

1. Check that the machine is idle enough to train (see §VI).

2. Load and validate the trace dataset. Malformed lines are reported and skipped.

3. Load the base model in 4-bit through Unsloth’s FastLanguageModel.

4. Attach LoRA adapters to the Qwen/Llama projection modules with gradient checkpointing.

5. Render each conversation through the model’s own chat template into plain text.

6. Train with TRL’s SFTTrainer in bf16.

7. Save the adapter and tokenizer, then write a run manifest (see §VII).

Heavy imports load lazily, so --help and the precheck return instantly even on a cold environment.

III The dataset: teacher-trace JSONL

The dataset is a JSONL file with one teacher LM call per line, in the format that critic export-traces (corpus repo) writes:

{"messages": [{"role": "system", "content": "You are a code reviewer."}, {"role": "user", "content": "Review this diff."}], "completion": "Looks solid.", "meta": {"ref": "ex-1", "model": "deepseek-chat", "cost_usd": 0.01}}

Each record carries three fields:

Field	Content
messages	The chat turns the teacher received (roles: system, user, assistant, tool). At least one.
completion	The text the teacher returned. Must not be empty.
meta	Provenance: example ref, teacher model, and cost_usd. All optional.

During training, to_training_messages appends the completion as the final assistant turn, which makes it the supervised target. Unknown extra keys are ignored, but unknown roles and empty completions fail validation.

The loader load_traces returns the valid records plus a list of per-line error strings. train.py prints the count of skipped malformed lines and continues. It exits only when no valid records remain. The trace_stats function summarizes the corpus (record count, distinct refs, character totals, per-model counts) for the run manifest.

The exporter writes train-split examples only. The frozen holdout stays unseen for critic eval, so do not add holdout traces to a training dataset.

IV Run configuration: TuneConfig

One small file declares a run completely, so runs are reproducible and reviewable. TuneConfig.from_file reads JSON, or YAML when pyyaml is installed. The main package environment has it, and the lean pod environment should use JSON.

Three fields are required:

Field	Meaning
base_model	HuggingFace ID of the student base. Must match the serving base exactly. There is no default.
adapter_name	Name of the produced adapter. Also its output subdirectory.
dataset_path	Path to the trace JSONL. ~ expands at load time.

The remaining fields carry recipe defaults:

Field	Default	Meaning
output_dir	~/local/models/loras	Parent directory for adapter output.
lora_r	16	LoRA rank.
lora_alpha	32	LoRA alpha. The default keeps the 2x-rank ratio from the template.
lora_dropout	0.05	LoRA dropout.
max_seq_len	8192	Maximum training sequence length. The binding VRAM knob on a 24 GB card.
epochs	1.0	Passes over the dataset. Ignored when max_steps is set.
max_steps	none	Hard step cap. Smoke runs set this small.
learning_rate	2e-4	AdamW peak learning rate.
batch_size	2	Per-device micro-batch size.
grad_accum	8	Gradient-accumulation steps.
seed	3407	RNG seed for the whole run.
load_in_4bit	true	QLoRA (4-bit base) instead of bf16 LoRA. 4-bit is what fits models of 4B parameters and up locally.

The effective batch size is batch_size * grad_accum. Unknown keys fail validation, so a typoed field name cannot silently train with defaults. Numeric bounds are validated as well: for example lora_r must be at least 1 and max_seq_len at least 128.

A minimal run config looks like this:

{
    "base_model": "Qwen/Qwen3-4B-Instruct-2507",
    "adapter_name": "example-adapter",
    "dataset_path": "~/local/traces/critic-train.jsonl",
    "max_steps": 200
}

V Running a training run

The Taskfile exposes three tasks, all rooted at pods/tune/:

1. Resolve the pod environment with task tune:sync. The first run downloads torch, so it is large.

2. Prove the loop with task tune:smoke. This runs a 5-step QLoRA on Qwen3-0.6B against a tiny synthetic dataset in a temporary directory.

3. Write a run config and start training with task tune:train -- --config runs/my-run.json.

On this box, wrap real runs in a memory scope (the house salt memory-guard rule):

systemd-run --user --scope -p MemoryMax=40G task tune:train -- --config runs/my-run.json

train.py takes these arguments:

Argument	Meaning
--config PATH	Path to a TuneConfig file (.json or .yaml).
--smoke	Run the synthetic 5-step loop instead. Mutually exclusive with --config.
--max-steps N	Override the step cap from the config.
--skip-precheck	Skip the busy-machine refusal. For cloud or burst runs where the box is not shared.

VI The machine precheck

Before any training, assert_ready refuses to start on a busy box. The training GPU (device 0) is shared with the desktop, and system RAM is the box’s real pressure point. The precheck reads nvidia-smi and /proc/meminfo, then raises PrecheckError listing every violated threshold:

Check	Refusal threshold
GPU memory	More than 6144 MiB in use
GPU utilization	More than 25 percent
System RAM	Less than 12 GiB available

The memory threshold tolerates the desktop (KWin/Xwayland hold about 2 to 3 GB) but rejects transient jobs such as marker OCR. Pass --skip-precheck only for cloud or burst runs on an unshared machine.

VII Outputs and serving the adapter

A run writes to output_dir / adapter_name (the TuneConfig.adapter_dir property). The directory holds the saved adapter, the tokenizer, and a run-manifest.json recording the full config, dataset stats, final training loss, step count, wall-clock seconds, peak VRAM, and the adapter path.

The default output_dir sits under the HuggingFace cache path, so adapters ride the HF-cache bind mount into the local vLLM container. To serve one, start vLLM on the exact student base with LoRA enabled (the enable_lora build arg, see Serving with vLLM), then load the adapter at runtime:

VLLM_MODEL=org/exact-base-checkpoint VLLM_TOOL_PARSER=hermes task vllm:up
curl -s http://localhost:8000/v1/load_lora_adapter \
  -H "Authorization: Bearer $LOCAL_VLLM_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"lora_name": "example-adapter", "lora_path": "/root/.cache/huggingface/loras/example-adapter"}'

Requests then select the adapter with "model": "example-adapter". Adding a gateway alias for an adapter is a separate, human-release-gated change under policies/versioning.md.

VIII Limits

• The LoRA target modules are fixed to the Unsloth-recommended Qwen/Llama set (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj). Other architectures need a code change, not a config change.

• The base_model must match the serving base checkpoint exactly, or the adapter will not apply.

• The pipeline has no evaluation stage. Quality measurement happens outside it, through critic eval on the frozen holdout.

• The smoke test proves the mechanics on a 0.6B model. It says nothing about adapter quality on real traces.