Model Catalogue #

The catalogue is the single source of truth for every inference model myModel knows about. Instead of hardcoding model paths and vLLM flags in serving code, you declare one ModelCard per model in a YAML file. A Registry loads those cards, enriches them with HuggingFace metadata, and the serving layer (serve.py) reads them to construct vllm serve launch commands.

What is the Model Catalogue?#

Every model — local or remote — is a ModelCard, a pydantic record with identifiers, type information, provenance metadata, and runtime parameters. Cards are grouped into YAML catalogue files that the Registry loads into memory at startup.

Two seed catalogues ship with the repo:

File	Contents
`mymodel/catalogue/data/optimizers.yaml`	Generative / transformer models (`CHAT`, `CODE`, `DRAW`, …)
`mymodel/catalogue/data/classifiers.yaml`	Reductive models (`EMBED`, `SCORE`, `SEGMENT`, …)

You can point the Registry at any YAML file, so you can maintain a private catalogue for experimental models without touching the seeds. The flow is:

YAML catalogue → Registry (loads ModelCards) → serve.py (builds vllm serve) → vLLM subprocess

The Ontology: Verbs, Modalities, and Sizes #

Every model is placed in the ontology by two coordinates: its Verb (what it does) and its modality pair (what it consumes and produces). A Size bucket records how big it is.

Verbs #

A Verb is a model’s function — the input-to-output transformation it performs. Each verb carries a default (input, output) modality pair inherited by every card with that verb (unless overridden).

Verb	Input → Output	What it does
`OMNI`	any → any	Omnimodal (any-to-any)
`EMBED`	text → vector	Text → embedding vector
`CHAT`	text\|image → text\|image	Conversational chat (multimodal)
`CODE`	text\|code → code	Code generation
`COMPLETE`	text → text	Text completion
`SUMMARIZE`	text → text	Summarization (collapses to `COMPLETE`)
`SEE`	image → text	Image captioning / VQA
`DRAW`	text → image	Text-to-image
`REDRAW`	image → image	Image-to-image
`LISTEN`	audio → text	Speech-to-text
`SPEAK`	text → speech	Text-to-speech
`PLAY`	text → music	Text-to-music
`REPLAY`	audio → audio	Audio-to-audio
`ANIMATE`	image\|audio → video	Image+audio → video
`SHOOT`	text → video	Text-to-video
`SCULPT`	text\|image → facets	3D mesh generation
`TRACK`	text\|image → points	3D pointcloud tracking
`BUILD`	text\|image → space	3D scene construction
`OCR`	pdf\|math → text	Document OCR
`PARSE`	html\|wiki → text	Web/wikitext parsing
`TREND` / `DETREND` / `RETREND`	text↔series / series↔series	Time series
`CHART` / `DECHART` / `RECHART`	text↔graph / graph↔graph	Visual graphs
`DESERIALIZE` / `SERIALIZE`	text↔json	Text ↔ JSON
`LABEL_TEXT` / `SCORE_TEXT`	text → class/score	Text classification/scoring
`LABEL_IMAGE` / `SCORE_IMAGE`	image → class/score	Image classification/scoring
`SEGMENT`	image → segments	Image segmentation
`SHADE`	image → regions	Depth / region mapping
`LABEL` / `SCORE`	any → class/score	Universal classification/scoring
`ENCODE`	text → model	Trainable encoder
`TRAIN`	model → model	Training step

Modalities #

A Modality is a Flag enum — a datatype a model can consume or produce. Models cluster by (input, output) modality pairs. Composite flags combine related atomic modalities:

Composite	Members
`BASIC`	`TEXT \| CODE`
`IMAGE`	`PHOTO \| ILLUSTRATION \| DATAVIZ \| FACE`
`AUDIO`	`SPEECH \| MUSIC`
`VIDEO`	`AVATAR \| CLIP`
`SPACE`	`FACETS \| POINTS \| SCENE`
`DOCUMENT`	`PDF \| MATH \| WIKI \| HTML`
`STRUCTURE`	`JSON \| SVG \| GRAPH \| SERIES`
`LABEL`	`CLASS \| SCORE \| VECTOR \| SEGMENTS \| REGIONS`
`MODEL`	`CLASSIFIER \| OPTIMIZER`

The full set of atomic modalities is: ANY, TEXT, CODE, PHOTO, ILLUSTRATION, DATAVIZ, FACE, SPEECH, MUSIC, AVATAR, CLIP, FACETS, POINTS, SCENE, PDF, MATH, WIKI, HTML, JSON, SVG, GRAPH, SERIES, CLASS, SCORE, VECTOR, SEGMENTS, REGIONS, CLASSIFIER, OPTIMIZER.

Parsing modality pairs #

Modality pairs are written as "input -> output" strings in YAML. The PAIR_RGX regex splits on ->, to, or 2 (case-insensitive): Modality.read_pair("text -> image") returns (Modality.TEXT, Modality.IMAGE). When a card omits modality, it inherits the verb’s default pair.

Sizes #

A Size is a coarse parameter-count bucket, parsed from a model’s _xs…_xl name suffix:

Size	Suffix	Typical scale
`XS`	`_xs`	< 1B params
`SM`	`_sm`	1–7B params
`MD`	`_md`	7–13B params
`LG`	`_lg`	13B+ params
`XL`	`_xl`	70B+ params

Size.get_size_suffix("gemma_md") returns Size.MD. Sizes are ordered (XS < SM < MD < LG < XL), enabling size-based filtering and display.

The ModelCard #

A ModelCard is a pydantic model — the persisted, HuggingFace-backed metadata record for one inference model. Here is the full field reference, grouped by category.

Identifiers #

Field	Type	Default	Description
`myname`	`str`	(required)	System name (e.g. `gemma_md`). Must be unique.
`hfname`	`str`	(required)	HuggingFace repo id (e.g. `google/gemma-7b-it`).
`family`	`str`	`''`	Family name for grouping related variants.

Type #

Field	Type	Default	Description
`verb`	`Verb`	`CHAT`	What the model does. Inherited from the YAML top-level key.
`modality`	`tuple[Modality, Modality]`	verb default	`(input, output)` modality pair. Inherited from `verb` if omitted.
`engine`	`str`	`''`	Model framework (`transformers`, `diffusers`, `remote`, etc.).
`hftask`	`str`	`''`	Auto-generated HuggingFace pipeline tag.
`usecase`	`str`	`''`	Manually-entered description of the model’s purpose.

Local-specific #

Field	Type	Default	Description
`size`	`Size`	`MD`	Parameter-count bucket. Auto-detected from `myname` suffix if omitted.
`f_size`	`str`	`''`	Total file size (e.g. `"9GB"`).
`params`	`str`	`''`	Parameter count (e.g. `"7B"`).
`weight`	`int`	`0`	Datatype bits (e.g. `16` for fp16, `4` for 4-bit).

Remote-specific #

Field	Type	Default	Description
`i_cost`	`float`	`0.0`	Input cost per million tokens.
`o_cost`	`float`	`0.0`	Output cost per million tokens.

Quantization-specific #

Field	Type	Default	Description
`parent`	`str`	`''`	HF repo of the parent (unquantized) model, for metadata lookup.
`target`	`str`	`''`	Specific quantized file to load from the repo.

Social metadata #

Field	Type	Default	Description
`c_time` / `m_time`	`date`	epoch	Creation / last-modified date on HuggingFace.
`usages` / `hearts`	`int`	`0`	HuggingFace download count / likes.
`license`	`str`	`''`	License identifier.
`papers`	`set[str]`	`set()`	arXiv paper IDs.
`tags`	`set[str]`	`set()`	Descriptive tags (library tags filtered out).

Architecture config (`HFConfig`)#

Field	Type	Default	Description
`m_type` / `m_auto`	`str`	`''`	`config.json` `model_type` / `transformers_info.auto_model` class.
`processor`	`str`	`''`	Processor class (e.g. `AutoTokenizer`).
`libraries`	`set[str]`	`set()`	Known HuggingFace library tags.
`pipeline` / `pipeline_tag`	`str`	`''`	Pipeline tags.
`architecture`	`str`	`''`	Architecture class name (e.g. `LlamaForCausalLM`).
`context_window`	`int`	`0`	Context window size.

Runtime parameters #

Field	Type	Default	Description
`local`	`bool`	`False`	Whether the model is locally hosted.
`build_args`	`dict`	`{}`	Engine-init kwargs. Become `vllm serve` CLI flags.
`infer_args`	`dict`	`{}`	Inference-time params (`temperature`, `max_tokens`, etc.).
`callbacks`	`list`	`[]`	Async callbacks executed at lifecycle points.
`files`	`list[str]`	`[]`	Model files on HuggingFace.

`build_args` and `infer_args`#

build_args are engine-init kwargs that serve.py translates into vllm serve CLI flags (e.g. dtype: half → --dtype half, trust_remote_code: true → --trust-remote-code). infer_args are inference-time parameters (temperature, max_tokens, etc.) consumed by clients, not the vLLM launcher.

Cards are nested under their verb key (top-level YAML), optionally grouped into families, with shared fields hoisted to the parent node.

The Registry #

The Registry class owns one YAML catalogue file plus its live ModelCard instances and a RegistryConfig.

Creating, listing, and looking up #

from mymodel.catalogue import Registry, OPTIMIZERS, RegistryConfig, DEFAULT_MODEL_DIR

registry = Registry(OPTIMIZERS)          # from a seed catalogue path
registry = Registry(Path('~/my/catalogue.yaml').expanduser())  # or any YAML file

for myname, card in registry.models.items():   # iterate all cards
    print(f'{myname}: {card.hfname}')

card = registry.models['gemma_md']              # look up by name

OPTIMIZERS and CLASSIFIERS are pre-exported Path constants pointing at the seed YAML files in mymodel/catalogue/data/. DEFAULT_MODEL_DIR is ~/local/models — where local model weights are stored.

Resolving layered args #

The resolve_args method layers defaults over a card’s own build_args and infer_args in the order: default → verb → family → myname. This lets you set engine defaults once and override per-verb, per-family, or per-model:

build_args, infer_args = registry.resolve_args(card)

HuggingFace integration #

await registry.refresh()  # re-fetch metadata for every local model

See HuggingFace Integration below for details.

Adding a new model #

await registry.add(hfname='meta-llama/Llama-3.2-3B-Instruct',
                    myname='llama_sm', verb='chat', family='llama')

add() creates a ModelCard, fetches HuggingFace metadata, registers it, and rewrites the YAML index.

Saving changes #

registry.write_index()  # serialize self.models back to the YAML file

write_index distills shared fields back into family and verb nodes, sorts cards by (verb, family, myname, size), and writes canonical YAML with blank separator lines.

Creating a New Model Entry #

Step 1: Choose a model from HuggingFace #

Find the model on huggingface.co. Note the repo id, e.g. Qwen/Qwen2.5-7B-Instruct.

Step 2: Determine its verb and modality pair #

For a conversational text model, the verb is CHAT (text/image → text/image). For a code model, CODE (text/code → code). For an embedding model, EMBED (text → vector).

Step 3: Determine its size category #

Use the parameter count to pick a Size: ~7B params → MD (_md). Name your card with the suffix: qwen_md.

Step 4: Write the ModelCard YAML entry #

Add the card under the appropriate verb in your catalogue file:

CHAT:
    families:
        qwen:
            engine: transformers
            license: apache-2.0
            qwen_md:
                hfname: Qwen/Qwen2.5-7B-Instruct
                size: md
                params: 7B
                weight: 16
                build_args:
                    dtype: half
                    gpu_memory_utilization: 0.9

Step 5: Test that the Registry loads it #

from mymodel.catalogue import Registry, OPTIMIZERS
r = Registry(OPTIMIZERS)
card = r.models['qwen_md']
print(card.verb, card.modality, card.hfname)
# CHAT (TEXT|IMAGE, TEXT|IMAGE) Qwen/Qwen2.5-7B-Instruct

Step 6: Test that serve.py can launch it#

uv run python -m mymodel qwen_md
curl http://localhost:5000/models   # check the management API
curl http://localhost:5001/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"qwen_md","messages":[{"role":"user","content":"Hello"}]}'

Step 7: Adjust build_args for your GPU #

If the model OOMs or is slow, adjust build_args — limit max_model_len to save VRAM, lower gpu_memory_utilization, etc.

Using the Catalogue with serve.py #

The serving layer (mymodel/serve.py) is a catalogue-driven vLLM launcher. It does not hardcode any model — everything comes from the Registry.

Model resolution #

Semioserve.__init__ takes a list of model specs (names or VERB:name pairs), loads the catalogue via Registry(catalogue), and resolves each name to a ModelCard:

self.registry = Registry(catalogue)
for spec in models:
    myname = spec.split(':', 1)[-1]
    card = self.registry.models[myname]

The _to_flags function translates a card’s resolved build_args dict into a vllm serve CLI flag list. Boolean True values become bare flags; False/None are skipped; all other values are stringified. So {"trust_remote_code": True, "dtype": "half"} becomes ['--trust-remote-code', '--dtype', 'half']. The full vllm serve argv also includes --served-model-name (the card’s myname), --host, --port, and optionally --api-key.

build_args → vLLM flags #

OpsStance: dev vs prd #

OpsStance is an enum with two members:

DEV — disables JWT auth on the management API, enables interactive prompts
PRD — enables AuthMiddleware (JWT auth) on the management API

The stance is selected via the --dev_mode CLI flag or the NUCLEUS_MODE environment variable.

Management API #

Semioserve exposes a thin Quart management API on the main port:

GET /models — active servers (with live health) and the full catalogue
GET /health — returns {"status": "ok", "models": <count>}

In production mode, these routes are wrapped in AuthMiddleware (JWT validation).

See Serving with vLLM for the full serving guide.

HuggingFace Integration #

The Registry fetches metadata from HuggingFace to enrich cards with social, provenance, and technical fields.

How it works #

The huggingface_hub library’s hf_api.repo_info(hfname, files_metadata=True) call retrieves a ModelInfo object. The registry splits this into two parse passes: parse_hf_metadata (social/provenance: creation date, last-modified date, download count, likes, license, parameter count) and parse_hf_data (technical: tags, papers, pipeline tag, engine, architecture, model type, auto-model class, processor, file listing).

Caching #

Fetched ModelInfo objects are pickled to ~/.local/models/hf_cache/<myname>.pkl. The refresh_days config (default 7) controls freshness. If you are offline, the registry will use cached pickles within that window; if no cache exists, refresh_hf_data will fail — but the YAML catalogue itself carries enough metadata for serve.py to launch models.

registry = Registry(OPTIMIZERS, RegistryConfig(refresh_days=30))
await registry.refresh()  # re-fetch stale cards

DEFAULT_LIBRARIES #

The DEFAULT_LIBRARIES frozenset contains well-known HuggingFace library tags (transformers, safetensors, pytorch, diffusers, etc.). During tag parsing, the registry partitions info.tags into library tags (matching this set) and descriptive tags. Library tags go into config.libraries; descriptive tags go into card.tags. arXiv paper IDs (matching arxiv:<digits>) are extracted into card.papers.

Step 6: Test that serve.py can launch it#

Using the Catalogue with serve.py#

Using the Catalogue with serve.py #