103. Model Weights 2 - Tensors, Layers and Checkpoints

The lesson

Weights are not a flat list; they are tensors with shapes. A linear layer holds a matrix that maps an input vector of one width to an output of another, so its shape is output width by input width. Attention holds separate projection matrices for queries, keys and values. Embeddings hold one vector per vocabulary entry. Reading a model's shape list is the fastest way to understand where its memory actually goes.

The embedding table is often the surprise. A 100,000-token vocabulary at 4,096 dimensions is about 410 million weights before anything else, and some models tie the output projection to it. Attention projections across dozens of layers add up quickly too. Feed-forward layers usually hold the majority of the parameters, which is why mixture-of-experts designs can have enormous totals while activating only a fraction per token.

A checkpoint is a directory or a file holding those tensors plus a config that describes the architecture and a tokenizer. Formats differ mainly in how tensors are laid out and named: safetensors stores a header of shapes and offsets followed by raw data, which allows memory-mapping so the operating system loads only the pages actually used. That is why a 40 GB model can start on a machine with less free RAM than its file size, and then fail later when it touches everything.

Quantized checkpoints are the same structure with different dtypes, plus scale tensors alongside. A 4-bit file still contains a tensor per layer; the values are packed, and the loader unpacks or the kernel reads them packed. This is why you cannot simply rename a Q4 file to Q8 or edit one in place: the layout and the metadata both encode the precision.

Practically, the shape view answers real questions. If a model supports a longer context, look for the position-encoding configuration. If it runs out of memory during generation rather than load, suspect the KV cache and its per-layer shapes. And if two checkpoints claim the same parameter count but different file sizes, the difference is precision, not architecture: compare the config's dtype fields before believing either label.

Check yourself