AirLLM: Run very large language models on a small GPU

What it does

A language model is built from many stacked layers. Instead of loading the whole model into GPU memory (VRAM), AirLLM first splits it into separate layers saved on disk, then loads and runs them one at a time. The creators say this lets 70B models run on a single 4GB GPU without quantization, distillation or pruning (common ways of shrinking a model), and report runs of DeepSeek V3 (671B) in 12GB. An optional 4-bit or 8-bit compression setting trades a little accuracy for speed.

How to use it

Input → action → output: in Python, call AutoModel.from_pretrained with a Hugging Face model ID or a local path, turn your prompt into tokens with the model's tokenizer, and call generate. You get the model's text reply back, the same way as with a regular Transformers model. Gated models need a Hugging Face token.

Common use cases

• Pass a Hugging Face model ID and a question such as the capital of the United States, and print the model's short text answer.
• Try a model far larger than your GPU memory on one consumer graphics card to compare its answers with a smaller model.
• Fine-tune adapters on a large Qwen model with the training example scripts while the frozen base weights stream from disk.