Picture published by the creator on GitHub.
Run huge mixture-of-experts models by streaming them from disk
Made by Vincenzo Fornaro
View profile on GitHub (opens in a new tab)An inference engine written in plain C that tries to run very large models — hundreds of billions to trillions of parameters — on hardware that could not normally hold them.
It treats graphics memory, system memory and your drive as one tiered pool and moves parts of the model between them as they are needed. That only works because these are mixture-of-experts models, where a small slice of the network handles each word and the rest sits idle. Nine model families are supported, one C file each, behind a shared chat, server and web interface. The project describes itself as a research platform as much as a tool.
You could use it to…
- Run a frontier-sized open model on hardware you already own
- Chat with a huge mixture-of-experts model through a web interface
- Watch live metrics on a dashboard while the model runs
Can I use this?
You'll need A capable desktop, fast storage, and a C toolchain · Setup needed
Worth knowing The project is explicit that it makes no promise about speed: with too little fast memory it simply gets slower. Its headline demonstration — a 744-billion-parameter model at about 4 tokens a second — was measured on six RTX 5090 cards, which is not consumer hardware in any ordinary sense. What it does guarantee is that it will never quietly reduce the model’s precision or change its routing to make something fit. Every performance figure here is the project’s own. Apache 2.0. BuildTube has not run or verified this software.
Why it's here
Not trending this week
Last seen on the GitHub trending list on 25 September 2026. It stays on BuildTube so you can still find it.
- 39,969 people have starred it on GitHub.
Numbers checked on 6 October 2026; not refreshed since.
Behind it
See the code on GitHubBuildTube has not run or verified this project. Everything above is written from what the creator published.
Made this?