Bottom line up front
Self-hosting is a memory decision before it is a speed decision. Work out whether the model's weights fit in the memory you have at the quantisation you intend to use, then add the context window, key-value cache, and runtime overhead, then check whether the model's licence and release notes actually permit your use. Published figures are a starting estimate, not a specification: confirm against the model's own release notes and measure on your own hardware before committing.
Size the memory budget before anything else
The first constraint is whether the weights fit. A commonly published rule of thumb is roughly half a gigabyte of memory per billion parameters at 4-bit quantisation, with full precision roughly doubling that. That figure is an estimate that circulates widely in community guides, not a specification, and it varies with architecture and quantisation scheme. Treat it as a starting point and confirm against the model's own release notes.
Quantisation is the main lever. A model that is impractical at full precision can become practical at 4-bit, with a quality trade-off that is real but model-specific. The useful question is not which quantisation is best in general, but which one your hardware can hold at a context long enough for your workload.
- Start from the parameter count and the quantisation you actually intend to use.
- Convert to a memory figure and compare it against available VRAM or unified memory.
- Record where the figure came from and whether it is a vendor claim or a community estimate.
Weights are only part of what your machine must hold
A model file that barely fits can still fail in practice, because the memory budget includes the weights plus context, the key-value cache, runtime buffers, and any concurrent requests. Longer context windows increase the cache roughly in proportion, so a configuration that works at a short context can exhaust memory at a longer one. If memory is tight, capping context is often a more honest trade-off than reducing model size.
Memory-mapping and offloading options let a runtime spill to system RAM or disk, which makes a model runnable without making it usable: hybrid execution where some layers run on the CPU can reduce throughput substantially. Runnability and usability are different thresholds, and conflating them is the most common way a self-hosting plan disappoints.
Treat throughput as something you measure, not something you look up
Published tokens-per-second figures vary by hardware, quantisation, context length, prompt length, and batching, and vendor or blog numbers are usually produced on a favoured configuration. They are useful for ballparking whether a runtime class is viable on your machine; they are not a prediction of what you will get. A small measured test on your own hardware is worth more than several comparison tables.
Record the conditions alongside any number you write down. A throughput figure without the hardware, quantisation, and context length attached is not evidence and cannot be compared to anything later.
- Run a short test on the actual hardware before buying or committing.
- Record hardware, quantisation, context length, and measurement method with the result.
- Separate interactive use from batch use, which have different acceptable speeds.
Check the licence and release notes, not just the model card
A downloadable model is not automatically a usable one. Licence terms, permitted use, attribution requirements, and commercial restrictions vary between models, and they are established by the licence file and the model's release notes rather than by the repository being public. Read the actual licence for the weights you intend to download, and keep a copy with the version you adopted.
This is where a directory helps and where it stops helping. A catalogue can surface which models exist and what a listing claims; it cannot determine whether a specific licence permits your use. That determination belongs with the licence text and, where the answer matters commercially, with a qualified adviser.
- Locate and read the licence for the exact weights and quantisation you plan to run.
- Record the model version, licence, and the date you checked it.
- Check the release notes for restrictions, known issues, and hardware guidance.