How to Pick the Right Open-Source LLM for Your Project

Pick the license first, then the size your hardware can hold, then test two or three candidates on your own tasks. Most "open-source LLMs" are open-weight models: you can download and run the weights, but the license may add conditions, and the training data is usually not published. Those conditions matter more for a product than a few points on a benchmark.

This guide uses license and size data checked on October 13, 2026, from each model's Hugging Face page and its publisher's license text. Models change quickly, so confirm the model card before you commit.

Open source, open weights, and why the difference matters

The Open Source Initiative published version 1.0 of its Open Source AI Definition. It asks for the same freedoms as open-source software: use the system for any purpose without asking permission, study how it works, modify it, and share it. For a machine-learning system it also requires three things under OSI-approved terms: detailed information about the training data, the complete code used to train and run the model, and the model parameters.

Few popular models meet all of that. Most publish weights and inference code but not enough about the training data. In practice you are choosing between three kinds of license:

• Standard open-source licenses such as Apache-2.0 or MIT applied to the weights. You can use, modify, and ship the model commercially, with the usual notice requirements.
• Custom community licenses, such as Meta's Llama license or Google's Gemma terms for Gemma 3. They let you use, modify, and redistribute the model, but they add conditions, and some of those conditions travel with the model when you redistribute it.
• Fully open releases, where the publisher also releases training data and code. AI2's OLMo is the clearest example in this list.

The main options, checked October 2026

The Llama licenses have two terms that surprise teams late. If your products had more than 700 million monthly active users in the month before the model's release date, you must request a separate license from Meta. And if you distribute the model or a product built on it, you must show "Built with Llama," and any model you build with Llama or its outputs and then distribute must include "Llama" at the beginning of its name. Both terms are in the Llama 3.1 and Llama 4 license files Meta publishes.

How much hardware a model needs

A model's weights take roughly its parameter count multiplied by the bytes stored per parameter. At 16-bit precision that is two bytes per parameter, so an 8B model needs about 16 GB just for the weights. Quantized to 4 bits, the same model needs about 4 GB. Your runtime also needs memory for the context window (the KV cache) and its own overhead, so leave headroom on top of these numbers.

That arithmetic narrows the list quickly:

• A laptop or a single consumer GPU suits models up to roughly 8B to 14B when quantized, and gpt-oss-20b according to its card.
• A single 80 GB data-center GPU fits gpt-oss-120b and 70B-class models when quantized.
• DeepSeek-R1 at full size needs several GPUs.

Pick a runtime that matches how you will serve it

The model is only half the decision. The software that runs it decides how easy it is to operate.

• Ollama (MIT) is the simplest way to run models locally from the command line or an API. The gpt-oss model card itself uses Ollama as its consumer-hardware example.
• llama.cpp (MIT) is a C/C++ inference engine, listed by Ollama as its backend, and the one to use when you want control over quantization formats.
• vLLM (Apache-2.0) is built for high-throughput serving, which matters once several users or services share one GPU.
• LocalAI (MIT) gives you an OpenAI-style API for local models, which helps when existing code already calls the OpenAI API.

If people rather than programs will use the model, add a chat interface such as LibreChat or AnythingLLM. Our list of open-source ChatGPT alternatives compares the options for that layer.

A shortlist process that works

1. Write down the job in one sentence, such as "summarize support tickets in English and Spanish" or "answer questions from our docs."
2. Remove models whose license your legal team will not accept. If you ship to customers, this usually leaves Apache-2.0 and MIT models, or a community license you have read in full.
3. Remove models your hardware cannot hold, using the memory arithmetic above.
4. Run the two or three survivors on 30 to 50 real examples from your own data and compare the outputs side by side. Public benchmarks measure general skill; your examples measure your job.
5. Measure speed on your hardware with your context length, because speed usually falls as the context grows.
6. Keep the prompt and test set, so you can rerun the comparison when a new model version appears.