API vs Self-Hosted Open-Weight SLMs: Weighing Cost, Latency, Privacy and Quality to Justify a Choice
You have a shortlist of models and evaluation results. The last decision is how to run the model: call a hosted API, or run open weights yourself, often as a small language model (SLM). This is the question leads and customers ask most often, and the one interviewers use to test judgment. A good answer is not 'APIs are easier' or 'open source is cheaper'. It is a trade-off grounded in the customer's numbers and constraints.
1. The deployment spectrum
This is not a binary choice. There are five common options, from least to most control:
| Option | What it is | You manage |
|---|---|---|
| Closed-model API | Frontier models available only through the vendor's API | Prompts, keys, cost |
| Hosted open-weight API | Open models served by an inference provider, such as Hugging Face Inference Providers | Prompts and model choice |
| Cloud-managed or dedicated endpoint | A model deployed for you in a chosen cloud region, sometimes on dedicated hardware | Region, scaling settings, some configuration |
| Self-hosted in your VPC or data centre | Open weights served on your own GPUs with vLLM, TGI or similar | Everything: hardware, serving, scaling, upgrades, monitoring |
| On-device or edge | A quantized SLM running on a laptop, phone or edge box with llama.cpp, Ollama or similar | The device fleet and model updates |
SLMs are usually models of a few billion parameters, often under about 10B. Quantized, they run on a single GPU, a CPU or even a laptop. They are weaker than frontier models in general, but on a narrow, well-defined task, especially after fine-tuning, they can be competitive.
2. The four dimensions
Cost
- APIs charge per token. There is no fixed cost, so they are ideal for prototypes, low or spiky traffic, and uncertain demand.
- Self-hosting has a fixed cost (GPU hours whether used or not) plus engineering and operations time, which teams often forget to count.
Fix a real ticket in a small codebase you didn't write: read the code, change only the functions you're given, pass the tests, then submit for hidden checks and a score.
Open the lab →