Should a Small Web Team Ever Self-Host an Open Model?

Should a Small Web Team Ever Self-Host an Open Model?

As of September 2026, usually no, but the exceptions are more real than they were a year ago. Open-weight models good enough for production work now fit on one GPU, and their licences allow commercial use. What has not changed is that running inference is an operations job, and most marketing teams do not have one.

The question keeps coming up because the models got genuinely good and the download numbers got very large. Hugging Face's own API reported just under 100 million all-time downloads for OpenAI's gpt-oss-20b, and about 6.8 million in the trailing 30 day window, when we checked on 27 September 2026.

So the interest is real. Here is how we think about whether it applies to a company whose core product is not AI infrastructure.

What Does "Open Weight" Actually Mean?

It means you can download the model's parameters and run it on hardware you control. It does not mean you can see how it was trained, and it does not automatically mean the licence is a standard open-source one.

The practical consequence is that the model stops being a service and becomes a file. Nobody can deprecate it out from under you, nobody changes its behaviour on a Tuesday, and no prompt leaves your network. Those three properties are the entire case for self-hosting, and they are worth real money to some companies and nothing at all to others.

The cost is that everything a provider used to handle becomes yours: capacity, uptime, upgrades, quantisation choices, batching, and the pager.

What Can You Genuinely Run on One GPU Now?

Quite a lot, and this is the part that changed. OpenAI's gpt-oss model card states that gpt-oss-120b has 117 billion parameters with 5.1 billion active, and that MXFP4 quantisation of the mixture-of-experts weights makes it "run on a single 80GB GPU (like NVIDIA H100 or AMD MI300X)."

The smaller sibling is more interesting for most teams. gpt-oss-20b has 21 billion parameters with 3.6 billion active, and OpenAI's card says it runs "within 16GB of memory" at the same quantisation. That is a workstation card, not a data centre.

There are several credible families, not one. Checking Hugging Face on 27 September 2026, Qwen3-32B was published in April 2025 under Apache 2.0, DeepSeek-V3.1 in August 2025 under MIT, Mistral Small 3.2 24B Instruct in June 2025 under Apache 2.0, and Gemma 3 27B Instruct in March 2025 under Google's Gemma licence. Newer versions of each family exist; the point is that this is a competitive field with permissive terms, not a single option.

Is the Licence Really Open?

Check it per model, because they differ, and the difference matters to your lawyers. gpt-oss-20b is Apache 2.0, which OpenAI's own model card describes as letting you "build freely without copyleft restrictions or patent risk." DeepSeek-V3.1 is MIT. Both are plain, well-understood open-source licences.

Gemma 3 is not. It ships under Google's own Gemma licence, which carries use restrictions that Apache and MIT do not. That is not a criticism of the model, which is good. It is a warning that "open weights" and "open source licence" are two different claims, and product teams routinely collapse them.

If you are going to depend on a model commercially, read its actual licence file once, properly, before you build. It is a twenty minute job that prevents a bad conversation later.

What Does Self-Hosting Cost That an API Does Not?

Five things, and only the first is obvious. Hardware, either bought or rented by the hour, paid whether or not you are using it. Serving infrastructure, which usually means running something like vLLM and understanding batching and context limits. Upgrades, because a new model version is a new deployment rather than a version string. Observability, because you now own latency and error rates. And the person, because all of the above needs an owner.

The economics are counterintuitive for low-volume use. An API charges per token, so a quiet automation costs almost nothing. A dedicated GPU costs the same at 3am with zero traffic. Self-hosting tends to win on unit cost only when utilisation is high and steady, which describes very few marketing automations.

The related trap is thinking you are avoiding lock-in. You are trading provider lock-in for stack lock-in, which we picked apart in avoiding AI provider lock-in.

When Does Self-Hosting Clearly Win?

Four situations, in our experience. When data genuinely cannot leave your network, because of a contract, a regulator, or a customer security review that will otherwise fail. When you need a frozen model that behaves identically in twelve months, which matters for anything audited. When volume is high and steady enough that per-token pricing has become the largest line in your bill. And when you are fine-tuning heavily enough that owning the weights is the point.

Notice that three of those four are compliance or contractual, not technical. That is the honest pattern. Most companies that should self-host know it because someone in procurement told them, not because they benchmarked latency.

The one technical case worth adding: a model doing something narrow, high-volume and boring, like classification or extraction, where a small model on cheap hardware beats paying frontier prices. We looked at that trade in small models against frontier models.

When Is It Clearly the Wrong Call?

When the motivation is cost and the volume is low. When nobody on the team has run a service with an on-call rotation. When the workload is spiky, because you will either over-provision or fall over. And when what you actually want is the newest frontier capability, which by definition is not the thing you can download.

There is also a quieter failure mode. A self-hosted model becomes the team's pet. It gets built, it works, one person understands it, and then that person leaves. An API integration in that situation is a file anyone can read. A GPU cluster is not.

We have watched small teams spend a quarter standing up inference for a use case a hosted endpoint would have covered in an afternoon. The models were not the problem. The time was.

What Is the Middle Option Most Teams Miss?

Someone else hosting the open model for you. You get the open weights, the permissive licence and the ability to move, without owning a GPU. Several inference providers serve the same open checkpoints behind an API, and because the model is a public artefact you can in principle move between them.

This is the option we recommend most often, and it is the one nobody asks about, because the debate is usually framed as self-host or closed API. It is neither. It is a hosted open model, and it keeps the substitutability that made open weights attractive in the first place.

Putting a gateway in front of it makes the substitution real rather than theoretical, which is the pattern we described in using an AI gateway on a web team.

What Should You Check Before You Trial Anything?

Read the model card, not a blog post about the model. OpenAI's gpt-oss card, for instance, is explicit that both models "were trained on our harmony response format and should only be used with the harmony format as it will not work correctly otherwise." That is a genuine integration constraint you would not guess from a benchmark table.

The same card notes that the models expose the full chain of thought to help with debugging, and says plainly it is "not intended to be shown to end users." If your plan was to stream reasoning into a product surface, that is a decision the vendor has an opinion about.

Then pick one narrow task, run it through the hosted version of the open model first, and compare output quality against whatever you use now. If quality fails there, hardware will not save it, and you have spent two days instead of two months.

So What Would We Actually Do?

For a normal B2B SaaS marketing team, we would not buy a GPU. We would pick an open-weight model with a plain licence, run it through a hosted endpoint, keep every call behind a thin internal interface so the provider is swappable, and revisit self-hosting only when a security review or a genuinely large bill forces the question.

For a company with a compliance constraint that rules out third-party inference, we would self-host, budget for an owner as a named role rather than a side project, and choose a permissively licensed model specifically so the legal review is short. The model is the easy decision in that scenario. The operations are the project.

The encouraging thing is that this is now a real choice with real options at both ends, which was not true two years ago. The discouraging thing is that "we could run our own model" has become a way to avoid deciding what the model is for.

If you are weighing this for a specific workload and want a straight answer rather than a hype cycle, we are happy to talk it through at phoenix.studio.