Private LLM Deployment on Your Own Servers
An open-weight language model on servers you control: requests are answered on those servers, and the model is chosen on your own questions.
Private LLM deployment puts the language model on servers you control, next to your knowledge base and your bots, so requests are answered on those servers and go to no outside AI service. Your security team can assess it against your own rules. You pay for it in hardware, and in some answer quality: the open models that fit on one or two servers can answer less well than the strongest cloud models. The pilot measures how well they answer your own questions before you commit.
Whether you need a closed network at all is answered on the AI platform page. This page is for when your data must not go to an outside AI service: where the model can run, which open-weight models fit, what hardware they need and what it all costs a month.
Four places the model can run
The model can run on your own servers, on a dedicated server at a host your security team approves, on a rented GPU server in a data centre in your jurisdiction, or as a cloud model hosted in your jurisdiction.
| Your own servers (on-premise) | Dedicated server at an approved host | Rented GPU server in your jurisdiction | Cloud model in your jurisdiction | |
|---|---|---|---|---|
| Where your data sits | own servers: inside your perimeter | dedicated server: with that host, under your contract | rented server: with the data centre, under your contract | cloud model: with the cloud vendor |
| Hardware | you buy it from your supplier; we write the configuration | the host's | rented by the month | none |
| Time to get the server | the longest, because it includes buying the hardware | weeks | days | days |
| What you pay | hardware, power and colocation (rented rack space in a data centre) | monthly rent | monthly rent (see the hardware table below) | per token |
| Fits | strict "no data outside" rules | your security team accepts a host under contract | a pilot on test documents; production too, if your rules allow it | your data may leave the company: see cloud LLM setup |
Only the first route keeps the data on hardware you own, inside your perimeter (in your building or your own rack in a data centre); in the next two it sits with a host or data centre you have a contract with. The routes combine well: pilot on a rented server with non-sensitive documents, then move inside your perimeter once the pilot passes.
Open-weight models compared
An open-weight model is one whose maker publishes the model itself, so you can run it on your own hardware. It runs on a GPU, a graphics card with its own memory, and that memory decides which models fit. Every model below is published under the Apache 2.0 licence, which permits commercial use (checked on each model's page on 23 October 2026). "B" is billions of parameters: a larger model usually answers better and needs more memory.
| Model (maker) | Size | Smallest GPU it fits |
|---|---|---|
| Qwen3.5-9B (Alibaba) | 9B | fits one 24 GB GPU |
| gpt-oss-20b (OpenAI) | 21B, mixture of experts | fits one 24 GB GPU (the maker says it runs within 16 GB) |
| Mistral Small 3.2 (Mistral AI) | 24B | fits one 24 GB GPU, compressed to 4-bit |
| Qwen3.6-27B (Alibaba) | 27B | fits one 24–32 GB GPU, compressed to 4-bit |
| Gemma 4 31B (Google) | 31B | fits one 24–32 GB GPU compressed to 4-bit; one 80 GB GPU uncompressed |
| Qwen3.6-35B-A3B (Alibaba) | 36B, mixture of experts | fits one 24–32 GB GPU, compressed to 4-bit |
| gpt-oss-120b (OpenAI) | 117B, mixture of experts | fits one 80 GB GPU (the maker's figure) |
| Qwen3.5-397B-A17B (Alibaba) | 397B, mixture of experts | needs a server with eight 80 GB GPUs compressed to 8-bit, or about four at 4-bit (our arithmetic from the published size) |
The GPU sizes count the model only; the text it reads and the number of users add to them. A mixture-of-experts model uses only part of itself for each word, so it answers faster than its size suggests, but the whole model still has to fit in memory. Compressed to 4-bit means a smaller copy that needs about a quarter of the memory of the uncompressed model for the model itself, and usually loses a little quality.
The maker's country can matter to your security team. If it rules out models by origin, that becomes a requirement in the sizing spec, and the table already has options from the US, Europe and China. A ranking of models narrows the list to two or three candidates; 20–30 questions from your own work decide between them.
Hardware for a private LLM, and what it costs a month
| What the AI does | Model class | Smallest GPU | Rent |
|---|---|---|---|
| Answers staff or customer questions from your documents | up to about 10B uncompressed, or up to 21B compressed (for example gpt-oss-20b) | 24 GB (RTX 4090 class) | rent about $640 a month |
| Better answers, documents with tables, calls to your systems | 24–36B, compressed to 4-bit | 24–32 GB (RTX 4090 / 5090 class) | rent about $640–860 a month |
| The same with lighter compression, longer documents, more users at once; or a 120B mixture-of-experts model its maker ships compressed (gpt-oss-120b) | 27–36B, compressed to 8-bit | 80 GB data-centre GPU (A100 / H100 PCIe) | rent about $1,290–2,080 a month |
| Large open models | 400B class, compressed to 8-bit (about half the GPUs at 4-bit) | a server with eight 80 GB GPUs | rent about $10,300–16,600 a month (eight A100 or H100 PCIe GPUs at the list rate) |
Rents are on-demand list prices at one US GPU cloud (RunPod, Secure Cloud) on 23 October 2026, for a full month of 720 hours. They show the order of magnitude: a host in your own jurisdiction, or a dedicated server under your own contract, charges its own rate, which the sizing spec quotes. If you buy the hardware instead, the rent becomes a one-time purchase plus power and colocation. We don't sell hardware: the sizing spec gives the exact configuration to put out for quotes and compares buying with renting at your load.
Memory needed = the model itself (plus the small model that indexes your documents) + (the text each user sends it, plus the passages pulled from your documents, × the number of users asking at the same moment). The tables count the model only. The pilot checks the rest under the load you expect.
Running cost of a private deployment
Running cost a month ≈ server rent (or, if you buy, depreciation, power and colocation) × number of servers + our support. There is no per-token bill: the server costs the same whether it answers a hundred questions a month or a hundred thousand, up to what one server can handle; beyond that you add a server.
| Example: one rented 80 GB GPU (estimate, 23 October 2026) | Month 1 | A steady month, one server |
|---|---|---|
| You pay Ksentra | setup from $10,000 + remote support from $1,000 | remote support from $1,000 |
| You pay the hosting provider, for one 80 GB GPU | about $1,290 | about $1,290 a month |
| Total with one rented 80 GB GPU | about $12,300 in month 1 | about $2,300 a month |
On one rented 24 GB GPU (about $640 a month) the same sums come to about $11,640 in month 1 and $1,640 a month after. Production on the optimal setup below doubles the hosting line, to about $2,580 a month for two 80 GB GPUs or $1,280 for two 24 GB GPUs, and its support is quoted above the $1,000 floor.
Two configurations to choose from:
- Minimal, enough for a pilot: one rented server, non-sensitive or synthetic documents (made-up ones that look like yours), a monthly check.
- Optimal, for production: two servers so one can take over if the other fails (with data-centre GPUs if you need a larger model), continuous monitoring and model updates. It is quoted after the sizing spec, above the support floor.
Remote support from $1,000 a month is the floor: a monthly check and a monthly note on cost and answer quality, with replies within business hours and no uptime guarantee.
Who runs it, and how we access your servers
You choose one of two arrangements in the sizing spec:
- Remote, the default. We install, update and check the platform through access your team grants and controls.
- Through your administrator. Your admin does the work on the servers, following our runbook (step-by-step instructions); we write the steps and review the results. Support for this arrangement is quoted separately.
Your deliverables
- The sizing spec, required on this path: where the model runs, two or three candidate models, the GPU and the monthly cost.
- Your knowledge base inside your perimeter, including the model that indexes your documents.
- The server configuration written for your supplier or host, then the software installed on it and connected to your chatbots, assistants and agents.
- The pilot's pass criterion, written into the spec before the pilot starts. The second payment (40% of the setup) is due only when the pilot passes.
What else every platform project includes, and what stays with you, is on the AI platform page.
The layer we already run
Chatbots on seven live sites, four of them other companies', run on our own engine (October 2026). It has a knowledge base and monitoring, and it calls a cloud model. In a private deployment we set the same layer up on your server around an open-weight model, with a local model indexing your documents. The paid sizing spec and the pilot check the fit on your own questions before you pay for the rest.
Frequently Asked Questions
Can a self-hosted model run with no internet connection at all?
Yes. On your own servers, the model and the knowledge base need no connection to outside AI services, and internal assistants and agents can run fully offline. A bot that talks to customers still needs its channel, your website or a messenger, but the requests are answered inside. With no connection at all, your administrator runs the platform with our runbook. How model and software updates reach the servers is agreed with your security team and written into the sizing spec.
How good are open models compared with cloud ones?
Open-weight models can answer less well than the strongest cloud models, and how much less depends on your task. We measure it rather than guess: two or three candidates run on 20–30 of your questions before you choose, and the pilot shows their answers to your questions side by side, so you see how well they do before the second payment.
Should we buy the hardware or rent it?
Renting starts in days and suits the pilot. Buying suits long, steady use and the strictest data rules, and adds power and colocation to the bill. The sizing spec compares both at your load. You buy from your own supplier; we write the configuration and install the software.
Who runs the private deployment after launch?
By default Ksentra runs it remotely, through access your team grants, with a monthly check and a note on cost and answer quality. Or your administrator runs it with our runbook.
How long is the pilot, and what counts as passed?
The pilot takes about three to four weeks, on non-sensitive or synthetic documents. The pass criterion is written into the sizing spec before the pilot starts: answer quality on your test set and response time with the number of users you expect at once. The second payment, 40% of the setup, is due only when the pilot passes.
Why is the sizing spec required for a private deployment?
The sizing spec is required because the hardware is bought or rented from it. Choosing a server before the models are tested on your questions risks paying for a GPU that is too small or far too large. The spec costs $1,500 and counts toward the setup.
Price
Sizing spec
$1,500
required; counts toward the setup
- Where the model runs, with alternatives
- 2–3 open-weight models tested on your questions
- The GPU and the monthly cost
- The pilot's pass criterion
Private deployment
from $10,000
setup · remote support from $1,000 / mo
- Open-weight model on servers you control
- Knowledge base inside your perimeter
- Server configuration and install
- Connected to your bots, assistants, agents and systems
- Pilot of 3–4 weeks on non-sensitive documents
You pay Ksentra for the sizing spec, the setup and support. You pay your hardware supplier or hosting provider directly; we don't resell hardware or licences. The setup is paid 40% when we sign, 40% when the pilot passes and 20% at go-live.
Need AI that keeps your data on servers you control?
Start with a free 30-minute call. Tell us what your data rules allow and what the AI should do, and we will show which model and server fit and what they cost a month.