Local LLM Setup for a Business: Ollama, Open WebUI, vLLM

A local LLM setup for a small business has three layers: a runner that loads the model (Ollama, llama.cpp or LM Studio), a chat page staff log into (Open WebUI or AnythingLLM), and an open-weight model sized to the box's memory. At 4-bit quantisation a model needs a little over half a gigabyte per billion parameters, so a 64GB box runs 20B to 30B models comfortably. vLLM replaces the runner when many users hit one GPU server.

This is the software page of the edge AI series. The box itself is covered in local AI hardware.

Which tool does what?

All of the runners above expose an OpenAI-compatible API, so a client's existing tools that talk to OpenAI can often be pointed at the local box by changing one address.

How do you set up Ollama and Open WebUI?

This is the stack most small installs end up on.

1. Install Ollama. It runs on macOS Sonoma or newer with Apple M-series chips, on Windows 10 22H2 or newer with NVIDIA and AMD Radeon support, and on Linux, according to Ollama's docs. There is also an official Docker image.
2. Pull a model. One command downloads it, for example a 20B-class model for a 32GB or 64GB box.
3. Check the API. Ollama serves a REST API on port 11434 and supports a subset of the OpenAI API at localhost:11434/v1.
4. Install Open WebUI. It describes itself as a self-hosted platform "built to run entirely offline" with support for Ollama, user groups with role-based access, LDAP and Active Directory, and local retrieval for document search.
5. Create accounts and groups. One admin (you, until handover), then a group per department so HR documents stay with HR.
6. Load a test folder. Use documents the client is happy to test with, not the sensitive archive, until the pilot is signed off.

Keep everything on the office network. LM Studio's own docs warn that binding its server to anything other than 127.0.0.1 "exposes the server beyond localhost; we recommend enabling authentication". The same applies to every runner: never expose port 11434 to the internet.

What does the Open WebUI licence allow?

Open WebUI changed its licence at version 0.6.6 on 19 April 2025. Code up to 0.6.5 stays under BSD-3. From 0.6.6 the licence adds a clause that forbids "altering, removing, obscuring, or replacing any Open WebUI branding", with three exceptions in the licence file:

• deployments with no more than 50 end users "within any rolling thirty (30) day period";
• written permission from the project;
• an enterprise licence.

The project's own summary is "Want to use Open WebUI for free? Just keep the branding." For an installer, that means you can install it for a 200-person firm at no cost, but you cannot sell it under your own name or the client's logo without an enterprise licence. AnythingLLM is MIT licensed if white-labelling matters, though its multi-user and white-label features are Docker only.

Which model fits which box?

Memory decides. The model has to fit in RAM (or VRAM on a graphics card) with room left for the conversation.

The rule of thumb: at 16-bit precision a model needs about 2GB per billion parameters. Hugging Face puts it as "roughly 2 * X GB of VRAM in bfloat16/float16 precision" for a model of X billion parameters. At 8-bit it is a little over 1GB per billion, and at 4-bit a little over half.

llama.cpp's own quantisation table shows it for real models:

On top of the weights, every conversation needs working memory called the KV cache. NVIDIA's example puts it at about 2GB for a 7B model at 4,096 tokens, and it grows with longer documents and more users at once. Leave a quarter of the memory free.

That gives a rough map:

OpenAI says gpt-oss-20b "can run on edge devices with just 16 GB of memory" and gpt-oss-120b runs on a single 80GB GPU. The example models are suggestions to test, not a ranking: test two or three on the client's own documents before you choose.

What does quantisation mean in plain English?

A model is billions of numbers. Quantisation stores each number with fewer bits. Think of a photo saved at lower quality: the file is much smaller and loads faster, and if you go too far you start to see the damage.

The common formats you will see in Ollama and llama.cpp are:

• Q8_0: about 8.5 bits per number. Close to the original, about half the size of 16-bit.
• Q4_K_M: about 4.9 bits per number. The usual default for local use, under a third of the 16-bit size.
• Below 4 bits: smaller still, with more visible loss. Worth it only to squeeze a bigger model onto a box.

llama.cpp's README says quantisation may introduce some accuracy loss, which is measured with perplexity or KL divergence. A bigger model at 4-bit usually beats a smaller model at 8-bit on the same memory, which is why Q4_K_M is the default. Test on the client's real questions, because loss shows up unevenly.

Which open models can a business use commercially?

Read the licence before you install. The ones most installers use:

• gpt-oss (OpenAI): Apache 2.0. 20B and 120B sizes.
• Qwen (Alibaba): "All our open-weight models are licensed under Apache 2.0", per the Qwen3 repository. The 2026 releases (Qwen3.5, 3.6 and 3.8) include 27B and smaller sizes.
• Gemma 4 (Google): released in April 2026 under Apache 2.0. The older Gemma 3 uses Google's own Gemma Terms of Use with a prohibited-use policy you must pass on to users.
• Llama (Meta): a community licence. Companies with over 700 million monthly active users must ask Meta for a licence, which no small client will hit, but the licence has other terms to pass on.
• Mistral: the small Ministral 3 models (3B, 8B and 14B) are Apache 2.0. Mistral Small 4, despite the name, has 119B parameters and needs data-centre GPUs.

Apache 2.0 is the easiest to explain to a client's lawyer. Keep a one-page record of which model and licence each client runs.

When do you need vLLM instead of Ollama?

When one box has to serve many people at the same moment. vLLM describes itself as "a high-throughput and memory-efficient inference and serving engine", using PagedAttention to manage conversation memory and continuous batching to answer many requests together.

The catch is setup. vLLM lists Linux as its OS and "does not support Windows natively", and on NVIDIA it needs a card of compute capability 7.5 or higher. For a 10-person office on a Mac mini, Ollama is the right answer. For a 60-person firm on a Linux GPU server, test vLLM.

How do you add document search (RAG)?

RAG (retrieval-augmented generation) means the system finds the relevant passages in the client's files and hands them to the model with the question, so answers come from the client's own documents. Open WebUI and AnythingLLM both have it built in.

It needs an embedding model, which turns text into numbers for searching. Two that run in Ollama: nomic-embed-text (274MB) and bge-m3 (1.2GB, more than 100 languages, a good pick for Singapore and Malaysia offices that mix English, Malay and Chinese). Ollama's docs advise using "the same embedding model for both indexing and querying", so pick one and keep it.

The office side of RAG, including who can see which documents, is in private AI for offices, clinics and law firms.

What does the stack cost compared with the cloud?

The software above is free. The costs are the box, its power and your time. On the cloud side, small API models are cheap per token: Anthropic lists Claude Haiku 4.5 at $1 per million input tokens and $5 per million output tokens, and its Team plan at $25 a seat a month billed monthly. A local box rarely beats a light API bill on price alone. It wins when the client pays per seat for many staff, or when the data cannot leave the building at all.

Run the client's numbers in the local AI vs cloud calculator.