What has to run locally – and what does not
"We want AI, but our data must not leave the building." I hear this often, and in that form it cannot be answered – because the question is framed too coarsely.
"AI" is not one operation but a chain of several. A typical setup contains text extraction, splitting into sections, vectorisation, search, and composing the answer. These steps have completely different requirements for confidentiality, quality, speed and cost. You do not decide once for all of it. You decide per step.
Four questions per step
What data actually flows through here? Not what data the company holds, but what this specific step sees. A search across a product catalogue moves the catalogue – public, already in the shop. Summarising support tickets moves customer names and complaints. Different case.
How good does the result have to be? Here is the honest part: locally runnable models have become good, but they are not as good as the large ones. For classification, extraction and vectorisation they are entirely sufficient. For tasks where the wording itself is the product, the gap is noticeable. Anyone claiming otherwise is usually selling hardware.
How often does this run? One operation a day is a different calculation than ten thousand. Vectorising a catalogue means sending every product through a model – locally that costs electricity and waiting, through an API it costs per call. At volume the maths tips quickly towards your own machine.
How fast does it have to be? A chat waiting for an answer does not tolerate eight seconds. An overnight import tolerates eight hours.
How I resolved it in my own setup
My product advisor illustrates the split well. Embeddings are produced locally: there are many of them, they are uniform, a local model is easily good enough for the job, and via an API it would be the most expensive part of the whole thing. The answer itself is composed by a large model over an API – that is where language quality counts, it is one call per question, and the data is public anyway, being my own product catalogue.
That boundary is not ideology, it is the outcome of four questions. For a client with personnel files it would sit somewhere else entirely.
The middle paths people overlook
Between "everything local" and "everything to an API" there is more than the debate suggests:
- Process locally, phrase externally. The search across confidential documents runs in house. What leaves is only the passage actually needed – not the corpus.
- Pseudonymise before sending. Replace names, customer numbers and addresses with placeholders and restore them after the answer. This automates reliably and changes the legal assessment considerably.
- Split by data class. The public part takes the fast route, the confidential part the local one. Two routes are not a flaw, they are an architectural decision.
What I advise companies
Do not start with "local or cloud". Start with: which operations do we actually need, and what data flows through each one? After that the decision usually makes itself – and often differently than expected at the start of the conversation.
Something else that may come out of it: for a large share of what companies plan to do with AI, confidentiality is not the problem at all. It is simply named first, because it is easier to say than "we do not know where our data is". That is then the real project – and it has little to do with models.