Running LLMs on-premises: a reference architecture for regulated organizations

Hardware Sizing

Everything else in this architecture depends on getting hardware sizing right first. The two numbers that matter most are usable VRAM per GPU and aggregate memory bandwidth across the cluster, because both determine which model sizes you can serve and at what latency. A rough planning rule: a model's weights need roughly two bytes of VRAM per parameter at common quantization levels, plus headroom for the key-value cache that grows with context length and concurrent users, plus overhead for the serving framework itself. Undersized memory forces aggressive quantization that can degrade output quality; oversized hardware wastes capital that could fund a second environment for staging and evaluation. We size for the busiest expected hour, not the daily average, and always reserve one node's worth of capacity for rolling upgrades so a deployment never requires full downtime.

Undersized memory forces aggressive quantization that can degrade output quality; oversized hardware wastes capital that could fund a second environment for staging and evaluation.

The Serving Layer

The serving layer is the component that actually loads model weights and answers inference requests, and this is where throughput and latency are won or lost. Frameworks such as vLLM and similar high-throughput serving engines use techniques like paged attention and continuous batching to serve many concurrent requests from the same GPU memory far more efficiently than naive request-by-request serving. In practice this means configuring batching aggressively enough to maximize GPU utilization without pushing per-request latency past what your application's users will tolerate, and running separate serving pools for latency-sensitive interactive workloads versus throughput-oriented batch workloads such as bulk document processing, since the two have conflicting tuning goals. Autoscaling should be driven by queue depth and time-to-first-token, not raw CPU usage, since GPU-bound inference rarely shows the signals a traditional web autoscaler expects.

API Gateway and Access Control

No application, internal or external, should talk to the serving layer directly. An API gateway sits in front of every model endpoint to enforce authentication, authorization, rate limiting and request or response logging consistently across every consumer. This is also where prompt-level policy controls belong: input validation, size limits, and routing rules that send a given request to the right model, context window size or safety configuration based on who is asking and what they are asking for. Access control should be scoped per application and per data classification, not just per user, so a procurement chatbot and a clinical decision-support tool, even if they share the same underlying model, never share the same permissions, logging retention or rate limits. Every request and response should be logged with enough context to support an incident investigation without having to reconstruct the prompt from application logs elsewhere.

RAG and Knowledge Retrieval

Retrieval-augmented generation lets a general-purpose model answer questions about your organization's own documents without retraining it, by fetching relevant passages from a vector database or search index and inserting them into the prompt at request time. The architecture decisions that matter most are how documents are chunked and embedded, how access control on the underlying documents is enforced at retrieval time rather than only at the application layer, and how retrieval quality is measured rather than assumed. A retrieval pipeline that ignores document-level permissions can turn a well-intentioned feature into a sensitive information disclosure problem, so the retrieval layer must enforce the same authorization rules as the source system it indexes. We also keep retrieval and generation observable as separate stages, so a poor answer can be traced back to bad retrieval versus a model reasoning failure, which are fixed in very different ways.

Observability

An on-premises LLM platform needs its own observability practice, distinct from standard application monitoring, because the failure modes are different. Latency should be tracked as time-to-first-token and total generation time separately, since they have different causes. Cost should be tracked per token and attributed back to the requesting application or business unit, both to control spend and to justify the infrastructure investment to stakeholders. Quality needs an ongoing evaluation pipeline, not a one-time benchmark at launch, that samples real production traffic against a rubric or a reference model and flags drift over time as usage patterns, document sets or underlying model versions change. We treat a sudden shift in any of these three signals, latency, cost-per-token or evaluation score, as an operational incident worth investigating, the same way a spike in error rate would be treated on any other production service.

Security Zones and Network Segmentation

We segment an on-premises AI platform into at least three network zones: an ingestion zone where external documents and data enter and are scanned before indexing, a serving zone where the GPU cluster, model weights and vector store live with no direct inbound access from application networks, and an application zone where client applications call the API gateway. Traffic between zones passes only through the gateway and defined data pipelines, never through direct database or GPU-node access, and each zone has its own logging and alerting baseline so an anomaly in one is visible without being drowned out by normal traffic in another. Model weights and embeddings at rest are encrypted, and administrative access to the serving zone requires separate, more tightly audited credentials than access to the application zone.

The Air-Gap Update Process

The hardest operational problem in a fully disconnected deployment is keeping models, dependencies and security patches current without a live connection to the public internet. Our process treats every update, a new model version, a patched serving framework, an updated vector database, as a signed, versioned package assembled and tested in a connected staging environment first. That package, along with its checksums, a software bill of materials and a rollback plan, is transferred across the air gap through a controlled, reviewed transfer process defined by the client's own security policy, then verified again inside the disconnected environment before it touches production. Nothing is pulled directly from the internet into the production environment at update time. Rollback is tested as part of every update, not treated as a theoretical fallback, because a disconnected environment cannot simply roll forward to a hotfix the way a cloud deployment can.

Nothing is pulled directly from the internet into the production environment at update time.