How my team runs four AI workloads in production — code review, security code review, grounded RAG chat, and code generation — on self-hosted LLMs (Ollama × NVIDIA DGX), behind one .NET abstraction, with sensitive data never leaving the organization.
Hosted LLM APIs are the fastest way to ship AI features — and a non-starter for a large part of enterprise work. Our source code, internal policies, incident history, and operational documents are exactly the inputs that make AI useful for engineering teams, and exactly the data that cannot cross the organization's boundary.
The naive answers both fail. "Just use the API" fails compliance. "Ban AI" fails the team — engineers were already losing hours to manual code review passes, repetitive security checklists, and hunting through documentation. The real requirement was: bring the model to the data, not the data to the model.
Every consumer — CI pipelines, the chat UI, IDE tooling — talks to one .NET gateway built on Microsoft.Extensions.AI's IChatClient abstraction. The gateway routes each request by data sensitivity: workloads touching internal code or documents go to open models served by Ollama on NVIDIA DGX hardware, fully on-premises; general-purpose requests can route to hosted models. Sensitivity is a per-request decision, not a per-project rewrite.
Every merge request gets a model pass before human review: correctness risks, N+1 queries, missing cancellation tokens, convention drift. Findings post as structured review comments — humans stay the approvers; the model clears the underbrush.
A separate pass with a security-specific prompt and ruleset: injection risks, authZ gaps, secret leakage, unsafe deserialization. Runs on-prem by necessity — the code being audited is exactly what can't be sent to an external API.
An assistant answering from our own knowledge: internal docs and repos are chunked, embedded on-prem, and retrieved per question. Answers cite their sources; "grounded" means the model is constrained to retrieved context, not free to improvise policy.
Scaffolding, tests, migrations, and boilerplate generated against our conventions, with repository context retrieved the same way the chat is grounded. Output is a starting point that compiles against our patterns — not generic samples from the internet.
| Decision | Why | The trade-off we accepted |
|---|---|---|
| Self-host on DGX | Data boundary is non-negotiable for code and internal docs | We own ops, upgrades, and capacity planning; models trail frontier quality |
| MEAI abstraction | Routing is config, not rewrites; providers are swappable | A thin layer to maintain; occasional provider-specific features need escape hatches |
| Structured output everywhere | CI can't parse prose — findings must be typed and validated | Stricter prompts; a retry-with-error loop for schema violations |
| Humans stay approvers | Model findings advise, never gate a merge alone | Slower than full automation — deliberately |
| Eval set before every swap | Model/quantization changes are deploys, not experiments | Maintaining golden datasets takes real time |
The unlock wasn't the model — it was treating AI infrastructure like any other production system: one abstraction, routed by data sensitivity, observable, and evaluated before every change.
Building something similar?
I'm happy to compare notes on running LLMs inside the boundary.