AI is evolving quickly, and software program engineers not must memorize syntax. Nonetheless, considering like an architect and understanding the expertise that allows techniques to run securely at scale is changing into more and more precious.
I additionally wish to replicate on being in my function a 12 months now as an AI Options Engineer at Cisco. I work with prospects each day throughout completely different verticals — healthcare, monetary providers, manufacturing, legislation corporations, and they’re all attempting to reply largely the identical set of questions:
- What’s our AI technique?
- What use circumstances truly match our information?
- Cloud vs. on-prem vs. hybrid?
- How a lot will it value — not simply at this time, however at scale?
- How can we safe it?
These are the actual sensible constraints that present up instantly when you attempt to operationalize AI past a POC.
Not too long ago, we added a Cisco UCS C845A to considered one of our labs. It has 2x NVIDIA RTX PRO 6000 Blackwell GPUs, 3.1TB NVMe, ~127 allocatable CPU cores, and 754GB RAM. I made a decision to construct a shared inside platform on prime of it — giving groups a constant, self-service surroundings to run experiments, validate concepts, and construct hands-on GPU expertise.
I deployed the platform as a Single Node OpenShift (SNO) cluster and layered a multi-tenant GPUaaS expertise on prime. Customers reserve capability by means of a calendar UI, and the system provisions an remoted ML surroundings prebuilt with PyTorch/CUDA, JupyterLab, VS Code, and extra. Inside that surroundings, customers can run on-demand inference, iterate on mannequin coaching and fine-tuning, and prototype manufacturing grade microservices.
This submit walks by means of the structure — how scheduling choices are made, how tenants are remoted, and the way the platform manages itself. The selections that went into this lab platform are the identical ones any group faces once they’re severe about AI in manufacturing.
That is the inspiration for enterprise AI at scale. Multi-agent architectures, self-service experimentation, safe multi-tenancy, cost-predictable GPU compute, all of it begins with getting the platform layer proper.
Preliminary Setup
Earlier than there’s a platform, there’s a naked steel server and a clean display screen.
Bootstrapping the Node
The node ships with no working system. Whenever you energy it on you’re dropped right into a UEFI shell. For OpenShift, set up usually begins within the Crimson Hat Hybrid Cloud Console by way of the Assisted Installer. The Assisted Installer handles cluster configuration by means of a guided setup stream, and as soon as full, generates a discovery ISO — a bootable RHEL CoreOS picture preconfigured in your surroundings. Map the ISO to the server as digital media by means of the Cisco IMC, set boot order, and energy on. The node will telephone dwelling to the console, and you may kick off the set up course of. The node writes RHCOS to NVMe and bootstraps. Inside a number of hours you will have a operating cluster.
This workflow assumes web connectivity, pulling photographs from Crimson Hat’s registries throughout set up. That’s not at all times an possibility. Most of the prospects I work with function in air-gapped environments the place nothing touches the general public web. The method there’s completely different: generate ignition configs domestically, obtain the OpenShift launch photographs and operator bundles forward of time, mirror the whole lot into an area Quay registry, and level the set up at that. Each paths get you to the identical place. The assisted set up is way simpler. The air-gapped path is what manufacturing appears to be like like in regulated industries.
Configuring GPUs with the NVIDIA GPU Operator
As soon as the GPU Operator is put in (occurs robotically utilizing the assisted installer), I configured how the 2 RTX PRO 6000 Blackwell GPUs are introduced to workloads by means of two ConfigMaps within the nvidia-gpu-operator namespace.
The primary — custom-mig-config — defines bodily partitioning. On this case it’s a blended technique, that means GPU 0 is partitioned into 4 1g.24gb MIG slices (~24GB devoted reminiscence every), GPU 1 stays entire for workloads that want the total ~96GB. MIG partitioning is actual {hardware} isolation. You get devoted reminiscence, compute models, and L2 cache per slice. Workloads will see MIG cases as separate bodily gadgets.
The second — device-plugin-config — configures time-slicing, which permits a number of pods to share the identical GPU or MIG slice by means of speedy context switching. I set 4 replicas per entire GPU and a pair of per MIG slice. That is what permits operating a number of inference containers facet by facet inside a single session.
Foundational Storage
The three.1TB NVMe is managed by the LVM Storage Operator (lvms-vg1 StorageClass). I created two PVCs as part of the preliminary provisioning course of — a quantity backing PostgreSQL and chronic storage for OpenShift’s inside picture registry.
With the OS put in, community stipulations met (DNS, IP allocation, all required A information) which isn’t coated on this article, GPUs partitioned, and storage provisioned, the cluster is prepared for the applying layer.
System Structure
This leads us into the principle matter: the system structure. The platform separates into three planes — scheduling, management, and runtime, with the PostgreSQL database as the one supply of fact.
Within the platform administration namespace, there are 4 at all times on deployments:
- Portal app: a single container operating the React UI and FastAPI backend
- Reconciler (controller): the management loop that repeatedly converges cluster state to match the database
- PostgreSQL: persistent state for customers, reservations, tokens, and audit historical past
- Cache daemon: a node-local service that pre-stages massive mannequin artifacts / inference engines so customers can begin shortly (pulling a 20GB vLLM picture over company proxy can take hours)
A fast word on the event lifecycle, as a result of it’s simple to complicate transport Kubernetes techniques. I write and take a look at code domestically, however the photographs are constructed within the cluster utilizing OpenShift construct artifacts (BuildConfigs) and pushed to the interior registry. The deployments themselves simply level at these photographs.
The primary time a element is launched, I apply the manifests to create the Deployment/Service/RBAC. After that, most modifications are simply constructing a brand new picture in-cluster, then set off a restart so the Deployment pulls the up to date picture and rolls ahead:
oc rollout restart deployment/<deployment-name> -n <namespace>That’s the loop: commit → in-cluster construct → inside registry → restart/rollout.
The Scheduling Airplane
That is the consumer dealing with entry level. Customers see the useful resource pool — GPUs, CPU, reminiscence, they choose a time window, select their GPU allocation mode (extra on this later), and submit a reservation.
GPUs are costly {hardware} with an actual value per hour whether or not they’re in use or not. The reservation system treats calendar time and bodily capability as a mixed constraint. The identical method you’d e-book a convention room, besides this room has 96GB of VRAM and prices significantly extra per hour.
Beneath the hood, the system queries overlapping reservations in opposition to pool capability utilizing advisory locks to stop double reserving. Primarily it’s simply including up reserved capability and subtracting it from complete capability. Every reservation tracks by means of a lifecycle: APPROVED → ACTIVE → COMPLETED, with CANCELED and FAILED as terminal states.
The FastAPI server itself is deliberately skinny. It validates enter, persists the reservation, and returns. It by no means talks to the Kubernetes API.
The Management Airplane
On the coronary heart of the platform is the controller. It’s Python based mostly and runs in a steady loop on a 30-second cadence. You may consider it like a cron job when it comes to timing, however architecturally it’s a Kubernetes-style controller chargeable for driving the system towards a desired state.
The database holds the specified state (reservations with time home windows and useful resource necessities). The reconciler reads that state, compares it in opposition to what truly exists within the Kubernetes cluster, and converges the 2. There aren’t any concurrent API calls racing to mutate cluster state; only one deterministic loop making the minimal set of modifications wanted to succeed in the specified state. If the reconciler crashes, it restarts and continues precisely the place it left off, as a result of the supply of fact (desired state) stays intact within the database.
Every reconciliation cycle evaluates 4 issues so as:
- Cease expired or canceled periods and delete the namespace (which cascades cleanup of all sources inside it).
- Restore failed periods and take away orphaned sources left behind by partially accomplished provisioning.
- Begin eligible periods when their reservation window arrives — provision, configure, and hand the workspace to the consumer.
- Preserve the database by expiring previous tokens and imposing audit log retention.
Beginning a session is a multi-step provisioning sequence, and each step is idempotent, that means it’s designed to be safely re-run if interrupted halfway:
The reconciler is the solely element that talks to the Kubernetes API.
Rubbish assortment can be baked into the identical loop. At a slower cadence (~5 minutes), the reconciler sweeps for cross namespace orphans corresponding to stale RBAC bindings, leftover OpenShift safety context entries, namespaces caught in terminating, or namespaces that exist within the cluster however haven’t any matching database document.
The design assumption all through is that failure is regular. For instance, we had an influence provide failure on the node that took the cluster down mid-session and when it got here again, the reconciler resumed its loop, detected the state discrepancies, and self-healed with out handbook intervention.
The Runtime Airplane
When a reservation window begins, the consumer opens a browser and lands in a full VS Code workspace (code-server) pre-loaded with the complete AI/ML stack, and kubectl entry inside their session namespace.
Fashionable inference engines corresponding to vLLM, Ollama, TGI, and Triton are already cached on the node, so deploying a mannequin server is a one-liner that begins in seconds. There’s 600GB of persistent NVMe backed storage allotted to the session, together with a 20GB dwelling listing for notebooks and scripts, and a 300GB mannequin cache.
Every session is a totally remoted Kubernetes namespace, its personal blast radius boundary with devoted sources and 0 visibility into another tenant’s surroundings. The reconciler provisions namespace scoped RBAC granting full admin powers inside that boundary, enabling customers to create and delete pods, deployments, providers, routes, secrets and techniques — regardless of the workload requires. However there’s no cluster stage entry. Customers can learn their very own ResourceQuota to see their remaining funds, however they’ll’t modify it.
ResourceQuota enforces a tough ceiling on the whole lot. A runaway coaching job can’t OOM the node. A rogue container can’t fill the NVMe. LimitRange injects sane defaults into each container robotically, so customers can kubectl run with out specifying useful resource requests. There’s a proxy ConfigMap injected into the namespace so consumer deployed containers get company community egress with out handbook configuration.
Customers deploy what they need — inference servers, databases, {custom} providers, and the platform handles the guardrails.
When the reservation window ends, the reconciler deletes the namespace and the whole lot inside it.
GPU Scheduling
Now the enjoyable half — GPU scheduling and truly operating hardware-accelerated workloads in a multi-tenant surroundings.
MIG & Time-slicing
We coated the MIG configuration within the preliminary setup, nevertheless it’s value revisiting from a scheduling perspective. GPU 0 is partitioned into 4 1g.24gb MIG slices — every with ~24GB of devoted reminiscence, sufficient for many 7B–14B parameter fashions. GPU 1 stays entire for workloads that want the total ~96GB VRAM for mannequin coaching, full-precision inference on 70B+ fashions, or something that merely doesn’t slot in a slice.
The reservation system tracks these as distinct useful resource sorts. Customers e-book both nvidia.com/gpu (entire) or nvidia.com/mig-1g.24gb (as much as 4 slices). The ResourceQuota for every session laborious denies the other sort. Should you reserved a MIG slice, you bodily can not request an entire GPU, even when one is sitting idle. In a blended MIG surroundings, letting a session unintentionally eat the unsuitable useful resource sort would break the capability math for each different reservation on the calendar.
Time-slicing permits a number of pods to share the identical bodily GPU or MIG slice by means of speedy context switching. The NVIDIA machine plugin advertises N “digital” GPUs per bodily machine.
In our configuration, 1 entire GPU seems as 4 schedulable sources. Every MIG slice seems as 2.
What which means is a consumer reserves one bodily GPU and may run as much as 4 concurrent GPU-accelerated containers inside their session — a vLLM occasion serving gpt-oss, an Ollama occasion with Mistral, a TGI server operating a reranker, and a {custom} service orchestrating throughout all three.
Two Allocation Modes
At reservation time, customers select how their GPU funds is initially distributed between the workspace and consumer deployed containers.
Interactive ML — The workspace pod will get a GPU (or MIG slice) connected immediately. The consumer opens Jupyter, imports PyTorch, and has speedy CUDA entry for coaching, fine-tuning, or debugging. Further GPU pods can nonetheless be spawned by way of time-slicing, however the workspace is consuming one of many digital slots.
Inference Containers — The workspace is light-weight with no GPU connected. All time-sliced capability is out there for consumer deployed containers. With an entire GPU reservation, that’s 4 full slots for inference workloads.
There’s a actual throughput tradeoff with time-slicing, workloads share VRAM and compute bandwidth. For improvement, testing, and validating multi-service architectures, which is precisely what this platform is for, it’s the best trade-off. For manufacturing latency delicate inference the place each millisecond of p99 issues, you’d use devoted slices 1:1 or entire GPUs.
GPU “Tokenomics”
One of many first questions within the introduction was: How a lot will it value — not simply at this time, however at scale? To reply that, you need to begin with what the workload truly appears to be like like in manufacturing.
What Actual Deployments Look Like
Once I work with prospects on their inference structure, no one is operating a single mannequin behind a single endpoint. The sample that retains rising is a fleet of fashions sized to the duty. You have got a 7B parameter mannequin dealing with easy classification and extraction, runs comfortably on a MIG slice. A 14B mannequin doing summarization and normal function chat. A 70B mannequin for complicated reasoning and multi-step duties, and possibly a 400B mannequin for the toughest issues the place high quality is non-negotiable. Requests get routed to the suitable mannequin based mostly on complexity, latency necessities, or value constraints. You’re not paying 70B-class compute for a activity a 7B can deal with.
In multi-agent techniques, this will get extra fascinating. Brokers subscribe to a message bus and sit idle till known as upon — a pub-sub sample the place context is shared to the agent at invocation time and the pod is already heat. There’s no chilly begin penalty as a result of the mannequin is loaded and the container is operating. An orchestrator agent evaluates the inbound request, routes it to a specialist agent (retrieval, code technology, summarization, validation), collects the outcomes, and synthesizes a response. 4 or 5 fashions collaborating on a single consumer request, every operating in its personal container throughout the similar namespace, speaking over the interior Kubernetes community.
Community insurance policies add one other dimension. Not each agent ought to have entry to each software. Your retrieval agent can speak to the vector database. Your code execution agent can attain a sandboxed runtime. However the summarization agent has no enterprise touching both, it receives context from the orchestrator and returns textual content. Community insurance policies implement these boundaries on the cluster stage, so software entry is managed by infrastructure, not software logic.
That is the workload profile the platform was designed for. MIG slicing enables you to proper measurement GPU allocation per mannequin, a 7B doesn’t want 96GB of VRAM. Time-slicing lets a number of brokers share the identical bodily machine. Namespace isolation retains tenants separated whereas brokers inside a session talk freely. The structure immediately helps these patterns.
Quantifying It
To maneuver from structure to enterprise case, I developed a tokenomics framework that reduces infrastructure value to a single comparable unit: value per million tokens. Every token carries its amortized share of {hardware} capital (together with workload combine and redundancy), upkeep, energy, and cooling. The numerator is your complete annual value. The denominator is what number of tokens you truly course of, which is totally a operate of utilization.
Utilization is probably the most highly effective lever on per-token value. It doesn’t cut back what you spend, the {hardware} and energy payments are fastened. What it does is unfold these fastened prices throughout extra processed tokens. A platform operating at 80% utilization produces tokens at practically half the unit value of 1 at 40%. Identical infrastructure, dramatically completely different economics. This is the reason the reservation system, MIG partitioning, and time-slicing matter past UX — they exist to maintain costly GPUs processing tokens throughout as many obtainable hours as attainable.
As a result of the framework is algebraic, you too can clear up within the different path. Given a identified token demand and a funds, clear up for the infrastructure required and instantly see whether or not you’re over-provisioned (burning cash on idle GPUs), under-provisioned (queuing requests and degrading latency), or right-sized.
For the cloud comparability, suppliers have already baked their utilization, redundancy, and overhead into per-token API pricing. The query turns into: at what utilization does your on-prem unit value drop beneath that fee? For constant enterprise GPU demand, the sort of steady-state inference site visitors these multi-agent architectures generate, on-prem wins.
Cloud token prices in multi-agent environments scale parabolically.
Nonetheless, for testing, demos, and POCs, cloud is cheaper.
Engineering groups typically must justify spend to finance with clear, defensible numbers. The tokenomics framework bridges that hole.
Conclusion
In the beginning of this submit I listed the questions I hear from prospects consistently — AI technique, use-cases, cloud vs. on-prem, value, safety. All of them ultimately require the identical factor: a platform layer that may schedule GPU sources, isolate tenants, and provides groups a self-service path from experiment to manufacturing with out ready on infrastructure.
That’s what this submit walked by means of. Not a product and never a managed service, however an structure constructed on Kubernetes, PostgreSQL, Python, and the NVIDIA GPU Operator — operating on a single Cisco UCS C845A with two NVIDIA RTX PRO 6000 Blackwell GPUs in our lab. It’s a sensible start line that addresses scheduling, multi-tenancy, value modeling, and the day-2 operational realities of protecting GPU infrastructure dependable.
This isn’t as intimidating because it appears to be like. The tooling is mature, and you may assemble a cloud-like workflow with acquainted constructing blocks: reserve GPU capability from a browser, drop into a totally loaded ML workspace, and spin up inference providers in seconds. The distinction is the place it runs — on infrastructure you personal, below your operational management, with information that by no means leaves your 4 partitions. In apply, the barrier to entry is commonly decrease than leaders anticipate.
Scale this to a number of Cisco AI Pods and the scheduling aircraft, reconciler sample, and isolation mannequin carry over immediately. The muse is identical.
Should you’re working by means of these similar choices — tips on how to schedule GPUs, tips on how to isolate tenants, tips on how to construct the enterprise case for on-prem AI infrastructure, I’d welcome the dialog.
I’m an AI Options Engineer at Cisco, specializing in enterprise AI infrastructure. Attain out at [email protected].
