Serve a model on your own GPUs

Control Room → GPU Management. Two A100 nodes, sixteen GPUs, discovered on the cluster and managed here: MIG profiles, pools, health. Every model in this chapter runs on them.

Allocation across the fleet, in use or reserved. The deployments you are about to see draw from this pool, and so does the one you will create.

Studio → LLM Deployment. Two models are running on NVIDIA Dynamo with vLLM, each pinned to GPU slices on the fleet. Status, framework, hardware and live traffic per deployment.

Click New deployment.

Three sources: Hugging Face Hub, NVIDIA NIM, and the Model registry, the in-cluster MLflow registry with staged weights. Click Model registry: no external credentials, nothing downloaded from outside.

A foundation model registered by the platform team, with its version and base. Click gemma-3-27b-it.

Click Continue · Runtime.

The deployment name is the model name callers send to the endpoint. We typed one; change it if you like.

The serving path is NVIDIA's Dynamo graph: frontend plus workers, profiled by AIConfigurator and sized to a latency target. Automatic mode lets the operator choose engine, tensor parallel and placement.

The node picker lists your fleet with what is free right now. The deployment takes A100s from ng-gpu-02; the MIG-sliced node beside it is for smaller models.

Click Continue · Review.

The review prices the request up front: GPU hourly rate, a baseline per deployment, an upper bound per month. The operator may use fewer GPUs than the budget allows.

GPU resources and cost start the moment the deployment is accepted. Tick the box.

Click Deploy.

The scheduler streams its stages: request accepted, hardware discovered, the model loaded, the first health check. On the demo the profiling is simulated in about thirty seconds; on a cluster it is real.

The deployment is Running. This panel records what Automatic mode chose: the engine, the topology and the GPU footprint, so the decision is on the record, not in someone's head.

The model answers on an OpenAI-compatible endpoint inside the cluster. One client discovers every deployed model; nothing is routed through a provider outside your boundary.

Click Playground.

A question for the model itself, no knowledge base attached. Change it if you like, then go on.

Click Send.

The reply streams from the deployment you just made, with tokens, time to first token and tokens per second measured on your hardware.

Click Gateway: the model is now a route on the AI Gateway, where agents, apps and API keys reach it under the organisation's policies and budgets.

You are the Builder in Development. Studio is not open to you here: a builder gets Studio only in an environment their team has been granted, and deploying a model needs the Admin's capability besides. The refusal is enforced by the platform, not by a note in a wiki.

Back as the Admin: the list now carries ops-assistant-llm, running beside the others, with compute cost accrued month to date per deployment. The same figures feed Budgets and Cost Analytics in the Control Room.