HPE GPU servers
What software do you need to run AI agents on HPE GPU servers?
To run AI agents on HPE GPU servers you need seven software layers above the hardware: a cluster layer, model serving, a gateway, a knowledge layer, an agent layer, guardrails with evaluation, and a place where staff use the agents. You can assemble these from open-source parts or use a platform that supplies them together.
The servers are the start
HPE GPU servers, bought alone or as part of an HPE AI Factory design, give you compute, storage and networking. That is the foundation. Nobody in the business can use a GPU server directly, so the work that remains is all software.
The question buyers ask after delivery is some version of "we have the GPUs, now what". The honest answer is that seven things have to exist before a member of staff can ask an agent to do a job.
The seven layers at a glance
| Layer | What it does | Typical parts |
|---|---|---|
| 1. Cluster | Schedules GPUs and runs containers | Linux, Kubernetes, NVIDIA GPU software |
| 2. Model serving | Loads models onto the GPUs and answers requests | NVIDIA NIM, vLLM |
| 3. Gateway | One controlled way in: access, budgets, logging | An AI gateway |
| 4. Knowledge | Lets agents use your documents and data, with permissions | Ingestion, a vector store, retrieval |
| 5. Agents | Where agents are built and run | Agent frameworks, tool connectors |
| 6. Guardrails and evaluation | Keeps agents inside policy and tests them before release | Guardrail rules, test sets |
| 7. Use | Where staff actually work with agents | A chat and workspace application |
Layer 1: the cluster
The cluster layer turns separate servers into one pool. It decides which job gets which GPU, restarts what fails, and lets several teams share the hardware. Most estates use Kubernetes with NVIDIA's GPU software. This layer is well documented by HPE and NVIDIA.
Layer 2: model serving
A model file does nothing until a serving engine loads it onto a GPU and answers requests. This layer decides how many requests a GPU handles at once and how fast answers come back. NVIDIA NIM and vLLM are common choices for open-weight models.
Layer 3: the gateway
Without a gateway, every team calls the model endpoints directly, with shared keys and no record. A gateway gives one way in. It checks who is asking, counts what they use, applies budgets and caps, and logs every call. This is the layer that makes it possible to charge departments for what they consume.
Layer 4: knowledge
A model knows nothing about your organisation. The knowledge layer reads your documents and data, indexes them, and retrieves the right passages when an agent needs them. The hard part is permissions: an agent answering for one person must not retrieve what that person may not see.
Layer 5: agents
This is where an agent is defined: its instructions, the tools it may call, the knowledge it may use and the steps it follows. Developers often start with an open-source framework. Business teams need a way to build without writing code. Production needs both to run under the same controls.
Layer 6: guardrails and evaluation
Guardrails check what goes into and comes out of an agent against policy, such as personal data rules or topics it must refuse. Evaluation tests each version of an agent against known questions before it is released. Without evaluation, nobody can say whether a change made the agent better or worse.
Layer 7: where staff use it
The last layer is the one the business sees. Staff need a place to find the agents they are allowed to use, talk to them, hand work over and see results. A project that stops at an API has not reached the people it was bought for.
Where projects stall
Teams usually get layers 1 and 2 working first. A served model feels like progress. Then the project stalls, because layers 3 to 7 are where the business requirements live: who may use which model, at what cost, on which data, under which rules, and with what record.
Each of those layers can be built from open-source parts. The cost is the joining: access control that matches across layers, one audit trail instead of five, and one place to see spend.
Build or use a platform
- Build it yourself when you have a platform engineering team, a narrow set of use cases, and time.
- Use a platform when several departments need agents, when audit and cost control are required from the first day, or when the team that bought the servers is small.
- A mixed approach is common: keep the cluster and serving layers you already run, and add a platform above them.
Questions to settle before you choose
- How many departments will use this, and do they need to be kept apart?
- Does any part of it have to run with no outside connection?
- Who has to sign off an agent before staff can use it?
- How will you show an auditor what an agent did?
- Who pays for the GPUs, and how will you divide the cost?
What Katonic covers
Katonic supplies layers 2 to 7 on servers the customer owns: open-weight models served on the customer's GPUs, the AI gateway with budgets per department, the knowledge layer, the agent studio, guardrails and evaluation, and the workspace staff use. It installs on the cluster layer and can run with no outside connection.
Read next
- Use case: token factory for Group IT →
- Use case: agent pilots to production →
- HPE Private Cloud AI or HPE GPU servers? →
- You bought the GPUs. Now what? →
- Katonic AI on HPE →
Questions and answers
Is a served model enough to run agents?
No. A served model answers requests. Agents also need controlled access, knowledge, tools, guardrails, testing and somewhere for staff to use them.
Can all of this run without internet access?
Yes. Every layer can run on the customer's own servers in a disconnected environment, provided the models and software are installed locally.
Do we have to choose one agent framework?
No. Agents built on different open-source frameworks can run side by side, as long as the platform underneath gives them the same gateway, guardrails and logging.
Which layer should we build first?
The cluster and model serving come first because everything depends on them. Put the gateway in next, before many teams start calling models directly, because it is hard to add control later.
How do we share GPUs between departments?
Through the gateway. Each department gets its own access, budget and cap, usage is metered per department, and the cost is charged back from those records.