NemoClaw on OpenShift

00 The ask

I've been writing and building in this space for more than a year, and this architecture is the first one I've assembled that feels like a complete enterprise answer rather than a collection of disconnected controls. Not because any one product got dramatically better, but because the pieces finally compose: an agent runtime worth isolating, a platform that can isolate it properly, a gateway that governs its tool traffic, an identity system that decides each call, a secrets engine that makes every credential temporary, and an observation layer that watches the whole thing, and the ability to have an automated way to kill a session or an agent or the mcp traffic.

The trigger was NVIDIA NemoClaw, announced in March 2026 and moving fast ever since. In my last post I ran an agent inside NVIDIA OpenShell on a single host and layered identity controls on top. The obvious next question, and a feeling that I couldnt get out of my head....what does this look like if we did an enterprise deployment? Like on OpenShift, on infrastructure an enterprise already operates, patches, and audits? Not a laptop demo or an EC2 instance or someone else's runtime. A cluster.

Answering that question can only be done one way, prove it: NemoClaw does not run on OpenShift today, not as a supported product. So section 01 is about what NemoClaw actually is, verified against NVIDIA's own documentation this week, and the rest of the post is about the architecture I built and proved around its agent runtime on a real OpenShift cluster. Every capability claim in this post is either cited to primary documentation or backed by my implementation with captured output from that build.

01 What NemoClaw actually is, verified

The name gets used loosely, so here is what NVIDIA says as of August 13, 2026. NVIDIA NemoClaw is an open source, Apache-2.0 project that NVIDIA's own README calls "an alpha project" and whose docs describe as an early-preview reference stack for running sandboxed AI agents, designed for a trusted operator on one host. One curl command installs the NVIDIA OpenShell sandbox runtime, local Nemotron models for inference, and one of three agent frameworks: OpenClaw (the default, now stewarded by the OpenClaw Foundation), Nous Research's Hermes, or LangChain Deep Agents. The release cadence is close to daily; v0.0.108 was tagged the day I started writing this. This gets deep...I have seriously learned so much in this build. Lookup Nous Research Hermes...really good stuff..anyway.

The sandbox underneath is OpenShell, and its default isolation is a container, not a VM: Landlock for filesystem rules, seccomp for syscalls (Seccomp filters which kernel system calls the agent can invoke), and a policy gateway that intercepts every egress and swaps credential placeholders for real values at the boundary, so the agent process never holds a usable secret. That resolve-token design is genuinely good, and I have said this before. 

Getting deeper into this build. OpenShell has official OpenShift install docs, but they are labeled experimental, they rely on the Kubernetes SIG Agent Sandbox controller rather than OpenShift sandboxed containers, and they currently require the privileged SCC with the guidance "use only for evaluation on a private network." NemoClaw itself has no Kubernetes documentation at all; its Kubernetes and multi-tenancy qualification is an open design-stage epic. Red Hat and NVIDIA have said publicly, in a March 2026 collaboration post and a May 2026 Red Hat Summit release, that they are integrating OpenShell with Red Hat AI and contributing upstream. That is a direction, publicly stated. However I had to take this on with as much as I have been doing in this space and building, its a great challenge.

So "NemoClaw on OpenShift" means something specific in this post: I took the agent runtime NemoClaw packages, the actual OpenClaw binary from NVIDIA's sandbox-base image, and ran it inside OpenShift's own isolation and governance stack. The literal nemoclaw onboard flow got close on the same cluster: ultimately failed, because its embedded gateway wants Docker bridge networking and the stock Kata guest kernel ships without the netfilter bridge modules. That gap is bounded, it is the kind of thing a custom Kata guest image fixes.
The architecture below does not depend on it.

02 A sandbox is not an authorization model

Here is the argument that carries the whole post, a sandbox decides what an agent can reach. It filters hosts, ports, files, syscalls. It cannot tell a harmless read from a destructive write on the same endpoint, because to the sandbox both are bytes to an allowed destination.
Authorization, deciding what this identity may do with this tool right now, is a different control, and it has to live somewhere with identity context.

The security community has been circling this conclusion for two years of Black Hat briefings. At Black Hat USA 2025 (August 2025), NCC Group's David Brauchler argued that "AI red teaming has proven that eliminating prompt injection is a lost cause," and that treating guardrails as a first-order control quietly introduces privilege-escalation paths. Last week at Black Hat USA 2026 (August 2026), a Roblox team presented a production incident in which "a hidden instruction in a GitHub Issue convinced Claude Code to upload Roblox's credentials to a public repository," and framed their fix around "the architecture that contains it, and three problems that sandboxing alone cannot solve." Check Point researcher put the diagnosis in one sentence, after disclosing 11 CVEs across six agent frameworks: "The core failure is architectural: frameworks often fail to keep attacker-controlled content in the data plane, allowing it to influence trusted orchestration, memory, state, routing, and system instructions." The model cannot be the boundary, so the architecture has to be.

My own testing shows the example...In the single-host build behind my previous post, I planted instructions in a real Jira ticket telling the agent to delete rows from a Databricks table. The model complied. The sandbox saw nothing wrong, because the delete went to an allowed host. What stopped it was the authorization layer: the delete action sat outside the identity's grant, so the exchange was refused before any credential existed. The injection beat the model and lost to the architecture.

  The model can be manipulated into making a request; it should not be able to manufacture the authority to fulfill it.

03 The layered architecture

Six components, each solving a problem the others cannot. Red Hat and HashiCorp are both IBM companies (Red Hat since July 2019, HashiCorp since February 2025), which is why this build you will not find anywhere else! one vendor family now spans the identity plane, the secrets plane, the platform, the isolation layer, and the observation layer, with commercial support behind each piece. 

Here is the whole thing in one picture. The animation follows a single tool call.

Red Hat enforces the verdict; IBM makes it. The gateway and Authorino are the policy enforcement point. The policy-service, Verify, and Vault are the policy decision point and the credential release. The agent is a workload that gets to ask, and a sandbox is where it waits for the answer.

04 Isolation: OpenShift sandboxed containers - within Kata

NemoClaw's own sandbox is a container, which means it shares the host kernel with everything else on the node. On a developer laptop that is a reasonable trade. On a shared GPU cluster serving multiple tenants, a kernel exploit inside the sandbox is a cluster problem, and no amount of Landlock policy changes that. OpenShift sandboxed containers closes exactly this gap: it runs a pod inside a lightweight VM via Kata Containers, with its own guest kernel, so a breakout from the agent's runtime is still trapped inside hardware virtualization. Version 1.12 shipped in April 2026 and took confidential containers on bare metal to GA, with attestation via the Red Hat build of Trustee and Technology Preview support for confidential NVIDIA Hopper GPUs. The direction of travel for agent isolation on this platform is clear and it is measured in hardware.

Running the real OpenClaw binary in a Kata pod took three manifests worth of decisions, and each one is a boundary. First the pod itself: runtimeClassName: kata, unprivileged, capabilities dropped, in a dedicated namespace.

# The agent runtime, in its own VM. Trimmed from the working manifest;
# the full file pins the image by digest and adds a readiness probe.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openclaw
  namespace: ${OPENSHIFT_NAMESPACE}
spec:
  template:
    metadata:
      labels: { app: openclaw, role: agent }   # role=agent binds the NetworkPolicy
    spec:
      runtimeClassName: kata                # the VM boundary, per pod
      serviceAccountName: agent-runtime
      containers:
        - name: openclaw
          image: ghcr.io/nvidia/nemoclaw/sandbox-base:v0.0.103
          env:
            # the agent reaches the model ONLY through the egress proxy;
            # the real key is injected outside the sandbox
            - { name: ANTHROPIC_BASE_URL, value: "http://llm-egress.${OPENSHIFT_NAMESPACE}.svc:8080" }
            - { name: ANTHROPIC_API_KEY,  value: "placeholder-llm-egress-injects-the-real-key" }
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }

What this protects: the node, and every other tenant on it. How to validate it: from inside the pod, uname -r returns a guest kernel that differs from the node's, and the pod's podIP differs from hostIP with hostNetwork unset. Both checks are recorded in my probe log as NEMOCLAW-P2, alongside the negative control that matters more than either: a pod requesting an unknown RuntimeClass is refused at admission instead of silently falling back to runc. Silent fallback is the failure that would make this whole layer decorative, so it is the thing to test first.

Second, reachability. The namespace carries a default-deny egress NetworkPolicy plus a narrow allowlist, and the allowlist is the entire reachable universe: the MCP gateway, the LLM egress proxy, and cluster DNS.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: agent-runtime-egress-allowlist
  namespace: ${OPENSHIFT_NAMESPACE}
spec:
  podSelector: { matchLabels: { role: agent } }
  policyTypes: [Egress]
  egress:
    # CRITICAL: namespaceSelector AND podSelector in the SAME to[] element.
    # Split into two elements they are ORed, and the allowlist blows open.
    - to:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: nemo-claw } }
          podSelector: { matchLabels: { gateway.networking.k8s.io/gateway-name: mcp-gw } }
    - to:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: nemo-claw } }
          podSelector: { matchLabels: { app: llm-egress } }
    - to:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: openshift-dns } }
      ports: [{ protocol: UDP, port: 5353 }, { protocol: TCP, port: 5353 }]

What this protects: It prevents an agent from reaching systems it has not explicitly been allowed to use. Think of the gateway as a controlled front door: the agent can use approved paths, but it cannot bypass the gateway to call outside services directly.

We verify this by testing both sides. From inside the agent’s isolated environment, direct connections to the public internet, the model provider, and the underlying MCP service all fail. The same requests work from a normal workload that is not running as an agent. That comparison matters: a failed request alone may mean the test is misconfigured; proving that the request works elsewhere shows the restriction is intentional.

We also plan for cases where an agent may eventually need elevated permissions inside its own isolated runtime. Those permissions are granted only to one specifically identified service account, through a narrowly scoped and auditable policy. The workload is still prohibited from accessing the host network or other host-level resources.

This is important because a privileged workload with access to the host network could bypass the network restrictions intended to contain it. Instead, the design keeps host access off-limits and relies on the isolated runtime to limit the impact of any permissions the agent needs. Finally, we verify that only the intended agent identity receives this exception—not every workload in the namespace.

05 Tool governance: Red Hat's MCP gateway

MCP is where agent risk concentrates, because it is where capability concentrates. At Black Hat Europe 2025 (December 2025), Tencent Zhuque Lab researchers reported "more than 500 unique vulnerabilities" across MCP clients and servers and told the audience plainly: "We will prove that MCP is the next major battleground for AI security." Fronting every MCP server with one governed, observable gateway is the structural response to that battleground, and it is no longer a do-it-yourself project.

In May 2026 Red Hat shipped an MCP gateway as a Technology Preview inside Red Hat Connectivity Link 1.4, productizing the upstream Kuadrant mcp-gateway project (v0.8.0). Technology Preview means what Red Hat says it means: no production SLA. Use it accordingly, and know that the underlying Kuadrant machinery, Gateway API plus Authorino for auth policy.

What the gateway does natively is federation and curation: it aggregates tools from many MCP servers behind one endpoint, lets you define virtual servers that expose a curated slice, and handles MCP protocol and sessions in an Envoy data plane. That governs what an agent can see. What it deliberately does not carry is a per-call authorization engine that understands RFC 9396 grants: it is an enforcement point, and it exposes exactly the right seam to attach a decision engine. That seam is Authorino's AuthConfig.

# The composition seam: Authorino authenticates the caller, calls out to
# the IBM policy service for the DECISION, gates on it, and injects the
# released credential upstream. The agent never sees a secret.
apiVersion: authorino.kuadrant.io/v1beta3
kind: AuthConfig
metadata:
  name: mcp-tool-authorization
spec:
  hosts: [mcp.internal, databricks-mcp.internal]   # every federated upstream hostname
  authentication:
    verify-jwt:
      jwt:
        issuerUrl: ${IBM_VERIFY_ISSUER}       # signature dies here with a 401
  metadata:
    policy:
      http:
        url: http://policy-service.${OPENSHIFT_NAMESPACE}.svc:8090/authorize
        method: POST
        bodyParameters:
          authorization: { selector: context.request.http.headers.authorization }
          mcp: { expression: 'request.body' }   # which tool, which arguments: per CALL
  authorization:
    allowed:
      patternMatching:
        patterns:
          - { selector: auth.metadata.policy.decision, operator: eq, value: allow }
  response:
    success:
      headers:   # the credential goes UPSTREAM, never back to the agent
        x-db-password: { plain: { selector: auth.metadata.policy.credential } }
    unauthorized:
      code: 403
      message: { selector: auth.metadata.policy.reason }  # tell the agent WHY

What this protects: the boundary between "the agent can see a tool" and "the agent may invoke this tool, with these arguments, right now." Validation is direct: call a permitted tool and get rows back; call a forbidden one on the same session and same bearer and get a 403 whose body names the policy reason; strip the bearer entirely and die at the JWT stage with zero policy-service log lines. 

06 Identity: IBM Verify decides every call

The human signs in from whatever IDP (does not have to be IBM Verify), in my build I used Microsoft Entra ID at the request of a customer, and that federation boundary is respected: every attribute the system needs about the user comes from token claims, never from a directory lookup that assumes local enrollment. The session token then becomes the subject of an RFC 8693 token exchange at IBM Verify, performed by the policy service on every single tool call. The exchange carries RFC 9396 authorization_details built from the specific tool and arguments, and what comes back is a token that names one action on behalf of one human, performed by one attested workload.

{
  "sub": "<the signed-in Entra user>",
  "act": { "sub": "spiffe://astra.internal/ns/nemo-claw/sa/orchestrator" },
  "scope": "databricks:read",
  "authorization_details": [{
    "type": "mcp_tool",
    "action": "databricks_query"   // one action, not a role
  }]
}

The act claim is the part I want newer readers to sit with. It is the difference between "a user's token got replayed by some process" and "this specific workload, whose identity was attested by the cluster, acted on behalf of this specific user for this specific action." Verify validates the actor's SPIFFE JWT-SVID against the cluster's published JWKS and refuses the exchange if the actor is not authorized to act for the subject. 

Rich Authorization Requests , where authorization_details types are declared and schema-validated in configuration. These same RFC 9396 pattern work against a Verify SaaS tenant or IVIA, and we are using Adaptive access, the risk-scored authentication context that can force step-up on a risky session, is documented for the SaaS 

07 Secrets: Vault makes every credential temporary

HashiCorp Vault moved to its 2.x line in April 2026, and the 2.0 release notes put SPIFFE-based workload identity on the headline list, which is exactly the primitive this architecture leans on. The Vault Secrets Operator is certified for OpenShift, and my cluster runs Vault Enterprise 2.0.4. Three of Vault's jobs are crucial, and each belongs to a different identity.

The agent's identity: the pod logs in with its projected Kubernetes service account token via the Kubernetes auth method, Vault validates it against the API server, and the returned Vault token carries one policy. That policy lets the agent mint its own ttl SPIFFE JWT-SVID and read the JWKS. Nothing else. Here it is in full:

# agent-svid: the agent may prove who it is. It may not touch credentials.
path "spiffe/role/nemo-claw/mintjwt" {
  capabilities = ["create", "update"]
}
path "spiffe/.well-known/*" {
  capabilities = ["read"]
}

# mcp-gateway: the enforcement plane may release credentials and revoke
# leases. It may NOT mint an agent identity. Compromising one identity
# does not compose into the other's authority.
path "verify-rar/creds/*" {
  capabilities = ["read", "create", "update"]
}
path "sys/leases/revoke" {
  capabilities = ["update"]
}

What this protects: the blast radius of a fully compromised agent. AKA the last mile! The validation is the simplest in the whole build because it is so plain: have the agent present its own Vault token directly to verify-rar/creds/dbx-read and watch Vault return 403 permission denied. The agent can be talked into anything; its Vault policy cannot.

The credential itself: after the Verify exchange succeeds, the policy service presents the resulting on-behalf-of token to Vault's verify-rar secrets engine (a custom plugin this blog has covered in depth), which checks the RFC 9396 grant a second time and mints a downstream credential with a 300-second lease. The gateway uses it, injects it upstream, and revokes the lease in a finally block, success or failure. The response envelope carries credRevoked populated from the actual revoke result. A field that can say false is a control.

[policy-service] ALLOW tool=databricks_query rar=databricks_read exchange=ok
  lease=verify-rar/creds/dbx-read/8kQz... ttl=300 credRevoked=true jti=1f37c0...
[policy-service] DENY  tool=databricks_delete reason=policy_deny   # same session, same bearer

And the audit trail: Vault's file audit device records every login, mint, and revoke with the entity that performed it, which is the backbone the correlation story in section 0.
Enable it on day one; retrofitting audit is how evidence gets lost.

08 Observation: Verify Identity Protection and the kill path

IBM Verify Identity Protection is the observation layer of this architecture: identity security posture management plus identity threat detection and response, agentless, watching activity across identity providers, MFA, SSO, and the service accounts that agents and gateways authenticate as. In an agent architecture its job is the questions the enforcement path cannot ask about itself. Which identities are accumulating grants? Which service account started authenticating from somewhere new? Which MFA enrollment quietly changed before a step-up was approved?

Detection without a response path is a dashboard, so the wiring that matters is the loop: a risk signal becomes a revocation. The mechanism is the OpenID Shared Signals Framework with CAEP events, the same pipeline this blog built for the MCP directly. On the cluster, a CAEP receiver accepts a session-revoked Security Event Token, empties the agent's Vault leases, and closes the gateway's gate for that subject. From event to enforced lockout is seconds, and the next call from the agent bounces.

09 The AWS leg: EC2 demo agents and correlated telemetry

Not everything runs on the cluster, deliberately. The live demo environment from my earlier builds runs on an EC2 host: a RHEL box carrying the sibling agent stack, its own Vault container, and the same IBM Verify tenant, brought up and down by scripts because a demo host that runs around the clock is a bill and an attack surface. It earns its place in this architecture by being the external runtime that proves the controls are not cluster folklore. The identity chain, Entra subject, Verify exchange, RAR grant, Vault mint, behaves identically from a VM in AWS, and its telemetry lands in different places, which is exactly what makes correlation worth practicing.

Three AWS-native signals do real work here. CloudTrail records the control-plane story: who started the instance, who changed its security group, which never contains more than SSH pinned to my current address. CloudWatch carries the operational story: a CPU-idle alarm auto-stops the host after enough quiet hours, so an abandoned demo shuts itself down. And the application story comes from the stack itself: eleven health endpoints, Vault seal state, and the token-exchange log. The CloudTrail versus CloudWatch split is worth internalizing once: one answers "who did what to the account," the other "what is the workload doing."

One incident from that host taught me more about cross-environment identity than any diagram. After the Vault container was re-created, every token exchange started failing with Verify's CSIAQ0197E while every health check stayed green. The cause: Verify validates the workload's SPIFFE SVID against a public JWKS mirror, the mirror was serving the previous Vault's keys, and nothing in the health surface covered "the identity provider can still verify us." The fix was mechanical; the lesson was structural. Identity freshness is a health check. The bring-up script now compares the live Vault's JWKS against the mirror and self-heals before anything else starts. Correlation is what found it: a Verify-side error code, a Vault-side key rotation, and an AWS-side container re-creation are three log lines in three systems that are one event.

Here is the shape of that correlation, assembled from the fields each layer really emits. This example is illustrative, with sanitized values, but every field name is one you have seen earlier in this post:

{
  "traceId": "one tool call, five systems",
  "user":   { "sub": "<entra-user>", "idp": "entra" },
  "agent":  { "act": "spiffe://astra.internal/ns/nemo-claw/sa/orchestrator",
              "pod": "openclaw-6d5f...(runtimeClass=kata)" },
  "tool":   { "name": "databricks_query", "decision": "allow", "rar": "databricks_read" },
  "vault":  { "lease": "verify-rar/creds/dbx-read/8kQz...", "ttl": 300, "revoked": true },
  "aws":    { "source": "ec2:i-***", "cloudtrail": "StartInstances by <operator>" },
  "risk":   { "signal": "none", "caepSubject": "<entra-user>" }
}

If you would rather run the whole architecture AWS-native, the platform story holds: Red Hat OpenShift Service on AWS is GA in both its hosted-control-plane and classic architectures, and OpenShift sandboxed containers' peer-pods mode extends Kata isolation into the cloud by backing each sandboxed pod with a cloud VM. I have not run the peer-pods variant of this build, so treat that pairing as documented capability, not my evidence.

10 The full flow, end to end

Everything above, in order, as a single read path. This is the flow my probes execute.

The sensitive-action variant is the same chain with a human inserted at the decision. A tier-3 write does not fail and does not proceed: the exchange returns a challenge, the transaction parks with a transaction id, and a one-time code goes to the email address taken from the signed-in user's token claims. The human approves; the flow resumes at the mint step; the credential still lives for five minutes; the evidence still lands in every log. I have watched this cycle complete live, code from a real inbox, on the cluster.

11 Validation

Security controls should be judged by what they stop, not by how they are configured or described.

• Unapproved tool. Same agent, same session, same bearer: databricks_query returns rows, databricks_delete returns 403 policy_deny before any external call, and the tool server is never contacted. 
• Unauthorized egress from the sandbox. From inside the Kata pod: the public internet, the model API, and the raw upstream MCP all fail to connect; the gateway and LLM proxy connect. The identical unlabeled pod reaches all of them.
• Vault token outside its policy. The agent's own Vault token asks for a credential directly and gets 403 permission denied; the gateway's token asks to mint an SVID and is refused the same way. After the 300-second lease expires or is revoked, replaying the minted credential fails at the backend; expect an auth error, and treat success as a sev-1 finding.
• High-risk identity event. Fire a CAEP session-revoked SET at the receiver: the subject's leases empty and the next tool call bounces within seconds. Recorded from a hand-fired event (REV probes); with Verify Identity Protection as the transmitter this becomes the automated response path.
• EC2 correlation. Stop and start the demo host, then follow one tool call across CloudTrail (instance lifecycle), the workload health surface, Verify's token-exchange log, and Vault's audit device. Expected: one narrative from four systems keyed by subject and time, and a stale identity artifact (my JWKS mirror incident) surfaces as an exchange error, not as a green dashboard.
• Sensitive action requires a human. Ask the agent for the tier-3 write: the transaction parks, a code arrives at the signed-in user's real inbox, a wrong code counts toward the kill threshold, and the right one resumes the flow through a fresh five-minute credential. 

12 What is next

Two threads point forward. First, agent identity at registry scale. Every SVID in my build already carries an agent_registry_id claim, stamped at mint time from a template the agent cannot influence, because I wanted the architecture ready for the day agent identities are catalogued, owned, and lifecycle-managed like any other enterprise identity. An agent identity registry, IBM's included, is the anticipated extension of this model: the registry answers "which agents exist and who owns them," and this runtime answers "what is this one allowed to do right now." I am building toward that seam.

Second, orchestration. Enterprises are not going to hand-assemble agents forever; platforms like IBM watsonx Orchestrate exist to build and run them at business-process scale. The direction this architecture is designed to support is exactly that pairing: an orchestration platform composing agents, each of which executes inside a runtime like this one, sandboxed, identity-bound, and governed per call. 

Nearer term, on the cluster itself: pointing inference at on-cluster NVIDIA NIM or vLLM instead of the egress proxy, which the Red Hat AI Factory with NVIDIA platform is built for and which changes nothing in the security architecture, and watching the confidential-GPU Technology Preview mature, because "the model's memory is encrypted even from the node" is the natural endpoint of the isolation story this post started.

13 Closing perspective

NemoClaw will keep moving; it shipped eight releases in the twelve days before the writing of the post. The OpenShift one-click is what im building: a VM boundary the platform enforces, an egress list with three entries, a gateway that asks an identity system before every tool call, credentials that expire before a meeting ends, and telemetry that lets one person reconstruct one action across five systems. None of that waits on NVIDIA's roadmap. All of it runs today, on supported enterprise products, with the seams documented above.