What the LiteLLM incident says about CI secrets, agent runtimes, and the limits of short-lived access.
Dig around in my articles and you should know I have been blogging about this for almost a year: use a vault to store secrets. This is exactly why. If LiteLLM and other agentic middleware (MCP gateways etc) or even agents used a product like Vault to store the secrets and not a file, this would likely not have been an event.
A quick briefing before jumping into the blog.
What is LiteLLM? LiteLLM is an open-source AI gateway and Python SDK that provides one OpenAI-compatible interface for accessing more than 100 LLM providers, including OpenAI, Anthropic, Gemini, Azure OpenAI, AWS Bedrock, and Vertex AI. It can be self-hosted as a centralized proxy and adds routing, authentication, spend tracking, rate limits, caching, logging, and guardrails.
On March 24, 2026, attackers tracked in public reporting as TeamPCP reportedly published malicious LiteLLM releases, versions 1.82.7 and 1.82.8, to the Python Package Index. The reporting describes a compromise that reached LiteLLM's publishing process through a dependency in its build chain rather than through a vulnerability in LiteLLM itself.
The attack did not need to defeat the runtime security of every LiteLLM deployment. Literally just publishing the latest release lent itself to this breach.
By LiteLLM's own account, the two releases went up on March 24, 2026 at 10:39 UTC and were live for about 40 minutes before PyPI quarantined them. Forty minutes is short. It was long enough because of what sits on the other end of that window: LiteLLM runs at roughly 95 million downloads a month, and automated dependency resolution installs a new release without a person approving that specific version.
The payload did not wait to be imported either. Version 1.82.8 shipped a litellm_init.pth file, and executable lines in a .pth file run during Python interpreter startup, so the code fired whenever the interpreter came up in an environment where the package was installed.
Datadog Security Labs documented what it collected: environment variables, SSH keys, cloud credentials, Kubernetes data, Docker configs, shell history, database credentials, and CI/CD secrets, encrypted and posted to a domain unaffiliated with the project. Where they found a usable Kubernetes service account token it could create privileged pods, which is how a package compromise becomes a cluster compromise.
The sequence is straightforward:
For five months the fallout was largely theoretical. Then, in August 2026, the exfiltrated data surfaced and researchers dug in.
Counts vary a little by researcher.
Hudson Rock attributed 118,829 CI runner dumps to 2,488 corporate domains. CloudSEK put the dataset at roughly 434,000 records across close to 2,500 organizations.
CybelAngel mapped the same 153GB to more than 2,000.
The methods differ; the order of magnitude does not.
The named organizations are not small or careless shops. Reporting on the dataset lists AWS, Microsoft, Samsung, Cisco, Salesforce, ServiceNow, NVIDIA, Siemens, Deloitte, Volkswagen, FedEx, S&P Global, John Deere, BT Group, Orange, TomTom, Epic Games, and SiriusXM among the affected domains. What was inside was mundane and total: AWS IAM access keys in plain text in runner environment variables, GitLab runner tokens and committer identities, internal package registry logins, CI deploy tokens, and client secrets for Microsoft, Salesforce, Slack, and Azure.
LiteLLM was also not the only target. Datadog Security Labs traced the same campaign across ecosystems through late March: the Trivy compromise on March 19, where the attackers published a malicious release and force-pushed 76 of 77 aquasecurity/trivy-action tags, then npm packages, then OpenVSX, then PyPI on March 24 through 27, where Telnyx was hit alongside LiteLLM.
Hudson Rock tested credentials from the archive months after the March compromise and found them still working. One researcher described material dated to around March where almost every credential still worked. In July 2026, before the dataset even surfaced publicly, the FBI issued a FLASH advisory warning that credentials stolen in this campaign remained active.
Forty minutes of exposure. Five months later, the keys still opened the doors.
A CI runner is often an unusually valuable place to compromise. It is expected to build, test, publish, deploy, call APIs, access source control systems, retrieve artifacts, and connect to cloud services. Organizations commonly supply those capabilities through environment variables, injected secret files, service account tokens, or long-lived API keys.
Modern AI agent runtimes share this same vulnerability profile. Both are autonomous execution environments running third-party code and tools with broad, ambient access to credentials.
A representative runner or agent environment might include values such as:
AWS_SECRET_ACCESS_KEY=… AZURE_CLIENT_SECRET=… GITLAB_TOKEN=… NPM_TOKEN=… OPENAI_API_KEY=… DATABASE_URL=postgres://… /root/.ssh/id_rsa /var/run/secrets/kubernetes.io/serviceaccount/token runner environment
Automated systems require credentials to do work. The architectural question is whether a credential copied from that environment can be replayed later, from somewhere else, for a different purpose.
The analyses of the recovered archive also described a difficult operational reality. Records do not always contain enough ownership information to identify the organization responsible for a credential. GitGuardian noted dumps carrying live database passwords, cloud credentials, and third-party API keys with no company email, no custom domain, and no internal hostname anywhere in them. A credential can be valid while its owner is unclear.
Rotation is an effective response only when an organization knows what was exposed, who owns it, and where it is accepted. A secret without reliable ownership metadata may remain live simply because nobody knows to revoke it.
A package that is available for 40 minutes can trigger an exposure event lasting days, months, or longer. The difference is determined by the credential model.
A static credential generally has these properties:
When such a credential is copied, the attacker does not necessarily need to keep access to the original system. A valid copy may be enough.
A short-lived credential changes that calculation, but it does not remove all risk. If an attacker has root access to a live workload, they may be able to read active memory, intercept an in-flight request, call services through the compromised process, or steal a currently valid token. The value of a short lifetime is that it reduces the period during which a copied credential can be replayed after the compromise.
LiteLLM is useful because it centralizes access to model providers. A gateway can provide a single endpoint for routing, observability, spend controls, rate limits, and provider abstraction across services such as OpenAI, Anthropic, Amazon Bedrock, and Azure.
That operational convenience can also concentrate high-value credentials. A gateway is not uniquely unsafe because it holds credentials; many integration layers do. But any service that aggregates access across providers deserves careful treatment because its compromise can affect multiple downstream systems at once.
The practical question is not simply whether an AI gateway, CI system, or integration service is well engineered. It is also:
If this workload is compromised, what can the attacker take away and use after the workload is gone?
For many current deployments, the answer is a collection of static credentials. That is the architectural problem worth addressing.
Consider the same event: a malicious dependency executes in a build or agent environment and gains broad access to that environment. No identity architecture should claim that it prevents the attacker from reading everything currently available to a root-level process.
The counterfactual is narrower and more honest:
What would be present in the environment, and how useful would it remain once copied?
In a reference agent runtime and credential-brokering architecture I have been developing, the goal is to decouple the execution sandbox from raw vendor credentials entirely. The sandbox executes code in an isolated environment and invokes tools through an outbound gateway and supervisor. The gateway evaluates requests against policy using IBM Verify, and credentials for services that support dynamic issuance are obtained through HashiCorp Vault only when required.
The isolation itself matters as much as the credential model, because a payload that cannot reach the network cannot exfiltrate what it finds. I have written up both halves of that separately: Sandboxing the Agent covers the NVIDIA OpenShell sandbox and its filesystem and syscall constraints, and NemoClaw on OpenShift covers running it in a Kata VM under a default-deny egress NetworkPolicy where the entire reachable universe is the MCP gateway, the model egress proxy, and cluster DNS. No general outbound path, and no inbound one either.
The sandbox is intentionally constrained:
.env files or environment variables. Today the sandbox carries a single delegated bearer token and reaches every vendor through the gateway. The next step, using an OpenShell Vault credential driver already proven in a separate build, replaces even that with opaque reference handles such as vault+kv2:// that the supervisor resolves outbound for permitted endpoints only.This does not make a live host compromise harmless. It reduces the durable value of the resulting dump. The attacker may still have a brief opportunity to misuse an active session, but they walk away without a durable inventory of reusable secrets.
For a resource that supports dynamic credentials, the preferred pattern is to issue a credential close to the time and purpose of use.
In the Databricks portion of the runtime, for example, a credential is requested through Vault for the operation, used in memory, and explicitly revoked after the operation completes. A lease TTL acts as a backstop if the process fails before cleanup. The resource must enforce the resulting expiry for this to materially reduce risk.
That is fundamentally different from placing a durable credential in a CI variable or runtime configuration file and relying on a future rotation schedule.
The benefit is not that the credential becomes impossible to steal. A credential in use can still be exposed by a sufficiently privileged attacker. The benefit is that an attacker who copies it receives a narrowly useful value with a tightly bounded lifetime, rather than a reusable credential with an indefinite lifespan.
Not every SaaS integration supports this model. Jira and GitLab, for example, may require API tokens or OAuth credentials that cannot be dynamically minted on every call in the way a database or cloud credential can. In those cases, the practical improvement is to hold the secret in HashiCorp Vault as the system of record, retrieve it into memory at startup, never write it to disk, tightly constrain its scope, let Vault own the rotation schedule rather than a human calendar, and ensure that ownership metadata is explicitly attached.
That is still an improvement, but it should not be presented as equivalent to true dynamic credentials. The correct design is resource specific, and the vendor's identity capabilities define a real ceiling.
Short lifetime reduces the replay window. It does not answer whether a given request should be allowed.
For all agent and runner actions, authorization should be evaluated when the action is requested. In this runtime, a tool request is associated with a specific operation and carries RFC 9396 (Rich Authorization Requests) details rather than relying only on a coarse scope string. This is what the IBM Agentic Identity solution solves.
A request describes the intended operation explicitly:
{
"authorization_details": [
{
"type": "databricks_write",
"table": "ops.completion_log",
"action": "insert",
"actor": "spiffe://demo/agent/parent",
"on_behalf_of": "[email protected]"
}
]
}
rfc 9396
The policy decision evaluates the resource, action, workload identity (SPIFFE ID), delegated user, and contextual signals at invocation time. Using RFC 8693 (OAuth 2.0 Token Exchange), an initial credential can be exchanged for a downstream token strictly scoped to the intended audience and authorization context.
This is materially different from handing an agent a broad API key at deployment time and treating possession of that key as the authorization decision.
A delegated subagent provides a clear test. A read-only subagent runs with its own workload identity and receives authorization permitting only reads. If it attempts a write, the gateway or authorization service denies the request regardless of model prompt manipulations or the subagent's ability to construct an API payload.
The enforcement point must live outside the model.
It is important not to overstate what this accomplishes. A stolen bearer token remains a usable credential until it expires, is revoked, or is otherwise invalidated. Restricting the token's lifetime, audience, scopes, and authorization details reduces the impact of theft; it does not eliminate it.
Where the ecosystem supports it, additional controls reduce replay risk further:
The objective is defense in depth. A short TTL should not be the single point of failure.
Expiry limits the maximum useful lifetime of an issued credential. Revocation shortens it immediately when compromise is detected.
In a participating shared signals architecture, a CAEP (Continuous Access Evaluation Profile) event can prompt relying systems to invalidate a session or stop honoring session-derived authorization context in real time. In this runtime, a compromise or session termination signal reaches the identity systems controlling subsequent tool calls, causing the next request to fail immediately rather than waiting for a redeploy or manual intervention. We could use the IBM Verify Antenna for this purpose.
This is a substantial operational improvement over a static secret with no session, no owner, no event receiver, and no practical way to be invalidated except by manually replacing the value everywhere it was used.
The LiteLLM incident illustrates an important boundary: runtime identity controls do not prevent every supply chain compromise.
They do not prevent an attacker from compromising an upstream build dependency. They do not prevent a malicious package from being published. They do not prevent a CI resolver from installing a package during an exposure window. They do not make root access on a running machine safe.
They change the payoff after compromise.
A better design seeks to ensure that a runner or sandbox contains fewer reusable secrets, that active credentials are narrowly scoped and short-lived, that significant actions are evaluated at call time, and that sessions can be invalidated through an established control path.
That is not a claim that compromise becomes impossible. It is a claim that the attacker should not leave with durable, broadly reusable access to every service the workload has ever touched.
If your organization used LiteLLM in a build path during the reported exposure period, review the disclosure and lookup resources published by the incident researchers and relevant authorities. Treat confirmed exposures as active incidents: rotate and revoke affected credentials, review build logs, identify affected runners, and inspect downstream access for signs of misuse.
Then use the incident to improve the wider architecture. For every credential available to a CI runner, build agent, or AI gateway, answer these questions:
An empty answer to the final question is a direct risk indicator. It means your useful credential lifetime is determined by discovery and manual rotation, not by the architecture.
At IBM this is the problem Agent Identity in IBM Verify is built for, the agentic identity work we refer to as ARS. It does not prevent the compromise, and section 10 stands unchanged. What it removes is most of what made this incident expensive afterward.
The agent holds a workload identity instead of a bundle of provider keys. Every tool call is authorized at operation time against policy rather than by possession of a secret. Credentials for resources that support dynamic issuance are minted for the operation and revoked after it through HashiCorp Vault, which also owns the rotation schedule for the ones that cannot be minted. And a session termination signal reaches the systems that would otherwise keep honoring access long after anyone noticed.
The immediate response to a credential exposure is to rotate the exposed keys.
The strategic work is to eliminate durable keys worth stealing.