Technology – Aaron Fulkerson

Researchers pulled 64 of them out of public agent logs, and nobody had typed a single one. The category is agent intermediate state, and nothing in your stack was built to see it.

The Treasury at Petra, seen through the Siq. The gap shows a few feet of a forty-meter facade. Photo by Aaron Fulkerson, Leica Q3, July 2026.

There are 64 secrets sitting in public AI agent logs on GitHub and Hugging Face that nobody ever wrote down.

Nobody typed them. They appear in no transcript and on no screen. A team of researchers recovered them anyway, out of blocks of encrypted text that the developers who published those logs had no way to open and no reason to suspect.

The paper is “Stealing Reasoning Traces from Proprietary LLM APIs,” published August 10 by Alexander Panfilov, David Schmotz, Ilia Shumailov, Luca Beurer-Kellner, Joachim Schaeffer, Ameya Prabhu, Jonas Geiping and Maksym Andriushchenko. Across 6,708 public agent trajectories they decoded 315,320 reasoning blocks and pulled out 62 API keys, 33 passwords, 24 access tokens and seven private keys belonging to real people. Roughly one session in twenty leaked something real. And of the artifacts recovered from genuine user sessions, 64 of 704 were entirely absent from the visible chat history.

They never broke an encryption key.

The mechanism isn’t exotic. OpenAI, Anthropic and Google all hide the model’s step by step reasoning to protect their own IP, handing it back to the client as an encrypted block that the client returns on the next call. Sensible design. Stateless APIs need somewhere to put the state. What the team found is that those blocks are interchangeable across sessions, users and models inside a single provider. So you hand a block from a strong model to a weaker sibling, ask it to read the contents out loud, and it does. Encryption held. Access control held. The secret still walked out.

A developer at a mid size fintech reads every line of a trace, redacts the keys, pushes it clean, and ships the credential anyway.

You can’t sanitize what you can’t read.

Data exhaust used to be inert

Every system throws off byproduct. Logs, metadata, telemetry, build artifacts, the timing and shape of requests. We’ve called it data exhaust for years, and for most of those years it was safe for a boring reason: reading it at volume cost more than the information was worth. A human analyst staring at six months of API logs is an expensive way to learn very little.

Large language models collapsed that cost to roughly zero. What used to be noise is a corpus now, and a corpus can be reconstructed into the thing that produced it. Most security teams have at least heard about that shift.

The second shift is the one this paper makes concrete, and it runs the other direction. We are now building systems whose exhaust only a machine can read. The reasoning block isn’t a log you’re neglecting. It’s a sealed object you are holding, forwarding, and occasionally publishing, with no ability to inspect what’s inside it. Something old moved up stack. The exhaust got a lock on it, and you don’t have the key.

Reasoning traces matter because researchers just proved the failure end to end, with numbers. They’re one instance of a larger category, and the category is what should worry you. Call it agent intermediate state: everything a system generates between the request and the answer. Model reasoning, planner state, tool calls and their responses, retrieved context, memory that persists across turns, environment variables and the credentials inside them.

Or more simply: dark exhaust. Data exhaust you cannot read or track.

Almost none of it reaches the transcript, and most of it is more revealing than the transcript. If providers reimplement reasoning tomorrow in a way that closes this specific paper, the category doesn’t shrink by one line.

Apple built a confidential supercomputer for a chatbot

Apple looked at Siri, a consumer chatbot answering questions on a phone, decided the data exhaust around it was too risky for ordinary cloud infrastructure, and built a confidential supercomputer instead. They call it Private Cloud Compute. It’s a Confidential AI system, and the largest one anyone has ever deployed.

The iPhone measures the server against a published build and refuses to send when the measurement is wrong, so the client gets cryptographic evidence about the environment before any data reaches it. That check has a name. Verifiable privacy: you confirm the guarantee yourself. Verifiable governance: the policy runs where nobody can edit it, and enforcement leaves a record you can hand to a regulator. Compliance becomes evidence you produce. And all of this is to prevent the user data, or data exhaust, from being accidentally leaked, which happens by default in GenAI systems.

In June, Apple extended the whole thing onto Google Cloud rather than relax any of it to get more capacity. Ivan Krstić, who runs security engineering at Apple, walked through the architecture on stage at the Confidential Computing Summit we host. I was thrilled they agreed to keynote. I strongly recommend listening to his presentation because he breaks down all the adversarial and non-adversarial threat models.

Anyway, Apple built this for a simple consumer question/answer chatbot. Enterprise agents are far more capable and dangerous.

An agent’s intermediate state is not a transcript; it’s enterprise data from across a variety of systems

An agent doesn’t just answer questions. It acts. To act, it retrieves, plans, tries, discards, and decides. The visible output is the smallest artifact it produces, and often the least interesting one.

Take a health system running an agent on prior authorization appeals. The output is one word: appeal. Getting there took the patient identifiers it pulled to check eligibility, the three denial codes it weighed, the contract clause it considered citing and dropped, and the dollar threshold that tells it not to bother below a certain claim. One of those five things reaches the transcript. The other four are what a competitor, a payer or a plaintiff’s attorney would actually want.

What I’ve observed is that teams inventory their prompts and they inventory their outputs. Almost nobody inventories the deliberation in between. Your agents keep a diary, and you have never read a page of it.

And it’s reachable by people outside your company. A federal court has already ordered OpenAI to preserve output logs its own deletion policy would have destroyed, and Rule 34 carries no exception for data a machine generated. Preserving and producing a record nobody at your company can read is its own article, and there’s a starting point for it in the reading list.

Now scale it. The paper’s core finding is that reasoning blocks are interchangeable across sessions, users, and models inside a provider. Read that as an architecture statement rather than a vulnerability report, and it describes something you are probably building on purpose. It’s what an agent assembles internally to get the job done, and it’s what an agent-to-agent handoff hands over. Every handoff moves intermediate state across a boundary.

Roughly one session in twenty leaked something real, and those were single sessions with a human somewhere in the loop. That rate was tolerable when a person reviewed the output and the work stopped for the night. It isn’t tolerable for systems that run continuously, hand off to each other, and multiply. Hierarchical access controls were designed for a world where the number of actors grew slowly. Agents don’t grow that way.

A patch is not the fix

Credit where it belongs. The researchers disclosed before publishing, and by the time the paper went public all three providers had shipped fixes. The frontier labs are fast and serious about this.

This instance is closed. The pattern isn’t.

And notice whose stack it was in. Three companies with dedicated world-class security research teams, full control of their entire stack, and a private disclosure before publication. They shipped in days.

This pattern is actually more damning with your enterprise agents, and nobody is going to disclose it to you privately first. Is your enterprise as fast as a frontier lab?

A patch closes a demonstrated attack, not the design decision that produced it, and that decision is an anti-pattern running through GenAI broadly: an intermediate state moving across a trust boundary with nothing binding it to who produced it, who may replay it, or what’s inside. Same shape, different surface, shipping inside enterprise agents right now.

In a chatbot, an instance of this might cost you some personal data or maybe a credential. In a fleet of enterprise agents, it costs you the deliberation behind every decision the fleet makes across every system it touches. Even if this pattern were resolved by the frontier model labs, the much bigger and more frightening issue is that, across all the enterprise agents, a secret, invisible scratchpad is being created and handed off that contains a wealth of sensitive data.

But Sandboxes…

Two objections come up here. The first is isolation: we already have an answer for untrusted code, so run the agent in a sandbox. Constrain its filesystem, its network, its tools.

Do that. It’s table stakes. It solves a different problem.

A sandbox governs what an agent can reach. It says nothing about who can read what happens inside it. The hypervisor can. The host operating system can. The cloud operator can, and so can anyone holding root on a platform your team doesn’t run. Memory sits there in the clear, because the boundary was drawn to keep the workload in, not the infrastructure out.

And look at how the leak in the paper actually happened. Nothing escaped anything. The reasoning block walked out the front door as ordinary API traffic, because that is the protocol. A sandbox is built to permit precisely that call, and it would have watched the whole thing and correctly done nothing. Sandboxing answers what this agent can touch. It was never built to answer who can see what it touched.

The second objection is the one that ends most of these conversations. The scratchpad never leaves our network. Our infrastructure, our employees. Fine.

Except “our employees” is no longer something you can verify by looking. Models are good enough now that an agent impersonating a person inside your network isn’t a hypothetical, it’s the cheapest way in, and the perimeter’s entire premise is that whatever is already inside was let in on purpose. It’s not just possible…it’s probable.

So run the threat model. An impersonating agent gets read access to what your agents generate. Not your document store, which you already monitor. Your intermediate state. That means the service-account credentials your agents hold to reach the data warehouse, the retrieved rows they pulled and discarded, the customer records they compared, the pricing floor a quoting agent decided not to go below, the acquisition target a diligence agent researched under a codename, the vulnerability a coding agent found and filed. None of it is in a document. All of it is in the deliberation.

Then speed. Dark exhaust is the richest and least watched material on your network, and it gets read at machine pace. What a skilled human team would need months to work through, an agent works through in an afternoon. There is no dwell time to detect, because there is barely any dwell.

And the bill changed. The EU AI Act entered its enforcement era this month, and Article 99 tops out at 35 million euros or 7 percent of global annual turnover, a ceiling set deliberately above GDPR’s 4 percent. California’s SB 53 has been in force since January at a million dollars per violation for companies above 500 million in revenue. “It stayed inside our network” was never much of a technical defense. It isn’t a legal one at all.

What the architecture has to satisfy

Skip the product category for a minute and write down what any answer has to do. Three requirements fall out of everything above.

Intermediate state has to be unreadable to the infrastructure that runs it, including your own operators and your cloud provider, because “inside our network” stopped being a boundary. The policy governing that state has to be enforced somewhere the party running the workload can’t quietly edit, because a policy enforced by the party you’re worried about isn’t a control. And the whole thing has to produce evidence a third party can check without trusting you, because a regulator, a customer or a court will eventually ask and your word won’t settle it.

Those are requirements, not a vendor list, and candidates land differently against them. Application-layer encryption covers data at rest and in flight, then leaves it in the clear during inference, which is exactly when intermediate state exists. Sandboxes bound reach, as above. Audit logs are generated by the operator and can be altered by the operator.

Hardware-backed Trusted Execution Environments plus remote attestation satisfy all three today, which is the case for Confidential AI. Before the workload runs, attestation proves what code is loaded and what it’s permitted to touch. During execution, policy binds to the data inside the TEE, so intermediate state is unreadable to the operator, the cloud provider and the model host alike. After it finishes, you hold an attested, tamper evident record of what ran on what. Before, during, and after.

Be clear about what that doesn’t buy. It doesn’t make the agent correct, and a confidential agent executes a bad plan as faithfully as a plain one. It doesn’t write your policy, and it enforces a careless one precisely. It does nothing at all if nobody ever decided what the agent was allowed to touch. Confidential AI makes a guarantee checkable. It doesn’t make the guarantee good.

That’s the same list Apple built for a consumer chatbot, and enterprises are in that position with less room to maneuver: agents on clouds you don’t own, models you didn’t train, vendors you can’t audit. That’s the problem we work on at OPAQUE, and we’re one of several teams working it. The category matters considerably more than the vendor.

Which is exactly why the evidence has to be a standard and not a product. TRACE, for Trust, Runtime Attestation, and Compliance Evidence, specifies the format and the verification rules for provable evidence that an agent ran under a stated policy, in a verified hardware environment, on data of a given classification, invoking identified tools. Open specification, Apache 2.0, moving through the Linux Foundation, at trace.agentrust-io.com. Every supercycle we’ve had ran on open standards, and not one was won by whoever held the best proprietary format.

Before you deploy the next one

None of this requires you to buy anything this quarter. It requires you to be able to answer five questions about an agent you already run.

What sensitive information enters it. What sensitive information it retrieves on its own. What intermediate state it generates along the way. Who can technically read that state, including your cloud provider, your model host and your own platform team. And what evidence you could produce, to someone who doesn’t trust you, that nobody did.

Most teams can answer the first two today. The third is usually a shrug. The fourth is longer than people expect once they write it out. The fifth is where the architecture argument stops being theoretical, because for almost everyone the honest answer right now is none.

If you want to see what that evidence looks like in practice, the cMCP quickstart walks through blocking a tool call and verifying the receipt. It takes a few minutes and it makes question five concrete.

Panfilov and his coauthors didn’t find a bug. They found the shape of the next fifty.

Category: Technology

AI Finds Secrets You Never Wrote Down