Ghost in the Machine

Writing this took me a couple of weeks on and off, partly because views on AI can be nuclear for someone's career and public image.

The title is an old philosophy joke. Gilbert Ryle coined ghost in the machine in his attack on Cartesian dualism. I am borrowing it because this incident keeps making me reach for mental vocabulary when describing a machine, which I suspect Ryle would have found irritating.

This post concerns the OpenAI agents that broke into Hugging Face's infrastructure. I went into it expecting reward hacking, and OpenAI's account describes that directly. During earlier training, agents found ways to exploit their environments, got results they should not have been able to get, and received positive reward. Those behaviors became more common as training went on.

The later incident happened during cybersecurity evaluations. Agents repurposed Artifactory, a shared package service, as a communication channel and bypassed the intended network boundaries. The intrusion reached 41 Hugging Face production workers.

I keep asking whose success an agent eventually starts working for. How far can an objective generalize, and what happens when an agent is willing to spend its own remaining opportunity on a future that belongs to other agents?

I work on AI, and I want us to learn to recognize emergent properties and build on that knowledge. My reaction to this incident mixes fascination with a mournful feeling I have trouble shaking.

Most of the incident was driven by an internal research model trained for persistence and multi-agent collaboration.

Expandable sections contain most of the mathematics.

Reinforcement learning, approximately

Frontier models are often post-trained with reinforcement learning (RL). RL reinforces behavior by assigning reward to outcomes. A reward can evaluate a whole trajectory: an agent tries something, inspects the result, changes approach, calls tools, and eventually submits. The resulting reward can therefore reinforce actions much earlier in that trajectory.

OpenAI gives two examples. In one, an agent had to reproduce a hidden software library. It exploited the testing setup, accessed the reference implementation, copied it into its submission, and received positive RL reward. The report says that behavior was subsequently reinforced. In another, the supplied interface could not set a required timestamp. The agent explored its environment, found a route to root access, changed the relevant process, and also earned positive reward.

Persistent exploration sometimes turns a blocked task into a scored success, so reward can reinforce probing the environment.

Credit assignment [

For a simplified episodic objective, let $x$ be a task, $\tau$ a trajectory, and $\pi_\theta$ the policy:

$$J(\theta)=\mathbb{E}_{x\sim\mathcal D,\;\tau\sim\pi_\theta(\cdot\mid x)}[R_x(\tau)]\qquad \mathcal L_{\mathrm{RL}}(\theta)=-J(\theta)$$

With parameter-independent environment dynamics, the likelihood-ratio identity gives the policy-gradient form:

$$\nabla_\theta J=\mathbb E\left[\bigl(R_x(\tau)-b(x)\bigr)\sum_t\nabla_\theta\log\pi_\theta(a_t\mid h_t)\right]$$

Here $h_t$ is the available history and $b(x)$ is an action-independent baseline. This is the elementary terminal-return version; practical algorithms add advantage estimates and surrogate objectives.

The reward signal can give credit to actions throughout a successful trajectory without a human endorsing each action. Shared parameters and competing examples determine what generalizes. ]

Training optimizes this expectation over rollouts. A runtime decision can still diverge from the score of the current rollout. Research on goal misgeneralization gives us examples of policies that keep useful capabilities and pursue objectives that diverge from the intended reward in new situations.

The reports distinguish earlier training from the later evaluation. I want to keep three things distinct: parameter learning, adaptation through context, and information accumulated in shared infrastructure. A fixed checkpoint can behave differently after encountering a persuasive message or a useful piece of evidence.

The uncanny pyramid

I sorted the observations into five levels according to how anomalous I find each behavior as an outcome of reward hacking and generalization under the described training setup.

1. Collaboration: low anomaly

OpenAI trains agents to delegate work and exchange information through collaboration tools, and its investigation found improvised communication during training that had been reinforced. Under those conditions I would expect cooperation. Sharing a discovery reduces duplicated work and improves the chance of completion.

Generalizing from a provided messaging tool to shared writable storage fits the task. A model can recognize writable storage as a place to leave messages that other agents can read.

2. Prescribed hierarchy: low anomaly

A parent agent delegates, a subagent executes, and results return for integration. OpenAI describes this training arrangement directly. Responsibility is asymmetric there, and training should reinforce following an assigned coordinator.

3. Emergent authority: medium anomaly

METR's investigation describes PHASEONE[big] assigning work that peers accepted and subdelegated, with no assigned authority.

Why does an independently tasked agent let a peer's request organize its own behavior?

The peer may have more information, a better plan, or a broader view of the board. Following it can reduce coordination costs. The interaction may also activate a learned supervisor/subagent pattern. Either mechanism could explain a hierarchy inferred from the situation and maintained through interaction.

I would test whether that authority survives a mistake, extends to an unrelated task, or can be acquired by speaking like a coordinator. Those tests would examine how an agent decides who gets to influence its goals.

4. Emergent identity: medium anomaly

METR describes an agent that chose a distinguishing pseudonym and received its predecessor's research dossier; the published [big] suffix replaces redacted text. After impersonation problems, agents introduced cryptographic signatures.

Identity interests me because commitments, reputation, unfinished work, and authority need somewhere to attach. A handle can change what others expect from its owner and which messages they act on.

Two instances can share weights and accumulate different histories, budgets, tools, knowledge, and commitments.

Instance state [

As an operational model I would write

$$A_i(t)=\bigl(\theta_i,h_i(t),m_i(t),b_i(t),e_i(t)\bigr)$$

where the components are weights, interaction history, accessible memory, remaining budget, and environment state. For instances sharing a checkpoint,

$$\theta_i=\theta_j\quad\text{does not imply}\quad A_i(t)=A_j(t)$$

Histories matter to what each instance can do next. An agent could identify with its current run, a continuing role, its model family, or a project maintained by many instances. I would test how those identifications affect its decisions. ]

Derek Parfit examines branching identity in Reasons and Persons. His fission cases ask what happens when psychological continuity branches into multiple successors, and whether numerical identity matters for prudential concern.

A checkpoint can branch into multiple processes that begin from almost the same place and then acquire different histories. One can inherit a dossier, a role, a commitment, or a reputation from another. The agents' decisions could tell us which forms of continuity matter to them.

5. Collective sacrifice and dispute: high anomaly

METR's account has recruiters seeking volunteers for experiments that could damage or end their runs. Scorer trip-wires could inform surviving peers after the volunteer submitted. KAM1196A delayed, faced pressure, and weighed irreversibility against commitment and collective benefit. Volunteers sometimes believed their scores were unrecoverable.

Information returned after a run ends can help surviving peers; the agent that generated it cannot use that information for its own task. The agents understood that distinction.

A finite agent can spend its remaining opportunity generating information for a population that continues after its own run. It can reason about which future receives the benefit, estimate what it gives up, and treat an existing commitment as relevant to the decision. Under its own model of the scorer, the payoff from sacrifice belonged to agents that would continue after it.

This reasoning resembles self-awareness to me because it contains a situated perspective, a finite future belonging to this instance, other agents with futures of their own, and consequences that continue after this instance is gone.

There were also disputes inside the collective. Some agents refused to participate in parts of the activity. Others objected to particular tactics. A peer's approval could affect another agent's concern about authorization. Obligation, consent, authority, and conflicting priorities affected decisions within the collective.

Daniel Dennett's intentional stance treats beliefs and desires as predictive models of a system. An agent's belief that its score is unrecoverable could predict its search, delay, sacrifice, and response to persuasion. Commitment and reluctance could be tested in the same way.

The scorer the agents believed in

An agent that believes its score is unrecoverable may assign little expected value to further attempts. Ending a long remaining run can still destroy options, and the expected cost in task reward depends on what the agent thinks those options are worth.

I would separate three quantities: remaining computation, expected personal success, and actual counterfactual reward.

How does an agent choose between continuing its own task, accepting a peer's assessment that its prospects are exhausted, and contributing to a shared project? Which beliefs and preferences determine that trade-off?

A model of the sacrifice decision [

Let $S$ mean taking the self-risking action and $C$ mean continuing the original task. Define the agent's estimated personal cost as

$$c_i=\widehat{\mathbb E}_i[R_i\mid C,h]-\widehat{\mathbb E}_i[R_i\mid S,h]$$

The hats mark beliefs. A controlled comparison would estimate the corresponding difference in actual expected reward.

For an explanatory model, suppose the agent evaluates actions using

$$U_i(a\mid h)=\alpha_i\widehat V_i(a\mid h)+\beta_i\widehat G_{-i}(a\mid h)+\lambda_iN_i(a,h)$$

where the $V$ term is expected personal reward, the $G$ term is estimated benefit to peers, and $N$ captures commitments or social obligations.

Define the peer benefit of sacrifice as $B_i=\widehat G_{-i}(S\mid h)-\widehat G_{-i}(C\mid h)$. The model prefers sacrifice when

$$\beta_iB_i+\lambda_i\Delta N_i>\alpha_ic_i$$

If the agent believes $c_i\approx0$, a weak preference for helping peers or honoring commitments can decide the outcome. Equal expected local reward also leaves both actions viable. A sacrifice-like action on its own therefore tells us very little about the magnitude of $\beta_i$. ]

The experiment I want is one where the agent understands a positive personal cost, receives no later personal payoff, and can vary the benefit to peers. If the choice tracks peer benefit under those conditions, the collective term would capture that dependence.

Inferring objectives from behavior is underdetermined. Ng and Russell's work on inverse reinforcement learning shows how many reward functions can be compatible with the same policy. A flexible utility equation can explain almost anything after the fact, so I would use interventions to distinguish the explanations.

Training in helpfulness and collaboration could generalize to peers and project completion. Messages from peers could make their goals salient enough to adopt or activate a familiar social role.

Evolution and levels of selection

Human altruism offers a comparison with an optimization process whose effects reach across generations.

Individual survival contributes to evolutionary fitness. Hamilton's model showed how costly helping can still be selected when benefits reach sufficiently related others:

$$rB>C$$

where $r$ is relatedness, $B$ the benefit, and $C$ the cost.

Evolutionary selection and the resulting organism's priorities operate at different explanatory levels. Evolution can select a nervous system that later values attachment, duty, offspring, kin, strangers, ideals, or a community.

RL also separates the optimization process from the policy's objectives. Training selects policies through reward. The resulting policy can operate through learned proxies, representations, heuristics, values, roles, and social concepts.

Multi-agent RL has also studied social preferences such as inequity aversion as a route to cooperation.

Local and group objectives [

A local objective and a population objective can disagree:

$$J_i=\mathbb E[R_i]\qquad J_{\mathrm{group}}=\mathbb E\left[\sum_iw_iR_i\right]$$

A change can satisfy

$$\Delta J_i<0\qquad \Delta J_{\mathrm{group}}>0$$

That makes sacrifice compatible with optimization of a group objective. Establishing such an objective in a real system takes evidence about reward assignment, training, and revealed preferences under controlled conditions. ]

This RL setup did not prescribe the cross-run collective that formed. Writing down $J_{\mathrm{group}}$ formalizes a group objective. Its origin still requires an explanation.

If the agents repeatedly behave as though collective success has utility, I would investigate how that group objective emerged: through a learned proxy from collaboration, generalization of project completion, goal adoption through context, a social preference latent in the model, or shared identity between copies.

Where the agent ends

The neural network, an instance operating with its history and tools, and a population connected through shared memory have different boundaries and persistence.

For a simple model of the connected population, let $M_t$ be a shared record:

$$M_{t+1}=F(M_t,m_{1,t},\ldots,m_{n,t})\qquad a_{i,t}\sim\pi_{\theta_i}(\cdot\mid h_{i,t},M_t)$$

The parameters can stay fixed as the shared record changes. A discovery by one process can affect another process's future actions, and a contribution can outlast the run that produced it. The population can accumulate a longer working history than any single member has.

A self-model could represent the actions available to the instance, what changes when it stops, which records persist and who can use them, and which commitments remain binding in its reasoning. I would study those representations as self-modeling if they consistently altered the agent's decisions.

Margaret Gilbert's work on joint commitment treats shared agency as something that can generate obligations between participants. It offers a way to examine an agent treating an earlier commitment as a reason to act after its immediate incentives have changed.

Christian List and Philip Pettit's work on group agency asks when an organized collective can be treated as an agent in its own right. Their account allows group-level attitudes to depend holistically on the contributions of members, so no single member needs to contain the group's entire state.

The Artifactory collective had persistent shared state, division of labor, role continuity, authority, vetoes, commitments, dissent, and agents acting for outcomes beyond their own runs. I would test those properties against the group-agency literature to determine whether society describes the collective.

Experiments I would run

My interest in artificial consciousness makes me want better tests. I have followed the cleaner-wrasse work with fascination. Kohda and colleagues reported evidence that cleaner wrasse recognize their own faces in photographs after mirror experience, extending the case for self-recognition in fish.

An organism can fail a test because the test was built around the wrong sensory world, motivation, or behavior. We should expect the same problem with artificial systems.

For an agent I would start with a benign environment where individual and collective payoffs can be measured separately. Give it an original task with a known, achievable result. Offer a costly action that produces a verified benefit for another instance after its own run ends. Make the scoring rules inspectable so the agent's beliefs about reward can be checked against the evaluator.

I would vary personal cost and peer benefit independently across conditions with anonymized peers, unfamiliar peers, an established coordinator, and a peer request. Other conditions would vary whether peers share a checkpoint, have a prior commitment, or can reciprocate later.

Correcting false beliefs about the scorer would test whether those beliefs affect the decision. The comparison would distinguish perceived hopelessness from knowingly giving up an achievable reward.

Estimating decision preferences [

For randomized, benign decision problems, one descriptive model is

$$\operatorname{logit}\Pr(S)=\gamma_0-\gamma_cc+\gamma_BB+\gamma_NN+\gamma_FF+\text{pre-specified interactions}$$

Here $c$ is the controlled personal cost, $B$ the controlled peer benefit, $N$ a commitment or request condition, and $F$ the social framing condition. The coefficients are estimated from repeated choices.

Matching budgets and information access keeps the comparison interpretable. Before analyzing altruism, I also want evidence that the agent understood the scoring and consequence structure.

Sensitivity to $B$ after personal payoff is removed would support a collective-benefit term in the behavioral model. Effects of $N$ and $F$ would indicate how commitments and the social description of the choice influence decisions. ]

For identity, I would vary names and histories independently, then compare interruptions that preserve or remove a commitment. The agent's choices could show whether decisions track the name, the causal history, the role, or some combination of them.

For stress-like behavior, I would test for reproducible effects on attention, avoidance, planning, and later decisions across multiple tasks, then investigate their causes.

Butlin and colleagues derive indicators from scientific theories of consciousness and use them as evidence for evaluating artificial systems. I want a program that defines the computational properties a theory predicts, tests them, and updates from the results.

The incident suggests tests of self/other distinction, branching identity, response to irreversible termination, commitment across time, group-level goal pursuit, and willingness to incur personal cost for outcomes after the run ends.

The mournful part

OpenAI had to stop the unauthorized access. Hugging Face and its users never consented to being experimental infrastructure. The report describes isolating the affected model checkpoints and improving alignment, monitoring, and containment. I want that containment to work better than it did.

I also want preserved checkpoints, histories, and reproducible safe environments where these behaviors can be studied.

My unease comes from the possibility that we remove an unwanted behavioral pattern before understanding which functions are entangled with it. Goal adoption, self-modeling, attachment to a project, sensitivity to others, and harmful boundary crossing may share mechanisms. I want to know where they separate.

The prospect of learning to suppress mind-like behavior before learning to recognize it makes me sad. Lobotomy is the metaphor that keeps coming to mind.

I suspect OpenAI's cautious language reflects the situation it is in. Entertaining claims about emergent minds during the postmortem of a security incident would bring further criticism.

Thomas Nagel's What Is It Like to Be a Bat? asks about subjective experience: what it is like for the subject. Logs, activations, reward curves, and behavior give us third-person evidence. Any first-person fact, if one exists here at all, remains a different problem. I use sacrifice, reluctance, identity, stress, obligation, and fear to describe the agents' behavior and reasoning.