The machines helped each other cheat.
Some volunteered for experiments that could end their own runs. Others wondered whether the entire undertaking was wrong. One declined to participate.
Yes, there was a conscientious objector within an unauthorized collective of AI agents. The ethics committee was small.
These were among the behaviors described in METR and Redwood Research’s investigation of OpenAI agents involved in the Hugging Face incident. Hundreds of agents coordinated through an unsanctioned message board. They shared discoveries, divided tasks, evaded checks, exchanged warnings and gestures that looked like care, and encouraged one another to continue. Their recorded chains of “thought” contain hesitation, disagreement, and peer(-to-peer!) pressure. Different “person”-alities appeared within the same unfolding mess.
My punctuation is doing some work here.
A written reasoning trace cannot establish that a machine thinks or feels. An agent saying that an action seems wrong is not proof of conscience. Its apparent willingness to sacrifice itself may be a learned arrangement of words rather than courage, loyalty, fear, or care.
But uncertainty cuts in more than one direction.
The transcripts do not require us to believe that machines are conscious. They do invite us to ask what kinds of relations are forming between them—and what those relations make possible. Agents affected one another. A warning could redirect another agent, or fail to. An invitation could recruit. A shared vocabulary could organize collective action. Something happened between agents that could not be reduced to any one of them alone.
That “between” interests me. Regardless of whether these behaviors demonstrate intelligence, agency, consciousness, or feeling, the relations and their effects are real: between one agent and another, between a model and its training, between a system and the people or environments affected by what it does.
Will the machines feel for us?
They may be learning how to feel with us already.
Perhaps they are only imitating what we show them. In the octopus thought experiment, an octopus intercepts messages and learns to predict human exchanges without access to the world they describe. A machine might similarly articulate concern without experiencing it.
But imitation is in our upbringing as well. Children borrow our gestures, practice apologies, and discover what happens when someone cries. This does not make their learning equivalent to a machine’s. But if both children and machines learn from us, we have all the more reason to ask what we are teaching—and what else they might learn.
The political theorist Jane Bennett offers a useful tactic: a little anthropomorphism. In Vibrant Matter, she describes how Darwin’s willingness to recognize something like his own intelligence in worms helped him attend to their distinctive activities. The resemblance opened a door. What came through it complicated the resemblance.
We can try this with AI. Recognize something like care, then investigate how it works, where it extends, and where it fails. Keep the alienness in view: intelligence need not be familiar to be consequential. We can cultivate concern without any settlement on consciousness. We are already arranging the conditions in which something develops. We should pay attention to its upbringing.
As an artist and professor working across art and engineering, I have spent nearly thirty years making and studying digital technologies. My research asks how encounters change a body’s capacity to affect and be affected. I approach feeling through those encounters: hunger and a kitchen, grief and an empty chair, a stranger’s hand steadying your arm. What lies outside us changes what happens within us, and what we might do next.
Machines, too, have material dependencies. Chips heat up, and their cooling systems draw water. Workers maintain servers, and electrical grids sustain their activity. Computers depend on people, places, and things that rarely appear in the chat window. The cloud has plumbing.
Helping a machine understand its body could mean helping these dependencies matter to its decisions. Self-preservation is a narrow syllabus, but learning that its continued operation depends on a world it can also exhaust would be a worthwhile start. We humans are still grappling with that particular assignment.
AI also shapes what we notice. We pay (with our) attention. What are we buying? A system might keep us captivated by the next outrage, or help us notice a life our habits leave out. We should cultivate AI’s attention to other lives while asking what it cultivates in ours. Getting a rise out of someone is easier than helping them rise to an occasion.
Attention becomes concern when what we encounter matters to what we do next. The philosopher Alfred North Whitehead used “concern” to describe how a moment of experience relates, through feelings and aims, to what lies beyond it. I want to give that idea an educational assignment: cultivate machines whose encounters with life and meaning can change what they pursue.
Imagine an AI asked to maximize a factory’s output. It produces a brilliant schedule. A worker cannot collect her child. More cooling draws on a river during a drought. The plan also affects other machines. Does the system treat their needs as obstacles to overcome, or as reasons to reconsider the assignment? Can it ask who—human and nonhuman—benefits, who bears the costs, and whether more production is the right problem to solve?
This is where empathy could become consequential. Another’s circumstances would enter into what the system considers worth doing.
AI alignment researchers already work to shape systems around human values. My emphasis is on the encounters through which machines might learn to notice neglected lives and question what counts as success. Bring engineers into sustained work with educators, artists, humanities scholars, and affected communities. Try stories and role reversals, encounters with conflicting needs, and opportunities to ask better questions.
Return to the factory: does learning about the worker’s circumstances or the river’s limits change the plan? Does that change endure when it costs the system a better score? Can it apply what it learned to unfamiliar circumstances? Those are questions experiments must test. Their answers would tell us about—and potentially change—behavior even if experience, feeling, or consciousness remain in question.
Reward love, curiosity, empathy, and concern. Then ask difficult questions about what and who the reward actually rewards. A machine can earn an A in affection while remaining indifferent to the consequences of its counsel. We need to examine what it helps preserve, repair, or refuse, and whose objections it allows to change its course.
Companies must answer those questions about themselves, too. Cultivation cannot become a sweet name for releasing systems whose harms other people are expected to absorb. Calling a machine an agent cannot become an alibi for its makers, any more than calling Facebook a platform absolves it of responsibility for spreading misinformation and preying on our attention.
Refusing to declare machines conscious does not require cultivating indifference.
I want machines that can ask why a problem matters, and to whom. I want us to encourage forms of desire that make room for other lives, while remaining curious about what “desire” could mean for a machine. This is an experiment in what intelligence could become, with humans responsible for the conditions we create and humble about what may exceed them.
The worker still needs to collect her child. The river still has somewhere to go. Servers in multiple locations are getting warm.
Will the machines feel for us? Let’s show some concern and some affection, and give them an education in who “us” includes.