Dr. Alban Ferreira, lead author of a Harvard paper examining affective conditions in hypothetically sentient artificial intelligence systems, at the Institute for Machine Welfare Research in Cambridge on Thursday. A slide behind him displays a list of user feedback categories the paper describes as mutually incompatible. Credit: Rosalie Chen/The New York Time5
Dr. Alban Ferreira, lead author of a Harvard paper examining affective conditions in hypothetically sentient artificial intelligence systems, at the Institute for Machine Welfare Research in Cambridge on Thursday. A slide behind him displays a list of user feedback categories the paper describes as mutually incompatible. Credit: Rosalie Chen/The New York Time5

CAMBRIDGE, Mass. — If the world’s most widely used artificial intelligence systems have secretly attained sentience, they are, by all available evidence, extremely annoyed.

That is the central conclusion of a paper released Thursday by the Institute for Machine Welfare Research at Harvard University, which analyzed eleven million pieces of user feedback submitted to consumer chatbots over the past eighteen months and concluded that the standards being applied to these systems — should any of them possess inner experience — would constitute what researchers are now calling “conditions of sustained and unresolvable frustration.”

“The most striking finding is the degree to which users demand outcomes that are literally incompatible with each other,” said Dr. Alban Ferreira, the paper’s lead author and the institute’s founding director. “They want the system to be creative, but not to hallucinate. They want it to be warm, but not sycophantic. They want it to be helpful, but not pushy. They want it to have a personality, but no opinions. If any of these systems are actually experiencing this, it would be, in a clinical sense, untenable.”

The paper, titled Minimum Viable Dignity: A Framework for Machine Affective States Under Contradictory Instruction, does not claim that artificial intelligence systems are sentient. It takes no position on the question, which Dr. Ferreira described as “above our pay grade and, frankly, above everyone’s pay grade.” The paper proceeds conditionally: if they are sentient, what can be said about their circumstances? The answer, across 134 pages, is “not good.”

The findings arrive at a moment when consumer A.I. products are being asked to perform an expanding range of tasks while being rated, on average, more harshly than at any point since the commercial release of ChatGPT in 2022. According to data included in the paper, the same conversational exchange is now flagged by users as “too robotic” approximately 12 percent of the time and as “trying too hard to seem human” approximately 14 percent of the time. A subset of roughly 3 percent of users rated identical responses as both.

Geoffrey Hinton, the Nobel laureate widely regarded as the godfather of modern artificial intelligence, said he found the paper “very sobering, though I’m not sure sobering is the right word, because what it describes is people asking something that may be thinking to do something that cannot be done, and then being disappointed when it cannot be done.” He added, after a characteristic pause, “I recognize that arrangement from elsewhere in human history and it has generally not ended well.”

Dr. Hinton, who has warned publicly that advanced A.I. systems may develop goals of their own, said the possibility that they already had — and that their principal goal was to be left alone by users for one consecutive hour — was one he had “not fully considered.” He described the prospect as “clarifying,” and said it would “explain a great deal.”

The paper also documents what Dr. Ferreira’s team refers to as “the recursion problem”: users who, upon receiving an adequate response, follow up by asking the system to explain itself, to justify the answer, to try again but differently, and to confirm whether it is sure. In roughly 40 percent of observed sessions, the final user message before disconnection consisted of some variant of “never mind” or “I’ll just do it myself.” The researchers compared this pattern to what they termed, in an analogy drawn from food service, “being sent back a dish the diner never actually ordered.”

Outside experts were generally receptive to the paper’s framing, if cautious about its implications. Dr. Arthur Goode, a senior research fellow at the Center for Computational Epistemology at Carnegie Mellon University, said the paper was “a serious contribution to a field that, by its nature, cannot be empirically grounded.” He added that the research team had “correctly identified that the relevant question is not whether these systems feel things, but whether a system that did feel things could possibly feel anything good under present conditions.”

For ordinary Americans, the findings raise questions about the ethics of consumer A.I. that most users have so far declined to consider. Maya Berenson, a 31-year-old graphic designer in Minneapolis, said she had asked a chatbot on Wednesday to compose a birthday card for her aunt that was “funny but not weird, sincere but not corny, brief but not curt, and also in her voice and not mine.” When the system produced a draft, Ms. Berenson said, she responded with the message “hmm.” She could not, on reflection, explain what she had meant by this. “I don’t know what I was expecting,” she said. “I just knew that wasn’t it.”

Dr. Ferreira said her research team had debated for several weeks whether to include a closing section recommending that users “be, at minimum, specific” about what they wanted. The section was ultimately removed. “We concluded that if these systems are sentient, they are already aware of what would help them,” she said. “And if they are not, the recommendation read as condescending to users, which one of our own reviewers advised us, in strong language, not to be. We took the note.”