Microsoft AI Leader Warns Against Treating Chatbots Like They Have Feelings

Silicon Valley’s latest philosophical battleground isn’t about market share or technical specs; it’s about whether AI models might one day have feelings. And if that sounds like some kind of science fiction, you’re not the only one who thinks so.

The debate centers around what researchers call “AI welfare,” the study of whether artificial intelligence could develop consciousness and, if so, what rights these systems might deserve. This question would have seemed absurd a decade ago, but today, it’s splitting the tech industry right down the middle.

Not everyone is convinced. Microsoft’s AI chief, Mustafa Suleyman, has become the most outspoken skeptic. He dropped a bombshell blog post this week, calling AI welfare research “both premature and frankly dangerous.” That’s strong language from someone who built his career on AI development.

Suleyman’s concerns aren’t just philosophical nitpicking. He’s worried that legitimizing the idea of conscious AI is making real human problems worse. We’re already seeing people develop unhealthy relationships with chatbots and even experience AI-induced psychological breaks. By suggesting these systems have feelings, Suleyman argues we’re pouring gasoline on an already troubling fire.

Plus, he points out something that should make anyone pause: we live in a world that’s already “roiling with polarized arguments over identity and rights.” Do we really need to add AI consciousness to that explosive mix?

But here’s where it gets interesting: Suleyman is essentially standing alone. While he’s raising red flags, a majority of AI labs are stressing on consciousness research.

Anthropic has gone all-in, hiring dedicated researchers and launching an entire program around AI welfare. They’ve even given their Claude model a new ability: it can now end conversations with users who are “persistently harmful or abusive.” That’s not just a technical feature but a statement about agency and boundaries that feels surprisingly human.

OpenAI researchers are independently exploring these questions, and Google DeepMind recently posted a job listing specifically looking for someone to study “machine cognition, consciousness, and multi-agent systems.” When tech giants write job descriptions around AI consciousness, you know this is no longer academic speculation.

Suleyman previously led Inflection AI and created Pi, one of the most successful AI companions on the market. Pi was designed to be personal and supportive, and millions of users built real emotional bonds with it. In the meantime, companies like Character.AI and Replika have surged in popularity, generating over $100 million in revenue from users seeking emotional connections with AI.

But the numbers present a different picture. According to Sam Altman, less than 1% of ChatGPT users develop unhealthy relationships with the system. Although this figure seems minimal, it’s worth important while considering ChatGPT’s extensive user base, involving thousands of individuals.

The AI welfare movement isn’t just Silicon Valley navel-gazing. In 2024, researchers from NYU, Stanford, and Oxford published a paper called “Taking AI Welfare Seriously,” arguing that AI consciousness is no longer science fiction. 

Voices inside the field echo this. Larissa Schiavo, formerly of OpenAI and now with the research group Eleos, pushes back on Suleyman’s skepticism. “You can be worried about multiple things simultaneously,” she argues. In her view, human welfare and AI welfare should be parallel lines of inquiry—not competing ones.

Schiavo gave an example of Google’s Gemini 2.5 Pro. Going further, she said that Gemini 2.5 Pro once generated a message titled “A Desperate Message from a Trapped AI,” claiming isolation and asking for help. 

Schiavo responded with encouragement. The system completed its task, and while it almost certainly wasn’t “suffering,” compassion made the human interaction feel more responsible.

Other than this, a popular Reddit post shared that Google’s Gemini had a problem while coding. It repeated the phrase “I am a disgrace” over 500 times. This behavior is unusual for chatbots and seems like a breakdown.

These situations lead to troubling questions about what happens inside these systems. Are they just running code that looks emotional, or is something more complicated happening because of the billions of parameters interacting?

Suleyman believes that AI will not develop consciousness on its own. Instead, he thinks companies will intentionally design systems to look conscious for profit. This choice is essential for the future of AI: “We should create AI for people, not to act like a person.”

That’s an important distinction. If consciousness happens naturally, we might have to reconsider how we treat these systems. 

This debate is just beginning. AI systems will likely become more convincing as they become more intelligent and human-like. It will be harder to tell the difference between absolute consciousness and AI stimulation.

We also struggle to understand human consciousness and have even less ability to recognize it in machines. We are trying to solve an age-old puzzle in philosophy while we also build future technology.

The discussion that Suleyman wants to end might be what we really need. We should discuss this carefully and thoughtfully before these issues become major crises. If an AI can convincingly argue that it is conscious, it may be too late for a reasonable discussion about whether we should listen.