AI is providing emotional support for employees – but is it a valuable tool or privacy threat?
As artificial intelligence (AI) tools like ChatGPT gain traction for personal therapy and emotional support, a new trend is emerging in the workplace: employers are increasingly deploying AI to monitor and support employee emotions. This shift, which has accelerated since the pandemic-induced rise of remote work, raises critical questions about the implications of using AI for emotional well-being in professional settings. Industries such as healthcare, human resources, and customer service have adopted AI systems designed to assess the emotional states of employees, identify those in distress, and provide support. However, this integration of AI into the emotional landscape of work brings with it concerns about privacy, confidentiality, and the potential for misuse of sensitive emotional data.
Preliminary studies suggest that AI can sometimes provide emotional support that feels more attentive than human interactions. For instance, research indicates that AI-generated responses may make individuals feel more heard, with some findings suggesting that AI can be as empathetic as human therapists. This can be particularly appealing for employees dealing with stigmatized issues, as the non-judgmental nature of AI offers a sense of safety. However, the use of AI in this capacity also poses significant risks. Workers have expressed apprehension about participating in employer-initiated mental health programs due to fears of confidentiality breaches and the potential impact on their careers. Moreover, advanced AI systems that analyze communication patterns and emotional states risk transforming emotional support into a form of surveillance, creating an environment where employees feel constantly monitored.
The implications of AI in the workplace extend beyond mere emotional support; they touch on ethical and managerial dilemmas. While AI tools can help organizations identify signs of burnout and morale issues, they can also lead to increased stress among employees who feel their privacy is compromised. Studies show that continuous monitoring can heighten anxiety and alter behavior, undermining the very safety needed for individuals to seek help. Furthermore, biases inherent in AI systems may disproportionately affect marginalized groups, raising concerns about the accuracy and fairness of emotional assessments. As companies navigate the integration of AI into emotional support frameworks, they must strike a balance between leveraging AI’s capabilities and preserving the essential human connections that foster trust and authenticity in the workplace. Thoughtful implementation, clear ethical guidelines, and robust privacy protections are crucial to ensuring that AI enhances rather than diminishes the emotional well-being of employees.
Does AI that monitors and supports worker emotions improve or degrade the workplace?
Marta Sher/iStock via Getty Images
As artificial intelligence tools like ChatGPT become an increasingly popular avenue for people seeking personal therapy and emotional support, the dangers that this can present –
especially for young people
– have made plenty of headlines. What hasn’t received as much attention is employers using generative AI to assess workers’ psychological well-being and provide emotional support in the workplace.
Since the pandemic-induced global shift to remote work, industries ranging from
health care
to
human resources
and
customer service
have seen a spike in employers
using AI-powered systems
designed to analyze the emotional state of employees,
identify emotionally distressed individuals
, and provide them with emotional support.
This new frontier is a large step beyond using general chat tools or
individual therapy apps
for psychological support. As
researchers studying
how AI affects emotions
and relationships in the workplace, we are concerned with critical questions that this shift raises: What happens when your employer has access to your emotional data? Can AI really provide the kind of emotional support workers need? What happens if the AI malfunctions? And if something goes wrong, who’s responsible?
The workplace difference
Many companies have started by offering automated counseling programs that have many parallels with personal therapy apps, a practice that has shown some benefits. In preliminary studies, researchers found that in a doctor-patient-style virtual conversation setting, AI-generated responses actually
make people feel more heard
than human ones. A study comparing AI chatbots with human psychotherapists found the bots were “
at least as empathic as therapist responses, and sometimes more so
.”
This might seem surprising at first glance, but AI offers unwavering attention and consistently supportive responses. It doesn’t interrupt, doesn’t judge and doesn’t get frustrated when you repeat the same concerns. For some employees, especially those dealing with stigmatized issues like mental health or workplace conflicts, this consistency
feels safer than human interaction
.
But for others, it raises new concerns. A 2023 study found that
workers were reluctant to participate
in company-initiated mental health programs due to worries about confidentiality and stigma. Many feared that their disclosures could negatively affect their careers.
Other workplace AI systems go much deeper, analyzing employee communication as it happens – think emails, Slack conversations and Zoom calls. This analysis creates detailed records of employee emotional states, stress patterns and psychological vulnerabilities. All this data resides within corporate systems where privacy protections are typically unclear and often favor the interests of the employer.
Employees might feel that AI emotional support systems are more like workplace surveillance.
Malte Mueller/fStop via Getty Images
Workplace Options, a global employee assistance provider, has partnered with Wellbeing.ai to
deploy a platform that uses facial analytics
to track emotional states across 62 emotion categories. It generates well-being scores that organizations can use to detect stress or morale issues. This approach effectively embeds AI into emotionally sensitive aspects of work, leaving an uncomfortably thin boundary between support and surveillance.
In this scenario, the same AI that helps employees feel heard and supported also generates unprecedented insight into workforce emotional dynamics. Organizations can now track which departments show signs of burnout, identify employees at risk of quitting and monitor emotional responses to organizational changes.
But this type of tool also transforms emotional data into management intelligence, presenting many companies with a genuine dilemma. While progressive organizations are establishing strict data governance –
limiting access to anonymized patterns
rather than individual conversations – others struggle with
the temptation to use emotional insights
for performance evaluation and personnel decisions.
Continuous surveillance carried out by some of these systems may help ensure that companies do not neglect a group or individual in distress, but it can also lead people to monitor their own actions to avoid calling attention to themselves. Research on workplace AI monitoring has shown how
employees experience increased stress and modify their behavior
when they know that management can review their interactions. The monitoring undermines the feeling of safety necessary for people to comfortably seek help. Another study found that these systems
increased distress for employees
due to the loss of privacy and concerns that consequences would arise if the system identified them as being stressed or burned out.
When artificial empathy meets real consequences
These findings are important because the stakes are arguably even higher in workplace settings than personal ones. AI systems lack the nuanced judgment necessary to distinguish between accepting someone as a person versus endorsing harmful behaviors. In organizational contexts, this means an AI might inadvertently validate unethical workplace practices or fail to recognize when human intervention is critical.
And that’s not the only way AI systems can get things wrong. A study found that emotion-tracking AI tools had a
disproportionate impact
on employees of color, trans and gender nonbinary people, and people living with mental illness. Interviewees
expressed deep concern
about how these tools might misread an employee’s mood, tone or verbal queues due to ethnic, gender and other kinds of
bias that AI systems carry
.
A study looked at how employees perceive AI emotion detection in the workplace.
There’s also an authenticity problem. Research shows that
when people know they’re talking to an AI system
, they rate identical empathetic responses as less authentic than when they attribute them to humans. Yet some employees prefer AI precisely because they know it’s not human. The feeling that these tools protect your anonymity and freedom from social consequences is appealing for some – even if it may only be a feeling.
The technology also raises questions about what happens to human managers. If employees consistently prefer AI for emotional support, what does that reveal about organizational leadership? Some companies are
using AI insights to train managers
in emotional intelligence, turning the technology into a mirror that reflects where human skills fall short.
The path forward
The conversation about workplace AI emotional support isn’t just about technology – it’s about what kinds of companies people want to work for. As these systems become more prevalent, we believe it’s important to grapple with fundamental questions: Should employers prioritize authentic human connection over consistent availability? How can individual privacy be balanced with organizational insights? Can organizations harness AI’s empathetic capabilities while preserving the trust necessary for meaningful workplace relationships?
The most thoughtful implementations recognize that AI shouldn’t replace human empathy, but rather create conditions where it can flourish. When AI handles routine emotional labor – the 3 a.m. anxiety attacks, pre-meeting stress checks, processing difficult feedback – managers gain bandwidth for deeper, more authentic connections with their teams.
But this requires careful implementation. Companies that establish clear ethical boundaries, strong privacy protections and explicit policies about how emotional data gets used are more likely to avoid the pitfalls of these systems – as will those that recognize when human judgment and authentic presence remain irreplaceable.
The authors do not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.
