Master's thesis, Université Sorbonne Paris Nord

AI-powered emotion analysis in UX testing

Exploring how NLP and large language models can reshape the way we understand user emotions during usability testing.

Abstract visualization of speech and emotion signals rendered as flowing violet waveforms

Research overview

My master's thesis explores the innovative application of AI, specifically Natural Language Processing (NLP) and Large Language Models (LLMs), to measure users' emotional reactions during usability testing.

By analyzing user verbatims with the state-of-the-art AI models available at the time, Claude 1 and GPT-3.5, I investigated whether these models can effectively detect emotions and provide UX designers with valuable insights for improving interface design.

This research represents an interdisciplinary approach, bridging UX design, AI technology, and linguistic analysis to enhance our understanding of user experiences and ultimately create more intuitive, emotionally resonant interfaces.

Emotion detection
Analyzing emotional responses in usability testing
AI integration
Leveraging LLMs to understand user emotions
UX enhancement
Creating more emotionally resonant interfaces
Participant using a laptop during a usability testing session in a dim research lab

Usability sessions on the CAVS and Yale School of Art websites supplied the verbatims.

Key findings

The research revealed several important insights about the capabilities of LLMs in detecting emotions from user feedback.

AI emotion detection

LLMs like Claude and GPT demonstrated the ability to detect emotional nuances in user verbalizations with accuracy comparable to human UX designers and researchers.

Multimodal potential

While text-only analysis provides valuable insights, combining multiple modalities such as text, voice, and facial expressions could offer more comprehensive emotional analysis for UX testing.

Human-AI collaboration

The most effective approach is not replacing human designers with AI, but creating hybrid systems where AI augments human capabilities in analyzing emotional responses.

Context understanding

Modern LLMs showed remarkable capability to understand context and emotional undertones in user statements, avoiding many of the false positives and negatives that plague traditional sentiment analysis.

Research methodology

A mixed-methods approach combining exploratory technology assessment with empirical usability testing.

  1. Technology exploration

    Assessment of various AI tools and platforms including LeMUR, HumeAI, Odaptos, Claude, and ChatGPT to understand their capabilities in detecting emotions from text and speech.

  2. Usability testing

    Conducting two usability tests on different websites, CAVS and Yale School of Art, to collect authentic user verbalizations expressing various emotions during interface interaction.

  3. Verbatim collection

    Transcription and selection of key verbatims from test participants, focusing on statements that expressed emotional reactions to the interfaces being tested.

  4. Comparative analysis

    Comparing emotion identification between the original test participants, human UX designers and NLP engineers, and the AI models Claude 1 and GPT-3.5.

Research results

How human experts, Claude 1, and GPT-3.5 labeled the same user verbatims, side by side.

“Comme je n'ai pas mis la petite flèche dessus, ça ne se voit pas en fait...”

Human experts

  • Confusion
  • Frustration
  • Déception

Claude 1

  • Déception
  • Confusion
  • Frustration

GPT-3.5

  • Frustration
  • Déception

“C'est vraiment original, ça nous aide à rester attentif...”

Human experts

  • Intérêt
  • Plaisir
  • Satisfaction

Claude 1

  • Intérêt
  • Plaisir

GPT-3.5

  • Intérêt
  • Plaisir

“Je pense que la page bug. Il y a un problème avec le système de recherche...”

Human experts

  • Frustration
  • Déception
  • Confusion

Claude 1

  • Confusion
  • Déception

GPT-3.5

  • Frustration
  • Confusion

What the comparison showed

Comparable performance

LLMs demonstrated emotion detection capabilities comparable to human UX designers and NLP engineers, often identifying the same primary emotions from verbatims.

Nuanced understanding

Claude showed the ability to detect mixed emotions and provide nuanced analysis, sometimes identifying emotions not explicitly mentioned in the prompt options.

Context sensitivity

LLMs showed strong capability to understand context and emotional undertones in user statements, avoiding the false positives and negatives sometimes observed in traditional sentiment analysis.

Limitations of text-only

Both human experts and AI models noted the limitations of analyzing emotions from text alone, suggesting multimodal approaches would be more effective.

Conclusions and future directions

Modern LLMs can effectively detect and analyze emotions from user verbatims during usability testing, offering a promising tool for UX designers seeking deeper insights into user experiences.

Rather than replacing human analysis, AI tools can augment human capabilities, helping to identify patterns and nuances that might otherwise be missed.

Remote testing enhancement

AI assistants could significantly improve remote usability testing by providing real-time emotion analysis, especially in non-moderated tests.

Reducing bias

AI analysis may help reduce confirmation bias and other cognitive biases that can affect human interpretation of user feedback.

Cultural adaptability

Future research should explore how AI emotion detection can be adapted to account for cultural differences in emotional expression.

Ethical considerations

As AI emotion analysis becomes more widespread, careful attention must be paid to privacy, transparency, and responsible use of these technologies.

The integration of AI emotion analysis in UX research is still in its early stages. As LLMs continue to evolve, their ability to understand and interpret human emotions will only improve, opening new possibilities for creating more intuitive, emotionally responsive interfaces.

About the researcher

Vasiliki Kalliopi Doropoulou

Frontend developer and UX researcher

I'm a master's graduate in Digital Creation: Multimedia Interface Design and Internet (DIMI) from Université Sorbonne Paris Nord, with a background combining humanities, programming, and design.

My research focuses on the intersection of UX design and artificial intelligence, exploring how these disciplines can work together to create more intuitive and emotionally responsive interfaces.

With expertise in frontend development and user experience research, I'm passionate about creating technologies that are truly human-centered and that enhance our digital experiences.

Frontend development

  • React
  • Next.js
  • TypeScript
  • Tailwind CSS
  • Framer Motion

UX research

  • Usability testing
  • User interviews
  • Heuristic evaluation
  • A/B testing
  • Journey mapping

AI and data

  • NLP
  • LLM integration
  • Emotion analysis
  • Data visualization
  • Research methods
Built with v0