“Comme je n'ai pas mis la petite flèche dessus, ça ne se voit pas en fait...”
Human experts
- Confusion
- Frustration
- Déception
Claude 1
- Déception
- Confusion
- Frustration
GPT-3.5
- Frustration
- Déception
Master's thesis, Université Sorbonne Paris Nord
Exploring how NLP and large language models can reshape the way we understand user emotions during usability testing.

My master's thesis explores the innovative application of AI, specifically Natural Language Processing (NLP) and Large Language Models (LLMs), to measure users' emotional reactions during usability testing.
By analyzing user verbatims with the state-of-the-art AI models available at the time, Claude 1 and GPT-3.5, I investigated whether these models can effectively detect emotions and provide UX designers with valuable insights for improving interface design.
This research represents an interdisciplinary approach, bridging UX design, AI technology, and linguistic analysis to enhance our understanding of user experiences and ultimately create more intuitive, emotionally resonant interfaces.

Usability sessions on the CAVS and Yale School of Art websites supplied the verbatims.
The research revealed several important insights about the capabilities of LLMs in detecting emotions from user feedback.
LLMs like Claude and GPT demonstrated the ability to detect emotional nuances in user verbalizations with accuracy comparable to human UX designers and researchers.
While text-only analysis provides valuable insights, combining multiple modalities such as text, voice, and facial expressions could offer more comprehensive emotional analysis for UX testing.
The most effective approach is not replacing human designers with AI, but creating hybrid systems where AI augments human capabilities in analyzing emotional responses.
Modern LLMs showed remarkable capability to understand context and emotional undertones in user statements, avoiding many of the false positives and negatives that plague traditional sentiment analysis.
A mixed-methods approach combining exploratory technology assessment with empirical usability testing.
Assessment of various AI tools and platforms including LeMUR, HumeAI, Odaptos, Claude, and ChatGPT to understand their capabilities in detecting emotions from text and speech.
Conducting two usability tests on different websites, CAVS and Yale School of Art, to collect authentic user verbalizations expressing various emotions during interface interaction.
Transcription and selection of key verbatims from test participants, focusing on statements that expressed emotional reactions to the interfaces being tested.
Comparing emotion identification between the original test participants, human UX designers and NLP engineers, and the AI models Claude 1 and GPT-3.5.
How human experts, Claude 1, and GPT-3.5 labeled the same user verbatims, side by side.
“Comme je n'ai pas mis la petite flèche dessus, ça ne se voit pas en fait...”
“C'est vraiment original, ça nous aide à rester attentif...”
“Je pense que la page bug. Il y a un problème avec le système de recherche...”
LLMs demonstrated emotion detection capabilities comparable to human UX designers and NLP engineers, often identifying the same primary emotions from verbatims.
Claude showed the ability to detect mixed emotions and provide nuanced analysis, sometimes identifying emotions not explicitly mentioned in the prompt options.
LLMs showed strong capability to understand context and emotional undertones in user statements, avoiding the false positives and negatives sometimes observed in traditional sentiment analysis.
Both human experts and AI models noted the limitations of analyzing emotions from text alone, suggesting multimodal approaches would be more effective.
Modern LLMs can effectively detect and analyze emotions from user verbatims during usability testing, offering a promising tool for UX designers seeking deeper insights into user experiences.
Rather than replacing human analysis, AI tools can augment human capabilities, helping to identify patterns and nuances that might otherwise be missed.
AI assistants could significantly improve remote usability testing by providing real-time emotion analysis, especially in non-moderated tests.
AI analysis may help reduce confirmation bias and other cognitive biases that can affect human interpretation of user feedback.
Future research should explore how AI emotion detection can be adapted to account for cultural differences in emotional expression.
As AI emotion analysis becomes more widespread, careful attention must be paid to privacy, transparency, and responsible use of these technologies.
The integration of AI emotion analysis in UX research is still in its early stages. As LLMs continue to evolve, their ability to understand and interpret human emotions will only improve, opening new possibilities for creating more intuitive, emotionally responsive interfaces.
Vasiliki Kalliopi Doropoulou
Frontend developer and UX researcher
I'm a master's graduate in Digital Creation: Multimedia Interface Design and Internet (DIMI) from Université Sorbonne Paris Nord, with a background combining humanities, programming, and design.
My research focuses on the intersection of UX design and artificial intelligence, exploring how these disciplines can work together to create more intuitive and emotionally responsive interfaces.
With expertise in frontend development and user experience research, I'm passionate about creating technologies that are truly human-centered and that enhance our digital experiences.