How to Pick Fine Grained Emotion Lexicon

Recently interested on the multi-class sentiment analysis, where to classify the text not only the polarity (Positive, Neutral, or Negative), but break into several class mood like happy, sadness etc. I come across few lexicon libraries, which are NRC, DepecheMood++, and Emo SenticMood. A simple quick assessment were executed on the dataset and lexicon libary to review the accuracy.

Photo by Leonardo Toshiro Okubo on Unsplash

Lexicon Library

NRC emotion lexicon is the English lexicon words that with 8 classes: anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. The idea of the mood is refer to Plutchik’s Wheel of Emotion.

DepecheMood++ is slightly different form NRC, where the mood classes are afraid, amused, angry, annoyed, dontcare, happy, inspired, and sad. The lexicon was built by extracting Rappler’s articles which were annotated according to the Mood Meter by Rappler’s readers

Emo SenticMood only has 6 classes which are anger, disgust, fear, joy, sadness, surprise

Benchmark Dataset

In order to determine the accuracy, following 3 datasets with label are used:

SamEval 2018 Task 1: Affect in Tweets (link)
SamEval 2019 Task 3: EmoContext: Contextual Emotion Detection in Text (link)
ISEAR - International Survey on Emotion Antecedents and Reactions (link)

Methods

All 3 datasets above download and simple data cleaning, NLP steps were executed on it. Then use the 3 different lexicon libraries to classify the benchmark dataset in order to compare the accuracy

Result

Following are the result for the 3 lexicon libraries on the 3 datasets. The yellow bar represent the expected mood from each benchmark dataset

SamEval 2018

By looking at the yellow bar in the figure, the dataset of SamEval 2018 only consists of the data with 4 moods (Anger, Fear, joy, Sadness) and the classes are not equal where the fear mood is slightly higher, but it should not impact the result.

From the NRC figure, we notice that the class labeling for NRC is quite correct. Although the accuracy is not very high (I think my data cleaning/NLP need to be enhanced more). but it is very obvious that somehow it is more accurate then other.

For the DepechaMood++ below. Although the mood label is not identical, for example the SamEval 2018 use fear but DepechaMood++ uses afraid. From the figure below, we can see for the

SamEval 2018 is anger; Inspired, Amused, anger are the top 3 mood classied by the DepecheMood++
SamEval 2018 is fear; Inspired, Amused, anger are the top 3 mood classied by the DepecheMood++
SamEval 2018 is joy; Inspired, Amused, dontcare are the top 3 mood classied by the DepecheMood++
SamEval 2018 is sadness; Inspired, Amused, annoyed are the top 3 mood classied by the DepecheMood++

From the result above, the DepecheMood++ does not seem to be very accurate to run on SamEval 2018 dataset

For the Emo Senticmood below. quite similar to DepecheMood++ where the mood joy are highest among all the mood.

SamEval 2019

Sam Eval 2019 is also not balance class where the other mood is quite high. However, the NRC still manage to classfy better then DepecheMood++ and Emo SenticMood.