Exploring Demographic Language Variations to ... - Semantic Scholar

Exploring Demographic Language Variations to Improve Multilingual Sentiment Analysis in Social Media Svitlana Volkova1 , Theresa Wilson2 and David Yarowsky1,2 , 1 Center

for Language and Speech Processing, Johns Hopkins University 2 Human-Language technology Center of Excellence

Motivation

S. Volkova, T. Wilson, D. Yarowsky (JHU)

Demographic Language Variations in Twitter

2 / 35

Motivation Demographic language variations (DLV) have been studied by socio-linguists for decades (Picard, 1997; Gefen & Ridings, 2005; Holmes & Meyerhoff, 2004; Macaulay, 2006; Tagliamonte, 2006).



2 / 35

Motivation Demographic language variations (DLV) have been studied by socio-linguists for decades (Picard, 1997; Gefen & Ridings, 2005; Holmes & Meyerhoff, 2004; Macaulay, 2006; Tagliamonte, 2006). DLV have been recently explored in personal email communication, blog posts, and public discussions (Boneva et al., 2001; Mohammad & Yang, 2011; Eisenstein et al., 2010; Bamman et al., 2012)



2 / 35

Motivation Demographic language variations (DLV) have been studied by socio-linguists for decades (Picard, 1997; Gefen & Ridings, 2005; Holmes & Meyerhoff, 2004; Macaulay, 2006; Tagliamonte, 2006). DLV have been recently explored in personal email communication, blog posts, and public discussions (Boneva et al., 2001; Mohammad & Yang, 2011; Eisenstein et al., 2010; Bamman et al., 2012) We propose to study differences in subjective language in social media to support commercial applications:



2 / 35

Motivation Demographic language variations (DLV) have been studied by socio-linguists for decades (Picard, 1997; Gefen & Ridings, 2005; Holmes & Meyerhoff, 2004; Macaulay, 2006; Tagliamonte, 2006). DLV have been recently explored in personal email communication, blog posts, and public discussions (Boneva et al., 2001; Mohammad & Yang, 2011; Eisenstein et al., 2010; Bamman et al., 2012) We propose to study differences in subjective language in social media to support commercial applications: personalized recommendation systems and targeted online advertising (Fan & Chang, 2009),



2 / 35

Motivation Demographic language variations (DLV) have been studied by socio-linguists for decades (Picard, 1997; Gefen & Ridings, 2005; Holmes & Meyerhoff, 2004; Macaulay, 2006; Tagliamonte, 2006). DLV have been recently explored in personal email communication, blog posts, and public discussions (Boneva et al., 2001; Mohammad & Yang, 2011; Eisenstein et al., 2010; Bamman et al., 2012) We propose to study differences in subjective language in social media to support commercial applications: personalized recommendation systems and targeted online advertising (Fan & Chang, 2009), detecting helpful product reviews (Ott et al., 2011),



2 / 35

Motivation Demographic language variations (DLV) have been studied by socio-linguists for decades (Picard, 1997; Gefen & Ridings, 2005; Holmes & Meyerhoff, 2004; Macaulay, 2006; Tagliamonte, 2006). DLV have been recently explored in personal email communication, blog posts, and public discussions (Boneva et al., 2001; Mohammad & Yang, 2011; Eisenstein et al., 2010; Bamman et al., 2012) We propose to study differences in subjective language in social media to support commercial applications: personalized recommendation systems and targeted online advertising (Fan & Chang, 2009), detecting helpful product reviews (Ott et al., 2011), tracking sentiment in real time (Resnik, 2013), S. Volkova, T. Wilson, D. Yarowsky (JHU)


2 / 35

Motivation Demographic language variations (DLV) have been studied by socio-linguists for decades (Picard, 1997; Gefen & Ridings, 2005; Holmes & Meyerhoff, 2004; Macaulay, 2006; Tagliamonte, 2006). DLV have been recently explored in personal email communication, blog posts, and public discussions (Boneva et al., 2001; Mohammad & Yang, 2011; Eisenstein et al., 2010; Bamman et al., 2012) We propose to study differences in subjective language in social media to support commercial applications: personalized recommendation systems and targeted online advertising (Fan & Chang, 2009), detecting helpful product reviews (Ott et al., 2011), tracking sentiment in real time (Resnik, 2013), large-scale, low-cost, passive polling (O’Connor et al., 2010). S. Volkova, T. Wilson, D. Yarowsky (JHU)


2 / 35

Motivation

Male ♂ and Female ♀ Twitter users use subjective terms differently: ♀+ “Chocolate is my weakness”



3 / 35

Motivation


♂− “Clearly they know our weakness. Argggg....”



3 / 35

Motivation





3 / 35

Motivation





4 / 35

Motivation





5 / 35

Motivation





6 / 35

Goal

I. Explore gender bias in the use of subjective language in Twitter:



7 / 35

Goal

I. Explore gender bias in the use of subjective language in Twitter: investigate multilingual subjective lexical variations;



7 / 35

Goal

I. Explore gender bias in the use of subjective language in Twitter: investigate multilingual subjective lexical variations; cross-cultural emoticon and hashtag usage.



7 / 35

Goal


II. Incorporate gender bias into models to improve sentiment analysis for English, Spanish, and Russian:



7 / 35

Goal


II. Incorporate gender bias into models to improve sentiment analysis for English, Spanish, and Russian: demonstrate that simple, binary features representing author gender are insufficient for gender-dependent sentiment analysis.



7 / 35

Data



8 / 35

Data



8 / 35

Data



8 / 35

Data



9 / 35

Data Automatic gender label prediction using user first name morphology (precision is above 0.98 across languages).



9 / 35

Data Automatic gender label prediction using user first name morphology (precision is above 0.98 across languages). Sentiment labels from Mechanical Turk (5 annotations per tweet):



9 / 35

Data Automatic gender label prediction using user first name morphology (precision is above 0.98 across languages). Sentiment labels from Mechanical Turk (5 annotations per tweet): Positive: Как же приятно просто лечь в постель после тяжелого дня... (It is a great pleasure to go to bed after a long day at work...)



9 / 35

Data Automatic gender label prediction using user first name morphology (precision is above 0.98 across languages). Sentiment labels from Mechanical Turk (5 annotations per tweet): Positive: Как же приятно просто лечь в постель после тяжелого дня... (It is a great pleasure to go to bed after a long day at work...) Negative: Уважаемый господин Прохоров купите эти выборы! (Dear Mr. Prokhorov just buy the elections!)



9 / 35

Data Automatic gender label prediction using user first name morphology (precision is above 0.98 across languages). Sentiment labels from Mechanical Turk (5 annotations per tweet): Positive: Как же приятно просто лечь в постель после тяжелого дня... (It is a great pleasure to go to bed after a long day at work...) Negative: Уважаемый господин Прохоров купите эти выборы! (Dear Mr. Prokhorov just buy the elections!) Both: Затолкали меня на местном рынке! но зато закупилась подарками для всей семьи :) (It was crowded at the local market! But I got presents for my family:-))



9 / 35

Data Automatic gender label prediction using user first name morphology (precision is above 0.98 across languages). Sentiment labels from Mechanical Turk (5 annotations per tweet): Positive: Как же приятно просто лечь в постель после тяжелого дня... (It is a great pleasure to go to bed after a long day at work...) Negative: Уважаемый господин Прохоров купите эти выборы! (Dear Mr. Prokhorov just buy the elections!) Both: Затолкали меня на местном рынке! но зато закупилась подарками для всей семьи :) (It was crowded at the local market! But I got presents for my family:-)) Neutral: Киев очень старый город (Kiev is a very old city).



9 / 35

Metrics Lexical Evaluation across Genders Term ti subjectivity: pti (subj|g) =


c(ti , P, g) + c(ti , N, g) , c(ti , g)


10 / 35


c(ti , P, g) + c(ti , N, g) , c(ti , g)

Term ti polarity: pti (+|g) =


c(ti , P, g) , c(ti , P, g) + c(ti , N, g)


10 / 35


c(ti , P, g) + c(ti , N, g) , c(ti , g)

Term ti polarity: pti (+|g) =

c(ti , P, g) , c(ti , P, g) + c(ti , N, g)

Polarity change across genders: ∆pt+i = |pti (+|F ) − pti (+|M)| s.t. subj tfti (F ) subj 1 − subj ≤ λ, tfti (M) 6= 0, tfti (M) λ controls term frequency similarity. S. Volkova, T. Wilson, D. Yarowsky (JHU)


10 / 35

Lexical Evaluation across Genders for English Terms: 3 - from LI , 4 - bootstrapped lexicon LB , and - hashtags



11 / 35




11 / 35




12 / 35




13 / 35

Lexical Evaluation for Spanish and Russian

Spanish: fiasco, triunfar (succeed) and #britneyspears used F + but M − ; horooriza (horrifies), #metallica and #latingrammy used F − but M + .



14 / 35

Lexical Evaluation for Spanish and Russian

Spanish: fiasco, triunfar (succeed) and #britneyspears used F + but M − ; horooriza (horrifies), #metallica and #latingrammy used F − but M + .

Russian: мечтайте (dream!), магический (magical) and совет (advice) used F + but M − ; исскушение (temptation), сложны (complicated), #iphones and #spartak (soccer team) used F − but M + .



14 / 35

How gender differences in subjective language can help subjectivity and polarity classification in social media?



15 / 35

Rule-based Subjectivity Classifiers

Gender-Independent (Riloff & Wiebe, 2003; Volkova et al., 2013): ~ · ~f ≥ 0.5, 1 if w RB GIndsubj = 0 otherwise.



16 / 35


Gender-Independent (Riloff & Wiebe, 2003; Volkova et al., 2013): ~ · ~f ≥ 0.5, 1 if w RB GIndsubj = 0 otherwise. Gender-Dependent: ( RB GDepsubj


=

1 0

if w~M · f~M ≥ 0.5 ∧ M, otherwise.


16 / 35


Gender-Independent (Riloff & Wiebe, 2003; Volkova et al., 2013): ~ · ~f ≥ 0.5, 1 if w RB GIndsubj = 0 otherwise. Gender-Dependent: ( RB GDepsubj


=

1 0

if w~F · f~F ≥ 0.5 ∧ F , otherwise.


17 / 35

Rule-based Subjectivity Classification Results Start with LI and incrementally add Emoticons, Adjectives, AdveRbs, Verbs, Nouns from LB .



18 / 35

Rule-based Subjectivity Classification Results Start with LI and incrementally add Emoticons, Adjectives, AdveRbs, Verbs, Nouns from LB .



19 / 35

Rule-based Polarity Classifiers

Gender-Independent (Riloff & Wiebe, 2003; Volkova et al., 2013): 1 if w~+ · f~+ ≥ w~− · f~− , RB GIndpol = 0 otherwise



20 / 35


Gender-Independent (Riloff & Wiebe, 2003; Volkova et al., 2013): 1 if w~+ · f~+ ≥ w~− · f~− , RB GIndpol = 0 otherwise Gender-Dependent: ( RB GDeppol =


1 0

~ ≥ w~M− · f M− ~ ∧ M, if w~M+ · f M+ otherwise


20 / 35


Gender-Independent (Riloff & Wiebe, 2003; Volkova et al., 2013): 1 if w~+ · f~+ ≥ w~− · f~− , RB GIndpol = 0 otherwise Gender-Dependent: ( RB = GDeppol


1 0

if w~F + · f F~+ ≥ w~F − · f F~− ∧ F , otherwise


21 / 35

Rule-based Polarity Classification Results Start with LI and incrementally add Emoticons, Adjectives, AdveRbs, Verbs, Nouns from LB .



22 / 35

Rule-based Polarity Classification Results Start with LI and incrementally add Emoticons, Adjectives, AdveRbs, Verbs, Nouns from LB .



23 / 35

Experimental Setup Gender-Independent features: V - unigram counts, LI , LB - set-count features from the original and bootstrapped lexicons, and E - emoticons ~f GInd = [LI , LB , E, V ]; subj ~f GInd = [L+ , L+ , E + , L− , L− , E − , V ]. pol I B I B



24 / 35

Experimental Setup Gender-Independent features: V - unigram counts, LI , LB - set-count features from the original and bootstrapped lexicons, and E - emoticons ~f GInd = [LI , LB , E, V ]; subj ~f GInd = [L+ , L+ , E + , L− , L− , E − , V ]. pol I B I B

Gender-Dependent joint features: ~f GDep−J = [LM , LM , E M , LF , LF , E F , V ]; I B I B subj ~f Dep−J = [LM+ , LM+ , E M+ , LF + , LF + , E F + pol I B I B M− F − F − LM− , LM− , LI , LB , E F − , V ]. I B ,E



24 / 35

Subjectivity Classification Results using SL



25 / 35




26 / 35




27 / 35

Polarity Classification Results using SL



28 / 35




29 / 35




30 / 35

Summary

Empirical study of differences in subjective language between male and female users in Twitter.



31 / 35

Summary

Empirical study of differences in subjective language between male and female users in Twitter. Analysis of hashtag and emoticon usage across cultures.



31 / 35

Summary

Empirical study of differences in subjective language between male and female users in Twitter. Analysis of hashtag and emoticon usage across cultures. Incorporating author gender as a model component can significantly improve subjectivity and polarity classification for multiple languages in social media.



31 / 35

Summary

Empirical study of differences in subjective language between male and female users in Twitter. Analysis of hashtag and emoticon usage across cultures. Incorporating author gender as a model component can significantly improve subjectivity and polarity classification for multiple languages in social media. Data: http://www.cs.jhu.edu/~svitlana/data/ data_emnlp2013.zip



31 / 35

References I Bamman, D., Eisenstein, J., & Schnoebelen, T. (2012). Gender in Twitter: styles, stances, and social networks. Computing Research Repository. Boneva, B., Kraut, R., & Frohlich, D. (2001). Using email for personal relationships: The difference gender makes. American Behavioral Scientist, 45(3), 530-549. Eisenstein, J., O’Connor, B., Smith, N. A., & Xing, E. P. (2010). A latent variable model for geographic lexical variation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’10) (p. 1277-1287). Fan, T. K., & Chang, C. H. (2009). Sentiment-oriented contextual advertising. Advances in Information Retrieval, 5478, 202-215.



32 / 35

References II Gefen, D., & Ridings, C. M. (2005). If you spoke as she does, sir, instead of the way you do: a sociolinguistics perspective of gender differences in virtual communities. SIGMIS Database, 36(2), 78-92. Holmes, J., & Meyerhoff, M. (2004). The handbook of language and gender. Blackwell Publishing. Macaulay, R. (2006). Pure grammaticalization: The development of a teenage intensifier. Language Variation and Change, 18(03), 267–283. Mohammad, S., & Yang, T. (2011). Tracking sentiment in mail: How genders differ on emotional axes. In Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA’11) (p. 70-79).



33 / 35

References III O’Connor, B., Eisenstein, J., Xing, E. P., & Smith, N. A. (2010). A mixture model of demographic lexical variation. In Proceedings of NIPS Workshop on Machine Learning in Computational Social Science (p. 1-7). Ott, M., Choi, Y., Cardie, C., & Hancock, J. T. (2011). Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (p. 309-319). Picard, R. W. (1997). Affective computing. MIT Press. Resnik, P. (2013). Getting real(-time) with live polling. (http://vimeo.com/68210812) Riloff, E., & Wiebe, J. (2003). Learning extraction patterns for subjective expressions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’03) (p. 105-112). S. Volkova, T. Wilson, D. Yarowsky (JHU)


34 / 35

References IV Tagliamonte, S. A. (2006). Analysing sociolinguistic variation. Cambridge University Press, 1st. Edition. Volkova, S., Wilson, T., & Yarowsky, D. (2013). Exploring sentiment in social media: Bootstrapping subjectivity clues from multilingual Twitter streams. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL’13) (pp. 505–510).



35 / 35