Troll and divide: the language of online polarization — TEI-tagged view

Almog SimchonpersName, William J BradypersName, Jay J Van BavelpersName (2022date). “Troll and divide: the language of online polarization.” PNAS Nexustitle·j.

Object of study — polarized languageterm·dom in tweets by Russian, Iranian and Venezuelan trollsterm and by American users

Research question — how trollsterm may sow intergroup conflict through polarized rhetoric

Contribution — a validated dictionaryterm·key of polarized languageterm·dom for social mediaterm·key

Full text — tagged

Trollterm and divide: the language of online polarizationterm·key Abstract The affective animosity between the political left and right has grown steadily in many countries over the past few years, posing a threat to democratic practices and public health. There is a rising concern over the role that “bad actors” or trollsterm may play in the polarizationterm·key of online networks. In this research, we examined the processes by which trollsterm may sow intergroup conflict through polarized rhetoric. We developed a dictionaryterm·key to assess online polarizationterm·key by measuring language associated with communications that display partisan bias in their diffusion. We validated the polarized languageterm·dom dictionaryterm·key in 4 different contexts and across multiple time periods. The polarizationterm·key dictionaryterm·key made out-of-set predictions, generalized to both new political contexts (#BlackLivesMatter) and a different social mediaterm·key platform (Redditterm·key), and predicted partisan differences in public opinion polls about COVID-19. Then we analyzed tweets from a known Russian trollterm source ( N = 383,510) and found that their use of polarized languageterm·dom has increased over time. We also compared trollterm tweets from 3 countries ( N = 79,833) and found that they all utilize more polarized languageterm·dom than regular Americans ( N = 1,507,300) and trollsterm have increased their use of polarized rhetoric over time. We also find that polarized languageterm·dom is associated with greater engagement, but this association only holds for politically engaged users (both trollsterm and regular users). This research clarifies how trollsterm leverage polarized languageterm·dom and provides an open-source, simple tool for exploration of polarized communications on social mediaterm·key.

Keywords: polarizationterm·key, trollsterm, social mediaterm·key We argue that trollsterm (including foreign actors) use social mediaterm·key to sow discord among Americans through political polarizationterm·key. We developed and validated an open-source linguistic tool to gauge polarized discourse on social mediaterm·key and found that 3 distinct trollterm populations, which hold antiAmerican views, used polarized languageterm·dom more than the average American user. In times of high political instability, misinformation, and disinformation, it is crucial to understand how the enterprise of online foreign interference operates. This research provides insight into the mechanism through which trollsterm function, and sheds light on the role of language in political warfare. It also provides a dictionaryterm·key for other scholars to study the online rhetoric of polarizationterm·key.

Trollterm and Divide: The Language of Online Polarizationterm·key A growing body of research suggests that the American public has become more polarized over the past few decades ( 1 , 2 ). These attitudes are mirrored in rising partisan antipathy; dislike toward members of the opposing ideology—a phenomenon known as “affective polarizationterm·key” ( 3 , 4 , 5 ). The consequences of polarizationterm·key include growing political radicalism ( 6 ), increased ingroup bias ( 7 ), and even different behavioral reactions to deadly pandemics ( 8 ). The alarming consequences of polarizationterm·key are by no means limited to America: India, Poland, Columbia, Bangladesh, Israel, Indonesia, Britain, and Brazil, are just some of the many countries facing growing levels of political polarizationterm·key ( 9 ), and some research attributes this intergroup conflict to the rise of social mediaterm·key (( 10 , 11 ); but see ( 12 , 13 )). In the current paper, we examine the language of online polarizationterm·key employed by regular citizens and internet trollsterm.

On social mediaterm·key, polarizationterm·key is often defined as emerging clusters of like-minded individuals who engage in confirmation bias and curate narratives congruent with their beliefs ( 14 , 15 ). The formation of like-minded social networks is particularly salient in social mediaterm·key platforms that deploy a news-feed algorithm (e.g. Facebook), or a computational formula that favors some pieces of content over others ( 16 ), creating filtered feeds of personalized content ( 17 ). The combination of ideological or partisan groups joining like-minded networks and receiving algorithmically determined political content may be amplifying polarizationterm·key ( 10 , 18 , 19 , 20 ). This trend has raised concerns that people may lose a shared sense of reality.

Although recent evidence suggests that general information consumption on social mediaterm·key might not be an echo-chamber for many users ( 21 , 22 ), there is nevertheless substantial evidence supporting the argument that segregated online communities emerge around politically contentious topics ( 23 , 19 , 24 , 25 , 26 , 27 , 20 ). Moreover, exposure to out-group partisans may even increase polarizationterm·key ( 28 ). A damaging effect of ideology-based homophily is enabling and fostering the spread of misinformation ( 29 , 14 ). Falsehoods appear to spread farther, faster, deeper, and more broadly than the truth on social mediaterm·key, especially for political news ( 30 ). As billions of people have opened social mediaterm·key accounts and use these platforms to get their news, it has also exposed them to a hotbed of conspiracy theories, misinformation, and disinformation ( 31 , 32 ). The rise of misinformation has fueled an international health crisis during the COVID-19 pandemic, leading the World Health Organization to declare this an “infodemic” of misinformation.

There has also been growing concern over the role bad actors may play in online polarizationterm·key and the spread of misinformation (e.g. antiquarantine messages during COVID-19; ( 33 )). For the past several years, cyberspace has been affected by organized groups of social mediaterm·key users, commonly referred to as “trollsterm,” who intentionally pollute online discourse. Since 2018, Twitterterm·key has been releasing the Twitterterm·key Transparency Report, archives of tweets authored by state-affiliated information operations ( https://transparency.twitterterm·key.com/en/information-operations.html ). The most famous of these operations is the Internet Research Agency (IRA), also known as a Russian “Trollterm Farm,” The IRA has engaged in online political tactics to sew intergroup conflict and influence US citizens during the 2016 presidential election ( 34 ) and British citizens prior to the Brexit vote ( 35 ). Similarly, other state-affiliated influence operations have been found in numerous countries, including Iran, Bangladesh, Venezuela, China, Saudi Arabia, Ecuador, the United Arab Emirates, Spain, and Egypt ( https://transparency.twitterterm·key.com/en/information-operations.html ). In the current paper, we developed and validated a polarizationterm·key dictionaryterm·key and examined whether the rhetoric used by these trollterm operations was highly polarized.

Some evidence suggests that trollsterm tend to take on far-right topics and stances, spreading hate speech and islamophobia ( 36 ). However, it would be inaccurate to say that trollsterm are only far-right leaning, and spreading conservative ideology may not even be their ultimate goal. Instead, their main goal appears to be creating polarizationterm·key and fostering social conflict within democracies. For instance, during #BlackLivesMatter discourse on Twitterterm·key Russian trollsterm were heavily engaged in spreading messages from the 2 ends of the debate; both anti-BLM and pro-BLM ( 37 ). The same pattern was observed during online antivaccine debates: trollsterm were found to echo both positions (pro and against vaccines; ( 38 )). Taken together, these data suggest that online trollsterm are attempting to polarize social mediaterm·key users during political discourse.

Overview The current research had 2 goals: (i) to create a dictionaryterm·key of polarized languageterm·dom (i.e. linguistic expressions that are associated with political polarizationterm·key) and (ii) to examine how this language has been used by trollsterm around the world. We began by building a simple tool to measure polarized languageterm·dom. Previous work studied polarizationterm·key through network analysis or by exploring topics known to be polarized ( 39 ). These methodologies have several advantages ( 40 , 41 ) but can be computationally expensive, create a barrier for adoption for behavioral scientists who lack the required technical expertise, and are most likely context-dependent which can undercut replicability ( 42 ). Here, we sought to validate a dictionaryterm·key of polarized languageterm·dom that would be applicable across numerous contexts. In what follows, we describe how the dictionaryterm·key was constructed, its validation using different topics and time periods, and how it tracks dynamic changes in partisan opinions during a time of national polarizationterm·key (the COVID-19 pandemic).

Next, we examined the online rhetoric of trollsterm and regular citizens using the polarizationterm·key dictionaryterm·key. We conducted a high-powered study using nearly 2,300,000 tweets from trollsterm in multiple countries and compared the results to a random sample of American Twitterterm·key users. To help determine if trollsterm were using polarized rhetoric more than the average American ( 38 , 43 ), we examined the levels of polarized languageterm·dom in their tweets when compared to a control group, and explored how levels of polarized languageterm·dom changed over time within each group. These studies suggest that polarized rhetoric was weaponized by online trollsterm during political discourse.

Method Data collection We used the SCI lab twitterterm·key database at Ben-Gurion University ( 44 ). Tweets were collected from all 50 states in the United States and the District of Columbia. We extracted tweets between November 2017 and December 2019. Trollsterm’ data was taken from the Twitterterm·key Transparency Report ( 45–47 ). Additional data collection was done using Twitterterm·key API 2.0 and the “academictwitteR” R package ( 48 ).

All research was conducted in accordance with the Departmental IRB committee at Ben-Gurion University and was ruled “exempt.”

Preprocessing Our sample size consisted of 2,306,233 original tweets in the English language (retweetsterm·dom were filtered out): 383,510 by Russian trollsterm, 329,453 by Iranian trollsterm, 85,970 by Venezuelan trollsterm, and 1,507,300 by American Controls (random sample from our Twitterterm·key database with no specific text search). Following the exclusion of retweetsterm·dom, English tweets constituted 34% of the Russian trollsterm dataset, 15% of the Iranian trollsterm dataset, and 1.25% of the Venezuelan trollsterm dataset.

For our content-matched analysis, we extracted the 20 most-frequent hashtags that appeared on politically engaged Russian trollsterm tweets (#MAGA, #tcot, #BlackLivesMatter, #PJNET, #news, #top, #mar, #topl, #Trump, #2A, #IslamKills, #WakeUpAmerica, #FAKENEWS!, #GOPDebate, #NowPlaying, #TCOT, #ccot, #amb, #sports, #TrumpTrain) and searched for tweets posted in the United States with the same hashtags. After the exclusion of retweetsterm·dom, politically engaged Russian trollsterm sample size was 55,726, and so was their politically matched American controls (55,726).

We could not use our sample of American Controls for Study 4 as it lacked engagement metrics. Therefore, we collected a new control sampleterm·mth, matched in time and without a specific text search (1,144,767).

All tweets had links, tags, and emoticons removed prior to any linguistic analysis. Text mining was done using the "quanteda" package ( 49 ) using R (Versions 3.6.3 and 4.0.3).

Study 1: development and validation of a polarizationterm·key dictionaryterm·key To develop a polarizationterm·key dictionaryterm·key, we synthesized data-driven methods and domain expertise. Specifically, we (i) explored the language associated with polarizationterm·key in a data-driven fashion; (ii) manually pruned the dictionaryterm·key; (iii) expanded the dictionaryterm·key by using GloVe word-embeddings ( 50 ); and (iv) employed manual trimming. The dictionaryterm·key contained 205 words (e.g. corruption, kill, lie, terrorists, political ,and stupid; see online materials for the full list) and its full development and psychometric properties are reported in the Supplementary Information . All the materials are publicly available on OSF https://osf.io/bm8uy .

Dictionaryterm·key validation We first validated our dictionaryterm·key on a subset of the original database used in its construction ( 19 ). The database included tweets about contentious political topics that showed a range of ingroup bias in their spread through social networks (i.e. they either were shared with only the political ingroup or spread to 1 or more outgroup members). We built the dictionaryterm·key on a randomly selected 80% of the original dataset ( N training set = 19,841) and tested it on the remaining 20% ( N test set = 5,008). This out-of-sample testing was conducted to ensure the predictive performance of the model and to avoid overfitting. Data preprocessing included removing all duplicates from the data and automatically deleting links and emojis from the text. A polarizationterm·key score was calculated based on the count of dictionaryterm·key words in the text, normalized by the tweet length. The means reported below represent the average percentage of the text that was found in the dictionaryterm·key (for a similar approach see LIWC; ( 51 ). Our analysis found that the dictionaryterm·key successfully discriminated between polarized and nonpolarized tweets from the test set ( M polarized = 6.70, SD polarized = 9.08, N = 3696 and M nonpolarized = 4.39, SD nonpolarized = 6.60, N = 1312), t (3156) = 9.79, P < 0.001, Cohen's d’ = 0.27. In other words, our dictionaryterm·key was able to determine which corpus was more likely to include polarized communications compared to another corpus.

To evaluate generalizability, we validated the polarizationterm·key dictionaryterm·key with a different political topic (i.e. different from the original research). We examined the effectiveness of the polarizationterm·key dictionaryterm·key in the context of the online #BlackLivesMatter (#BLM) discourse between December 2015 and October 2016, which focused on issues of racial justice in the United States. Prior work had studied the flow of information in #BLM tweets by using a machine learning clustering technique to identify distinct Twitterterm·key communities and quantifying the spatial retweetterm·dom flow within and between clusters ( 37 ). The original dataset included 58,698 tweets ( https://github.com/leo-gs/ira-reproducibility ), and we were able to retrieve 24,747 tweets out of the original sample from Twitterterm·key's API. Like in the prior validation, messages were categorized with regard to the spread of information; whether the tweets showed ingroup bias (retweeted within 1 political cluster), or not (retweeted by a user from the other cluster, as classified by the authors). We applied our dictionaryterm·key to the posts we were able to retrieve, and again we observed that ingroup bias messages contained more polarized languageterm·dom than messages that diffused between clusters ( M ingroup bias = 5.54, SD ingroup bias = 5.68, N = 24,077 and M diffused = 4.83, SD diffused = 5.09, N = 670), t (716.06) = 3.58, P < 0.001, Cohen's d’ = 0.13. This helped establish the generalizability of our dictionaryterm·key to a novel political topic.

Beyond testing out-of-sample generalizability, we also tested cross-platform generalizability. We tested the polarizationterm·key dictionaryterm·key on the platform Redditterm·key using a wider range of political topics. Redditterm·key is an online social mediaterm·key platform that consists of many discussion forums, or communities, called subreddits , including several communities devoted to politics ( 52 ). We extracted up to 1,000 messages from 36 political communities with established ideologies (18 from each political side). As a control group, we sampled up to 1,000 messages from 18 other communities, randomly sampled from a list of popular subreddits ( https://github.com/saiarcot895/reddit-visualizations ). We collected 53,859 posts between June 2015 and December 2018 from the Pushshift Redditterm·key API ( 53 ). Following data cleaning, our sample size consisted of 49,230 original posts. We applied the polarizationterm·key dictionaryterm·key on the Redditterm·key sample and conducted a one-way between-group ANOVA. A planned comparison between the political groups revealed a significant difference between the control and the other political communities ( M left = 2.38, SD left = 4.61, N = 17,005; M right = 2.57, SD right = 5.34, N = 15,859; and M control = 0.97, SD control = 3.44, N = 16,366), t (49,227) = 34.81, P < 0.001, Cohen's d’ = 0.31. More information is reported in the Supplementary Information . In other words, the rhetoric in political Redditterm·key groups was more polarized than apolitical Redditterm·key groups.

As a more stringent sensitivity test, we replaced the randomly sampled control group with a “neutral” reference of contentious topics. We extracted messages from the popular subreddit NeutralPolitics ( www.redditterm·key.com/r/NeutralPolitics ), a redditterm·key community devoted to factual and respectful political discourse. This sample consisted of 9,984 posts between April 2016 and December 2018 (9,772 after data cleaning). A planned comparison between the political groups revealed a significant difference in polarized rhetoric between NeutralPolitics and the other political communities ( M left = 2.38, SD left = 4.61, N = 17,005; M right = 2.57, SD right = 5.34, N = 15,859; and M neutral = 2.24, SD neutral = 4.49, N = 9,772), t (42,633) = 4.12, P < 0.001, Cohen's d’ = 0.04. See Supplementary Information for more details. This suggests that polarized rhetoric was reduced among the redditterm·key community focused on respectful political discourse (although we note that the effect size here is very small).

To determine if our dictionaryterm·key would track dynamic changes in polarized public opinions over time, we compared polarized languageterm·dom with polls about US citizens’ concern about the COVID-19 pandemic. The data were collected from a representative panel by Civiqs ( https://civiqs.com/results/coronavirus_concern ), an online polling and analytics company. Recent polls have revealed clear partisan differences between Democrats and Republicans in reported concerns about the COVID-19 pandemic—such that Democrats are consistently more concerned about the pandemic than Republicans ( 54 ). We tested whether the language in tweets about coronavirus was associated with the partisan discrepancy in public opinion about COVID-19. We calculated a “partisan difference score” from 2020 February 25th until 2020 April 14th by subtracting the daily Republican net concern from the daily Democratic net concern, as reported by Civiqs (the specific question was “how concerned are you about a coronavirus outbreak in your local area?”). The poll was based on responses from 22,256 respondents and included measures to avoid demographic and ideological biases.

To compare Twitterterm·key language to partisans’ concern, we collected 553,876 Twitterterm·key messages from the United States within these dates that used the terms “covid” or “coronavirus.” We then applied the polarizationterm·key dictionaryterm·key to the tweets and aggregated by date. We found that polarized languageterm·dom on social mediaterm·key, measured by the mean % of wordsterm·mth from our dictionaryterm·key contained in the tweets, was positively associated with partisan differences in concern about the COVID-19 pandemic over time, r (48) = 0.45, P = 0.001, see Figure 1 . A post hoc analysis revealed that the correlation between poll responses and twitterterm·key language was strongest when Twitterterm·key language was lagged by 8 days (i.e. poll t0 , twitterterm·key t8 ) r (40) = 0.67, P < 0.001 (for a full lag table of 16 days, see Table S1, Supplementary Material ). In other words, polarized rhetoric about COVID-19 mirrored polarizationterm·key in public opinion over the early phase of the pandemic. This also suggests that the polarizationterm·key dictionaryterm·key may be useful in detecting future patterns of public opinion.

Dynamic polarizationterm·key changes in polls of COVID-19 concern and polarized languageterm·dom on Twitterterm·key. The solid line represents partisan differences in COVID-19 concern ( N = 22,256), and the dashed line represents the degree of polarized discourse on Twitterterm·key ( N = 553,876, dashed line). Values on the X -axis represent the time, and values on the Y -axis represent standardized scores of the variables. The functions have gone through a locally estimated scatterplot smoothing (span = 0.33, degree = 1). Shaded areas around the regression line denote 95% CI.

Taken together, these 4 sets of analyses (cross-validation, out of set validation, cross-platform validation, and predictive validation) provide converging validity for the dictionaryterm·key, showcasing its ability to capture political polarizationterm·key in language across 4 different contexts. For a summary of all validation steps, see Table 1 .

Summary of validation steps. Effect sizes correspond to Cohen's d’ or Pearson's r . All tests are significant at P < 0.001.

Study 2 Study 2a: polarizationterm·key in Russian trollsterm Russian trollsterm, or anonymous social mediaterm·key accounts that are affiliated with the Russian government, were active around highly contentious political topics around the world, including in the United States and Britain ( 34 , 35 ). With the release of the Twitterterm·key Transparency Report, a sample of the Russian and other countries’ operations were officially disclosed and used to study the role of trollsterm in amplifying political polarizationterm·key ( 37 , 38 , 55 ). Therefore, we hypothesized that state-affiliated trollsterm would use more polarized languageterm·dom on social mediaterm·key compared to ordinary Twitterterm·key users. We also examined how polarized languageterm·dom may have changed over time. For instance, if trollsterm' levels of polarized languageterm·dom are increasing over time, it would imply that trollsterm are spending increased energy toward tactics that sow discontent and aim to influence polarized discourse. On the other hand, levels of polarized languageterm·dom might be increasing among American Twitterterm·key users as well, similar to trends of affective polarizationterm·key ( 5 ).

Results We compared Twitterterm·key messages posted by trollsterm to an American control sampleterm·mth (collected from across the United States through the Twitterterm·key API). We only used original tweets that were posted in the English language and were most likely aimed for an international/American audience. The comparison was matched for the same time range (2016 November 23–2018 May 30). We applied the polarizationterm·key dictionaryterm·key, which was generated from and validated on different datasets (see Study 1) to extract polarizationterm·key scores. First, we found that Russian trollsterm ( M = 2.37, SD = 5.14, and N = 61,413) used significantly more polarized languageterm·dom than tweets sent by the control sampleterm·mth ( M = 1.47, SD = 5.35, and N = 516,525), t (78,081) = 40.96, P < 0.001, and Cohen's d = 0.17. These results suggest that trollsterm are leveraging polarized languageterm·dom to push conflict among the US citizens in the context of political discourse. For the top 25 most used words adjusting for their frequency (tf-idf), see Figure S1 (Supplementary Material) .

However, not all trollsterm are equal. Research suggests that Russian trollsterm could be classified into 5 distinct types: Right, Left, News, Hashtag Gamers, and Fearmongers ( 56 ). It could be argued that a cleaner analysis would only constitute Left and Right trollsterm, and should be contrasted with a politically engaged American sample. Therefore, we used the Russian Trollterm classification (Shared in partnership with FiveThirtyEight on https://github.com/fivethirtyeight/russian-troll-tweets ) ( 56 ), and matched an American sample for their content (via hashtag use, see Method section), posting time (January 2015–May 2018) and quantity. Again, we find that politically oriented Russian trollsterm use significantly more polarized languageterm·dom than their politically matched American sample (Russian trollsterm: M = 5.16, SD = 8.00, and N = 55,726; American controls: M = 2.91, SD = 6.84, and N = 55,726), t (108,836) = 50.61, P < 0.001, and Cohen's d = 0.30 (for a robustness check, see Supplementary Materials ).

To determine if polarized languageterm·dom is increasing over time, we sampled 1,507,300 tweets that were posted between November 2016 and December 2019 in the United States. These tweets were pulled randomly from BGU's SCI lab twitterterm·key database (sampling approach described in the Method section), with no specific text search. We applied the polarizationterm·key dictionaryterm·key to the text and aggregated by months. We conducted a weighted linear regression with monthly observations as the weighting factor. We found that Russian trollsterm used far more polarized languageterm·dom as time progressed ( b = 0.03), R 2 = 0 .46, F (1, 69) = 58.85, and P < 0.001, Moreover, this was a strikingly large effect size. We did not find the same pattern among American control users, ( b = −0.001) R 2 = 0 .05, F (1, 35) = 1.90, and P = 0.178 (see Figure 2 ). This suggests that trollsterm are increasing the use of polarized languageterm·dom much faster than ordinary users, independent groups correlation comparison z = 5.06, 95% CI [.55, 1.21], and P < 0.001.

Scatter plot of the average polarized score by Twitterterm·key sample. We examined monthly polarized languageterm·dom in American controls ( N = 1,507,300; blue), and trollsterm from Russia ( N = 383,510; red), Venezuela ( N = 85,970; yellow), and Iran ( N = 329,453; green). Values on the Y -axis represent the average % of polarized languageterm·dom in the month. The size of the dots corresponds to the monthly sample size. Shaded areas around the regression line denote 95% CI. Note that the Y -axis is fixed to 0–5, data points exceeding this limit are not shown in the figure; the regression lines take these observations into account. Results indicate that trollsterm from Russia and Venezuela have been increasing their use of polarized rhetoric, but Americans have not.

This finding suggests Russian trollsterm have increased their use of polarized rhetoric, but the average US Twitterterm·key does not show evidence of mirroring the type of language used by the trollsterm. This could be because trollsterm are only reaching and influencing the most politically active Twitterterm·key users, or that the average user expresses polarized attitudes in different ways. However, we note that the time frame for trollsterm and controls is not identical. As such, any differences in these trends should be treated as tentative. That said, in a post hoc analysis conducted on the same time frame (2016 November 23–2018 May 30), Again, Russian trollsterm used far more polarized languageterm·dom as time progressed ( b = 0.03), R 2 = 0 .51, F (1, 17) = 17.48, and P < 0.001 while American control users did not, ( b = 0.006), R 2 = 0 .16, F (1, 17) = 3.14, and P = 0.094 (however note the small sample sizes in this analysis).

Study 2b: polarizationterm·key in Venezuelan and Iranian trollsterm We, next, sought to see if this pattern of polarized languageterm·dom generalized to other political contexts and countries. Given Russia's effort at online political warfare ( 57 ), we also tested whether polarizationterm·key attempts extended to other political actors. Russia, Iran, and Venezuela all hold antiAmerican views and share warm relationships with each other ( 58 , 59 , 60 ). Therefore, these countries may have incentives to meddle with American politics. We analyzed trollsterm from these nations to see if they were using similar polarized rhetoric to sow conflict with Americans.

Results We compared Twitterterm·key messages posted by Venezuelan and Iranian trollsterm (identified by Twitterterm·key 1 ) to a neutral American control sampleterm·mth. Again, we only used original tweets that were posted in the English language which were most likely aimed for an international/American audience. The paired comparisons were again matched for the same time range. In both the countries we examined, the tweets sent by trollsterm used significantly more polarized languageterm·dom than tweets sent by American control samplesterm·mth ( P s < 0.001), see Table 2 . For the top 25 most-used words adjusting for their frequency (tf-idf), see Figure S1 (Supplementary Material) .

Means, SDs, sample sizes, and time range for each trollterm group comparison with American controls. The table consists of t statistics, degrees of freedom, and Cohen's d’ . All the t tests are significant at P < 0.001.

Following the same analysis as in Study 2a, we conducted a weighted linear regression with monthly observations as the weighting factor, and found a diverging pattern between populations of trollsterm: Whereas trollsterm based in Venezuela used more polarized languageterm·dom as time progressed ( b = 0.02), R 2 = 0 .19, F (1, 91) = 20.91, and P < 0.001, Iranian trollsterm used less polarized languageterm·dom ( b = -0.03), R 2 = 0 .33, F (1, 85) = 41.06, and P < 0.001, see Figure 2 . Therefore, any trends in polarized languageterm·dom might be specific to the foreign nation involved.

Study 3: exploratory topics of polarizationterm·key In Study 2, we showed that foreign agents from various countries strategically used polarized languageterm·dom in social mediaterm·key communications, and in a majority of cases we see an increase over time in these attempts. One remaining question is what specific forms of polarized rhetoric are leveraged by different groups. Investigating the themes associated with polarized languageterm·dom could shed light on the social–psychological processes capitalized by trollsterm, and generate better intuition on their strategies. Therefore, we conducted an exploratory analysis in which we decomposed the dictionaryterm·key into different factors to determine whether these factors could contribute to our understanding of the trends in online polarized rhetoric.

As in Study 1, we used GloVe word embeddings as a high-dimensional representation of the words in the polarized dictionaryterm·key ( 50 ) and conducted hierarchical clustering analysis. We observed that in the highest level of division, the algorithm clustered the words in a manner that is somewhat consistent with the theoretical separation between issue and affective polarizationterm·key (see Figure 3 and Figure S2, Supplementary Material ). See Supplementary Information for more information.

Semantic space representation of the Polarizationterm·key Dictionaryterm·key. X - and Y -axes represent t-SNE dimensionality reduction of GloVe embeddings. Words in red mark the first cluster (“Affective”) and words in blue mark the second cluster (“Issue”).

Scholars have made the conceptual distinction between issue polarizationterm·key—an ideological, policy-based political divide, and affective polarizationterm·key, i.e. dislike, distrust, and general animosity of political partisans toward the other political side ( 5 , 61 , 62 ). This distinction is roughly reflected in the dictionaryterm·key. For example, while the Issue subcomponent addresses ideological and policy keywords (e.g. liberal , conservative , socialism , and gun-control ), the Affective component references instances of negative moral–emotional words (e.g. kill , destroy , and cheaters ; ( 19 )), and distinct ethnic and religious groups (e.g. Muslim and Jews ). We should note that this is our own interpretation of the clusters and other theoretical mappings may fit as well (e.g. Affective could be interpreted as partisan taunting ( 63 ); therefore, the labels Affective and Issue should be treated as relatively fuzzy concepts.

With these divisions in mind, we tested whether language associated with issue vs. affective polarizationterm·key was associated with differential language use among trollsterm and ordinary users.

Results We applied the 2 subsets of the polarizationterm·key dictionaryterm·key on the social mediaterm·key messages posted by trollsterm and a random sample of American users. As in Study 2, we compared polarizationterm·key levels between the groups (paired comparisons matched for the same time range). In all the countries we examined, the tweets sent by trollsterm used significantly more polarized languageterm·dom than tweets sent by American control samplesterm·mth ( P s < 0.005), both on affective and issue polarizationterm·key, see Table 3 and Figure S3 (Supplementary Material) . Temporal analyses are reported in the Supplementary Information .

Means, SDs, sample sizes, and time range for each trollterm group comparison with American controls by Issue and Affective polarizationterm·key components. The table consists of t statistics, degrees of freedom, and Cohen's d’ . All the t tests are significant at P < 0.005.

In the current exploratory study, we showed that the polarizationterm·key dictionaryterm·key is composed of 2 subcomponents that map onto theoretical elements of polarizationterm·key (Issue and Affective). In addition, we showed that all trollterm groups use more polarized languageterm·dom than a random sample of American social mediaterm·key users and that this holds for both affective and issue polarizationterm·key (although effect sizes of issue polarizationterm·key are substantially larger).

Study 4: polarized languageterm·dom and engagement Studies 2 and 3 demonstrated a link between polarized languageterm·dom use and trollterm accounts on social mediaterm·key. Indeed, our findings are consistent with the idea that trollsterm sow discord among Americans by using polarized languageterm·dom in conversations with others. Yet our results are agnostic to whether polarized languageterm·dom creates divisions vs. merely reflects an existing polarized state. To address this ambiguity, in the current study we investigate the extent to which polarized languageterm·dom is associated with increased engagement If polarized languageterm·dom used by trollsterm draws more engagement from ordinary social mediaterm·key users, it would demonstrate that when trollsterm seed polarized languageterm·dom online, users become active agents in spreading the polarized messaging among groups.

Engagement of polarized languageterm·dom is an important metric because engagement with political content online is generally associated with high levels of ingroup bias; that is, it is far more likely to be shared within the political ingroup than in the outgroup ( 18 , 19 , 16 ). For example, 1 of the key predictors of engagement on Twitterterm·key and Facebook in the political context is outgroup animosity ( 64 ). Whether intentional or not, if polarized languageterm·dom used by Trollsterm is associated with increased engagement, it would suggest that Trollsterm’ language use has potential to exacerbate division among political users (even if users were already divided).

Results To test whether polarized languageterm·dom is associated with selective engagement in political discourse, we utilized the Russian Trollsterm Classification data ( 56 ) from Study 2. We only included observations for which we had an engagement metric (i.e. retweetterm·dom count, N = 118,215). For ease of interpretation, we counted the number of words in the dictionaryterm·key and used it to predict the retweetterm·dom count in a negative binomial generalized linear model. We find that for every polarized wordterm·dom in a tweet, retweetsterm·dom increase by 72%, IRR = 1.72, CI [1.64, 1.79], and P < 0.0001. Next, we added an interaction by trollterm category. We conducted an Analysis of Deviance and found both main effects and interaction term significant (Polarized Languageterm·dom: χ 2 (1) = 5263 , P < 0.0001; Trollterm Category: χ 2 (7) = 373,563 , P < 0.0001; and Polarized Languageterm·dom * Trollterm Category: Category: χ 2 (7) = 2223 , P < 0.0001). In a planned comparison we find that this effect stems directly from political trollsterm, Political vs. non Political trollsterm ratio = exp(1.35*10 15 ), 95% CI [exp(5.03*10 14 ), exp(2.2*10 15 )], and P < 0.0001, see Figure 4 .

Polarized languageterm·dom predicts retweetsterm·dom in political Russian trollsterm. The graph depicts the number of retweetsterm·dom predicted for a given tweet as a function of polarized languageterm·dom present in the tweet and type of trollterm. Bands reflect 95% CIs. For varying Y -axes, see Figure S5 (Supplementary Material) .

We take these results as evidence that polarized languageterm·dom is indeed polarizing, however, there is no reason to assume this effect applies strictly to trollsterm. We conducted the same analysis on samples of politically engaged controls (Study 2, N = 55,726), and a new sample of American controls for which we obtained engagement metrics ( N = 1,144,767). Again, we find that in the politically engaged controls there is a positive association between polarized languageterm·dom and retweetsterm·dom, such that for every polarized wordterm·dom in a tweet, retweetsterm·dom increase by 39%, IRR = 1.39, 95% CI [1.35, 1.48], and P < 0.0001. However, in a random sample of Americans we do not find a significant association IRR = 1.19, 95% CI [0.80,1.77], and P = 0.390.

We should note that these analyses are usually done with the number of followers as a covariate, yet retrospective information was only available for the trollsterm’ dataset. For transparency, we show here the analysis controlling for the covariate. After adding followership in the trollsterm analysis we find the same pattern of results, however the effect size diminishes: IRR = 1.61, 95% CI [1.57, 1.67], and P < 0.0001; planned contrasts: Political vs. non Political trollsterm ratio = exp(7.6*10 9 ), CI [exp(3.6*10 9 ), exp(1.21*10 10 )], and P < 0.0001.

Overall, these results indicate that polarized languageterm·dom is associated with greater traction on social mediaterm·key, but only in political contexts. Since the probability of a political message to be retweeted within the political ingroup is far greater than the outgroup, we take this as evidence that polarized languageterm·dom is not only a marker for a static polarized state, but contributes to the polarizationterm·key process.

Discussion We developed and validated a dictionaryterm·key of polarized languageterm·dom used on social mediaterm·key. We validated this dictionaryterm·key using 4 strategies and showed it consistently detected polarized discourse on Twitterterm·key and Redditterm·key on multiple topics and corresponded well to the dynamics of partisan differences in attitudes towards the COVID-19 pandemic. We found that state-affiliated trollsterm from Russia and other countries use more polarized languageterm·dom than a random sample of American users and that while the language of Russian and Venezuelan trollsterm have used more polarized rhetoric with time, levels of polarized languageterm·dom in American controls did not increase. We found that our data-driven dictionaryterm·key taps into distinct theoretical elements of polarizationterm·key, and that trollsterm from all tested countries use more polarized rhetoric in both issue and affective factors (broadly denied). Lastly, we showed that polarized languageterm·dom is associated with more traction on social mediaterm·key, but only in political contexts; this finding suggests that polarized languageterm·dom advances polarizationterm·key and not merely reflects it.

These results expand on prior work documenting trollsterm’ attempts to pollute the online environment with polarized content and sow discord among Americans ( 65 ). We provide novel evidence that this mission spans several countries that hold antiAmerican views. Prior research has revealed that when exploring the clusters of polarized topics, trollsterm are often found in the centroids of these clusters, driving the partisan discourse on both ends ( 37 , 38 , 55 ). Our research extends these findings; we found that trollsterm share controversial content and engage in highly polarized issues, but that they also use higher levels of polarized languageterm·dom as a tool in their discourse. In addition, we found that polarized languageterm·dom is associated with greater engagement, however, this association only holds for politically engaged users—both trollsterm and controls. This is consistent with a view that trollsterm’ use of polarized languageterm·dom is intended and weaponized in order to sow polarizationterm·key, however our methods are not sufficient to draw such causality.

Questions remain as to the extent of influence of trollsterm’ social mediaterm·key presence on real people. However, it is important to note that even a small number of agents with aggressive attitudes can have a substantial influence on the majority view, a process called “information gerrymandering” ( 66 ). Exposure to polarizing attitudes even produced by a small number of agents can have a devastating effect on political compromise in a social network; such findings suggest that trollsterm have the ability to influence many of the users on social networks. Furthermore, recent evidence suggests that trollterm's messages propagate to mainstream media and are represented as “the voice of the people” ( 67 ). This way, trollsterm win twice: once when they share the polarized content, and then again when it is being echoed on other media platforms, creating a polarizing loop.

However, some are skeptical of the change trollsterm may impose on people's attitudes. A recent paper followed over 1,200 American Twitterterm·key users for the course of 1 month in late 2017. The authors found that only a small fraction of users interacted with Russian trollsterm, and they did not observe any change in partisan attitude during that time among these users ( 68 ). In a study that explored the domestic effect of Russian trollsterm (i.e. messages that were targeted inwards to Russian users), it was found that trollsterm were trying to promote a progovernment agenda and dissolve government criticism ( 69 ); nevertheless, trollsterm were only successful at the latter, suggesting their influence is restricted in scope. While our results cannot speak to causal factors, we do find that while levels of polarized languageterm·dom were rising in Russian trollsterm, this was not the case among American users. Future research is required to understand the precise impact trollsterm have in reference to specific political events.

Given the evidence on the growing polarizationterm·key and partisan antipathy in the American public ( 5 ), we also explored whether polarized discourse on social mediaterm·key would increase with time among a sample of American users. We did not find evidence to support this hypothesis; levels of polarizationterm·key did not increase across time, suggesting that polarized discourse among average American users did not grow between November 2016 and December 2019. These results are consistent with other findings that do not find evidence for increased polarizationterm·key during this brief time frame ( 70 ). This could suggest that polarized discourse has not changed, that it has reached a plateau, or that American users’ way of expressing polarized languageterm·dom has changed slightly over time. Discerning between these possibilities is an important endeavor for future research.

This paper also introduced the polarizationterm·key dictionaryterm·key and showcases its validation and application in studying political polarizationterm·key. The dictionaryterm·key is easy to use and can be utilized externally with LIWC ( 51 ), or with the example code provided in the Supplementary Information for R . Having a quantifiable measure of polarized languageterm·dom in social mediaterm·key messages is a quick way to estimate polarizationterm·key levels that aligns with other current practices, wherein researchers relied on computationally extensive network analyses, or narrowed down to a specific partisan topic to carry out their studies.

The current study has several limitations. The polarizationterm·key dictionaryterm·key has been built on data collected in 2015 and on 3 polarized topics. Therefore, it is subjected to bias about topics that were timely in 2015 and is potentially restricted in its scope. We attempt to get around this limitation by expanding the lexicon using word-embeddings and testing its validation over multiple time periods. Nonetheless, language is highly dynamic on social mediaterm·key and our dictionaryterm·key should always be validated when applied to a new context. Given its data-driven development, it also includes some terms that may not seem strictly polarized (e.g. people ). Therefore, if being used by other researchers, we recommend using it comparatively by having a baseline corpus and measuring amounts of polarized languageterm·dom between groups to get a relative estimate.

A potential issue is with the authenticity of early social mediaterm·key accounts identified as trollsterm. Some countries use hacked, purchased, or stolen accounts. Early data, therefore, may not have originated with the nation in question. While this was probably not the case with the Russian trollsterm dataset, it could be the case with some Venezuelan or Iranian content, and may have biased our polarizationterm·key over time analyses. That said, we employed a weighted regressions analysis that takes into account the relatively sparse nature of early messages (and therefore, downweights their importance). These analyses complement the Russian sample and provide a wider, descriptive view of how different trollterm populations use polarized languageterm·dom.

In addition, this work has focused primarily on quasi-experimental manipulations or correlational methodology. Future work should examine if there are causal factors that increase or decrease polarizationterm·key. For instance, given the potential influence that the design of social mediaterm·key can have on moralized language ( 29 ), it is possible that specific design feature changes could impact polarizationterm·key language. For instance, down-weighting polarized languageterm·dom on social mediaterm·key news feeds might influence attitudes such as partisan antipathy.

Conclusion Taken together, this research offers a tool to detect and understand the use of polarized rhetoric on social mediaterm·key. In times when it seems like we have reached toxic levels of polarizationterm·key in America, it is increasingly important to continually develop tools to study and combat the potentially polarizing influence of foreign agents in American politics.

Funding This work was partially supported by research grants through Grant number 61378 from the John Templeton Foundation to J.V.B.

Supplementary Material Click here for additional data file.

Acknowledgments The authors would like to thank Dr Michael Gilead for resource assistance, and to the Civiqs company and Dr Kate Starbird for data sharing. We would like to acknowledge members of the NYU Social Identity and Morality Lab for comments on a previous version of this manuscript.

Notes Competing Interest: The authors declare no competing interest.

Contributor Information Almog SimchonpersName, Department of Psychology, Ben-Gurion University of the Negev, POB 653, Beer Sheva 8410501, Israel; School of Psychological Science, University of Bristol, BS8 1TU, Bristol, UK.

William J BradypersName, Department of Psychology, Yale University, CT 06520-8205, New Haven, CT, USA.

Jay J Van BavelpersName, Department of Psychology, New York University, New York, NY 10003, USA; Center for Neural Science, New York University, New York, NY, USA.

Authors' Contribution A.S. and W.J.B. conceived and designed the experiments; A.S. and W.J.B. performed the experiments; A.S. and W.J.B. analyzed the data; A.S., W.J.B., and J.V.B. contributed materials/analysis tools; and A.S., W.J.B., and J.V.B. wrote the paper.

References Associated Data Supplementary Materials Click here for additional data file.