One of my favorite ways to waste time is reading internet comments. You can observe all kinds of angry people there and wonder at their pathologies. While I was still in school I particularly enjoyed reading the comments section of the Daily Princetonian because of its relevance to me. Though I only worked for a brief time formatting the print edition of the paper, I got to know some of the journalists, opinion writers, and editors through that work and through classes. Writing for a college newspaper requires a thick skin; the commenters are almost always out to get you.

At some point I decided I would try to learn something about internet news and comments, perhaps so I could convince myself all those hours were not wasted. A basic question we can ask about news and comments is what are the paper and readers talking about? This kind of question, of what topics underly a corpus, is what probabilistic topic modeling can help answer. A frequently used method for topic modeling is latent Dirichlet allocation (Blei, Ng, & Jordan, 2003). There have been many extensions to this original model, including some that focus on internet posts and their comments. In this post I describe a model introduced in (Yano, Cohen, & Smith, 2009) and apply it to a corpus of Daily Princetonian articles and their comments.

Data

I focus in particular on the opinion pieces and letters of the Daily Princetonian from fall 2013 to spring 2015–opinion pieces because in my experience these columns tend to attract the most, and the most rabid, comments, and that timespan because the Prince migrated its content to Wordpress in spring 2013 and all comments from before then have been lost. Using beautifulsoup and Selenium I scraped 962 articles from the opinion section with a total of 3576 comments. Using the tm package in R I removed capitalization, whitespace, numbers, and punctuation, stemmed the words using the Porter stemmer (Porter, 1980), and converted the documents to the bag of words representation required by Latent Dirichlet allocation. The data used in this analysis, as well the code for the model, is available at my repo comment_type_topic_model.

Model

(Yano, Cohen, & Smith, 2009) describe this model as LinkLDA. I do not use the other model they propose, CommentLDA, because it depends on commenter identity. The Disqus commenting system that the Prince uses allows guest commenters who can use anything as their usernames; thus it’s not possible to identify guests, who posted about 40% of the comments I consider here.

The generative process of the model is:

$$\begin{align*} \theta_d &\vert \alpha \sim Dirichlet(\alpha) \\ z_{dn} &\vert \theta_d \sim Multinomial(\theta_d) \\ \beta_k &\vert \eta \sim Dirichlet(\eta) \\ w_{dn} &\vert z_{dn}, \beta \sim Multinomial(\beta_{z_{dn}}) \\\ z'_{dcn} &\vert \theta_d \sim Multinomial(\theta_d) \\ \beta'_k &\vert \eta' \sim Dirichlet(\eta') \\ w'_{dcn} &\vert z'_{dcn}, \beta' \sim Multinomial(\beta'_{z'_{dcn}}) \end{align*}$$

where $w_{dn}$ is the $n$th observed word in article $d$, $z_{dn}$ is the latent topic assignment of $w_{dn}$, $\theta_d$ is the topic proportion parameter of article $D$, $w'_{dcn}$ is the $n$th observed word in comment $c$ in article $d$, $z'_{dcn}$ is the latent topic assignment of $w'_{dcn}$, $\beta_k$ is the article distribution over words of topic $k$, $\beta'_k$ is the comment distribution over words of topic $k$, and $\eta$, $\eta'$, and $\alpha$ are the Dirichlet prior parameters for $\beta_k$, $\beta'_k$, and $\theta_d$, respectively.

And in plate notation:

I implemented a collapsed Gibbs sampler for the model with Rcpp.

Article and Comment Topics

At first I was going to choose the number of topics K in some disciplined way, like evaluating based on held out perplexity, but that requires various approximations and I got lazy. So I just settled on 40 topics. I ran 2000 iterations of the Gibbs sampler. I used the term score metric described in (Blei & Lafferty, 2009) to determine an order for topic visualization. What follows are 10 topics I found coherent and interesting. I list the words that are among the 20 highest by term score in both the article and comment versions of the topic to give a sense of topic identity. I also take the difference between article and comment term scores and list the words with the greatest difference to convey which words are most likely to appear in articles rather than in comments for the same topic. Likewise I use the difference between comment and article scores to illustrate which words are most likely to appear for comments rather than articles. I have labeled each topic myself, based on my own judgment.

The differences between the article and comment distributions for each topic are interesting but cryptic if you don’t read the Prince. I have read the Prince and can offer some insight. In the Israel-Palestine topic for instance, the articles are more likely to discuss events or initiatives related to the conflict, like referenda on whether to divest from Israeli companies. Meanwhile in the comments people focus on identities and emphasize conflict, usually by accusing Israel of being an oppressive apartheid state and accusing Muslims of being hateful killers who want to destroy Israel. In the sexual assault topic, the opinion writers usually decry rape culture, while in the comments people warn against false accusations of rape and blame the victim for being drunk or just regretting poor decisions. Generally the opinion writers are students and relatively liberal in outlook, while many commenters are not even students and often reactionary.

References

1. Porter, M. F. (1980). An algorithm for suffix stripping. Program, 14(3), 130–137. Retrieved from http://dblp.uni-trier.de/db/journals/program/program14.html#Porter80
2. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. J. Mach. Learn. Res., 3, 993–1022. Retrieved from http://dl.acm.org/citation.cfm?id=944919.944937
3. Yano, T., Cohen, W. W., & Smith, N. A. (2009). Predicting Response to Political Blog Posts with Topic Models. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics (pp. 477–485). Stroudsburg, PA, USA: Association for Computational Linguistics. Retrieved from http://dl.acm.org/citation.cfm?id=1620754.1620824
4. Blei, D. M., & Lafferty, J. D. (2009). Topic Models.