Skip to content Skip to navigation

Debating the Methods in Matt Jockers's Macroanalysis

On September 3rd we had our second meeting of the Stanford Digital Humanities Reading Group, in which we discussed Matt Jockers’s new book, Macroanalysis: Digital Methods & Literary History. Because Jockers is a former colleague, a co-founder of Stanford’s Literary Lab, and a friend to several people in the reading group, I went into this meeting anxious that we might all be too happy with his book to sustain ninety minutes of conversation. I was very wrong. Jacqueline Hettel, whose Ph.D. research focused on text analysis of domain-specific language using corpus linguistics, prompted a vigorous debate about the methods Jockers uses in Macroanalysis. Hettel’s primary critique is that the statistical methods behind topic modeling, word frequencies, and other methods that undergird the book’s chapters are heavily dependent upon a set of assumptions common to NLP, Chomsky, and other primarily American approaches to understanding language.

Topic models, for example, rely on the assumption that Bayesian analysis can accurately describe how language works. When Jockers, in chapter 7 (“Nationality”) relies on the mean usage of the word “the”, he assumes that language has a Gaussian distribution. Hettel prefers a log-likelihood method, among others, owing to her training in the school of linguistic thought exemplified by her major professor William Kretzschmar, who follows John Firth and others in what is known as the “London School”. I am not a statistician nor a linguist, so it did not occur to me that the statistical methods Jockers uses might be controversial or, more importantly, that they make assumptions about the nature of language. This topic led the group to consider the purpose of the book, the audience, and its relationship to more traditional modes of literary scholarship. Is the evidence in support of Jockers’s argument meant to get at some truth, and hence tangled up with the problems of scientism, or, as Blevins asked, is the evidence he presents more akin to the sort we find in a close reading, where the force of argument is driven by a persuasive narrative?

Moreover, Jockers, whose Ph.D. is in English and who is a scholar of Irish literature, is not a linguist or statistician either. Was his use of tools like relative work frequencies, topic models, and part-of-speech taggers a conscious choice that reflects his understanding of the nature of language or an unconscious one borne of an ignorance of this other realm of quantitative studies of language? In using methods that edge so close to those that have been used in linguistics for a relatively long time compared to the newness of quantitative text analysis in digital humanities, Jockers prompts us to think about how such scholarship may overlap disciplines in which we lack enough expertise to even understand what our choices entail. Indeed, my uneasy truce with the use of topic models for literary analysis stems from my keen awareness of how little I understand about the assumptions behind Latent Semantic Analysis.

We must also consider the context not only of the language under study, a point Hettel emphasized, but also the context in which the methods were chosen. Perhaps owing to the work of Stanford’s Natural Language Processing group, led by Christopher Manning and Dan Jurafsky, Jockers was led when he began his work in this area towards tools like the Stanford POS Tagger and others that imply certain language models of which those of us using such tools are not fully aware.

We also touched upon the nature of classification in digital humanities, which Ted Underwood has written about recently on his blog. In Macroanalysis, Jockers regularly classifies texts according to genre, nationality, or gender, but one of these things is not like the others. Genre, as studies in neuroscience and cognitive literary theory have shown (and which is the topic of my own dissertation), is not a static box into which texts may be placed, but instead a network of associations more in line with Jauss’s “horizon of expectations”, which emphasizes the reader’s prior knowledge and the interrelated nature of features in literary works. There is little acknowledgement in Jockers’s book that genres possess an internal structure or that this structure is not accessible via machine-learning classification methods as currently deployed by digital humanists.

This objection returns us to the core question about the book’s audience and purpose. I have seen several people on Twitter note that they could use this book as something like a textbook for a digital humanities course. And, as Worthey rightly noted, Jockers does a masterful job of leading potential skeptics “by the nose” from simple, seemingly straight-forward (though still novel) analyses to the more arcane world of network graphs derived from topic models and stylometrics. I suspect that this book has in mind at least both the audience of skeptics and the already converted who want to know precisely what algorithms he used to derive his findings.

The book prompted an invigorating conversation that touched on these and other issues that, as digital humanities scholars, we should engage with regularly and critically. Our next discussion will focus on the collaboratively-authored 10 PRINT, a very different kind of book. If you are in the Bay Area and free that day, we invite you to join our discussion.

Attendees: Cameron BlevinsJason HepplerJacqueline HettelKarl GrossnerMichael WidnerGlen Worthey

Reviews of Macroanalysis:

Mike Kestemont, LLC, http://llc.oxfordjournals.org/content/early/2013/08/22/llc.fqt056.full

Scott McLemee, IHE: http://www.insidehighered.com/views/2013/05/01/review-matthew-l-jockers-macroanalysis-digital-methods-literary-history

Scott Weingart: http://www.scottbot.net/HIAL/?p=34566

Matthew Wilkins, LA Review of Books: http://lareviewofbooks.org/review/an-impossible-number-of-books/

Comments

Enjoyed this post; it sounds like a fabulous interdisciplinary reading group, with a mixture of talents that ensured y'all could hold Macroanalysis to a very high standard.

I'm not going to try weighing in on any of the fascinating questions here; they're good questions, and I haven't made my mind up yet about most of them. But I just want to briefly bring out something that's implicit here -- and in Matt Wilkens' excellent review in LARB -- which is that Jockers has raised the bar for this whole area of discussion. In fact, "raised the bar" might be an understatement, because five years ago I don't recall this particular bar existing.

E.g., I wouldn't have known that topic modeling existed if it weren't for Matt. And more broadly, it's worth pausing to remark the novelty of the whole notion that literary historians are going to have to have a conversation about corpus linguistics, computational linguistic models, and different theories of statistics. Those are absolutely appropriate standards for this book. But they're not standards that, four years ago, I could have imagined we'd be discussing.

To indulge a silly metaphor, it strikes me that the reception of Macroanalysis is sort of analogous to a sonic boom, where the speed of an object's own motion creates a wave that builds up right ahead of it and becomes the resistance it has to overcome. I'm not sure that my physics is quite accurate there, but you see what I mean.

 

Agreed, which is why I would use it as the central book in a text-analysis course if I were to teach one. I loved the book for how it provides a model for this type of work in what, as far as I can tell, is an entirely new way; I haven't seen anything on this topic that even approaches the level of detail, sophistication, and clarity that Matt provides. The way the book edges up to so many disciplines is one of its strongest points, in my opinion, but also one that can open it up to these sorts of critiques. Or, to frame it differently, the book's importance also rests in the way it can force us to consider all these different disciplines, to show how they are increasingly connected. To use a different metaphor, it's as if the book wrenches open a door that was, previously, only barely ajar.

 

I guess I'm curious as to the nature of the objection Hettel makes.  Is it that Jockers is not laying bare his assumptions, or that his assumptions are incorrect?  I also wonder at the contention that Chomskian linguistics and NLP somehow constitute a united set of "American methods."  Chomsky has repeatedly criticized NLP--and the use of statistical methods in general--as constituting only a weak "engineering success."

 

 

Stephen, while I have great respect for what Jockers is doing with this book (in fact, I was constantly finding myself saying, "Yes, exactly!" through the end of Chapter 3, I would say that my objection is a mixed bag of what you describe. Yes, I do wish that he would make explicit his assumptions, while also acknowledging that there is an alternative text analysis methodology with its own statistical implications. And yes, personally, I think that the language he is analyzing would benefit from thsoe alternative approaches. Furthermore, his economics metaphor is much more in line with the perspective of language behaving as a complex system and that language production behaves logarithmically.

Also, regarding Mike's paraphrase of Chomskian linguistics and NLP as being "American Methods," that was something presented in opposition to the London School. And yes, while Chomsky has repeatedly criticized NLP, NLP's approaches are rooted in generative linguistic assumptions. 

Despite all of my "objections," I think that this book is having a positive impact. Primarily, Matt's book has enabled us to start having these conversations and to realize that there are different ways to perform this kind of litereary analysis.

I wanted to clarify (or actually, to amplify) what Mike W. writes about our truly outstanding discussion of Macroanalysis: what struck me wasn't so much that the debate was critical in some overwhelmingly negative way, but rather that that it was rigorous, lively, engaging, and definitely sustainable for our 90-minute discussion (and more).  It could have been just a boring old lovefest for our good old friend's great new book.  But we went way, way beyond that.

I absolutely loved Matt's book (which I had also read in draft form -- as someone snarked in the discussion, my favorite part was seeing my name in the acknowledgements! which I cannot deny!).  But as Mike points out, love for the book extended beyond the merely personal.  I should let the critics speak for themselves, but I can report that even those with the deepest objections seemed to have an immense admiration for the book: for its tone, its sense of mission, its "raising the bar" of discussion (or maybe even revealing the bar to us), as Underwood and Wilkins write.  

My sense was that the criticisms were meant as something along the lines of: it would be a shame if such an important book's reception were to suffer because some  strident Firthian shot holes in some inherent statistical or linguistic assumption that had gone unacknowledged.  (Again, I won't attempt to respond to Ramsay's great question, or to paraphrase the critiques, which are rather beyond my ken -- although I think Mike does an outstanding job of that here.)

I also wanted to elaborate on something that Mike hints at, and which strikes me as an extremely important contribution of Macroanalysis: its stunning combination of new (and newish) methodologies and practices (still very much up for debate, criticism, clarification, reification, etc.) with real, deep literary-critical and literary-historical knowledge.  (I'm not saying we lack such people in the DH community -- on the contrary! -- but only that Matt in this book is a particularly good example.)  

To go beyond the obvious examples of Matt's particular expertise in Irish and Irish-American lit, I thought his embrace and reading of the Russian Formalists through a DH lens (and vice versa) was remarkably sophisticated, and really without compare in our literature.  (In previous DH work there has of course been a little Russian Formalist name-checking here and there, but nothing like this.)  Matt and I have talked about this topic quite a lot over the years, so I surely have a skewed and personally-inflected view -- but I continue to find this aspect of his work a true inspiration.

Finally, not to boast or anything -- but it is such a privilege to be in the same room with these colleagues (many of them still very new to the profession), talking about the work of one of very own!  Only about half of us Stanford DHers were able to make it to this particular discussion; I'm sure things would have been even livelier with more.  But it was great, great, great.  

Oh, and we'll keep doing this.

I'm intrigued by the division within linguistics that Hettel describes; if there were a link to a blog post or something like that explaining it, I'd be interested in following up and learning more.

My initial impression (which could be wrong) is that we're looking at a tension that isn't purely within linguistics, but also involves disciplines like machine learning.

And I don't think this tension is at all unique to Macroanalysis; it's a broader story about the history of humanistic text mining. I first got into text mining through MONK, which I suspect was heavily shaped by corpus linguistics; log-likelihood was at that point our default way of assessing relative overrepresentation and identifying the characteristic vocabulary of a particular corpus.

But in the last five years I've seen people relying more and more on machine learning methods (e.g. training classification models and evaluating the utility of words as features in those models, which is what happens ultimately in Chapter 7 of Macroanalysis.)I've also been moving in that direction myself, although I still read corpus linguistics. It's helpful to be reminded that this could be a controversial move; that's a tension I'll want to acknowledge. But I think it's vital that humanists not be timid here. Often we're going to have to take a stand in debates between other disciplines. There's just no way for us to say, "well, I'm not a linguist or a statistician, so I'll outsource these questions to the experts." There are always multiple disagreeing experts, so ultimately it's on us to decide which methods are more useful in our research.