With this post we continue our reading of Cathy O’Neil’s Weapons of Math Destruction. (If you’d like to catch up with the reading schedule, click here. All posts for this reading, including the schedule one, are grouped here.)
Here I’ll summarize this week’s chapters, then offer some discussion questions.
But first, some book club business. Last week’s reading elicited some fine comments on that post (scroll down). Elsewhere regular book club reader Tom Haymes blogged this meditation on the human role in data analytics, starting with a local baseball example. Mike Richichi posted about last week’s reading, thinking about education and data.
Beyond the book club, Slate has a good article sketching out the recent history of disillusionment with and criticism of big data. And I just finished Shattered, an account of the 2016 Clinton presidential campaign, which argues that overreliance on data models played a key role in that defeat (here’s my review).
Onward!
Chapter 2, “Shell Shocked: My Journey of Disillusionment”
This section dives into the financial world in search of WMDs (the titular weapons of math destruction), while continuing the book’s autobiographical theme.
Finance: O’Neil explains how hedge funds like the one she worked for use math to make money. This includes applying historical data models to the present and future. It also involves working with ever more complex derivatives and securities, which lead up to the 2008 financial meltdown. Along the way an important distinction appears:
[T]he subprime mortgages that piled up during the housing boom… were not WMDs. They were financial instruments, not models…
But when banks starting loading mortgages… into classes of securities and selling them, they were relying on flawed mathematical models to do it. The risk model attached to mortgage-backed securities was a WMD. [emphases added] (40-41)
While we can find many problems with the financial strategies that led to the 2008 crisis, O’Neil focuses on one, with regard to data models: scale. (42)
One final point: the chapter points out that the people running these financial models, as well as the financial houses, tend to be drawn from economic and/or intellectual elites, including “elite universities like MIT, Princeton, or Stanford”, and become eager devotees of economic Darwinism. They are filled with confidence and marked by rewards, which confirms to them the virtue of their successes. However,. “it looks very much to the outside like a combination of gaming a system and dumb luck.” (47-8)
Autobiography: we follow O’Neil as she moves from academia to a hedge fund, then, disillusioned, takes up other positions to try to understand, and do something about, WMDs.
Chapter 3, “Arms Race: Going to College”
Here the book shifts to college ranking systems, and how they have changed the world of higher education. The chapter focuses on the famous and notorious U.S. News & World Report ranking service, deeming it to be “a bona fide WMD.” (54)
O’Neil criticizes the service for incentivizing gaming the system (54, 62, 66), for not incorporating costs (59), for relying on proxies (52-3), and for its scale (“[i]t forces everyone to shoot for exactly the same goials, which creates a rat race”, 58). The desirability of gaming the system encourages unethical behavior (62). She also draws attention to the “Flutie effect” whereby a successful campus athlete or team boosts student applications. (57)
This chapter also faults US News for taking advantage of escalating social anxiety about academics and economic status, “fe[eding] on these beliefs, fears, and neuroses” (60). I’m reminded of Tressie Cottom’s theory of the “education gospel“. The success of this WMD – growing a major audience, altering institutional behavior – has also led to spinoff businesses (64).
There are two more points which end the chapter. First, in terms of economic inequality, the author points out that the intensity of such a WMD rewards wealthy families, and punishes “the vast majority of Americans, the poor and middle-class families who don’t have thousands of dollars to spend on courses and consultants.” This speeds another feedback, loop, whereby the former win benefits which solidify their position, while opening wider the gap with the latter. (65)
Second, O’Neil reminds us that the Obama administration’s drive to produce an alternative scorecard flopped, partly due to strong resistance from higher ed. The result is an anti-WMD, one with open data, available to outside queries, and capable of individualization. “Think of it: transparent, controlled by the user, and personal. You might call it the opposite of a WMD.” (67)
Questions
- If creating or running a WMD is so profitable, how can we push back against them?
- Do you find other university ranking schemes to be preferable to the US News one, either personally or within this book’s argument?
- At one point the author suggests that gaming the US News ranking might not be bad for a university, as “most of the proxies… reflect a school’s overall quality to some degree” (58). Do you agree?
Next up: for October 30, chapter 4, “Propaganda Machine: Online Advertising” and chapter 5, “Civilian Casualties: Justice in the Age of Big Data”.
First of all, trying to rank colleges and universities is like trying to rank symphonies (the music, not the orchestra) or renaissance paintings. I’ll give Mona Lisa a five out of seven but deductions for not telling a clear story of exactly why she’s smiling like that…
I think we tend to underestimate the importance of narratives when we look at WMDs. Sure, if you know how to play them you can make some money, but at the end of the day they are creating false narratives that feed on the Zeitgeist. If your social standing is dictated by what college you send your kids to, US News provides a useful proxy. If you think college is necessary to achieve a better life and Phoenix seems to be offering you a golden ticket, you ignore the fact that its fool’s gold. If you want to believe Obama was born in Kenya and was a secret Muslim, there’s algorithm on Facebook that will feed that narrative for you. If you think trading on the margins will make you rich, you will ignore the underlying fundamentals because the corporate narrative tells you to.
Understanding the mechanics of how this works is extremely useful and a necessary first step but until we start questioning those narratives we will always be open to exploitation and manipulation by the latest technology meant to shape our Weltanshauung (world view). Until we start recognizing that every legitimate college and university is filled wth people who are dedicated to bettering mankind through a deeper understanding of just how we fit into these narratives, we will be susceptible to narrative manipulation made more efficient and effective by technology (that’s what technology does). Until we give the good teachers in our world a more effective voice, we will continue to be subject to false narratives.
The narrative means a lot because there are very good and powerful algorithms that can save lives, enrich our experiences of the world, and help educate us. The difference between a good algorithm and a WMD is usually found in the narrative, not the technology.
Excellent point about narratives, Tom. (That’s where Chris Newfield is working.)
Sometimes data and story are opposed, or complementary.
I think you make a fine point but is this anything new? Hasn’t narrative always shaped our worldview? In our youth did we not get narratives that supported demonstrated biased accounts of reality? I presume that the access to more information and the ability of computers to apply grouping algorithms to that information may be clumsy (but improving), but overall is it not similar to what an editor at a newspaper or press secretary or professor used to posit as the truth?
With both Shattered and WMD there is this odd passivity thing going on. These things are happening all by themselves, and there is no real sense of motivation.
With WMD it feels like “that darn math!” and there is a constant apology of “oh, people probably had good intentions…” with no real exploration of if these effects are bugs or features. It would have been interesting exploring what groups are unaware of these effects versus what groups are aware but benefit from those same effects.
With Shattered that lament from HRC seems odd, given that at every turn during the campaign she was actively dismissive of so many of the people she assumed would support her…until even her campaign figured out that people meant it when they said she would not get their votes. In her world does she seriously believe people simply owed her their allegiance? It’s one thing to say she wailed about not understanding but I see precious little attempt on her part to ever try understanding.
Do you think O’Neil distributes blame among data experts (scientists, programmers), enterprise managers (police chiefs), and the very wealthy?
I don’t think O’Neil distributes nearly enough blame, and early on makes several statements that sound like apologetics: “well-intentioned people”, “despite good intentions”, etc. There is definitely a vibe that it hurts her to imagine that these things are features, not bugs. I’m not sure she believes anyone can be so evil as to want to grow those “side effects”. It seems pretty clear to me that these algorithms are doing precisely what they are designed to do in very efficient manners and are constantly tweaked to make their owners more and more profitable still.
I suspect her background in banking means it’s difficult for her to square that she was working for fantastically greedy people and that the people she may well have considered friends have some very nasty motives driving them.
Pingback: Book Club—“Weapons of Math Destruction”, Part 2 – Mike Richichi Dot Net
Reblogged this on As the Adjunctiverse Turns and commented:
As CEW progresses, don’t forget to keep up with all factors affecting higher education, academic labor, their future, and current, continuing influence on your workplace and working conditions
Anyone know about what has happened with universities or colleges that have opted out of high profile ranking systems like those of US News & World Report? The NYT had a piece in 2007: http://www.nytimes.com/2007/06/20/education/20colleges.html, and the Wikipedia mentions this opting out: https://en.wikipedia.org/wiki/U.S._News_%26_World_Report_Best_Colleges_Ranking#Criticism.
I’m not seeing many signs of it.
Pingback: _Weapons of Math Destruction_, part 3 | Bryan Alexander