Book Report: Weapons of Math Destruction

“Predictive models are, increasingly, the tools we will be relying on to run our institutions, deploy our resources, and manage our lives. But … these models are constructed not just from data but from the choices we make about which data to pay attention to – and which to leave out. These choices are not just about logistics, profits, and efficiency. They are fundamentally moral. If we back away from them and treat mathematical models as a neutral and inevitable force, like the weather or the tide, we abdicate our responsibility.” (Weapons of Math Destruction)

Title: Weapons of Math Destruction – How Big Data Increases Inequality and Threatens Democracy

Authors: Cathy O’Neil

Publisher: Crown

Publication Date: 2016

Origin: I was interested in reading Weapons of Math Destruction for two main reasons: first, I’ve worked with biggish data, and might do so again, so I want to make sure I do so responsibly; second, I’ve been learning about hidden factors that influence our lives, and wanted a more fully developed understanding of the role of big data- and machine learning-enabled algorithms.

Summary: Let’s start with a quote from p13: “Big Data has plenty of evangelists, but I’m not one of them. This book will focus sharply in the other direction, on the damage inflicted by WMDs [Weapons of Math Destruction] and the injustice they perpetuate. We will explore harmful examples that affect people at critical life moments: going to college, borrowing money, getting sentenced to prison, finding and holding a job. All of these life domains are increasingly controlled by secret models wielding arbitrary punishments.”

“Big Data has plenty of evangelists, but I’m not one of them. This book will focus sharply in the other direction, on the damage inflicted by WMDs [Weapons of Math Destruction] and the injustice they perpetuate.”

O’Neil begins her exploration of WMDs with an Introduction that sets the stage and begins to show how WMDs create, contribute to, and exacerbate social injustice: “The privileged, we’ll see time and again, are processed more by people, the masses by machines.“

But why is this dichotomy bad?

Next, in Bomb Parts: What is a Model?, O’Neil touches on (lack of) model interpretability in WMDs and introduces their three elements: opacity, scale, and damage.

The next chapter, Shell Shocked: My Journey of Disillusionment, describes her experiences building and observing WMDs in the financial sector, during which time she saw their dark side and realized the social dangers that they create.

From there, O’Neil shows how WMDs impact our lives, often in subtle and nefarious ways, using a range of examples:

• Arms Race: Going to College
• Propaganda Machine: Online Advertising
• Civilian Casualties: Justice in the Age of Big Data
• Ineligible to Serve: Getting a Job
• Sweating Bullets: On the Job
• Collateral Damage: Landing Credit
• No Safe Zone: Getting Insurance
• The Targeted Citizen: Civic Life

Finally, O’Neil brings everything together into a Conclusion that implores us to act responsibly: “Big Data processes codify the past. They do not invent the future. Doing that requires moral imagination, and that’s something only humans can provide. We have to explicitly embed better values into our algorithms, creating Big Data models that follow our ethical lead. Sometimes that will mean putting fairness ahead of profit.”

My Take: I was saddened by Weapons of Math Destruction, but was neither surprised nor shocked. I’ve worked with big data and sensitive information professionally, and I’d like to think I’m more aware than most of social problems, so the journey that O’Neil took me on wasn’t especially mind-blowing.

That being said, it was sufficiently rich in detail and diversity as to open my eyes wider to the pervasiveness of WMDs and the risks that they pose. So while I wasn’t surprised or shocked, I’ve definitely gained a greater understanding of the issues and a more informed appreciation of just how pernicious things have already become.

Wherever possible, I’ve tried to spread the word about Weapons of Math Destruction – both about the book specifically and about the concept in general.

Wherever possible, I’ve tried to spread the word about Weapons of Math Destruction – both about the book specifically and about the concept in general. Shortly after completing it, I was in attendance at a local Competitive & Market Intelligence peer-to-peer group and the topic veered towards big data; I wasted no time in promoting O’Neil’s book and message, and I was happy to see more than a few folks write down the title.

More recently, at the Deloitte TMT Predictions session, machine learning was discussed. During the discussion, the speaker (Deloitte’s Duncan Stewart) showed this image:

The point was to illustrate the risk of over-reliance on algorithms to make our decisions. Duncan asked the audience if anyone knew why such a reliance was a bad thing.

Having read Weapons of Math Destruction, and after looking around to see if anyone else wanted to answer, I stuck up my hand and (when prompted) briefly explained that these algorithms codify biases. For this contribution, Duncan verbally awarded me a gold star. Huzzah!

Why am I so eager to promote Weapons of Math Destruction? Because I think it raises an issue that is vitally important to the future of our societies.

Quoting O’Neil, from p118: “We’ve seen time and again that mathematical models can sift through data to locate people who are likely to face great challenges, whether from crime, poverty, or education. It’s up to society whether to use that intelligence to reject and punish them – or to reach out to them with the resources they need. We can use the scale and efficiency that make WMDs so pernicious in order to help people. It all depends on the objective we choose.”

So what objective will we choose?

Read This Book If: …You work with data of any reasonable magnitude, or algorithms that rely on data (personally, I think we’ve got a professional obligation to do so); or, simply, if you want a glimpse into how opaque algorithms are coming to exert enormous (and often very negative) influence on our lives as individuals and on society as a whole.

Notes and Quotes

Introduction

“The privileged, we’ll see time and again, are processed more by people, the masses by machines.“

• p6 outlines some key factors for systems to work (e.g., large sample sizes, representative samples, feedback loops, etc.): “Indeed, if we were to analyze teachers with the statistical rigor of a search engine, we’d have to test them on thousands or even millions of randomly selected students. Statisticians count on large numbers to balance out exceptions and anomalies. (And WMDs, as we’ll see, often punish individuals who happen to be the exception. Equally important, statistical systems require feedback – something to tell them when they’re off track.”
• p8 starts to touch on why inequality is an output of these systems: “The privileged, we’ll see time and again, are processed more by people, the masses by machines.“
• p13: note to self

Bomb Parts: What Is a Model?

“Models, despite their reputation for impartiality, reflect goals and ideology.”

• p21: “Models, despite their reputation for impartiality, reflect goals and ideology.”
• p28: “The first question: Even if the participant is aware of being modeled, or what the model is used for, is the model opaque, or even invisible?”
• p29: “WMDs are, by design, inscrutable black boxes. That makes it extra hard to definitively answer the second question: Does the model work against the subject’s interest? In short, is it unfair? Does it damage or destroy lives?”
• p29: “The third question is whether a model has the capacity to grow exponentially. As a statistician would put it, can it scale? This might sound like the nerdy quibble of a mathematician. But scale is what turns WMDs from local nuisances into tsunami forces, ones that define and delimit our lives. As we’ll see, the developing WMDs in human resources, health, and banking, just to name a few, are quickly establishing broad norms that exert upon us something very close to the power of law.”
• p31: “So, to sum up, these are the three elements of a WMD: Opacity, Scale ,and Damage.”

Shell Shocked: My Journey of Disillusionment

“I saw a growing dystopia, with inequality rising. The algorithms would make sure that those deemed losers would remain that way. A lucky minority would gain ever more control over the data economy, raking in outrageous fortunes and convincing themselves all the while that they deserved it.”

• p38, reminiscent of Taleb’s Black Swans: “We could crunch numbers with the best of the best. But what if the frightening tomorrow on the horizon didn’t resemble any of the yesterdays? What if it was something entirely new and different? That was a concern, because mathematical models, by their nature, are based on the past, and on the assumption that patterns will repeat.”
• p47, reminiscent of Smart People Should Build Things: “In fact, I saw all kinds of parallels between finance and Big Data. Both industries gobble up the same pool of talent, much of it from elite universities like MIT, Princeton, or Stanford. These new hires are ravenous for success and have been focused on external metrics – like SAT scores and college admissions – their entire lives. Whether in finance or tech, the message they’ve received is that they will be rich, that they will run the world.”
• p48, sounds like me (but I’ve been called worse)! “Those who objected were regarded as nostalgic Luddites.”
• p48 reminded me of Success and Luck, and – in particular – how it talks about successful people not even being aware of the winds that were at their back: “I saw a growing dystopia, with inequality rising. The algorithms would make sure that those deemed losers would remain that way. A lucky minority would gain ever more control over the data economy, raking in outrageous fortunes and convincing themselves all the while that they deserved it.”

Arms Race: Going to College

“Proxies are easier to manipulate than the complicated reality they represent.”

• p55: “When you create a model from proxies, it is far simpler for people to game it. This is because proxies are easier to manipulate than the complicated reality they represent.”
• p55, in the next paragraph: “As people game the system, the proxy loses its effectiveness. Cheaters wind up as false positives.”
• p65: “The result is an education system that favors the privileged. It tilts against needy students, locking out the great majority of them – and pushing them down a path toward poverty. It deepens the social divide.”

Propaganda Machine: Online Advertising

“Vulnerability is worth gold. It always has been.”

• p70: “Anywhere you find the combination of great need and ignorance, you’ll likely see predatory ads.”
• p72: “Vulnerability is worth gold. It always has been.”
• p72: “Once the ignorance is established, the key for the recruiter, just as for the snake-oil merchant, is to locate the most vulnerable people and use their private information against them. This involves finding where they suffer most, which is known as the ‘pain point.'”

Civilian Casualties: Justice in the Age of Big Data

“The result is that we criminalize poverty, believing all the while that our tools are not only scientific, but fair.”

• p89 hits on a common misconception: “More data, it’s easy to believe, is better data.”
• p91, on the impact of crime prediction algorithms and other related approaches: “The result is that we criminalize poverty, believing all the while that our tools are not only scientific, but fair.”

Ineligible to Serve: Getting a Job

“We can use the scale and efficiency that make WMDs so pernicious in order to help people. It all depends on the objective we choose.“

• Neat! From page 113: “The ideal way to circumvent such prejudice is to consider applicants blindly. Orchestras, which had long been dominated by men, famously started in the 1970s to hold auditions with the musician hidden behind a sheet… Since then, the percentage of women playing in major orchestras has leapt by a factor of five.”
• p118 make a crucially important point. Will it be profit or some greater good? “We’ve seen time and again that mathematical models can sift through data to locate people who are likely to face great challenges, whether from crime, poverty, or education. It’s up to society whether to use that intelligence to reject and punish them – or to reach out to them with the resources they need. We can use the scale and efficiency that make WMDs so pernicious in order to help people. It all depends on the objective we choose.”

Sweating Bullets: On the Job

“But eventually, an industry standard will emerge, and then we’ll all be in trouble.”

• p134, alluding to fragility and locked-in bias: “Throughout the tech industry, many companies are busy trying to optimize their white collar workers by looking at the patterns of their communications. The tech giants, including Google, Facebook, Amazon, IBM, and many others, are hot on this trail. For now, at least, this diversity is welcome. It holds out the hope, at least, that workers rejected by one model might be appreciated by another. But eventually, an industry standard will emerge, and then we’ll all be in trouble.”

Collateral Damage: Landing Credit

“As e-scores pollute the sphere of finance, opportunities dim for the have-nots.”

• p159: “And what does that mean for us? With the relentless growth of e-scores, we’re batched and bucketed according to secret formulas, some of them fed by portfolios loaded with error. We’re viewed not as individuals but as members of tribes, and we’re stuck with that designation. As e-scores pollute the sphere of finance, opportunities dim for the have-nots.”

No Safe Zone: Getting Insurance

“Nearly a half century later, however, redlining is still with us, though in far more subtle forms.”

• p162, after describing redlining: “Nearly a half century later, however, redlining is still with us, though in far more subtle forms. It’s coded into the latest generation of WMDs… They punish the poor, and especially racial and ethnic minorities. And they back up their analysis with reams of statistics, which give them the studied air of evenhanded science.”

The Targeted Citizen: Civic Life

“By tweaking its algorithm and molding the news we see, can Facebook game the political system?”

• p180, eerily prescient (or perhaps it was just obvious, informed foresight) given what we now know about how Facebook was weaponized during the United States’ 2016 presidential election: “Nearly half of [American adults] count on Facebook to deliver at least some of their news, which leads to the question: By tweaking its algorithm and molding the news we see, can Facebook game the political system?”

Conclusion

“We have to explicitly embed better values into our algorithms, creating Big Data models that follow our ethical lead. Sometimes that will mean putting fairness ahead of profit.”

• Again we see (if our eyes are open) how the skewed perception described in Success and Luck naturally arises, p200: “The quiet and personal nature of this targeting keeps society’s winners from seeing how the very same models are destroying lives, sometimes just a few blocks away.”
• p204, with a crucial point and an important consequence: “Big Data processes codify the past. They do not invent the future. Doing that requires moral imagination, and that’s something only humans can provide. We have to explicitly embed better values into our algorithms, creating Big Data models that follow our ethical lead. Sometimes that will mean putting fairness ahead of profit.”
• p218: “Data is not going away. Nor are computers – much less mathematics. Predictive models are, increasingly, the tools we will be relying on to run our institutions, deploy our resources, and manage our lives. But as I’ve tried to show throughout this book, these models are constructed not just from data but from the choices we make about which data to pay attention to – and which to leave out. These choices are not just about logistics, profits, and efficiency. They are fundamentally moral. If we back away from them and treat mathematical models as a neutral and inevitable force, like the weather or the tide, we abdicate our responsibility.”