Weapons of Math Destruction - Cathy O'Neil

Amazon link

Cathy O'Neil is a data scientist who taught before moving to DE Shaw as a quant, and started the Lede Program in Data Journalism at Columbia. Currently she writes for Bloomberg View, and her blog mathbabe; she also authored Doing Data Science (published by O'Reilly).

With a short afterword added in the 2017 edition, O'Neil writes an equally important, equally timely book for the next decade. She writes about the problems of algorithmic bias and how they "increase inequality and threaten democracy." She defines such problematic algorithms as Weapons of Math Destruction (WMD) which she characterizes as having 3 traits: (1) opacity — when people, including the programmers themselves, don’t know the inner workings of the WMD; (2) scale, and (3) damage.

Such models often lack feedback and so in this way "define their own reality, and use it justify their results." This lack of feedback is a huge flaw in predictive algorithms in that they suffer severely from confirmation bias a result: never knowing when their wrong, they continue to think that they're always right. This result is worsened by the possibility of feedback loops. For example, in the case of criminal behavior prediction: people deemed to be at high risk of such behavior get longer convictions, which in turn increase their chances of recidivism.

O'Neil paints a scary image of how such algorithms systematically target and exploit the poor, and as consequently drop them deeper into the poverty trap from which they already find it difficult to escape from.

In (college) education — which has seen a 500%  increase in cost between 1985-2013 or 4x inflation — services such as rightstudent can tell you which students are likely to either be able to pay full tuition or to require government assistance (which results in additional grant money for the school).

Vatterott College used this to their advantage. A 2012 Senate report detailed a recruiter manual's instructions to target "Recent Divorce. Low Self-Esteem. Experienced a Recent Death. Physically/Mentally Abused Recent Incarceration. Drug Rehabilitation. Dead-End Jobs- No Future" link. This targeting of the vulnerable pays off: Corinthian College (which filed C11 in 2015 after this came to light and federal funding was withdrawn) spent $120M on a 30-man marketing team which pursued 2.4m leads and generated $600M in revenue.

Meanwhile, schools also know that rankings matter, and that these rankings can be gamified. As a result, King Abdulaziz University paid academics to change their affiliation on the Thomson Reuters academic citations, link, while law schools offered $20/h temp jobs to new graduates to improve post-graduate employment rate, link.

Finally, these degree mills aren't even paying off for their graduates. Using fictitious degrees (n=9000), researchers found that fake job applicants from for-profit universities had a lower response rate than equivalent fake applicants from community colleges, link

This process: schools game ranking algorithms to look better >>> schools target the vulnerable with promises of social mobility >>> graduates leave with more debt and fewer prospects

In health and car insurance: nearly one dollar of every five we earn feeds the healthcare industry, link

Yet smart scheduling software and cost-averse companies have resulted in a gig economy where many employees are kept from working < 30 hours, thereby disqualifying them from company healthcare. And for those who do qualify, the industry uses them for data collection and algorithmic augmentation and penalizes them when they don't cooperate: somebody insured by Anthem Insurance who doesn't participate in the wellness program (for which he must accrue points by logging in, filling out surveys, etc.) would pay an extra $50/m; employees at Michelin who don't mean glucose, cholesterol, triglyceride targets pay an extra $1000/y; employees at CVS had to report body fat, blood sugar, blood pressure, and cholesterol, or pay $600/y, link

This has resulted in a world where "all it takes is a single accident or illness for them to miss a payment on a loan." which >>> lower credit score >>> higher living costs + harder to get a job >>> poverty trap

Car insurance: "in Florida adults with clean driving records and poor credit scores paid an average $1,552 more than the same drivers with excellent credit and a drunk driving conviction", link, and Allstate's pricing schemes had at least one hundred thousand pricing schemes, consequently some receive discounts of up to 90% off the average rate, while others face an increase of 800%,  link

In hiring practices: despite a meta analysis of various selection processes showing personality tests to be 1/3 as predictive as cognitive exams, which were ruled illegal in 1971 Griggs v Duke Power Company, personality tests are still used in some hiring practices: a law suit vs Kroger is still pending, tests administered by Kronos, link

Additionally, O'Neil talks about the risk to democracy that WMDs can pose: with a FB campaign estimated to have increased 2012 turnout by 340,000, link, the scoring and targeting of individual voters in swing states, only exacerbates the existing Electoral College system, and disenfranchises voters from every other state


O'Neil surmises correctly that the way going forward requires a deeper look at our priorities and values, and making sure our methods reflect these. For example, while our legal traditions (rightfully) prioritize fairness over efficiency, "from a modeler's perspective, the presumption of innocence is a constraint."

Dismantling the current paradigm of efficiency over fairness requires readdressing the way our society looks at profits, something Rutger Bregman covers excellently in Utopia for Realists (book notes, look at Salary =/= Societal Value, link).

She also correctly suggests that Data Scientists need a version of the medical profession's Hippocratic Oath (to do no harm). Models have blind spots, and these reflect the judgements and priorities of their creators, e.g. avionics models ignore streets, tunnels, buildings. "Models are opinions embedded in mathematics." It seems Bloomberg's Data Science team did fulfill this in 2017

Finally, she notes in her book that Google and Facebook are not yet WMDs (perhaps disagreeing only with respect to time with Franklin Foer and Scott Galloway in World Without Mind and The Four respectively), but definitely have the potential to become so. 73% of Americans believe search results to be accurate and impartial (Pew 2012, link); while 62% of Facebook users aren't aware of the new feed algorithm (Karrie Karahalios 2013 study, n=40, link)