In technology, the ingredients are data, the instructions are algorithms, and the meal is a decision. But unlike a meal, algorithms recognize patterns to base future decisions. For example, Discover Weekly playlists on Spotify use algorithms to recognize the patterns of music each user prefers to then make a decision in the form of a personalized playlist based on this data.
But as the world continues to digitize, algorithms are employed to make more and more far-reaching decisions, oftentimes using inaccurate, biased, or incomplete data. For example, police departments use predictive software to more efficiently identify crime, but because of bias and incomplete data, an algorithm that was supposedly “race-neutral” targeted black neighborhoods at twice the rate of white neighborhoods. In another example, the Department for Work and Pensions (DWP) in the United Kingdom, the body responsible for providing disability benefits, used corrupt and biased data that led to thousands of disabled citizens losing coverage and £100 million spent on appeals.
Because algorithms operate by collecting data and recognizing patterns on which future programming decisions will be based, it is essential to understand who and what show up more commonly, and the integrity of the data. For example, having fewer women or people of color in a database means algorithms based on those databases categorize these populations as “less likely” or “riskier.”
Consequently, algorithms do not only mirror perceived realities but can also amplify them. In a deeply unequal world, the consequences can be severe. For example, Dr. Galdón-Clavell explained that if a woman applies for a mortgage, she is understood by the training data as a riskier client because women historically have less access to banking systems and therefore less representation and history as loan recipients in the database.
