When researchers decide what to use as a counterfactual they are making a decision about?

Andy Lo Pò

Life-changing decisions are happening in the dark. Machine-learning algorithms now determine decisions from loan applications to cancer diagnoses. In France, they place children in schools. In the US, they determine prison sentences. They can set credit scores and insurance rates, and decide the fate of job candidates and university applicants.

But these programs are often unaccountable. To arrive at their decisions, machine-learning algorithms automatically build complex models based on big data sets, so that even the people using them may not be able to explain why or how a particular conclusion is reached. They’re a black box.

The AI researcher Sandra Wachter is working to drag them into the light. The 32-year-old is a research fellow at both the Oxford Internet Institute and the Alan Turing Institute in London. She trained as a lawyer in her native Austria and attributes her interest in technology to her grandmother, one of the first three women admitted to a technical university in the country. “She shaped my thinking about technology as something that was powerful and interesting, and could be used for good,” she says.

Now Wachter works on the legal and ethical implications of AI, machine learning and robotics. She acts as a link between the makers of technology and the judges and policymakers who will create a legal framework for it. Her work is about “striking a fair balance”, she says, and figuring out how we can reap the benefits of technology without jeopardising our human rights and privacy.

Wachter believes we should have the legal right to know why algorithms come to specific decisions about us. But there’s a clash, as software owners often claim that increasing transparency could risk revealing their intellectual property, or help individuals find loopholes to game the system. Wachter, along with her colleagues Brent Mittelstadt and Chris Russell, has come up with a compromise: counterfactual explanations.

Counterfactuals are statements of how the world would need to be different in order for a different outcome to occur: if you earned £10,000 a year more, you would have got the mortgage; if you had a slightly better degree, you would have got the job.

“If I don’t get a loan, I don’t necessarily care so much how the algorithm works. I actually just want to know why I didn’t get the loan and have some guidance on how to improve,” Wachter says. Counterfactual explanations can provide this by finding the smallest possible change that would have led to the model predicting a different outcome.

Crucially, counterfactual explanations give answers about why a decision was made without revealing the guts of the algorithm. The black box stays sealed. “It finds the sweet spot between meaningful information and protecting intellectual property rights and trade secrets,” Wachter says.

In September 2018 Google implemented Wachter’s suggestion in What-If, a new feature of its open-source machine-learning web application TensorBoard. What-If allows anyone to analyse a machine-learning model and create counterfactual explanations for its outcomes – no coding required.

When algorithms get it wrong

Google, 2018: Researchers at Cornell University found that setting a user’s gender to female resulted in them being served fewer ads for high-paying jobs.

Durham police force, 2017: An algorithm to predict reoffending was rolled back due to concerns that it discriminated against people from certain areas.

China, ongoing: The state monitors many aspects of an individual’s life – such as employment and hobbies – to give a score based on “trustworthiness”.

Chicago Police, 2013-2017: A project to identify the risk of being involved in a shooting labelled people as previous offenders based on where they lived.

Google, 2013: A study showed searches with “black sounding” names were more likely to turn up ads for services such as criminal background checks.

As well as shedding light on specific decisions, counterfactual explanations can be used to test algorithms for fairness, for example by checking whether changing someone’s race or gender affects a decision (which would be illegal). “It’s a major step forward in terms of transparency and accountability,” Wachter says.

She warns, however, that transparency in the decision-making process is “only one side of the coin of accountability”. “It gives you some idea why a decision has been made a particular way, but it doesn’t mean the decision is justified or legitimate,” she says.

One problem is a lack of transparency around what data is being used to train decision-making algorithms in the first place. The payday lender Wonga claimed to use more than 7,000 data points to assess how likely applicants were to default on a loan. In China, social credit scores are based on a person’s online behaviour and their network of friends.

If algorithms are building assumptions about us based on variables we’re not even aware of, “it becomes a human rights problem”, Wachter says. We might expect to be turned down for a job based on our employment history, but not because of the newspapers we like on Facebook. She says companies should be required to demonstrate that the data points they use to make decisions are both relevant and lead to accurate predictions.

Wachter’s latest work focuses on the “right to reasonable inferences”. Under GDPR, the sweeping European data protection regulations implemented in 2018, everyone in the European Union has the right to access and amend any data a company holds about them. We also have the “right to be forgotten” – the ability to request that online search results about us are removed. But Wachter argues that, as well as access to our data, we should be able to ask for and correct any assumptions that companies have made about us based upon its contents. “It’s the logical counterpart to the right to be forgotten,” she says. “You also have a right on how to be seen.”

More great stories from WIRED

– Inside the dark web's biggest hitman-for-hire website

– How the London Underground will be carbon neutral by 2050

– An energy giant will offer all-you-can-eat electricity in 2019

– The everyday things you can do to prevent climate change

– Inside the British Army's secret information warfare machine

Get the best of WIRED in your inbox every Saturday with the WIRED Weekender newsletter

What research design creates a situation where the subjects serve as their own controls?

What is required for a research study to be considered a true experiment quizlet?

Which type of quantitative study provides the strongest evidence for causal inference?

Which of the following is a key characteristic of a research design that allows you to make a causal inference?