Along with the GDPR, the EU also bought "Right to Explanation" for algorithmic decision making. Basically, a user has the right to ask for an explanation on an algorithmic decision. This regulation has been mostly buried in the deluge of GDPR content but it's an important shift in decision making. There's a nice paper which outlines the implications and HN too had a discussion on it couple of years ago [0, 1].
back
2 comments
That's actually really interesting! I think it really is a good law, although it may render some things really difficult for engineers.
I remember when Alphago was beating the go master, we couldn't really tell why it made certain moves.
But it certain makes sense to require an explanation for things that affect us in our lives.
Yea, I think showing your work is substantially more important when people's lives and finances are on the line as opposed to the thrill of victory for a go master.
If you're making algos for that directly affect another human's basic needs, I think it's really important to be able to back up why the algo did what it did. I feel like you're probably opening yourself up to massive liability (regardless of region your operating in) if you can't explain to a court why your system did X to a person that resulted in them suing your company.
how does this work with neural networks when the "explanation" is going to be a list of weights?
You cannot then use such a network to make a questionable decision. In other words, they better be robust or you might drown in lawsuits or face a big fat one.
Which is the exact result people want. We want explanations that are valid and understandable in relatively plain language, using postulate logic.
It's not actually hard to come up with a valid and understandable explanation from a neural net, or from any other black box decision maker as long as the decision maker is consistent.
Let's take simplified case, to make visualization easier. Suppose the black box is deciding whether to approve mortgages, and it only has three inputs: income, credit score, and mortgage amount.
You can visualize the inputs as points in a 3-dimensional space, each point representing a unique combination of income, credit score, and mortgage amount.
Now imagine in that space we place green dots at all the points that represent inputs that result in approval of the mortgage application. Do nothing at the points for inputs that represent rejected inputs.
Need to give someone an explanation of their rejection? Find their point in that space, and then find points on the green region that are nearest their point, and tell them what would need to change to move their point to those green points.
I'd suggest, in this 3-dimensional space, to look at four green region points. (1) the nearest green point reached moving from their point parallel to the income axis, (2) same but using credit score axis, (3) same but using mortgage amount axis, and (4) the nearest green point.
In the context of a decision not made by an algorithm, would you find acceptable an explanation like "many repeated studies from well-known and reputed institutions and universities have concluded that X leads to Y, and therefore as a way of providing Y to our customers, we have chosen to do X"?
In case your answer is yes, would it be acceptable to use the track record of a particular network and similarly trained networks, and why or why not?