Explainability has become a very hot issue
within the very hot domain of Artificial Intelligence (AI). Here is my
perspective on the issue. As usual, my views are based on a systems
engineering (and real world application) approach ... oh, and these are my own
personal views and don't represent those of my employer.
Most of the current excitement (I won't use
the word hype) around AI stems from the publicity surrounding Machine Learning
and, in particular, Deep Learning. Deep Learning algorithms are not "explainable"
and so ... with everyone assuming that AI is achieved by simply throwing data
at Deep Learning algorithms ... it is no surprise that there is a lot of
concern about explainability.
In reality, we need to think about systems and
not algorithms (I say that quite a lot).
Consider,
for example, a driverless car comprising multiple AI sub systems. There may be a vision system that
identifies and classifies vehicles. There
may be a laser range finder that calculates the distance to objects around the
vehicle. There may be a GPS
system that identifies the location of the vehicle and this system will be
supported by mapping data. There
may be a noise classification system that identifies the sound of sirens. I would expect all this data to be
fused together and fed into a decision engine and I would expect the output of
the decision engine to be explainable. I
would expect the system to state, "I
made the decision to pull over to the side of the road because the noise
classifier detected a siren approaching from behind, the vision system
identified an emergency vehicle approaching from behind, the road was straight
and it was a safe place to stop without causing an obstruction". Within this explanation there are
"classifications" that may have come from an AI sub system that
cannot fully explain it's reasoning. For
example, the vision system may not be able to explain how it identified an
emergency vehicle approaching. Whilst we may feel that is unacceptable, it is important to accept that there are many situations when we as human beings cannot explain own own image or sound
recognition. However, even when the classifier cannot explain its reasoning, it should be possible to design a Machine Learning system so that it
retrieves the training data that influenced its classification decisions. The vision system may not be able to
explain its classification but it should be able to say, "here are the closest matching
images in my training data".
Finally, remember that there are many different Machine Learning algorithms. Rule induction system such as C4.5 generate human readable rules. As such, these algorithms are able to explain their reasoning.
In
summary, whilst certain algorithms cannot explain their decision making,
overall systems should be engineered to be explainable and machine learning
systems should be able to support their decisions with training evidence.
