The Comprehensible Artificial Intelligence project group is a collaboration between the University of Bamberg and Fraunhofer IIS. The group researches the potential of explainable machine learning. Why was the group founded in the first place? What are the practical applications of your findings?
Prof. Schmid: Explainable AI refers to methods that allow to make the decisions of AI systems transparent and explainable. This applies in particular to systems based on machine learning. Complex neural networks can be opaque and their decisions difficult to explain, even for the developers themselves.
Currently, extensive research is being conducted on different methods to highlight and visualize the key aspects of the input data that influenced the neural network’s decision. Take, for example, a diagnosis based on a medical image. The developer can use visualizations to determine whether the learned model is accurate or overfitted, for example, because it is correlating arelevant attribute with an irrelevant attribute, such as the background color. Users – in this case, medical professionals – are more likely to need explanations of the diagnosis itself, and these types of explanations are often best expressed verbally. In the medical example, the diagnostic decision could be explained in a way which relates to the input data, for example, that a particular type of tumor was identified due to the location and type of tissue affected.
Dominik Seuss: Right now, we’re working on pain recognition, for example. There isn’t much data available for this use case, so neural networks tend to identify false correlations in the data. In the worst cases, they simply try to “memorize” the training data. So, we use visualization techniques, or techniques such as layer-wise relevance propagation and others, that show us which parts of the input image had the greatest impact on the neural network’s decision. In the next stage, we can integrate existing knowledge from psychology experts into the network modelling process. We can do that by building targeted restrictions into the learning process. This means that we prevent the network from learning potential correlations that are physiologically impossible. This allows us to produce robust models, even when we only have limited data available.
Prof. Schmid: Of course, explainability is not only relevant to medical diagnostics; it is also critical for all kinds of fields in which decisions can have serious consequences, for example when it comes to controlling production processes or mobility and autonomous vehicles.