Explainable and Reliable Artificial Intelligence (ERAI)

Recent high-profile successes in Machine Learning have mostly been achieved using deep neural networks, which yield ‘black-box’ input-output mappings that can be challenging to explain to users. Especially in the medical, military and legal fields, black-box machine-learning techniques are unacceptable, since decisions may have a profound impact on peoples’ lives. As reported in recent news, AI may amplify the biases that already pervade our society without strict ethical standards, unbiased data collection, and algorithmic bias considerations.

Research focus and application

Research within the ERAI theme Investigates different ways to make intelligent systems better explainable and more reliable. Some of the research foci are:

• Analyzing whether the input-output relations of a system can provide high-level correlationbased explanations of the black-box AI system.
• Logic-based systems can provide explanations and are able to reason with (legal) regulations that should be adhered. Integrating logic-based approaches with machine learning approaches is one possible way to realize explainable artificial intelligence, and is an important challenge for the near future.
• Learning explainable models instead of learning a mapping may improve explainability with minor or no decrease in the quality of predictions / decisions / recommendations. • Improving the understanding of deep neural networks. Especially w.r.t. the influence of input and intermediate layers on the outputs, information flow through the network, detecting changes that can make the system collapse and determining the tolerance of the system.
• Using machine learning and causal inference to explain and understand the relationship between variables in high-dimensional and complex data.
• Developing model-checking tools for properties of artificial intelligence and control systems in safety-critical engineering, medical interventions, and legal compliance.
• Mediating effects of individual characteristics on explanation effectiveness. How to adapt explanations to user specific factors that may influence the effectiveness of explanations. E.g. how individual user characteristics (e.g., cognitive abilities, expertise) influence the effectiveness of interactive explanation interfaces.
• New evaluation criteria and methodologies. It is also important to study the effects of explanations over time. However, decision support and recommender systems research to date has primarily studied the impact of recommending individual items, or conducted analysis based on historical data. E.g. using simulations to model interactions with recommendations, or the effects of different algorithms and algorithm parameters.
• Explanation interfaces to deal with biases. Human decision-making is rarely completely rational and suffers from biases, such as confirmation or availability biases. These biases can be considered in the design of explanations interfaces. For example, automatically generated explanations can be used to support credibility assessments, supporting people who are having trouble making assessments about controversial content they find online.
• Explanations for groups. Explanations can also be helpful in situations where it is difficult to reach an agreement, e.g., when people have diverging preferences in a group of tourists. This also raises interesting questions about the tension between privacy and transparency of explanations in group decision making.
• Making reliable prediction with confidence bounds on the error.