Systematic review of user experience and trust improvement evaluations on healthcare oriented explainable artificial intelligence
Iñaki Soto-Rey, Samantha Cramer
Abstract
Many different AI systems have been developed in recent years. The field of explainable AI is designed to shed light on the black box modles of the various AIs. However, the explainable AI is rarely developed in relation to the needs of the users, but is based on the technical side that the AI structure provides. Our goal is to explore the various scientific researches that have been conducted on explainable AI in terms of user satisfaction. This goal is to be achieved through a systematic review based on the PRISMA guidelines.
Before start
Before starting the Systematic Review we studied the PRISMA guidlines, in order to proceed and complete the Systematic Review according to these guidlines.
Steps
Background
The reviews subjective is to identify the amount of evaluations done on the usability, user satisfaction, user experience and trust of XAI. The domain of interest is the medical field.
Methodology - Literature Search
General search query: ("XAI" OR "X-AI" OR ("Explainable" AND "AI") OR ("Explainable" AND "artificial intelligence") AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND ("Evaluation")
Search restrictions: publication between 01.01.2017-31.12.2022
Data extraction from the following data bases on 02.05.2023, with the following results:
PubMed:
Query: ("XAI" OR "X-AI" OR ("Explainable" AND "AI") OR ("Explainable" AND "artificial intelligence")) AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND "Evaluation"
Formats saved: PubMed
Results: 17
PubMed Central (PMC):
Query: ("XAI" OR "X-AI" OR ("Explainable" AND "AI") OR ("Explainable" AND "artificial intelligence")) AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND "Evaluation"
Formats saved: MEDLINE
Results: 2996
Science Direct:
Query: ("XAI" OR "X-AI" OR ("Explainable AI") OR ("Explainable artificial intelligence")) AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND ("Evaluation")
Formats saved: RIS
Results: 1296
Google Scholar:
Website: Website:
using the Website for a quick overview and the Query: find article... with all words: “Evaluation” ; …and one of the words: “XAI” “X AI” “Explainable AI” “Explainable artificial intelligence”
Words contained in Title.
Results: 83
Problem: articles can’t be extracted from google scholar, therefor the Literature Search for this databank was done using the following software:
Software
| Value | Label |
|---|---|
| Harzing's Publish or Perish | NAME |
| Tarma Software Research Ltd | DEVELOPER |
| https://harzing.com/resources/publish-or-perish/ | LINK |
| 8.8.4275.8412 | VERSION |
Due to Publish and Perish set up extraction of max. 1000 results possible.
Publish or Perish search:
Query in Title words: (“XAI" OR “X-AI” OR “Explainable AI” OR "Explainable artificial intelligence”) AND (“Evaluation”)
Citation records included Result: t: 80 records
Formats saved: csv ; bibtex
filter out the results included in the last part of the query: AND (“Usability” OR “satisfaction” OR “experience” OR “trust”) by looking for the keywords in the .csv file
Results: 6
Extracted Literature was imported into Civati (version 6), excluding Duplicates during import process.
Software
| Value | Label |
|---|---|
| Citavi | NAME |
| Swiss Academic Software GmbH | DEVELOPER |
| https://www.citavi.com/de | LINK |
| 6.15 | VERSION |
Fulltexts were collected for results imported into Citavi, using the "find Fulltext"-function of Citavi as well as manual searches in the web. The searches resulted in:
-
3869 results including fulltext
-
395 no fulltext
Methodology - Screening
Inclusion:
-
health care information systems
-
scientific evaluation in terms of usability, user satisfaction, user experience or trust and XAI
-
scienticfic publication (poster/abstract/article/book)
Exclusion:
-
non health care publications
-
non english publications
-
non scientific publications
The screening process was done individually by two seperate reviewers. Individual screening results were then compared.
Disclaimer: The results are at risk of bias due to the subjective opinion of the reviewers, which we aim to avoid by reviewing the papers individually. Furthermore, risk of bias exists, due to the fact we might not find all fitting publications with our search query, even though we have developed it to the best of our knowledge.
