Designing a trustworthy AI means teaching it how to trust, and be trusted.

“A very simple aspect that influences trust is transparency. If you have an autonomous vehicle that is acting in the environment and you go and ask it ‘why did you make this action?’ If it is unable to tell you and it is unable to make this justification, then it doesn’t matter how reliable it is.”

In a world where ‘smart’ systems are a day-to-day part of life, from phones to vehicles, for people to feel comfortable using these systems they need to be trustworthy. But what does trustworthiness even mean when it comes to artificial intelligence, and how can it be designed?

Prof. Hussein Abbass at UNSW Canberra’s School of Engineering and Information Technology is investigating these questions to help ensure new and powerful technologies can be utilised to their full potential.

Prof. Abbass looks at ‘smart autonomous systems’: technologies that can not only make decisions about their behaviour based on the information in front of them, but can learn from those decisions to come up with solutions to problems they are encountering for the first time.

One example of an autonomous system is an autonomous vehicle, like a driverless car, which can sense its environment and navigate it without human input. In a smart autonomous vehicle, the vehicle isn’t just responding to its environment based on preprogrammed information; it can learn from each situation and respond appropriately even in situations it hasn’t specifically been programmed for.

When dealing with smart autonomous systems, trustworthiness is vital.

“When we talk about trust we are really talking about risk and delegation—you trust when you are dealing with an entity that you can delegate to in situations of high risk,” said Prof. Abbass.

“A very simple aspect that influences trust is transparency. If you have an autonomous vehicle that is acting in the environment and you go and ask it ‘why did you make this action?’ If it is unable to tell you and it is unable to make this justification then it doesn’t matter how reliable it is.”

So, in order to be considered trustworthy, an autonomous system must appear trustworthy. Designing trustworthiness into autonomous systems means understanding what will help a user trust it.

One of the main aspects of trustworthiness Prof. Abbass works on is ‘machine trust’: for a system to be trustworthy, it needs to be able to assess the trustworthiness of another system, or its user.

“Imagine the following: if you are dealing with an investment agent, someone to invest on your behalf, and if people tell you this agent is unable to assess the trustworthiness of the investors they are dealing with, how can you trust them?” he said.

Ensuring the trustworthiness of smart autonomous systems can have major consequences for how we use technology in both military and non-military contexts.

“A smart home means that we have autonomous entities in our house and they all interact in a network… If any of these entities are not trustworthy then the system will collapse,” said Prof. Abbass.

“The risk then is that we lose trust in the system. Even though the system is capable of doing great things, we don’t use it.”