Draft:ASAQ - Artificial Social Agent Questionnaire

ASAQ - Artificial Social Agent Questionnaire

The Artificial Social Agent Questionnaire (ASAQ) is an instrument designed to measure human experiences when interacting with artificial social agents (ASAs). ASAs are, for instance, chatbots, virtual agents, conversational agents, and social robots. The ASAQ provides measurements for assessing constructs like believability, sociability, usability, and trust. The instrument has two versions: a 90-item long version and a 24-item short version. The questionnaire has been translated into English, Chinese Mandarin, Dutch, and German.

Development of the ASAQ began in 2018 at the Intelligent Virtual Agent conference in Sydney, Australia. The ASAQ was developed by an international working group. They followed a community-driven approach. This means that the community's interest determined what the questionnaire aimed to measure, rather than a specific theory.

Questionnaire Constructs

The ASAQ is structured around 19 core constructs, each capturing a particular aspect of the human-agent interaction experience. Three of these constructs are further divided into a combined total of eleven dimensions.

19 constructs of the ASAQ

No.	ID	Construct Name	Construct Definition
1		Agent Believability	The extent to which a user believes that the artefact is a social agent
1.1	HLA	Human-Like Appearance	The extent to which a user believes that the social agent appears like a human
1.2	HLB	Human-Like Behavior	The extent to which a user believes that the social agent behaves like a human
1.3	NA	Natural Appearance	The extent to which a user believes that the social agent's appearance could exist in or be derived from nature
1.4	NB	Natural Behavior	The extent to which a user believes that the social agent's behaviour could exist in or be derived from nature
1.5	AAS	Agent's Appearance Suitability	The extent to which the agent's appearance is suitable for its role
2	AU	Agent's Usability	The extent to which a user believes that using an agent will be free from effort (future process)
3	PF	Performance	The extent to which a task was well performed (past performance)
4	AL	Agent's Likeability	The agent's qualities that bring about a favourable regard
5	AS	Agent's Sociability	The agent's quality or state of being sociable
6		Agent's Personality	The combination of characteristics or qualities that form an individual's distinctive character
6.1	APP	Agent's Personality Presence	To what extent the user believes that the agent has a personality
6.2		Agent's Personality Type*	The particular personality of the agent
7	UAA	User Acceptance of the Agent	The willingness of the user to interact with the agent
8	AE	Agent's Enjoyability	The extent to which a user finds interacting with the agent enjoyable
9	UE	User's Engagement	The extent to which the user feels involved in the interaction with the agent
10	UT	User's Trust	The extent to which a user believes in the reliability, truthfulness, and ability of the agent (for future interactions)
11	UAL	User Agent Alliance	The extent to which a beneficial association is formed
12	AA	Agent's Attentiveness	The extent to which the user believes that the agent is aware of and has attention for the user
13	AC	Agent's Coherence	The extent to which the agent is perceived as being logical and consistent
14	AI	Agent's Intentionality	The extent to which the agent is perceived as being deliberate and has deliberations
15	AT	Attitude	A favourable or unfavourable evaluation toward the interaction with the agent
16	SP	Social Presence	The degree to which the user perceives the presence of a social entity in the interaction
17	IIS	Interaction Impact on Self-Image	How the user believes others perceive the user because of the interaction with the agent
18		Emotional Experience	A self-contained phenomenal experience. They are subjective, evaluative, and independent of the sensations, thoughts, or images evoking them
18.1	AEI	Agent's Emotional Intelligence Presence	To what extent the user believes that the agent has an emotional experience and can convey its emotions
18.2		Agent's Emotional Intelligence Type*	The particular emotional state of the agent
18.3	UEP	User's Emotion Presence	To what extent the user believes that his/her emotional state is caused by the interaction or the agent
18.4		User's Emotion Type*	The particular emotional state of the user during or after the interaction with the agent
19	UAI	User Agent Interplay	The extent to which the user and the agent have an effect on each other

Notes: the numbering following <construct number>.<dimension number>. In italics are the constructs that are measured indirectly through dimensions. * Dimension not measured in the ASAQ.

Scale and perspective

The constructs and dimensions are assessed through a series of statements (i.e., questionnaire items), where participants indicate their level of agreement on a seven-point scale. Responses range from -3 ("strongly disagree") to +3 ("strongly agree"), with 0 representing a neutral stance ("neither agree nor disagree"). Using first and third-person perspective, the ASAQ can be used to assess a user's own experience with an ASA or to evaluate someone else's interaction with an agent.

The ASAQ instrument

ASAQ Representative Set

The ASAQ Representative Set serve as the normative datasets for interpreting ASAQ scores, providing context for understanding how an ASA scores across constructs and dimensions. The ASAQ Representative Set 2024 consists of 1,066 participant ratings of 29 agents using a third-person perspective. The representative set offers researchers benchmarks for comparing their ASA's scores against familiar agents, enabling interpretation through percentile ranks or relative positioning. It also supports study planning by providing effect size estimates and guidance for sample size decisions.

ASAQ Charts

The profile of two agents from the ASAQ Representative Set 2024, Amazon Alexa and NAO (robot) . Above: the ASAQ chart. Below: the percentile ASAQ chart comparing the representative set 2024. Here, the grey area indicates scores below or above the representative set.

Two types of ASAQ charts can be used to visualise an ASA's interaction profile. The ASAQ Chart displays scores on the original scale, ranging from -3 to +3, reflecting the raw mean responses for each of the 24 constructs and dimensions. The Percentile ASAQ Chart presents the same constructs using percentile ranks, allowing researchers to compare their ASA's performance against the ASAQ representative set. In both charts, the centre shows the overall ASAQ score or its corresponding percentile score.