Speaking our language does not automatically make an AI tool usable. This poster examines how usability evaluation can inform the development of conversational AI through AURA, an evaluation project focused on JackDaw, an LLM-based tool combining natural-language interaction with geospatial data.
Introduced through an open call, AURA brought user evaluation into a development process in which human-centred design and recurring usability evaluation were not embedded. Across three phases involving 81 participants, the study combined usability testing, stakeholder workshops and community-based evaluation.
Findings revealed barriers to basic interaction: all nine participants in the initial usability study struggled to discover the area-selection control, and six confused zooming with selecting an area. The mean System Usability Scale score was 58.9. Sharing preliminary findings prompted developers to prioritise interface changes. Subsequent evaluation showed shorter area-selection times and higher task-completion scores, while assessed answer accuracy remained essentially unchanged. These descriptive comparisons highlight the distinction between improving interaction and improving AI output.
Connecting these observations with HCI literature, the poster argues that conversational capabilities do not replace established usability principles or the need for iterative, human-centred design. Its contribution is a practice-based account of how user evidence entered development decisions, alongside actionable recommendations: test early, base design decisions on evidence, and plan repeated evaluation throughout development.
We propose small, focused usability studies interspersed with opportunities to revise and retest. The poster invites discussion on making this connection between user experience and development decisions a planned component of AI development.
We are excited to invite you to the ACM GreekCHI-CYCHI Symposium 2026, held on November 6-7, 2026 jointly in Athens, Thessaloniki, Nicosia and Online.