short-paper

Designing evaluation beyond evaluating design: measuring success in cross-cultural projects

Authors:

Jennyfer Lawrence Taylor,

Alessandro Soro,

Margot Brereton,

Anita Lee Hong,

Paul RoeAuthors Info & Claims

OzCHI '16: Proceedings of the 28th Australian Conference on Computer-Human Interaction

Pages 472 - 477

https://doi.org/10.1145/3010915.3010965

Published: 29 November 2016 Publication History

Abstract

This paper considers the issue of evaluation in cross-cultural contexts, where the focus on Western principles of technology quality can be problematic. We present a case study of designing a Digital Community Noticeboard with an Australian Aboriginal community, and discuss how its evaluation may be better approached in terms of 'designing futures'. Rather than viewing evaluation as the validation of a designed artefact, our approach is focused on mapping progress towards a vision for a future world that shapes, and is shaped by the noticeboard, as the two have mutually evolved. The resulting insights go beyond the technical and lead to a better understanding of how the noticeboard may fit with everyday practices, local values and social protocols, and issues of sustainability. This shift in perspective helps to foreground these socio-technical factors and research problems that may otherwise pass unnoticed.

References

[1]

Bailey, J. E., & Pearson, S. W. (1983). Development of a tool for measuring and analyzing computer user satisfaction. Management Science, 29(5), 530--545.

Digital Library

[2]

Bessarab, D., & Ng'andu, B. (2010). Yarning About Yarning as a Legitimate Method in Indigenous Research. International Journal of Critical Indigenous Studies, 3(1), 37--50.

[3]

Bevan, N. (2001). International standards for HCI and usability. International Journal of Human-Computer Studies, 55(4), 533--552. article. http://doi.org/http://dx.doi.org/10.1006/ijhc.2001.0483

Digital Library

[4]

Bidwell, N. J. (2016). Moving the centre to design social media in rural Africa. AI and Society, 31(1), 51--77. http://doi.org/10.1007/s00146-014-0564-5

Digital Library

[5]

Bidwell, N. J., Standley, P.-M., George, T., & Steffensen, V. (2008). The Landscape's Apprentice: Lessons for Place-centred Design from Grounding Documentary. In Proceedings of DIS 2008 (pp. 88--98). inproceedings, New York, NY, USA: ACM. http://doi.org/10.1145/1394445.1394455

Digital Library

[6]

Bohm, V., & Wolff, C. (2014). A review of empirical intercultural usability studies. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8517 LNCS(PART 1), 14--24. http://doi.org/10.1007/978-3-319-07668-3-2

[7]

Brereton, M., Roe, P., Amagula, T., Bara, S., Larara, J., & Hong, A. L. (2013). Growing Existing Aboriginal Designs to Guide a Cross-Cultural Design Project. In Proceedings of INTERACT 2013 (pp. 323--330). Berlin: Springer.

[8]

Brereton, M., Roe, P., & Hong, A. L. (2012). Evolving a relationship for cross-cultural participatory innovation. In Proceedings of Participatory Innovation Conference 2012 (pp. 1--5). Melbourne: Swinburne University.

[9]

Brereton, M., Roe, P., Schroeter, R., & Hong, A. L. (2014). Beyond Ethnography: Engagement and Reciprocity as Foundations for Design Research Out Here. In Proceedings of CHI 2014 (pp. 1183--1186). New York City, New York: ACM.

Digital Library

[10]

Dillon, A. (2002). Beyond usability: process, outcome and affect in human-computer interactions. Canadian Journal of Library and Information Science. JOUR.

[11]

Ehn, P. (2008). Participation in design things. Conference on Participatory Design, 92--101. http://doi.org/10.1145/1795234.1795248

Digital Library

[12]

Ehn, P. (2016). Introduction to Making Futures. JOUR.

[13]

Gorman, T., Rose, E., Yaaqoubi, J., Bayor, A., & Kolko, B. (2011). Adapting Usability Testing for Oral, Rural Users, 1437--1440.

[14]

Greenberg, S., & Buxton, B. (2008). Usability evaluation considered harmful (some of the time). Proceedings of CHI 2008, 111. http://doi.org/10.1145/1357054.1357074

Digital Library

[15]

Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2--3), 61--83. article. http://doi.org/10.1017/S0140525X0999152X

[16]

Hewett, T. T., Baecker, R., Card, S., Carey, T., Gasen, J., Mantei, M., ... Verplank, W. (1992). ACM SIGCHI Curricula for Human-Computer Interaction (techreport). New York, NY, USA: ACM.

Digital Library

[17]

Irani, L., Vertesi, J., Dourish, P., Philip, K., & Grinter, R. (2010). Postcolonial computing: a lens on design and development. In Proceedings of CHI 2010 (pp. 1311--1320). New York City, New York: Association for Computing Machinery. http://doi.org/10.1145/1753326.1753522

Digital Library

[18]

Kermarrec, A.-M., & Le Merrer, E. (2012). Offline Social Networks: Stepping Away from the Internet. In Proceedings of the Fifth Workshop on Social Network Systems (p. 14:1--14:2). inproceedings, New York, NY, USA: ACM. http://doi.org/10.1145/2181176.2181190

Digital Library

[19]

Light, A. (2015). Troubling Futures: can participatory design research provide a constitutive anthropology for the 21st century? Interaction Design and Architecture(s) Journal, (26), 81--94.

[20]

Müllert, N., & Jungk, R. (1987). Future Workshops: How to create desirable futures. London, United Kingdom: Institute for Social Inventions. JOUR.

[21]

Nielsen, J. (1993). Usability engineering. (A. Press, Ed.). book, Academic Press.

[22]

Preece, J., Rogers, Y., & Sharp, H. (2015). Interaction Design: Beyond Human-Computer Interaction. West Sussex: Wiley.

[23]

Redstr, J., Petersen, M. G., Lar, H., & J.K., J. R. (2008). Introduction to special issue on the aesthetics of interaction. ACM Transactions on Computer-Human Interaction, 15(3), 1--5. http://doi.org/10.1145/1453152.1453153

Digital Library

[24]

Simonsen, J., & Robertson, T. (2012). Participatory Design: An introduction. Routledge International Handbook of Participatory Design. Abingdon-on-Thames: Routledge International.

Digital Library

[25]

Soro, A., Brereton, M., Lee Hong, A., & Roe, P. (2015). Bi-Cultural Content Publication on a Digital Noticeboard: A Design and Cultural Differences Case Study. In Proceedings of OzCHI 2015 (pp. 217--221). New York City, New York: Association for Computing Machinery. http://doi.org/10.1145/2838739.2838813

Digital Library

[26]

Soro, A., Brereton, M., Taylor, J. L., Lee Hong, A., & Roe, P. (2016). Cross-Cultural Dialogical Probes. In Proceedings of AFRICHI 2016. New York City, New York: Association for Computing Machinery.

Digital Library

[27]

Soro, A., Lee Hong, A., Shaw, G., Roe, P., & Brereton, M. (2015). A Noticeboard in "Both Worlds" Unsurprising Interfaces Supporting Easy Bi-Cultural Content Publication. In Proceedings of CHI 2015 (pp. 2181--2186). New York City, New York: Association for Computing Machinery.

Digital Library

[28]

Suchman, L. (2002). Located accountabilities in technology production. Scandinavian Journal of Information Systems, 14(2), 91--105.

Digital Library

[29]

Taylor, J. L., Soro, A., Hong, A. L., Roe, P., & Brereton, M. (2016). Designing For Cross-Cultural Perspectives of Time. In Proceedings of AfriCHI 2016. New York City, New York: Association for Computing Machinery.

Digital Library

[30]

Tuhiwai Smith, L. (2012). Decolonizing Methodologies (2nd Editio). London: Zed Books.

[31]

Verran, H., Christie, M., Anbins-King, B., Van Weeren, T., & Yunupingu, W. (2007). Designing digital knowledge management tools with Aboriginal Australians. Digital Creativity, 18(3), 129--142. http://doi.org/10.1080/14626260701531944

[32]

Winschiers-Theophilus, H., Chivuno-Kuria, S., Kapuire, G. K., Bidwell, N. J., & Blake, E. (2010). Being Participated - A Community Approach. In Proceedings of PDC 2010 (pp. 1--10). New York City, New York: Association for Computing Machinery. http://doi.org/10.1145/1900441.1900443

Digital Library

[33]

Winschiers, H. (2006). The Challenges of Participatory Design in a Intercultural Context: Designing for Usability in Namibia. In PDC (pp. 73--76). inproceedings.

[34]

Zimmerman, J., Forlizzi, J., & Evenson, S. (2007). Research through design as a method for interaction design research in HCI. In Proceedings of CHI 2007 (pp. 493--502). inproceedings, New York, NY, USA: ACM. http://doi.org/http://doi.acm.org/10.1145/1240624.1240704

Digital Library

Cited By

Taylor JCouncil WSoro ARoe PBrereton M(2019)A Relational Approach to Designing Social Technologies that Foster Use of the Kuku Yalanji LanguageProceedings of the 31st Australian Conference on Human-Computer-Interaction10.1145/3369457.3369471(161-172)Online publication date: 2-Dec-2019
https://dl.acm.org/doi/10.1145/3369457.3369471
Taylor JSoro ABrereton M(2018)New literacy theories for participatory designProceedings of the 15th Participatory Design Conference: Full Papers - Volume 110.1145/3210586.3210588(1-13)Online publication date: 20-Aug-2018
https://dl.acm.org/doi/10.1145/3210586.3210588
Taylor JSoro ARoe PLee Hong ABrereton MMandryk RHancock MPerry MCox A(2018)“Debrief O'Clock”Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173882(1-14)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3173882
Show More Cited By

Index Terms

Designing evaluation beyond evaluating design: measuring success in cross-cultural projects
1. Human-centered computing

Recommendations

Designing For Cross-Cultural Perspectives of Time
AfriCHI '16: Proceedings of the First African Conference on Human Computer Interaction

We present ongoing work that seeks to understand cross-cultural perspectives of time, and reflect these temporalities in the participatory design of a cross-cultural Digital Community Noticeboard. Time is a socio-cultural phenomenon that is experienced ...
Evaluating HCI Research beyond Usability
CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems

Evaluating research artefacts is an important step to showcase the validity of a chosen approach. The CHI community has developed and agreed upon a large variety of evaluation methods for HCI research; however, sometimes those methods are not applicable ...
Evaluation Beyond Usability: Validating Sustainable HCI Research
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

The evaluation of research artefacts is an important step to validate research contributions. Sub-disciplines of HCI often pursue primary goals other than usability, such as Sustainable HCI (SHCI), HCI for development, or health and wellbeing. For such ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

OzCHI '16: Proceedings of the 28th Australian Conference on Computer-Human Interaction

November 2016

706 pages

ISBN:9781450346184

DOI:10.1145/3010915

Conference Chairs:
Henry Duh
University of Tasmania Christopher Lueg, University of Tasmania
,
Christopher Lueg
University of Tasmania
,
Program Chairs:
Mark Billinghust
University of South Australia
,
Weidong Huang
University of Tasmania

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

IEEE-SMCS: Systems, Man & Cybernetics Society
Australian Comp Soc: Australian Computer Society
Data61: Data61, CSIRO
ICACHI: International Chinese Association of Computer Human Interaction
Infoxchange: Infoxchange
HITLab AU: Human Interface Technology Laboratory Australia
James Boag: James Boag
Tourism Tasmania: Tourism Tasmania
HFESA: Human Factors and Ergonomics Society of Australia Inc.
IEEEVIC: IEEE Victorian Section
UTAS: University of Tasmania, Australia

In-Cooperation

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Australian Research Council

Conference

OzCHI '16

Sponsor:

IEEE-SMCS
Australian Comp Soc
Data61
ICACHI
Infoxchange
HITLab AU
James Boag
Tourism Tasmania
HFESA
IEEEVIC
UTAS

OzCHI '16: The 28th Australian Conference on Human-Computer Interaction

November 29 - December 2, 2016

Tasmania, Launceston, Australia

Acceptance Rates

Overall Acceptance Rate 362 of 729 submissions, 50%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
207
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)0

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Taylor JCouncil WSoro ARoe PBrereton M(2019)A Relational Approach to Designing Social Technologies that Foster Use of the Kuku Yalanji LanguageProceedings of the 31st Australian Conference on Human-Computer-Interaction10.1145/3369457.3369471(161-172)Online publication date: 2-Dec-2019
https://dl.acm.org/doi/10.1145/3369457.3369471
Taylor JSoro ABrereton M(2018)New literacy theories for participatory designProceedings of the 15th Participatory Design Conference: Full Papers - Volume 110.1145/3210586.3210588(1-13)Online publication date: 20-Aug-2018
https://dl.acm.org/doi/10.1145/3210586.3210588
Taylor JSoro ARoe PLee Hong ABrereton MMandryk RHancock MPerry MCox A(2018)“Debrief O'Clock”Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173882(1-14)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3173882
Taylor JSoro ARoe PHong ABrereton M(2018)From Preserving to Performing Culture in the Digital EraDigitisation of Culture: Namibian and International Perspectives10.1007/978-981-10-7697-8_2(7-28)Online publication date: 10-May-2018
https://doi.org/10.1007/978-981-10-7697-8_2
Taylor JSoro ARoe PLee Hong ABrereton MMark GFussell SLampe Cschraefel mHourcade JAppert CWigdor D(2017)Situational WhenProceedings of the 2017 CHI Conference on Human Factors in Computing Systems10.1145/3025453.3025936(6461-6474)Online publication date: 2-May-2017
https://dl.acm.org/doi/10.1145/3025453.3025936
Soro ABrereton MTaylor JLee Hong ARoe P(2017)A Cross-Cultural Noticeboard for a Remote Community: Design, Deployment, and EvaluationHuman-Computer Interaction - INTERACT 201710.1007/978-3-319-67744-6_26(399-419)Online publication date: 20-Sep-2017
https://doi.org/10.1007/978-3-319-67744-6_26

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten