On February 23, we co-hosted the event Ethics Matters together with ethix and the UX Research Guild Zurich. The online mini-conference was attended by 300-400 UX designers, developers, and interested professionals, and has brought to light topics around ethical and responsible design. It has achieved to create a sense of community that is rare in online events of such size.
Girls in Tech Switzerland had the honor to organize a panel discussion about diversity in user testing – a topic we believe is often overlooked when talking about ethics in tech. In the following, you can witness the discussion from the online conference, where we will talk about the practical challenges as well as possible solutions.
Slides that have accompanied the discussion can be found here.
Prelude: When talking about diversity, we cannot ignore the fact that panel discussions and events about ethics in tech are often dominated by women. For us, diversity is not a problem of a lack of women, it’s a lack of diversity in general. And diversity goes beyond gender. The discussion about ethical design is lacking the voices of men in the tech industry and could highly benefit from it.
Lisa: In your opinion, what is the main goal or purpose of user testing?
Janett: From a software development perspective, the goal of user testing is to make sure that we are doing our job right. We are testing the software in order to clarify whether it is doing what the user expects.
Tiffany: In my research, I am using user studies to find the underlying reasons why an interface works or why it fails. I am analyzing human behaviors with regard to technology.
Josefin: When designing products, we are making certain assumptions about the use of them. With user testing, we can test these assumptions.
Lisa: Why aren’t there any rules/laws in place for (user) testing in tech like in other industries? (e.g., medicine, psychology, food, etc.)
Janett: I believe there are historical reasons. Tech was always a thing on the side but these days it now affects our lives much more than in the past, so maybe we’re a bit behind in this respect. We need to start thinking about rules and laws here.
Tiffany: Another reason is also that different user testing is targeted towards different user groups. There are no rules that can apply to all and that’s why it’s quite difficult to have rules and laws for user studies in tech. User groups, purpose, evaluation methods, etc all vary depending on your target group. In academic research, in terms of ethics, we need ethical approval from the university before any user study is done. Research papers only get accepted to conferences and published in journals when they meet a certain standard regarding user testing. So there are already some mechanisms.
There are no rules that can apply to all and that’s why it’s quite difficult to have rules and laws for user studies in tech.
Lisa: As a master’s student, I attended a top-tier conference in user interface and software technology. I was astonished to see that many papers were presented with user studies that had a relatively small number of participants and were very non-diverse, e.g. 11 students participated from which 10 were male. How do you explain this?
Tiffany: The recruitment pool is a very important limitation. Many recruitment pools at universities tend to always be the same type of focused group (same backgrounds etc.). This limits the potential for user studies and should be avoided by the researcher. I always try to obtain at least 30 participants per study, but it’s not always easy.
I recently did a study where I needed tourists as a user group to test eye-tracking and speech recognition interfaces. As I did my study in the lab here in Zurich, most participants were German-speaking. This had an impact on the results of the study, the user group was not diverse enough in regards to languages. However, I am limited by the location, and it is much harder to find participants that are not native German speakers around here.
Josefin: I am in general more concentrated on the qualitative approach. The software that I am testing targets a specific user group (architects) so I always need to reach out to my network for participants. When I used to work as a consultant for a pharmaceutical company, we weren’t even allowed to do testing because the product was secretive. This is also something that we shouldn’t forget. Not every product can actually be tested before the release.
Janett: We often test with other developers in the team that have not seen the interface yet. The challenge is always where do you get your participants from, balancing the effort vs. what you gain from it. We often do usability tests at conferences which is very valuable. In these cases, we often don’t know who is going to show up so we usually end up with a diverse selection of people. In general, how software is developed today requires quick turn arounds, so there isn’t always time to do extensive user testing.
Don’t try to trick people. Tell the user “I’m not testing you, I am testing my work”.
Lisa: Agile software development often means that we need these quick turn arounds and user testing takes time, so it is a challenge to embed it in the development process. As a developer, I need to be aware that I always tend to design and develop for myself. I’m the first person to test an interface because I am testing my own code. Eli Pariser said this very well in a recent podcast series, that developers especially in Silicon Valley tend to be white males and they tend to develop for themselves. And that is partly the reason for some of the problems we are seeing in digital products. So I am wondering, how can we make user testing and the software development process more diverse?
Josefin: One important aspect is to do your user research before diving into testing. When you did your homework on researching the user requirements, then testing will be easier. It’s also hard to test for everything. Think of a device like Alexa. My mom speaks German with a heavy Swedish accent, but how can you ensure Alexa understands her when she speaks German?
If you have clear user requirements and do user research, in the testing phase you can then validate your assumptions. The ethical groundwork needs to be done first to catch these things before they lead to bias in the products.
Lisa: How do you find the balance between diversity and the target audience? Would it help to have more diverse design and development teams?
Tiffany: It’s important to clearly define the users that you want to test with. It is impossible to include all kinds of users in one user study. It also depends on how you define diversity. How do you make sure your user group is diverse? In terms of gender, it’s important to ask users not just whether they identify as female/male, but also give other options, and in the best case not even mention gender. For nationality, the same applies. It’s better in a user study to treat everyone equally, so as not to ask about their nationality in the first place.
Lisa: Larger tech companies are typically outsourcing user testing to scale it up. Public surveys, e.g. on LinkedIn, are becoming more popular these days. What do you think of these approaches?
Janett: This depends a lot on what you want to find out. I think in general it’s better to watch users using the interface rather than ask them about it.
Josefin: I personally prefer qualitative data to quantitative. When it’s out of context it’s hard to track. The problem with public surveys, e.g. “Which design do you like better?” is that it’s uncontextualized and someone looking at it does not have the big picture.
Lisa: Did you ever consider A/B testing?
Janett: I think if we had the technical means then it’s interesting. …but qualitative data is better. I would also be really careful about how to interpret quantitative data correctly.
Lisa: What is the future of user testing? What can we do as individuals or as a community to ensure products are designed and developed with every user in mind?
Tiffany: The voice of the community is missing about this topic. Recruitment pools with different types of participants would help in academia. We need to take diversity into consideration while we are writing and reviewing papers and ask for it. Change is already happening. At last year’s CHI (Computer-Human Interaction) conference, a paper about racial bias has won the best paper award. The awareness is slowly coming to the research community.
Josefin: I think it’s already a start that we’re having this conference today. Talking about these topics is already a good start. Don’t try to trick people. Tell the user “I’m not testing you, I am testing my work”. People tend to apologize for what they did when they behave in a way that was not anticipated by the researcher.
Lisa: Why is user testing not always a priority in software development?
Janett: Anyone in tech has experienced it, user testing is the first thing that’s skipped when time is short. Today I did a quick user testing session that took 15min and yes, it’s not as diverse but it’s better than doing no user testing at all.
I believe it’s extremely important to surround yourself with as many diverse people as possible. We often connect to people who are similar to us and stay in our bubble. We then don’t know about those users that are reading right to left, for example or users that are color blind. The more diversity you have in your surroundings, the more you have these different experiences in your mind when you’re designing.
I believe it’s extremely important to surround yourself with as many diverse people as possible.
A huge thanks to our panelists and all participants for this interesting discussion. We hope that we can shine a critical but optimistic light on the topic of diversity in user testing. Let’s keep up the conversation about ethical design. You can sign up for the ethix newsletter to receive any further information and upcoming events by ethix.
Authors: Lisa Stähli, Taisha Fabricius