076: User Research? Game On with Steve Bromley

Jason Ogle: Defenders I have with me today, Steve Bromley, user researcher expert. He’s an author. He’s written a couple of books now, building user research teams and soon to be released at the time of this recording, how to be a games researcher.

Steve works with game studios to help them get insight from players to make games better. That’s a very important thing, just like you don’t know how people are using your software, you don’t know how players are using your games. So, I’m excited to learn a little bit more about that.

He’s worked with Sony, he’s done a lot of stuff there. So he’s got a lot of experience and I’m super excited to talk to him more about that.

And here’s a little fun fact. I like to start with these before we jump in as a child, Steve was once on the Disney channel advertising land yachting. It’s so random, but really pretty spectacular.

Steve Bromley: It’s the only time I’ve ever done land yachting. It’s quite an experience to do it on TV for the first time. I don’t know if you’ve ever done it yourself.

Jason Ogle: No. What is land yachting for our education?

Steve Bromley: You take a yacht. Well, actually that’s unfair. You take a canoe you put some wheels on it and a sail on it, and then you just go down the beach as fast as you can. Obviously it requires a very windy day, but yeah, it’s exciting.

Jason Ogle: That’s awesome. So Steve officially welcome to User Defenders: Podcast. I’m super excited to have you on the show today, my friend.

Steve Bromley: Fantastic. Thank you for having me on Jason. I’m really excited to be here today.

Jason Ogle: We’re going to talk a lot about user research. Around games too, because that is really where you focused a lot of your attention. And I know you’ve done traditional software as well, so I’m interested to really glean and I know the Defenders too, especially around what is games, user research.

So we’ll, we’ll jump into that for sure. But before that, I want to start with the most important question that I’m going to ask. I think during our time today,

Steve Bromley: I’m looking forward to this.

Jason Ogle: How to mere mortals get their hands on a PS5??

Steve Bromley: Oh yes. So I gathered the easiest way is to take a temporary job as a delivery driver and look out for someone ordering a PS5 and then just don’t deliver it. I gather there’s a lot of that going on.

Jason Ogle: I have a feeling there is, Oh man, imagine, creating a product that people want that bad. I mean, that’s pretty spectacular, right?

Steve Bromley: It’s a really nice and to have isn’t it? I imagine there’s lots of businesses who would like to be in that position where as fast as you can make them, people will buy them. And, yeah, it’s exciting time, I guess, with the new console generation, lots of people paying a lot of attention to what’s coming out and to there’s a lot of hype around it as well, which is exciting as well.

Jason Ogle: Definitely eat your heart out Apple. Right?

Steve Bromley: Exactly.

Jason Ogle: So I just want to reflect for a minute. I grew up in the eighties. I missed that generation so much. The more, some more time goes by the more I miss the eighties. But I just remember like going down the street, and they had one of the early Atari 2600’s.

And I saw Pac-Man on the screen for the first time. And they had this little black joystick with one red button. That was it. The neighborhood kids were all playing this game and I was just enamored.

I couldn’t believe that this was possible. if you look at Pac-Man now you’re like, crud, that’s really cheesy, the graphics and everything. But back then, man, this was cutting edge stuff. This was outlandish. This was one of those aha moments, right? Wow. Games inside of my home. On a console.

And so I reminisce and it hits a soft spot in my heart because those were my roots there, you know, and all that to say we ultimately ended up getting a console. It probably took another year or two, but we got one for Christmas. That was one of the best Christmases ever.

All that to say, Atari has really led the way I think, in bringing video games into the home and being kind of the pioneers of that. If I’m not mistaken, I’d love to know more. Maybe if I’m missing something, but also I’m really curious because UX in general is still a pretty new, nascent even movement in building software. It still is. You can look back, you can trace back to Don Norman and the eighties, and he was onto something, right? Don Norman coined the term user experience worked at Apple.

I know Atari did research on their games, right? So it’s not necessarily new, but it’s new to the masses. Let’s say that, right. This is still a relatively new area to the masses. So all that said, I’m really curious.

What are some of the differences and similarities to how Atari conducted user research, in the seventies to say how Sony or Microsoft do it today?

Steve Bromley: Yeah, I think that’s a great summary of the history of games, user research as well. Because games is such a small field and so well documented, I think, Games User Research you’re very lucky that we can see that history back. We can see, okay, this is the company that did it first and how they did it.

And then what happened next? And then as it grew with games like Crash Bandicoot. So you asked about the similarities, the work that Atari was doing in the seventies is still fundamentally very similar to what we’re doing today. They had employed a lady called Carol Kantor who went to arcades in the first place to watch people playing games for real in the arcades. What she was doing there is the same as what user researchers do today.

We’re looking to see if people understand how the game works, what they’re meant to do. Is it fun? Is it too difficult? Is it too easy? Balancing the game correctly? And that is still a lot of what we do. One of the problems that every user research field has. Especially games is the people who design games have to care about games.

They have stories similar to yours about their first encounter with games, and they play a lot of games all day long and they become very different to a typical player just because the amount of exposure they have to games in general and the game they’re working on specifically. And so there’s still that tremendous value of getting some players in, watching the players experience your game, realizing how the experience they’re having is different to what the designer intended the experience to be, and then making changes to the game to bring that alignment between what we think players should understand and should do, and what they’re actually able to do that doesn’t mean that the field hasn’t evolved since the seventies you also asked about what’s, what’s different between.

When Atari, we’re watching people play games back in the seventies and what Sony or Microsoft are up to today with this kind of play testing. Some of the areas where it has developed, include bringing it earlier in the development process. So one of the real problems, if you’re going to watch people play a game in arcade is.

That game is already made. It’s been printed on circuit boards and released everywhere. And if you do notice that game, isn’t working in the way you intended, you can’t re-do much at that point. You can’t send out patches in those days or post-launch updates. So the problem is the things you’ve learned in the arcade.

You can only apply to your next game. You can’t apply to this game. What Games User Researchers have done over the last decade or two decades is move earlier in that development process. So looking at prototypes and in development software, so that it gives our game developers the opportunity to react to the findings for that game and fix the game that they’re working on rather than just get learning some things that you might apply in their next game.

I guess another area that games use research has evolved since the seventies is the methods available to us. So that original method of one-to-one testing, where we have a player, we watch them play and we look at what they understand or don’t understand them. We ask some probing questions is still extremely valuable, and one of the most impactful types of research we can run. But there’s other types of questions that our game developers have that you can’t answer with that kind of one-to-one session.

Like. How difficult is this level that will take some sort of quant measurement to looking at how many times the players fail, or how many times the people die, or how are they rating this level?

And to run that kind of study, you need a lot of people to come in to test the game. And that wasn’t really possible in the same way in the arcades in the seventies, you could only do one to one studies at a time rather than these kind of mass testings that it’s possible now. But as I say, the fundamentals are still very similar.

Jason Ogle: Yeah, that’s really insightful. I remember going to the arcades in the early eighties and man Dragon’s Lair when I saw Dragon’s Lair it was like a cartoon you could control. And I was like blown away, man. I was like, I just want to put all my quarters in this game, even though it was super difficult, because if you don’t move that joystick at the right second you’re dead.

Right. And, but you get to see a cool animation at least to that end. But yeah, you touched on some interesting points that I hadn’t really considered with video games like I do with traditional software and even print. I think we really take the technologies we have available to us for granted. younger Defenders coming in, growing up with smartphones and things, and really high-performing computers and I really want you to know how blessed you are to be building software and designing software in this age, because everything was so much harder earlier on, right?

Like even print. I’m not to get too off track here, but I think about print in like the early days of advertising, right. They, everything was hand done. Everything was hand align, your typography, you’re leading the reason they call it leading is that they actually put strips of lead underneath the letters to space it out, to give it some readability. We just do two clicks on a computer and that’s it, and then we can send it up print. And then I think about software websites and you know, that’s where it kind of, a lot of this started moving from was a in website development at the infancy of the web, and then it became robust software solutions, and e-commerce so I’m just thinking around the fact that when you print something, if you misspell one word, like a headline or something, and you print 10,000 of those, guess what? You just wasted a lot of money and a lot of time to produce something that was wrong and you can’t fix it.

The only way you can fix is to start over and then spend a bunch more money to get those 10,000 printed right, but with software you make a misspelling, you can go in and you can correct that in about 10 seconds, upload it.

And then the whole world may not have even noticed. And then I just going to the video game subject. Arcade that’s where it was before Atari helped bring entertainment, video game entertainment into our homes. When they were doing research, they had to go into the arcades, talk about the early guerilla research, right?

Like they had to go into the arcade, watch young people likely play these arcade games and just try to learn how they can make it better and bugs. How do you even know. Really like, right. I mean, you could probably test it in like your lab or whatever, or in your garage, wherever you’re building these things.

But you probably don’t discover a lot of the bugs until you actually spent all this money all this time. Shipping that game into a huge hardware console with graphics and everything. And then you got to make it again after you learn. And then you’ve got to go. I’m sorry to pontificate.

I hope I’m not like pontificating too much, but I just feel like this is something we really ought to kind of take a step back, and really kind of be grateful…show some gratitude about how lucky and blessed we are to be living in an age of agile and swift development, and that ties right into research, right?

Steve Bromley: Yeah, exactly. And it’s not just the tech that we are, the things we’re testing that have changed over time and allowed us to make updates and iterate much quicker, but also the technology available to researchers early on the work that PlayStation were doing to run usability studies on a Crash Bandicoot, for example, they were recording those sessions on VCR tape so they could watch them back later.

It’s really interesting too, that mashup of. How, what is incredibly simple these days for a researcher to share their screen, to give control to a participant, all those things that you have to do 10 times a day, just wasn’t possible back then. And you had to do a lot more low tech solutions to these kinds of things.

Jason Ogle: That’s awesome, man. I, it’s neat to learn about the history of I’ve honestly never thought about it and I’m not a gamer, now, but I really appreciate, and again, like I said, I grew up with video games and I stayed up all night trying to solve Legend of Zelda with my buddies, you know, like drinking six pack of Jolt Cola’s trying to solve legend of Zelda or Sonic the Hedgehog and all that stuff. This is really hitting a soft spot for me, man is kinda it’s really is a it’s. It’s very sentimental for me. And so I appreciate the craft and I know. My son is wanting a PS5 so bad, just like pretty much the rest of the world.

But I could not believe, like not having looked at a video game in quite a while to having seen the display at Target, and I just can’t believe how far technology has brought gaming. I really appreciate all of the work, all the effort that goes into creating this entertainment.

Steve Bromley: Yeah, it’s a really interesting medium, I guess, to build on your point about the changes as well. One of the things that makes games particularly exciting is the technology available to you. It’s usually one of the first fields that sees new tech, like virtual reality headsets, a big thing in games for the last few years, hasn’t really broken through to the rest of the consumer market yet.

Things like the Wii a few years ago, which use a lot of motion gaming, and when that was very big, about a decade ago, again, as a researcher, it gives you new challenges that the rest of consumer software won’t see for another 10 years or more. So some lovely research objectives and questions to work on there.

Jason Ogle: I kind of teased it a little bit and you kind of touched on it a little bit in your response to my user research history question, but I’m curious because I’m sure there are many, but what are the inherent differences, and/or complexities with doing UXR, I’ll call it UXR, which is stands for user research for games versus general software. Can you dive into some of that for us?

Steve Bromley: Yeah, there’s two areas that I think are particularly interesting. I guess the first one is the objectives. What are the research questions that you are being set as a researcher traditionally in software, you hear research objectives about efficiency. Like can someone create a spreadsheet quickly, or can they order their taxi on a taxi ordering app within a few seconds and not get stuck.

Efficiency is a lot less relevant for games. The most efficient game I could imagine is they replace Mario with a button that you press the button says, save the princess and you’d press the button and save the princess. Honestly, that’s extremely efficient, but that’s not the point of the games. Games are meant to have friction and meant to throw challenges that you can overcome.

And so as a researcher, as well as that typical usability can people do things you’re often brought questions about fun. Is this game fun, and do people enjoy it? That’s reasonably hard to measure, especially compared to efficiency, which you can measure things like time or errors. For fun, it’s a lot more subjective and requires interesting study designed to experience to understand, and then measure that experience.

Things like benchmarking against other games or, trying to understand what are the reasons that make this game fun, rather than just relying on a player, giving it nine out of 10 on a survey And another area I think is really interesting, and that difference between games user research and other forms of user research is some of the setups that you require.

And some of the constraints that games introduce. So, games have very large marketing budgets. I think some it’s well-known that Microsoft, for example, spent more on the marketing of Destiny, one of their big games from a few years ago than the actual development. And what that means is that gained are very interested in secrecy.

They want to be very clear what information is getting out in the public and stop anything from getting out. As user research works on prototype software or in development software, that means some of the methods that are used a researcher would normally want to apply like remote studies, where you can screen share with someone playing at home.

Or sending someone something and then using analytics to measure it aren’t so possible because of that risk of leaks or stuff. Getting out in the public domain before it’s ready. That means that you have to come up with work around. I think it’s very common in the big studios like Ubisoft or Microsoft or Sony is having a playtest lab where they have 10 or 20 or 50 pods set up to bring players in. So you can start to answer quant questions by getting lots of people to play at the same time, but they come to you. Whereas typically you’d want to do that remotely with other types of software. Again, it’s just different approaches and different challenges are answering your research objectives that I think make games a really interesting subject to work on. Yeah, it’s it’s a great area.

Jason Ogle: That’s great. There is definitely a lot of different questions to ask than creating the next car buying software or whatever. Right? Methods, and you touched on it slightly, I think, with benchmarking, but I’ve asked this question certainly several times in the history of the show. And I’ve never really been satisfied with the answers. So no pressure, Steve, but you made a good point about, how measuring delight, It’s very difficult, but it’s very important to the success of a video game. How have you been able to do that? What are some of the best ways that you’ve found to actually measure the delight of an experience?

Steve Bromley: Yeah, as you can imagine, it’s the type of questions that game developers come to you all the time. They say, do people like my game and is it fun? And can you give it some sort of score? Everyone likes scores, I guess, especially for games because they get reviewed on websites and in magazines. And the review has a score.

The whole industry expects scores, and also they assume that some sort of correlation between well players get a high score. So the reviewers is definitely going to get a high score. Whereas actually they’re quite different people and they’re not necessarily going to be the case. And you’re right.

That it’s really hard to measure delight or fun. So what a lot of teams will start with is asking for scores, but scores are ultimately not that useful. If you go to a developer and you say, well, your game is seven out of 10. That’s not so actionable. What are you going to do about it? You’re like, okay, so could be better? It could be worse? But it’s seven. Great. Instead, one of the things that we think is more useful than just that fund score is understanding why the game is fun. And that can be very difficult. It’s very easy to lead people. If you ask them, do you like the graphics? They will give you an opinion about the graphics, but that didn’t necessarily mean it was a thing that they cared about in the first place.

So there’s some techniques that I know some user research teams use that employ a thing called grounded theory, where you try not to prompt the player or the participant about the subject. You let them talk freely about what they thought about the game, and then you identify, Okay, the concepts they talked about were the graphics and the jumping and the weapons.

And then you have some things that you can measure after that. So you can take each of those topics that they brought up by themselves, and you can create a survey or do some sort of quant study to see how important these things are and how many people, whether people like tell they didn’t like to avoid that risk of leading people.

So I guess to come back your point about the difficulty with measuring delight is extremely difficult. It’s a quant question, so you need a lot of people to do it. It’s very easy to lead people down the wrong answer, but the approach I’d recommend is interviewing people to find out what’s important with them and then measuring how many people that’s important to, to create the size of those impacts.

Jason Ogle: I’m going to get to this question, but I’m going to ask it later, but I, I want you to put that in your back pocket about how many, how many users. Okay. So just kind of priming you there. Okay. But, I think it’s interesting what you said about measuring delight factor and even just showing graphics to somebody like, I think anybody can appreciate good graphics.

I think we kind of know the difference now. Especially in, again, a thankfully it took some time, but thankfully, a design driven technology world. The average user understands, whether an app, whether a software, whether a game is designed well or not.
And I think that’s a good thing. It puts pressure on us as designers and researchers to get this right. And if not to keep trying to get it right, thankfully, like we said earlier, we can do that with software. But, it’s funny, you were talking about like, yeah, the graphics are great, but do you like it, right?

Like, would you, is this fun? Would you play this game? And then it made me think about how in 2006, Taylor Hicks one American Idol, he beat Chris Daughtry to win American Idol. And guess what? Nobody bought his record when it came out.

Steve Bromley: Yeah. That, that really, that really highlights the different, doesn’t it between making sure you’re measuring the thing that’s important and what American Idol measures is. Do you want this person to win a pop contest? Yeah. Great. But it doesn’t measure, will people buy the record of this person?

That’s a different thing and you’re measuring the wrong thing in that case.

Jason Ogle: Exactly. I have a rant about that, Defender’s, I’ll post it in the uh, the show notes. I gotta say a little humble brag here. I told Seth Godin about it because he’s been a big inspiration of mine. And he read the article and he left a comment on it.
So that was one of the highlights of my very short writing career there.

Steve Bromley: I will check it out. Yes.

Jason Ogle: Yeah. I’ll post it. I’ll send you the link after, and I posted it in the show notes, but I talk about this kind of case study, and it’s a bit of a rant again about being picked. Cause a lot of us going all the way back to childhood in school, we want to be picked for the kickball team. We don’t want to be picked last, even if we’re picked like second or third from last that’s better. But my whole point was don’t wait to be picked, pick yourself. But you also mentioned about scoring and I couldn’t help, but think about, seven out of 10, 10, what.

Right. What’s the criteria there. And I think about, you know, Defenders and I know they’ll, many of you are looking for a job in UX. You’re either in transition or you’re trying to get your foot in the door for the first time. When you look around, you’ll see portfolio examples that have skills on them, that folks are measuring themselves and use of like Photoshop or Figma or any of these tools. Right? My big question is. nine out of 10. What? Right. That’s why I’m not, I don’t know if I’m crazy about those. When I see those on a portfolio, I was like, yeah, but what are you measuring against?

So I, that’s just a little rant there too. A little mini rant on that. Maybe reconsider that or try to test it with others, see if that is effective or not, but I just, you just want to go, well, what’s the scale?
What are you measuring against?

Steve Bromley: Yeah, everyone likes measurements. Cause they’re easy to look at and easy to compare, but often it’s very difficult to make the thing you’re measuring actually be the important impacts that you’re trying to have on the world.

Jason Ogle: That’s a good point. And then when you’re trying to pitch your work to, to stakeholders, to executives, they wanna see something. And so I can understand why score. It has some sort of score or some sort of quant would help convince them of the hard work that you’ve done. And you’ve been able to do synthesize that and I, I know we’re going to try to get into that a little bit later, too.

But before that I’m gonna challenge you. I’m gonna challenge the Defenders a little bit around gaming because. Like I said, I appreciate the craft. I appreciate, I know how much work goes into this.

I’m I appreciate that. There’s a lot of really talented folks that are able to make a livelihood out of creating really groundbreaking and fun experiences for other humans. I appreciate that. Okay. I am going to challenge this a little bit because ethics are very important.

Especially for Defenders listening, talk about fighting for the user, right? Like that’s what really, what the the show, the name of the show is where User Defenders. Right? I want to tell a quick story. My good friend’s younger brother. He had everything going for him. He was engaged to be married. He had a great job. Life seemed really great for him, and then he started playing this game called EverQuest, also known as “EverCrack”.

Okay. So I’m telling you, within a matter of months of just nonstop play of this game, he ended up letting himself go stop taking care of himself. His beard got really long and unkept his fingernails grew. He lost his job. He lost his fiance. She left him. It all fell apart really fairly quickly, seemingly. He kind of became like a Howard Hughes.

Have you ever seen the Aviator movie or read the Howard Hughes? He kind of became kind of a recluse. I’m not blaming the video but, I think there’s an interesting correlation to some of these life events and his unrelenting habit of playing this online role-playing game.

And then there’s of course, there’s the story of Shawn Wooley who was also an EverQuest player. And in 2001, I actually hadn’t heard about this. I wasn’t even looking for this when I was researching, but it came up. I will say he did have some mental and emotional disabilities. So I will put that out there.

Schizoid personality disorder. But he killed himself. His mother found him. Basically with his head down right in front of the computer and the computer screen had EverQuest on there. So she did a little correlation too on this, and it wasn’t just that day. He had just shut his family out for months before that he wouldn’t let anybody into the house.

He’d stopped cleaning his house and just shut the world out. And he got emotionally involved with another player in this game It was assumed that there was a player on the other side that he had actually romantically completely fallen for.

And it was assumed that he was rejected somehow by this fictitious. I mean, that’s a real person, but this was a character in a game that rejected him. And we know, I mean, rejection hurts, man. It hurts a lot. So, and if you have again, combined with some of the mental and emotional disabilities, this fellow had he’s 21 or something at this time.

And this was in 2001. He shot himself in front of the computer. What I’m getting at here, and my long-winded way of asking a question around this. Should video game companies and console providers, should they take more responsibility and/ or action to address the potentially addictive and destructive qualities of some video games. Do you think that these are designed for attention, man, they’re designed to just like social media. I’m not going to let social media off the hook. I’ve had issues. I’ve had to quit some of the platforms. I’m wearing a shirt right now that I made It says wonderful servant. Terrible master. And it’s got a smartphone on it, massaging the squirrel there. I know that, and Nir Eyal, a former two-time guest, he wrote a book called “Hooked” with good intentions to try to help software designers create products that will help people using psychology to help people get good habits, but just like anything, can be used for good or bad. So, with great power comes great responsibility.

Steve Bromley: It’s a great topic. And you asked about responsibility. I agree entirely that us as designers and people who work in the design field, even though my job title isn’t designer, we have to be responsible for the consequences of what we’re designing. You’re probably very familiar with dark patterns, that idea of you can create hooks in, in software or in games that encourage you to see things that are aren’t necessarily healthy or good for them. And one of the issues that games has faced is it’s found a lot of these hooks much earlier because it’s about entertainment, things that took the web or internet until the last decade to learn games have been applying.

They’ve seen mechanics to encourage people to play more, to play longer for many years now, as designers, we need to recognize the impact that it really has on our users and, stand up for the users to stop it. One of the reasons I think this occurs a lot and we talked about metrics already today, is that focus on metrics.

If, for example, we’re a massively multiplayer game like EverQuest. If their designers are briefed with, okay, we want you to come up with a game that people will play 18 hours a day, and they’ll check in first thing at the end of the day and last thing at the end of the day. You can create some mechanics that do that.

And if you’re focused exclusively on meeting those metrics, you will make something that’s ultimately unhealthy for our users or our players. As designers, we need to be brave enough to step up and recognize these things question the metrics, whether the things that we’re measuring are the things that are important to obviously the things are important to our business. Businesses would like people to do this, but are they healthy for our users and be brave enough to say no or find mitigating ways to make it better for our users when this occurs. That often can be very difficult. People have to put their job’s on the line to stand up and say no, I’m not going to do this. But, ultimately I think it’s part of the designers and user researchers, because we’re in the design field as well, responsibility to be looking out for users because no one in business is going to, if we’re not going to.

Jason Ogle: Yeah, I appreciate that. Let’s continue to advocate for the users at great cost, possibly sometimes. I mean, you may have to say I don’t want to design this. I don’t want to build this. I don’t want to research around this. You may need to at times, and that’s up to you use your ethics, use your instinct Defenders to weigh that out, and I’m not pigeonholing video games again, like I said, I appreciate it. A lot of these things can be edge cases, right. But the fact is the fact remains that there are potentials around our software that we design that is meant to bring engagement, meant to grab attention, and sometimes we can lose our sense of reality. Right? And then virtual reality is coming along. Like we have to be careful, I guess what I’m saying is we just, we have to be careful and we have to take a stand and we have to tactfully try to educate our business folks.

Around some of the ethical issues of it. I think that’s our job. I don’t expect our CEO. I don’t expect our VP’s, I don’t expect our CFOs. I don’t expect any of these folks to necessarily understand or be educated around behavioral science and around some of these dangers. But I think that’s our job to educate ourselves and to really be advocates for the users.

Steve Bromley: I agree entirely. Without going too deep on this is ultimately it’s a very difficult thing to do in every way. It’s difficult for designers, particularly designers early in their career to say no, because they are worried often quite rightly with some, some tech companies that you might lose your job.

If you say, I’m not going to do this. But, then all the way up to the top as well. So, CEOs and the people who are setting these metrics or asking for more engagement or more login or more time, they are incentivized by things like the stock market and the success of the business. It’s their job to do unethical things.

And so some people do need to stand up against that. And then when yeah, no one else is going to do it apart from our design friends.

Jason Ogle: Yeah, I appreciate that. Cue the Upton Sinclair quote, it is difficult to get a man to understand something when his salary depends on his not understanding it.

Right on. Well, thanks for, I appreciate you indulging me on that and and answering that and, , this isn’t a FUD factor, a fear, uncertainty doubt. I’m not trying to put any sort of dark connotation on video games at all.

I’m not, I I’m just, I just felt like that was an important area to address. And I appreciate your response to that, Steve.
So with that, I’m curious, because I know, I mean, games, this is a big industry. Obviously nobody can get us stinkin’ PS5. That’s a good problem to have if you make a product, right?

That’s a good problem. So I have a feeling. That a lot of the Defenders listening, especially our, our UXR Defenders. I have a feeling that they’re intrigued about even pivoting or even pursuing a path along games, user research You’re the guy to talk to about this. You wrote the book on it.

So I’m going to say, I haven’t read it yet, but I’m going to recommend reading Steve’s book when it’s available. We’ll have some, maybe we can do a giveaway or something. I don’t know. That’d be cool to try to explore that.

Steve Bromley: Yeah, definitely.

Jason Ogle: Definitely get ahold of his book. It’s coming out pretty soon within the next couple of months, I think. Is that correct?

Steve Bromley: Yes. It’s out in February. So, by the time people are listening to this, it will be available in shops, yes.

Jason Ogle: We’ll have a link in the show notes, Defenders. So, what’s the recommended path for one to become a video game UX researcher?

Steve Bromley: The first thing I guess to caveat is it’s very difficult field to get into even getting a user research job anywhere. I guess isn’t the most easy task. There’s a lot of people who want to work in user research, but there’s very few user research dedicated roles out there and that’s even more so in the field of games, typically only light.

I think the maturity of games, although, as we talked about in the history thing, it has started very early. I think it’s adoption has been slower in type games and other types of software. One of the barriers I think for that is games are an artistic medium, and so if you are a game designer, you are an artist creating art.

And so that discussion that user researcher has about, okay, how do we incorporate an understanding of the players experience and combine that with a piece of art can be more nuanced and difficult to get through than in other fields, like e-commerce where no one considers creating their website unfair.

I’m sure many people do consider it a great art, but some people would, I would not consider creating a website art in the same way that game designers think what they do they are doing is art. So because uh, because of that Games User Researchers have to do a lot more advocacy work and explain what they’re doing, which UX’ers have to do everywhere. I understand, but the consequence is there are not that many places that employ Games User Researchers often it’s the largest studios, so Activision or Ubisoft or Edos, or I just talked about Microsoft and Sony. And that means that someone who wants to be a Games User Researcher, often has to make big life moves, like move, where they live, for example, to make that sort of commitment.

And then not that many roles out there. Because of that, people who are interested in being Games User Researchers have to be extremely proactive, but there’s lots of opportunities out there. So, for the last five years, I ran a mentoring scheme related to Games User Research, where we paired people who are already in industry.

With people who are interested in joining the industry or just at the beginning of their career and taking advantage of opportunities like that, or in the new book, we give a lot of examples about here are some ways to practice those user research skills without having the job so that when that opening opens up and do you have to apply for that job, you have a lot of relevant experience that you can talk about and you will ace that interview because competitive few jobs out there. One of the advantages though for Games User Research is that because it’s quite a small community, it is very active and very friendly.

And there’s communities like games research, and User Experience special interest group who do things such as hosting this mentoring scheme. I’ve mentioned they have a discord channel where they have a regular event. They run conferences and they do lots of things to make the field more accessible and to help people understand actually, what is the job and how can I get those skills so I can ace an application for it.

Jason Ogle: Oh, that’s great. appreciate you mentioning that because, in some of my research before our interview, I did see that there are groups out there there’s a community for everything Defenders, as we talked about in the last episode as well, with Jessica Gaddis we talked about community, finding your community, they’re there and they’re possibly looking for you to, to lead them.
So I, I it’s great. And it’s not surprising to me that there are communities around this niche of video games user research. So we’ll link to that in the show notes, Steve, cause I have a feeling that to me, like it’s networking is so important and networking with people that are doing what you want to do.

It really makes a big difference because a they’re aware of you and B, they may be able to help you. They may be able to answer questions. They may be able to find opportunities that you didn’t even know existed yet because they’re the first to hear.

Steve Bromley: Yeah, exactly that. I should also mention they have a job’s board so it will be the place where you find opportunities because every UX’er is like this, but the Games User Research community, we are all strong advocates for Games User Research. We want to tell people why it’s interesting, want to tell people why it helps make games better. And because of that, that community of often very experienced user researchers. Are happy to spend their time talking to people about the field to help people join, to get people excited about Games User Research. And so, yeah, just reach out because there are people there who, who are really happy to share everything they know about the video ads.

Jason Ogle: That’s awesome. I feel like it’s important for me to round out my ethics statement. Video games can actually be a lifeline for some people too. Like I, this just occurred to me as I was thinking even along the groundbreaking work I know firsthand Microsoft has done for disabled folks to be able to play video games it’s astonishing it’s actually moving to see some of the commercials when you see this kid, he can hardly move, but guess what? He can play these games. He can bring entertainment, bring joy into his life. I think it’s important for me to round out there’s good and bad. I mean, you can use anything for good and bad, but I think the good always wins. The good will always win. That touches me every time I see those commercials. I’m sure that Sony’s doing things too. I just haven’t heard as much around their console as I have with Microsoft X-Box and how they’re really reaching the accessibility community and those with special needs.

Steve Bromley: Yeah, there’s been a really exciting I guess advocacy and interest in that games, accessibility space for perhaps the last five years. Things like custom hardware, like the work that X-Box has done has been really exciting and also a lot of software things. So through advocacy work from user researchers and other accessibility advocates, a lot of big game to now building in accessibility settings into the game.

So, if you have low vision or mobility issues, or you’re hard of hearing. The games will now many big games like The Last Of Us 2 which came up last year, have all the settings required so that you can still enjoy the game in the way that designers intended. Obviously that journey has been difficult because games do have friction and challenges, parts of it, a naive view that some game designers might have had in the past is that our accessibility is just making games easier and we don’t want to make games easy, because difficulty is the point. But, I think through education and advocacy, we’ve started to move the dial on that so that people recognize it’s not making the game easier, it’s making it so people can experience the game that you want them to experience in the way that has appropriate difficulty for them, which is a great ethos.

Jason Ogle: That’s so great, man. That just kinda came to my mind. I was like, I need to share this because this is Every time I see those commercials, every time I hear about the work that’s being done so that everyone is welcome. Everyone feels included to be able to experience these magnificent games that are available, I, it brings a lot of joy into a lot of folks’ lives. I really think that’s important. I appreciate that we got to touch on that a little bit.

I kind of teased this a little bit earlier about synthesis and synthesizing data. I have done I still do. I do use a research. I’m not a specialist like yourself. I’m a designer. I’m an Interaction Designer. That’s my primary specialty, and I love it. I love thinking about how folks use an interface and how to make it better, how to make it easier, take friction out. I love thinking about that in designing solutions around that. But, I also do appreciate User Research very much, because there’s so many benefits. Any of us that we dive in and we start learning about user research, start doing it like, oh my gosh, this makes so much sense. Why did we wait so long to really make this an important part of our design discipline? So all that to say, I saw this in your book. I was like, Oh my gosh. He’s like, how did you know? In Building User Research Teams. You said, User Research is a specialty, and if you’re doing design and being tasked to do user research, you’re probably going to be able to do both, but not both really well.
And I’m like, dude that’s me. I love to hear your thoughts on that too, but I am setting this up about synthesizing. I’ll be honest, I love talking to people. I love this. This is why I do a podcast. I love talking to you, and the one-on-one stuff is really great.

And when I do user interviews, I enjoy the face to face. I enjoy watching them use and even struggle with products, although it’s humbling. Once I get through all that, I honestly dread the synthesis of all the data I’ve gathered. It sometimes feels like digging a ditch to me. How do you feel about it? And can you offer any advice to help me and any other Defenders listening to love it more and reduce the pain of synthesizing more?

Steve Bromley: Yeah, I totally empathize, and I recognize that when you’ve got hundreds of post-its in front of you or hundreds of lines of texts, of what you’ve written and observed from a session that can be overwhelming and intimidating to think about, okay, how am I going to dive into this and come back with some, something interesting and relevant and actionable from it?

I think I feel it’s really important is the preparation that you do before you go into the study. So that when you end up with your data, it’s ready and easy to use. To be specific about that, whether it’s a usability test where you’re looking at some software and you know, okay, we’re going to have these screens, or if it’s an interview where we know, okay, we’re going to talk about this topic.

Let we talk about this topic. If you recognize that beforehand, you can catch your notes and put them in that section live. So every post that I write about what they thought about the graphics I’ll put in this graphics section for it example. In the book, I talk about my fondness of Mind Maps and I find Mind Maps already helpful for that, because you can predetermine what topics you’re going to talk about, capture your notes life against those topics, and then when you realize, and you’re coming to an analysis, I’ll actually, I put it in the wrong section, it’s the work of seconds to move it from this area to this area. Whereas, that’s not possible if you’re looking at raw transcripts or of fiddling with pictures or those kinds of things.

So, by doing that rough sort live as you’re capturing your notes, it helps. Mean that when you do come to synthesis, it’s already vaguely in the right sort of area, and a lot less intimidating to deal with. You still do need to then go re-read through each section, work out, okay, what are the relevant observations about this theme?

Or what did we learn about this theme and draw out the meaning from that, and then make sure you’ve communicated that meaning effectively, but it’s a lot less intimidating if you’ve done that preparation to anticipate what we’re going to talk about, and then put it into the right section. So, in general, I guess separating it into discrete tasks really helps prepare where you’re going to capture the notes, capture the notes in those sections, then group them the observations inside that section.

So you’ve only got a few observations, then reword it, treating it as distinct tasks really helps break down that intimidating amount of synthesis you have ahead of you.

Jason Ogle: Oh, that’s great advice, Steve, thank you. I selfishly received that feeling that a lot of the Defenders listening are also really paying attention to that as well. And so thank you for that. So, what about reports, do you find that you generally use the same template or do you have like several templates that you shift around and why?
Kind of things like that real quick.

Steve Bromley: No, that’s a great question. So I do often fall back on the same templates and on the website for Building User Research Teams, which I think is buildinguserresearchteams.com. We have all these templates available to download. But, yeah, it can be lazy to rely on templates as well. Ultimately, the point of that report is to communicate what you, as the researcher know with the person who needs to do something about it.

And that requires just having a good relationship with your designer your product manager, whoever the audience is, and then finding what works with you, whether that is a report. Can be a conversation. Often I just go through the mind map with the person I’m talking to, and you don’t need to write it up. Yeah, what works with your relationship is what’s important.

Jason Ogle: Very nice. Jakob Nielsen wrote an article on Nielsen Norman Group. I’m sure you’ve read it about how many users is enough and the takeaway from that article on research that he’s done on the fact that six or seven, generally is enough users for most of what you want to learn, because of the law of diminishing returns. Once you start exceeding seven, eight, nine, 10, you start to see the same results or similar results.

I think the implication there is, you might be wasting your time, if you keep talking and spending an hour, it takes me at least an hour to do a one-on-one. I’m more of an introverted personality. I’m an ambivert. So it’s very draining for me. Emotionally is draining. It’s exhausting. So I never scheduled more than two moderated interviews a day personally. Cause I just can’t take it. I’m like I’m ready to fall asleep after the first one. So, all that to say, what’s your take on this on that statement and that article from Jakob Neilsen, I quite call him Uncle Yakob, and, and how many users is enough, Steve?

Steve Bromley: I have a classic UX’er answer to that question, which is it depends. And what

Jason Ogle: I knew you were going to say that.

Steve Bromley: It depends on your research objective. So. Yep. I actually say everyone’s in the article and there’s some important parts about the article that we need to keep in mind when we’re deciding is five to eight users enough he’s describing for usability studies.

So. If we’re testing some software. Yes. After five to eight users, we will have seen a whole bunch of issues and it’s probably more worth what you’re while fixing those issues. And then retesting then going on for another 10 or 20 users, because at that point, you know what the issues are. You’re just wasting time when you could be fixing usability problems.

But some bits that we might forget about when we’re just quoting five to eight users, Is it just depends on the research objectives. Not every study is a usability study. Sometimes you want to learn about user behavior in the real world. So for example, I mentioned a taxi app earlier. If you were inventing a taxi app, you would want to understand, okay, how do people go about ordering taxis currently?

And what’s their experience. And for those types of studies, you want to first identify, okay, what types of users exist out there? Taxi are people who are ordering a taxi. Oh, different from the taxi driver, so that at least two audiences, and we need to talk to both of those. And then how many of each audience do we need to speak to depends on too many factors that you can’t guess in advance. So. You’ll start speaking to people and after five to eight users, five to eight taxi drivers, for example, you should take a call on, okay, am I hearing the same thing again and again? Have I understood everything about this, or am I still learning new things?

And I haven’t yet hit that saturation point where it’s just repeating the same information again. And for those type of studies you can’t come up with a hard and fast rule in the same way, because you don’t know how different that audience are until you get in there with them and start talking to them, and then only then we know, okay, now I can see stop when I’ve heard it all before.

Jason Ogle: Awesome, I appreciate that. So, before I get into the Super Seven, I like to go below the surface and I think that’s why the Defenders like to listen to the show because it’s not just about design.

It’s not just about research. It’s not just about UX. I think that, and I believe, I don’t just think, I believe that being a great designer and UX person, begins and ends with being a great human. It really does and empathy that’s all part and parcel, right? This work requires empathy for our users and advocating for them, fighting for them.

I want to know Steve, what’s your why? Like, why do you do what you do?

Steve Bromley: Oh, that’s a great question. It’s a really interesting space to work in. All you do all day as a user researcher is understand real people’s life’s experience in different contexts. And, that’s tremendously interesting. There’s always new things about learning, okay, these are all real people doing real things, and they’re all different from me.

And so I guess it exposes you to how broad users as a category are, and what people are like in the real world. And I think that’s endlessly interesting. I also think that it’s nice to be able to make experiences better for people. What you’d learn from any usability study is that there’s problems with almost every piece of software. Everything is confusing to someone and, by doing this job, we can make things less confusing to people and reduce the amount of pain in the world. So I think it’s that combination of learning how interesting the world is, and also trying to make that world slightly better for just that few number of people who are using your software.

Jason Ogle: Oh, thanks man. That’s good. That’s inspiring. Yeah, and curiosity is really a big part of that, right? You really need to stay curious and hungry to really continue to thrive and do good in this area.

Steve Bromley: Yeah, exactly. I guess you can’t assume you’ve heard it all because then you stop listening to people. And then, what’s the point of running these studies if we’re not paying attention?

Jason Ogle: That’s good stuff. So, let’s jump into the super seven before we wrap up here.
Steve, what’s your UX superpower?

Steve Bromley: Oh yes. So. I think it’s reflection. I think we haven’t talked about yet is I I’ve been very lucky and worked with some very good researchers in the past. Say for example David Tisserand was one of the senior researchers who I worked with at PlayStation. And what he taught me was how important it was to recognize that research is a semi-structured process.

You always do recruitment. You always design your study. You always moderate a session. You won’t analyze your findings, and always present it. And if you recognize it, it’s all these discrete phases. You can then interrogate. Okay, what went well about each of these phases? What didn’t go well and what can we do differently next time to make the study go better or to learn more interesting things.

And I think with that core skill of reflecting on the process and then making changes to do it differently next time. That’s all you need. You can start from no research process at all and iterate your way up to it very quickly. And, I guess this is one of the themes of the first book Building User Research Teams is how do you recognize those phases and how do you continually iterate to make them better?

Jason Ogle: Oh, very cool. Reflection. That’s great. And I think in the world that we live in, that’s again, constantly vying for our attention. I think that seems to be somewhat of a lost skill. And a lost art. I think that we’re constantly moving. Um, This is an assumption, but I’ll speak for myself. I feel like the world we live in, which is wonderful and we’re, grateful to have the technologies we have, again, as mentioned previously, but with that, we tend to forget to slow down, even stop and just reflect and think. And, I appreciate that because that’s not just a good UX skill. That’s a good life skill for Pete’s sake. So yeah. So that’s, I wanted to just kind of comment on that. I think that that’s a great skill altogether.

Steve Bromley: Yeah, something we can practice in every aspect of our life. Like you say.

Jason Ogle: What’s your UX kryptonite, conversely?

Steve Bromley: Hmm. I guess the thing I don’t. Like to see and try and avoid is the idea of inefficiency, which I guess plays back to those reflection themes. So, I think fundamentally I, like perhaps many people, am quite lazy. And, so I’d like to deliver the results in the most effective way without wasting time or wasting effort. What I’m trying to achieve through this reflection is identify what can we do differently or stop doing that is a waste of our time. Because of that inefficiency in the design process is something that I try and avoid.

Jason Ogle: There’s a lot of different points to that that are interesting to me. One of them is in UX. We’re told to fail fast, right? We should be encouraged to fail. And yet, some, especially higher ups may look at failure as inefficiency. So there’s like a weird dichotomy there.

Right? It’s like, you really should be trying to fail, but fail fast. I think we’ve framed failure in the wrong light, especially again, older school folks, our higher ups or what have you. They may not feel like there’s enough room in the budget be inefficient at least initially.

Like, can you speak to that? Do you have any thoughts on that, that dichotomy there?

Steve Bromley: Yeah, I guess it takes a certain amount of design maturity to recognize the value of failing. And to recognize that by doing this, we are ultimately gonna get to our destination faster. Because, we have learnt something from this experience, we’re going to do something differently, and we have that same North Star goal that we’ve always been aiming for, but our approach is different.

And then, through education about the design process, we can hopefully help people understand it. Although, we’ve called this a failure, this isn’t a step back. It’s a step towards the goal that we’re trying to reach.

Jason Ogle: Yeah. I like that. I think Facebook’s mantra for a long time was “Move fast and break things”. I don’t think that’s helped our field. I don’t think that’s helped software very much. I’m of the belief that you should, we should move slow and fix things, but the reality is some of the biggest innovators in the world are maybe at heart, they might be lazy bums.

Steve Bromley: I know I am.

Jason Ogle: They’re tired of doing things manually. And I mean, I’m the same way. So you know, there’s a lot to glean there, I appreciate that, Steve.

This is a fun one. What would your UX superhero name be?

Steve Bromley: Yes. The slightly corny name, Reflector. So in, in honor of our idea of let’s look at what we’re doing in the research process, let’s measure its success. Let’s adjust and learn and make changes. I think that idea of reflecting on the research process is that ethos of the first book and something that I’m still extremely enthusiastic about.

Jason Ogle: That’s great. I love that. Reflector. Can’t wait to see your art Eli Jorgensen’s art.

So, what’s one habit you believe contributes to your success?

Steve Bromley: Yeah, I, I’m a big fan of documenting what I’m doing as I’m going. So, I’ve been lucky in my roles where I’ve been in a position to set up a number of user research teams. And, by writing the templates and the processes, as I’m establishing them, that becomes a learning resource that as the team scales and grows, people can learn from what you’re doing.

And, you wouldn’t have time to go back to these things. So, after you start to hire people and grown your team. You’ll never have time to go back and write up how we do research or how we recruit participants or how we analyze things. So, doing that as you go along, I think is tremendously valuable, and the thing that’s paid off, a number of times in my career,

Jason Ogle: That’s great, man. Yeah. Documenting what you’re doing. That’s good.

What’s your most invincible UX resource or tool you could recommend to our listeners?

Steve Bromley: Is a book appropriate or

Jason Ogle: Absolutely, yeah.

Steve Bromley: A book I think is extremely useful. Perhaps you’ve said it before, but Just Enough Research by Erika Hall. I think if you’re, it’s fantastic, isn’t it. If you’re a solo researcher or a member of someone representing research and organization. It’s quite a brief book, and it just tells you everything you need to know about, here’s the ethos behind research, here’s the activities that you could be doing. And, it is a no no-nonsense guide to explaining, this is how you be a user researcher. I recommend it to everyone who joined the team that I’m on or who I work with as a user researcher.

Jason Ogle: So my last question for you, Steve is, what is your best advice for Defenders looking to get their foot in the door as a UXR or user researcher? And let’s even say as a games, user researcher.

Steve Bromley: Yeah, I think this goes back to some of the things we talked about already today. So, my advice is find community, the community for games, user research. But, as you said, there’s a community in every aspect of UX and design. I imagine. Be open with that community, tell people you’re new, you’re learning and just be proactive, enthusiastic about, okay, I’m on my journey. I’m going to make some mistakes, but taking advantage of those people around you who are there to help. It’s extremely valuable. Yeah, so that’s my advice for people who are interested in becoming games user researchers.

Jason Ogle: Awesome. That’s great advice. Before we wrap, do you have anything else you want to share or anything else you want to leave us with?

Steve Bromley: No, I. Thank you very much for taking opportunity to talk about this today. As I’ve mentioned, a number of times the games user research community we talked about things, research and user experience, special interest group. That’s fantastic. So check them out. We also mentioned the book, How to be a Games User Researcher, a lot of the things you talked about go into more depth in the book.

So that’s also a great place to look. Or if people want to chat about games, user research, I’m always available on Twitter. Steve underscore Bromley.

Jason Ogle: Perfect. And Defenders definitely keep an eye out for the book. I’ll link to it in the show notes. So you’ll be able to pick it up there. Is it going to be on Amazon? I, I imagine so. Okay, great. So Defenders, if you pick it up through the show notes page, I have an affiliate with Amazon, so I’ll get a, a really small piece of that pie at no additional cost to you. So please buy it from the show notes, userdefenders.com/076. I believe. I hope I got that right.

So, yeah. Uh, Steve, thank you so much, man, for being here. This has really been insightful. I’ve learned a lot personally and I know the Defenders have as well.

This is really exciting stuff. I feel like this area of user research is very nascent. I feel like there’s a lot of growth happening here and we’ll continue along this trajectory again.

Try to get a PS5 Defenders, try, just try. I dare you. And if you have one already, you’re you’re lucky. So, but yeah, this is a really interesting, exciting field. I’m going to continue to keep a pulse on it and definitely excited to read your book, Steve myself. So that’s that’s about it, man. Thank you so much. And as always last but not least, I just want to say, fight on my friend.

Hide transcript