Focus Tests

Ugh, time flies so fast. Been so busy lately. But I figured I might as well post again before I get bruised up from all the pokes I've been getting lately for neglecting this page! I'll have to tell you what I've been up to in my next post. For now, I might as well finish this draft that has been sitting here for months!

When I was studying at the Ubisoft Campus, one of my design teachers' main job (when not teaching) was running focus tests. He had this big team to perform the tests and I thought it seemed a bit of an overkill. Then a few months ago, I had to organize and run my first focus test. Was I ever glad he had explained to us how he used to run his. At the end of the test, I had to admit that not only everything he did in his tests was justified, but the wealth of information you could gather was unbelievable.

We wanted to assess the difficulty and appeal of a new feature we're about to put in the game for the 5-12 year old players. Usually, booking a room and getting it all setup for your needs seems to be the toughest part as everyone and their brother always want the same room you need. But in this instance, that was cakewalk. The toughest parts were:

- Rounding up enough kids (boys and girls) of the right age group
- Putting together a good test for us that would also retain their interest
- Coming up with a comprehensive (but non-leading) questionnaire

When the kids arrived, I remember getting a major knot in my stomach. As I'm the only woman on the team, the guys all decided (without consulting me!) that I would be the best suited to deal with the kids! I love kids but have none of my own and have never taken care of so many at once. Fortunately, they were the very receptive "I don't need to cling to my momma's skirt or start crying for nothing in the presence of strangers" type. I was more nervous than they were. They were just eager to start playing :P

Quick explanation of what the test was about. Insert a few "there are no bad answers or performance" type speech, then we're ready to start. I was nearly pulling my hair in the first few minutes. The kids were so impatient that while I was logging them in, they would start hitting random keys on their keyboard or click that little X at the right corner, or refresh their screen... So yeah, I had to log a few of them back in, 2-3 times in a row! >.<

Eventually, we're all there and ready to roll. The kids start playing and remembering my teacher's description of his own tests, I start moving from one kid to the other, observing their reaction, body language, unconsciously spoken words, comfort (or lack thereof!) with the mouse, etc. It was incredibly revealing.

The boys were extremely vocal. So you didn't need to be next to them as much to know if they were happy or annoyed. A lot of mumbling (both positive and negative), heavy sighs of exasperation or shouts of triumph. The girls on the other hand were a lot quieter. They just focused on their task and their frustration or happiness was usually shortly displayed on their face: a short pout, a pleased smile. While the guys would flat out ask how long they have to keep doing this or if they can stop when they would get annoyed, girls would simply get restless on their chair or looking around to see if others had stopped, thus making it okay for them to stop too.

Performing the one-on-one post-test interview helped clarify a few things, but nothing imho was more revealing than their body language. But more importantly, I got a whole new respect for play tests because of all the things you take for granted but that isn't so obvious for your players. The most eye opening one was regarding chat.

We had a 6 yo girl trying to answer a 9 yo girl who had asked her a question in chat. The 6 yo was taking forever to type. Her spelling being very limited, she was trying to type things phonetically. Since we had a very strict language filter to protect the kids, half the words the 6 yo typed were being rejected, making it even harder for her. By the time she was ready to hit send, the 9 yo had tired of waiting and walked away. The 6 yo was so depressed, it was heartbreaking.

Other silly things such as the kids don't play this specific mini-game not because they don't like it but because it requires a dexterity with the mouse that their tiny hand doesn't have. Just slowing it down a little changed it from frustratingly hard to fun and exciting. The funniest thing though was this girl staring at her screen for a few minutes then clearly starting to be annoyed. When we asked what was the problem, she said "it's broken, it's been loading forever". When we asked why she thought it was loading, she pointed to the progress bar of the challenge she was working on and surely enough it looked a lot like a loading bar though it wasn't!

I'm now a strong believer in focus tests. But like in every other type of survey, when trying to interpret the results, you need to be as objective as possible and not try to make them say what you want instead of what really is. In the end, it's all about adjusting the game in a way that will make it appealing and enjoyable for the players.