Getting the Most from Virtual Playtesting

Having been involved in a large number of online playtesting sessions over the past couple of months I have a few thoughts on how we can all get the most out of virtual playtesting sessions.

We all (well, many of us) have a lot more time on our hands; that is one of the reasons virtual playtesting is on the increase. But time remains a limited commodity. If 4 designers meet to playtest each other’s 4 player games, and each game takes 2 hours that is potentially a mammoth 8 hour playtest session. I appreciate that many of us have more time on our hands at the moment, but anything we can do to cut down wasted time, will benefit all of us, and will in fact lead to better feedback. I have lost count of the number of playtests I have been involved in, where the feedback has been dominated by how long the game took, or how slowly it played.

So, a bit like my recent blog post on how to give good feedback, this post is first as a reminder to myself, about how to respect the time I am asking of fellow playtesters. However, I think that some of these suggestions are based around smoothing the overall process and trying to reduce time being unnecessarily wasted. They certainly are not intended to be rules – but I have framed them as questions that I might want to have thought about, going into a virtual playtesting session.

1. Are Steam and Tabletop Simulator up to date?
2. Have I set the table up already? And for which player count(s)?
3. Have I been realistic with the requested time slot?
4. What player count do I need?
5. What is the purpose for this playtest?
6. How am I going to teach the game?
7. Am I committed to managing the allotted time well?
8. Is the ruleset fixed? What should I do if the game is broken?
9. How will I be managing discussion?
10. How do I want to hear feedback?
11. After Feedback

Before the Playtest

Are Steam and Tabletop Simulator up to date?

As online traffic has increased dramatically over the past couple of months, in order to save bandwidth, many programs on Steam have changed the way they offer updates. Rather than filtering those down continuously, they are now only downloaded when the program is opened. I have lost count of the number of times I have tried to open TTS only to find that it has taken 10 minutes or longer to update and open.

I would suggest anyone joining an online playtest group, opens Steam and Tabletop Simulator (and even Discord) at least 15 minutes before the scheduled time, and if you have not opened either recently, maybe allow even longer.

Have I set the table up already? And for which player count(s)?

I have joined some playtests where playtesters arrive to see neatly stacked decks of cards and tiles and the first 10 or 15 minutes is taken up by the designer setting up the game to play.
Tabletop Simulator offers unlimited save points such that it is easy to save a different set up for each player count in the game. You can create a huge play table and have the game set up for different player counts at each end. You can create tutorial areas with duplicated components so you can show players what the game might look like.

You do not need to assume that players are sat around a table. Think about it. If it is easier to read player boards a certain way up, then everything can be the same way up for every player.
Don’t treat the virtual space the same way as your kitchen table and don’t pretend that you only have one virtual copy of the game which needs to be put away at the end of each session. 

Have I been realistic with the requested time slot?

Playing virtually always takes longer. I’d estimate at least 30-50% longer than playing physically. If we add on time to explain the rules and time for feedback at the end then you could easily expect a playtest to take twice as long as playing a physical game. We all tend to underestimate, but we should all try and get better at this.

An indicator that you have taken the limitations of virtual environment seriously is to compare the length of an ‘in-person’ playthrough with the time requested. “My game plays physically in about 45 minutes. I’d expect the playtest (with feedback) to be over in about 90 minutes.” 

What player count do I need?

This should depend on where a game is in development.
I think we all want to get as many people as possible to play our games – that is why we design games. And we also want as much feedback as possible – that is why we playtest games. But more players take longer and during the earliest stages of playtesting it is often the case that feedback tends to reach consensus.

At this point I can’t help but look at the maths…
If a game takes about 20 minutes per player then a 3-player game will take 60 minutes and a 4-player game 80 minutes. But in terms of the committed resource of playtester time, the former takes 180 player minutes and the latter 320 player minutes. And a 5-player game, 500 player minutes! This means that a 4-player playtest adds 140 player minutes to a 3-player playtest and the 5-player game adds 320 minutes (i.e. a whole new 4-player playtest!).
If you are going to try and put as much into the community as you take out, then you could ‘owe’ 6 to 8 hours after a 5-player playtest.    
Unless you are really looking at scaling, do you need the full complement of players? As a designer you might legitimately have a reason for playtesting at a particular count, but if you are at the point where you need to test with 5 or more players, I would expect the gameplay to be very solid at lower player counts, and the focus should be on minor tweaks to adjust for the number of players not basic mechanics.

What is the purpose for this playtest?

It might be the case that I just want to get another playtest under my belt, or have other designers cast a critical eye over it, but depending on the stage of development, it can also be helpful to focus on a particular aspect of the game. Do I have a specific part of my game that I want to test?
Have a look at what Ignacy Trzewiczek says about the differences between playing and playtesting a game. 
He also distinguishes between the objectives that he will share with playtesters and those he will keep to himself.
Or similar thoughts from our own David Digby here.

You might want to ask one of the playtesters to fulfil a particular role “Can you please try and get as many points as possible from the corn strategy?”

“I’d like it if you can just focus on combat and preventing other players from trading.”

This can make your playtest immensely more valuable, but it also has the additional benefit of giving your playtesters (who may well be designers in their own right) something valuable to do. No longer are they just making up the numbers, but they are taking a role in the development process. They have a job to do and something to report back on.
As a playtester given such a task I would not feel like agency was removed, I would feel valued and empowered.
In addition, players tasked with a specific objective might not need to fully understand the entire ruleset to make meaningful decisions. I do not need to understand the nuances of the trading system if my role is to try and stop other players from trading by blowing them up.

During the Playtest

How am I going to teach the game?

This is sometimes the biggest time sink in a playtest. I have spent 45 minutes listening to rules before I ever touched a card.
I have shared before, that I feel the most useful piece of feedback – something that a designer can immediately learn from and put into practice – is how well did they teach the game. And frankly most of us aren’t great at it.
One reason for this is that we are trying to give other designers as much information as possible to play the game.
I think that we need to approach early playtests, especially those conducted in a virtual environment, with the mindset of a demo rather than a ‘teach’.

First however, I should admit, I am bad at learning the rules of a game. Really bad!
Actually, I think I do myself a disservice. I am really bad at learning the rules of a game the way most people teach. When people just point at different parts of the board, explaining what goes here or what that does I involuntarily switch off. In the virtual world this is even more the case. I cannot always see where the ‘teacher’ is pointing, or even looking. I have learned that if I wait until the game actually starts, I can usually learn the rules during turn 1 when I start to see things moving round the board. I usually accept that my first play is a bit of a learning game. Although I am fully aware that many players want to know everything before they start I think the best way to teach everyone at the table (whatever their learning style) is to control a turn or two, taking people’s actions for them and explaining why and how.
This tends to be the way the pros teach, or the way publishers will run a good demo at a convention. I’d even suggest the fact that so many people gravitate towards the pro teachers on YouTube who generally tend towards my preferred methods, suggests that I am not in the minority.

I think one of the reasons for not wanting to do this is to give players their own agency. We don’t want to take their turns for them.

But;

• Done well, I could demo 2 turns each of a 4 player game, in a few minutes. If players did not like the position in the game they have arrived at, we could reset the game and start again, in less time than most rules explanations take.   
• This takes us through the less interesting parts of a game very quickly. Many games tend to get more interesting from turn 3 or beyond. Unless your game is an exception, or you really need a playtest to focus on the first turn or two, then you might find this gets you to the best parts of the game, and the most useful part of a playtest, more quickly.
• Finally – but perhaps most importantly; Playtesters are not here to play your game. They are here to playtest your game. If the two are not different in your mind, then you might want to reconsider.   

Am I committed to managing the allotted time well?

Many playtests overrun. It’s expected. But should we collectively aim to be better at managing our time?

Knowing when to call a game – either because the playtest has yielded enough information or because the game end has become predictable is a useful discipline to exercise. And it is far better to call a playtest early than wait until playtesters have become frustrated and bored.   

Also, managing the turn structure can be helpful. Encouraging players to say “ turn” when they have finished, or encouraging the next player to start, while the previous player is still dealing with the steps of moving components around, which become more time consuming in a virtual interface, can help the game move at a clip.
This does two things – both of which are to your benefit as a designer. First it avoids wasting the limited time you have – both yours and your fellow designer/playtesters. Secondly, it helps the game to play almost as fast as it would in the real world. This means that comments about the pace of the game and how long it feels are less likely; but if they are made, they are more likely to be relevant.

Is the ruleset fixed? What should you do if the game is broken?

I love changing the rules of a game during a playtest. It makes me feel that the playtest is achieving something – especially if the new version works better. Obviously, this depends on the game, where you are in development and whether the needed changes are clear, but if there is an obvious problem, pursuing the playtest to the end might not be the best use of time, and making a direct A B test within a game can generate immediate and useful feedback.

After the Playtest

How will I be managing discussion?

I have already written a blog post about how I try and give feedback. As the designer, and host of a playtest session, you are in a position to control the way playtesters respond to some extent.
Some discussions can start during the game – especially if they give players something to do when they are not playing. Asking about game play choices and decisions (often revealing the secret player information that players would usually keep to themselves) seems to be perfectly valid during a playtest. John Brieger refers to this as a “Think-Aloud Protocol” in his playtesting talk to the Ignite Conference.

However, if discussions begin to slow down gameplay then they should probably be shut down until the end. That’s not to say that the end won’t be brought forward; but if players have to wait 5 minutes for their turn because the player preceding them is carefully dismantling your action point mechanism that is probably not a good idea. 

How do I want to hear feedback?

When the play finishes can be when the playtest ‘proper’ starts, and how the designer manages the post-match analysis can be important.
Some designers may just sit back and listen to the discussion at this point, but that is to give up the opportunity to control the way this discussion takes place, and even the feedback that may be received. It should not be seen as a negative if the designer wants to impose a certain order on proceedings, rather than opening the floodgates to whatever the loudest voices have to say.
Developing a local culture within any playtest group will be helpful to all.

If the designer is trying to control which feedback comes when, they should outline this at the start – playtesters will be more likely to sit on their thoughts, if they know that they will have an opportunity in due course.

A general introductory question can help start the ball rolling, if players aren’t immediately forthcoming. I like to start with “How could I have taught the game better?” or “Was there anything in the rules explanation which didn’t make sense at first?”
This can also help set a tempo – moving round the table and giving each playtester the opportunity to speak; thereby avoiding one playtester dominating the discussion and leaving very little for others to say.

If the discussion allows, then I will continue to work through specific questions, managing the input from playtesters and asking for input from those who have not spoken.
If you have given playtesters a focus at the start of the playtest this may be easier – if they have been asked to run a particular strategy then they will naturally comment on how that part of the game felt, and may in fact be the only person able to give feedback to that specific aspect.

I don’t think I have been involved in a playtest where at some point, the discussion does not become more free form, as participants unload all their ideas about how the game could be improved. While it is important for the designer to listen to all input, it is not necessary to be defensive or justify design decisions you have already made. It is well known that while it is important to try and understand how your playtesters feel about the game, and why they may feel that way, there is no obligation to accept the solutions they offer.

After Feedback

It goes without saying that you should have written notes during the feedback process.
Playtesting should not be about just ticking off another play of the game, but about improving it. I’m sure I am not alone in having playtested a game, and then forgotten the feedback received until it was raised again in a subsequent playtest. You might not iterate after each playtest, but if a playtest has revealed nothing that you want to reconsider then you probably aren’t playtesting hard enough!

Finally, don’t forget to thank your playtesters.
Processing the advice given on the spot, suggesting what you might try in a subsequent iteration of the game, is one way to show your appreciation.
But perhaps the best way to really show your appreciation is to fully engage with the community where you are playtesting and immediately jump into another designer’s game. It doesn’t happen very often – but you don’t want to be that playtester who shows up, plays their game, and then vanishes.  

This is the first in a series of blog posts about virtual playtesting. Others here;
Playtesting in Pyjamas
Giving Playtest Feedback
Getting the Most from Virtual Playtesting

Since writing this I have been involved in establishing the Virtual Playtesting Group. We meet on Discord on Thursday evenings (6pm UK time) until late, and playtest in Tabletop Simulator or Tabletopia.
With moderators in time zones from Eastern Europe to the Pacific, we welcome you to bring your online prototypes to playtest with other designers.

Here is an invite to the Discord Server.
https://discord.gg/Ze9mBWc