The illusion of skill: how to make a better game in less time

The following article is a converted script from my video “How to Make a Better Game in Less Time”.

Hi, I’m Casey “Boz” Weeks and I’ve been making games for over ten years now.

Way back when I started making games, I spent two years agonizing over every detail of my first game. Once I had a map, skill tree, boss fights, hours of content, and a tutorial, it was finally ready for playtesting.

Now if you’re an indie dev and that last sentence didn’t make you cringe, this article is for you.

You can also watch the video version of this article here.

Many Indie devs, especially beginners, underestimate the real value of playtesting.I wrote this to amplify the importance of getting feedback and give you some ideas on how to approach it.

The short answer to why you should playtest: Playtesting increases the quality of a game design per hour of development.

In other words, you can make a better game in less time.

Quality of Game = Dev Time + (Dev Time * Playtesting)

Before we begin:

Playtesting often gets mixed in with Quality Assurance, so I want to point out the distinction.

Quality Assurance testing (or QA), is about Functionality. Finding bugs, typos and other errors.

Whereas playtesting is about putting your game in front of fresh players to see where the magic happens. Playtesting is for proving appeal.

If you don’t playtest, my hope is that the first thing you do after reading is schedule one. And if you do playtest, then I hope this article helps you playtest even harder.

Most indie devs seem to know they should be playtesting, but, whenever I ask if they are, I often hear a lot of excuses – myself included. The excuses usually center around the following sentiment: There’s no time!

But what if I told you that the designers of the most successful games consider playtesting the highest priority in game development?

Part 1: Everybody’s Doin’ It

Mike Ambinder, Valve’s former Principal Research Scientist says “For us, playtesting is the most important part of the game development process. … it is the dominant factor that shapes our decisions about what to release and when to release it.”

I reached out to him to see if he had more to say on the subject.

Mike Ambinder:

Every game design is an hypothesis, and playtests are experiments.The goal is to get as much information as you can to make the most informed decisions possible about the design of your game.The goal is to make a game that is appealing to the people who play it.Best way to do that is to put the game in front of people who are not you. Then iterate and improve the games as a consequence. The goal is to get it in front of people as quickly as we can and see if we’re on the right track.

Mike emphasizes that playtesting is a scientific tool to assess uncertainty and that the answers lie within your players. He had a lot of other amazing things to say, so we’ll hear more from him later.

I often compare game design to blindly stumbling forward in the dark. In this metaphor, playtesting is like a distant lighting strike, briefly illuminating the landscape, allowing you to check your path and make adjustments. (or reveal you’re about to walk off a cliff)

Playtesting is like checking to make sure you’re not about to walk off a cliff.

Schell Games has the following guiding principle: “Playtest Constantly. It’s the Only Way to Know.”

In a 2024 GDC talk, Francisco Souki, the Principal Game Designer of Schell Games credited the success of their games to their rapid player testing workflow. (We’ll come back to their workflow later)

“Playtest Constantly. It’s the Only Way to Know” exemplifies this idea of constantly checking your path in the dark. Playtesting steers your development process towards what players connect with. What the Schell team realizes is that playtesting prevents you from planning too far into the future on just a hunch.

Richard Lemarchand, co-lead of the Uncharted series, says that while his focus in game design today is playtesting, it wasn’t always seen as an invaluable tool.

In an gamesuserresearch.com interview he said, “in the ’90s and even in the early 2000s, playtesting and especially user research, wasn’t very prevalent.”

In fact, it wasn’t until he was working for Naughty Dog in the late 2000s that he says the importance of play testing really “clicked” for him.

The struggle to make playtesting a part of the dev process is touched on in the book “The Secret Science of Games”. The author, John Hopson, has been a research lead for Halo, World of Warcraft, Overwatch, Destiny, Fortnite, and more.

In his book, he writes about how during the development of Halo, Bungie had an initial hesitance to bring playtesting into the fold. But as they began to see the positive results, successive titles allowed playtesting to become more and more integrated with the development process.

Knowing it took the industry a couple decades to catch on to playtesting makes me feel a little less dumb for waiting two years to start testing my own game.

I reached out to John to see if he had more insight into why it took the industry so long to embrace playtesting and he pointed to the growth of online feedback post-release.

Before internet reviews, game developers used to ship games and rarely hear from players.

Now, when a game is released, they receive an onslaught of player feedback on the internet.

He believes that today it’s easier to convince game developers to incorporate a more structured version of player feedback earlier in the process to avoid problems with angry players at release.

This may be more of a AAA problem than an indie problem, but if you needed another reason to playtest: just imagine an onslaught of angry reviews of something you could’ve easily avoided by collecting feedback before launch.

Part 2: How to Trust Yourself

Daniel Kahneman was a nobel prize winner who (despite being a psychologist) became known as the “grandfather of behavioral economics”. If you’ve ever heard of cognitive biases before, he and Amos Traversky were the first to introduce them.

Kahneman says, “Intuition feels just the same when it�’s wrong and when it’s right”

Kahneman writes that academic researchers have repeatedly confirmed that professionals often contradict their own prior judgments when given the same data on different occasions. Kahneman calls this “The Illusion of Skill”

But that doesn’t mean you can’t trust your intuition.It’s not until you are confronted with failure, adapt, and then find what works that you begin to build an intuition you can trust.

Kinda sounds like playtesting, right?

Mike Ambinder:

There are some of the best game designers in the world at Valve. They’re wrong all the time. Just like everybody is wrong all the time. As smart as we hope we are.The smartest game designers in the world still benefit from playtesting.

I’ve worked with some of the best in the world at Valve and they felt the same way.

In his book, Thinking Fast and Slow, Kahneman says intuitions must be based on real expertise. And in order to develop expertise, experience isn’t enough – it must exist in a context of rapid and clear feedback.

We can turn that into the following algorithm:

Trust in your Intuition = Time Spent Gaining Experience * Speed of Feedback * Clarity of Feedback

More feedback and experience, better intuition.

Imagine a computer programmer and a wealth advisor, both with 20,000 hours of experience in their respective fields.

Which of the two do you think has accumulated the most expertise in their field? Who can make the best predictions about the results of their recommendations?

Kahneman was once given a rare opportunity to analyze 8 years of data for 25 wealth advisors.

Here’s what he discovered:

“…differences in skill were not to be found. The results resembled what you would expect from a dice rolling contest, not a game of skill.”

The reason for this? Clarity of feedback.

The stock market is unpredictable by design. If it was possible to predict the market, it would be completely upended – it wouldn’t work. The feedback is too ambiguous to know exactly what you did wrong or right. There’s little room to adapt.

The computer programmer on the other hand? The compiler will almost immediately tell them if the code worked or not. And they can measure how code revisions improve performance data.

The data is unambiguous and clear.

So whose expertise do you trust more?

Let’s turn that formula into terms of your current game design:

It’s probably obvious from looking at the algorithm that the more often you test and the clearer the feedback, the more you can trust your game design.

Think of playtesting as a multiplier buff on your insight score per hour of game design.

Conversely – if you test too infrequently, you can counteract the amount of time you’ve spent working on your game. (remember that bit about a crack of lightning revealing you’re about to walk off a cliff?)

In other words, playtesting helps you get more productivity out of each hour of game design.

Okay, we know we want rapid and clear feedback, but what does that look like?

Part 3: Speed & Clarity

How often should you be testing?

Well, that depends where you are in the project and your resources.

Remember earlier when I mentioned Schell Games and their rapid player testing workflow?

Francisco Souki says their team aims to have a build ready by the end of every week. Their testing team then puts the build in front of outside testers over the weekend. They start every Monday discussing what the test sessions taught them and then plan what to focus on for the week. All of this will culminate in a new build on Friday, ready for testing.

Others wait til they’re nearing a milestone on a feature and try to iterate as fast as possible at the end.

In my projects, we often enter milestones where we are working on a feature for a month or two, and as it NEARS completion, I begin scheduling tests.

I then aim to playtest as often as I can make updates between playtests. This could be anywhere from a new test within an hour to a test each day to taking a week to make more serious changes.

Obviously, your ability to test will depend on how long it takes for you to build what you’re wanting to test. But you also have to consider balancing the time you spend managing playtesting with the time you spend building your game.

In the case of Schell Game, while they have people who handle the testing and create reports for the designers, the designers still need to watch or attend the tests for best results.

You choose a style that works best for you, but my advice is: don’t get caught up thinking everything has to be perfect before you can test. (more on that later)

How many people should you test with?

Mike Ambinder:

The short and unsatisfying answer is as many as you can. You make do with the constraints you have. The more people you get data from, the more insight you will be able to accumulate.But, if you only get four people, then you make the best of those four people. It’s just being cognisant that if you haven’t surveyed a large amount of people, then your data is going to be noisier.

Fortunately, the more often you test, the more perspectives you bring to your design.

Clarity of FeedbackUnlike the stock market, Game Development can receive much clearer feedback. But, there’s a trick to it.

Adriaan de Jongh touches on this topic in a 2017 GDC talk called Playtesting: Avoiding Evil Data. To summarize, there are different ways of testing that can give you useless, confusing, or incomplete data. In the talk  he discusses how to get the most from your playtesting.

He talks about ways to collect analytics:

Surveys, in-game player data, and heat maps.

These can tell you where players are having problems, and it’s good feedback to collect, but it’s far from the whole story. What this data doesn’t tell you is WHY players are having problems.

He says, by far, the best way to get clear feedback, to understand the WHY, is to watch people play your game.

Here’s a link to his video: https://youtu.be/6EUeYu0aPn4?si=N-Iw_EIRbjiN__hy&t=0s

I reached out to Adriaan to ask if he had more advice specifically on how to improve clarity in feedback. He emphasized two main tips:

Don’t focus on what players say, focus on WHY they are saying it

These are powerful tips, and I want to dig into each separately.

1 – Be Open to Feedback

This involves getting out of your own way, which is a lot harder than it sounds.

Getting clear feedback is a journey of self-discovery. It’s about building skill with listening, empathy, player psychology, and overcoming your inner biases.

Adriaan says

“…it’s likely that some parts feel ‘core’, something that shouldn’t change. it takes real courage to look at someone playing, recognize that that core isn’t good enough or what you had hoped it would be, and make (drastic) changes to improve the game’s experience.”

You have to balance what you wanted from the design, and how the players are reacting to it. It’s a compromise between your vision and what your audience really wants. There’s a dangerously fine line between your ego and good results, and it’s experience and data that will help you navigate this problem.

Mike Ambinder:

Another way to think about it, is that I never want to be afraid to find out if my idea is a good one or a bad one. I should create that feedback.If I’m making a game for other people, I should want to get the game in front of other people as quickly as I can. To find out if it’s working for them. If it’s working for me, that’s great, but I’m not the audience – other people are. And so I should want to find out quickly, and effectively, and efficiently as I can: Is it working? And if I’m afraid to do so, then I’m probably afraid of my game design.

2 – Don’t focus on what players say, focus on WHY they are saying it

While everything playtesters say is valuable, it’s also important to take their feedback with a grain of salt to prevent yourself from being misled by what players are saying.

Mike Ambinder:

People are not great at explaining why they do what they do. Obviously, what people say be very helpful. But I should for sure would want to correlate anything someone says with their behavior in a game. If they’re like, “No! That was easy!” But they died like 35 times, well, maybe it wasn’t so easy.

When you combine watching someone play while listening to what they say, it puts everything into context.

Adriaan says

“the clearest feedback you can get is to see how someone plays your game. everything else is very easily a distraction.”

Testing for WHYThis is where the ethnographic interview methodology comes into play.

This methodology is built to get the most WHY out of a test session. Here’s the key mindset: The playtester is the teacher, and you the student. They aren’t the ones being tested, YOU and YOUR DESIGN are being tested. Make sure they know that. Empower them to be the teacher.

If my playtesters feel dumb, it’s because *I* am dumb.

As they play, encourage playtesters to speak their thoughts aloud and to ask questions, but don’t tell them how to solve problems, at least, not immediately. Just say “mhmm” and scribble notes like a student attending a lecture would. When a player tells you what to change, ask them why they are suggesting the change – you want to understand the WHY.

While they play, you can stop the player to ask some questions, but your main job is to shut up and watch. Be very careful with your questions. Avoid prescribing solutions – don’t ask questions like “What would you think about a double jump?” Instead, ask something like “How did you feel about the movement? What problems did you encounter?”

You want to tease out information to understand the real problems rather than accidentally confirm your biases. Though, feel free to break free of these rules once you feel like the test is over. I would love to talk about playtesting processes and tips more, but I think that’s a topic for another time.

Okay, now you know that successful game designers base their process around playtesting calling it, “The Only Way to Know”, and how to aim for frequent, clear feedback.

Maybe now you’re thinking “Okay, I need to playtest, but….”

Part 4: No Buts, No Excuses

It’s time for common excuses for not playtesting!

Excuse: “My game isn’t in a place that it’s ready to playtest yet”.

In the example I started this article with – I waited 2 years to start testing my game.And after just a handful of tests, it immediately became apparent that the core assumption of the mechanic was just plain wrong, and that I had to completely redesign. At the time I thought the game needed to be “done” before I could test it. That assumption added a year of development.

Test EarlyYour FIRST goal when designing a game is to get it to a place it can be tested – first by you and then by someone who has never seen your game before. If you’re months into development and you haven’t tested your core assumptions yet, you’re in danger of having planned too far into the future on a hunch.

Test EarlierI once spent three months programming a design idea only to finally realize how painfully awful it was when it was up and running.

I threw it out, moved to paper and programmed a tiny tool to track data.

Within 2 days I had a way better design.

Within a week, I had an amazing design.

I sometimes test games before even debugging. So long as there are no show-stoppers, I can guide players around bugs.

In super early testing, if I only want to test the core mechanics and game loop, I’ll break a golden rule of playtesting and shamefully skip in-game tutorials altogether – I just explain like it’s board game night during the test session. This saves me the extra dev time of updating a tutorial alongside a design I’m fumbling with – allowing for faster iteration cycles.

While writing this, I found the following quote by the Interaction Design Foundation:

“Make sure users want to use the app. You can make it usable later.”

The key here is: you want to spend as little time as possible chasing the wrong ideas and as much time as possible nurturing the best ones. Always ask yourself: What is the smallest testable version of my game or feature?

Test early, test often.

“It’s the Only Way to Know.”

Excuse: People will take my idea

No one wants your idea. Ideas are free. They’re useless on their own. Everyone believes their own idea is better. It’s execution that counts.

If you’re worried you don’t have a better idea, I have an exercise for you: force yourself to make a list of 50 new ideas every day until you overcome this fear.

If you’re still concerned – have a lawyer create an NDA for you and have testers sign it.

To find out if your idea is even worth something to players, show off your game early and often.

Excuse: No one will test my game!!!

Finding testers may not be as hard as you think.

You can ask friends and family to let you watch them play for 30 minutes while you take notes.

Join a Community of Game Developers and trade testing each other’s games.

Start a Discord server where regularly scheduled testing happens at the same time every week and build it over time.

Start a newsletter and offer early-build testing as a signup bonus.

If it’s a mobile game, take your phone to a party – the drunker the tester, the more open and honest the feedback and the more they will struggle with difficult design.

Find people who write reviews for similar games and add them as friends or try to contact them.

Ask as many people as you can to be play testers, because after all:
“It’s the Only Way to Know.”

Excuse: I’m afraid to receive feedback

No one ever actually admits this, but it’s a common concern hiding in all of us. It may be the core concern fueling the previous excuses discussed. Showing something we’ve put our heart an soul into to others can make us feel very vulnerable.

But remember: You are not your work. And your work was created to share with the world. To enrich other people’s lives. It’s not about you – it’s about connecting to them, but… in a way that brings you satisfaction, so stay true to yourself. Your work doesn’t have to be for everyone.

In the book “Board Game Design Advice: From the Best in the World” more than 60 game designers were asked what it’s like to have a bad test session and all of them said a variation of “Great! That’s when I really learn a lot.”

That means your worst case scenario “they hate it!” could also be viewed as the best case scenario “Here are things to work on!”. The true worst case scenario is you don’t get any feedback at all.

The more you protect yourself, the weaker you become.

When Enough is EnoughSo the question really isn’t SHOULD YOU PLAYTEST and more when should you stop play testing? How do you know your game is done?

Mike Ambinder:
Does anybody know that it’s done? No. Nobody knows. What we’re after though, is maximizing our likelihood of success. Yes, you could iterate and playtest forever and never release anything and that would be a poor outcome. But what you get with playtesting, is an identification of a bunch of problems that need fixing and an understanding of the things people find appealing about your game. And at some point, you’re gonna start hearing a lot more of the positive and a lot less of the negative. And you’re going to start feeling a lot more confident. So I don’t look at it as a gatekeeping device. No, what it is is just the device that steadily, and hopefully increases your confidence. If the feedback you’re getting is “there aren’t really any major problems”, that’s a pretty good milestone.

One way to know:

If after the playtest, people don’t want to stop playing, or start asking when they can play again – that’s a good way to know your design is working.

To summarize, why should you playtest?

Prevents you from planning too far into the future on a hunch

Confront your hypotheses and adapt to your players.

Build trust in your intuition

Get more productivity out of each hour of game design.

Become more self-reflective about your inner biases towards your designs

Increase your empathy for players

Gain intuitive understanding of user experience best practices

Mike Ambinder:

Everybody’s goal is to just make an experience that people want to play. So go find out if what you’re making is something people want to play. And continue to do that, over and over until you get to a point where people wanna play your game. If you’re unsure about what to do, okay, get more informed. Go gather data. Playtesters will be happy to give it to you.

Now get out there and playtest!