The Benefits of Having a Tester Pool
by Stephen - 11th July 2022
The primary benefit of having a tester pool is the retention of good testers.
That's it folks. See you next time! I joke, of course; don't leave. But that really is the bottom line, up front. The main reason for starting and maintaining a tester pool is to retain your best and most effective testers.
Now don't take this to mean that a tester pool is just a list of your favourite testers. Far from it. IF that were the case, you'll be missing out on a bunch of really awesome perks. But we'll get to those in just a minute.
Only focusing on results without making sure that you are retaining your best testers means that it will be harder to replicate that same success in future tests. You will slightly improve your gains in the present and achieve less results in the long term.
Of course you won't get retention just by having a pool. There's more to it than that (which will be the subject of a future post), but having a pool is sine qua non. It is the essential first step. Other efforts to retain your best testers cannot be made, or are rendered futile, if you do not maintain your own tester pool.
Retention brings about all sorts of benefits depending on the type of pool you have and how you manage it.
At their core, they're just a list of people, but how that list grows and changes is what makes all the difference.
Let's say you only add people to your pool who are frequent testers of yours. Testers you trust and know will give you the quality of feedback and the level of results you need. I would argue that it wouldn't really be a pool more than a database of trusted testers. But for simplicity's sake and ease of reference, let's call this a Closed Pool. By contrast, consider a list to which anyone can add themselves, and let's call this an Open Pool. This is more akin to a mailing list of sorts.
Now I know my coined terms might already have some pre-existing connotations (such as "open" meaning you can apply to but not necessarily be added to), but that's not really important here. I just want to set some reference points to describe a type of pool that is a hybrid of these two. I'm sure many designers use this kind of pool already, but let's clearly lay it out in terms of how it works so we can then extrapolate our benefits from that.
Let's call this pool, and forgive my lack of creativity, a General Pool. A General Pool has open applications, meaning anyone can apply/ask to join at any time. This does not necessarily mean that they are automatically added to the pool. The designer gets to review the request before deciding whether to include them or not. This way you are able to eliminate obvious no-fits, such as in cases of significant language barriers, people you know you do not want to include, etc... However, unless you have a serious objection to including that person, the idea is to accept as many of the applicants as possible. These are people who, for some reason or other, have shown an interest in your designs and in testing for you, which is very important.
How you manage a General Pool determines the degree to which you can benefit from it. I would recommend you do two specific things:
You can just as easily keep a list on a simple document on your computer or phone. An online document (such as using Google Docs) would of course be more convenient and portable (and free).
If you're somewhat familiar with spreadsheet software such as Microsoft Excel (or Google Sheets; again: portable, free) and can write a couple of simple formulas, you should go for that. It will immediately level up your game as you can automate a lot of things and cross reference most of your data. Let me know if you're interested in this kind of thing and want me put together a post on how to leverage formulas and tables to supercharge your tester pool.
Of course, you can also opt to use a tool like Pattern Orchard which does it all for you right out of the box. It saves you time and makes your information clear to understand and access. You should check it out.
Shameless plug notwithstanding, maintaining a General Pool is not hard, and the rewards are huge.
Clarity and ease of use are very important if you want your tester pool to be an asset to your process.
You don't just have access to people, but to interested people who have already declared that they want to test for you. Some of them you already know a lot about and have an existing relationship with. This is extremely powerful, and avoids a lot of issues down the line that are so prevalent in tests that mainly source their testers from public calls.
Of course this isn't to discourage having public calls, but these are meant to supplement the test, not fill it. They're also an opportunity to grow your pool by asking test applicants whether they'd like to join the pool as well.
Let's not ignore the (often underappreciated) time savings you get by:
Finally, and although subtle it's definitely very powerful, a pool gives you an opportunity to build relationships with your testers, which is all kinds of good. Better relationships equals better results.
In future we'll discuss how to grow, maintain and curate your pool, how to take care of your testers, develop your relationships etc... In the meantime I recommend you start using a General Pool and start managing it as suggested above. I promise, you'll notice the difference.
Stephen
That's it folks. See you next time! I joke, of course; don't leave. But that really is the bottom line, up front. The main reason for starting and maintaining a tester pool is to retain your best and most effective testers.
Now don't take this to mean that a tester pool is just a list of your favourite testers. Far from it. IF that were the case, you'll be missing out on a bunch of really awesome perks. But we'll get to those in just a minute.
Results and retention
An effective pattern test is one that achieves the results you need. These could be testing your design assumptions, eliminating all sorts of errors, improving clarity of instructions, promotion, etc...Only focusing on results without making sure that you are retaining your best testers means that it will be harder to replicate that same success in future tests. You will slightly improve your gains in the present and achieve less results in the long term.
Of course you won't get retention just by having a pool. There's more to it than that (which will be the subject of a future post), but having a pool is sine qua non. It is the essential first step. Other efforts to retain your best testers cannot be made, or are rendered futile, if you do not maintain your own tester pool.
Retention brings about all sorts of benefits depending on the type of pool you have and how you manage it.
All pools are not created equal
Tester pools come in different shapes and sizes, depending on what a designer intends to use it for.At their core, they're just a list of people, but how that list grows and changes is what makes all the difference.
Let's say you only add people to your pool who are frequent testers of yours. Testers you trust and know will give you the quality of feedback and the level of results you need. I would argue that it wouldn't really be a pool more than a database of trusted testers. But for simplicity's sake and ease of reference, let's call this a Closed Pool. By contrast, consider a list to which anyone can add themselves, and let's call this an Open Pool. This is more akin to a mailing list of sorts.
Now I know my coined terms might already have some pre-existing connotations (such as "open" meaning you can apply to but not necessarily be added to), but that's not really important here. I just want to set some reference points to describe a type of pool that is a hybrid of these two. I'm sure many designers use this kind of pool already, but let's clearly lay it out in terms of how it works so we can then extrapolate our benefits from that.
Send me the Challenges of Running Pattern Tests eBook
Learn how successful designers are dealing with the top five problems of running pattern tests.
Discover a novel solution to make the process simpler and more effective.
Discover a novel solution to make the process simpler and more effective.
How you manage a General Pool determines the degree to which you can benefit from it. I would recommend you do two specific things:
- Clearly label or otherwise be able to identify your trusted testers. Consider them VIPs.
- Curate your list from time to time, especially after completing a pattern test. There will be a dedicated blog post on this soon, as it is very important. To go over it briefly:
- rate your testers, on a scale of your choosing
- keep detailed comments on anything you think is relevant
- keep a historical performance record (what tests did they apply to and/or participate in, and if they did, what was their performance like?)
Start your pool now
This might seem like a lot, but all you need is a pen and some paper. At a very basic level, there's not much you need to maintain a tester pool.You can just as easily keep a list on a simple document on your computer or phone. An online document (such as using Google Docs) would of course be more convenient and portable (and free).
If you're somewhat familiar with spreadsheet software such as Microsoft Excel (or Google Sheets; again: portable, free) and can write a couple of simple formulas, you should go for that. It will immediately level up your game as you can automate a lot of things and cross reference most of your data. Let me know if you're interested in this kind of thing and want me put together a post on how to leverage formulas and tables to supercharge your tester pool.
Of course, you can also opt to use a tool like Pattern Orchard which does it all for you right out of the box. It saves you time and makes your information clear to understand and access. You should check it out.
Shameless plug notwithstanding, maintaining a General Pool is not hard, and the rewards are huge.
Clarity and ease of use are very important if you want your tester pool to be an asset to your process.So what about those rewards?
Now you have direct access to a list of people you can issue test calls to. Kind of like a mailing list, although there's quite a bit more to it than that. You don't need to worry about social media or post discovery in a sea of irrelevant content.You don't just have access to people, but to interested people who have already declared that they want to test for you. Some of them you already know a lot about and have an existing relationship with. This is extremely powerful, and avoids a lot of issues down the line that are so prevalent in tests that mainly source their testers from public calls.
Of course this isn't to discourage having public calls, but these are meant to supplement the test, not fill it. They're also an opportunity to grow your pool by asking test applicants whether they'd like to join the pool as well.
Let's not ignore the (often underappreciated) time savings you get by:
- having a pre-existing audience that saves you having to start from scratch every time in order to reach and source testers
- not having to vet the same people (as in the case of frequent collaborators) over and over again every time they apply for a test
Finally, and although subtle it's definitely very powerful, a pool gives you an opportunity to build relationships with your testers, which is all kinds of good. Better relationships equals better results.
- You get used to each other & ways you work, improving the effectiveness of the testing process
- The testers learn your system and become comfortable working within it, making the experience better for everyone. They learn what you're looking for and you learn what they need from you
- You get an early heads up when something unexpected happens, allowing you to adjust rather than hearing about it last minute, if at all, which might mess up your plans
- Your testers are more inclined to go the extra mile, as are you
- They are more likely to help you promote the pattern once it's been released. Their messages will be genuine, rather than the forced canned phrasing of a singular social media post they feel obliged to upload
In future we'll discuss how to grow, maintain and curate your pool, how to take care of your testers, develop your relationships etc... In the meantime I recommend you start using a General Pool and start managing it as suggested above. I promise, you'll notice the difference.
Stephen
Comments powered by Talkyard.