geordie racer wrote:
I don't think it was a solid playtest.
So few playtests are, IMO. I've been involved with several -- mostly for indie game writers.
Some background first. I have experience running usability tests for users of web applications. It's part of my day job.
So I'm used to creating "scripts" or designing activities to test different parts of a design or feature and then observing the user in action as objectively as possible, recording results and then gathering data from a sample of participants. Then analyzing and presenting that data in some meaningful fashion.
Playtests aren't that. Most playtests are groups gathering around and having fun with a game. That's not as hard as you might think. Once in a while, they might hit a logjam with a mechanic -- which results in a change to the rules most times.
But most playtests aren't designed to stress the system. Or to gather data in a meaningful fashion. Or present any sort of analysis of that data to figure out really what went wrong, where, why and what people are getting hung up on.
Because a lot of the time, people just get stuck on something and they think this one thing is a solution when, in reality, it's this other thing that's throwing them off. And that solution might actually make something worse because, well, it's a new variable that's untested.
So, while a good marketing exercise, I think it's important for Joseph to stick to his guns with DCC. Sure, some interesting ideas might develop from the feedback he receives. But he's not getting any real data from us. Just hearing where (or if) we're having problems.
Open playtests, IMO, are more a test of (A) are the rules comprehensible and (B) are there any glaring mechanical horrors that somehow slipped through the cracks. The feedback should be taken with a minivan-sized grain of salt.
Just my opinion. But I'll point out that 4e went through how many hundreds of "playtesters" without ever realizing that a combat that lasts 90 minutes is too long, skill challenges (as printed in DMG1) were broken, the stats for monsters were off (see adjustments in MM3) and PCs failed to effectively scale with the monsters (see the Expertise feats for more on that)?
So what exactly were they "testing"?
Not to diss on 4e. Their playtesting shared the same flaw as most RPG playtesting. Not to say I don't want to playtest DCC. I'm planning on it. I just recognize it involves more "play" than "test".