Continued testing via the hive
We've had a fair amount of press recently on our extended uTest use;
So the questions remains, is the UTest service living up to the hype?
Well the quick answer is yes and no. It appears to be another tool in the tester's arsenal rather than a silver bullet. It's very useful for particular aspects of testing and not so useful for others.
This is because of the current way the interface works, and how the releases are issued to the testing community. Let's say for instance that you are working on a web application that needs testing. You get your release ready and load up your uTest account with a fair amount of dollars (say $500 to start with).
Once you hit that go live button you are instantly in a race. The uTesters are essentially racing against each other to find the easiest bugs to report because they get paid a standard amount for each bug, so the first in gets all the easy bugs to spot and recreate. After the easy ones are gone instantly, the testers then start to look for iterative bugs, so for instance if they find a cross site scripting bug on one field in the form, they register a bug for each of the fields in the form for the same (essentially) cross site scripting bug. This quickly starts to eat up the budget.
To counter this you are allowed to update the specification of testing to say something along the lines of "no more field validation bugs please". You have to be quick though, if you notice a load of bugs coming through with the same essential issue, you need to update your spec for testing as fast as possible, and with a big team finding bugs you are pretty much in a race condition trying to stop multiple similar bugs from being logged.
Once all the easier bugs and iterative bugs are gone, the more complex bugs seem to start appearing assuming you still have some budget left. This is where the application starts to come into its own and the power of the masses starts to become apparent.
The benefits of this kind of testing are clear. We now check our applications don't have any of the common uTest bugs and are already tested for the cross site scripting and other common bugs the community seems to go for immediately, so now more of our budget goes on the more hard to find issues. If you are thinking of trying out uTest I would say go for it. It makes your internal testers improve the product before it's released to the community, but there is a period where you are getting used to the system, and that can be a little expensive.
To be fair though uTest are continually changing the process in order to address some of these issues, and I expect the next testing round to be smoother.
I don't think we are going to scrap hudson, but it's a good tool to know about and comes in handy for tight deadlines.