Howdy,
Here's another little summary of some of the ways in which the beta results are used...
It's a very complex statistical process that the pscyhometricians use, but I DO know a little of it:
* If virtually everyone gets an answer wrong, or everyone gets an answer write, the question might be eliminated. We DO want a mix of hard and easy questions, but we don't want IMPOSSIBLE questions.
*If it appears that the *wrong* people get an answer right, and the *right* people get an answer wrong, the question is probably thrown out. What I mean by this is that, let's say that the people who score very well on the beta exam all tend to get the same answer incorrect, while the people who did NOT score well were more likely to get that answer right. This tells us that the question is probably poorly worded, in such a way that the more experienced folks look at it differently, and *not* in the way we intended.
* Overall timing is taken into account, I think, more than any single question's timing. You will obviously have a mix of quick questions and long questions, so as long as the overall balance works out for the timing... I don't think that the time it takes for any particular question really determines whether that question is too hard. Unless, I imagine, that it is in the extreme.
* The individual comments submitted by the beta testers are given a LOT of value! If enough people complain about a particular question, even if the question is accurate, it may be removed.
* Other comments that have come in, including here on javaranch, will be taken into consideration as well.
* The beta results are also used to determine the overall pass percentage (i.e. how many you have to get right in order to pass). I do not know how they really determine that.
There is an overall goal that the exam should NOT be a 'walk in the park', in order to keep the value of the exam high (so that it does NOT become just a piece of paper, and a piece of paper that *anyone* can pay $150 to get). But I have no idea what kind of analysis goes into the final decision on that.
Hope that helps
good luck to all as you wait for your results...
-Kathy