How is Beta Test data used?

[Gary Mann]

Does anyone know (and is willing to share) how the beta test data is used to develop the actual exam? I'm presuming that the SCBCD cert test will have fewer questions and so take less time, so is the process now to analyze how we all did on each of the questions and select those that differentiate the best? i.e. if 95% of beta testers answered a question rightly or wrongly, then perhaps it's not a good question.
Also, did the automated exam take timings on how long we spent on each question and could that be factored in? e.g. one of the EJB-QL questions was a bit of an IQ test and took extra time, but was not, at least in my opinion, a good way of determining understanding of the concepts.
Or am I completely on the wrong track? Just interested how it all works.

[Peter Storch]

Originally posted by Gray Man:
I'm presuming that the SCBCD cert test will have fewer questions and so take less time.

I can help you at least at the number of questions question. Kathy posted in http://www.coderanch.com/t/157855/java-EJB-SCBCD/certification/Back-exam

But remember, the real exam will have only 1/3 of the questions! So the beta testers had to answer every possible variation of questions where on the real exam you might have only a small handful of questions on any particular topic.

[Kathy Sierra]

Howdy,
Here's another little summary of some of the ways in which the beta results are used...
It's a very complex statistical process that the pscyhometricians use, but I DO know a little of it:
* If virtually everyone gets an answer wrong, or everyone gets an answer write, the question might be eliminated. We DO want a mix of hard and easy questions, but we don't want IMPOSSIBLE questions.
*If it appears that the *wrong* people get an answer right, and the *right* people get an answer wrong, the question is probably thrown out. What I mean by this is that, let's say that the people who score very well on the beta exam all tend to get the same answer incorrect, while the people who did NOT score well were more likely to get that answer right. This tells us that the question is probably poorly worded, in such a way that the more experienced folks look at it differently, and *not* in the way we intended.
* Overall timing is taken into account, I think, more than any single question's timing. You will obviously have a mix of quick questions and long questions, so as long as the overall balance works out for the timing... I don't think that the time it takes for any particular question really determines whether that question is too hard. Unless, I imagine, that it is in the extreme.
* The individual comments submitted by the beta testers are given a LOT of value! If enough people complain about a particular question, even if the question is accurate, it may be removed.
* Other comments that have come in, including here on javaranch, will be taken into consideration as well.
* The beta results are also used to determine the overall pass percentage (i.e. how many you have to get right in order to pass). I do not know how they really determine that.
There is an overall goal that the exam should NOT be a 'walk in the park', in order to keep the value of the exam high (so that it does NOT become just a piece of paper, and a piece of paper that *anyone* can pay $150 to get). But I have no idea what kind of analysis goes into the final decision on that.
Hope that helps
good luck to all as you wait for your results...
-Kathy