Win a copy of Testing JavaScript Applications this week in the HTML Pages with CSS and JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
  • Campbell Ritchie
  • Bear Bibeault
  • Ron McLeod
  • Jeanne Boyarsky
  • Paul Clapham
  • Tim Cooke
  • Liutauras Vilda
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • fred rosenberger
  • salvin francis
  • Piet Souris
  • Frits Walraven
  • Carey Brown

How is Beta Test data used?

Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Does anyone know (and is willing to share) how the beta test data is used to develop the actual exam? I'm presuming that the SCBCD cert test will have fewer questions and so take less time, so is the process now to analyze how we all did on each of the questions and select those that differentiate the best? i.e. if 95% of beta testers answered a question rightly or wrongly, then perhaps it's not a good question.
Also, did the automated exam take timings on how long we spent on each question and could that be factored in? e.g. one of the EJB-QL questions was a bit of an IQ test and took extra time, but was not, at least in my opinion, a good way of determining understanding of the concepts.
Or am I completely on the wrong track? Just interested how it all works.
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Originally posted by Gray Man:
I'm presuming that the SCBCD cert test will have fewer questions and so take less time.

I can help you at least at the number of questions question. Kathy posted in

But remember, the real exam will have only 1/3 of the questions! So the beta testers had to answer every possible variation of questions where on the real exam you might have only a small handful of questions on any particular topic.

Cowgirl and Author
Posts: 1589
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Here's another little summary of some of the ways in which the beta results are used...
It's a very complex statistical process that the pscyhometricians use, but I DO know a little of it:
* If virtually everyone gets an answer wrong, or everyone gets an answer write, the question might be eliminated. We DO want a mix of hard and easy questions, but we don't want IMPOSSIBLE questions.
*If it appears that the *wrong* people get an answer right, and the *right* people get an answer wrong, the question is probably thrown out. What I mean by this is that, let's say that the people who score very well on the beta exam all tend to get the same answer incorrect, while the people who did NOT score well were more likely to get that answer right. This tells us that the question is probably poorly worded, in such a way that the more experienced folks look at it differently, and *not* in the way we intended.
* Overall timing is taken into account, I think, more than any single question's timing. You will obviously have a mix of quick questions and long questions, so as long as the overall balance works out for the timing... I don't think that the time it takes for any particular question really determines whether that question is too hard. Unless, I imagine, that it is in the extreme.
* The individual comments submitted by the beta testers are given a LOT of value! If enough people complain about a particular question, even if the question is accurate, it may be removed.
* Other comments that have come in, including here on javaranch, will be taken into consideration as well.
* The beta results are also used to determine the overall pass percentage (i.e. how many you have to get right in order to pass). I do not know how they really determine that.
There is an overall goal that the exam should NOT be a 'walk in the park', in order to keep the value of the exam high (so that it does NOT become just a piece of paper, and a piece of paper that *anyone* can pay $150 to get). But I have no idea what kind of analysis goes into the final decision on that.
Hope that helps
good luck to all as you wait for your results...
Stinging nettles are edible. But I really want to see you try to eat this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
    Bookmark Topic Watch Topic
  • New Topic