Comparative evaluation of multiple choice question formats. Introducing a knowledge score

OBJECTIVE: Over the last 6 years different multiple choice question (MCQ) formats have been used in postgraduate examinations for trainees in psychiatry. In phase 1 - K-type True/False (T/F) items with negative marking; in phase 2 combined T/F and type-A one-best answer (OBA) questions without negative marking; in phase 3 exclusively OBA without negative marking. The study compares the gross scores (GS) obtained with different MCQ formats, and introduces knowledge score (KS).

METHODS: The study was conducted in the Saudi Council for Health Specialties, Riyadh, Kingdom of Saudi Arabia from 1996 to 2002. The mean percentile scores obtained by all postgraduate trainees sitting any Part I or Part II Saudi Board Examination in Psychiatry were subjected to a comparative analysis.

RESULTS: A total of 110 candidates sat 18 examinations returning 143 papers containing a total of 32,375 MCQ options. Phase 1 generated lowest overall mean GS (47.8%), phase 3 occupied an intermediate position (53.1%) and phase 2 produced the highest score (68.3%). The KS, to the contrary, generated strikingly similar results for all the 3 phases (47.8, 50.5 and 49.5%) indicating that the marked differences in the GS were probably related to benefits obtained from guessing in the absence of negative marking. In this respect, the OBA produced considerably higher KS scores than the T/F, presumably due to its facilitating extra benefits from cueing, partial knowledge and judgement.

CONCLUSION: Different MCQ formats generate dissimilar quantitative results. The OBA format seems superior to the T/F format in crediting judgement and application of knowledge. In non-negatively marked MCQ tests, the suggested KS provides results comparable to those of negatively marked tests. Pass marks in MCQ tests should be calibrated according to the used format.