Authors: Siew Kim Kwa, Zainab Majeed, Shane Varman.
ABSTRACT
Introduction: Assessment is an integral aspect of teaching. One-best-answer (OBA) items, if properly constructed are able to drive learning. In-house OBA items are notoriously poorly-constructed. The role of a central vetting committee is to review test items and ensure that they adhere to expected standards. Hence, the objective of this audit is to determine whether central vetting has improved the construct quality of OBA items.
Methods: We audited the psychiatry end-of posting OBA items from before and after central vetting to compare the quality of the items before and after central vetting was instituted. Quality was evaluated on appropriateness of test content, items with higher cognition and items without flaws. A standard was not set for this first audit.
Results: Seventy six of 181 psychiatry OBAs items retrieved from 2011 to August 2012 had undergone first level (department) vetting only and the remainder 105 (58.0%) had two levels of vetting; department and central vetting committee (CVC). Appropriateness of content increased from 92.1% to 98.1%. Items with higher order thinking doubled from 21.1% to 42.9%. Items with clinical scenario increased by 8.4% to 78.1%. Logical ordering of options however, remained around 50%. Two-level vetting markedly reduced problematic lead-in questions (67.1 to 13.3%), non-homogenous options (42.1 to 9.5%), vague and implausible options (39.5 to 6.7%), and spelling and grammar mistakes (19.7 to 5.7%).
Conclusion: Two-level vetting had improved the quality of OBAs and should be continued. This could be enhanced by training all Faculty on writing quality OBA items and careful selection and empowerment of CVC members. A re-audit is to be conducted after Faculty training.
Keywords: Assessment, Vetting, One-best-answer items, MCQ, Quality assurance.
Citation: IeJSME 2017 11(3): 3-9
