Establishing an MQC Score
The concept of the minimally qualified candidate (MQC) score is familiar to most candidates, but the process by which MQC scores are established tends to cause confusion. Continuing with our series of articles that (hopefully) help demystify the CAS admissions process, here is a description of the steps involved in setting MQC scores.
A few months before the exams are administered,there is a pass mark panel meeting for each exam. These meetings generally include the chairs and vice chairs of the exam as well as a handful of other committee members and general officers of the examination committee.
Dr. Richard Fischer, who is an expert on psychometrics and test development, also participates in these meetings to provide guidance on general testing philosophy (learn more about Dr. Richard Fischer and his role in the CAS on the CAS Website). The fundamental purpose of a pass mark panel meeting is to establish the a priori MQC score for each question.
The first step in this process involves painting a picture of the minimally qualified candidate. Which topics do we believe the MQC should have full mastery of, and which topics require average mastery? Though these rankings are not published, the committees do assign a relative value to each exam topic. For example, those who took Exam 5 probably won’t be shocked to learn that Bornhuetter-Ferguson items will require a higher MQC score, as a percentage of total points, than techniques for estimating unpaid loss adjustment expenses. In addition to topic importance, question difficulty is also taken into account in setting MQC scores. Thus, the MQC score for a Bornhuetter-Ferguson question will likely be higher than the score for a question of comparable difficulty that involves calculating unpaid LAE. For example, if each question were 3 points, the BF may have an MQC score of 2.5 while the unpaid LAE may have an MQC score of 1.5.
Once the picture of the minimally qualified candidate – which only changes season-to-season if there are new learning objectives or knowledge statements – is complete and understood by the panel, the pass mark panel goes through each question and sets the a priori MQC score. This process generally involves quite a bit of debate among panel members, and it’s not uncommon to see chairs fly across the room. Ultimately though, the a priori mark is established.
The next phase involves determining the a posteriori MQC scores, and that occurs during the grading sessions (famously held at an undisclosed desert location). Based on candidate performance by question, the scores set at the pass mark panel meeting are reevaluated. To be clear, the MQC scores don’t automatically move to meet actual candidate scores. Instead, the graders are asked to select a score for their question based on the responses that they have seen. This is done blindly, with the graders having no knowledge of the MQC scores on which the pass mark panel agreed. The exam committee officers then reconcile every question, taking the a priori score and the graders’ a posteriori estimates into consideration. In some cases the a priori score is changed, and in other cases it isn’t.
For example, say a particular question tests a topic of medium importance, and the pass mark panel believed that the question was straightforward and thus assigned it a relatively high MQC score. However, in actuality candidates struggled much more with the question than was anticipated. Because the candidate population performed differently than expected, the officers may adjust the MQC score for that question. This process is called reconciliation and is done for each question for which the a priori score differs from that of the graders. Questions with the largest difference are reconciled first, but ultimately every question is reconciled before the final pass score is established.
For most exams, the final pass mark is simply determined by summing up the a posteriori MQC scores for each question on the exam. In some instances though, the pass mark is adjusted to reflect unanticipated factors. For example, the examination committee works hard to create exams that allow candidates sufficient time to demonstrate their knowledge, but in some cases the committee misestimates the overall length of the exam. In such circumstances, the committee may make a bulk adjustment to the final pass mark.
As this article hopefully highlights, much debate, discussion, and reconsideration go into selecting final MQC scores, and the Exam Committee considers this process to be very important.
If you have any questions or comments, please send us a note by filling out the Candidate Liaison Committee feedback form. We always look forward to hearing from candidates on the admissions process, as well as on any other topic.
