Great Expectations? How MQCs Should Perform on Exams

If you've paid much attention to how exams are graded, you've probably seen the term minimum qualified candidate (MQC). Future Fellows wanted to know, what does the CAS really expect of the MQC on an exam? To that end, we interviewed the general officer of grading and pass mark panel within the Syllabus and Exam Committee. All quotes are attributed to that officer.

The formal concept of the MQC was adopted around 10 years ago. Prometric, an educational consultant, recommended it as an "objective standard to determine whether or not a candidate had an understanding of the source material well enough to be considered qualified." Today the idea is fundamental to the creation and grading of every CAS exam. Here's how it fits into the six steps of the exam process:

1. The Exam Committee assigns papers or sections of the syllabus to exam writers. They write the questions, using the wording of the learning objectives on the syllabus as a guide to ascertain what level of Bloom's Taxonomy the questions should be.
2. The Exam Committee collects and reviews the questions. Using the syllabus weights as a guide, they select the questions and create the exam.
3. Now the MQC standard comes into play. The Pass Mark Panel is a group of experts on the material, including individuals involved with different parts of the exam process, such as past writers and graders. They ask, "What is the essential material that we think a future Fellow or Associate would need to know from this material to be considered to be a fully credentialed actuary?" Based on the answers to that question, they define how the MQC should be expected to perform on each individual question. They factor in the difficulty as well as the Bloom's level, and note any nuances in the question before selecting a final MQC score.
4. The CAS administers the exams.
5. The exams are graded. Part of that process involves a discussion of the MQC score for each question, in which graders identify any issues that the Pass Mark Panel did not foresee. Some adjustments to the MQC score may be made at this point. For example, maybe a majority of candidates interpreted a question differently than expected due to unintentional ambiguous wording. For more details on the grading process, refer to "After the Exam: A Behind-the-Scenes Glimpse of the CAS Grading Process" in the last issue of Future Fellows, available online.
6. Candidate scores are evaluated against the pass mark, which is basically the ultimate MQC score, and we find out who passed.   

As you can see, the MQC score is "an objective measure that does try to take feedback into it at various phases but is not based on candidates' performance."

The MQC standard is reevaluated annually for each exam, but it usually doesn't change much unless there are changes to the syllabus (such as a change to a learning objective or a source material).

"The expectation is not that an MQC should be able to answer every question at every level of difficulty."

High school teachers often use Bloom's Taxonomy as a guide for writing exams. The idea is that a C student should be able to answer most questions from the knowledge, comprehension, and application levels of Bloom's, while an A student can answer questions from the evaluation and synthesis levels. A test would therefore have questions from all levels of Bloom's in order to allow students of all levels to demonstrate their abilities and earn the appropriate letter grade.

In many ways, a CAS exam is like a classroom test. Exam construction is dictated "exclusively based on the syllabus," and the syllabus uses verbs in the learning objectives that should indicate at what level of Bloom's Taxonomy that objective might be tested. Reading through an exam syllabus you should see a variety of verbs from all levels of Bloom's. Therefore, there should be some sections where the MQC will "knock this portion of the exam out of the park, but this [other] portion is a little more abstract…so we don't think they should necessarily get 100% on this every single time."

In fact, if a question is really high level, the MQC score may be set very low or even at zero. There has to be some room to distinguish oneself above the minimum in Minimum Qualified; those questions with a low MQC score will differentiate the high achievers.

Just like in the classroom, it would be unexpected, but it is theoretically possible for 100 percent of candidates to pass an exam under the MQC standard. Rather than setting a pass rate, the MQC score sets a bar for candidates to reach or exceed: "They aren't competing against themselves; they're competing against the MQC standard."