After the Exam: A Behind-the-Scenes Glimpse of the CAS Grading Process

Exam day is over and for CAS candidates, eight weeks of waiting begins (actually seven weeks for the last exam sitting). Some candidates breathe a sigh of relief, knowing that their hours of studying probably paid off. Others feel a sense of dread towards results day and do their best to put it out of their minds. But what actually happens behind the scenes between the day of the exam and the day that the list of passing candidate numbers is released? And once the Examiner’s Report is posted, what happens if a candidate disagrees with the model answer or thinks they had a valid alternative?

0-2 Weeks After the Exam — Collection and Defective Questions

After the exam, CAS candidates’ long journey of studying and sitting for the exam is complete, but for the exams themselves the voyage is just beginning. All around the world, from Philadelphia to Hong Kong, the answer sheets are collected and make their way to the CAS office in Arlington, Virginia. Once all of the envelopes arrive, staff must verify that the candidate numbers on the envelopes match the numbers on the answer sheets within, and they also must confirm that the numbers match the records of the exam proctor from each examination center. The answer sheets are now ready to be sorted by question and scanned in preparation of being sent to the graders.

While the CAS staff is busy sorting through thousands of exams, candidates have a chance to voice their concern about specific questions from the exam they just took. The CAS Syllabus and Examination Committee makes every effort to provide a perfect exam, but in rare cases a typo, ambiguity, or other flaw may be present in a question. If a candidate feels that a question was defective, they are invited to write to the Syllabus and Examination Committee within two weeks after the examination.

Members of the committee will review the model solution(s) that was provided to them and determine if there are any modifications that are needed. They will also carefully review every comment and decide if they need to take action. In some cases, the model solution(s) may be altered in order to accept an additional answer, or the question may be considered defective and not counted at all. Sometimes, the syllabus may even be changed for future exams in order to eliminate confusing or contradictory information in the source material.

2-5 Weeks After the Exam — Initial Scoring

In order to maximize the consistency in grading for CAS exams, the scanned answer sheets are sorted by question and every answer sheet for a given question is sent to a team of graders. First, each of the graders independently scores around 20 of the responses, using the model solution and their own knowledge. Then, they compare these initial scores between themselves in order to establish a consistent grading scale for the question. Once they agree on the scale and how to interpret the model solution, the graders independently assign scores to each of the responses. The graders compare their scores for each response and must ensure that the scores are within a certain margin of consistency. If not, they will have to discuss the grading of that response further in order to come to a consensus.

Finally, the graders record all of the scores for their question in a spreadsheet and send them to the Syllabus and Examination Committee.

5-6 Weeks After the Exam — Grading Session and Final Pass Mark

During this time, the Syllabus and Examination Committee, along with all of the graders for every CAS exam, meet in person for the grading session. The graders make modifications to the proposed solutions and provide sample responses as well as common mistakes, which will eventually be published in the Examiner’s Report. Once again, the graders review their assigned questions in order to determine what score a minimally qualified candidate (MQC) would achieve on that question. While a MQC score for each question is established during the construction of the exam, the graders may feel that it should be adjusted based on the hundreds of responses and solutions that they have studied (See December 2013 Future Fellows article for more detail on establishing a MQC score). Similarly, the pass mark for the exam was established months before, but the Examination Committee now has a chance to potentially revise it based on adjustments to the MQC score from the graders, any unforeseen solutions or defective questions, and modifications to the grading rubric. At this point, the candidates’ scores are compared to the pass mark, and any exam whose score is within a certain range from the pass mark automatically triggers further review. For these close candidates, each question is reviewed one final time by the graders, who must reach an exact agreement on what score the candidate deserves. Once all of the scores are compiled and the grading session is complete, the chair of the Syllabus and Examination Committee will give a detailed explanation of the exam and results to the CAS vice president of admissions as well as the Canadian Institute of Actuaries representative, who will give the final approval of the pass mark.

6-9 Weeks After the Exam — Results Release and Appeals Process

Approximately seven to eight weeks after the exam, CAS candidates will receive an email with a randomly generated URL to a webpage containing the candidate numbers of those who passed the exam. About a week later, all candidates, including those who passed, will be sent their numeric score (0-5 is failing and 6-10 is passing), and the examiner’s reports will be released on the CAS website. CAS candidates can now access their official grade report by logging into their CAS profile. The release of the examiner’s reports triggers the final phase of the exam results process: appeals. Candidates have two weeks from the release of the Examiner’s Report to appeal questions on the exam. Appeals must pertain to a specific question from the exam, and can either explain why the model answer from the examiner’s report is incorrect or why the candidate believes that their alternative solution is also correct. Candidates must give a detailed explanation in either scenario. Once the appeal is received, the graders for that question along with the Syllabus and Examination Committee will review the explanation, and the decision they reach on a score will now be now final.

The grading process for this round of exams is complete, but it will start over again in about four short months!