Exam Process Overview

There is much uncertainty surrounding actuarial exams. Will this paper be tested? What will the pass ratio be? How hard is the next exam? Unfortunately, we can’t help you with any of those questions. But, the process to create, implement, and grade an exam can be illuminated.

Note that this article only pertains to the upper-level exams (5 through 9).

At the most fundamental level, the exam process begins with the CAS Board of Directors and Executive Council. The Board sets basic education policy for the CAS, while the Executive Council oversees operational issues. In determining basic education policy and operational issues, the Board and Executive Council incorporate feedback from the Examination Committee, the Syllabus Committee, the Education Policy Committee, as well as various task forces that have been commissioned.

Once the education policy objectives have been outlined, it is up to the Syllabus Committee to create the exam syllabi. This involves determining learning objectives, knowledge statements, and selecting readings for each syllabus.

Creating an Exam

Now we’re at the fun part—creating an upper-level CAS exam. Before we get into details though, it is helpful to clarify the structure of the Examination Committee. The Examination Committee is the largest CAS committee, and the vast majority of committee members are assigned to one specific exam, either as a part chairperson, a vice chairperson, or a member. You can think of each exam as essentially being its own subcommittee. The remainder of the Examination Committee is comprised of the chairperson, assistant chairperson, consultants, and a handful of general officers. (Side note: If you know that someone is a member of the exam committee, do not ask which exam they work on. Examination Committee members can only confirm that they are on the committee—they are prohibited from providing any further detail.)

Based on the learning objectives, knowledge statements and syllabus readings, members of each exam subcommittee create several test items (questions). The items are written in pairs—that is, two members are assigned selected learning objectives and must jointly create questions based on those objectives. For each question they create, the item writing teams also propose the point value, a minimally qualified candidate score, a suggested response or responses, a grading rubric, and various diagnostics such as Bloom’s level and estimated time to fully answer the question.

Once the item writers have completed their questions, it is time to compile the initial draft of the exam. The process for doing this is actually evolving, and there are currently differences in how the various exams are produced. Historically, the part chairperson, vice chairpersons, and several experienced members of each exam subcommittee would create a draft exam. Recently, however, item writing summits have been held for Exams 5 and 6. All item writers for that exam are invited to attend the summit. The CAS is in the process of assessing the feasibility of conducting item writing summits for all upper-level exams.

After the initial draft exam is created, it is sent to consultants on the Examination Committee. Consultants are generally actuaries who are experienced members of the Examination Committee. The consultants review the proposed exam and recommend changes or improvements to the test items.

The next step involves assessing the exam in its entirety. It is presented to a recent Fellow or Fellows who played no role in creating the draft exam. The purpose of this step is to review the exam from a more critical “candidate” eye. The process helps the exam committee assess the length of the proposed exam and change or clarify questions based on feedback. The exam is also sent to a copy editor to check the spelling and grammar of each item.

The final draft of the exam is then submitted to the general officers and chairperson of the Examination Committee for approval. The proposed exam is also sent at this point to the pass mark panel, which helps to determine a preliminary pass mark for each exam as well as provide any other feedback.

Exam Administration

Now we’ve reached everyone’s favorite step—exam administration! Around early May and late October of each year, actuarial candidates around the world sit for the exams that have been meticulously drafted by the Examination Committee. (This is generally followed by drinking heavily, but that is the subject for another article.)

A (hopefully) infrequent, but very important step after exam administration is identifying defective questions. The Examination Committee works very hard to construct an error-free exam, but unfortunately invalid or incomplete questions do sometimes make it into an exam. If you believe that you have identified a defective question on an exam, please notify the CAS of the suspected error within two weeks of the exam.

After the exam responses are returned to the CAS Office, sorted, photocopied, and sent to the graders, the most labor-intensive portion of the process begins—exam grading. In the weeks after the exam is administered, each question is evaluated by a grading pair or triplet who must agree on each candidate’s score within some small margin. Note that the graders may or may not have been involved in writing exam questions. Once all papers have been initially graded, the committee members meet in a central location to finalize scores for each candidate. At this point, the grading teams must agree on a final score for each candidate near the pass mark. This is an iterative process in which candidate papers that are within a few points of the pass mark are re-graded (and re-graded) to ensure an accurate score.

Grading the Exam

Finally, based on the advice of item writers, graders, consultants, and the pass mark panel, as well as candidate performance by question, the Examination Committee recommends a proposed pass mark score for the exam. For each exam, the part chairperson creates a summary report that includes the proposed pass mark score and key statistics. The report is submitted to the vice president-Admissions for approval. Then comes everyone’s second favorite part of the exam process—anxiously awaiting results while hitting Refresh on the CAS website every few seconds!

For further information on the exam process, we encourage you to review past Future Fellows articles on exam-related issues and to read Pat Teufel’s open letter on the Fall 2011 exam results.