Exam Length — From Exam Writing to Candidate Feedback

by Celeste Bremen, ACAS, Candidate Liaison Committee Representative

Editor's Note: Only committee members' titles will be used in this article in order to preserve anonymity.

Exam time is a topic close to candidates' hearts. Over the course of several months, candidates invest hundreds of hours studying for the one day where they have four hours to show what they have learned. Candidates are keenly aware of time ticking on exam day, but it's not just those taking the exam who take exam length seriously — members of the Syllabus and Examination Committee also devote much attention to this important topic. To learn more about how exam length is determined, the CLC spoke with a few members of the committee.   

Exam length considerations begin at the question-writing phase and go through several checkpoints in the editing process. A committee general officer for writing notes that, as each question is constructed, question writers estimate the number of minutes they believe the minimally qualified candidate (MQC) needs to complete the question. This estimate includes not just time to answer, but time to read and think as well as work out calculation-intensive problems.   

Subject matter experts and other question writers also review questions. Additionally, each exam has volunteers called pre-testers who take the exam under exam conditions. Each pre-tester group is required to give feedback on the time needed to complete the questions. "Significant weight is placed on each groups' feedback, with the most weight given to the recent FCASs doing the pre-test," says a Syllabus and Examination Committee part chairperson. "We try to follow the exam process as closely as possible. They [pre-testers] have never seen the exam before; we highly value their feedback."   

The committee also takes into account candidates' comments and changes to the exam structure when estimating the time to complete questions. For example, the addition of Bloom's taxonomy over the past several years requires more reading and thinking, which affects the estimated time to complete a question. After reviewing completion time estimates for each question and considering writer and volunteer feedback, the committee makes adjustments to the exam. When compiling the final version of the exam, the exam committee must balance not only the time estimated to complete the questions but also syllabus coverage. Striking the right balance between completion time and exam material can be challenging, especially for exams with more diverse material as this may require asking a greater variety of questions. However, the Exam Committee always strives to ensure adequate coverage of the syllabus, and thus a fairer exam for candidates, while still ensuring that the final time given to take the exam is more than the time estimated to complete it.   

Another challenge for the exam committee is the recent addition of integrative questions, or IQs, to the upper-level exams. "The more feedback we get, the better we get at estimating the time needed for these questions," says a writing general officer. A changing syllabus and new types of questions create additional complexity in estimating exam length; the committee is constantly testing for and anticipating these challenges in order to create a balanced exam for candidates.

The feedback process doesn't end once the exam has been created. After taking the exams, candidates are asked to complete an exam survey. The Syllabus and Examination Committee also takes into account exam survey comments when considering the time needed to complete the exam. Part chairs read every survey comment for an exam. "The survey may seem like a black box, but we spend a lot of time reviewing candidate comments," says one part chair. For example, for the spring 2018 sitting of Exam 9, more than 80 percent of candidates responded to the exam survey.

But it is not just this written feedback that the committee uses to assess exam length. They also analyze candidates' exam results to see how many questions were left unanswered. Exam graders are required to note when candidates leave questions entirely blank. (Data on candidates attempting questions but receiving no points is also tracked separately.) The committee uses this information to better understand the appropriateness of the exam length and whether the unanswered questions were due to time or some other factor. Thus, grading is also a key metric.

One part chair comments that sometimes there may be a "disparity between how you feel coming out of an exam [and answer the exam survey] and how you perform in comparison with MQC expectations."

The time needed to complete the exam can be subjective, but the Syllabus and Examination Committee seriously considers several elements to improve the quality of the exams. Changes to the exam process can provide additional challenges, but the Syllabus and Examination Committee is constantly incorporating these changes, monitoring candidate exam performance and weighing feedback to improve this process.