How Are Exam Questions Created?

by Rachel Hunter, FCAS, Candidate Liaison Committee

Volunteers, volunteers, volunteers! You've probably heard Future Fellows and other CAS communications talking about the critical role of volunteers in the examination process. In honor of National Volunteer Week, the CAS published an infographic in April showing that nearly 1,000 Fellows volunteer as part of the examination process. So Future Fellows decided to take a moment to focus on the work of the volunteers in the Syllabus & Examination Committee who help to write the exams. Exam writers put a great deal of time and care into their volunteering commitment. The art of crafting examination problems is a creative pursuit that takes most volunteers out of the normal structure of their day jobs and into the critical role of helping to assess candidate understanding of the syllabus.

How does a volunteer, who usually has little or no experience as an educator, go about developing quality exam questions? They are supported both by other volunteers with experience and by the improvements to processes that have been developed over the years by the CAS based on feedback from candidates and others on the examination process, including the examination questions.

New volunteers are asked to attend a training session that orients them to the process and to the typical style in which exam questions are written. Also included are some key dos and don'ts of exam question creation that have come from CAS experience as well as external consultants who have helped the CAS improve the examination process. After that, the full volunteer group, including seasoned writers with many years of experience on one exam or multiple exams, works to start crafting first drafts of their questions. These draft questions are reviewed by others on the writing team and questions are rewritten, clarified, and refined several times. The draft includes model solutions that are designed to help the graders later on.

Question reviewers refer to the portions of the syllabus that the question writer is drawing from and consider the following when providing comments to the writers:

  • Does the question cover anything not covered by the syllabus Learning Objectives and the associated readings?
  • Does the question have a potential for ambiguity?
  • Is the question written in a way that is consistent with the exam editorial guidelines, including the use of adverb prompts? (See "The Importance of Adverbs on Exams".)
  • Are the point values assigned to the solution appropriate?
  • Is the proposed solution correct and could there be other valid solutions?
  • What is the difficulty of the question for the minimally qualified candidate?
  • At what Bloom's level is the question testing the candidate's understanding?
  • How long will the question take to complete, including reading the question?

Finally, writers and reviewers meet in person at the writing summit and continue to refine the questions. For new and seasoned writers, this experience results in an opportunity to further improve questions and to build skills in question creation that will follow those who continue to volunteer the next year.

At the end of the writing summit, the exam writers typically have many high-quality questions that they have identified from refining the initial drafts. The work shifts to exam creation. Key members of the exam committee continue to work together to select which problems will ultimately make it on to the next exam. The goal is to choose problems that will test the candidate on a broad spectrum of syllabus content, at a variety of difficulties, and at a mix of Bloom's levels. This draft exam goes through even more review, including exposure to pre-testers, before being finalized for the next exam sitting. The original question writers may be involved again if there are proposed revisions to their questions selected for the exam.

While volunteering to write for exams is a big commitment, returning year after year also gives volunteers an excellent opportunity to network with others in the industry and deepen their understanding of the content on the exams. Most volunteers would like the exams to continue to improve in quality, and many of the CAS leadership started their volunteering with the exam committee. New Fellows are an important volunteering resource for the CAS, so please keep it in mind when you complete your exams.

If you are still a candidate and would like to help improve exams, do reach out and share your constructive feedback. One way is to contact the Candidate Liaison Committee through our online feedback form. We'd be happy to share candidate thoughts or work with the Syllabus & Examination Committee to try to address your questions in an upcoming Future Fellows issue.