TBE Post-Mortem: What Happened, and What Will Happen Next?
Future administrations of TBE have been deferred based on findings of the TBE Investigation Task Force.
After months of anticipation and preparation, the CAS administered its first Technology-Based Exam (TBE) this May. To both the CAS's and candidates' dismay, it did not go as planned. What happened and what does it mean for the future of TBE? I talked to three candidates who took the exam, Chloe Marshinski, Lee Drinkwater, and Dan Richard, and the current CAS Vice President of Admissions Jeanne Crowell to find out.
Before the exam itself, both the CAS and the candidates put in countless hours of preparation. The CAS spent months dedicated to working out and communicating all the details. The students prepared for the content of the exam, but also prepared for the new format of the exam. In addition to doing practice problems in Excel, this involved ensuring that candidates had the right technology so they could access the exam on exam day. That was the tricky part.
Some candidates bought Chromebooks at the CAS's recommendation, which was later withdrawn when the CAS found out that Chromebooks wouldn't work for the testing environment after all. Chloe and the four other candidates at her company planned to use their corporate laptops, but when the compatibility check came out 1-2 weeks before the exam, the secure browser didn't work. "We were calling tech support at PSI and they were saying 'oh don't worry, it'll work on test day, that's just an issue on our end.' We were skeptical that it was actually going to work on test day, so our IT department helped us figure it out and found workarounds to make it work. Our IT department put in so much work to help us," said Chloe.
Dan also planned on using his corporate laptop. Once the CAS and PSI figured out that corporate laptops wouldn't work for the exam, he decided to use a friend's laptop. "I didn't really use my last week to study much at all. My last week was spent almost entirely dodging all the hurdles that were placed in front of me with the TBE environment," he said.
Needless to say, exam day did not go as the CAS or candidates hoped. The Actuarial Outpost is filled with posts from students about their experience taking the exam: from technical difficulties preventing a student from taking the exam at all, to having minimal issues, to a proctor giving a candidate unlimited time to finish the exam. The candidates I talked to also had a range of experiences: One had no issues once they got into the exam, but another had a 14-hour exam process riddled with technical difficulties and had only a Clif Bar for sustenance for the entire day. Thankfully, all the candidates I talked to were able to get into the exam and finish it.
When asked whether the technological issues affected their ability to respond to the content of the exam, students' expectations seemed to play a role. Dan felt confident in the material going into the exam, but being unexpectedly locked out of the exam for a few hours shook his confidence. "You're worried about auto-failing, and at this point you're almost to the point of near crying, thinking you wasted six months of your life and you're going to fail because of all these technical difficulties, and it's hard to compose yourself once you're back in the exam environment. So I noticed that my first few questions didn't really go so well. I had to skip them and come back to them. Somewhere in the middle of the exam I got to a question I was more comfortable with, and I got a few of them in a row so I kind of settled down, and I could go back and I felt like I got my focus back."
Lee had a different approach. "People I knew had told me that they weren't able to get into their exam before my time slot even started, so I was expecting that things could potentially fall apart. I figured my only chance to not have a miserable day was just to roll with it as best I could." He contacted PSI to ask if he could play Xbox while he waited to get access to his exam; he could, so he did. The CAS was also trying to roll with it. Jeanne gave huge kudos to the CAS staff, who were trying to help talk candidates through it, even though there was not much that they could do to help with connectivity issues. "Candidates would call into the CAS office in tears, especially in the first sitting, and the staff would [try to help calm them down]" Jeanne said. "The CAS staff really went through a lot. And they did it with their heads held high. I have the utmost respect for them."
Once the exam was over, feedback came gushing in. As Jeanne puts it, "There was an outpouring of outrage by members and candidates that we let this happen. And understandably so. We fully understand that. And we were outraged as well. But there was also this underlying, 'What can we do about it?'" The CAS staff and volunteers were all determined to help make this right.
It was quickly decided that a makeup exam had to be given. The plan for a makeup was to use the exam written for the Fall sitting, which was still in its draft stages; the Exam Committee now had to finalize one exam within weeks and write an entirely new exam in time for the Fall sitting. The decision to use in-person proctors as opposed to virtual ones posed another challenge, since more proctors would be needed to watch each candidate's screen than would normally be needed for a pen and paper exam. To work through all the details, a myriad of decisions that had been deliberated over for the first sitting had to be decided in a much shorter timeframe to get the exam to candidates. Pulling it all off took an enormous effort on behalf of the CAS staff and volunteers in a short amount of time. Jeanne also made sure to give huge kudos to all the volunteers and staff who went above and beyond to make it happen.
The effort seems to have paid off. From the perspective of the candidates I talked to, the makeup exam was a much better experience than the first sitting. They had few (if any) technical issues and definitely preferred having an in-person proctor. Every candidate was afforded the opportunity to take the exam, which the CAS considers a success.
So after their experience with TBE, were candidates put off of the idea of an exam in Excel? Apparently not. All the candidates that I talked to agreed that Excel was preferred to pen and paper, and they hoped that exams would continue to be offered in this format in the future. Jeanne mentioned that the graders also had a much easier time working in Excel, and that the format even allowed them to remove the margin for calculator error that's usually built into pass marks (this is why the pass mark was higher for this exam than for previous sittings). There seems to be consensus that Excel-based exams are the right direction for the CAS to be heading — it's just a matter of figuring out how to get there.
As was announced in a press release on July 19, Exam 5 will not be TBE this fall. The CAS is taking the time to review what happened and learn from it before offering another TBE exam. Jeanne mentioned several issues that are being evaluated: processes for using live proctors; working with a vendor to determine the best way to have candidates take their exams on the same day; and solving the issue of latency between candidate's computers and the virtual machine. When asked about the fate of the other exams, Jeanne assured me that no other exams would be offered TBE before a successful sitting with Exam 5 offered through TBE. The intent is to offer Exam 5 in the spring 2019. The CAS will ensure that students know what format an exam is in before opening registration for it, as was demonstrated with the Fall 2018 sitting — the CAS delayed registration for Fall exams by a month so that they could decide on the fate of TBE before candidates registered.
Candidates certainly seem to be in favor of future TBE exams, and the CAS is listening. Jeanne said that the CAS will continue to listen to the candidate perspective through the Candidate Liaison Committee. So what do you think? If you'd like to provide feedback, contact us via the CAS feedback form.
