Is Online Test-Monitoring Here to Stay?

When the coronavirus pandemic began, Femi Yemi-Ese, then a junior at the University of Texas at Austin, began attending class and taking exams remotely, from the apartment that he shared with roommates in the city. A former Division 1 football player, majoring in kinesiology, Yemi-Ese had never suffered from anxiety during tests. “Being in sports for as long as I was, and getting yelled at by coaches, I don’t get stressed much,” he said. He was initially unconcerned when he learned that several of his classes, including a course in life-span development and another in exercise physiology, would be administering exams using Proctorio, a software program that monitors test-takers for possible signs of cheating. The first time Yemi-Ese opened the application, positioning himself in front of his laptop for a photo, to confirm that his Webcam was working, Proctorio claimed that it could not detect a face in the image, and refused to let him into his exam. Yemi-Ese turned on more lights and tilted his camera to catch his face at its most illuminated angle; it took several tries before the software approved him to begin.

Like many test-takers of color, Yemi-Ese, who is Black, has spent the past three semesters using software that reliably struggles to locate his face. Now, whenever he sits down to take an exam using Proctorio, he turns on every light in his bedroom, and positions a ring light behind his computer so that it shines directly into his eyes. Despite these preparations, “I know that I’m going to have to try a couple times before the camera recognizes me,” he said. When we first spoke, last November, he told me that, in seven exams he’d taken using Proctorio, he had never once been let into a test on his first attempt. Adding sources of light seems to help, but it comes with consequences. “I have a light beaming into my eyes for the entire exam,” he said. “That’s hard when you’re actively trying not to look away, which could make it look like you’re cheating.”

Proctorio, which operates as a browser plug-in, can detect whether your gaze is pointed at the camera; it tracks how often you look away from the screen, how much you type, and how often you move the mouse. It compares your rate of activity to a class average that the software calculates as the exam unfolds, flagging you if you deviate too much from the norm. Meanwhile, Proctorio is also monitoring the room around you for unauthorized faces or forbidden materials. At the end of the exam, the professor receives a report on each student’s over-all “suspicion score,” along with a list of moments, marked for an instructor to review, when the software judged that cheating might have occurred.

Last spring, during a Zoom meeting with a professor, Yemi-Ese learned that the software had flagged him for moving too much. “I feel like I can’t take a test in my natural state anymore, because they’re watching for all these movements, and what I think is natural they’re going to flag,” he told me. His dread of the software only increased after he was kicked out of an exam when a roommate dropped a pot in the kitchen, making a clang that rang through their apartment. (Proctorio says that its software does not expel users from exams for noise.) By the time his professor let him back into the test, he had lost a half hour and his heart was racing. “I had to try to calm down,” he said. He feared that, if he showed physical signs of anxiety, Proctorio was “going to send the video to the professor and say that suspicious activity is going on.” The software, he said, “is just not accurate. So I don’t know if it’s seeing things that aren’t there because of the pigment of my skin.”

Yemi-Ese’s grades dropped precipitously early in the pandemic, a problem he attributed in large part to Proctorio. He took several tests while displaced from his home by the winter storm that devastated Texas in February, which forced him to crash with a series of friends. (The situation, in addition to its other challenges, deprived him of his usual light setup.) By the end of his senior year, Yemi-Ese was still struggling to get admitted to every Proctorio exam. Still, he managed to raise his grades back to pre-pandemic levels, even in classes that required Proctorio. “After I figured out nothing was going to change, I guess I got numb to it,” he said.

When college campuses shut down in March, 2020, remote-proctoring companies such as Proctorio, ProctorU, Examity, and ExamSoft benefitted immediately. (In a survey of college instructors conducted early in the pandemic, ninety-three per cent expressed concern that students would be more likely to cheat on online exams.) Some of these companies offer live proctoring underwritten by artificial intelligence. These include ProctorU, which said, in December, that it had administered roughly four million exams in 2020 (up from 1.5 million in 2019), and Examity, which told Inside Higher Ed that its growth last spring exceeded pre-pandemic expectations by thirty-five per cent. Fully algorithmic test-monitoring—which is less expensive, and available from companies including Proctorio, ExamSoft, and Respondus Monitor—has expanded even faster. Proctorio’s list of clients grew more than five hundred per cent, from four hundred in 2019 to twenty-five hundred in 2021, according to the company, and its software administered an estimated twenty-one million exams in 2020, compared with four million in 2019.

The surge in online-proctoring services has launched a wave of complaints. A letter of protest addressed to the CUNY administration has nearly thirty thousand signatures. Anti-online-proctoring Twitter accounts popped up, such as @Procteario and @ProcterrorU. One student tweeted, “professor just emailed me asking why i had the highest flag from proctorio. Excuse me ma’am, I was having a full on breakdown mid test and kept pulling tissues.” Another protested, “i was doing so well till i got an instagram notification on my laptop and i tried to x it out AND I GOT FUCKING KICKED OUT.” A third described getting an urgent text from a parent in the middle of an exam and calling back—“on speaker phone so my prof would know I wasn’t cheating”—to find out that a family member had died. “Now proctorio has a video of me crying,” the student wrote.

Other anecdotes call attention to the biases that are built into proctoring programs. Students with dark skin described the software’s failure to discern their faces. Low-income students have been flagged for unsteady Wi-Fi, or for taking tests in rooms shared with family members. Transgender students have been outed by Proctorio’s “ID Verification” procedure, which requires that they pose for a photograph with an I.D. that may bear a previous name. In video calls with live proctors from ProctorU, test-takers have been forced to remove bonnets and other non-religious hair coverings—a policy that has prompted online pushback from Black women in particular—and students accessing Wi-Fi in public libraries have been ordered to take off protective masks.

Jarrod Morgan, the chief strategy officer of ProctorU, told me that his company was in need of “relational” rather than technical changes. “What we will own is that we have not done a good enough job explaining what it is we do,” he said. Sebastian Vos, the C.E.O. of ExamSoft, denied that his company’s product performed poorly with dark-skinned people. “A lot of times, there are issues that get publicly printed that are not actually issues,” he said.

On December 3rd, six U.S. senators sent letters to Proctorio, ProctorU, and ExamSoft, requesting information about “the steps that your company has taken to protect the civil rights of students,” and proof that their programs securely guard the data they collect, “such as images of [a student’s] home, photos of their identification, and personal information regarding their disabilities.” (Proctorio wrote a long letter in response, defending its practices.) On December 9th, the nonprofit Electronic Privacy Information Center submitted a complaint to the attorney general of D.C. against five proctoring companies, arguing that they illegally collect students’ personal data. More recently, several students in Illinois have sued their institutions for using the software, alleging that it violates their rights under a state law that protects the privacy of residents’ biometric data.

Several institutions, including Harvard, Stanford, McGill, and the University of California, Berkeley, have either banned proctoring technology or strongly discouraged its use. (Harvard urged faculty to move toward open-book exams during the pandemic; if professors felt the need to monitor students, the university suggested observing them in Zoom breakout rooms.) Since last summer, several prominent universities that had signed contracts with Proctorio, including the University of Washington and Baylor University, have announced decisions either to cancel or not to renew those contracts. Meanwhile, rising vaccination rates and schools’ plans to reopen in the fall might seem to obviate the need for proctoring software. But some universities “have signed multi-year contracts that opened the door to proctoring in a way that they won’t just be able to pull themselves out of,” Jesse Stommel, a researcher who studies education technology and the editor of the journal Hybrid Pedagogy, said. “They have committed to paying for these services for a long time, and, once you’ve made a decision like that, you rationalize using the software.” (Several universities previously listed as customers on Proctorio’s Web site told me that they planned to reassess their use of proctoring software, but none had made determinations to end their contracts.)

Mike Olsen, the C.E.O. of Proctorio, said that his company could hold on to its COVID-era gains by presenting itself as a flexible, inexpensive alternative to traditional testing venues. “If you’re not wasting faculty and T.A.s’ time to proctor exams, schools save money,” he said. (Proctorio does require that instructors spend what can be hours reviewing footage of possible cheating. According to the Gazette of Iowa, an internal audit of fall, 2019, classes by the University of Iowa indicated that nearly three-quarters of faculty declined assistance in reviewing Proctorio alerts, and, “of those courses for which instructors rejected assistance, only 14 percent of alerts were reviewed.”) Universities would see value in Proctorio, Olsen said, even after in-person learning resumed: “No one enjoys taking tests, but, if we can let students do it in the environment they choose, that’s better customer satisfaction for the schools.” For now, the tens of millions of exams being proctored by software represent only a small fraction of over-all testing. “If we’re doing twenty-five million exams this year, I still think that’s only a drop in the bucket,” Olsen said. “For us, as a company, it’s an opportunity.”

In the spring of 2020, Tiffany Chu, an undergraduate at the University of British Columbia, stayed up all night studying for her economics midterm, then made a haphazard attempt to tidy her room in advance of the Proctorio “room scan”—a three-hundred-and-sixty-degree pan of her space that she would complete with her Webcam before starting the test, to prove that she didn’t have banned materials handy. (Unsure of what to do with her textbooks, she hid them under her comforter.) Proctorio also required Chu to verify her identity by holding up an I.D. card, and to prove that her Webcam was working by posing for a photograph. The software prompted her with a text command: “Get ready to smile!” But, when she clicked the link to start her test, she couldn’t get in.

She dashed off a panicked note to her professor and contacted Proctorio’s tech support via chat. As the minutes crawled by without a response, her anxiety started to build. “plEASE. MY GRADES ARE ON THE LINE,” she wrote in the chat after ten minutes of silence. A tech support employee, identified as Roy, replied “hi” before going idle. Chu typed frantically into the void:

MA’AM
SIR
I didnt pull an all nighter for this to happen
no wait come back

After two minutes, Roy reappeared and advised her to uninstall and reinstall Chrome; she did this several times, to no avail. Finally, with ten minutes left in the hour-long exam, Chu’s professor e-mailed her granting her permission to take the test another time. A few hours later, she completed the exam on her father’s laptop.

In the weeks that followed, Chu noticed posts critical of Proctorio on her university’s Reddit forum, r/UBC. Students wrote that the software made them anxious and miserable, that it invaded their rooms and violated their privacy. Chu posted a screenshot of part of her chat with Roy to the Reddit page. “Roy went MIA. Roy deserves a Nobel Peace prize for being a real one. thank you Roy,” she wrote.

Less than two hours after Chu posted the photo, a Reddit user named artfulhacker entered the thread and accused her of spreading “fake news” about Proctorio. In since-deleted comments, artfulhacker identified himself as Mike Olsen, the C.E.O. “I was furious we would leave a student hanging before an exam, but my team quickly pointed me to the transcript and I just had to jump in,” he wrote, publishing a partial log of the support chat to prove that Roy had, in fact, returned and tried to help Chu. “If you’re gonna lie bro... don’t do it when the company clearly has an entire transcript of your conversation,” Olsen wrote. “Shame on you.”

A few days later, Olsen publicly apologized. He sent Chu a private message on Reddit, including his e-mail address and urging her to contact him if she experienced “another technical hiccup.” Chu, who felt guilty about exposing a support technician, wrote back to apologize for “acting childish.” But, when Olsen replied asking for her permission to tell the press and the U.B.C. administration that they had “resolved the issue,” she didn’t answer. She still thought that Olsen had committed an ethical breach by posting her chat log. She still didn’t know why the program wouldn’t work on her laptop, or whether the same issues might recur. “He was trying to get me to de-villainize him,” she said, “and to say it’s been resolved, when it hasn’t been.” (U.B.C. has since decided to stop using algorithmic-proctoring tools, including Proctorio.)

Of the competing proctoring-software companies, Proctorio has gained outsized visibility in part because of its rate of growth, aided by a partnership with McGraw-Hill that was fortuitously announced in February, 2020. But the most salient difference between Proctorio and its peers may be Olsen. Last April, he pressured Hybrid Pedagogy to retract an article by Shea Swauger, a librarian and researcher at the University of Colorado Denver, that was critical of his company. (The journal refused.) In September, Olsen filed a lawsuit against Ian Linkletter, a learning-technology specialist at U.B.C., claiming copyright infringement after Linkletter tweeted publicly accessible company videos. (Linkletter told me that he has exhausted his life savings fighting the suit.) In October, Olsen pursued takedowns of a computer-engineering student’s tweets and Pastebin posts that pointed out potential security weaknesses in Proctorio’s handling of students’ private data. (The student is now suing Proctorio, alleging that the company misrepresented its copyright claims.)

When I first met Olsen, on Microsoft Teams last November, he sat in front of a white wall in his four-year-old son’s playroom, which he had converted into a cramped office. Cubby shelves stacked with Legos peeked into the frame. Olsen has low, heavy brows and the clipped hair and broad shoulders of a onetime jock. He spent his undergraduate years at Arizona State University, studying aerospace engineering and competing in hackathons. After graduating, in 2011, he took a job helping to build A.S.U.’s fast-growing online-degree program. While there, he was introduced to the concept of remote proctoring, which then consisted solely of live observation conducted by proctors in international call centers.

This practice struck Olsen as an inefficiency ripe to be rationalized. Paying a human proctor was expensive (around twenty-five dollars per test). Olsen thought that an algorithm could do the job cheaper and better. (Schools that sign up with Proctorio enter a yearlong contract for a flat fee, with some contracts valued at around half a million dollars, for unlimited use.) Olsen also believed that students would feel more comfortable being watched by a computer than by a remote proctor. “I kept hearing that this technology we were using was really bad. I wanted to solve a problem,” he said. In 2014, he pitched A.S.U. on developing an algorithmic proctoring service. When the university turned him down, he struck out on his own.

Even before the pandemic, Proctorio was growing quickly, roughly doubling its revenue and user count every year since its founding. Olsen describes Proctorio as a means of protecting the value of a college’s sales pitch. “For most institutions, the integrity is the brand,” he said. “Every university sells the same product—calculus is calculus—so the brand means everything. You can’t have a big old cheating scandal come out.”

Although most educators assume that cheating is more common when exams are online, research has suggested that the prevalence may not vary much from in-person exams. Stories about online cheating often rely on the say-so of proctoring companies, as was the case with a recent Washington Post article, which cited ProctorU to suggest that cheating had increased nearly eight-fold during the pandemic. Instead, the variable that most reliably drives cheating is pressure—of the kind that students feel when a single test determines a semester’s grade, for example, or when a certifying exam decides a career.

Evidence for the effectiveness of proctoring software is limited. One peer-reviewed study, from 2018, found that students who used Proctorio had G.P.A.s that were 2.2 per cent lower on average than those who didn’t—a possible indication that the system had prevented dishonesty. But the ways the software increases stress could also account for the difference, according to a 2019 study that found students were more likely to bomb proctored tests if they already suffered from anxiety. (Stories of students circumventing the software raise additional questions about its efficacy; on TikTok, videos of students finding ways around proctoring programs constitute a thriving genre.)

Online proctoring arose less from pedagogical needs than from economic exigencies. Online degrees provide revenue to cash-strapped institutions, and cheating is most likely an issue with assessments that require the least labor from instructors, such as multiple-choice tests. (Enrollment in online degree programs increased twenty-nine per cent between 2012 and 2018, and is projected to continue growing after the pandemic.) Chris Gilliard, a visiting research fellow at the Harvard Kennedy School’s Shorenstein Center, told me that paying a proctoring service for an online exam “is simpler, and scales better, than the things I think schools and instructors should do, like rethinking what assessment means,” or assigning creative work that takes longer to grade. A real rethinking of testing, Gilliard said, would involve “systemic changes that are not going to happen in the ten-day hiatus between when we went to lockdown and when school started back.”

Studies have shown that existing facial-recognition systems are up to a hundred per cent more likely to misidentify Asian and Black faces than they are white ones, and even more liable to misread Native Americans—a reflection of the fact that the software was trained using a data set in which white faces were overrepresented. The same systems perceive men more clearly than they do women or nonbinary people, and midlife adults better than seniors or children.

Proctorio previously referred to its program as using “facial recognition” technology, but switched to calling it “facial detection” in the past year, as bans on technology of the former type proliferated. For Olsen, the difference in terminology reflects an important distinction. Facial-recognition programs that have, for example, brought about the wrongful arrests of Black men in multiple cities are designed to match the face in front of the camera to an image in a database. Proctorio’s algorithm, by contrast, simply identifies whether a face is present and then tracks the direction of its gaze. The Federal Trade Commission, however, includes Proctorio’s feature under its definition of “facial recognition.”

If the software has trouble seeing dark-skinned students, it may be more likely to flag them for suspicious behavior, mistakenly claiming that they looked away from their screens because of its faltering grasp on their faces. So, when the program sends its report to the teachers, it could stoke unconscious bias. “Even if a teacher were to look at those videos and see that there wasn’t cheating, if they consistently see Black students with high suspicion ratings, that can play into how they treat them and grade them on other assignments,” Shea Swauger said. Because the University of Colorado Denver leaves the use of the software to the discretion of faculty, Swauger lobbies his colleagues not to employ it. “I ask them, ‘Would you care more if a few students cheated or if a few students were discriminated against?’ ” he said.

Students with disabilities have also claimed that the software compounds preëxisting forms of discrimination against them. Grace Massamillo, another student at U.T. Austin, was partway through her first quiz using Proctorio when her screen flashed and she was ejected from the test. Massamillo, who has optic-nerve atrophy, had been doing what she always does when she can’t read something on her computer: using the “pinch zoom” function to enlarge the text. But “resizing” is one of the actions that Proctorio flags, to catch “students who attempt to hide unauthorized materials behind the browser,” according to a marketing overview. “My first thought was, if I can’t zoom, I am going to fail,” Massamillo, a neuroscience major, told me.

The professor of the course, an introduction to medical terminology, told her that she had to use Proctorio’s zooming function, although it doesn’t enlarge the text as much as her usual method does. On the first big exam, Massamillo pieced together some of the questions using guesswork, and rushed through her answers to minimize the strain on her eyes. But the second exam included an X-ray. Massamillo inverted her computer’s color scheme to see it better; Proctorio flashed a stop sign, and she gave up, afraid of being barred from the test. “I was having trouble focussing, because I was thinking, Is my exam going to be void?” she said. After the next exam, she dropped the class. “I decided it wasn’t worth it,” she said.

Asking for accommodation has become more difficult during the pandemic, Massamillo said, because she hasn’t met the professor she’s e-mailing; she worries about appearing like she’s trying to game the system. “When they can’t get to know you as a good student, it furthers the weird distrust everyone is feeling,” she said. “I felt like I was fighting to prove my academic integrity more than my knowledge.” (When asked whether university administrators were aware that some students had experienced difficulty using Proctorio, a spokeswoman for U.T. Austin wrote, in a statement, “The university is aware of issues that have been raised with online proctoring apps. . . . Due to limited alternatives and time, the university renewed our Proctorio contract but we are actively assessing options.” She added that the university directs its faculty to grant exemptions to students who have trouble with Proctorio.)

Olsen has hired two auditors to investigate two of the most commonly cited flaws with Proctorio: BABL AI, a research consultancy, to look for bias against people of color and other minorities, and My Blind Spot, a nonprofit advocacy organization, to assess whether the technology is fully accessible to people with disabilities. “Even though everyone has made me out to be Darth Vader, I care about students,” Olsen told me. “Something like this has to exist, but there is a good way to do it—a better way. I think we’re the solution that’s doing that.” Olsen also asked BABL AI to help build a new generation of Proctorio’s software that is more responsive to race and gender variations.

It’s unclear how well a program that works in part by comparing students with both a class average and the median of a larger data set—encoding its preference for a physically and neurologically typical test-taker—can be adapted to accommodate disability. Albert Rizzi, the C.E.O. of My Blind Spot, told me that, although the program needs to be compatible with screen readers and other tools, students also need to advocate for themselves. Some differences, such as motor tics, will necessarily trip the software, he said, and, in such cases, the students have “a responsibility to articulate their needs.”

Cathy O’Neil, a leading figure in the nascent field of auditing algorithms, sees a need for accountability that goes beyond asking students to self-identify. “It’s not enough to say that people for whom this sucks should be excused, but there’s no policy in place even for that,” she said. “The burden has to be on the people running it, not the people for whom it has failed.”

On a Monday in early December, a month after we first spoke, Olsen greeted me over Zoom from his son’s playroom, bearing a noticeably uneven beard. He’d been letting his young niece trim it, he said, because he was avoiding the barber. In the course of two hours, Olsen’s son made three appearances, including one to ask for snacks and another to point out the arrival of the postman—the kind of background activity that Proctorio would flag.

When he makes the case for online proctoring, Olsen points to institutions whose students enter high-stakes professions. “I agree with the critics that you shouldn’t apply Proctorio to every kind of exam,” he said. “But, then, you have people who can literally kill people as students—like, at the pharmacy school, in their second year, they’re in the hospital giving out drugs.” He was referring to the University of Colorado’s Skaggs School of Pharmacy, which started working with Proctorio in 2014; Lisha Bustos, the manager of instructional design at the school, noted that her students do not counsel patients on medication until their third year. “We wanted to provide our students with the flexibility to take their exams wherever they wanted,” she said. “And then we want to make sure that our environment is as secure as it was when we required students to come to the classroom.”

Olsen said that many of the teachers who have turned to Proctorio in the past year don’t seem to understand how it works. This was his explanation for an experience shared by Arielle Brown, a student at North Carolina A. & T. State University, which purchased Proctorio as part of a McGraw-Hill package. Brown tweeted a screenshot of an irate, all-caps e-mail from one of her professors. “A STUDENT IN 6 MINUTES HAD 776 HEAD AND EYE MOVEMENTS,” the professor wrote to her class. She interpreted the fact that Proctorio registered these movements as proof that they constituted “negative behavior.” “I would hate to have to write you up for online cheating,” she threatened. “I told McGraw-Hill to kick that teacher off,” Olsen told me. “They basically came back and said, ‘We can’t.’ ”

Now he is in the process of writing an acceptable-use policy for schools and faculty, which will require that professors receive training before using the software. He is also developing a system for students to report negative experiences, “which will at least give us some insights into potential abuses,” he said. “If we feel like something crosses the line, we can take action.” Olsen believes that much of the criticism of Proctorio will dissipate once in-person learning resumes. He predicts that “students will be given the choice between taking the exam in their dorm room or a testing center,” he said, whereas “in the pandemic people were forced to use this. That’s where a lot of the friction happened.”

Olsen was hesitant to specify what he would see as an unethical use of Proctorio, but said that he didn’t always approve of the way institutions used certain features, such as the room scan. “You have some institutions who will just say, ‘Scan your desk,’ and then you have others who say, ‘Show your whole bedroom,’ or ‘Show under your bed.’ I don’t know where we can draw the line.” He pointed out that other proctoring companies also offer the room-scan feature. “I would lose business if I got rid of it.”

“We’re just providing the tech version of what already existed in the classroom,” he added. “The critics are saying that we’ve created a new context of mistrust, but there was always a level of mistrust.” Just then, his four-year-old wandered into the frame again. Netflix had auto-paused the animated series “Pocoyo” to ask, “Are you still watching?” The boy needed his dad’s help to get his show to play again.