Science and technology

Babbage

Technology monitor

Beating cheating

Mar 3rd 2011, 16:39 by The Economist online

Computer analysis of test-takers' answers to standarised tests is revealing cheats

TO SAY that cheats never prosper is to elevate hope over experience. Modern technology, in the form of miniature cameras, smart phones and the internet, means stealing answers and sharing them has never been easier. Indeed, the problem has got so bad that, on March 1st, the Japanese government asked the country’s universities to ban mobile phones from the rooms they use to conduct their entrance exams. Nor are students the only cheats. Teachers, whose salaries often depend on the success of their charges, are not above dropping the odd hint about what is the right answer before or during a test—or even correcting test papers after the event. And the invigilators who police the actual exam are not always immune to bribery, either. Conversely, however, technology allows cheats to be detected more easily than before. An arms race has thus developed between cheats and exam setters.

Data-forensics software, developed by exam-setting firms like Prometric, of Baltimore, Maryland, and Caveon, of Midvale, Utah, detects cheating by calculating the probabilities of particular patterns of answers being honest. This software exploits the relentless logic of combinatorial statistics. A correct answer is a correct answer, of course, but unless a candidate answers all questions correctly (itself an event that might at least raise suspicions), the mix of right and wrong answers provides information that can point to collaboration. If two candidates’ patterns of answers are similar or identical, warning flags go up. If more than two, hanky-panky is a racing certainty. 

In tests where the candidate is allowed to change his mind about an answer, the pattern of changes also provides information. Several candidates making the same change is suspicious. So is a case where all changes are from wrong to right.

Sudden improvements in scores by an individual candidate, compared with previous attempts, also raise an electronic eyebrow—particularly if the new result is from a different exam hall. Crossing an international border to take a test is suspicious, too, and doubly so if it is from a place generally reckoned clean to one generally reckoned corrupt. American candidates who fly all the way to, say, India, for their exams may thus find themselves with some explaining to do.

A company called Kryterion scrutinises matters even more closely than that. Kryterion administers its tests online, and the invigilators sit at its headquarters in Phoenix, Arizona, watching test-takers around the world through webcams and never meeting them in person. The opportunities for envelopes stuffed with banknotes to change hands are thus minimised. Software that recognises facial features and keystroke rhythms stops candidates being impersonated by professional exam sitters. Remote computers are “locked down” with security software, to prevent unauthorised windows from being opened. Invigilators warn or disqualify test-takers whose eyes or hands wander in suspicious ways. And the software also alerts them if difficult questions are being answered suspiciously quickly, or if two test-takers’ answers match too closely for comfort. 

Nor does the scrutiny stop when the test is over. Caveon and Prometric operate “web patrol” software that hunts day and night for illicitly revealed test information. And Prometric, for one, has an additional security measure. It inserts a unique question in some individual exam papers. If that question later appears online, only a handful of people would know to put it there. 

Prometric now detects so much cheating that, every week, it investigates about 20 of its 5,000 test centres around the world. Of those, around five a week have to be shut down permanently, while numerous administrators and invigilators at even those places which survive investigation are fired for lax security or accepting bribes, and undisclosed numbers of tests-takers are charged with cheating. How different from the days when an answer scribbled in biro on the palm of a sweaty hand was all that stood between a candidate’s success and failure.

You must be logged in to post a comment.
Please login or sign up for a free account.
1-16 of 16
Mar 3rd 2011 8:50 GMT

The article fails to notice that the error lies in exams and tests instead.

If a student can find the answer on his internet-equipped mobile during the test, then he can do the same in real life and probably needed not to memorize the answer at all.

Too many tests and exams rely on regurgitating useless data. Such exams are easier to create than ones testing skills and qualities, tickle vanity of dusty academics, but are outdated in the internet age.

Engineer no 1 wrote:
Mar 4th 2011 12:38 GMT

Mr. Common Sense,

I do agree with your argument that most exams rely on regurgitating useless data, but test-takers are expected to know that useless rudimentary data. If in real life you are relying on internet to find very basic science skills then you better not appear for exams. Moreover, before putting any correct answers on the internet someone has to find those answers manually.

However, Testing academic abilities of millions of students is not easy task.

Anjin-San wrote:
Mar 4th 2011 1:53 GMT

The reason for the sudden request by the Japanese government on March 1st was the spate of online cheating at 4 universities (all by the same individual) utilizing real-time crowdsourcing of test answers via mobile phones. The individual in question was arrested yesterday (March 3rd), but I suspect he is actually an accomplice for the true beneficiary (his girlfriend maybe?) who merely had to browse the website in question without leaving a trace.

HWLanier wrote:
Mar 4th 2011 6:40 GMT

This is a fascinating example of evolving intelligence. From a different perspective this student was utilizing modern technology and techniques. Our intelligence is layered, we learn short-cuts and equations to speed the process of discerning our surroundings and understanding. What we have here is an innovative example of such, initially judged as cheating.

HWLanier wrote:
Mar 4th 2011 7:03 GMT

My claim above is far too strong, for example copying & pasting into google and the same onto the test, why you wouldn't even have to know what classroom you were in -- yet there's a facet to this that is an example of utilizing the technology existent. Makes one wonder how to frame the tests in the future because these events are only going to be more common. Yet it's not all bad is what i'm sayin'. Brings to mind the intro to Wicksteed's Economic treatise.

luso_star wrote:
Mar 4th 2011 4:03 GMT

In my university in Portugal, I was allowed to use anything I wanted. I used to take my laptop do my exams and be on msn during them.
Of course with unlimited access to information you better know before hand what you are looking for, otherwise by the time you find it, the exam is over. Its a process of making people smart and efficient rather than momorizers.

IMB9 wrote:
Mar 4th 2011 5:21 GMT

The problem is "meritocracy". The earning ability of millions of people depends on how well they do on some random tests. Whether these test are indicators of the true abilities of the test-takers, who knows.

Example: X is a very good student and test taker. He gets into the best bussiness school in the world and from then on he has the guarantee of a high power, high paying job. He will be well paid to play with other people's money (and to boss others around) all his life. Obviously, after the test the true abilities of X will never be assessed again (the success of a manager is often the success of his/her subordinates, b.s. and connections matter, etc.).

So: if cheating on one test can put you on a path to financial and professional success many people will keep trying. The problem, in my opinion (after successfuly taking many life-changing tests) is not the cheating, but the sistem that judges the people (and implicitly sentences them) basen on some test score.

math-phys wrote:
Mar 4th 2011 8:13 GMT

Dear Sir
The great problem fully missed on this quite sensitive issue is the growing of specialized groups outside the academy and schools involving professional students , instructors, officers,militaries etc, all given support (solving exams's questions!) to the branche of Professional criminal Mafias specialized in selling "help" in real time and on line for all those students ready to cheat exams -really high tech electronic gadgets are used -professional spy stuf !
Certainly, things are getting more and more subtle and worse!. By now, it is widely suspected that there are many Instructors involved in such Cheat Mafias who receive considerable amount of bribery dirty money and benefits just to make exams more and more difficults ,but with the clear target on stimulating such new criminal mafia market on Colleges and Universities.
But worse of all is that all those students using such criminal protocol of "helping in exams and class works assignements" in an apparent innocent way , end up sooner or later deeply involved in others criminal Mafias activities branches like ;Making pornographic adult movies, traffic of drugs, prostitution , etc...The mobsters only "help" those students that are ready to get dirty as "security key " of the cheat action !.Very devilishly indeed !.

KevinSchnider wrote:
Mar 4th 2011 11:25 GMT

My thermodynamics professor had a much better way of dealing with this problem... and it required no fancy cameras, monitoring, or software.

He made every exam was 'open world'... which means that ANY resource can be used during the exam. However, his tests were absolutely brutal and proctored under a strict time limit. If a student didn't immediately know exactly what they were doing, it would be impossible to finish the exam within the hour.

In the end, all those textbooks, laptops, friends, notes, and whatever else didn't really help anyone. Unless you knew your stuff and exactly where to look, you would run out of time trying to 'find' the answers in your textbook / internet / whatever...

Next_rim wrote:
Mar 5th 2011 4:23 GMT

KevinSchnider,

I agree with you that it is possible to make up the test in a way that excludes any possibility of cheating. And some professors get really good at it. I had a couple of those back in college.

However, the problem arises with standardized tests that are taken by endless bullions of people, like admission exams and certificate exams. Unlike in a course exam, you can't possibly test all the material, or score open answers in reasonable time.

It is possible to cheat on a standardized exam simply because it is standardized. And there is no easy way out of this. If you raise the difficulty level, exam takers will guestimate too much, and you will have to curve the admission score accordingly. If you engage in an arms race, students will eventually outsmart you, since they are the younger and more tech savvy generation anyway.

I think the best solution is to deviate from a standardized test, and go back to individual interview-like exam, where you solve a set of problems, then stand up in front of a commission and defend your answers. Each university then has the option of making up their own exams to match expectations and select students with a set of skills and knowledge suitable for the particular program.

Mar 5th 2011 10:07 GMT

@NextRim

Videoconferencing makes your system semi viable but it still restricts the pool of qualified applicants. By all agreeing to accept the scores from a similar test, universities save themselves the expense of having to do personal interviews for 4000 applicants every year. Harvard or Yale can afford to but small universities rely on the tests to screen out the huge number of applicants.

Furnald Hall wrote:
Mar 5th 2011 5:24 GMT

One might suppose cheating one's way into a university, particularly where the expected work level is going to be well above one's abilities, is like jumping from the frying pan into the fire -- understandable for the prestige it brings among peers, but ultimately a self-defeating and irrational act.

Here in Ontario, Canada.....ummmmm........how to phrase this.... happily, there now seems to be a prominent cottage industry that changes this dismal picture. Once admitted, those attempting such an illicit end-run know, there is a not-even-very-discreet-any-more service sector at one's disposal of "essay research and editing services" (they advertise heavily on the power poles around the various university campuses, especially the University of Toronto downtown)that can help such "students" get their desired degrees effortlessly (even post-graduate degrees), and without any need for even an unbearably tedious modicum of traditional skill development. I personally see no evidence that the universities themselves care enough to separate out, and stop, this cohort of students from graduating. As a strategy, therefore, cheating your way into a good Ontario university, "facilitating" ones graduation by the same methods, then squeezing out more deserving candidates in the competition for good jobs, appears to pay off and appears quite doable.

Is the situation the same in other countries, I wonder?

tocharian wrote:
Mar 5th 2011 9:54 GMT

Many students cheat because they see that that's the way to succeed in this greedy world. They see it happening in the financial world (Wall Street, investment banks, advertising, ...) in the political world especially on an international scale (doesn't China cheat by not respecting copyright laws?).
If you don't want our children to cheat we have to totally overhaul societal values. Kids learn from grown-ups.

Mar 5th 2011 11:24 GMT

Perhaps one problem is the fact that the standardized test market has been cornered by a relatively small number of businesses. It wouldn't be so easy to cheat on the Educational Testing Service's exams were it not for the fact that the vast majority of students seeking entry to US graduate programs have to take the GRE, one of their standardized tests which is administered by Prometric.

boontee wrote:
Mar 6th 2011 2:59 GMT

Assessing a student’s capability and potential can come in a variety of ways. A single paper-and-pencil test (especially the multiple choice/completion) is not the best measurement, it should be phased out gradually. Instead, a triangulation of assessment procedures would be more valid and reliable, though that will take a lot of good examiners’ effort and time. Nevertheless, it will reduce and minimize the effect of cheating.

Cheating in tests is as old as the test itself. With the advent of new communication technology, it could get worse, not to mention the increasingly widespread of plagiarism nowadays.

Honesty is virtuous. But how many of us are? (btt1943, vzc1943)

rotorhead1871 wrote:
Mar 6th 2011 4:46 GMT

the test should examine thinking skills, not memorized levels of data.

1-16 of 16

About Babbage

In this blog, our correspondents report on the intersections between science, technology, culture and policy.

Follow Babbage on Twitter »

Advertisement

Advertisement

Latest blog posts - All times are GMT
Et tu, Zagreb?
From Eastern approaches - 1 hrs 18 mins ago
A pointed resignation
From Newsbook - March 6th, 9:47
Trains and partisanship
From Gulliver - March 5th, 20:49
Opponents unknown
From Multimedia - March 4th, 22:30
More from our blogs »
Products & events
Stay informed today and every day

Subscribe to The Economist's free e-mail newsletters and alerts.


Subscribe to The Economist's latest article postings on Twitter


See a selection of The Economist's articles, events, topical videos and debates on Facebook.

Advertisement