Academia.eduAcademia.edu
Changes in publication culture and the Stapel fraud case Ruud Abma Faculty of Social Sciences Utrecht University The Netherlands July 2012 (version 8 Aug. 2012) r.abma@uu.nl Recent events At the end of August of last year, the social psychologist and professor of consumer behavior at Tilburg University, Diederik Stapel, was confronted by the principal of his university with evidence that he had fabricated data, used for papers in social psychology journals. This evidence was gathered by three of Stapel’s PhD students, who had grown suspicious by both the results – that were always ‘too good to be true’ – and his evasive or intimidating behavior when asked for the original data. On 7 sept., Stapel confessed to the fabrication of data. Stapel was fired on the spot and the forgery was made public by Tilburg University. Also, a committee was installed to coordinate investigations into Stapel’s work at three universities: Amsterdam, Groningen and Tilburg. The committee is chaired by Pim Levelt, emeritus professor of psycholinguistics and former president of the Royal Netherlands Academy of Sciences. The results of the committee’s investigations are published on a special website (https://www.commissielevelt.nl/) with regular updates about the proceedings.1 Up until now it has been established that the majority of Stapel’s publications in the last ten years are fraudulent. As was to be expected, the news received an extensive media coverage, which was accompanied by critical remarks about the state of the art of social psychology. Psychologists immediately responded by emphasizing that Stapel was an exception and that psychological research in 99% of the cases was conforming to the rules of scientific integrity. 1 To arrive at the conclusion whether or not a particular publication (article or chapter in the dissertation) is fraudulent, the Committee, supported by statistical experts, takes the following steps: 1. 2. 3. 4. 5. 6. The data, as far as available, are examined for conspicuous oddities. The provided datasets are analyzed and its outcomes compared to the reported results in the publication. The provided questionnaires and stimulus material are checked against the provided data and against contents of the article. The publication itself is examined for suspicious irregularities. Information regarding the publication by the coauthor is taken into account, partially in reaction to questions posed by the Committee given the analyses so far. Judgment of Mr. Stapel about the publication is taken into account. The Committee concludes that a publication is fraudulent, when at least one of the studies based on experimental data in the article is fraudulent. Fraud in a study is concluded when the study is based on fictitious data, when the reported analysis was not performed on the collected data, when connections between questionnaires, data and publication results are lacking or when the presented results are extremely unlikely, next to impossible, in combination with other indications of fraud. Some of the separately mentioned reasons for fraud are more compelling than others, but it is the combination of these kinds of points that led the committee to the conclusion of fraud. 1 Others, including methodologists, pointed out that there is a whole ‘grey area’ between accepted types of data cleaning and outright scientific misconduct. In this grey area, a vast number of researchers would fall prey to the temptation of bending the rules, using procedures such as ‘cherry picking’ and ‘data cooking’. When last month another social psychologist, Dirk Smeesters, also a professor in the field of consumer behavior (Erasmus University Rotterdam), confessed to having manipulated his data, the reputation of social psychology took another blow. Of course, scientific misconduct is not limited to social psychology or the social sciences. At about the same time that Stapel’s fraud was revealed, a professor in vascular surgery was found out to have faked results on a massive scale. This also caused a stir in the media, but nothing compared to that caused by the Stapel case. Apparently, there is something about (social) psychology that fascinates the public and thereby generates an extraordinary amount of media attention. Some social psychologists (including Diederik Stapel) try to use this public interest to their advantage. For instance, a few weeks before Stapel’s unmasking, his colleague Roos Vonk sent out a press release, presenting a study done together with Stapel that showed that ‘meat eaters’ tend to be more aggressive than vegetarians. This is typically the kind of ‘facts’ that some experimental social psychologists come up with to get media attention. As soon as Stapel’s fraud was revealed, Roos Vonk dissociated herself from the study. Investigating the association between eating meat and aggressive behavior had been her idea, she said, but the actual research and the analysis of the data had been done by Diederik Stapel – at least that is what he told her. Working alone This was typical of the way Stapel worked. Together with a colleague or a PhD student, he would invent a research theme and hypothesis, and construct an experimental design. Subsequently, Stapel would volunteer to put it to the test – by himself. He claimed that he had a privileged access to high schools where subjects for his experiments and research assistants to help him collect the data were widely available. Or he claimed that he had data from previous research that were fit to answer the new research question. After such a solo intermezzo, he would return with the data already ordered in neat tables or even with readymade statistical analyses, gave them to his colleagues or students and / or worked together with them in writing the research article, which was, in his view, ‘the real work’ that PhD students should learn to master. On 31 October 2011, the Levelt Committee published an interim report, concluding that at least 30 journal publications, (co)authored by Stapel, had been based on fabricated data and that these fraudulent practices had been going on since at least the year 2004. Thereupon, Stapel withdrew his doctor’s degree and made a public statement, saying that: ‘social psychology is a great, important and solid discipline that offers splendid and unique insights into human behavior and therefore still deserves plenty of attention. (…) I have used improper means to produce attractive results. In modern science, the level of ambition is high and the competition for scarce means is huge. During the last years, this pressure has gotten the better of me. I did not cope adequately with the pressure to score, to publish (…). I wanted too much too fast. In a system where there is a lack of control, where people usually work alone, I have taken a wrong turn.’ 2 In this statement, it is striking that Stapel refers to flaws in the system (pressure to publish, lack of control, etc.), thereby neglecting that of his own volition he published considerably more than officially required or even necessary to make promotion. Also, he obscured that he evaded every attempt of control or collaboration regarding his data collection. So, what had been going on? Who was Stapel? How did his career develop? The history of Diederik Stapel Amsterdam (1985-1994-1999) Diederik Alexander Stapel was born in 1966 in the village of Oegstgeest, close to Leiden. After high school, he studied psychology and communication at the University of Amsterdam. After receiving his masters degree in both (cum laude, in 1991), he studied in the UK and the USA, after which he returned to Amsterdam to become a PhD student at the department of social psychology. This department had during the 1980s grown into a veritable research machine and Stapel was eager to learn the tricks of the trade. PhD students did not only learn how to conduct experiments, they also learned how to write successful publications. Special courses ‘how to publish?’ were developed by the British social psychologists Russell Spears and Tony Manstead, who had been invited to join the department. This was all new to Dutch social psychology, where teaching was the main task and publications – if at all – were in Dutch and focused (for instance) on human relations and the role of the mass media. Instead, starting in the 1980s, research became the main goal, preferably targeted at the role of social influences on the individual, with students as research subjects. Spears, who has a reputation of being a ‘publication canon’, says in retrospective: ‘During the 1990’s we had a golden generation of PhD students. We encouraged them all along to submit papers at the highest ranking journals in our field. (…) I believe that we implicitly have boosted the level of ambition of these students.’ In April 1997 Stapel got his PhD at the University of Amsterdam with the dissertation What we talk about when we talk about knowledge accessibility effects. Was Stapel already faking data at the time? The subcommittee investigating his Amsterdam period claims there is evidence that he did, based on the statistical patterns in his publications. It cannot be proven however, because the data have been destroyed. Moreover, Stapel, who has agreed to cooperate with the committee, states that he does not remember any fraudulent actions on his part from that period. Nevertheless, one feature of his usual conduct is already apparent: in the process of data collection and analysis he mostly works alone. Besides being a productive experimental psychologist, Stapel is also a creative person, interested in art, novels and poetry, and in acting (as a high school student he appeared in a Dutch sitcom). This probably helped him to come up with original and playful ideas for his scientific work. He was also fascinated by the effects of fraud on the image of psychology. After the clinical psychologist René Diekstra had been accused of plagiarism, Stapel, in collaboration with 3 Russell Spears and Wim Koomen, sent out an electronic questionnaire to Dutch colleagues in social psychology. In one condition, the plagiarizer was named ‘a psychologist’, in the other ‘a clinical psychologist’. As expected, in the latter condition the effects on the image of the discipline were considered less negative by the respondents. Stapel et al. (1999) concluded that by adequate manipulation even expert researchers of social influence (i.e. social psychologists) can fall prey to the phenomena they study. A striking conclusion, in the view of the massive deception he subjected his fellow social psychologists (including his own PhD students) to. Groningen (2000-2006) In 2000, at age 33, Stapel was appointed professor of cognitive social psychology, at the University of Groningen. Here, he perfected his strategies for maximizing his publication output, together with a variety of co-authors, 35 in total, mostly from the field of social psychology. In his inaugural lecture, held on 11 September 2001 (!), he presented social psychology as a brand of psychology that couples interest in everyday phenomena with precision and experimental rigour. While physics is the king of science, social psychology is the queen, with a razor in one hand (to cut away theoretical and rhetorical riff-raff) and a photo camera in the other (as a sign of its interest in daily life). Stapel explicitly rejects ‘grand theory’: ‘we should applaud the lack of grand theories in social psychology’; what we need is specific, small-scale theories. In Groningen, Stapel created his own network, mostly PhD students, and worked hard to improve his personal position in the faculty. In doing so, he elbowed across his colleagues in his department, including the one that had invited him to come to Groningen in the first place (Bram Buunk). For the first time, Stapel made enemies. His playfulness was gone, he meant business. Colleagues who had worked with him in Amsterdam during the 1990s, found that he had changed: ‘He had lost his former passion and curiosity, and had become cynical’. In informal conversations, he would denounce publication practices in social psychology as ‘empty’ and being driven by perverse stimuli. In a lecture for a group of former student friends from the Amsterdam students’ corps, he called science a form an undertaking that was primarily aimed at self-satisfactory activities (‘Wetenschap als zelfbevrediging’). Apparently, he had gotten second thoughts about the mechanisms of making a career in the social sciences. In spite of the full running of his publication machine, he was denied status of top professor in Groningen. Tilburg (2006-2011) When the university of Groningen thwarted his career ambitions, Stapel applied for a full professorship at Tilburg University, which also allowed him to enter the field of economics: he became professor of consumer science and director of TIBER, the Tilburg Institute for Behavioral Economics Research.2 2 NB Tilburg University was less prestigious than Groningen, where he, moreover had been a professor of cognitive social psychology. 4 In 2008, he gave his inaugural lecture ‘Searching for the soul of the economy’. Here, he contrasts the abstract nature of economic grand theories with the contingent mechanisms of social psychology: ‘It is the context that determines whether you cheat or not. You cheat when you’re angry, if the game lets you, if you don’t want to lose from your older brother, or if you play against your six year old daughter and do not want to win.’ Everything is relative, and social psychology shows you how. So, what then about the game of social psychology? What is the context here that determines whether you cheat or not? Maybe Stapel was angry and therefore decided to test if ‘the game’ permitted him to cheat. The least you can say is that he was ambivalent. In his inaugural lecture, he lashed out to the same publication culture that helped him to his professorships in Groningen and Tilburg: ‘Like the consumer that sees bargains and shopping-streets everywhere, the scientist that is rewarded to the publication sees potential papers everywhere. In the long run, this strategic behavior is not in the interest of the forum of science. It leads to scientific pornography, the result of a quick climax. It leads to trendy and conformist science at the expense of originality and creativity.’ The intrinsic pleasure of writing a good research article will be replaced by the extrinsic reward of the publication score. Why not let scientists pay for each article?, Stapel asks, that way they will only publish their best work. Two souls in one breast? So, it seems there are two Diederik Stapels: one that conforms to the high impact publication culture, with trendy articles that guarantee a good score in both scientific journals and popular media, and another that rebels to this culture, in the name of quality of content and long term satisfaction – corresponding to the intrinsic value of science. The inner friction between the two might have caused the anger, the cynicism and eventually the scientific misconduct of professor Diederik Stapel. How then might this misconduct have originated and developed? In a comment in Nature, Jennifer Crocker refers to the 1963 studies of Milgram, the results of which show ‘how easy it is to take the first small step on the slippery slope that ends with violation of our norms and values, and how hard it is to stop’. That first tiny step is significant: ‘Every minor transgression – dropping an inconvenient data point, or failing to give credit where it is due – creates a threat to self-image.’ According to the Levelt-report, Stapel might have started his fraudulent activities already as a PhD student – and was not found out! That must have been both rewarding and disappointing – as if your work was not being paid attention to! The reward was of course getting compliments for the almost perfect results of the experiments – they were ‘too good to be true’, as was often said to him. With a mixture of charm and power (charisma backed up by a strong formal position) he succeeded in curbing any doubts about the origin of the data that might come up in the heads of students and colleagues. The system of science When Stapel’s fraud finally was found out, the self-purifying capacities of the scientific community were hailed. However, behind this applause there lurked a disturbing thought: 5 how is it possible that Stapel had his way with data and publications for at least 10, but probably more than 15 years? Doesn’t this mean that the edifice of (social) science, with peer review as its foundations and impact scores as its floors, is starting to crumble? The interim Levelt-reports give an astonishing view on the pitfalls of peer review: ‘very doubtful results’, ‘highly implausible results’, ‘unbelievably high factor loadings’, ‘results extremely unlikely’ are among the verdicts of the committee’s statistical experts on by now some 50 journal articles with Stapel as one of the authors. The Stapel fraud case has prompted authorities in academia to argue for stricter rules and regulations and better education of students to prevent scientific misconduct. This sounds sympathetic, but is this evidence-based policy? There is at least one counter example: the professor who taught ethics to psychology students at Tilburg University was … Diederik Stapel. Apparently, knowing how to do the right thing is not automatically followed by the appropriate conduct. The Stapel case has also sparked a debate on habits in publishing and choosing topics for research. As it appears, the splendid results displayed in Stapel’s publications were seldom followed by attempts to replicate these results. Compared to producing novel results, replication is not ‘sexy’, and journals are not interested in publishing replication research. Also, reanalyzing and recalculating data from earlier research is not done (or not often done). Especially in the field of psychology, researchers tend to keep their data to themselves, a practice that in the future will become impossible, because of new regulations by subsidizers of research. Conclusion What is to be learned from this all? First of all, even if Stapel’s fraude is an isolated case – which it probably is not – his misconduct is at least in part the result of an unfortunate combination of perverse incentives and lack of scrutiny (by peer reviewers). Second, this unfortunate combination has a widespread existence within – at least – the field of social psychology, and is therefore influencing a great number of researchers. The pages long lists of publications by even young researchers show that in their daily practice they identify scientific quality with number of publications. As long as this system of reward and promotion keeps its grip on researchers, the risk of sliding down the slope from ‘data cleaning’ to ‘data falsification’ is in existence. As the philosopher of science David Hull (1998, 30) said ‘Melodramatic as allegations of fraud can be, most scientists would agree that the major problem in science is sloppiness. In the rush to publish, too many corners are cut too often.’ A third lesson to be learned does evolve from the Stapel case in a more indirect manner. It has to do with theoretical coherence. Half a century ago, E.G. Boring typified the field of psychology as an exploded confetti factory: research results flying everywhere like colourful bits and pieces, and no theoretical structure to hold them together. While Stapel in his two inaugural lectures applauded this focus on small-scale research and theories (‘theorettes’, as he called them), other social psychologists deplore the lack of the theoretical integration in their field. 6 It seems, however, that their attempts of catching and combing the confetti pieces – as exemplified in the recent two volume Handbook of theories of social psychology (Van Lange et al., 2011) – are undermined by the structural determinants of the current science system: the replacement of quality with quantity, a reward system based on number of publications and journal policies driven by impact envy. References - Broad, W. & N.J. Wade (1983) Betrayers of the truth. Fraud and deceit in the halls of science. New York: Simon and Schuster. - Fanelli, D. (2009) How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data. PLoS ONE, 4(5), e5738. - Fanelli, D. (2010). Do Pressures to Publish Increase Scientists’ Bias? An Empirical Support from US States Data. PLoS ONE, 5(4), e10271. - Heilbron, J. (2005) Wetenschappelijk onderzoek: dilemma’s en verleidingen. Amsterdam: KNAW. - Levelt, W.J.M. et al. (2011) Interim-rapportage inzake door prof. dr. D.A. Stapel gemaakte inbreuk op wetenschappelijke integriteit. Tilburg University. - Radder, H. (ed.) (2010) The commodification of academic research. Science and the modern university. Pittsburgh: Pittsburgh University Press. - Vugt, M. van (2011, September 10). What to Do About Scientific Fraud in Psychology? Psychology Today. Naturally Selected. 7