The evaluation of scholarship in academic promotion and tenure processes: Past, present, and future

Introduction

There is some question as to whether the academic system, and its means of evaluating the worth of its faculty’s contributions, has kept pace with the rapid evolution of technology and communications (e.g., Genshaft et al., 2016; Howard, 2013; Piwowar, 2013; Sanberg et al., 2014), as well as with societal goals such as ensuring equal opportunities for employment and career advancement regardless of gender, ethnicity, or other personal characteristics (e.g., Johnsrud & Jarlais, 1994; López et al., 2018; Menges & Exum, 1983; Whittaker et al., 2015). Some common complaints about academia, including those focused on lack of reproducibility (Open Science Collaboration, 2015), problems with peer review (Ross-Hellauer, 2017; Smith, 2006; Tennant et al., 2017), and the lack of access to research, could conceivably be reduced by building mechanisms that capitalize on freely available online communications and information-sharing tools. There is a clear desire by some, if not many, to make changes to the ways in which we organize academic activity and the dissemination of its products (e.g., Ellison & Eatman, 2008; O’Meara, 2014; O’Meara et al., 2015; Sid & Richardson Foundation Forum, 1997). However, there are barriers to change.

Chief amongst these barriers are the incentive structures currently in place for faculty career advancement. As Buttliere (2014) pointed out, “the problem is an ineffective reward system which makes doing the prosocial action … bad for the individual because it less efficiently achieves high impact work and thus promotion” (p. 1). To address the problems of academia, and conceptualize how its scholarly communication system might be improved to promote the active sharing of information and support a more efficient and transparent approach to conducting research, it is essential to understand the explicit rules of the game. To aid in this understanding, this paper offers a synthesis of the literature on review, promotion, and tenure (RPT) practices in the United States and Canada.

Identifying RPT issues and areas for reform

The literature shows that there has been discontent, especially on the part of faculty but also in administrators, regarding the methods used to evaluate faculty for tenure and promotion (e.g., Diamond & Adam, 1998; Gordon, 2008; Harley et al., 2010). The major concerns about the RPT process can be grouped into two themes. Firstly, many faculty experience a dissonance between the apparent focus of RPT evaluation on research and publication, versus their actual work responsibilities, which often result in spending over half of their work hours on teaching (e.g., Diamond & Adam, 1998). Some also spend a great deal of time on service activities, which are barely recognized in the RPT process (e.g., Foos et al., 2004; Mamiseishvili et al., 2016). Secondly, faculty are concerned about the amount or type of publishing that is expected of them, the way their published works are assessed, and that the venues in which they are expected to publish (i.e., prestigious international and national journals, and university presses) don't have the capacity to support the amount of publication that universities want from their faculty (e.g., Adler et al., 2009; Brembs et al., 2013). Both of these factors can lead to frustration for university faculty.

That both of these concerns involve the publishing of research is not a coincidence: the challenges in scholarly communications and those of career advancement are intricately linked. To reduce the incongruence experienced by those wanting to both appropriately communicate their scholarship and advance successfully in their careers, it is necessary to understand the process that rewards activities within academia, especially as it pertains to publishing practices. A greater understanding of the RPT process may reveal an effective and efficient means for change. Changing the RPT process might lead to a reduction of the reliance on publication prestige or easily manipulated citation metrics, a restructuring of the peer review system, and even to an improvement in the quality, affordability, and flexibility of format in publishing venues. We begin by examining research that discusses how research, teaching, and service are viewed in RPT, and how the process places an emphasis on research and publications.

Research, teaching, and service in the review process

In general, candidates for tenure and promotion are judged based on their research and publications, teaching effectiveness, and service. Although explicit weights for each aspect are not typically provided in RPT guidelines and policies, most faculty, across disciplines, assume that a strong research and publication record is necessary, and lack thereof cannot be compensated for by excellence in teaching and service (Green, 2008; Harley et al., 2010; Youn & Price, 2009). This pattern has remained consistent since at least the 1990s.

Tenure and promotion requirements have changed over time: in the 1980s most university departments wanted to see excellence in at least one of research, teaching, or service (Gardner & Veliz, 2014), and then a shift occurred in which excellence in teaching and service was no longer sufficient to earn tenure (Youn & Price, 2009). By the 2000s, excellence in all three was expected with the most focus placed on research. This trend began to elicit concerns: in a 1989 survey of 5000 faculty (from two- and four-year U.S. academic institutions, across a spectrum of different disciplines), 68% agreed with the statement, “At my institution we need better ways, besides publications, to evaluate the scholarly performance of the faculty” (The Carnegie Foundation for the Advancement of Teaching, 1989; p. 52). Although 71% of these faculty preferred teaching to research, publishing was considered a dominant factor in determining faculty career success (The Carnegie Foundation for the Advancement of Teaching, 1989). Similarly, in 1989-1990, the Higher Education Research Institute surveyed over 35 000 faculty who taught undergraduate courses, representing all categories of higher education institutions in the U.S., and 44% of faculty at public universities felt that institutional demands for research productivity interfered with their ability to teach effectively (Astin et al., 1991).

Fairweather (1993) analyzed surveys of over 4000 faculty in four-year colleges and universities and found that research productivity best predicted success in promotion, tenure, and salary increases across institution types and disciplines. Teaching was rarely a contributing factor to RPT success, and, in some cases, salary appeared to be negatively influenced by teaching hours. Commentary at the time reflected these ideas: an article appeared in The Chronicle of Education entitled “Teaching Awards: Aid to Tenure or Kiss of Death?” and another article commented, “Some professors … regard the Teacher of the Year Award as the kiss of death … I personally know three different professors at three different institutions who have gotten the Teacher of the Year Award and were then told that their contracts would not be renewed” (Sowell, 1990, p. 69).

Quantity, quality, and prestige of publications for RPT

If it is clear that research and publications are presently the most important components of the review process, then what should academics focus on: quantity, or quality? Or is it about seeking prestige? Publications in the most prestigious venues are not necessarily those of the highest research quality; other factors such as the editors’ perceived novelty and importance of the findings also determine likelihood of acceptance for publication.

Some aspects of the evaluation of publications in RPT appear to have remained relatively consistent over the past few decades. In the 1990s and early 2000s, several studies found that those evaluating faculty for promotion or tenure preferred to focus more on the quality of their research and impact of publications as opposed to the quantity of papers (Cronin & Overfelt, 1995; Estabrook & Warner, 2003). Department chairs liked to see “value,” “quality,” “legitimacy,” and “weight” in the publications (Andersen & Trinkle, 2001). Publishing in peer-reviewed journals was, and remains, a key to demonstrating research quality (Acord & Harley, 2013; Andersen & Trinkle, 2001; Cronin & Overfelt, 1995; Harley et al., 2010; King et al., 2006; Seipel, 2003) and the quality of the peer review offered by particular journals is also an important consideration (Andersen & Trinkle, 2001).

Despite consistent value being placed on research quality and peer-reviewed publications, there is some concern that RPT research and publication requirements are gradually increasing, resulting in greater workloads and an imbalance between varied job responsibilities and the reality that faculty are expected to produce more papers and books than ever before. Estabrook & Warner (2003) provided evidence that standards for publishing in book-centric disciplines had increased based on the reports of faculty in History, English, and Anthropology departments who had received tenure. Similarly, academics in other disciplines feel pressure to publish particular numbers of articles due to RPT policy (Walker et al., 2010). There can be either formal, or informal and verbally communicated, expectations regarding the number of articles required for tenure or promotion. For instance, King et al. (2006) found that at UC Berkeley it is typical to need three or four peer reviewed articles per year to succeed in RPT applications in biostatistics and chemical engineering. To achieve full professor in chemical engineering, one needed about twenty papers in major journals as well as widespread and international recognition in their research specialty. Similarly, Foos et al. (2004) reported that 27% of US geoscience departments had guidelines regarding the number of publications needed to earn tenure: the requirement was 3.7 publications on average, with a range between one and twelve.

Harley et al. (2010) found that across various disciplines, those assessing faculty for tenure or promotion were looking for numerous and exceptional publications that represent significant progress in their field of study, are deemed high in quality by both internal and external reviewers, and can be described as “groundbreaking,” “indicative of sustainable scholarship,” and “lauded by the larger community of scholars” (p. 7). It can be difficult to quantify exactly how many journal articles are necessary for tenure within and across different disciplines – the guidelines are not always specific, and can allow for some flexibility, especially in order to take quality into consideration.

Regarding top-tier versus second-tier institutions, Harley et al. (2010) found that some faculty perceived second-tier institutions to have less stringent publication requirements. The list of acceptable journals and presses was thought to be more inclusive, fewer publications were needed, and more emphasis was placed on teaching. Similarly, economics department chairs revealed that more prestigious departments required more second-tier publications to make up for the lack of publishing in a top-ten journal (Liner & Sewell, 2009). However, some faculty in Harley et al.'s study thought that the requirements at top-tier research universities influenced the policies of lower-ranked institutions, with lower-ranked institutions attempting to move up the rankings by increasing their research presence.

Faculty rank can influence the career advancement process from both the applicant and the evaluation sides. On the applicant side, the evaluation process can be qualitatively different for tenure applications versus applications for promotion to full professor. Harley et al. (2008); Harley et al. (2010) found that assistant professors applying for tenure feel pressure to publish only in high-impact, high-prestige venues, whereas associate professors may publish in more varied formats, even including encyclopedias or electronic resources. It may be associate professors applying for promotion who pave the way towards inclusiveness of different media in the RPT process – this group, having already been granted tenure, has a tendency to be more open-minded towards non-traditional forms of publishing (Harley et al., 2010). That being said, Liner & Sewell (2009) reported that in economics departments, those applying for promotion to full professor had to compensate more than those applying for tenure if they lacked publications in top-tier journals.

There is also evidence that shared authorship can influence the value placed on publications in RPT evaluations, which can be a cause for concern in fields of research that are increasingly collaborative in nature (e.g., Soares, 2015). Walker et al. (2010) found that journal article authors ranked journal impact factor, number of publications, and order of authorship as most crucial for tenure and promotion, whereas the number of authors on a paper was less of a concern. At some universities, only the first or corresponding authors received credit in the RPT process, whereas in other institutions, second and third authorship was rewarded. First or corresponding authors tended to benefit the most towards promotion, tenure, and/or financial compensation (Mahoney, 1985; Seipel, 2003; Wren et al., 2007).

Modern approaches to evaluating research output

Numerous advances have occurred in scholarly communication over the last decades, some of which include online publication and databases, academic use of social networks, and analytic tools aimed at quantitatively assessing the reach of individual publications. Specific metrics have been developed that have the potential to reflect the influence of a candidate's publications in their field of specialty more accurately than the impact factor. Are such alternate citation measures considered in RPT evaluations?

Indeed, some institutions have begun to consider additional citation metrics, such as counts per journal article, in their decision making process (Reinstein et al., 2011). Such citation searching may be required in the RPT application and it may not be an easy task for the candidate to carry out. The amount of support available from the university library varies across institutions (Dagenais Brown, 2014), although there are freely available online resources that provide guidance in choosing and interpreting scholarly literature metrics for different situations (e.g., http://www.metrics-toolkit.org). Indeed, some have predicted there will be a movement towards using alternative metrics (altmetrics) to assess the influence of research findings for RPT (Darling et al., 2013; Piwowar, 2013). Altmetrics can involve such measurements as views, discussion posts, or social media shares, of either the original research articles or other products that result from the research, such as datasets.

The idea of altmetrics is still quite new – the term itself was coined only in 2010 – and so the integration of these alternate measures of research communication with RPT processes remains in flux (Howard, 2013). Some view altmetrics as a potentially informative addition to RPT evaluations, but there are concerns regarding the value of the data. For instance, a low-quality publication in a broadly interesting, or new and exciting field of research may generate a lot of online “buzz”, whereas a high-quality publication in a niche field may attract far less attention. Accordingly, although Gruzd et al. (2011) found a majority (65%) of library and information science faculty agreed that online social media use should be considered in the tenure and promotion process, most were unsure of exactly how such professional social media use should be formally evaluated. Also, 73% of faculty in this study stated that online social communication tools have significantly influenced how they use traditional information sources. This widespread, but currently informal, use of social media (including forums like Twitter, Mendeley, and blogs) has become an integral part of how some academics stay informed on progress in their fields, and can even help to accelerate the pace of scientific discovery. Despite this, only a minority (12% of faculty) in the Gruzd et al. (2011) study reported that their tenure and promotion procedures acknowledged so-called alternate forms of scientific communication.

Accordingly, there is little published evidence of RPT procedures directly acknowledging academic service involving outreach to the academic and public communities. In fact, Harley et al. (2008); Harley et al. (2010) found that across a number of disciplines at research-intensive institutions, pre-tenured faculty were encouraged to focus on high-impact publishing and not invest too much time on committee work, public engagement, or writing in non-traditional formats such as commentaries or blogging. Although raising scholarly visibility with blogs, working papers, or preprints may indirectly help a tenure application, Harley et al. (2010) reported that these items are not typically included in tenure applications, and may be considered neutral or even negative in the review process. Similarly, Goldstein & Bearman (2011) found little emphasis on community service or engagement in the RPT process at medical schools. In general, these types of activities, along with the sharing of unpublished work and using social media such as tweeting, haven't been valued by tenure and promotion committees but there is some indication this might begin to change (Fox, 2012; Gruzd et al., 2011; Piwowar, 2013).

One example of scholarly social media being considered in RPT evaluation is that of the Mayo Clinic – starting in early 2016, digital portfolios were allowed in evaluations for promotion (Cabrera et al., 2017). Cocchio & Awad (2014) reported that across medical, nursing, and pharmacy programs, deans have varying views regarding the value of social media in the evaluation of scholarly activity. 31% of these deans were of the opinion that high viewership of scholarly works increased academic merit, and 52% thought peer review of materials published online would also add value. It seems that the consideration of social media and altmetrics in RPT practice would be facilitated by implementing clear-cut structures for evaluation, and including the well-accepted trait of peer-review in assessing value.

Importantly, Harley et al. (2010) found that engagement with the public is generally valued across disciplines and by institutions. There is recognition for faculty who facilitate public education or find other ways to give back to the public as a way to acknowledge taxpayer funding. However, attempts to become a public figure aren't without their risk. Traditionally, some departments view negatively those who attempt to popularize their research niche (“An interview with Aaron Barlow, editor of Academe, the magazine of the American Association of University Professors,” n.d.). And, some academics view high levels of public engagement as only appropriate for those who have already been granted tenure and are well known to academics in their field; faculty may garner criticism if their public persona is not balanced with significant research contributions (Harley et al., 2010). However, Aaron Barlow argues that any academic who has succeeded in having their work taken seriously by the public is likely to also be taken seriously in RPT (“An Interview with Aaron Barlow,” n.d.).

It has also been suggested that universities should shift to formally recognizing the translational value of academic research in the RPT process (Sanberg et al., 2014). In general, about half of faculty agree that the societal impact of one’s scholarly work should be a key RPT consideration (Wolff et al., 2016). Specifically, patents, licensing, and commercialization could be credited in order to encourage faculty to engage in use-oriented research that has the potential to positively affect society. Further, about 35% of faculty believe data should be credited equally to academic publications in RPT evaluations, and 37% believe software/code should be equally credited (Wolff et al., 2016). Sanberg et al. (2014) report that an apparent minority of US institutions have integrated these ideas into RPT policy, and changes in policy on this theme have likely been slow because they have been initiated primarily at the level of individual departments (bottom-up) rather than that of the institution (top-down).

Beyond RPT guidelines

To reform the RPT process, it might be logical to begin by examining how the process has been instituted in formal guidelines. However, it seems that RPT guidelines can be unclear (Smesny et al., 2007), or purposefully vague, to allow for flexibility in each applicant's situation. Although promotion and tenure committees usually do attempt to use objective measures, in reality, the procedures, criteria, and weights used can vary between applicants and between departments (Claxton, 2005; Walker et al., 2010). Macfarlane (2007) observed that institutions typically don't specify weights to convey which of their tenure and promotion criteria are the most important. May (2005) found that all promotion and tenure documents from several research-focused universities addressed research, teaching, and service, but the language of the policy tended to be very broad as to allow for interpretation. All documents that May reviewed had specific requirements with regard to publication of research findings, but the expectations for teaching were less clear and more variable, and the definitions of service requirements were the most vague.

Faculty rank and institution type may also affect the way one views the RPT process itself. Diamantes (2004) reported that tenured faculty perceived that the requirements were well communicated, but untenured faculty expressed a degree of uncertainty regarding the expectations. Estabrook & Warner’s (2003) study on Anthropology, English, and History departments, however, found no relationship between faculty age or tenure status and opinion on whether a book should be required for tenure. And, Gordon (2008) found that faculty at research and hybrid (research/teaching) universities report less confusion about publishing requirements than faculty at teaching universities. She provided different examples of faculty from research universities who had specific guidelines for publishing (e.g., six publications in six years) versus guidelines that were difficult to interpret, as one respondent wrote: “It is 45% of my responsibility allocation, but I’m not sure that tells the whole story. I think its more that I need to have quality or quantity of pubs. I’m not sure how they can translate that into a percentage” (p. 64). King et al. (2006) described the RPT process in chemical engineering at UC Berkeley as having vague and ambiguous written guidelines – even requirements for publication were not clearly stated.

Despite ambiguous guidelines, faculty in King et al.’s (2006) study reported a clear understanding of how to succeed in career advancement, indicating the value of informal communication within the department in supporting its members. In survey responses, faculty expressed the opinion that vague requirements are understandable because the RPT process is “unquantifiable,” and that “if I'm doing my job right, tenure should come along with it” (p. 39; King et al., 2006). Acker & Webber (2016) similarly reported that in Ontario, Canada, many candidates found the rules for tenure criteria lacking in clarity. And, Prottas et al. (2017) found that faculty in the northeastern USA experienced a lack of clarity, and perceived unfairness, in their tenure criteria and in their institutions’ decision making processes. In the UC Berkeley Anthropology Department, it was acknowledged that the process for career advancement can be unclear to junior faculty, therefore it is the responsibility of the department chair to explain tenure expectations to new hires (King et al., 2006).

Harley et al. (2010) also received reports of considerable flexibility in tenure and promotion judgement at research universities. Excellent quality in research and publication was most important and could override unwritten rules about the numbers of journal articles, books, or citations required. Special forms of scholarly evidence, such as the products of interdisciplinary research, creative pursuits, and many practices more common in the arts, can require special attention by reviewers. Harley et al. noted that RPT policy had built-in mechanisms to credit these types of activities as appropriate, and that each RPT application receives a great deal of attention in its adjudication. May (2005) concluded that the paucity of particular weights or values for any particular aspect of tenure or promotion applications leads to decisions being made by individuals and committees using their own “weighted judgement for each given criteria,” or by viewing all evidence together to make a prediction about the applicant's potential for making ongoing and substantial scholarly contributions.

Estabrook & Warner (2003) also found evidence of tenure and promotion committees deviating from policy in making career advancement decisions in the somewhat variable disciplines of Anthropology, English, and History. Here, it is generally expected that faculty members will have published a scholarly book or monograph prior to making a tenure application. However, most official promotion and tenure guidelines indicate that either a book or a considerable number of substantial and peer reviewed publications may be accepted. When Estabrook and Warner interviewed 17 department chairs in these disciplines, the chairs consistently acknowledged the option given in the RPT guidelines, but stated that most faculty (with the exception of those in a few specific subfields) needed to publish a book to receive tenure.

One way to describe the relationship between RPT policy and the way the RPT process is actually carried out is to acknowledge the difference between institutional policy and departmental expectations. Institutional policy tends to be broad with many potential criteria for faculty to meet in order to earn career advancement. Departments may pick and choose from the institutional framework which criteria are their particular deal-breakers, and which items can be overlooked in favor of other candidate qualities and contributions. This, of course, can also lead to differences in the RPT process not only between institutions, but between departments within the same institution, as reported by Andersen & Trinkle (2001).

Just as departments within an institution can vary in their RPT practices, so can departments of the same discipline across different institutions. For instance, Liner & Sewell (2009) surveyed 125 economics department chairs regarding their consideration of applications for both tenure and advancement to full professor, and found variability between them in the degree of reduction of credit for paper co-authorship. Reports from faculty in the field of English-language literature echo this theme, with one faculty member in King et al.’s (2006) study stating that the norms for advancement in the field “vary wildly” (p. 23). Although the general opinion was that across institutions the differences were substantive, faculty were clear on what was required within their own institution. Overall, it seems that policies provide a framework, but that RPT decisions are made on a case-by-case basis with considerable allowances made for differences from the norm.

Conclusions: The future of RPT

Expectations and practices for review, promotion and tenure have shifted significantly over the last few decades. Although there are differences across institutions, disciplines, and faculty ranks, it is clear that faculty in many contexts are feeling increasing pressure to focus on research at the expense of teaching and service (Otten et al., 2015; van Dalen & Henkens, 2012). In recent years there has been an effort to help the pendulum swing back the other way by allowing for consideration of more varied measures of performance (e.g., altmetrics or non-traditional publishing mediums), but these efforts have not been entirely successful in offsetting oversimplified approaches such as points schemes based on journal impact factors. As a result, those faculty who wish to value activities beyond traditional research publications in so-called high-prestige venues may face barriers to career advancement.

Although there are frustrations with RPT practices, this doesn’t mean the RPT process is fixed as it is today. The noticeable shift towards greater emphasis on research and particular types of publications, along with the documented efforts to counteract those trends, are signs that RPT practices do not go uncontested. Part of this challenge to the current status-quo was the San Francisco Declaration on Research Assessment (DORA), drafted at the Annual Meeting of The American Society for Cell Biology in December 2012 (Cagan, 2013) and since signed by over 450 organizations and almost 12,000 individuals (DORA, n.d.). The declaration makes several recommendations that are directly aimed at pushing back on some of the trends in researcher assessment highlighted in this review. In particular, they recommend, among other things, that researchers, and those involved in assessing research: 1) “Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion or funding decisions;” 2) “Be explicit about the criteria used to reach hiring, tenure and promotion decisions, clearly highlighting, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published;” and 3) “When involved in committees making decisions about funding, hiring, tenure, or promotion, make assessments based on scientific content rather than on publication metrics.” The global effect of these recommendations on changing the current RPT practices, however, remains largely unknown.

DORA has inspired much updating of policy and shifting of opinions away from the use of journal impact factors, but there is still a great need for action to elicit change in the actual procedures used in RPT evaluations. In working with DORA, Curry (2018) has observed numerous instances of RPT procedures that are maintaining the dominance of the impact factor in determining the value of research. The next step will focus on moving beyond declarations and focusing on finding ways for institutions and funding agencies to change their evaluation practices in the spirit of the declaration (Curry, 2018).

Although DORA is promoting change in procedures for evaluation of academic research contributions, the issue of imbalance within the academic “trifecta” of research, teaching, and service remains. Faculty seem accepting of the idea that research may count more towards RPT than the other two elements, but failure to reward teaching and service devalues faculty work in these areas. It may be time to evaluate whether our institutions of higher education and mechanisms of scholarly communication can reflect Boyer's (1996) scholarship of engagement, in which scientific discovery (research) is a crucial function of the university, but so are functions deriving from teaching and service, such as the sharing of information across disciplines, the sharing of knowledge with students and the public, and the application of information to real world problems.

Because RPT criteria strongly influence where faculty will place their focus, RPT reform may be one of the most successful ways to effect change in the academic system. We believe there are two natural best next steps to devising an updated system for evaluating scientific merit: 1) to deepen our understanding of faculty and administrative perceptions of the current reward system and desires moving forward (see also Desrochers et al., 2018); and 2) to assess the relationship between the content of current RPT documents and their actual operationalization into existing practices. Together with the foundation of information presented in this review, progress in these directions will provide insight into how RPT should be reformed, and whether there may be additional targets for change within the academic system.

Data availability

No data is associated with this article.