Too often in economics, where you post can be more important than what you post.
That’s the theory explored in a new study co-authored by Nobel Prize-winning economist James J. Heckman and Sidharth Moktan, a predoctoral fellow at the Center for Human Development Economics. The University of Chicago researchers found that tenure and award committees often base their decisions on how often applicants publish in the “top five” journals in the field. This practice not only concentrates career advancement in the hands of a select group of editors, many of whom are long-serving, but does so at the expense of innovative economic research.
“Relying on publication counts in ‘top-ranked’ journals encourages crude careerism in young economists,” said Heckman, Henry Schultz Emeritus Service Professor of Economics and the College. “It shifts their focus from basic research to blatantly strategizing about the research focus and favorite topics of long-time journal editors.
“Relying on rankings rather than reading to promote and reward young economists subverts the essential process of evaluating and rewarding innovative original research.”
The “top 5” refers to the main economic journals most crucial for the academic and professional success of young university students: The American economic journal, Econometricsthe Journal of Political Economy, the Economics Quarterly Review and the Review of economic studies. These journals are chosen by a process that weights the number of citations of all articles in the journal; in other words, it judges an item by the company it holds.
Heckman and Moktan’s working paper, released Oct. 1 by the National Bureau of Economic Research, found that academics who wrote a “top five” paper are 90% more likely to be tenured during a given year. Those numbers jump to 260% for two of these articles and 370% for three.
“Junior and senior researchers often bring up the top five when evaluating someone,” Moktan said. “Even if it’s in casual conversation, they’ll be like, ‘Oh, how many ‘top five’ do they have?'”
Assess the role of bias
According to Heckman and Moktan, writing for a top five journal involves more than producing the best research possible. Their study argues that to maximize the chances of placement, researchers are encouraged to tailor their work to individual publishers, who, consciously or not, are guided by their own biases.
In addition to tracking seniority rates, Heckman and Moktan’s study tracks authors’ affiliation in the “top five” journals from 2000 to 2016. For example, Heckman sits on the editorial board of the Journal of Political Economy, published by University of Chicago Press. Perhaps unsurprisingly, he pulled 14.3% of his papers during the aforementioned period from authors linked to the University of Chicago. the Economics Quarterly Reviewwhich is edited in the Department of Economics at Harvard University, drew almost a quarter of its articles (24.7%) from its own affiliates, plus 13.9% of its articles from affiliates of the Massachusetts Institute of Technology.
In contrast, the Review of economic studies, which has a higher turnover rate on its editorial board, has much weaker ties to individual universities. From 2000 to 2016, the publication was most heavily tied to New York University and Northwestern University affiliates, from which it drew 7.3% and 7.0% of articles.
The scope of the problem widens when universities use the “top five” journals as a proxy for determining tenure. It is no longer simply a question of who is published in certain journals, say the authors; instead, flaws in the editorial process are amplified into career hurdles, which can be difficult for outsiders to overcome without ties to the “top five” editors and the referees they select.
The “top five” do not have a monopoly on quality work either. According to Heckman and Moktan, some of the most influential work in economics is published by other journals. Although the “top five” papers produce more citations on average, these numbers are skewed by outliers. Moreover, senior scholars who rely on the “top five” to judge their colleagues often do not publish themselves in these journals once they are tenured. Relying on journals, Heckman said, encourages caution rather than creativity in young researchers seeking employment.
The “top five” journals encourage researchers to focus on follow-up and replication of work, research that is easy to assess for immediate publication, but does not push the boundaries of economics. Often, the kinds of innovative projects that would challenge accepted ideas are too long or too data-intensive to fit into the “top five” review format.
“Research is inherently risky because you’re trying to find answers to questions that haven’t been answered,” Moktan said. “Sometimes the answers aren’t exciting. But serious evaluations require seasoned researchers to read articles and understand them, and why this line of research is important.
Heckman and Moktan suggest that tenure committees devote more resources to careful reading of published and unpublished articles, rather than relying on the reputation of journals as a substitute for careful reading. This method could incentivize each individual institution to pursue more unconventional research instead of relying on papers channeled through the “top five” publications. Expanding the pool of influential publications beyond five journals could also alleviate the problem.
They also suggest a more radical solution: move away from conventional journals in favor of open source formats such as arXIV and PLOS ONE, which are used in the hard sciences. Such a change would offer researchers the opportunity to share their ideas sooner and get real-time peer review, an approach that could be more welcoming to off-the-shelf ideas.
“The current posting and reward system doesn’t encourage creativity,” Heckman said. “It delays the publication and dissemination of new ideas. It centralizes power in the hands of a small group of editors, prevents open discussion, and stifles dissent and debate. It needs to be changed. »