Retrospectives from 20 Years of JMLR
Fabian Pedregosa, Tegan Maharaj, Alp Kucukelbir, Rajarshi Das, Valentina Borghesani, Francis Bach, David Blei, Bernhard Schölkopf
21 February 2022
In 2000, led by editor-in-chief Leslie Kaelbling, JMLR was founded as a fully free and open-access platform for publishing high-quality machine learning research. Twenty-one years later, JMLR publishes more than 250 papers per year and is one of the premiere publishing venues in the field of artificial intelligence. How did a community-driven journal, without any financial or managerial support from traditional publishing companies, become a leading journal in the field? Celebrating more than 20 years of history, we take a look back at the story of JMLR and the lessons that can be learnt from it.
Outline:
1. How JMLR works
2. Papers, decisions and publication time
3. The Human Cost of Sustaining a Growing Field
4. Mirroring trends and biases of the
field
5. Outlook
6. Credits
1. How JMLR works
In summary. JMLR runs almost entirely on volunteer labor.[1] The JMLR team consists of three Editors-in-Chief (EiCs), two Managing Editors, an Editorial Assistant, a Production Editor, and a Webmaster, along with the Advisory Board and a large group of currently 133 of Action Editors (AEs). The AEs are senior researchers in the field (typically tenured or equivalent), recruited by invitation, and it's by relying on their expertise that JMLR can have such a small and agile management team. Expert AEs decentralize much of the editorial work that is typically centralized in EiCs in other journals – EiCs assign each paper to an AE, and from there on the AE takes responsibility for finding reviewers and making the final decision.
In detail. Authors upload submission to JMLR's own submission system, hosted at MIT (see section 3 for a precise cost estimate). This system was initially written by Christian Sheltonin 2003. It has served us remarkably well, as the system continues to be in use today with minor improvements by subsequent managing editors. It is written in Perl and Python, and uses a PostgreSQL database.
Each manuscript is scanned by an Editor-in-Chief. If the Editor-in-Chief finds the paper out of scope, they will desk-reject the paper, otherwise they will assign the paper to an action editor (AE) who has expertise in the area of the paper. The AE decides whether to send the paper for a full review.
If the paper goes out for review, the AE solicits 2-3 expert reviewers to write detailed technical reviews of the paper. After review, the AE can then accept the paper as is, reject the paper, ask for minor revisions, or propose "reject with encouragement to resubmit", i.e. ask for major revisions.
If the paper is accepted, the production editor prepares the camera-ready version that is then uploaded to the website. Papers are accepted on a rolling basis and uploaded as they are processed, grouped into one Volume per year. The managing editors support all users of the submission system (authors, reviewers, and the editorial board) throughout the reviewing process; the webmaster takes care of uploading the final documents and is responsible for the maintenance of the public webpage.
Besides the main publication track, JMLR also has a track for open source software contributions called Machine Learning Open Source Software (MLOSS). These papers have dedicated action editors and a page limit of 4 pages, while main track JMLR papers don't have page limit. Reviewers of MLOSS papers are also asked to follow a different set of reviewing criteria. JMLR has also hosted special issues for topics of timely relevance. In recent years, JMLR has also become an umbrella for different publications that we won't cover in this blog post, such as the Proceedings of Machine Learning Research (PMLR) and the recently-announced Transactions on Machine Learning Research (TMLR), which will start accepting submissions in March 2022.
2. Papers, decisions and publication time
The number of papers submitted and accepted has been steadily increasing throughout the years: from 10 papers published in 2002 to 290 papers published in 2021
Below we show the number of unique submissions (that is, excluding resubmissions) since 2003.[2]Color indicates the final decision of the paper:
- desk reject, for papers rejected by the action editor or editor-in-chief without sending the paper for review.
- reject, for papers whose final decision after review is either reject or reject with encouragement to resubmit (but whose authors decided to not resubmit), and
- accept, for papers that will be published on the website.
Figure 1. Number of submissions across the years, color coded according to the associated editorial decision.
This plot shows the dramatic growth experienced in recent years. While it took 8 years for the number of submissions to double in size from 2004 to 2012, it only took 2 years from 2018 to 2020 for these to double again.
Acceptance rate. JMLR's only criteria for acceptance is quality, i.e., it does not enforce an annual acceptance rate. The action editor is responsible for deciding whether the paper is up to the standard of the journal. The following figure shows the yearly evolution of the different editorial decisions (desk reject, reject, accept).
Figure 2: For each year, we show the percentage of submissions that received an editorial decision of desk reject vs. reject vs. accept.
The percentage of accepted papers has been steadily declining since the 33% acceptance rate of 2007 to the current 17% acceptance. The trend can be explained by a constant strive for quality in face of an increasing number of submissions.
Time to receive the first round of reviews. Papers that are sent to reviewers can take a variable amount of time to come back to authors. Below we plot the median, 25-75 and 10-90 percentile for the number of days it took for authors to receive the first round of reviews (hence desk rejected papers are excluded).
Figure 3. Days from submission to decision for papers that are sent to reviewers. The dark line shows median, while lighter intervals represent 25-75 and 10-10 percentile respectively.
This median delay has unfortunately been steadily increasing throughout the years, reaching 187 days in 2021. It's a priority for us to reduce the delay times without sacrificing the quality of the review process. The section below explores the cost of such an endeavor.
3. The Human Cost of Sustaining a Growing Field
The storage and bandwidth needs of the journal, although increasing throughout the years, remain negligible. The full jmlr.org website, including non-public under-review papers, backups, and a PostgreSQL database, occupies 49GB. Currently, MIT provides this storage for free, but it would have a cost of around $100 per month using a standard Cloud Platform.
The most precious resource is the human workforce. To ensure that published papers are technically sound and of the highest quality, JMLR relies on a group of experts all volunteers. In 2021, JMLR counted 1938 active reviewers (93% of the workforce),[3] 133 action editors (6%), and 8 members of the editorial board (0.3%), which includes the Editors-in-Chief, and technical time-consuming roles such as webmaster, production editor and managing editor.
To handle the increasing workload, the number of action editors and reviewers has been steadily increasing through the years. The figure below shows the number of Submissions, Action editors and Reviewers.
Figure 4: Number of action editors and reviewers throughout the years.
These figures highlight the increased labor that the action editors have taken in recent years: in 2010 the average number of submissions per action editor was 4.5, while in 2020 it was more than double, 11.7.
4. Mirroring trends and biases of the field
According to recent estimates,[4] only 12% of leading machine learning researchers are female and only 22% of jobs in artificial intelligence are held by them, with even fewer holding senior roles.
We sought to understand the representation of female scientists in the journal. Unfortunately, JMLR does not track gender information, so we appealed to the indirect means of inferring gender from first names, which only provides a rough estimate. We inferred gender for both action editors and corresponding authors. Below we plot the gender proportion of female AEs and authors. We only plot this data since 2012, which is when the number of AEs surpassed 100 members:
Figure 5: Percentage of female action editors and corresponding authors.
The years 2013-2016 are characterized by an exceptionally low number of female AEs, below 10%. This number has been increasingly steady throughout the last few years, reaching 17% in 2021. Although this is an improvement on previous years, we're far from the gender balance that we strive to achieve. This is an aspect we're committed to improving.
5. Outlook
When it was founded in 2002, the first editors of JMLR sought to create an independent and open-access journal, with the most minimal operating costs. This was a radical and visionary experiment, and it was a success. Today, JMLR is a top journal in AI and ML, while remaining free, open, and community-driven.
Since 2002, the fields of AI and ML have grown and thrived, and JMLR has grown along with them. Of course, the increase in submissions means increased demands on all the volunteers’ time and energy. JMLR is indebted to the immense efforts of the leadership team, the action editors, and the many reviewers. Thank you for making JMLR such a great journal and keeping JMLR free.
6. Credits
Citing. Please consider citing this article as
Fabian Pedregosa, Tegan Maharaj, Alp Kucukelbir, Rajarshi Das, Valentina Borghesani, Francis Bach, David Blei, Bernhard Schölkopf, "Retrospectives from 20 Years of JMLR", https://www.jmlr.org/news/20_years.html
BibTex entry:
@misc{pedregosa2022retrospectives,
title={Retrospectives from 20 Years of JMLR},
author={Pedregosa, Fabian and Maharaj, Tegan and Kucukelbir, Alp and Das, Rajarshi and
Borghesani, Valentina and Bach, Francis and Blei,David and Sch{\"o}lkopf, Bernhard},
url={https://jmlr.org/news/2022/retrospectives.html},
year={2022}
}
Acknowledgements. We would like to thank Leslie Pack Kaelbling, Lawrence Saul and Barbara Engelhardt for providing feedback on this blog post.
[1] The only paid job at JMLR is that of the part-time editorial assistant, who provides email support to users of the journal and ensures the manuscripts flow smoothly through the submission and review system.
[2] Although the journal was established in the year 2000, the current submission system was created in 2003 thus prior statistics are not available.
[3] We count active reviewers as those that have performed at least one review during the year 2021.
[4] Bridging The Gender Gap In AI, Forbes magazine.
| © JMLR 2024. |