Scaling the Journal of Open Source Software (JOSS)

Daniel S. Katz, Lorena A. Barba, Kyle Niemeyer, Arfon M. Smith

The Journal of Open Source Software (JOSS) started with one editor-in-chief (EiC) and 10 topic editors, and in our first year, we published about 100 papers. Three years later, the rate of publication has increased to 300 papers per year. How did we scale up to this point, and how can we continue scaling the journal?

Scaling to-date

Today, we have one editor-in-chief (EiC), three associate editors-in-chief (AEiCs), and 23 topic editors. We recognize past service of editors that step down, for any reason, as emeritus editors on our website (currently, six). The journal is currently receiving about one new submission per day, a 3.5x increase in 2 years.

What scales and what doesn’t – or – what we did to scale to where we are now:

Assigning papers to editors: when a paper is submitted, it enters the “incoming” queue, from which an AEiC moves it to a Pre-Review issue and assigns it to a handling editor. We scaled this task, initially handled by the EiC, to the EiC and 3 AEiCs who rotate on-duty responsibilities weekly. The overall per-person workload for this task is thus manageable.
Assigning papers to reviewers: after a Pre-Review issue starts, the assigned handling editor has the task of finding and assigning reviewers to the paper. This task can scale up by 1) recruiting additional editors so that each editor has a manageable workload, as long as 2) there are sufficient appropriate reviewers available and 3) the editor can find them. JOSS currently assigns a minimum of two reviewers/paper. As we receive more submissions each year, more reviewers are needed. Although we can ask authors of accepted papers to review new submissions, with many more active papers simultaneously, we will need better tracking of reviews contributed, to avoid overburdening some reviewers, and we may need a better way to find appropriate reviewers than the keyword searches of existing volunteer reviewers and the personal knowledge and search skills of the editors that we currently use.
Overseeing active reviews: editors and AEiCs maintain a close eye on active reviews to keep things moving along, moderate conversations, and clarify the appropriate process when needed. This currently scales well for handling editors, who have a dashboard showing the state of their assigned papers with key activity stats. The AEiC on rotation is aided by the “In Progress Papers” dashboard, with a manageable workload to keep reviews from becoming stuck; this scales acceptably at this point with AEiC weekly rotations.
Accepting and publishing papers: the one AEiC on duty each week completes the final tasks needed after reviewers and handling editor recommend acceptance. Our current rotation system avoids excessive long-term workload on any one person, but may not scale to a much larger number of papers accepted each week, as part of the AEiC’s job is final proofreading and detail checking, which can be time consuming.
Editorial robot: Much of the editorial process is assisted by our editorial bot ‘Whedon’ which carries out a number of automated processes on behalf of editors (see this post for more details). These automated steps include preparing the final version of a publication, checking references for missing or malformed DOIs, adding the software release version and archival DOI to the metadata, building an XML file for Crossref metadata, and executing the Crossref DOI registration. All of these steps are possible with simple commands issued to Whedon in the review thread on GitHub. For example, typing the command @whedon accept in a comment on the Review issue will automatically compile the paper PDF from the author’s source in markdown, check the references, and create the Crossref XML, providing links to these files for the AEiC’s final visual check.
Overall process overview: We built a JOSS editor dashboard to 1) help the AEiCs balance editor loads, review unassigned submissions, and understand the status of all in-progress reviews. The dashboard also 2) helps editors keep track of all current submissions (in addition to an automated weekly email reminding editors of current assignments).

Future scaling

If the number of submissions increased by another factor of 3 or even 10, we would need to make additional changes to handle the increase. Some of these could be:

Increase the number of AEiCs

If we do this, we might also need to change the schedule we now have. Our idea of rotating a single AEiC on duty at any one time likely wouldn’t still work for assignment of papers to editors; we might need multiple AEiCs at a time with that assignment of papers to AEiCs done in either a round-robin or random manner.

A decreased rotation time might work for accepting papers and publishing them, but the round-robin and random assignment would also work.

Alternatively, each paper that comes in might be assigned to an AEiC for all work on the paper, from assigning an editor through to acceptance and publication, though this would remove the workload pauses that AEiCs currently have during their off-duty periods.

Further increase number of editors

This should scale reasonably well.

Developing additional tooling for handling the editorial process

As both the number of AEiCs and editors increase, additional tooling likely will be needed, including:

  • Semi-automated tools for authors to suggest reviewers from a pool of volunteers and previous JOSS authors, and for editors to choose reviewers from these same sources as well as from the community outside JOSS
  • Semi-automated tools for assignment of papers to editors
  • Semi-automated checking of status for both editors and AEiCs
  • Reliable statistics on reviewer contributions to avoid reviewer fatigue

Better support

We could off-load some of the AEiC work (e.g., helping authors who have problems with formatting papers or other issues) either to a paid staff member, or more likely, we could set up support mechanisms for authors to be helped by other authors or editors, perhaps by Stack Overflow or something similar. Given that we are working in an open source environment, this would be a way of getting some members of the community to take on a little more, which might later lead to some becoming editors.

An alternative to scaling JOSS itself would be to scale up by starting more JOSS-like journals, even using the same infrastructure, perhaps by discipline, by programming language, etc. (Note to commercial publishers: We would still make these open access and free to publish; we don’t mean that you should start JOSS-like journals and charge APCs.)

References

Smith AM, Niemeyer KE, Katz DS, et al. Journal of Open Source Software (JOSS): design and first-year review. PeerJ Computer Science. 2018;4:e147. doi:10.7717/peerj-cs.147

Smith A. Chatops-Driven Publishing Arfon Smith. Published online February 28, 2019. Accessed July 8, 2019. https://www.arfon.org/chatops-driven-publishing