This week JOSS reached a big milestone – publishing our 2000th paper! It also happens to be our 7th birthday, and we thought we’d take this opportunity to review our submission stats from the last few years, discuss some of the changes to JOSS we’ve made of late, and reflect on some of the challenges we have faced as a journal.

Submission summary

Everything discussed here is derived from the amazing work of one of our editors (thanks Charlotte!) who created our submission analytics page which is built nightly based on data from the JOSS API. If you want to dig more into this analysis, the source code is available for you to do so.

High level publication stats

2000 papers over 7 years means, on average, we’ve published 285 papers per year. An average isn’t quite right though of course, with our throughput being substantially lower in the early days.

Year 1 (May 2016 – May 2017): 57
Year 2 (May 2017 – May 2018): 138
Year 3 (May 2018 – May 2019): 254
Year 4 (May 2019 – May 2020): 345
Year 5 (May 2020 – May 2021): 338
Year 6 (May 2021 – May 2022): 369
Year 7 (May 2022 – May 2023): 366

Note that JOSS closed for submissions between March 2020 and May 2020 due to the COVID-19 pandemic. This likely accounts for the drop in publications in year 5 (May 2020 – May 2021). Looking at the high-level breakdown, it seems like we’ve reached some kind of plateau of around one paper published per day. If we were a business looking to grow revenue this lack of year over year growth might be of concern. However, as a volunteer-run journal, this is OK with most of us :-)

Submission scope and rejections

In July 2020 we introduced a test for ‘substantial scholarly effort’ for all new submissions. You can read more about the motivations for this in our blog post but this clearly had an effect on our rejection rate both in the pre-review stage, and during/post review.

We now reject between 15-30% papers at the pre-review stage, and around 5% during the actual review process itself (note this is in line with our goal to provide a constructive review process, improving submissions such that they can be accepted).

Authorship

JOSS is no longer dominated by single author submissions. In fact, we are seeing some evidence of more authors per submission with the fraction of submissions with more than 5 authors now approaching 25%.

Citations

More than 200 papers have been cited > 20 times. About a third have never been cited. A few papers have been cited a lot: Welcome to the Tidyverse currently has nearly 6000 citations.

Papers in more than 6000 different venues have cited a JOSS paper, with the most common being bioRxiv, JOSS, Scientific Reports, Monthly Notices of the Royal Astronomical Society, and The Astrophysical Journal.

Editor and Reviewer statistics

2093 unique individuals have contributed reviews for the 2000 papers published in JOSS, including 8 amazing individuals who have contributed more than 10 reviews each!

JOSS currently has 77 editors, 6 of whom are track Editors-in-Chief, and one Editor-inChief. 112 editors have served in total on the editorial board.

Unfortunately, our reviews are getting slower. We’re not really sure why, but this has been a noticeable change from our earlier days. Our average time now spent in review is approaching four months, whereas pre-COVID it was under three.

Software statistics

JOSS reviews are primarily about the software, and so it would be remiss of us not to talk about that. Python is still the #1 language for JOSS submissions, used in part for well over half of published papers (~1200 out of 2000). R is #2 at 445 submissions, and C++ #3 (although of course C++ and C may be used together with another language).

Anecdotally we’re seeing an increase in the number of authors submitting to JOSS to declare their submission ‘ready for use’ or declaring as a v1.0 release. This is also potentially supported by the peak close to 0 days between the repository creation date and the submission to JOSS.

The MIT, GPL (v3), and BSD 3-Clause licences are still used for the majority of submissions.

Investments in tooling and infrastructure

JOSS has benefited from the kind support of the Alfred P. Sloan Foundation, the Gordon and Betty Moore Foundation, and a number of small development grants from NumFOCUS. This support has enabled JOSS to invest further in our infrastructure and tooling, highlights of which we describe below:

Editorialbot for all of your GitHub-based review needs

Our former bot Whedon has now become Editorialbot, which is much more than a rename. Editorialbot is now its own open source framework that can be used to manage review-like interactions on GitHub (i.e., not just publishing workflows). Currently, Editorialbot is used by rOpenSci, JOSS, SciPy Proceedings, with more coming soon. Thanks to Juanjo Bazán for all of his great work here!

As part of this work we’ve also extracted all of the legacy capabilities from the Whedon codebase to run as a series of GitHub Actions workflows.

Investments in our Pandoc-based publishing pipeline

JOSS has always used Pandoc to produce our PDF papers and Crossref XML metadata, but the way we used it was… hacky. Over the past few years we’ve been fortunate to work directly with one of the Pandoc core team (Albert Krewinkel) to implement a number of improvements to how we use Pandoc within the JOSS publishing system, and also to contribute new capabilities to Pandoc itself, including improved ConTeXt support, support for JATS outputs, and more sophisticated handling of author names in the Pandoc frontmatter.

Editorial tooling improvements

Last but not least, we’ve also made a bunch of small (and not so small) changes to the way that we handle submissions as an editorial team, including implementing tracks (and appointing a track EiC for each) and a reviewer management tool for searching for appropriate reviewers and tracking reviewer assignments.

Thank you!

Over these seven years of operations, JOSS simply wouldn’t work without the dedicated volunteer editors and reviewers (and of course the authors submitting their work).

References

Höllig J, Kulbach C, Thoma S. TSInterpret: A Python Package for the Interpretability of Time Series Classification. Journal of Open Source Software. 2023;8(85):5220. doi:10.21105/joss.05220

Smith AM. Minimum publishable unit. Published online July 7, 2020. doi:10.59349/dfy0f-3y061

Wickham H, Averick M, Bryan J, et al. Welcome to the Tidyverse. Journal of Open Source Software. 2019;4(43):1686. doi:10.21105/joss.01686

Once again, we’re looking to grow our editorial team at JOSS!

Since our launch in May 2016, our existing editorial team has handled over 2000 submissions (1838 published at the time of writing, 215 under review) and the demand from the community continues to be strong. JOSS now consistently publishes a little over one paper per day, and we see no sign of this demand dropping.

New editors at JOSS are asked to make a minimum 1-year commitment, with additional years possible by mutual consent. As some of our existing editorial team are reaching the end of their term with JOSS, the time is right to bring on another cohort of editors.

Background on JOSS

If you think you might be interested, take a look at our editorial guide, which describes the editorial workflow at JOSS, and also some of the reviews for recently accepted papers. Between these two, you should be able to get a good overview of what editing for JOSS looks like.

Further background about JOSS can be found in our PeerJ CS paper, which summarizes our first year, and our Editor-in-Chief’s original blog post, which announced the journal and describes some of our core motivations for starting the journal.

More recently we’ve also written in detail about the costs related with running JOSS, scaling our editorial processes, and talked about the collaborative peer review that JOSS promotes.

How to apply

Firstly, we especially welcome applications from prospective editors who will contribute to the diversity (ethnic, gender, disciplinary, and geographical) of our board.

✨✨✨ If you’re interested in applying please fill in this short form by 22nd December 2022. ✨✨✨

Who can apply

We welcome applications from potential editors with significant experience in one or more of the following areas: open source software, open science, software engineering, peer-review.

The JOSS editorial team has a diverse background and there is no requirement for JOSS editors to be working in academia. Unfortunately individuals enrolled in a PhD program are not eligible to serve on the JOSS editorial team.

Selection process

The JOSS editorial team will review your applications and make their recommendations. Highly-ranked candidates will then have a short (~30 minute) phone call/video conference interview with the editor(s)-in-chief. Successful candidates will then join the JOSS editorial team for a probational period of 3 months before becoming a full member of the editorial team. You will get an onboarding “buddy” from the experienced editors to help you out during that time.

References

Smith AM, Niemeyer KE, Katz DS, et al. Journal of Open Source Software (JOSS): design and first-year review. PeerJ Computer Science. 2018;4:e147. doi:10.7717/peerj-cs.147

Katz DS, Barba LA, Niemeyer K, Smith AM. Cost models for running an online open journal. Published online June 4, 2019. doi:10.59349/g4fz2-1cr36

Katz DS, Barba LA, Niemeyer K, Smith AM. Scaling the Journal of Open Source Software (JOSS). Published online July 8, 2019. doi:10.59349/gsrcb-qsd74

JOSS is continuing to grow, and we are looking to add more editors with expertise in the area of astronomy & astrophysics.

Since our launch in May 2016, our existing editorial team has handled nearly 1900 submissions (1684 published at the time of writing, 205 under review) and the demand from the community continues to grow. In particular, we have seen an increase in the number of astronomy & astrophysics submissions, beyond the capacity of our current editorial team.

Editors at JOSS make a minimum 1-year commitment, with additional years possible by mutual consent. With some of our existing editorial team reaching the end of their terms with JOSS and this increase in submissions, the time is right to bring new editors on board.

Background on JOSS

If you think you might be interested, take a look at our editorial guide, which describes the editorial workflow at JOSS, and also some of the reviews for recently accepted papers. Between these two, you should be able to get a good overview of what editing for JOSS looks like.

Further background about JOSS can be found in our PeerJ CS paper, which summarizes our first year, and our Editor-in-Chief’s original blog post, which announced the journal and describes some of our core motivations for starting the journal.

More recently we’ve also written in detail about our commitment to the Principles of Open Scholarly Infrastructure the costs related with running JOSS, scaling our editorial processes, and talked about the collaborative peer review that JOSS promotes.

Of specific interest to this call, we also have a collaboration with the American Astronomical Society (AAS) Journals to provide a parallel review for submissions with a significant software component.

Who can apply

We welcome applications from potential editors with research experience in astronomy and astrophysics, including but not limited to, open source software development for astrophysical simulations, data reduction, or statistical methods.

Members of the JOSS editorial team have diverse backgrounds, and we welcome JOSS editors from academia, government, and industry. We especially welcome applications from prospective editors who will contribute to the diversity (ethnic, gender, disciplinary, and geographical) of our board. We also value having a range of junior and senior editors.

How to apply

✨✨✨ To apply please fill in this short form by 31 July 2022. ✨✨✨

Selection process

The JOSS editorial team will review your applications and make recommendations. Highly-ranked candidates will then have a short (~30 minute) phone call/video conference interview with a current editor. Successful candidates will then join the JOSS editorial team for a probationary period of 3 months before becoming a full member of the editorial team. You will get an onboarding “buddy” from the experienced editors to help you out during that time.

References

Smith A. Announcing The Journal of Open Source Software - Arfon Smith. Published online May 5, 2016. Accessed July 12, 2022. https://www.arfon.org/announcing-the-journal-of-open-source-software

Smith AM, Niemeyer KE, Katz DS, et al. Journal of Open Source Software (JOSS): design and first-year review. PeerJ Computer Science. 2018;4:e147. doi:10.7717/peerj-cs.147

Katz DS, Smith AM, Niemeyer K, Huff K, Barba LA. JOSS’s Commitment to the Principles of Open Scholarly Infrastructure. Published online February 14, 2021. doi:10.59349/m5h23-pjs71

Katz DS, Barba LA, Niemeyer K, Smith AM. Cost models for running an online open journal. Published online June 4, 2019. doi:10.59349/g4fz2-1cr36

Katz DS, Barba LA, Niemeyer K, Smith AM. Scaling the Journal of Open Source Software (JOSS). Published online July 8, 2019. doi:10.59349/gsrcb-qsd74

Smith AM. A new collaboration with AAS publishing. Published online December 19, 2018. doi:10.59349/wj1gg-tsg49

JOSS is continuing to grow, and we are looking to add more editors again. We’re especially interested in recruiting editors with expertise in bioinformatics, neuroinformatics/neuroimaging, material science, ecology, machine learning & data science, and the social sciences.

Since our launch in May 2016, our existing editorial team has handled over 1800 submissions (1200 published at the time of writing, 170 under review) and the demand from the community continues to grow. The last three months have been our busiest yet, with JOSS publishing more than one paper per day, and we see no sign of this demand dropping.

Editors at JOSS make a minimum 1-year commitment, with additional years possible by mutual consent. With some of our existing editorial team reaching the end of their terms with JOSS and this increase in submissions, the time is right to bring on another cohort of editors.

Editing for a journal during a pandemic

After a pause on submissions in early 2020, JOSS has been open for submissions during most of the pandemic. We recognize that making time for volunteer commitments such as JOSS is especially challenging at this time and are taking steps to reduce the load on authors, editors, and reviewers and continually striving to find the right balance between accommodating the very real challenges many of us now face in our daily lives, and providing a service to the research software community.

Editing for JOSS is a regular task, but not one that takes a huge amount of time. JOSS editors are most effective if they are able to check in on their submissions a couple of times per week. Our goal is that a JOSS editor handles about three submissions at any one time making for about 25 submissions per year.

Background on JOSS

If you think you might be interested, take a look at our editorial guide, which describes the editorial workflow at JOSS, and also some of the reviews for recently accepted papers. Between these two, you should be able to get a good overview of what editing for JOSS looks like.

Further background about JOSS can be found in our PeerJ CS paper, which summarizes our first year, and our Editor-in-Chief’s original blog post, which announced the journal and describes some of our core motivations for starting the journal.

More recently we’ve also written in detail about our commitment to the Principles of Open Scholarly Infrastructure the costs related with running JOSS, scaling our editorial processes, and talked about the collaborative peer review that JOSS promotes.

Who can apply

We welcome applications from potential editors with significant experience in one or more of the following areas: open source software, open science, software engineering, peer-review, noting again that editors with expertise in bioinformatics, neuroinformatics/neuroimaging, material science, ecology, machine learning & data science, and the social sciences are most needed.

Members of the JOSS editorial team have diverse backgrounds and we welcome JOSS editors from academia, government, and industry. We especially welcome applications from prospective editors who will contribute to the diversity (ethnic, gender, disciplinary, and geographical) of our board. We also value having a range of junior and senior editors.

How to apply

✨✨✨ To apply please fill in this short form by 23 April 2021. ✨✨✨

Selection process

The JOSS editorial team will review your applications and make recommendations. Highly-ranked candidates will then have a short (~30 minute) phone call/video conference interview with the editor(s)-in-chief. Successful candidates will then join the JOSS editorial team for a probationary period of 3 months before becoming a full member of the editorial team. You will get an onboarding “buddy” from the experienced editors to help you out during that time.

Thanks to our editors who are stepping down

A few of our editors are completing terms and stepping down from editorial duties at JOSS. Lorena A Barba (@labarba), Kathryn Huff (@katyhuff), Karthik Ram (@karthik), and Bruce E. Wilson (@usethedata) have been amazing editors to have on the team and we will miss them very much!

References

Smith A. Announcing The Journal of Open Source Software - Arfon Smith. Published online May 5, 2016. Accessed July 12, 2022. https://www.arfon.org/announcing-the-journal-of-open-source-software

Smith AM. Reopening JOSS. Published online May 18, 2020. doi:10.59349/4tz9w-yq369

Smith AM, Niemeyer KE, Katz DS, et al. Journal of Open Source Software (JOSS): design and first-year review. PeerJ Computer Science. 2018;4:e147. doi:10.7717/peerj-cs.147

Katz DS, Smith AM, Niemeyer K, Huff K, Barba LA. JOSS’s Commitment to the Principles of Open Scholarly Infrastructure. Published online February 14, 2021. doi:10.59349/m5h23-pjs71

Katz DS, Barba LA, Niemeyer K, Smith AM. Cost models for running an online open journal. Published online June 4, 2019. doi:10.59349/g4fz2-1cr36

Katz DS, Barba LA, Niemeyer K, Smith AM. Scaling the Journal of Open Source Software (JOSS). Published online July 8, 2019. doi:10.59349/gsrcb-qsd74

The Journal of Open Source Software (JOSS) is committed to the Principles of Open Scholarly Infrastructure, and here we summarize our status in doing so, followed by a more detailed discussion of how we do so, as well as explaining some when we do not, and some work in progress.

This document was assembled by Daniel S. Katz, Arfon Smith, Kyle E. Niemeyer, Kathryn D. Huff, and Lorena A. Barba, and reviewed and approved by the active JOSS editorial board and topic editors, and the Open Journals Steering Council.

Summary

Governance
💛 Coverage across the research enterprise
💛 Stakeholder Governed
💚 Non-discriminatory membership
💚 Transparent operations
💚 Cannot lobby
💛 Living will
💚 Formal incentives to fulfil mission & wind-down

Sustainability
💚 Time-limited funds are used only for time-limited activities
💛 Goal to generate surplus
💛 Goal to create contingency fund to support operations for 12 months
💚 Mission-consistent revenue generation
💚 Revenue based on services, not data

Insurance
💚 Open source
💚 Open data (within constraints of privacy laws)
💚 Available data (within constraints of privacy laws)
💚 Patent non-assertion

(💚 = good, 💛 = less good)

Discussion

Governance

💛 Coverage across the research enterprise

• It is increasingly clear that research transcends disciplines, geography, institutions and stakeholders. The infrastructure that supports it needs to do the same.

Research software is essential to all types of research, and in response, JOSS’s coverage includes research software in any discipline, from any place, and from any institution.

The scope of what we publish is only limited by a few guidelines. JOSS publications must be research software of sufficient scholarly effort, which means that some software essential to research is excluded because it is either not research software, (e.g., a C compiler) or too small (e.g., a few hundred lines of Python that implement an existing tool or that provides a wrapper to access a fine-grained data source.)

JOSS strives for broad coverage of research disciplines in its editorial board, to better serve a broad community of authors.

💛 Stakeholder Governed

• A board-governed organisation drawn from the stakeholder community builds more confidence that the organisation will take decisions driven by community consensus and consideration of different interests.

Open Journals is fiscally sponsored by NumFOCUS, and has a documented governance structure. The steering council, being a small group, is limited in its representation, in terms of geographic, ethnic, gender, and organizational diversity. The editorial board members mostly represent North America and Europe, are mostly white, are mostly male, and are mostly hands-on researchers, primarily from universities and national laboratories.

💚 Non-discriminatory membership

• We see the best option as an “opt-in” approach with a principle of non-discrimination where any stakeholder group may express an interest and should be welcome. The process of representation in day to day governance must also be inclusive with governance that reflects the demographics of the membership.

Additions to the editorial board, which is the first layer of governance, are made via selections from responses to open calls and are non-discriminatory.

💚 Transparent operations

• Achieving trust in the selection of representatives to governance groups will be best achieved through transparent processes and operations in general (within the constraints of privacy laws).

JOSS is publicly transparent, much more so than most other journals. Few issues are not publicly open, but these are generally open to all editors, including initial discussions about potential changes to the journal and discussions about the scope of submissions that may or may not be accepted for review. Even though some of these discussions may not occur in the open, we do publish the decisions themselves, along with an explanation.

💚 Cannot lobby

• The community, not infrastructure organisations, should collectively drive regulatory change. An infrastructure organisation’s role is to provide a base for others to work on and should depend on its community to support the creation of a legislative environment that affects it.

The vast majority of time and effort used at JOSS is operational, with improvements of JOSS next, and finally some publicity about JOSS, which could be considered lobbying. However, this lobbying is mostly done by the JOSS editors because they think it is important to them, not to JOSS. The fact that they are also JOSS editors is a consequence of their feelings about the importance of recognizing contributions to research software, which also leads them to talk about this with others.

💛 Living will

• A powerful way to create trust is to publicly describe a plan addressing the condition under which an organisation would be wound down, how this would happen, and how any ongoing assets could be archived and preserved when passed to a successor organisation. Any such organisation would need to honour this same set of principles.

As discussed below, there are circumstances in which we would consider the mission of JOSS fulfilled and the journal no-longer necessary. While we have not documented a plan for winding down JOSS, we believe that the core assets associated with the journal (software, article metadata, papers) are appropriately preserved as part of our ongoing operations. The articles published in JOSS are persistently archived such that an end of the journal will not affect the scholarly record.

💚 Formal incentives to fulfil mission & wind-down

• Infrastructures exist for a specific purpose and that purpose can be radically simplified or even rendered unnecessary by technological or social change. If it is possible the organisation (and staff) should have direct incentives to deliver on the mission and wind down.

JOSS views itself as a temporary solution to provide a means for software developers and maintainers to receive credit for their work, and to have this work (research software) improved by the process of open peer review. We look forward to a time when software papers are not needed, when software is directly recognized and cited, and when software peer review (potentially using a future version of our criteria and processes) is more widespread. JOSS is volunteer-run as a service to the community, and most of the volunteers will be happy when a solution like JOSS is no longer needed, because software has found a more direct avenue to be valued and counted in the scholarly record.

Sustainability

💚 Time-limited funds are used only for time-limited activities

• Day to day operations should be supported by day to day sustainable revenue sources. Grant dependency for funding operations makes them fragile and more easily distracted from building core infrastructure.

JOSS does not depend on grants for regular operations, but has attracted grant funding for specific activities to improve the tooling that facilitates running JOSS. We make effective use of time-limited funds such as grants to support enhancements to our services.

💛 Goal to generate surplus

• Organisations which define sustainability based merely on recovering costs are brittle and stagnant. It is not enough to merely survive, it has to be able to adapt and change. To weather economic, social and technological volatility, they need financial resources beyond immediate operating costs.

As described in our blog post on the topic, our operational costs are deliberately very low. We currently do not generate a surplus and have no plans to. We also do not employ any staff and so “economic, social and technological volatility” would be expected to have limited impact on JOSS.

💛 Goal to create contingency fund to support operations for 12 months

• A high priority should be generating a contingency fund that can support a complete, orderly wind down (12 months in most cases). This fund should be separate from those allocated to covering operating risk and investment in development.

We have sufficient funds available today to support our operations for substantially longer than 12 months. Allocating some of these to a formal contingency fund is something we are considering but have not yet done. As a fiscally sponsored project of NumFOCUS, JOSS can receive donations from individuals, and we can kick off a fund-raising campaign at short notice. JOSS can also apply for NumFOCUS Small Development Grants, which are awarded several times per year.

💚 Mission-consistent revenue generation

• Potential revenue sources should be considered for consistency with the organisational mission and not run counter to the aims of the organisation. For instance…

JOSS revenue comes from three sources: a small amount from donations, a small amount from the American Astronomical Society (AAS) as fees for reviews of the software linked to AAS publications, and a larger amount from grants to the journal related to demonstrating its effectiveness, promoting the importance of research software, and recognizing research software’s contributors. These sources of revenue are fully consistent with the mission of JOSS.

💚 Revenue based on services, not data

• Data related to the running of the research enterprise should be a community property. Appropriate revenue sources might include value-added services, consulting, API Service Level Agreements or membership fees.

JOSS receives no revenue for its data, which is completely open, but rather receives revenue for its services and for its community impact.

Insurance

💚 Open source

• All software required to run the infrastructure should be available under an open source license. This does not include other software that may be involved with running the organisation.

All of JOSS’ tools are open source and available on GitHub under the Open Journals organization. This includes the JOSS website, our editorial bot Whedon, and the document production toolchain. Some of the collaboration tools we use as an editorial team are not open (e.g., GitHub, Slack, Google Docs), but these are not critical to the functioning of the journal and could be replaced by open alternatives.

💚 Open data (within constraints of privacy laws)

• For an infrastructure to be forked it will be necessary to replicate all relevant data. The CC0 waiver is best practice in making data legally available. Privacy and data protection laws will limit the extent to which this is possible

Our papers and the (Crossref DOI) metadata associated with them are available on GitHub, with an open license. We deposit open citations with Crossref, and archive papers and our reviews with Portico.

💚 Available data (within constraints of privacy laws)

• It is not enough that the data be made “open” if there is not a practical way to actually obtain it. Underlying data should be made easily available via periodic data dumps.

Our papers and the (Crossref DOI) metadata associated with them are available on GitHub, with an open license. These data are easily accessible to all motivated to make use of them.

We could potentially create data exports of the JOSS web application database; however this would just be an alternative representation of the data already available.

💚 Patent non-assertion

• The organisation should commit to a patent non-assertion covenant. The organisation may obtain patents to protect its own operations, but not use them to prevent the community from replicating the infrastructure.

JOSS has no interest in patents, other than resisting the creation of patents that might prevent us from operating freely.

References

POSI. The Principles of Open Scholarly Infrastructure. The Principles of Open Scholarly Infrastructure. Accessed February 21, 2021. https://openscholarlyinfrastructure.org/

Today we reached a huge milestone at JOSS – we published our 1000^th paper! JOSS is a developer friendly, free-to-publish, open-access journal for research software packages. Publishing 1000 papers (and reviewing the corresponding 1000 software packages) over the past ~4 years has been no small feat. This achievement has been possible thanks to the efforts of our journal team and community of reviewers who have all given their time to make JOSS a success. We take this opportunity to review some of what we’ve learnt over the past four years and outline some plans for the future.

A brief recap

Much has been written¹ on the topic of research software, and the challenges that individuals face receiving credit for their work. Software is critical for modern research, yet people who invest time in writing high-quality tools often aren’t well rewarded for it. The scholarly metrics of the “impact” of a researcher’s work do a poor job of supporting software (and more).

JOSS was created as a workaround to some of the challenges for supporting software development in academia. Launched in May 2016, JOSS provides a simple, reliable process for receiving academic career credit (through citation of software papers) for writing open source research software. Authors write and submit a short article (usually under 1000 words) about their software, and JOSS reviews the paper and the software², reviewing it for a variety of qualities including function, (re)usability, documentation³.

In establishing JOSS, we wanted the editorial experience to be very different from a traditional journal, and developer friendly – short papers authored in Markdown, review process on GitHub, open process and documentation – while at the same time following best practices in publishing depositing first-class metadata and open citations with Crossref, archiving papers and reviews with Portico, leaving copyright for the JOSS papers with authors and more.

We describe the journal this way: JOSS is an open access (Diamond OA) journal for reviewing open source research software. With a heavy focus on automation, our open and collaborative peer review process is designed to improve the quality of the software submitted and happens in the open on GitHub.

Some lessons learned publishing our first 1000 papers

JOSS is meeting a need of the research community

Something unclear when starting JOSS was whether demand from the research community would be sufficient. A few years in, we can safely conclude that JOSS is meeting a real need of the academic community.

It has taken us a little over four years to publish 1000 papers and before pausing submissions for two months starting in early March 2020 (to give relief to our volunteers during the pandemic), we were projecting to reach this milestone sometime in June this year. It took us a little under a year to publish our 100th paper, and an additional 8 months to reach our 200th. In that time we’ve grown our editorial board from an initial group of 10 to more than 50 today.

People are reading and citing JOSS papers

Well over half of all JOSS papers have been cited, and many have been cited hundreds of times.

While not designed for it, JOSS is proving a useful resource for people interested in new research software: JOSS currently receives ~10,000 monthly visitors to the journal website and provides language (e.g., https://joss.theoj.org/papers/in/Python) and topic-based (e.g., https://joss.theoj.org/papers/tagged/Exoplanets) search filters and feeds.

People are key

Every journal relies upon the expertise, knowledge, and time of their reviewers and JOSS is no different. 935 individuals have contributed reviews for our first 1000 papers, many having reviewed multiple times.

💖💖💖 THANK YOU 💖💖💖

Like many journals, as the number of submissions grows, JOSS has had to scale its human processes. Over the past few years we’ve added many more editors and associate editors-in-chief to enable papers to be handled efficiently by the editorial team. Simultaneously, we’ve developed our editorial robot Whedon from being an occasional assistant during the review process to being the backbone of the whole JOSS editorial process.

Automation is important

A big part of keeping our costs low is automating common editorial tasks where at all possible. The primary interface for editors managing JOSS submissions is a GitHub issue with the assistance of our Whedon bot who supports a broad collection of common editorial tasks. Other than having reviewers, editors and authors read papers, Pandoc-generated proofs are the final version, and no additional copy editing is done before a paper is published. PDF proofs and Crossref metadata are generated automatically by Whedon, and when the time comes, deposited with Crossref and published automatically too.

When starting JOSS, we thought that automation could be a big part of how things would work if the journal became successful. 1000 papers in, we believe it has been an absolutely critical part of our operations. We call this chatops-driven publishing.

Financial support

JOSS is committed to providing a high-quality service to the community at no cost for authors or readers (Diamond/Platinum Open Access). We’re transparent about our operating costs and have written about cost models for operating an online open journal.

While JOSS’ operating costs are modest, we’ve benefited from the support of a number of organizations including NumFOCUS (the ‘Open Journals’ organization is a sponsored project of NumFOCUS), the Gordon and Betty Moore Foundation, and the Alfred P. Sloan Foundation. It’s also possible to donate to JOSS if you would like to support us financially.

To the future!

With 1000 papers published and a further ~170 papers under review JOSS is busier than ever and there’s little sign of demand from the community slowing.

Over the next year or so, we’re going to be investing resources in a number of key areas to enable JOSS to scale further, improve the experience for all parties, and help others reuse the infrastructure we’ve developed for JOSS. We’ve captured much of what we want to achieve in this public roadmap. All of this will be possible thanks to a new grant from the Alfred P. Sloan Foundation, some highlights include:

Smarter reviewer assignment and management: Finding reviewers for a JOSS submission is still one of the most time-intensive aspects of the JOSS editorial process (though a large fraction of those we ask to review tend to accept, as they are excited by our mission). We think there’s lots of opportunity for substantially improving the success rate of finding potential reviewers through automation. Making sure we’re not overloading our best reviewers will also be an important aspect of this work.

A major refactor of our editorial bot Whedon: Whedon is a critical part of our infrastructure but has become hard to maintain, and almost impossible for other projects to reuse. We’re planning to rework Whedon into a general framework with a set of reusable modules for common editorial tasks.

Investments in open source: JOSS relies upon a small number of open source projects such as Pandoc and pandoc-citeproc to produce scholarly manuscripts (PDFs) and metadata outputs (e.g., Crossref and JATS). We’re going to work with the Pandoc core team to generalize some of the work we’ve done for JOSS into Pandoc core.

For many of us on the editorial team JOSS is a labor of love, and it has been quite a ride growing JOSS from an experimental new journal to a venue that is publishing more close to 500 papers per year. For those of you who have helped us on this journey by submitting a paper to JOSS or volunteering to review, thank you ⚡🚀💥.

References

Anzt H, Cojean T, Chen YC, et al. Ginkgo: A high performance numerical linear algebra library. JOSS. 2020;5(52):2260. doi:10.21105/joss.02260

Katz DS, Barba LA, Niemeyer K, Smith AM. Scaling the Journal of Open Source Software (JOSS). Published online July 8, 2019. doi:10.59349/gsrcb-qsd74

Smith AM. Call for editors. Published online December 21, 2018. doi:10.59349/546tr-5p719

Katz DS, Barba LA, Niemeyer K, Smith AM. Cost models for running an online open journal. Published online June 4, 2019. doi:10.59349/g4fz2-1cr36

Smith A. Chatops-Driven Publishing

Arfon Smith. Published online February 28, 2019. Accessed August 31, 2020. https://www.arfon.org/chatops-driven-publishing

Jiménez RC, Kuzak M, Alhamdoosh M, et al. Four simple recommendations to encourage best practices in research software. F1000Res. 2017;6:876. doi:10.12688/f1000research.11407.1

Cohen J, Katz DS, Barker M, Chue Hong N, Haines R, Jay C. The Four Pillars of Research Software Engineering. IEEE Softw. 2021;38(1):97-105. doi:10.1109/MS.2020.2973362

Katz DS, Druskat S, Haines R, Jay C, Struck A. The State of Sustainable Research Software: Learning from the Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.1). JORS. 2019;7(1):11. doi:10.5334/jors.242

tl;dr – JOSS is introducing new submission criteria whereby submissions under 1000 lines of code will automatically be flagged as potentially being out of scope, and those under 300 lines desk-rejected. This blog post describes some of the motivations behind this decision.

Sometime in 2020, JOSS will publish its 1000^th paper – an incredible achievement by a volunteer team of editors and reviewers.

Since its inception a little over four years ago, the primary goal of JOSS has always been to provide credit for authors of research software. Quoting from the original blog post announcing JOSS:

The primary purpose of a JOSS paper is to enable citation credit to be given to authors of research software.

One challenge we’ve always struggled with as an editorial team is defining clear guidelines for submissions allowed (and not allowed) in JOSS. Our current submission criteria are available online and include language about software having to be a significant contribution, feature complete, and having an obvious research application. In these criteria we also explicitly exclude a category of software we generally call “minor utilities”.

The challenge of defining a unit of publication credit

We think of JOSS essentially granting “1 publication credit” for each software package that is reviewed and published in the journal. In empirical research, a publication is often the result of years of work. In engineering research, rarely does a paper represent less than one year of work. Other fields may vary, but let’s say that a scientific paper resulting from work measured in just months is rare or exceptional.

Since the earliest days of the journal, there has been a range of views within the editorial team on what level of effort we should require from authors for a submission to be allowed in JOSS – some JOSS editors feeling that every useful piece of software should be considered, others believing that the “bar” for publishing in JOSS should be higher than it currently is.

Building trust in JOSS

As an editorial team we want JOSS papers to count the same as any other publication in the CV of researchers who write software. With career credit the stated primary reason for starting JOSS, if this isn’t true then the mission of the journal is at risk.

In reality this means that our editorial policy requires us to balance two competing needs:

1. Providing an excellent service to authors by offering peer-review of their software and the opportunity to receive career credit for their work.
2. Building the trust of an existing academic culture to accept a JOSS paper as equal to any peer-reviewed journal paper.

These two aspects are in tension with each other because while we would dearly love to publish any and all research software in JOSS regardless of size/scale and level of effort to implement, part of building and maintaining that trust with the community relies on us ensuring that our published authors can continue to expect JOSS papers to “count” in their future merit reviews and promotions.

Updates to our submission requirements and scope-checking procedures

Over the past couple of years, as the number of submissions to JOSS has grown, we’ve found that our existing submission criteria and protocol for rejecting papers as out of scope have been taking a significant fraction of our editorial team time. With a volunteer team of editors, it’s essential that we use their time carefully and an update to our procedures for handling scope assessments is long overdue.

Going forward we’re going to adopt the following new process:

Automatically flagging small submissions

As part of the pre-review process, incoming submissions that are under 1000 lines of code¹ (LOC) will be automatically flagged as potentially out of scope by the EiC on rotation.

Submissions under 300 lines² of code will be desk rejected with no further review.

Mandatory “Statement of need” section in JOSS papers

While this has always been preferred, a clear Statement of need section in the JOSS paper is usually extremely valuable in helping the editorial team understand the rationale for the development of the software.

Gauging the scholarly content of the software as part of the review

Reviewers will be asked if the software under review is a “substantial scholarly effort” and guidelines will be provided on how they can make that assessment.

A streamlined editorial review process

Rather than each paper that is potentially out-of-scope being discussed in a separate thread on our editorial mailing list, JOSS is going to move to a weekly review of such papers by our editorial team. Topic editors will be asked to review papers flagged as potentially out of scope in their area and help the EiC team make a decision.

References

Smith A. Announcing The Journal of Open Source Software - Arfon Smith. Published online May 5, 2016. Accessed July 7, 2020. https://www.arfon.org/announcing-the-journal-of-open-source-software

On March 12th of this year, we suspended new submissions to JOSS in order to reduce the load on our volunteer team of editors and reviewers. We revisited this suspension in early April and decided to continue the pause on new submissions for at least another month. Over the past two weeks, we’ve been reviewing our current status and discussing what it might look like to reopen JOSS.

After collecting inputs from the whole JOSS editorial team about their respective availabilities and discussing what to do next we’ve decided to reopen JOSS in a “reduced service mode” from May 20th, 2020.

Reduced service mode

So what does a “reduced service mode” mean exactly? Essentially this is about adjusting our editorial processes to give authors, editors, and reviewers more time to carry out their activities during the JOSS review process.

We expect that JOSS will be operating in this mode for the foreseeable future with the following new editorial processes:

Revised (extended) review time: Before the suspension, we encouraged reviewers to complete their reviews in two weeks. In this reduced service mode, we’re asking that reviewers try to complete their reviews in six weeks. Reviewers are of course welcome to complete their reviews faster than this and given the iterative nature of JOSS reviews we still encourage everyone to start their review as soon as possible.

Reduced paper throughput: As we reopen JOSS, we plan to limit the number of incoming submissions that will be assigned to a handling editor (and therefore go out for review). In these first few weeks we’re planning on a limit of five papers per week, with papers after this held in a first come, first served waitlist pre-review issue on GitHub.

Re-assign the submissions with unavailable editors: A small number of our editors are currently completely unable to edit due to other commitments. As we reopen JOSS, those papers will be reassigned to a new handling editor.

Growing our editorial team: Back on February, we issued a call for new editors which received an excellent response. Over the coming weeks we will begin on-boarding new editors to help grow the capacity of JOSS.

What this means for JOSS

During the pause on submissions, we’ve managed to publish 37 papers from the existing backlog which is an amazing feat in itself 💖. Since instigating the pause however, activity across all of JOSS had been markedly reduced.

JOSS relies upon editors and reviewers volunteering their time to help authors improve their research software in an iterative, open review process. For many of us, JOSS is a labor of love, and an opportunity to give back to the communities we work in.

While we can never know exactly how the COVID-19 pandemic is affecting every JOSS author, editor, and reviewer, we hope that with these changes to our editorial processes we have found the right balance between accommodating the very real challenges many of us now face in our daily lives, and providing a service to the research software community.

References

Barba LA, Katz DS, Moerman KM, Niemeyer K, Thyng K, Smith AM. Call for editors. Published online February 10, 2020. doi:10.59349/k5yqv-7hj89

Once again we’re looking to grow our editorial team at JOSS. We’re especially interested in recruiting editors with expertise in bioinformatics, material science, physics, R/statistics, and the social sciences.

Since our launch in May 2016, our existing editorial team has handled over 800 submissions (830 published at the time of writing, 119 under review) and the demand from the community continues to grow. The last three months have been our busiest yet, with JOSS publishing a little over one paper per day, and we see no sign of this demand dropping.

New editors at JOSS are asked to make a minimum 1-year commitment, with additional years possible by mutual consent. As some of our existing editorial team are reaching the end of their term with JOSS, the time is right to bring on another cohort of editors.

Background on JOSS

If you think you might be interested, take a look at our editorial guide, which describes the editorial workflow at JOSS, and also some of the reviews for recently accepted papers. Between these two, you should be able to get a good overview of what editing for JOSS looks like.

Further background about JOSS can be found in our PeerJ CS paper, which summarizes our first year, and our Editor-in-Chief’s original blog post, which announced the journal and describes some of our core motivations for starting the journal.

More recently we’ve also written in detail about the costs related with running JOSS, scaling our editorial processes, and talked about the collaborative peer review that JOSS promotes.

How to apply

Firstly, we especially welcome applications from prospective editors who will contribute to the diversity (ethnic, gender, disciplinary, and geographical) of our board.

✨✨✨ If you’re interested in applying please fill in this short form by 4th March 2020. ✨✨✨

Who can apply

We welcome applications from potential editors with significant experience in one or more of the following areas: open source software, open science, software engineering, peer-review, noting again that editors with expertise in bioinformatics, material science, physics, R/statistics, and the social science are most neded.

The JOSS editorial team has a diverse background and there is no requirement for JOSS editors to be working in academia. Unfortunately individuals enrolled in a PhD program are not eligible to serve on the JOSS editorial team.

Selection process

The JOSS editorial team will review your applications and make their recommendations. Highly-ranked candidates will then have a short (~30 minute) phone call/video conference interview with the editor(s)-in-chief. Successful candidates will then join the JOSS editorial team for a probational period of 3 months before becoming a full member of the editorial team. You will get an onboarding “buddy” from the experienced editors to help you out during that time.

References

Smith A. Announcing The Journal of Open Source Software - Arfon Smith. Published online May 5, 2016. Accessed December 19, 2018. https://www.arfon.org/announcing-the-journal-of-open-source-software

Smith AM, Niemeyer KE, Katz DS, et al. Journal of Open Source Software (JOSS): design and first-year review. PeerJ Computer Science. 2018;4:e147. doi:10.7717/peerj-cs.147

Katz DS, Barba LA, Niemeyer K, Smith AM. Cost models for running an online open journal. Published online June 4, 2019. doi:10.59349/g4fz2-1cr36

Katz DS, Barba LA, Niemeyer K, Smith AM. Scaling the Journal of Open Source Software (JOSS). Published online July 8, 2019. doi:10.59349/gsrcb-qsd74

2019 has been a big year for JOSS: This year, we’ve already published 300 papers and are on target to reach ~360 papers by the end of the calendar year. We’ve substantially improved our website and made a number of important changes to our editorial team to help us scale further.

In the last 12 months we’ve issued two calls for new editors (one in December 2018 and one in August 2019), and we’ve not done a very good job of announcing when new editors join the team so let’s fix that…

Two new associate editors in chief

In the last few months, we’ve added two associate-editors-in-chief with Kevin M. Moerman and Kristen Thyng stepping into these new roles ⚡.

New editors joining the JOSS team

Twenty one(!) new editors have joined the JOSS editorial team thus far in 2019. Ordered by the date they joined they are:

Viviane Pons: Mathematics, Computer Science
Associate professor at Université Paris-Sud. Mathematician, computer scientist and strong defender of open-source and open science in general. Contributor and user of the SageMath software. Member of the OpenDreamKit European project for open-source development in Mathematics.

Jack Poulson: Numerical optimization, numerical linear algebra, PDEs, high-performance computing, lattice reduction
Independent computational scientist running Tech Inquiry and Hodge Star Scientific Computing. Previously, research scientist at Google and assistant professor of mathematics at Stanford. His research interests: software engineering of high-performance mathematical libraries (e.g., conic optimization, lattice reduction, determinantal point processes, numerical PDEs), their connections to pure mathematics (e.g., differential geometry, conic analysis, representation theory).

George K. Thiruvathukal: HPC, software engineering, programming languages, systems, computational science, digital humanities
Professor of computer science at Loyola University, Chicago, and visiting faculty at the Argonne National Laboratory Leadership Computing Facility. Research interests: high-performance & distributed computing, cyber-physical systems, software engineering, programming languages and systems, history of computing, computational and data science, computing education, and ethical/legal/social issues in computing. Past editor-in-chief of IEEE Computing in Science and Engineering.

Lorena Pantano: Small RNAseq, RNAseq, miRNA, isomiRs, visualization, genomics, transcriptomic, non-codingRNA, data integration
Research scientist at Harvard T.H. Chan School of Public Health. Focused on genomic regulation and data integration, with more than a decade of experience in biological data analysis and contributing to novel algorithms to improve the quantification and visualization of genomic data.

Juanjo Bazán: Astrophysics, Mathematics
Astrophysics researcher at CIEMAT, mathematician and software engineer currently developing chemical evolution models for galaxies. Juanjo has worked as advisor on open source policies and contributed code to many popular libraries like Rails and Astropy. He is member of the founder team of Consul, the most widely used open sourced citizen participation software.

Bruce E. Wilson: Ecology, remote sensing, information sciences, material sciences
Manager for the Oak Ridge National Laboratory Distributed Active Archive Center for Biogeochemical Dynamics (ORNL DAAC) and Adjunct Professor of Information Sciences at the University of Tennessee, Knoxville. Originally trained as a chemist and statistician. Spent a few years as an Enterprise Architect. Research interests in citations, linked data, reproducible science, identity, cybersecurity, data reuse, and long-term data preservation.

Leonardo Uieda: Geoscience, Geophysics, Data Visualization
Geophysicist researching methods for determining the inner structure of the Earth from geophysical observations, mainly disturbances in the Earth’s gravity and magnetic fields. Developer of open-source software for processing, modeling, and visualizing geophysical data. Currently Visiting Research Scholar at the University of Hawai’i at Mānoa working on Generic Mapping Tools.

Alex Hanna: Social sciences, politics
Alex Hanna is a computational social scientist working on machine learning curriculum at Google. She received her PhD in sociology from the University of Wisconsin-Madison. Her research has focused on how new and social media has changed social movement mobilization and political participation. More recently, she has been interested in issues of fairness, accountability, and transparency in sociotechnical systems.

Charlotte Soneson: Bioinformatics, data visualization, transcriptomics, reproducible research
Research Associate at the Friedrich Miescher Institute for Biomedical Research in Basel, Switzerland, with a research background mainly in development and evaluation of analysis methods for transcriptomics data. Developer and maintainer of several open-source R packages for analysis, quality assessment and interactive visualization of high-throughput biological data.

Monica Bobra: Heliophysics, Data Science
Research scientist at Stanford University in the W. W. Hansen Experimental Physics Laboratory who studies the Sun and space weather as a member of the NASA Solar Dynamics Observatory science team and contributes to Heliopython and SunPy.

Yuan Tang: Machine Learning, Distributed Systems, Cloud Computing
Senior software engineer at Ant Financial, building AI infrastructure and AutoML platform. He’s a committer of TensorFlow, XGBoost, and Apache MXNet, maintainer of several Kubeflow projects, and author of numerous open source software projects. He’s also the author of the best-selling book TensorFlow in Practice, which is the first book teaching TensorFlow in Chinese and has been translated to several other languages such as traditional Chinese and Korean.

Olivia Guest: Computational cognitive modeling, cognitive science, cognitive neuroscience
Olivia Guest is a computational modeler in cognitive science and neuroscience. She creates and evaluates computational accounts for categorization and conceptual representation in healthy adults, patient groups, infants, and animals. She is also interested in using computational modeling and data science broadly in theoretical as well as applied contexts.

Marie E. Rognes: Applied mathematics, PDEs, numerical methods, biomechanics
Marie E. Rognes is Chief Research Scientist at Simula Research Laboratory, Oslo, Norway. Her research focuses on numerical methods for partial differential equations, software for scientific computing, with applications in biomechanics and neuroscience. She is a core member of the FEniCS and Dolfin-adjoint Projects.

Vincent Knight: Mathematics, Applied mathematics, Game Theory, Stochastic processes, Pedagogy, Python
Vince is a mathematician at Cardiff University. He is a maintainer and contributor to a number of open source software packages and a contributor to the UK Python community. His research interests are in the field of game theory and stochastic processes and he also has a keen interest in pedagogy. Vince is a fellow of the Software Sustainability Institute and is interested in reproducibility and sustainability of scientific/mathematical research.

Matthew Sottile: Computer science, programming languages, applied mathematics
Research scientist working on program synthesis tools for high performance computing at Noddle.io, and adjunct faculty at the Washington State University Department of Mathematics and Statistics. His PhD work in computer engineering focused on measurement and analysis methods for performance analysis of high performance systems. He believes that open and reproducible research software artifacts are critical to the advancement of computational science.

Mark A. Jensen: Bioinformatics and Computational Biology, Cancer Genomics, Population and Evolutionary Biology, Databases and Data Architecture, Software Development Lifecycle Management
Director of Data Management and Interoperability at the Frederick National Laboratory for Cancer Research, he leads efforts to design, build, and maintain scientist-friendly research data systems that integrate clinical and multiomic data across thousands of cancer patient-donors. He is active in open source software development and a supporter of the FAIR (Findable, Accessible, Interoperable, Reusable) movement in scientific data management. A molecular evolutionary biologist by training, he served as an Associate Editor for the Journal of Molecular Evolution from 2008-2013.

Melissa Weber Mendonça: Applied Math, Numerical Optimization, Numerical Linear Algebra, Fortran, Python, MATLAB, Julia, Teaching
Applied mathematician, working with numerical optimization and numerical linear algebra at the Federal University of Santa Catarina, in Brazil. Interested in open science, open research, and teaching practices in mathematics and computer science.

Katy Barnhart: Geology, Geophysics, Earth Surface Dynamics, Terrain Analysis
Research scientist in Earth surface dynamics at the University of Colorado Department of Geological Sciences and Cooperative Institute for Research in Environmental Sciences. Her research focuses on integrating observations, model development, and model analysis to understand the evolution of the Earth’s surface. She is a core developer of the Landlab toolkit.

William Rowe: Bioinformatics, genomics
Bioinformatician working at the University of Birmingham, UK. Research areas include data sketching for genomics, variation graphs, and metagenome profiling. Currently working on long-read sequencing applications for real-time genomic epidemiology.

Dan Foreman-Mackey: Astrophysics, probabilistic programming, data science, python
Dan Foreman-Mackey is an Associate Research Scientist at the Flatiron Institute in the Center for Computational Astrophysics. His research program focuses on the development and application of probabilistic data analysis techniques to make novel discoveries and solve fundamental problems in astrophysics.

Marcos Vital: R, biostatistics, data visualization, open science, quantitative ecology, evolutionary biology, conservation biology, teaching
Biologist, quantitative ecologist and open science enthusiast at Universidade Federal de Alagoas, Brazil, where he coordinates the Quantitative Ecology Lab. He is interested in a wide range of topics related to evolutionary ecology and conservation biology, but also in other fields such as active learning methods and digital games in education.

Editors retiring from the team

In 2019, a few of our editors have stepped down from editorial duties at JOSS: Jason Clark, Pjotr Prins, and Yo Yehudi have already retired from the JOSS editorial team and Roman Valls Guimera, Melissa Gymrek, and Lindsey Heagy will be retiring from the team soon. JOSS only works because of the volunteer efforts from our editors and reviewers and we’re deeply indebted to them 💖.

We thanks Jason, Pj, Yo, Roman, Melissa, and Lindsey for all of their contributions to JOSS!

References

Smith AM. Call for editors. Published online December 21, 2018. doi:10.59349/546tr-5p719

Barba LA, Katz DS, Niemeyer K, Smith AM. Call for editors. Published online August 20, 2019. doi:10.59349/z1fj7-x7z68