Criteria for being package reviewer?

Software-Review
1

The README of the onboarding repo now states very clearly why to review packages for rOpenSci, but it does not mention any criteria for becoming a reviewer.

I’ve for instance been asked by someone with programming experience but no R package development experience whether they could become a reviewer. Should the README include a sentence about who can become a reviewer (so that potential reviewers don’t feel intimidated before volunteering)? And if it does, what should this sentence be?

1 Like
2

Some disjointed thoughts:

  • We are trying to move towards having two reviewers for all reviews at this point. With two reviewers, I am comfortable with one reviewer having experience with package development, and another having R programming experience and domain knowledge related to the package’s uses.

  • That said, I don’t want to “split” the review - as in the case of journal paper reviews - I’d like all reviewers to review the package comprehensively, though from their particular perspective.

  • One thing that may help is moving to a more robust “checklist” for reviewers that’s closely tied to the packaging guide. That’s what we’re moving towards in the JOSS-harmonization revision. @hrbrmstr’s geoparser review is also a great example of checklist-type. A checklist is no substitute for experience, but it would help a new reviewer, especially when they are paired with a more experienced one.

  • We have been discussing reviewer training. I guess the very short version would be to ask that reviewers read our packaging guide and Hadley’s R Packages.

  • I was about to say we should start adding volunteers to our database, but I see @sckott has already started. I guess we should add to the notes something about package development experience.

2 Likes
3

I should also probably write down somewhere what I look for and how I go about looking for reviewers:

What I look for:

  • Some package development experience (searching CRAN and the potential reviewer’s github page for evidence they’ve developed a package)
  • Some domain experience (again, often looking at their github page, home page or publications to find some experience with code that’s, say, geospatial or does text-mining or whatever the domain of the package is).
  • No conflicts of interest - sometimes I find cross-citations/dependencies between the package and a reviewer and I ask in the recruitment e-mail if they are involved somehow.
  • I try to balance my sense of the potential reviewer’s experience against the complexity of the package.
  • Diversity - with two reviewers both shouldn’t be cis white males.

Where I look:

  • I check our database for people who haven’t reviewed in six months
  • I ping our #onboarding slack channel for recommendations on who is good. (This is 90% a conversation between the editors and a bit of other rOpenSci leadership, but is open to anyone on the Slack)
  • I scour the ROpenSci website and unconf listings for people within our community already but haven’t reviewed.
  • I look for people active on on twitter, usually on the #rstats hashtag, and also in the feed of @RWomenTaskforce
  • I search for authors of related packages on r-pkg.org, then look for them on GitHub, Twitter, or a website for some evidence that they’re interested in openness or R community activities. Even if not, I still blind e-mail some.
2 Likes
4

Thanks for the question @maelle

Should we add a section to the README for onboarding about what we look for in reviewers?

5

I added a short bit (mindful that the README is getting long):

To volunteer to be one of our reviewers, please e-mail us at
info@ropensci.org or contact us on Twitter at @rOpenSci. We are always
looking for more reviewers with both general package-writing experience and
domain expertise in the fields packages are used for.

Maybe that should be an “or”, not an “and”?

6

@noamross is it that bad that the README is getting long, since there is a table of contents at the beginning? Regarding package development experience, maybe you could add a link to Hadley’s R Packages book since you mentioned it here?

By the way the description of the repo is very short (“rOpenSci Onboarding”) -> would more details, e.g. “Each issue = 1 review” be useful?