The README of the onboarding repo now states very clearly why to review packages for rOpenSci, but it does not mention any criteria for becoming a reviewer.
I’ve for instance been asked by someone with programming experience but no R package development experience whether they could become a reviewer. Should the README include a sentence about who can become a reviewer (so that potential reviewers don’t feel intimidated before volunteering)? And if it does, what should this sentence be?
Some disjointed thoughts:
We are trying to move towards having two reviewers for all reviews at this point. With two reviewers, I am comfortable with one reviewer having experience with package development, and another having R programming experience and domain knowledge related to the package’s uses.
That said, I don’t want to “split” the review - as in the case of journal paper reviews - I’d like all reviewers to review the package comprehensively, though from their particular perspective.
One thing that may help is moving to a more robust “checklist” for reviewers that’s closely tied to the packaging guide. That’s what we’re moving towards in the JOSS-harmonization revision. @hrbrmstr’s geoparser review is also a great example of checklist-type. A checklist is no substitute for experience, but it would help a new reviewer, especially when they are paired with a more experienced one.
We have been discussing reviewer training. I guess the very short version would be to ask that reviewers read our packaging guide and Hadley’s R Packages.
I was about to say we should start adding volunteers to our database, but I see @sckott has already started. I guess we should add to the notes something about package development experience.
I should also probably write down somewhere what I look for and how I go about looking for reviewers:
What I look for:
- Some package development experience (searching CRAN and the potential reviewer’s github page for evidence they’ve developed a package)
- Some domain experience (again, often looking at their github page, home page or publications to find some experience with code that’s, say, geospatial or does text-mining or whatever the domain of the package is).
- No conflicts of interest - sometimes I find cross-citations/dependencies between the package and a reviewer and I ask in the recruitment e-mail if they are involved somehow.
- I try to balance my sense of the potential reviewer’s experience against the complexity of the package.
- Diversity - with two reviewers both shouldn’t be cis white males.
Where I look:
- I check our database for people who haven’t reviewed in six months
- I ping our #onboarding slack channel for recommendations on who is good. (This is 90% a conversation between the editors and a bit of other rOpenSci leadership, but is open to anyone on the Slack)
- I scour the ROpenSci website and unconf listings for people within our community already but haven’t reviewed.
- I look for people active on on twitter, usually on the #rstats hashtag, and also in the feed of @RWomenTaskforce
- I search for authors of related packages on r-pkg.org, then look for them on GitHub, Twitter, or a website for some evidence that they’re interested in openness or R community activities. Even if not, I still blind e-mail some.
Thanks for the question @maelle
Should we add a section to the README for
onboarding about what we look for in reviewers?
I added a short bit (mindful that the README is getting long):
To volunteer to be one of our reviewers, please e-mail us at
firstname.lastname@example.org or contact us on Twitter at @rOpenSci. We are always
looking for more reviewers with both general package-writing experience and
domain expertise in the fields packages are used for.
Maybe that should be an “or”, not an “and”?
@noamross is it that bad that the README is getting long, since there is a table of contents at the beginning? Regarding package development experience, maybe you could add a link to Hadley’s R Packages book since you mentioned it here?
By the way the description of the repo is very short (“rOpenSci Onboarding”) -> would more details, e.g. “Each issue = 1 review” be useful?