In addition to requesting / requiring code in relevant submissions, one could also imagine journals recruiting reviewers specifically to evaluate code / reproducibility.
While it is obviously not reasonable to expect such a reviewer to exhaustively evaluate the validity of large and complicated software, there are some very basic and easy things that could be checked with minimal effort.
For example, I recently participated in a three day reproducibility workshop at NIH which attempted to teach researchers about the principles and best practices of reproducibility by reproducing ~10 bioinformatics / genomics papers which appeared to have all of the information / code necessary to be easily reproduced. Within 2 minutes of looking at the RMarkdown code for the very first paper, it was obvious that it had no hope of ever being run (referenced variables not defined anywhere in the file).
Some things that would be easy to check:
- Are all variables defined?
- Is package information captured?
- Is the code documented at some minimal level?
- Does the file reference directories / data that are unique to some user’s system? (e.g.
setwd("/home/jsmith"))
In about ten minutes of downloading a script / software and attempting to get it running, you could at least make sure it passes these minimum requirements.