I wanted to check in and discuss what we should be recommending for testing of API packages for rOpenSci submissions. I find a lot of diversity in how these packages are tested and dealt with in CI, and I’d like to have some consensus.
Many times people just use
skip_on_cran() for everything, and have Travis-CI act as CRAN. In this case. Travis-CI status is somewhat meaningless. We can’t count on it to give package status. We can usually run tests (and measure test coverage) locally, as I do on package submission, but we won’t see breakage as packages get updated during the review or during our regular testing of accepted packages. Alternatively, Travis-CI can use NOT_CRAN, but this will often give false errors due to connectivity issues.
Karina Marks and @gabor’s recent blog post about mocking gives a neat alternative that makes API calls CRAN compatible and not dependent on API service status. However, we want to know if there’s an error caused by changes to the API being wrapped, as well.
Here are some potential strategies:
- Separate tests, both mocked and not (which can be run alternately depending on whether on CRAN)
- Mocked tests, with a check that returns a message, not an error, for when a website can’t be reached
- Explicit tests that check that the data returned from an API is the correct form, before the package processes it.
- Just ask authors to be explicit in package submission what tests are running on Travis/CRAN and which are being run locally.
P.S. Is there a way to automatically test example code even if it’s wrapped in