Given experience in other langs (Ruby, Python, etc.) it seems mocking in tests is definitely the standard. And makes test suites run faster, etc. Of course testing the API itself is often still a concern, so need to do both.
I'm nearly there with a useable first version of
vcr https://github.com/ropenscilabs/vcr ported from Ruby. It will be super flexible, allowing mocking of HTTP requests, matching patterns based on base URL, paths, query params, headers, any combination of the previous, etc.
Separate tests, both mocked and not (which can be run alternately depending on whether on CRAN)
perhaps a set of triggers is best, so test if on CRAN or not and internet avail or not. Then, if on CRAN or if internet not avail., then use mocked data, if not on CRAN and internet avail use full tests, or maybe only full tests when on Travis e.g.
Mocked tests, with a check that returns a message, not an error, for when a website can't be reached
Agree, this is useful, since checks can still pass, but then we still know if the web service is down
Explicit tests that check that the data returned from an API is the correct form, before the package processes it.
Perhaps, but i guess the exported functions that call the API will presumably account for this, though maybe not fail well when the API response is not in correct form depending on how code is written
Just ask authors to be explicit in package submission what tests are running on Travis/CRAN and which are being run locally.
Hmm, maybe we could scan the pkg code and see which fxns make http requests by looking for e.g.,
httr::GET, etc. then we'd know which tests make requests?
In the long term, as some have mentioned, its probably best to get all of our software to separate out the code that does the HTTP request and the code that parses the data, so they can be tested separately. I'm not sure what the best pattern is to do this, but could explore this separately.
devtools::run_examples(run = FALSE) but presumably there's a way to do it from R check, but not sure what the flag would be.