I am working on some code that extracts total radiance of country regions using polygons from http://www.gadm.org. I have expanded it to be able to calculate total radiance at the lowest sub-national level and crops the tiles to the country outline for 1. faster future data extraction and 2. possible display. Currently also playing around with a shiny interface that allows exploration of country data and possibly integration with other data sources and simple model building. As I keep going I am wondering if it maybe could grow all the way into an R package.
So far I have the satellite package seems closest in functionality, but this is a much more low level interface, from what I have seen - it deals with raw data and applying atmospheric corrections etc. It also seems mainly to deal with Landsat (as opposed to SNPP/VIIRS) although it seems the package has the potential to expand to other satellite data products.
NB: I am also considering offering the option of the package connecting to a commercial service for those who want the data without having to download and process the tiles and am not sure how this is viewed by rOpenSci/CRAN?
Is your code in a private repo right now? I’d advise putting code into a package now, can’t hurt.
Some clarifying questions:
Where is the radiance data coming from? it’s not clear. sounds like you’re only getting spatial polygons from gadm
When you say this is a much more low level interface - do you mean your code is lower level, or satellite is?
For the commercial data, we do have some R pkgs that work with data that you need an API key for, but they are all as far as I can remember, open to anyone that wants to use them without paying. Would users have to pay?
Thanks for getting back and for the advice. The code is currently in a private repo on github. It still needs a fair amount of work. I will at the very least put it into the public space.
The satellite package is lower level than what I am working on.
My idea is that it will be free to use if the user downloads the imagery from the NASA website and polygons from GADM and processes locally. However, there will be an option to obtain an api key from the website which will allow one to download pre-processed data such as the radiance stats and already cropped and masked images.
That is free? or users have to pay to get an API key?
If it’s free, that all sounds good to me, and I think may be a good candidate for ropensci. but I want my co-editors to chime in here @maelle@noamross
I haven’t worked it all out, but since the data is identical to what you would get if you run the package, I was thinking of charging for preprocessed data. Charges may vary based on volume/frequency of downloads and may include a free tier. Does this scenario fit the bill? I would be interested in finding out how other packages are doing it.
I think a core useful functionality of the package that would interest us would be: functions to download, cache, and pre-process the raw data from NOAA. For instance, a function could take a bounding box or polygon as an argument, along with with date and image series, download the appropriate tile if it’s not already cached and return a matrix or raster of the requested area. (Similar to the ccafs package) Anything on top of that, such as calculating irradiance across the tile or related statistics of interest, image processing, would be gravy in the package, and in fact we avoid methods-heavy packages as we have less expertise in reviewing field-specific statistical implementations. We would prefer that the package aims to perform the download/clip type tasks as speedily and efficiently as one can on local machines, rather than drive users to a service (and can help with how to do that). But if the service is providing a useful additional task or making the process faster for people, even if its just hosting much smaller tile sets to make user work flow more efficient, I have no problem with it having a commercial model. If core functionality is in the paid service, given that the data is free we wouldn’t accept the package (though others, such as cloudyr or JOSS, would).
I know you can already grab and import data from GADM using raster::getData() (I think they provide it as .Rdata files), not sure if that’s built in to any of other or our own packages.
Thanks for your input @noamross. All the core functionality is available in the package i.e. download, cache and pre-process data from NOAA. Currently just like you have said, one supplies the image series, country code(s), and the year(s) and month(s) required and the function will download the appropriate tiles and polygons, crop and mask the tiles to the country outlines and then extract data at the lowest country admin level. There are other tidbits such as a function that build a map file that can be used to configure a mapserver to serve the cropped rasters (used by the shiny app).
The service can be seen as more of a remote cache. Some of the larger country polygons can take a while to crop and mask and this would simply save time. If the data extraction is not important then it can also be part of the service. Since the site also has a shinydashboard interface to the data, it can serve as an example of how the package can be used.
If I am not wrong, ggmap also has the option to pull polygons from GADM possibly in .rds format. In my package, I download ESRI shapefiles since I found it easier to navigate the country admin levels. However, I am not attached to any particular method.
Following up on the discussion here. If there is any interest, I would like to enlist support for this project. The code is available at https://github.com/chrisvwn/shiny_nightlights. There is no sample data and currently all it does is download, clip and extract the latest version of satellite images (SNPP/VIIRS) though there are placeholders for the older version (DMSP/OLS) and support for future versions. If there is any interest I can provide more info on what does what.
I am happy to announce that the package Rnightlights is now available on CRAN. It is still in alpha and any suggestions and bug reports are very welcome.