Does a SQL workflow fit within the drake / targets framework, or does that go counter to its design?
There are a few challenges I’m running into, and I’m curious how those with more experience with drake/targets think about them:
In Drake, if I change an input file for a data-frame, it would be reconstructed from that file. In a SQL workflow, I would be more interested in persisting the diffs. Moreover changes in the database table aren’t tied within Drake, but it makes perfect sense for the top-down process to be async updated. That is, the data frame that the table is loaded from won’t update if the database table changes, and would need refreshed.
In my workflow for populating the database, target granularity is harder when thinking about SQL. I have a table that I want to update multiple times, as the process is populating these fields based on the prior steps. This multiple update method seems to work against Drake/Targets, though I suppose I could just return the new column with the keys each time? Still, it seems like what I want to do is refresh my local copy of the table after each persistent change.
Are there any complex examples of using Drake/Targets for SQL projects that both (a) populate a database from scratch, and (b) then update it based on input files? And do you think this fits, or is this more of an event driven problem and needs a different framework?