Forum Discussion
Dataset incremental refresh from flow not working?
And of course the moment I posted this it finally clicked. Folding issue with how i transformed the data. My bad. Had to make changes to the source table so that they weren't needed on the import.
Dataflows are glorified CSV files. Or Parquet, if you want. Their purpose is to shield the downstream systems from poor performance of the data source.
It is technically possible to have an incremental dataset feed off an incremental dataflow. But is it necessary? I would say no. Dataflows are designed to deliver maximum spool performance when consumed. There's no need for the dataset to be picky.
On the other hand, dataflows add one more point of breakage to your process. If you have an incremental refresh dataset and it is performing well on its own, then there is no need for a dataflow, or chained dataflows that seem to be all the rage now.
And no, you cannot feed a report directly from a dataflow. The dataset part is mandatory.
Good to know, but then i'm still left with "why won't it incrementally refresh" since it is possible.
I'm testing on the dataset directly now and having issues there as well, so i'm not sure what could be going on that would allow me to incrementally refresh the flow, but not the dataset. It's as if it can't load the data into the dataset but has no problem loading the flow.
It is a ton of data for sure, but I'm not sure how to get it published at this point. I've tried loading all the data intially and it seems to change nothing?
And of course the moment I posted this it finally clicked. Folding issue with how i transformed the data. My bad. Had to make changes to the source table so that they weren't needed on the import.