Forum Discussion
Dataset incremental refresh from flow not working?
And of course the moment I posted this it finally clicked. Folding issue with how i transformed the data. My bad. Had to make changes to the source table so that they weren't needed on the import.
Perhaps i've misunderstood the purpose of a data flow, but I was attempting to use them to better consolidate our data across workspaces and for future reporting.
Is it just not possible to incrementally refresh a dataset that is getting it's data from a data flow? Or is there some way to make a report dependent on a data flow without using a dataset in betwen?
Dataflows are glorified CSV files. Or Parquet, if you want. Their purpose is to shield the downstream systems from poor performance of the data source.
It is technically possible to have an incremental dataset feed off an incremental dataflow. But is it necessary? I would say no. Dataflows are designed to deliver maximum spool performance when consumed. There's no need for the dataset to be picky.
On the other hand, dataflows add one more point of breakage to your process. If you have an incremental refresh dataset and it is performing well on its own, then there is no need for a dataflow, or chained dataflows that seem to be all the rage now.
And no, you cannot feed a report directly from a dataflow. The dataset part is mandatory.