Forum Discussion
Dataflow Incremental Refresh Update Specific Partitions
I believe this is required feature for many.
In my case, I have huge table loaded into my dataflow and would like to process the partitions seperately when I need reprocess all the partitions.
Only option now is loading all of the partitions sequentally and then incrementally after that.
Can someone confirm that there is no way of processing dataflow partitions sepperately?
Hey Solvisig,
- Can you please elaborate this in details regarding only way available. I am stuck in same situation. I have a dataflow 0.8 Million rows. After setting up the incremental refresh when I refresh the dataflow, it timeouts in 3 hours. Thinking to do the partitioning in SQL server then bringing the table in dataflow. Will there be any way then to refresh the partitions one by one? If yes, please do tell. Also, if there are any other ways, it will be really helpful. Thanks in advance.
The only option is using the refresh button which refreshes all the partitions sequentially for the initial load and then incrementally after that. There is currently no way to have control over your partitions when using dataflows. Thats why I am not using dataflows for many of my workloads so I use standard datasets instead and can easily manage the partitions in tabular editor and other tools.
However with Fabric now being the new hot thing, this must be on the development backlog.This is definitely a required feature. I need to move a dataset currently working under incremental refresh and managed through XMLA endpoint to a dataflow, so we can improve the refresh performance by doing all of the transformation within dataflows. I cannot currently see a way to manage the partitioning, beyond the initial setup of the incremental refresh.
