Forum Discussion
Dataflow Incremental Refresh Update Specific Partitions
Could I have a link to the documentation page please?
No, I can't see the Dataflow in the XLMA Endpoint. I only see the Datasets.
Here is a screenshot of the capacities settings:
Are the Dataflows hidden from the list, but I still connect with the right "Database" or "Initial Catalog" name.
Do I connect using Analysis Services or SQL Database or a different one?
Hello,
I'm also very interested in this topic.
Given a dataflow, which has incremental refresh set up (keeping last 12 months of data, with incremental refresh of 2 months), meaning every exeution refresh only current month and previous, I would like to have a way to refresh only one partition 4 month ago.
For Datasets this is easy
- Datasets are stored in underneath Analysis services. This analysis service is accessible via SSMS(SqlServerManagementStudio) using workspace XMLA endpoint.
- From SSMS, refresh to single partition can be executed, so you are in full control of which partition you want to refresh. In my example, I could manually refresh my partition of 4 months ago.
For Dataflows:
- This is fully diferent story. Dataflows are stored in ADLS, not in Analysis Services, so obviously they cannot be reached from SSMS via analysis services connection. So same approach used for datasets do not apply to dataflows.
- Question is: Is there any way to refresh a dataflow concrete partition? (I know how to do full refresh, and how to run incremental refresh for last n partitions. But questio is how to process only one partition, which is not in the range of the incremental refresh last last n partitions).
Many thanks
I believe this is required feature for many.
In my case, I have huge table loaded into my dataflow and would like to process the partitions seperately when I need reprocess all the partitions.
Only option now is loading all of the partitions sequentally and then incrementally after that.
Can someone confirm that there is no way of processing dataflow partitions sepperately?Hey Solvisig,
- Can you please elaborate this in details regarding only way available. I am stuck in same situation. I have a dataflow 0.8 Million rows. After setting up the incremental refresh when I refresh the dataflow, it timeouts in 3 hours. Thinking to do the partitioning in SQL server then bringing the table in dataflow. Will there be any way then to refresh the partitions one by one? If yes, please do tell. Also, if there are any other ways, it will be really helpful. Thanks in advance.
The only option is using the refresh button which refreshes all the partitions sequentially for the initial load and then incrementally after that. There is currently no way to have control over your partitions when using dataflows. Thats why I am not using dataflows for many of my workloads so I use standard datasets instead and can easily manage the partitions in tabular editor and other tools.
However with Fabric now being the new hot thing, this must be on the development backlog.
