Forum Discussion
Dataset incremental refresh from flow not working?
And of course the moment I posted this it finally clicked. Folding issue with how i transformed the data. My bad. Had to make changes to the source table so that they weren't needed on the import.
"but it eventaully times out 12 hours+ later." I would doubt that, the default hard timout limit on the Power BI service is 600 minutes.
Can you explain the reasoning for trying to feed your dataset from a dataflow? If you are already using incremental refresh in your dataset then you don't need the dataflow?
Perhaps i've misunderstood the purpose of a data flow, but I was attempting to use them to better consolidate our data across workspaces and for future reporting.
Is it just not possible to incrementally refresh a dataset that is getting it's data from a data flow? Or is there some way to make a report dependent on a data flow without using a dataset in betwen?
Dataflows are glorified CSV files. Or Parquet, if you want. Their purpose is to shield the downstream systems from poor performance of the data source.
It is technically possible to have an incremental dataset feed off an incremental dataflow. But is it necessary? I would say no. Dataflows are designed to deliver maximum spool performance when consumed. There's no need for the dataset to be picky.
On the other hand, dataflows add one more point of breakage to your process. If you have an incremental refresh dataset and it is performing well on its own, then there is no need for a dataflow, or chained dataflows that seem to be all the rage now.
And no, you cannot feed a report directly from a dataflow. The dataset part is mandatory.
Good to know, but then i'm still left with "why won't it incrementally refresh" since it is possible.
I'm testing on the dataset directly now and having issues there as well, so i'm not sure what could be going on that would allow me to incrementally refresh the flow, but not the dataset. It's as if it can't load the data into the dataset but has no problem loading the flow.
It is a ton of data for sure, but I'm not sure how to get it published at this point. I've tried loading all the data intially and it seems to change nothing?
And of course the moment I posted this it finally clicked. Folding issue with how i transformed the data. My bad. Had to make changes to the source table so that they weren't needed on the import.