Dataset incremental refresh from flow not working?

[dungar]

Setup-

I have a workspace name Flows, with a Dimensions flow and a SalesData flow. 

The dimensions flow contains various dimensions, and full refreshes every day, the SalesData flow contains various sales tables (2 we need for this purpose), 1 of which incrementally refreshes (stores data going back to 2000, refreshes the past 14 days, Sales table). 

The SalesData flow takes about 10 mintues to refresh.

Problem-

I made a report, that pulls a few dimensions from the Dimension flow, and the Brand and Sales table from the SalesData flow.  I then configured incremental refreshing on the Sales table in the report to be basically the same as the Sales table in the Flow (store back for years, refresh past 14 days).  Further I set the initial parameters to pull life to date so i could do a local refresh to get the majority of the data right at the start.

I then published this report + dataset to the Flows workspace.  I tried triggering a quick refresh to test, and it just "hangs".  By hanging i mean it acts as if it's refreshing, but it eventaully times out 12 hours+ later.

Have i configured something wrong or is there some sort of limitation that i'm running into?  My suspicon is that even though i've got incremental refresh configured on both the flow and the dataset, that for some reason it's not actually incrementally refreshing the dataset and trying to do the whole thing (possibly as an initial load?), but I can't really see where/how it's failing or what it's trying to pull.  Further I feel like having set the intial params to force the local pbix to have all the data should avoid that problem, but maybe not.

Any advice on how to get this to work?  I'm pretty close to finally getting this all cleaned up but I'm concerned that I won't be able to intially load my larger models.  In this case it's some 43 million records worth of data, but the pbix itself is only 300k.  I know i'm going to have MUCH larger datasets when i'm done testing though, as i've pushed past the 1 gig limit on PRO before.

[lbendlin]

"but it eventaully times out 12 hours+ later."  I would doubt that, the default hard timout limit on the Power BI service is 600 minutes.

Can you explain the reasoning for trying to feed your dataset from a dataflow? If you are already using incremental refresh in your dataset then you don't need the dataflow?

[dungar]

Perhaps i've misunderstood the purpose of a data flow, but I was attempting to use them to better consolidate our data across workspaces and for future reporting.

Is it just not possible to incrementally refresh a dataset that is getting it's data from a data flow?  Or is there some way to make a report dependent on a data flow without using a dataset in betwen?

[lbendlin]

Dataflows are glorified CSV files. Or Parquet, if you want. Their purpose is to shield the downstream systems from poor performance of the data source.

It is technically possible to have an incremental dataset  feed off an incremental dataflow. But is it necessary?  I would say no.  Dataflows are designed to deliver maximum spool performance when consumed. There's no need for the dataset to be picky.

On the other hand, dataflows add one more point of breakage to your process.  If you have an incremental refresh dataset and it is performing well on its own, then there is no need for a dataflow, or chained dataflows that seem to be all the rage now.

And no, you cannot feed a report directly from a dataflow. The dataset part is mandatory.

[dungar]

Good to know, but then i'm still left with "why won't it incrementally refresh" since it is possible.

I'm testing on the dataset directly now and having issues there as well, so i'm not sure what could be going on that would allow me to incrementally refresh the flow, but not the dataset.  It's as if it can't load the data into the dataset but has no problem loading the flow.

It is a ton of data for sure, but I'm not sure how to get it published at this point.  I've tried loading all the data intially and it seems to change nothing?

[dungar · Accepted solution]

And of course the moment I posted this it finally clicked.  Folding issue with how i transformed the data.  My bad.  Had to make changes to the source table so that they weren't needed on the import.

[lbendlin]

Advanced incremental refresh and real-time data with the XMLA endpoint in Power BI - Power BI | Micr...