Forum Discussion
Adding Large Dataflow Entities To Dataset
yes, deployment pipelines allow you to parameterize the date sources. It is preferable to have data sources that support query folding.
Another option for you would be to evaluate mixed storage mode with aggregations.
One more constraint that I forgot to mention - due to the size of the different fact tables, we're expecting to probably run into issues with incrementally refreshing a dataset that contains all of these tables as they would all be refreshed at the same time.
With dataflows, we're able to stagger each refresh throughout the day.
Incremental refresh is done by data source, not by dataset.
ok when I say Datasource I mean "Power Query Query resulting in a Power BI table". Surely you have separate queries for them, right?
Incremental refresh can be configure independently for each of these Power BI tables. Granted, they all have to share the RangeStart and RangeEnd parameters, so this may be a moot point.
lbendlinthanks for your assistance with this topic. Really appreciated
Based on our discussion so far, I take it that there's no way to get data from an extremely large Dataflows Entity directly into PBI Desktop?
I also just came across the deployment pipelines feature. I haven't used this feature before. Can I change the Dataflow that a dataset points to in the different stages (Dev, Test, and Prod)? If so, I can build a dataflow that uses a small subset of the original full dataflow and use that for Dev/Test, then just change the datasource to the full dataflow in Prod.
If all else fails, I'll take the dataset approach.