Forum Discussion
Adding Large Dataflow Entities To Dataset
yes, deployment pipelines allow you to parameterize the date sources. It is preferable to have data sources that support query folding.
Another option for you would be to evaluate mixed storage mode with aggregations.
two notes:
1. Have you tried to access the dataflow via direct query? This is now possible, but I don't know if it helps in your scenario
2. Any chance of normalizing the source data / splitting it into separate dataflows ?
I believe the 10 GB limit is either already lifted or about to be lifted. The idea is that a dataset will use up the available memory of a SKU and then swap data in and out. That allows working with datasets that are larger than half the memory size of the SKU (people tend to forget that each dataset always occupies twice the RAM)
lbendlinthanks for the reply.
I've tried DirectQuery preview feature. Unfortunately, it keeps timing out at the visualization level.
The 2.4 billion rows is the normalised data. I've split out the historical data into separate tables, so this is only the most recent month's data.
The news about the dataset restrictions being lifted is great to hear! If I cannot get the dataflow to work, I will look at this option. The big downside of using a dataset is the moment there's a new fact table or dimension, we have to add it in Desktop, then reupload to Service. If this happens well after the initial dataset upload, then all the fact tables will need to be manually updated (we split out the fact tables into history (which remains static) and current (which can be refreshed incrementally). This enables us to load all the data without the server timing out).
You may want to look at ALM Toolkit which will allow you to make structural changes (within reason) without requiring a full data reload. (Discaimer: This requires XMLA read/write to be enabled on your SKU. May be incompatible with other preview options)
One more constraint that I forgot to mention - due to the size of the different fact tables, we're expecting to probably run into issues with incrementally refreshing a dataset that contains all of these tables as they would all be refreshed at the same time.
With dataflows, we're able to stagger each refresh throughout the day.
Incremental refresh is done by data source, not by dataset.
