Forum Discussion
Adding Large Dataflow Entities To Dataset
yes, deployment pipelines allow you to parameterize the date sources. It is preferable to have data sources that support query folding.
Another option for you would be to evaluate mixed storage mode with aggregations.
lbendlinthanks for the reply.
I've tried DirectQuery preview feature. Unfortunately, it keeps timing out at the visualization level.
The 2.4 billion rows is the normalised data. I've split out the historical data into separate tables, so this is only the most recent month's data.
The news about the dataset restrictions being lifted is great to hear! If I cannot get the dataflow to work, I will look at this option. The big downside of using a dataset is the moment there's a new fact table or dimension, we have to add it in Desktop, then reupload to Service. If this happens well after the initial dataset upload, then all the fact tables will need to be manually updated (we split out the fact tables into history (which remains static) and current (which can be refreshed incrementally). This enables us to load all the data without the server timing out).
Super UserYou may want to look at ALM Toolkit which will allow you to make structural changes (within reason) without requiring a full data reload. (Discaimer: This requires XMLA read/write to be enabled on your SKU. May be incompatible with other preview options)