Forum Discussion
Dataflows and large datasets
I haven't used incremental refresh in datasets but have in dataflows. They operate separately--incrementally load data into your dataflow, and incrementally load data into your dataset.
For development and prototyping, I create a "sample" dataflow with 1,000 records from each entity in my source dataflow and use that in Power BI Desktop. When I'm ready, I switch to the full dataflow. Of course all the data still has to be downloaded to Power BI Desktop just to refresh and publish up to the cloud again. I agree this is inefficient and I hope this functionality is evolving within the product team.
I also separate my "ingest" dataflows from my "final" dataflows that are used in downstream dataflows/datasets. So you could insert this sample filter in the "final" dataflow, and then just remove it when you want the full entity loaded. This is nice because you can publish with the sample set, update the dataflow, and then refresh the dataset to pull in the full entity.
Apologies for the delay, but thanks a lot for the detailed response, that’s good to know.
On reflection, I think I might be able to use dataflows in conjunction with the REST API in order to achieve what I’m looking for, if a dataset is based on one or more dataflows.
This would be my suggested workflow for changes to a given dataset, using the API as much as possible:
- Save 2 copies of the dataset, one suffixed _bkp (for backup) and one suffixed _dev (development)
- Change the incremental load policy to 1 month (or whatever time period provides a small enough example to work with)
- Download the _dev dataset
- Refresh the data from the flow, so that it only covers the limited time period
- Make all required changes
- Upload the updated _dev dataset
- Reset the incremental policy of the dataflow to normal
- Refresh dataflow
- If changes all appear correctly, copy _dev over the production dataset
- Delete _dev, and archive _bkp
If there are problems, then additional steps would of course be necessary to address them.
I would be happy to implement all this - however, given the fact that the new deployment pipeline is imminent, I’m not sure it would be worth the effort.