Forum Discussion
SAP BW connector batch sizing dataflow
Hi
I have a SAP query that downloads daily to a dataset but i cant set it up successfully in a dataflow.
Its large! and the dataflow always runs out of memory. i think the dataset is successful due to the batch sizing. 50K lines at a time.
Does anyone know if its possible to add this option to a dataflow?
Dan
Hi djwbaker ,
You can use incremental refresh with dataflows to resource consumption. Please refer to https://docs.microsoft.com/en-us/power-query/dataflows/incremental-refresh and if you have a premium capacity, you can set Max memory used for dataflow in the capacity : https://docs.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-premium-workload-configuration#refining-dataflow-settings
There is no option that can eanble batch sizing or you can talking about container Size, it is used for ETL processes. The mashup engine splits your query logic into these containers and they can process in parallel.
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Best Regards,
Dedmon Dai
1 Reply
Hi djwbaker ,
You can use incremental refresh with dataflows to resource consumption. Please refer to https://docs.microsoft.com/en-us/power-query/dataflows/incremental-refresh and if you have a premium capacity, you can set Max memory used for dataflow in the capacity : https://docs.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-premium-workload-configuration#refining-dataflow-settings
There is no option that can eanble batch sizing or you can talking about container Size, it is used for ETL processes. The mashup engine splits your query logic into these containers and they can process in parallel.
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Best Regards,
Dedmon Dai