Improve Databricks / Apache Spark Dataflow

[joshua1990]

Hi experts!

I'm looking for some guidelines, tips and tricks etc to improve a dataflow that is based on a Databricks / Spark connection.

Since this type of source does not support query folding, the refresh takes some time.

I already have reduced the number of steps to a minimum. In general, there are just simples filter for the time horizon etc. No transformations are done.

[v-yangliu-msft]

Hi  @joshua1990 ,

This is the related document, you can view this content:

https://medium.datadriveninvestor.com/my-dataset-is-taking-too-long-to-refresh-843fd8fd4a51

https://data-marc.com/2020/06/09/how-to-keep-your-power-bi-dataflows-organized-and-optimized/

https://docs.microsoft.com/en-us/power-bi/guidance/power-bi-optimization

Best Regards,

Liu Yang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[joshua1990]

No ideas?