Setup incremental refresh on Databricks source

[mhani]

Hi Community,

I recently created a database in Databrick, which I want to use in my Power BI Report.

One of the tables contains a huge amount of data, as to why I would like to implement Incremental refresh on the Dataset.

However, I struggle to get the incremental refresh working on this source.

I am using the method where I import the data, filter the data based on two parameters created (RangeStart & RangeEnd). I close and apply and then set up the Incremental refresh on the Desktop. Here I want to keep the last 5 years of data, and then update the most recent date daily. It doesn't seem like this is working, as the refresh time is around the same as when doing the full refresh.

Therefore, do any of you have some experience implementing incremental refresh using a Databricks source?

Michael

[mhani]

Hi Essjay,

I am not completely sure that it worked. However I can see that there are partitions in SSMS and it also use very low amount of compute. However it still take a lot of time. 

So the answer to you would be, I am not sure 🙂

[mhani]

Hi Ibendlin,

Thanks for your reply.

How do I see these partitions?

Michael

[lbendlin]

In SSMS or DAX Studio.

[mhani]

Thanks!

I found it 🙂

[lbendlin · Accepted solution]

There is a choreography to this. Every time you make meta data changes the incremental refresh will restart from scratch.

 It doesn't seem like this is working, as the refresh time is around the same as when doing the full refresh.

The ONLY way to know if it works or not is to check how many partitions the table has, and if these partitions are aggregated over time according to your rules.

And then there's this: Incremental refresh for semantic models and real-time data in Power BI - Power BI | Microsoft Learn

[Essjay]

@mhani : After checking the partitions via SSMS or DAX studio were you able to confirm if it worked for  you? I am in a similiar situation and although I see partitions , it doesn't seem to load data as required.

TIA!