Forum Discussion

mhani's avatar
mhani
Frequent Visitor
2 years ago
Solved

Setup incremental refresh on Databricks source

Hi Community,

 

I recently created a database in Databrick, which I want to use in my Power BI Report.

 

One of the tables contains a huge amount of data, as to why I would like to implement Incremental refresh on the Dataset.

 

However, I struggle to get the incremental refresh working on this source.

 

I am using the method where I import the data, filter the data based on two parameters created (RangeStart & RangeEnd). I close and apply and then set up the Incremental refresh on the Desktop. Here I want to keep the last 5 years of data, and then update the most recent date daily. It doesn't seem like this is working, as the refresh time is around the same as when doing the full refresh.

 

Therefore, do any of you have some experience implementing incremental refresh using a Databricks source?

 

Michael

Need Help
Tips and Tricks

6 Replies

  • lbendlin's avatar
    lbendlin
    Super User
    2 years ago

    There is a choreography to this. Every time you make meta data changes the incremental refresh will restart from scratch.

     

     It doesn't seem like this is working, as the refresh time is around the same as when doing the full refresh.

     

    The ONLY way to know if it works or not is to check how many partitions the table has, and if these partitions are aggregated over time according to your rules.

     

    And then there's this: Incremental refresh for semantic models and real-time data in Power BI - Power BI | Microsoft Learn

    • Essjay's avatar
      Essjay
      Helper I
      2 years ago

      mhani : After checking the partitions via SSMS or DAX studio were you able to confirm if it worked for  you? I am in a similiar situation and although I see partitions , it doesn't seem to load data as required.

       

       

      TIA!

  • mhani's avatar
    mhani
    Frequent Visitor
    2 years ago

    Hi Ibendlin,

     

    Thanks for your reply.

     

    How do I see these partitions?

     

    Michael

  • mhani's avatar
    mhani
    Frequent Visitor
    2 years ago

    Hi Essjay,

     

    I am not completely sure that it worked. However I can see that there are partitions in SSMS and it also use very low amount of compute. However it still take a lot of time. 

     

    So the answer to you would be, I am not sure 🙂