Regarding Incremental Refresh in Power BI with databricks as source

mkravi ,My Initial thoughts

Can you check if you can do your incremental setting before adding a conditional column? After source change datatype to datetime and do incremental setting.

Hi Amit 

Thanks for your quickest response, appreciate the same.

even when i delete the additional column then also the view native query shows disabled 

The column on which i have to apply  the incremental custom filter is already datetime 

Do you think databricks will support query folding as so far i didn't see. 

And what is your opinion in doing incremental refresh even when the query folding is disabled .

Hey I am having the same issue - have you resloved it and how? Thanks.

HI Tiff92 ,mkravi ,amitchandak
I am also facing similar issue any one resolved this..?

Regards

I'm having the same issues here. It seems as if Power BI is not able to perform query folding any other than simple  renaming or casting and thus is not able to use the RangeStart and/or RangeEnd parameters in the query that is send out to the source system. In our case this makes incremental refresh completely useless, as refreshing seems only to take longer since it has several partitions and fires multiple exactly the same queries on the source system, which are subsequently handled by the Power BI engine and put into partitions.

I managed to get it to work! I came across this post while looking for a solution so I hope this helps someone.

My solution was to use Databricks.Query instead of Databricks.Catalogs to load the table and write SQL statements to filter the table on the M code using Advanced Editor.

let
    Source = Databricks.Query(ServerHostName, HTTPPath, [Catalog=null, Database=null, 
    EnableAutomaticProxyDiscovery=null]),
    Table = "SELECT * FROM hive_metastore.bl.testtable",
    FilterRangeStart = " WHERE LastChanged >= timestamp '" &  DateTime.ToText(RangeStart, "yyyy- 
    MM-ddTHH:mm:ss") & "'", 
    FilterRangeEnd = " AND LastChanged < timestamp '" &  DateTime.ToText(RangeEnd, "yyyy-MM- 
    ddTHH:mm:ss") & "'", 
    Navigation = Source(Table & FilterRangeStart & FilterRangeEnd)
in
    Navigation

The parameters RangeStart and RangeEnd were not recognised as a timestamp so I had to specify they were a timestamp and convert them to text format in the exact same way as a timestamp.
 

I was also able to get incremental refresh working with Azure Databricks.  Only I have a custom sql query that uses RangeStart and RangeEnd.  Here is the where clause of the custom query...
where date_created >= try_to_timestamp('"& Text.From(RangeStart) &"', 'M/d/yyyy h:mm:ss a')
and date_created < try_to_timestamp('"& Text.From(RangeEnd) &"', 'M/d/yyyy h:mm:ss a')