Incremental loading from MySQL database to fabric Lakehouse using Dataflow

[ajomat]

Hello everyone,

I have a query about performing an incremental loading from MySQL database to Fabric Lakehouse, but it seems like my queries are not foldable. I read from some other internet sources that without foldable queries incremental refresh is not possible and also query folding is not supported in MySQL. How can I get around this? Is there any alternative solution ? Thanks in advance...

[audreygerred]

Hello! What steps do you have applied in Power Query? Assuming your source supports query folding, it is likely that one of the steps you applied broke the query folding. You can try to right click on the steps in Power Query and see if View Native Query is greyed out. If it is greyed out on the first step (Source) then this trick won't work to identify which step is breaking your query folding. If it is not greyed out, keep checking each step - when you get to a step the is greyed out this is the step that broke your query folding. You will need to remove it.

[ajomat]

Hi,

Thank you for the response.
It seems like my queries are not foldable at the source.

I followed the same step in SQL server database and the query was folding at source (the source was not greyed out and was marked as green - Screenshot below). It worked there.

But when I used MySQL database the source is greyed out and the incremental refresh is not working anymore. This is what I could see in the window (screenshot below).

[Anonymous]

Hi @audreygerred ,thanks for the quick reply, I'll add further。

Hi @ajomat ,

Regarding your question, the documentation does not state that Mysql supports query folding. If query folding is missing, then incremental refresh loses its meaning. There is no specific solution for the moment, you can only move the data to a data source that supports query folding. For example, SQL Server.

Query folding on native queries - Power Query | Microsoft Learn

[ajomat]

Thank you @Anonymous for the response. 
Yes, it seems like there is no other solution at the moment. I hope Microsft will rectify this in the next update.