Forum Discussion
Dataflow with incremental refresh doing a full table load x16 times
With the information given, the first place I'd be checking is to ensure query folding is occurring.
I assume query folding works with MySQL the same as MSSQL?
My understanding is, without query folding, the advantages of incremental refresh are redundant.
With the information given, the first place I'd be checking is to ensure query folding is occurring.
I assume query folding works with MySQL the same as MSSQL?
My understanding is, without query folding, the advantages of incremental refresh are redundant.
After messing around with it I think you're right, but I'm confident everything was folding eariler (before I put a keep-rows filter for testing). I'll see if it runs correctly now though
Keep an eye on the query folding icons.
I had issues this morning with a merge (Inner Join) not folding but Left Join does (at least that's what the icon suggested).
Also, another issue to watch out for, if using Comparer.OrdinalIgnoreCase, it will break the query folding. As well as many other things. 😆
Accepting this as the solution cause I believe this was ultimatly the issue. Having a new PipelineExpection: IO Timeout issue that's preventing me from verifying it though
