Forum Discussion
Dataflow with incremental refresh doing a full table load x16 times
I'm working on adding incremental refresh to a fact table in my dataflow, unfortunately when this dataflow does it's next refresh I see 16 processes on my mysql instance that are all performing the same query (full table load). This fact table has over 100m rows and takes 3 hours to refresh without this feature and pipeline is failing to handle loading this table 16 times.
I feel like the problem is that these queries are supposed to have time contraints to break the data being loaded into 16 parts, but those parameters aren't making it in.
Anyone have any thoughts on how to deal with this?
With the information given, the first place I'd be checking is to ensure query folding is occurring.
I assume query folding works with MySQL the same as MSSQL?
My understanding is, without query folding, the advantages of incremental refresh are redundant.
4 Replies
With the information given, the first place I'd be checking is to ensure query folding is occurring.
I assume query folding works with MySQL the same as MSSQL?
My understanding is, without query folding, the advantages of incremental refresh are redundant.
After messing around with it I think you're right, but I'm confident everything was folding eariler (before I put a keep-rows filter for testing). I'll see if it runs correctly now though
Keep an eye on the query folding icons.
I had issues this morning with a merge (Inner Join) not folding but Left Join does (at least that's what the icon suggested).
Also, another issue to watch out for, if using Comparer.OrdinalIgnoreCase, it will break the query folding. As well as many other things. 😆
Accepting this as the solution cause I believe this was ultimatly the issue. Having a new PipelineExpection: IO Timeout issue that's preventing me from verifying it though
