Microsoft Fabric - Dataflow Gen2 - Merge Queries

[jlakshminaraya]

Hello - 

Here are my observations when i use "Merge Queries" option in Microsoft Fabric Dataflow GEN2 with Innner join.

1. Dataflow getting failed

2. Query folding is not working with Merge Queries option.

3. Generally Daflow GEN2 refresh took at max 40 min before use "Merge Queries" option.

    But with "Merge Queries" option refresh took more than 4 hrs and it gets failed.

4. Data volume in each table is 13000 & 10000. This is very reasonable count.

Could you please help me with right solution to overcome the issue mentioned above (i.e., Best way to use Merge Queries option in Dataflow GEN2)

Thanks in advance.

Thanks,

Lakshmi Narayana

[v-cboorla-msft]

Hi @jlakshminaraya 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[v-cboorla-msft]

Hi @jlakshminaraya 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks.

[pqian_MSFT]

@jlakshminaraya if you found your merges aren't folding, the best way to deal with this is to change to the ELT pattern - create a dataflow with two STAGED entities representing the raw data to be merged, refresh it once. Then create a new dataflow and use the Dataflows connector to bring those two staged entities in - at this point they have compute from LH\SQL and you should be able to easily merge them with inner joins.

[miguel]

Hi Lakshmi,

It really depends. With the information provided I'm not sure that we can share that many guidelines. The underlying issue could be related to any of the following:

• Your data source is unable to fold the inner join and its subsequent expand operation
• The connector that you're using might not have the right capabilities to fold the full folding that you're trying to perform
• If you're using a gateway, perhaps try to upgrade to the latest version of the gateway
• There might be operations in your query which might be preventing the folding from happening

Below is a list of articles that might be of help in regards to understand how things can get evaluated for your query and how to try and optimize them:

• Understanding query evaluation and query folding in Power Query - Power Query | Microsoft Learn
• Query folding examples in Power Query - Power Query | Microsoft Learn
• Query folding indicators in Power Query - Power Query | Microsoft Learn
• Query plan - Power Query | Microsoft Learn
• Best practices when working with Power Query - Power Query | Microsoft Learn

If you can share some steps on how we could repro the scenario on our end, that would help us pinpoint exactly what could be happening. If perhaps you'd rather use your own dataflow, I'd recommend creating a support ticket so an engineer can take a closer look at this but, again, with the information provided it is not possible to reach a conclusion as to what might be the root cause of the issue.

[v-cboorla-msft]

Hi @jlakshminaraya 

Thanks for using Microsoft Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Appreciate your patience.

Thanks