topic Re: Dataflow from saleforce object to warehouse with incremental refresh not working in Data Engineering

Dataflow from saleforce object to warehouse with incremental refresh not working

jigneshpatel — Thu, 02 Jan 2025 10:26:56 GMT

Hi,

I want to bring data from salesforce object to data warehouse in fabric workspace. I am using datalfow gen2 for this.

When I set "Incremental refresh" on source table I get following error.

A query with incremental refresh is not folding.
Please make sure the query is folding, then re-publish the dataflow.

There is no additional implementation, just source is set to salesforce object and destination is set to warehouse existing table with replace option selected.

Is there any solution to this? or we can not use incremental refresh for salesforce objects.

also to note, when we use powerbi desktop to load data from salesforce object connector the incremental refreshworks but it does not work in the fabric dataflow.

Jignesh

Re: Dataflow from saleforce object to warehouse with incremental refresh not working

goldenarm253 — Thu, 02 Jan 2025 11:25:10 GMT

Hi,

reference: Validating Findings on Optimizing Incremental Refresh with Microsoft Fabric

Last week, I conducted tests and observed that Salesforce objects are incompatible with Gen2 incremental refresh in Microsoft Fabric. If query folding isn't enabled, scheduled refresh times can triple. The issue stems from Salesforce using SOQL rather than standard SQL. A better approach might be setting up a pipeline to handle SOQL efficiently.

I’m also attaching a link to my post. Nono Chen shared some best practice ideas in the discussion, which might provide additional insights.

I hope this helps!

Re: Dataflow from saleforce object to warehouse with incremental refresh not working

Anonymous — Mon, 06 Jan 2025 04:35:48 GMT

Hi @jigneshpatel,
Thanks for reaching out to the Microsoft fabric community forum.

The issue you're facing is related to query folding, which is essential for incremental refresh in Dataflow. Salesforce objects are sometimes problematic when it comes to query folding due to how the Salesforce connector in Power Query often does not support query folding due to API limitations. If the query is not folding, the dataflow cannot use incremental refresh. To check the Query Folding Capability ensure that the query from Salesforce supports query folding. In Power Query, right-click on any step in the query editor and check if the "View Native Query" option is available. If it is not, query folding is not happening.

Here are some links which may be useful in understanding Query Folding :
https://learn.microsoft.com/en-us/power-query/query-folding-examples

https://learn.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-understand-optimize-refresh

https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-incremental-refresh

If this post helps, then please consider Accept it as the solution and give a Kudos to help the other members find it more quickly. If you have any other queries you can reach out to community forum for assistance.

Thank you.

Re: Dataflow from saleforce object to warehouse with incremental refresh not working

goldenarm253 — Mon, 06 Jan 2025 12:12:58 GMT

Thank you for your response.

Salesforce does not allow query folding. After 'View data source query,' all Salesforce objects in the navigation are greyed out, as shown below in the screenshot. This behavior has been duplicated in Gen2, Excel, and Power BI

Gen2 dataflow

PowerBI

I am designing a data pipeline and testing refresh times with promising results.

Re: Dataflow from saleforce object to warehouse with incremental refresh not working

jigneshpatel — Fri, 10 Jan 2025 11:04:22 GMT

HI,

You mentioned that Salesforce connector in Power Query often does not support query folding due to API limitations

But, on PowerBI desktop when I do same it supports query folding and also it supports incremental refresh. But in fabric dataflow it does not.

jignesh

Re: Dataflow from saleforce object to warehouse with incremental refresh not working

goldenarm253 — Wed, 15 Jan 2025 13:26:38 GMT

Apologies for my delayed response, and thank you for taking the time to reply.

I exclusively use the Fabric backend for all reports, as it provides improved governance for data management.

Re: Dataflow from saleforce object to warehouse with incremental refresh not working

Anonymous — Wed, 29 Jan 2025 09:54:22 GMT

Hi @jigneshpatel,

You are right. Power BI Desktop and Fabric Dataflow handle query folding differently, even when using the same Salesforce connector. This difference comes from how Power BI Desktop interacts with the connector versus how Dataflow processes queries in the cloud. For example PowerBi Desktop uses the Power Query engine locally while Fabric Dataflow runs transformations in the cloud, where query folding behavior may differ.

If I misunderstand your needs or you still have problems on it, please feel free to let us know.

Best Regards,
Hammad.
Community Support Team

If this post helps then please mark it as a solution, so that other members find it more quickly.

Thank you.

Re: Dataflow from saleforce object to warehouse with incremental refresh not working

Anonymous — Mon, 03 Feb 2025 06:13:29 GMT

Hi @jigneshpatel,

As we haven’t heard back from you, so just following up to our previous message. I'd like to confirm if you've successfully resolved this issue or if you need further help?

If yes, you are welcome to share your workaround and mark it as a solution so that other users can benefit as well. If you find a reply particularly helpful to you, you can also mark it as a solution.

If you still have any questions or need more support, please feel free to let us know. We are more than happy to continue to help you.
Thank you for your patience and look forward to hearing from you.

Re: Dataflow from saleforce object to warehouse with incremental refresh not working

Anonymous — Mon, 10 Feb 2025 04:59:52 GMT

Hi @jigneshpatel,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution so that other community members can find it easily.

Thank you.

Re: Dataflow from saleforce object to warehouse with incremental refresh not working

goldenarm253 — Mon, 10 Feb 2025 13:25:09 GMT

Hi,

To my knowledge incremental refresh for Salesforce in a Gen2 dataflow is not possible at this time in fabric. It seems the incremental refresh is only possible with Power Bi desktop application.

Re: Dataflow from saleforce object to warehouse with incremental refresh not working

Anonymous — Fri, 14 Feb 2025 10:56:10 GMT

Hi @jigneshpatel,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.