Forum Discussion

jigneshpatel's avatar
jigneshpatel
Regular Visitor
1 year ago
Solved

Dataflow from saleforce object to warehouse with incremental refresh not working

Hi,

 

I want to bring data from salesforce object to data warehouse in fabric workspace. I am using datalfow gen2 for this. 

 

When I set "Incremental refresh" on source table I get following error.

 

A query with incremental refresh is not folding.
Please make sure the query is folding, then re-publish the dataflow.

There is no additional implementation, just source is set to salesforce object and destination is set to warehouse existing table with replace option selected.

 

Is there any solution to this? or we can not use incremental refresh for salesforce objects.

 

also to note, when we use powerbi desktop to load data from salesforce object connector the incremental refreshworks but it does not work in the fabric dataflow.

 

Jignesh

 

 

  • Hi,

     

    To my knowledge incremental refresh for Salesforce in a Gen2 dataflow is not possible at this time in fabric.  It seems the incremental refresh is only possible with Power Bi desktop application.

12 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi jigneshpatel,

    As we haven’t heard back from you, so just following up to our previous message. I'd like to confirm if you've successfully resolved this issue or if you need further help?

    If yes, you are welcome to share your workaround and mark it as a solution so that other users can benefit as well. If you find a reply particularly helpful to you, you can also mark it as a solution.


    If you still have any questions or need more support, please feel free to let us know. We are more than happy to continue to help you.
    Thank you for your patience and look forward to hearing from you.

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi jigneshpatel,

    I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution so that other community members can find it easily.


    Thank you.

  • goldenarm253's avatar
    goldenarm253
    Frequent Visitor

    Hi,

     

    To my knowledge incremental refresh for Salesforce in a Gen2 dataflow is not possible at this time in fabric.  It seems the incremental refresh is only possible with Power Bi desktop application.

  • goldenarm253's avatar
    goldenarm253
    Frequent Visitor

    Hi,

    reference: Validating Findings on Optimizing Incremental Refresh with Microsoft Fabric 

     

    Last week, I conducted tests and observed that Salesforce objects are incompatible with Gen2 incremental refresh in Microsoft Fabric. If query folding isn't enabled, scheduled refresh times can triple. The issue stems from Salesforce using SOQL rather than standard SQL. A better approach might be setting up a pipeline to handle SOQL efficiently.

     

     

    I’m also attaching a link to my post. Nono Chen shared some best practice ideas in the discussion, which might provide additional insights.

    I hope this helps!

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi jigneshpatel,
    Thanks for reaching out to the Microsoft fabric community forum.

    The issue you're facing is related to query folding, which is essential for incremental refresh in Dataflow. Salesforce objects are sometimes problematic when it comes to query folding due to how the Salesforce connector in Power Query often does not support query folding due to API limitations. If the query is not folding, the dataflow cannot use incremental refresh. To check the Query Folding Capability ensure that the query from Salesforce supports query folding. In Power Query, right-click on any step in the query editor and check if the "View Native Query" option is available. If it is not, query folding is not happening.

    Here are some links which may be useful in understanding Query Folding :
    https://learn.microsoft.com/en-us/power-query/query-folding-examples

    https://learn.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-understand-optimize-refresh

    https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-incremental-refresh

     

    If this post helps, then please consider Accept it as the solution and give a Kudos to help the other members find it more quickly. If you have any other queries you can reach out to community forum for assistance.

    Thank you.

    • jigneshpatel's avatar
      jigneshpatel
      Regular Visitor

      HI,

       

      You mentioned that Salesforce connector in Power Query often does not support query folding due to API limitations

      But, on PowerBI desktop when I do same it supports query folding and also it supports incremental refresh. But in fabric dataflow it does not.

       

      jignesh

      • Anonymous's avatar
        Anonymous
        Not applicable

        Hi jigneshpatel,

         

        You are right. Power BI Desktop and Fabric Dataflow handle query folding differently, even when using the same Salesforce connector. This difference comes from how Power BI Desktop interacts with the connector versus how Dataflow processes queries in the cloud. For example PowerBi Desktop uses the Power Query engine locally while Fabric Dataflow runs transformations in the cloud, where query folding behavior may differ.

         

        If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

        Best Regards,
        Hammad.
        Community Support Team

         

        If this post helps then please mark it as a solution, so that other members find it more quickly.

        Thank you.

    • goldenarm253's avatar
      goldenarm253
      Frequent Visitor

      Thank you for your response.

       

      Salesforce does not allow query folding. After 'View data source query,' all Salesforce objects in the navigation are greyed out, as shown below in the screenshot.  This behavior has been duplicated in Gen2, Excel, and Power BI

       

      Gen2 dataflow

       

      PowerBI

       

      I am designing a data pipeline and testing refresh times with promising results.

       

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi jigneshpatel,

    May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

     

    Thank you.

    • Lu333's avatar
      Lu333
      New Member

      Hi

       

      I hope you are good. I have the same issue but my data source is SQL Server, how can I solve this?

  • Hi jigneshpatel,

     

    goldenarm253 is right - native incremental refresh for Salesforce in Fabric Dataflow Gen2 isn't supported because the Salesforce connector doesn't reliably fold the date-range predicates back into SOQL in the Gen2 runtime.

     

    For server-side filtering, Power BI Connector for Salesforce on AppExchange handles the filter configuration through a UI inside Salesforce:

     

    Why it might fit:

    Happy to set up a quick demo if it'd help. Just reach out via the support link.

     

    Cheers,

    Metrica Team.