Power BI Dataflows and Salesforce Objects help needed.

incremental refresh does not require query folding. It's prefered but not required.

What kinds of error message are you getting?  what's the cardinality of your objects?

In our environment we use Informatica to pull the SalesForce object data into our enterprise data lake on a schedule (multiple times a day).  It is quite a bit more robust than trying to do this from Power BI.

Hey robarbie ,

As lbendlin  stated, you might be trying to push a rock uphill on this one and a third party tool might be the way to get the data more reliably. There are a number of third party tools that can extract the data in bulk, perhaps more efficiently, than can multiple data flows.  The issue here is the ability (and speed) with which you can get all of the data that you need that you eventually use within Power BI.  There are a number of tools (like Salesforce) that make exporting their data from the native tool difficult.

That said, with the limitations of number of concurrent queries the trick with Salesforce is to time out all of your dataflows.  You mentioned you keep them to 8 or so at a time.  By just the nature of the setup, you then have to have multiple dataflows that do NOT refresh at the same time as each other (since that would put you over 8 and closer to your maximum concurrent connections issue).  You have to be absolutely sure that the first dataflow with the 8 connections is totally done (and cleared from the Salesforce connection cache) before the next dataflow with connections begins.

I am not sure what incremental refresh issues that you might be having as you need to have a date field that you are using to do the incremental criteria.  If you are using a date field then you should be ok with the incremental.  BUT, again, you have the multiple simultaneous connections issue then that might be why your incremental is not working.

In the end, I think that the "secret" is to get the data out and I think that more often than not the best method is to get a full data dump on a regular schedule.  You might find a third party tool or create your own export using the built in Salesforce tools (like "Data Loader export").  

Hi robarbie,

On the 10-query concurrent limit: this is a Salesforce-side governor limit, so no connector removes it entirely. But you can reduce how close you get to it by pulling less data per query - server-side filtering (only the rows/columns each object actually needs) means lighter, faster queries that clear the connection pool quicker, so your staggering has more headroom.

Power BI Connector for Salesforce on AppExchange is one option that handles the filtering on the Salesforce side rather than relying on Power Query folding:

Why it might fit:

• Server-side filtering configured in Salesforce - lighter queries, less pressure on the concurrency limit
• Handles custom objects and large object sets
• No 2,000-row limit on reports (if you mix in report-based pulls)
• 30-day free trial
• Setup docs: https://metricasoftware.com/docs/salesforce/
• Responsive support if you get stuck: https://metricasoftware.com/docs/salesforce/contact-support/

Happy to set up a quick demo on your object setup. Just reach out via the support link.

Cheers,

Metrica Team.