Forum Discussion

Bokazoit's avatar
Bokazoit
Continued Contributor
1 year ago

Handle "large" amount of data using API and incremental refresh

I write "large" because the intial load is 3.5 mio rows, and that is imho not large. But my problem is this:   It is Business Central and I fetch data using API.   I would like to use incremental...
Need Help
Bokazoit's avatar
Bokazoit
Continued Contributor
1 year ago

So what You are saying is that the limit is in the engine, because I have a working load of the dataset using my current data warehouse in a Pro enviroment, but switching to Power BI connection is a bad idea?

v-karpurapud's avatar
v-karpurapud
Community Support
1 year ago

Hi,

Connecting directly to Business Central is not necessarily a bad approach, but it behaves differently than using a data warehouse. Business Central uses OData APIs, which aren’t designed for large analytical workloads. Even with query folding, API refreshes can be slower and use more resources, often hitting Pro workspace limits. If you want to stay within Pro, it’s more reliable to stage Business Central data in your data warehouse or Azure SQL/Data Lake and load from there. Alternatively, you can move the dataset to a Premium Per User or Premium capacity workspace for better resources and more consistent incremental refreshes.

Regards,
Microsoft Fabric Community Support Team.

  • Bokazoit's avatar
    Bokazoit
    Continued Contributor
    1 year ago

    I will try that PPU suggestion