dynamics CRM Performance issue When should we upscale BI architecture

Connecting Power Bi to Dynamics CRM on the cloud we are experiencing download that takes very long time

The connectivity and network issue have been checked and we can uplaod donwlod lage files form internet

I woudl search for some experience or feed-back on when we should look for architecture with dawarehouse server beteen Dynamics CRM and Power BI?

What could we take as high level assomption of volume and complexity that could resonably manage with direct connectin only

If we look at number of rows, colums, entities, etc. what would be the bounderies to consider scalble architecture?