Data Upload Issue

[JyotiBora]

 Hi Team,

I am trying to fetch data in Power BI desktop using data flows in import mode. It was working fine until we tried adding 1 more column. The dataflows are fine, giving the results but due to that column, data size got increased and now it's giving memory error in Power BI Desktop. I am not sure what should I do? Which memory this refers.

Any help is appreciated.

Thanks

[Balqis_Adan]

What is your Power BI License type?
Free
Pro
Premium Per User (PPU)
Microsoft Fabric / Premium Capacity
And what is your Computer RAM?
8GB
16GB
32GB
This will help give you the exact right solution for your specific situation.

please tell us to gain your the solution sin your probleem from RAM

[stoic-harsh]

Hey @JyotiBora,

As per your ss, Nielsen alone has ~16M rows. (Add a new column × 16M rows) × 5 tables loading simultaneously, and your RAM spikes during import. That's the culprit.

As an immediate fix, disable parallel loading: 

• Go to File > Options & Settings > Options > Current File > Data Load > Under "Parallel loading of tables", select One (disable parallel loading).

Consider incremental refresh, a long-term fix (RAM problem solved + Refresh time drops significantly):

• Add RangeStart / RangeEnd parameter on your date column.
• Define the incremental refresh policy in PBI Desktop.
• Publish to Service.

On DirectQuery, it will solve the memory issue too, but kills report performance at runtime. Not a great trade-off for 16M rows, I'd say (until you club incremental refresh and Direct Query ofc).

Hope this helps!

Best,

Harshit

[tharunkumarRTK]

@JyotiBora 

1. Create a M Query parameter.

2. Reduce the amount of data that you load in to power bi desktop using a conditional logic based on the parameter value.

3. Complete your development activities. 

4. Publish your report to power bi service.

5. Change the parameter value in power bi service. And supply necessary data source connection credentails or connection mapping.

6. Trigger an on demand data refresh.

7. Once the refresh is completed. If you want you can download the report again and perform any other activities in your file. 

Checkout this page for sample steps: Chris Webb's BI Blog: Limit The Amount Of Data You Work With In Power BI Desktop Using Parameters An... 

Connect on LinkedIn

Read my blogs here: TechieTips

Did I answer your question? Mark my post as a solution!
If I helped you, click on the Thumbs Up to give Kudos. Proud to be a Super User!

[krishnakanth240]

Hi @JyotiBora 

Error refering to Power BI Desktop RAM usage during import refresh process and not only dataflow itself. As issue started after adding new column to 16 million row table, column will increase the model size due to cardinality.

Try removing unnecessary columns, reduce data volume before loading, ensure correct datatypes,  and confirm usage of 64-bit Power BI Desktop. If the dataset continues to grow then consider DirectQuery

[danextian]

How cardinal is the new column for it to be causing a memory issue during refresh? When this happens to me, I'd normally just clear the data cache.

[JyotiBora]

I have tried that already. But it's not working